X-Git-Url: https://git.tokkee.org/?a=blobdiff_plain;ds=inline;f=Documentation%2Fgit-fast-import.txt;h=d5119678b59492db4651006391fd693a1a12601a;hb=afc05f9f13beded8caf15d8e58d06fd64e0f7808;hp=939ec4652bea90888006adf7fd6860e43c5439ed;hpb=b578e509d3d346513efda14606886a55174e5181;p=git.git diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt index 939ec4652..d5119678b 100644 --- a/Documentation/git-fast-import.txt +++ b/Documentation/git-fast-import.txt @@ -3,7 +3,7 @@ git-fast-import(1) NAME ---- -git-fast-import - Backend for fast Git data importers. +git-fast-import - Backend for fast Git data importers SYNOPSIS @@ -62,7 +62,18 @@ OPTIONS Dumps the internal marks table to when complete. Marks are written one per line as `:markid SHA-1`. Frontends can use this file to validate imports after they - have been completed. + have been completed, or to save the marks table across + incremental runs. As is only opened and truncated + at checkpoint (or completion) the same path can also be + safely given to \--import-marks. + +--import-marks=:: + Before processing any input, load the marks specified in + . The input file must exist, must be readable, and + must use the same format as produced by \--export-marks. + Multiple options may be supplied to import more than one + set of marks. If a mark is defined to different values, + the last file wins. --export-pack-edges=:: After creating a packfile, print a line of data to @@ -165,6 +176,15 @@ results, such as branch names or file names with leading or trailing spaces in their name, or early termination of fast-import when it encounters unexpected input. +Stream Comments +~~~~~~~~~~~~~~~ +To aid in debugging frontends fast-import ignores any line that +begins with `#` (ASCII pound/hash) up to and including the line +ending `LF`. A comment line may contain any sequence of bytes +that does not contain an LF and therefore may be used to include +any detailed debugging information that might be specific to the +frontend and useful when inspecting a fast-import data stream. + Date Formats ~~~~~~~~~~~~ The following date formats are supported. A frontend should select @@ -221,7 +241,7 @@ been well tested in the wild. + Frontends should prefer the `raw` format if the source material already uses UNIX-epoch format, can be coaxed to give dates in that -format, or its format is easiliy convertible to it, as there is no +format, or its format is easily convertible to it, as there is no ambiguity in parsing. `now`:: @@ -278,6 +298,11 @@ and control the current import process. More detailed discussion This command is optional and is not needed to perform an import. +`progress`:: + Causes fast-import to echo the entire line to its own + standard output. This command is optional and is not needed + to perform an import. + `commit` ~~~~~~~~ Create or update a branch with a new commit, recording one logical @@ -291,8 +316,8 @@ change to the project. data ('from' SP LF)? ('merge' SP LF)? - (filemodify | filedelete | filedeleteall)* - LF + (filemodify | filedelete | filecopy | filerename | filedeleteall)* + LF? .... where `` is the name of the branch to make the commit on. @@ -314,13 +339,17 @@ commit message use a 0 length data. Commit messages are free-form and are not interpreted by Git. Currently they must be encoded in UTF-8, as fast-import does not permit other encodings to be specified. -Zero or more `filemodify`, `filedelete` and `filedeleteall` commands +Zero or more `filemodify`, `filedelete`, `filecopy`, `filerename` +and `filedeleteall` commands may be included to update the contents of the branch prior to creating the commit. These commands may be supplied in any order. -However it is recommended that a `filedeleteall` command preceed -all `filemodify` commands in the same commit, as `filedeleteall` +However it is recommended that a `filedeleteall` command precede +all `filemodify`, `filecopy` and `filerename` commands in the same +commit, as `filedeleteall` wipes the branch clean (see below). +The `LF` after the command is optional (it used to be required). + `author` ^^^^^^^^ An `author` command may optionally appear, if the author information @@ -349,17 +378,16 @@ their syntax. `from` ^^^^^^ -Only valid for the first commit made on this branch by this -fast-import process. The `from` command is used to specify the commit -to initialize this branch from. This revision will be the first -ancestor of the new commit. - -Omitting the `from` command in the first commit of a new branch will -cause fast-import to create that commit with no ancestor. This tends to be -desired only for the initial commit of a project. Omitting the -`from` command on existing branches is required, as the current -commit on that branch is automatically assumed to be the first -ancestor of the new commit. +The `from` command is used to specify the commit to initialize +this branch from. This revision will be the first ancestor of the +new commit. + +Omitting the `from` command in the first commit of a new branch +will cause fast-import to create that commit with no ancestor. This +tends to be desired only for the initial commit of a project. +Omitting the `from` command on existing branches is usually desired, +as the current commit on that branch is automatically assumed to +be the first ancestor of the new commit. As `LF` is not valid in a Git refname or SHA-1 expression, no quoting or escaping syntax is supported within ``. @@ -374,7 +402,7 @@ Here `` is any of the following: + The reason fast-import uses `:` to denote a mark reference is this character is not legal in a Git branch name. The leading `:` makes it easy -to distingush between the mark 42 (`:42`) and the branch 42 (`42` +to distinguish between the mark 42 (`:42`) and the branch 42 (`42` or `refs/heads/42`), or an abbreviated SHA-1 which happened to consist only of base-10 digits. + @@ -452,18 +480,18 @@ in octal. Git only supports the following modes: In both formats `` is the complete path of the file to be added (if not already existing) or modified (if already existing). -A `` string must use UNIX-style directory seperators (forward +A `` string must use UNIX-style directory separators (forward slash `/`), may contain any byte other than `LF`, and must not start with double quote (`"`). If an `LF` or double quote must be encoded into `` shell-style quoting should be used, e.g. `"path/with\n and \" in it"`. -The value of `` must be in canoncial form. That is it must not: +The value of `` must be in canonical form. That is it must not: * contain an empty directory component (e.g. `foo//bar` is invalid), -* end with a directory seperator (e.g. `foo/` is invalid), -* start with a directory seperator (e.g. `/foo` is invalid), +* end with a directory separator (e.g. `foo/` is invalid), +* start with a directory separator (e.g. `/foo` is invalid), * contain the special component `.` or `..` (e.g. `foo/./bar` and `foo/../bar` are invalid). @@ -471,8 +499,9 @@ It is recommended that `` always be encoded using UTF-8. `filedelete` ^^^^^^^^^^^^ -Included in a `commit` command to remove a file from the branch. -If the file removal makes its directory empty, the directory will +Included in a `commit` command to remove a file or recursively +delete an entire directory from the branch. If the file or directory +removal makes its parent directory empty, the parent directory will be automatically removed too. This cascades up the tree until the first non-empty directory or the root is reached. @@ -480,9 +509,60 @@ first non-empty directory or the root is reached. 'D' SP LF .... -here `` is the complete path of the file to be removed. +here `` is the complete path of the file or subdirectory to +be removed from the branch. See `filemodify` above for a detailed description of ``. +`filecopy` +^^^^^^^^^^^^ +Recursively copies an existing file or subdirectory to a different +location within the branch. The existing file or directory must +exist. If the destination exists it will be completely replaced +by the content copied from the source. + +.... + 'C' SP SP LF +.... + +here the first `` is the source location and the second +`` is the destination. See `filemodify` above for a detailed +description of what `` may look like. To use a source path +that contains SP the path must be quoted. + +A `filecopy` command takes effect immediately. Once the source +location has been copied to the destination any future commands +applied to the source location will not impact the destination of +the copy. + +`filerename` +^^^^^^^^^^^^ +Renames an existing file or subdirectory to a different location +within the branch. The existing file or directory must exist. If +the destination exists it will be replaced by the source directory. + +.... + 'R' SP SP LF +.... + +here the first `` is the source location and the second +`` is the destination. See `filemodify` above for a detailed +description of what `` may look like. To use a source path +that contains SP the path must be quoted. + +A `filerename` command takes effect immediately. Once the source +location has been renamed to the destination any future commands +applied to the source location will create new files there and not +impact the destination of the rename. + +Note that a `filerename` is the same as a `filecopy` followed by a +`filedelete` of the source location. There is a slight performance +advantage to using `filerename`, but the advantage is so small +that it is never worth trying to convert a delete/add pair in +source material into a rename for fast-import. This `filerename` +command is provided just to simplify frontends that already have +rename information and don't want bother with decomposing it into a +`filecopy` followed by a `filedelete`. + `filedeleteall` ^^^^^^^^^^^^^^^ Included in a `commit` command to remove all files (and also all @@ -538,7 +618,6 @@ lightweight (non-annotated) tags see the `reset` command below. 'from' SP LF 'tagger' SP SP LT GT SP LF data - LF .... where `` is the name of the tag to create. @@ -582,12 +661,14 @@ branch from an existing commit without creating a new commit. .... 'reset' SP LF ('from' SP LF)? - LF + LF? .... For a detailed description of `` and `` see above under `commit` and `from`. +The `LF` after the command is optional (it used to be required). + The `reset` command can also be used to create lightweight (non-annotated) tags. For example: @@ -626,29 +707,40 @@ intended for production-quality conversions should always use the exact byte count format, as it is more robust and performs better. The delimited format is intended primarily for testing fast-import. +Comment lines appearing within the `` part of `data` commands +are always taken to be part of the body of the data and are therefore +never ignored by fast-import. This makes it safe to import any +file/message content whose lines might start with `#`. + Exact byte count format:: The frontend must specify the number of bytes of data. + .... 'data' SP LF - LF + LF? .... + where `` is the exact number of bytes appearing within ``. The value of `` is expressed as an ASCII decimal integer. The `LF` on either side of `` is not included in `` and will not be included in the imported data. ++ +The `LF` after `` is optional (it used to be required) but +recommended. Always including it makes debugging a fast-import +stream easier as the next command always starts in column 0 +of the next line, even if `` did not end with an `LF`. Delimited format:: A delimiter string is used to mark the end of the data. fast-import will compute the length by searching for the delimiter. - This format is primarly useful for testing and is not + This format is primarily useful for testing and is not recommended for real data. + .... 'data' SP '<<' LF LF LF + LF? .... + where `` is the chosen delimiter string. The string `` @@ -657,6 +749,8 @@ fast-import will think the data ends earlier than it really does. The `LF` immediately trailing `` is part of ``. This is one of the limitations of the delimited format, it is impossible to supply a data chunk which does not have an LF as its last byte. ++ +The `LF` after ` LF` is optional (it used to be required). `checkpoint` ~~~~~~~~~~~~ @@ -665,7 +759,7 @@ save out all current branch refs, tags and marks. .... 'checkpoint' LF - LF + LF? .... Note that fast-import automatically switches packfiles when the current @@ -684,6 +778,32 @@ process access to a branch. However given that a 30 GiB Subversion repository can be loaded into Git through fast-import in about 3 hours, explicit checkpointing may not be necessary. +The `LF` after the command is optional (it used to be required). + +`progress` +~~~~~~~~~~ +Causes fast-import to print the entire `progress` line unmodified to +its standard output channel (file descriptor 1) when the command is +processed from the input stream. The command otherwise has no impact +on the current import, or on any of fast-import's internal state. + +.... + 'progress' SP LF + LF? +.... + +The `` part of the command may contain any sequence of bytes +that does not contain `LF`. The `LF` after the command is optional. +Callers may wish to process the output through a tool such as sed to +remove the leading part of the line, for example: + +==== + frontend | git-fast-import | sed 's/^progress //' +==== + +Placing a `progress` command immediately after a `checkpoint` will +inform the reader when the `checkpoint` has been completed and it +can safely access the refs that fast-import updated. Tips and Tricks --------------- @@ -753,7 +873,7 @@ to remove the dummy branch. Import Now, Repack Later ~~~~~~~~~~~~~~~~~~~~~~~~ As soon as fast-import completes the Git repository is completely valid -and ready for use. Typicallly this takes only a very short time, +and ready for use. Typically this takes only a very short time, even for considerably large projects (100,000+ commits). However repacking the repository is necessary to improve data @@ -777,6 +897,15 @@ This will take longer, but will also produce a smaller packfile. You only need to expend the effort once, and everyone using your project will benefit from the smaller repository. +Include Some Progress Messages +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Every once in a while have your frontend emit a `progress` message +to fast-import. The contents of the messages are entirely free-form, +so one suggestion would be to output the current month and year +each time the current commit date moves into the next month. +Your users will feel better knowing how much of the data stream +has been processed. + Packfile Optimization --------------------- @@ -813,8 +942,8 @@ Memory Utilization ------------------ There are a number of factors which affect how much memory fast-import requires to perform an import. Like critical sections of core -Git, fast-import uses its own memory allocators to ammortize any overheads -associated with malloc. In practice fast-import tends to ammoritize any +Git, fast-import uses its own memory allocators to amortize any overheads +associated with malloc. In practice fast-import tends to amortize any malloc overheads to 0, due to its use of large block allocations. per object @@ -871,7 +1000,7 @@ per active tree ~~~~~~~~~~~~~~~ Trees (aka directories) use just 12 bytes of memory on top of the memory required for their entries (see ``per active file'' below). -The cost of a tree is virtually 0, as its overhead ammortizes out +The cost of a tree is virtually 0, as its overhead amortizes out over the individual file entries. per active file entry @@ -899,4 +1028,3 @@ Documentation by Shawn O. Pearce . GIT --- Part of the gitlink:git[7] suite -