![[tokkee]](http://tokkee.org/images/avatar.png){kind=avatar}
index a92ae6d1a36f800256a477f23c8280e2717b13c9..c29a4f8126442e06873eab18c96a093c6bd9f373 100644 (file)
with the newly imported data.
The fast-import backend itself can import into an empty repository (one that
with the newly imported data.
The fast-import backend itself can import into an empty repository (one that
-has already been initialized by gitlink:git-init[1]) or incrementally
+has already been initialized by linkgit:git-init[1]) or incrementally
update an existing populated repository. Whether or not incremental
imports are supported from a particular foreign source depends on
the frontend program in use.
update an existing populated repository. Whether or not incremental
imports are supported from a particular foreign source depends on
the frontend program in use.
This information may be useful after importing projects
whose total object set exceeds the 4 GiB packfile limit,
as these commits can be used as edge points during calls
This information may be useful after importing projects
whose total object set exceeds the 4 GiB packfile limit,
as these commits can be used as edge points during calls
- to gitlink:git-pack-objects[1].
+ to linkgit:git-pack-objects[1].
--quiet::
Disable all non-fatal output, making fast-import silent when it
--quiet::
Disable all non-fatal output, making fast-import silent when it
+
An example value is ``Tue Feb 6 11:22:18 2007 -0500''. The Git
parser is accurate, but a little on the lenient side. It is the
+
An example value is ``Tue Feb 6 11:22:18 2007 -0500''. The Git
parser is accurate, but a little on the lenient side. It is the
-same parser used by gitlink:git-am[1] when applying patches
+same parser used by linkgit:git-am[1] when applying patches
received from email.
+
Some malformed strings may be accepted as valid dates. In some of
received from email.
+
Some malformed strings may be accepted as valid dates. In some of
+
Frontends should prefer the `raw` format if the source material
already uses UNIX-epoch format, can be coaxed to give dates in that
+
Frontends should prefer the `raw` format if the source material
already uses UNIX-epoch format, can be coaxed to give dates in that
-format, or its format is easiliy convertible to it, as there is no
+format, or its format is easily convertible to it, as there is no
ambiguity in parsing.
`now`::
ambiguity in parsing.
`now`::
This particular format is supplied as its short to implement and
may be useful to a process that wants to create a new commit
right now, without needing to use a working directory or
This particular format is supplied as its short to implement and
may be useful to a process that wants to create a new commit
right now, without needing to use a working directory or
-gitlink:git-update-index[1].
+linkgit:git-update-index[1].
+
If separate `author` and `committer` commands are used in a `commit`
the timestamps may not match, as the system clock will be polled
+
If separate `author` and `committer` commands are used in a `commit`
the timestamps may not match, as the system clock will be polled
This command is optional and is not needed to perform
an import.
This command is optional and is not needed to perform
an import.
+`progress`::
+ Causes fast-import to echo the entire line to its own
+ standard output. This command is optional and is not needed
+ to perform an import.
+
`commit`
~~~~~~~~
Create or update a branch with a new commit, recording one logical
`commit`
~~~~~~~~
Create or update a branch with a new commit, recording one logical
('from' SP <committish> LF)?
('merge' SP <committish> LF)?
(filemodify | filedelete | filecopy | filerename | filedeleteall)*
('from' SP <committish> LF)?
('merge' SP <committish> LF)?
(filemodify | filedelete | filecopy | filerename | filedeleteall)*
- LF
+ LF?
....
where `<ref>` is the name of the branch to make the commit on.
....
where `<ref>` is the name of the branch to make the commit on.
and `filedeleteall` commands
may be included to update the contents of the branch prior to
creating the commit. These commands may be supplied in any order.
and `filedeleteall` commands
may be included to update the contents of the branch prior to
creating the commit. These commands may be supplied in any order.
-However it is recommended that a `filedeleteall` command preceed
+However it is recommended that a `filedeleteall` command precede
all `filemodify`, `filecopy` and `filerename` commands in the same
commit, as `filedeleteall`
wipes the branch clean (see below).
all `filemodify`, `filecopy` and `filerename` commands in the same
commit, as `filedeleteall`
wipes the branch clean (see below).
+The `LF` after the command is optional (it used to be required).
+
`author`
^^^^^^^^
An `author` command may optionally appear, if the author information
`author`
^^^^^^^^
An `author` command may optionally appear, if the author information
Omitting the `from` command in the first commit of a new branch
will cause fast-import to create that commit with no ancestor. This
tends to be desired only for the initial commit of a project.
Omitting the `from` command in the first commit of a new branch
will cause fast-import to create that commit with no ancestor. This
tends to be desired only for the initial commit of a project.
+If the frontend creates all files from scratch when making a new
+branch, a `merge` command may be used instead of `from` to start
+the commit with an empty tree.
Omitting the `from` command on existing branches is usually desired,
as the current commit on that branch is automatically assumed to
be the first ancestor of the new commit.
Omitting the `from` command on existing branches is usually desired,
as the current commit on that branch is automatically assumed to
be the first ancestor of the new commit.
+
The reason fast-import uses `:` to denote a mark reference is this character
is not legal in a Git branch name. The leading `:` makes it easy
+
The reason fast-import uses `:` to denote a mark reference is this character
is not legal in a Git branch name. The leading `:` makes it easy
-to distingush between the mark 42 (`:42`) and the branch 42 (`42`
+to distinguish between the mark 42 (`:42`) and the branch 42 (`42`
or `refs/heads/42`), or an abbreviated SHA-1 which happened to
consist only of base-10 digits.
+
or `refs/heads/42`), or an abbreviated SHA-1 which happened to
consist only of base-10 digits.
+
* A complete 40 byte or abbreviated commit SHA-1 in hex.
* Any valid Git SHA-1 expression that resolves to a commit. See
* A complete 40 byte or abbreviated commit SHA-1 in hex.
* Any valid Git SHA-1 expression that resolves to a commit. See
- ``SPECIFYING REVISIONS'' in gitlink:git-rev-parse[1] for details.
+ ``SPECIFYING REVISIONS'' in linkgit:git-rev-parse[1] for details.
The special case of restarting an incremental import from the
current branch value should be written as:
The special case of restarting an incremental import from the
current branch value should be written as:
`merge`
^^^^^^^
`merge`
^^^^^^^
-Includes one additional ancestor commit, and makes the current
-commit a merge commit. An unlimited number of `merge` commands per
+Includes one additional ancestor commit. If the `from` command is
+omitted when creating a new branch, the first `merge` commit will be
+the first ancestor of the current commit, and the branch will start
+out with no files. An unlimited number of `merge` commands per
commit are permitted by fast-import, thereby establishing an n-way merge.
However Git's other tools never create commits with more than 15
additional ancestors (forming a 16-way merge). For this reason
it is suggested that frontends do not use more than 15 `merge`
commit are permitted by fast-import, thereby establishing an n-way merge.
However Git's other tools never create commits with more than 15
additional ancestors (forming a 16-way merge). For this reason
it is suggested that frontends do not use more than 15 `merge`
-commands per commit.
+commands per commit; 16, if starting a new, empty branch.
Here `<committish>` is any of the commit specification expressions
also accepted by `from` (see above).
Here `<committish>` is any of the commit specification expressions
also accepted by `from` (see above).
If an `LF` or double quote must be encoded into `<path>` shell-style
quoting should be used, e.g. `"path/with\n and \" in it"`.
If an `LF` or double quote must be encoded into `<path>` shell-style
quoting should be used, e.g. `"path/with\n and \" in it"`.
-The value of `<path>` must be in canoncial form. That is it must not:
+The value of `<path>` must be in canonical form. That is it must not:
* contain an empty directory component (e.g. `foo//bar` is invalid),
* end with a directory separator (e.g. `foo/` is invalid),
* contain an empty directory component (e.g. `foo//bar` is invalid),
* end with a directory separator (e.g. `foo/` is invalid),
complete set of bytes which normally goes into such a signature.
If signing is required, create lightweight tags from within fast-import with
`reset`, then create the annotated versions of those tags offline
complete set of bytes which normally goes into such a signature.
If signing is required, create lightweight tags from within fast-import with
`reset`, then create the annotated versions of those tags offline
-with the standard gitlink:git-tag[1] process.
+with the standard linkgit:git-tag[1] process.
`reset`
~~~~~~~
`reset`
~~~~~~~
....
'reset' SP <ref> LF
('from' SP <committish> LF)?
....
'reset' SP <ref> LF
('from' SP <committish> LF)?
- LF
+ LF?
....
For a detailed description of `<ref>` and `<committish>` see above
under `commit` and `from`.
....
For a detailed description of `<ref>` and `<committish>` see above
under `commit` and `from`.
+The `LF` after the command is optional (it used to be required).
+
The `reset` command can also be used to create lightweight
(non-annotated) tags. For example:
The `reset` command can also be used to create lightweight
(non-annotated) tags. For example:
+
....
'data' SP <count> LF
+
....
'data' SP <count> LF
- <raw> LF
+ <raw> LF?
....
+
where `<count>` is the exact number of bytes appearing within
`<raw>`. The value of `<count>` is expressed as an ASCII decimal
integer. The `LF` on either side of `<raw>` is not
included in `<count>` and will not be included in the imported data.
....
+
where `<count>` is the exact number of bytes appearing within
`<raw>`. The value of `<count>` is expressed as an ASCII decimal
integer. The `LF` on either side of `<raw>` is not
included in `<count>` and will not be included in the imported data.
++
+The `LF` after `<raw>` is optional (it used to be required) but
+recommended. Always including it makes debugging a fast-import
+stream easier as the next command always starts in column 0
+of the next line, even if `<raw>` did not end with an `LF`.
Delimited format::
A delimiter string is used to mark the end of the data.
fast-import will compute the length by searching for the delimiter.
Delimited format::
A delimiter string is used to mark the end of the data.
fast-import will compute the length by searching for the delimiter.
- This format is primarly useful for testing and is not
+ This format is primarily useful for testing and is not
recommended for real data.
+
....
'data' SP '<<' <delim> LF
<raw> LF
<delim> LF
recommended for real data.
+
....
'data' SP '<<' <delim> LF
<raw> LF
<delim> LF
+ LF?
....
+
where `<delim>` is the chosen delimiter string. The string `<delim>`
....
+
where `<delim>` is the chosen delimiter string. The string `<delim>`
immediately trailing `<raw>` is part of `<raw>`. This is one of
the limitations of the delimited format, it is impossible to supply
a data chunk which does not have an LF as its last byte.
immediately trailing `<raw>` is part of `<raw>`. This is one of
the limitations of the delimited format, it is impossible to supply
a data chunk which does not have an LF as its last byte.
++
+The `LF` after `<delim> LF` is optional (it used to be required).
`checkpoint`
~~~~~~~~~~~~
`checkpoint`
~~~~~~~~~~~~
....
'checkpoint' LF
....
'checkpoint' LF
- LF
+ LF?
....
Note that fast-import automatically switches packfiles when the current
....
Note that fast-import automatically switches packfiles when the current
repository can be loaded into Git through fast-import in about 3 hours,
explicit checkpointing may not be necessary.
repository can be loaded into Git through fast-import in about 3 hours,
explicit checkpointing may not be necessary.
+The `LF` after the command is optional (it used to be required).
+
+`progress`
+~~~~~~~~~~
+Causes fast-import to print the entire `progress` line unmodified to
+its standard output channel (file descriptor 1) when the command is
+processed from the input stream. The command otherwise has no impact
+on the current import, or on any of fast-import's internal state.
+
+....
+ 'progress' SP <any> LF
+ LF?
+....
+
+The `<any>` part of the command may contain any sequence of bytes
+that does not contain `LF`. The `LF` after the command is optional.
+Callers may wish to process the output through a tool such as sed to
+remove the leading part of the line, for example:
+
+====
+ frontend | git-fast-import | sed 's/^progress //'
+====
+
+Placing a `progress` command immediately after a `checkpoint` will
+inform the reader when the `checkpoint` has been completed and it
+can safely access the refs that fast-import updated.
+
+Crash Reports
+-------------
+If fast-import is supplied invalid input it will terminate with a
+non-zero exit status and create a crash report in the top level of
+the Git repository it was importing into. Crash reports contain
+a snapshot of the internal fast-import state as well as the most
+recent commands that lead up to the crash.
+
+All recent commands (including stream comments, file changes and
+progress commands) are shown in the command history within the crash
+report, but raw file data and commit messages are excluded from the
+crash report. This exclusion saves space within the report file
+and reduces the amount of buffering that fast-import must perform
+during execution.
+
+After writing a crash report fast-import will close the current
+packfile and export the marks table. This allows the frontend
+developer to inspect the repository state and resume the import from
+the point where it crashed. The modified branches and tags are not
+updated during a crash, as the import did not complete successfully.
+Branch and tag information can be found in the crash report and
+must be applied manually if the update is needed.
+
+An example crash:
+
+====
+ $ cat >in <<END_OF_INPUT
+ # my very first test commit
+ commit refs/heads/master
+ committer Shawn O. Pearce <spearce> 19283 -0400
+ # who is that guy anyway?
+ data <<EOF
+ this is my commit
+ EOF
+ M 644 inline .gitignore
+ data <<EOF
+ .gitignore
+ EOF
+ M 777 inline bob
+ END_OF_INPUT
+
+ $ git-fast-import <in
+ fatal: Corrupt mode: M 777 inline bob
+ fast-import: dumping crash report to .git/fast_import_crash_8434
+
+ $ cat .git/fast_import_crash_8434
+ fast-import crash report:
+ fast-import process: 8434
+ parent process : 1391
+ at Sat Sep 1 00:58:12 2007
+
+ fatal: Corrupt mode: M 777 inline bob
+
+ Most Recent Commands Before Crash
+ ---------------------------------
+ # my very first test commit
+ commit refs/heads/master
+ committer Shawn O. Pearce <spearce> 19283 -0400
+ # who is that guy anyway?
+ data <<EOF
+ M 644 inline .gitignore
+ data <<EOF
+ * M 777 inline bob
+
+ Active Branch LRU
+ -----------------
+ active_branches = 1 cur, 5 max
+
+ pos clock name
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ 1) 0 refs/heads/master
+
+ Inactive Branches
+ -----------------
+ refs/heads/master:
+ status : active loaded dirty
+ tip commit : 0000000000000000000000000000000000000000
+ old tree : 0000000000000000000000000000000000000000
+ cur tree : 0000000000000000000000000000000000000000
+ commit clock: 0
+ last pack :
+
+
+ -------------------
+ END OF CRASH REPORT
+====
Tips and Tricks
---------------
Tips and Tricks
---------------
When committing fixups, consider using `merge` to connect the
commit(s) which are supplying file revisions to the fixup branch.
When committing fixups, consider using `merge` to connect the
commit(s) which are supplying file revisions to the fixup branch.
-Doing so will allow tools such as gitlink:git-blame[1] to track
+Doing so will allow tools such as linkgit:git-blame[1] to track
through the real commit history and properly annotate the source
files.
through the real commit history and properly annotate the source
files.
Import Now, Repack Later
~~~~~~~~~~~~~~~~~~~~~~~~
As soon as fast-import completes the Git repository is completely valid
Import Now, Repack Later
~~~~~~~~~~~~~~~~~~~~~~~~
As soon as fast-import completes the Git repository is completely valid
-and ready for use. Typicallly this takes only a very short time,
+and ready for use. Typically this takes only a very short time,
even for considerably large projects (100,000+ commits).
However repacking the repository is necessary to improve data
even for considerably large projects (100,000+ commits).
However repacking the repository is necessary to improve data
~~~~~~~~~~~~~~~~~~~~~~~~~
If you are repacking very old imported data (e.g. older than the
last year), consider expending some extra CPU time and supplying
~~~~~~~~~~~~~~~~~~~~~~~~~
If you are repacking very old imported data (e.g. older than the
last year), consider expending some extra CPU time and supplying
-\--window=50 (or higher) when you run gitlink:git-repack[1].
+\--window=50 (or higher) when you run linkgit:git-repack[1].
This will take longer, but will also produce a smaller packfile.
You only need to expend the effort once, and everyone using your
project will benefit from the smaller repository.
This will take longer, but will also produce a smaller packfile.
You only need to expend the effort once, and everyone using your
project will benefit from the smaller repository.
+Include Some Progress Messages
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Every once in a while have your frontend emit a `progress` message
+to fast-import. The contents of the messages are entirely free-form,
+so one suggestion would be to output the current month and year
+each time the current commit date moves into the next month.
+Your users will feel better knowing how much of the data stream
+has been processed.
+
Packfile Optimization
---------------------
Packfile Optimization
---------------------
------------------
There are a number of factors which affect how much memory fast-import
requires to perform an import. Like critical sections of core
------------------
There are a number of factors which affect how much memory fast-import
requires to perform an import. Like critical sections of core
-Git, fast-import uses its own memory allocators to ammortize any overheads
-associated with malloc. In practice fast-import tends to ammoritize any
+Git, fast-import uses its own memory allocators to amortize any overheads
+associated with malloc. In practice fast-import tends to amortize any
malloc overheads to 0, due to its use of large block allocations.
per object
malloc overheads to 0, due to its use of large block allocations.
per object
~~~~~~~~~~~~~~~
Trees (aka directories) use just 12 bytes of memory on top of the
memory required for their entries (see ``per active file'' below).
~~~~~~~~~~~~~~~
Trees (aka directories) use just 12 bytes of memory on top of the
memory required for their entries (see ``per active file'' below).
-The cost of a tree is virtually 0, as its overhead ammortizes out
+The cost of a tree is virtually 0, as its overhead amortizes out
over the individual file entries.
per active file entry
over the individual file entries.
per active file entry
GIT
---
GIT
---
-Part of the gitlink:git[7] suite
+Part of the linkgit:git[7] suite