filter-branch: Big syntax change; support rewriting multiple refs

We used to take the first non-option argument as the name for the new
branch.  This syntax is not extensible to support rewriting more than just
HEAD.

Instead, we now have the following syntax:

	git filter-branch [<filter options>...] [<rev-list options>]

All positive refs given in <rev-list options> are rewritten.  Yes,
in-place.  If a ref was changed, the original head is stored in
refs/original/$ref now, for your inspecting pleasure, in addition to the
reflogs (since it is easier to inspect "git show-ref | grep original" than
to inspect all the reflogs).

This commit also adds the --force option to remove .git-rewrite/ and all
refs from refs/original/ before filtering.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
index eaea82d0a690154223d0553603a08cf1fa27c8ce..915258f4105c69d35b70d0da67a8826ff2c5f6c8 100644 (file)
--- a/Documentation/git-filter-branch.txt
+++ b/Documentation/git-filter-branch.txt
@@ -12,7 +12,7 @@ SYNOPSIS
        [--index-filter <command>] [--parent-filter <command>]
        [--msg-filter <command>] [--commit-filter <command>]
        [--tag-name-filter <command>] [--subdirectory-filter <directory>]
-       [-d <directory>] <new-branch-name> [<rev-list options>...]
+       [-d <directory>] [-f | --force] [<rev-list options>...]
 
 DESCRIPTION
 -----------
@@ -26,10 +26,9 @@ information) will be preserved.
 The command takes the new branch name as a mandatory argument and
 the filters as optional arguments.  If you specify no filters, the
 commits will be recommitted without any changes, which would normally
-have no effect and result in the new branch pointing to the same
-branch as your current branch.  Nevertheless, this may be useful in
-the future for compensating for some git bugs or such, therefore
-such a usage is permitted.
+have no effect.  Nevertheless, this may be useful in the future for
+compensating for some git bugs or such, therefore such a usage is
+permitted.
 
 *WARNING*! The rewritten history will have different object names for all
 the objects and will not converge with the original branch.  You will not
@@ -38,8 +37,9 @@ original branch.  Please do not use this command if you do not know the
 full implications, and avoid using it anyway, if a simple single commit
 would suffice to fix your problem.
 
-Always verify that the rewritten version is correct before disposing
-the original branch.
+Always verify that the rewritten version is correct: The original refs,
+if different from the rewritten ones, will be stored in the namespace
+'refs/original/'.
 
 Note that since this operation is extensively I/O expensive, it might
 be a good idea to redirect the temporary directory off-disk, e.g. on
@@ -142,6 +142,11 @@ definition impossible to preserve signatures at any rate.)
        does this in the '.git-rewrite/' directory but you can override
        that choice by this parameter.
 
+-f\|--force::
+       `git filter-branch` refuses to start with an existing temporary
+       directory or when there are already refs starting with
+       'refs/original/', unless forced.
+
 <rev-list-options>::
        When options are given after the new branch name, they will
        be passed to gitlink:git-rev-list[1].  Only commits in the resulting
@@ -156,14 +161,14 @@ Suppose you want to remove a file (containing confidential information
 or copyright violation) from all commits:
 
 -------------------------------------------------------
-git filter-branch --tree-filter 'rm filename' newbranch
+git filter-branch --tree-filter 'rm filename' HEAD
 -------------------------------------------------------
 
 A significantly faster version:
 
--------------------------------------------------------------------------------
-git filter-branch --index-filter 'git update-index --remove filename' newbranch
--------------------------------------------------------------------------------
+--------------------------------------------------------------------------
+git filter-branch --index-filter 'git update-index --remove filename' HEAD
+--------------------------------------------------------------------------
 
 Now, you will get the rewritten history saved in the branch 'newbranch'
 (your current branch is left untouched).
@@ -172,25 +177,25 @@ To set a commit (which typically is at the tip of another
 history) to be the parent of the current initial commit, in
 order to paste the other history behind the current history:
 
-------------------------------------------------------------------------
-git filter-branch --parent-filter 'sed "s/^\$/-p <graft-id>/"' newbranch
-------------------------------------------------------------------------
+-------------------------------------------------------------------
+git filter-branch --parent-filter 'sed "s/^\$/-p <graft-id>/"' HEAD
+-------------------------------------------------------------------
 
 (if the parent string is empty - therefore we are dealing with the
 initial commit - add graftcommit as a parent).  Note that this assumes
 history with a single root (that is, no merge without common ancestors
 happened).  If this is not the case, use:
 
--------------------------------------------------------------------------------
+--------------------------------------------------------------------------
 git filter-branch --parent-filter \
-       'cat; test $GIT_COMMIT = <commit-id> && echo "-p <graft-id>"' newbranch
--------------------------------------------------------------------------------
+       'cat; test $GIT_COMMIT = <commit-id> && echo "-p <graft-id>"' HEAD
+--------------------------------------------------------------------------
 
 or even simpler:
 
 -----------------------------------------------
 echo "$commit-id $graft-id" >> .git/info/grafts
-git filter-branch newbranch $graft-id..
+git filter-branch $graft-id..HEAD
 -----------------------------------------------
 
 To remove commits authored by "Darl McBribe" from the history:
@@ -208,7 +213,7 @@ git filter-branch --commit-filter '
                done;
        else
                git commit-tree "$@";
-       fi' newbranch
+       fi' HEAD
 ------------------------------------------------------------------------------
 
 The shift magic first throws away the tree id and then the -p
@@ -238,14 +243,14 @@ A--B-----C
 To rewrite only commits D,E,F,G,H, but leave A, B and C alone, use:
 
 --------------------------------
-git filter-branch ... new-H C..H
+git filter-branch ... C..H
 --------------------------------
 
 To rewrite commits E,F,G,H, use one of these:
 
 ----------------------------------------
-git filter-branch ... new-H C..H --not D
-git filter-branch ... new-H D..H --not C
+git filter-branch ... C..H --not D
+git filter-branch ... D..H --not C
 ----------------------------------------
 
 To move the whole tree into a subdirectory, or remove it from there:
@@ -255,7 +260,7 @@ git filter-branch --index-filter \
        'git ls-files -s | sed "s-\t-&newsubdir/-" |
                GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
                        git update-index --index-info &&
-        mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' directorymoved
+        mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
 ---------------------------------------------------------------
 
 

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index 0d000ed306caad7be70856c070a362f09439e48d..0ff3475525a0d50739f218abbac91eb2dc926430 100755 (executable)
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh
@@ -78,6 +78,8 @@ filter_msg=cat
 filter_commit='git commit-tree "$@"'
 filter_tag_name=
 filter_subdir=
+orig_namespace=refs/original/
+force=
 while case "$#" in 0) usage;; esac
 do
        case "$1" in
@@ -85,6 +87,11 @@ do
                shift
                break
                ;;
+       --force|-f)
+               shift
+               force=t
+               continue
+               ;;
        -*)
                ;;
        *)
@@ -126,24 +133,43 @@ do
        --subdirectory-filter)
                filter_subdir="$OPTARG"
                ;;
+       --original)
+               orig_namespace="$OPTARG"
+               ;;
        *)
                usage
                ;;
        esac
 done
 
-dstbranch="$1"
-shift
-test -n "$dstbranch" || die "missing branch name"
-git show-ref "refs/heads/$dstbranch" 2> /dev/null &&
-       die "branch $dstbranch already exists"
-
-test ! -e "$tempdir" || die "$tempdir already exists, please remove it"
+case "$force" in
+t)
+       rm -rf "$tempdir"
+;;
+'')
+       test -d "$tempdir" &&
+               die "$tempdir already exists, please remove it"
+esac
 mkdir -p "$tempdir/t" &&
+tempdir="$(cd "$tempdir"; pwd)" &&
 cd "$tempdir/t" &&
 workdir="$(pwd)" ||
 die ""
 
+# Make sure refs/original is empty
+git for-each-ref > "$tempdir"/backup-refs
+while read sha1 type name
+do
+       case "$force,$name" in
+       ,$orig_namespace*)
+               die "Namespace $orig_namespace not empty"
+       ;;
+       t,$orig_namespace*)
+               git update-ref -d "$name" $sha1
+       ;;
+       esac
+done < "$tempdir"/backup-refs
+
 case "$GIT_DIR" in
 /*)
        ;;
@@ -153,6 +179,29 @@ case "$GIT_DIR" in
 esac
 export GIT_DIR GIT_WORK_TREE=.
 
+# These refs should be updated if their heads were rewritten
+
+git rev-parse --revs-only --symbolic "$@" |
+while read ref
+do
+       # normalize ref
+       case "$ref" in
+       HEAD)
+               ref="$(git symbolic-ref "$ref")"
+       ;;
+       refs/*)
+       ;;
+       *)
+               ref="$(git for-each-ref --format='%(refname)' |
+                       grep /"$ref")"
+       esac
+
+       git check-ref-format "$ref" && echo "$ref"
+done > "$tempdir"/heads
+
+test -s "$tempdir"/heads ||
+       die "Which ref do you want to rewrite?"
+
 export GIT_INDEX_FILE="$(pwd)/../index"
 git read-tree || die "Could not seed the index"
 
@@ -174,6 +223,8 @@ commits=$(wc -l <../revs | tr -d " ")
 
 test $commits -eq 0 && die "Found nothing to rewrite"
 
+# Rewrite the commits
+
 i=0
 while read commit parents; do
        i=$(($i+1))
@@ -234,22 +285,75 @@ while read commit parents; do
                $(git write-tree) $parentstr < ../message > ../map/$commit
 done <../revs
 
-src_head=$(tail -n 1 ../revs | sed -e 's/ .*//')
-target_head=$(head -n 1 ../map/$src_head)
-case "$target_head" in
-'')
-       echo Nothing rewritten
+# In case of a subdirectory filter, it is possible that a specified head
+# is not in the set of rewritten commits, because it was pruned by the
+# revision walker.  Fix it by mapping these heads to the next rewritten
+# ancestor(s), i.e. the boundaries in the set of rewritten commits.
+
+# NEEDSWORK: we should sort the unmapped refs topologically first
+while read ref
+do
+       sha1=$(git rev-parse "$ref"^0)
+       test -f "$workdir"/../map/$sha1 && continue
+       # Assign the boundarie(s) in the set of rewritten commits
+       # as the replacement commit(s).
+       # (This would look a bit nicer if --not --stdin worked.)
+       for p in $((cd "$workdir"/../map; ls | sed "s/^/^/") |
+               git rev-list $ref --boundary --stdin |
+               sed -n "s/^-//p")
+       do
+               map $p >> "$workdir"/../map/$sha1
+       done
+done < "$tempdir"/heads
+
+# Finally update the refs
+
+_x40='[0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f]'
+_x40="$_x40$_x40$_x40$_x40$_x40$_x40$_x40$_x40"
+count=0
+echo
+while read ref
+do
+       # avoid rewriting a ref twice
+       test -f "$orig_namespace$ref" && continue
+
+       sha1=$(git rev-parse "$ref"^0)
+       rewritten=$(map $sha1)
+
+       test $sha1 = "$rewritten" &&
+               warn "WARNING: Ref '$ref' is unchanged" &&
+               continue
+
+       case "$rewritten" in
+       '')
+               echo "Ref '$ref' was deleted"
+               git update-ref -m "filter-branch: delete" -d "$ref" $sha1 ||
+                       die "Could not delete $ref"
        ;;
-*)
-       git update-ref refs/heads/"$dstbranch" $target_head ||
-               die "Could not update $dstbranch with $target_head"
-       if [ $(wc -l <../map/$src_head) -gt 1 ]; then
-               echo "WARNING: Your commit filter caused the head commit to expand to several rewritten commits. Only the first such commit was recorded as the current $dstbranch head but you will need to resolve the situation now (probably by manually merging the other commits). These are all the commits:" >&2
-               sed 's/^/       /' ../map/$src_head >&2
-               ret=1
-       fi
+       $_x40)
+               echo "Ref '$ref' was rewritten"
+               git update-ref -m "filter-branch: rewrite" \
+                               "$ref" $rewritten $sha1 ||
+                       die "Could not rewrite $ref"
        ;;
-esac
+       *)
+               # NEEDSWORK: possibly add -Werror, making this an error
+               warn "WARNING: '$ref' was rewritten into multiple commits:"
+               warn "$rewritten"
+               warn "WARNING: Ref '$ref' points to the first one now."
+               rewritten=$(echo "$rewritten" | head -n 1)
+               git update-ref -m "filter-branch: rewrite to first" \
+                               "$ref" $rewritten $sha1 ||
+                       die "Could not rewrite $ref"
+       ;;
+       esac
+       git update-ref -m "filter-branch: backup" "$orig_namespace$ref" $sha1
+       count=$(($count+1))
+done < "$tempdir"/heads
+
+# TODO: This should possibly go, with the semantics that all positive given
+#       refs are updated, and their original heads stored in refs/original/
+# Filter tags
 
 if [ "$filter_tag_name" ]; then
        git for-each-ref --format='%(objectname) %(objecttype) %(refname)' refs/tags |
@@ -286,6 +390,8 @@ fi
 
 cd ../..
 rm -rf "$tempdir"
-printf "\nRewritten history saved to the $dstbranch branch\n"
+echo
+test $count -gt 0 && echo "These refs were rewritten:"
+git show-ref | grep ^"$orig_namespace"
 
 exit $ret

diff --git a/t/t7003-filter-branch.sh b/t/t7003-filter-branch.sh
index 4ddd656e847e90e8f9f373e40dc41c39809d816f..bc6e2ddb19093fbdc9e4f94cbec71d369f0251ed 100755 (executable)
--- a/t/t7003-filter-branch.sh
+++ b/t/t7003-filter-branch.sh
@@ -30,24 +30,24 @@ test_expect_success 'setup' '
 H=$(git rev-parse H)
 
 test_expect_success 'rewrite identically' '
-       git-filter-branch H2
+       git-filter-branch branch
 '
-
 test_expect_success 'result is really identical' '
-       test $H = $(git rev-parse H2)
+       test $H = $(git rev-parse HEAD)
 '
 
 test_expect_success 'rewrite, renaming a specific file' '
-       git-filter-branch --tree-filter "mv d doh || :" H3
+       git-filter-branch -f --tree-filter "mv d doh || :" HEAD
 '
 
 test_expect_success 'test that the file was renamed' '
-       test d = $(git show H3:doh)
+       test d = $(git show HEAD:doh)
 '
 
-git tag oldD H3~4
+git tag oldD HEAD~4
 test_expect_success 'rewrite one branch, keeping a side branch' '
-       git-filter-branch --tree-filter "mv b boh || :" modD D..oldD
+       git branch modD oldD &&
+       git-filter-branch -f --tree-filter "mv b boh || :" D..modD
 '
 
 test_expect_success 'common ancestor is still common (unchanged)' '
@@ -69,7 +69,8 @@ test_expect_success 'filter subdirectory only' '
        git rm a &&
        test_tick &&
        git commit -m "again not subdir" &&
-       git-filter-branch --subdirectory-filter subdir sub
+       git branch sub &&
+       git-filter-branch -f --subdirectory-filter subdir refs/heads/sub
 '
 
 test_expect_success 'subdirectory filter result looks okay' '
@@ -89,7 +90,8 @@ test_expect_success 'setup and filter history that requires --full-history' '
        test_tick &&
        git commit -m "again subdir on master" &&
        git merge branch &&
-       git-filter-branch --subdirectory-filter subdir sub-master
+       git branch sub-master &&
+       git-filter-branch -f --subdirectory-filter subdir sub-master
 '
 
 test_expect_success 'subdirectory filter result looks okay' '
@@ -100,7 +102,8 @@ test_expect_success 'subdirectory filter result looks okay' '
 '
 
 test_expect_success 'use index-filter to move into a subdirectory' '
-       git-filter-branch --index-filter \
+       git branch directorymoved &&
+       git-filter-branch -f --index-filter \
                 "git ls-files -s | sed \"s-\\t-&newsubdir/-\" |
                  GIT_INDEX_FILE=\$GIT_INDEX_FILE.new \
                        git update-index --index-info &&
@@ -108,9 +111,10 @@ test_expect_success 'use index-filter to move into a subdirectory' '
        test -z "$(git diff HEAD directorymoved:newsubdir)"'
 
 test_expect_success 'stops when msg filter fails' '
-       ! git-filter-branch --msg-filter false nonono &&
-       rm -rf .git-rewrite &&
-       ! git rev-parse nonono
+       old=$(git rev-parse HEAD) &&
+       ! git-filter-branch -f --msg-filter false &&
+       test $old = $(git rev-parse HEAD) &&
+       rm -rf .git-rewrite
 '
 
 test_expect_success 'author information is preserved' '
@@ -118,7 +122,8 @@ test_expect_success 'author information is preserved' '
        git add i &&
        test_tick &&
        GIT_AUTHOR_NAME="B V Uips" git commit -m bvuips &&
-       git-filter-branch --msg-filter "cat; \
+       git branch preserved-author &&
+       git-filter-branch -f --msg-filter "cat; \
                        test \$GIT_COMMIT != $(git rev-parse master) || \
                        echo Hallo" \
                preserved-author &&
@@ -129,7 +134,8 @@ test_expect_success "remove a certain author's commits" '
        echo i > i &&
        test_tick &&
        git commit -m i i &&
-       git-filter-branch --commit-filter "\
+       git branch removed-author &&
+       git-filter-branch -f --commit-filter "\
                if [ \"\$GIT_AUTHOR_NAME\" = \"B V Uips\" ];\
                then\
                        shift;\
@@ -148,4 +154,9 @@ test_expect_success "remove a certain author's commits" '
        test 0 = $(git rev-list --author="B V Uips" removed-author | wc -l)
 '
 
+test_expect_success 'barf on invalid name' '
+       ! git filter-branch -f master xy-problem &&
+       ! git filter-branch -f HEAD^
+'
+
 test_done