Archives/git-filter-repo
2
0
Fork 0
mirror of https://github.com/newren/git-filter-repo.git synced 2024-11-15 06:12:55 +00:00
Code Issues Packages Projects Releases Wiki Activity
Quickly rewrite git repository history (filter-branch replacement)
384 Commits 11 Branches 13 Tags 5.5 MiB
Python 64.4%
Shell 34.2%
Makefile 1.4%
cdb7b77f07
Go to file
Elijah Newren cdb7b77f07 filter-repo: repack with --source or --target 
When using --source or --target in combination with filtering paths,
users were surprised out how large the resulting repository was.  The
usage of --source and --target were turning off repacking; while we
don't want repacking for partial history rewrites and --source and
--target turn on some of the other features we want with partial history
rewrites, repacking is something that we still want turned on.

Reported-by: Alexey Volkov <alexey.volkov@ark-kun.com>
Signed-off-by: Elijah Newren <newren@gmail.com>
2020-05-28 08:16:12 -07:00
contrib/filter-repo-demos contrib: fix special character handling in filter-lamely 2020-04-03 09:42:57 +02:00
Documentation git-filter-repo.txt: fix extraneous space 2020-05-27 23:47:50 -07:00
release setup.py: test for FileExistsError on symlink 2020-02-11 20:19:01 +01:00
t filter-repo: fix ugly bug with mixing path filtering and renaming 2020-05-25 12:35:34 -07:00
.gitignore .gitignore: ignore the test result directories 2019-12-27 09:25:25 -05:00
COPYING filter-repo: re-license 2019-07-16 13:15:52 -07:00
COPYING.gpl Update GPL license file 2019-12-22 18:33:01 +01:00
COPYING.mit filter-repo: re-license 2019-07-16 13:15:52 -07:00
git_filter_repo.py filter-repo: simplify import in lib-usage examples 2019-04-26 07:56:03 -07:00
git-filter-repo filter-repo: repack with --source or --target 2020-05-28 08:16:12 -07:00
INSTALL.md INSTALL: streamline a bit and guide folks to package managers 2020-05-18 09:25:11 -07:00
Makefile Makefile: a few sanity checks for releasing 2020-03-23 16:04:07 -07:00
README.md Documentation: add more detailed explanation of safety checks and --force 2020-05-19 14:52:51 -07:00

README.md

git filter-repo is a versatile tool for rewriting history, which includes capabilities I have not found anywhere else. It roughly falls into the same space of tool as git filter-branch but without the capitulation-inducing poor performance, with far more capabilities, and with a design that scales usability-wise beyond trivial rewriting cases. git filter-repo is now recommended by the git project instead of git filter-branch.

While most users will probably just use filter-repo as a simple command line tool (and likely only use a few of its flags), at its core filter-repo contains a library for creating history rewriting tools. As such, users with specialized needs can leverage it to quickly create entirely new history rewriting tools.

Table of Contents

Prerequisites

filter-repo requires:

  • git >= 2.22.0 at a minimum; some features require git >= 2.24.0
  • python3 >= 3.5

How do I install it?

git-filter-repo is a single-file python script, which was done to make installation for basic use trivial: just copy it into your $PATH.

See INSTALL.md for things beyond basic usage or special cases. The more involved instructions are needed if you

  • are working with a python3 executable named something other than "python3"
  • want to install documentation (beyond the builtin docs shown with -h)
  • want to run some of the contrib examples
  • want to create your own python filtering scripts using filter-repo as a module/library

How do I use it?

See the user manual. If you prefer learning from examples, the simple example below may be of interest, and the user manual has an extensive examples section.

Why filter-repo instead of other alternatives?

This was covered in more detail in a Git Rev News article on filter-repo, but some highlights for the main competitors:

filter-branch

BFG Repo Cleaner

  • great tool for its time, but while it makes some things simple, it is limited to a few kinds of rewrites.

  • its architecture is not amenable to handling more types of rewrites.

  • its architecture presents some shortcomings and bugs even for its intended usecase.

  • fans of bfg may be interested in bfg-ish, a reimplementation of bfg based on filter-repo which includes several new features and bugfixes relative to bfg.

  • a cheat sheet is available showing how to convert example commands from the manual of BFG Repo Cleaner into filter-repo commands.

Simple example, with comparisons

Let's say that we want to extract a piece of a repository, with the intent on merging just that piece into some other bigger repo. For extraction, we want to:

  • extract the history of a single directory, src/. This means that only paths under src/ remain in the repo, and any commits that only touched paths outside this directory will be removed.
  • rename all files to have a new leading directory, my-module/ (e.g. so that src/foo.c becomes my-module/src/foo.c)
  • rename any tags in the extracted repository to have a 'my-module-' prefix (to avoid any conflicts when we later merge this repo into something else)

Solving this with filter-repo

Doing this with filter-repo is as simple as the following command:

  git filter-repo --path src/ --to-subdirectory-filter my-module --tag-rename '':'my-module-'

(the single quotes are unnecessary, but make it clearer to a human that we are replacing the empty string as a prefix with my-module-)

Solving this with other filtering tools

By contrast, BFG Repo Cleaner is not capable of this kind of rewrite, it would take considerable effort to do this safely with fast-export/fast-import (especially if you wanted empty commits pruned or commit hashes rewritten), and filter-branch comes with a pile of caveats (more on that below) even once you figure out the necessary invocation(s):

  git filter-branch \
      --tree-filter 'mkdir -p my-module && \
                     git ls-files \
                         | grep -v ^src/ \
                         | xargs git rm -f -q && \
                     ls -d * \
                         | grep -v my-module \
                         | xargs -I files mv files my-module/' \
          --tag-name-filter 'echo "my-module-$(cat)"' \
	  --prune-empty -- --all
  git clone file://$(pwd) newcopy
  cd newcopy
  git for-each-ref --format="delete %(refname)" refs/tags/ \
      | grep -v refs/tags/my-module- \
      | git update-ref --stdin
  git gc --prune=now

Some might notice that the above filter-branch invocation will be really slow due to using --tree-filter; you could alternatively use the --index-filter option of filter-branch, changing the above commands to:

  git filter-branch \
      --index-filter 'git ls-files \
                          | grep -v ^src/ \
                          | xargs git rm -q --cached;
                      git ls-files -s \
                          | sed "s%$(printf \\t)%&my-module/%" \
                          | git update-index --index-info;
                      git ls-files \
                          | grep -v ^my-module/ \
                          | xargs git rm -q --cached' \
      --tag-name-filter 'echo "my-module-$(cat)"' \
      --prune-empty -- --all
  git clone file://$(pwd) newcopy
  cd newcopy
  git for-each-ref --format="delete %(refname)" refs/tags/ \
      | grep -v refs/tags/my-module- \
      | git update-ref --stdin
  git gc --prune=now

However, for either filter-branch command there are a pile of caveats. First, some may be wondering why I list five commands here for filter-branch. Despite the use of --all and --tag-name-filter, and filter-branch's manpage claiming that a clone is enough to get rid of old objects, the extra steps to delete the other tags and do another gc are still required to clean out the old objects and avoid mixing new and old history before pushing somewhere. Other caveats:

  • Commit messages are not rewritten; so if some of your commit messages refer to prior commits by (abbreviated) sha1, after the rewrite those messages will now refer to commits that are no longer part of the history. It would be better to rewrite those (abbreviated) sha1 references to refer to the new commit ids.
  • The --prune-empty flag sometimes misses commits that should be pruned, and it will also prune commits that started empty rather than just ended empty due to filtering. For repositories that intentionally use empty commits for versioning and publishing related purposes, this can be detrimental.
  • The commands above are OS-specific. GNU vs. BSD issues for sed, xargs, and other commands often trip up users; I think I failed to get most folks to use --index-filter since the only example in the filter-branch manpage that both uses it and shows how to move everything into a subdirectory is linux-specific, and it is not obvious to the reader that it has a portability issue since it silently misbehaves rather than failing loudly.
  • The --index-filter version of the filter-branch command may be two to three times faster than the --tree-filter version, but both filter-branch commands are going to be multiple orders of magnitude slower than filter-repo.
  • Both commands assume all filenames are composed entirely of ascii characters (even special ascii characters such as tabs or double quotes will wreak havoc and likely result in missing files or misnamed files)

Design rationale behind filter-repo

None of the existing repository filtering tools did what I wanted; they all came up short for my needs. No tool provided any of the first eight traits below I wanted, and all failed to provide at least one of the last four traits as well:

  1. [Starting report] Provide user an analysis of their repo to help them get started on what to prune or rename, instead of expecting them to guess or find other tools to figure it out. (Triggered, e.g. by running the first time with a special flag, such as --analyze.)

  2. [Keep vs. remove] Instead of just providing a way for users to easily remove selected paths, also provide flags for users to only keep certain paths. Sure, users could workaround this by specifying to remove all paths other than the ones they want to keep, but the need to specify all paths that ever existed in any version of the repository could sometimes be quite painful. For filter-branch, using pipelines like git ls-files | grep -v ... | xargs -r git rm might be a reasonable workaround but can get unwieldy and isn't as straightforward for users; plus those commands are often operating-system specific (can you spot the GNUism in the snippet I provided?).

  3. [Renaming] It should be easy to rename paths. For example, in addition to allowing one to treat some subdirectory as the root of the repository, also provide options for users to make the root of the repository just become a subdirectory. And more generally allow files and directories to be easily renamed. Provide sanity checks if renaming causes multiple files to exist at the same path. (And add special handling so that if a commit merely copied oldname->newname without modification, then filtering oldname->newname doesn't trigger the sanity check and die on that commit.)

  4. [More intelligent safety] Writing copies of the original refs to a special namespace within the repo does not provide a user-friendly recovery mechanism. Many would struggle to recover using that. Almost everyone I've ever seen do a repository filtering operation has done so with a fresh clone, because wiping out the clone in case of error is a vastly easier recovery mechanism. Strongly encourage that workflow by detecting and bailing if we're not in a fresh clone, unless the user overrides with --force.

  5. [Auto shrink] Automatically remove old cruft and repack the repository for the user after filtering (unless overridden); this simplifies things for the user, helps avoid mixing old and new history together, and avoids problems where the multi-step process for shrinking the repo documented in the manpage doesn't actually work in some cases. (I'm looking at you, filter-branch.)

  6. [Clean separation] Avoid confusing users (and prevent accidental re-pushing of old stuff) due to mixing old repo and rewritten repo together. (This is particularly a problem with filter-branch when using the --tag-name-filter option, and sometimes also an issue when only filtering a subset of branches.)

  7. [Versatility] Provide the user the ability to extend the tool or even write new tools that leverage existing capabilities, and provide this extensibility in a way that (a) avoids the need to fork separate processes (which would destroy performance), (b) avoids making the user specify OS-dependent shell commands (which would prevent users from sharing commands with each other), (c) takes advantage of rich data structures (because hashes, dicts, lists, and arrays are prohibitively difficult in shell) and (d) provides reasonable string manipulation capabilities (which are sorely lacking in shell).

  8. [Old commit references] Provide a way for users to use old commit IDs with the new repository (in particular via mapping from old to new hashes with refs/replace/ references).

  9. [Commit message consistency] If commit messages refer to other commits by ID (e.g. "this reverts commit 01234567890abcdef", "In commit 0013deadbeef9a..."), those commit messages should be rewritten to refer to the new commit IDs.

  10. [Become-empty pruning] Commits which become empty due to filtering should be pruned. If the parent of a commit is pruned, the first non-pruned ancestor needs to become the new parent. If no non-pruned ancestor exists and the commit was not a merge, then it becomes a new root commit. If no non-pruned ancestor exists and the commit was a merge, then the merge will have one less parent (and thus make it likely to become a non-merge commit which would itself be pruned if it had no file changes of its own). One special thing to note here is that we prune commits which become empty, NOT commits which start empty. Some projects intentionally create empty commits for versioning or publishing reasons, and these should not be removed. (As a special case, commits which started empty but whose parent was pruned away will also be considered to have "become empty".)

  11. [Become-degenerate pruning] Pruning of commits which become empty can potentially cause topology changes, and there are lots of special cases. Normally, merge commits are not removed since they are needed to preserve the graph topology, but the pruning of parents and other ancestors can ultimately result in the loss of one or more parents. A simple case was already noted above: if a merge commit loses enough parents to become a non-merge commit and it has no file changes, then it too can be pruned. Merge commits can also have a topology that becomes degenerate: it could end up with the merge_base serving as both parents (if all intervening commits from the original repo were pruned), or it could end up with one parent which is an ancestor of its other parent. In such cases, if the merge has no file changes of its own, then the merge commit can also be pruned. However, much as we do with empty pruning we do not prune merge commits that started degenerate (which indicates it may have been intentional, such as with --no-ff merges) but only merge commits that become degenerate and have no file changes of their own.

  12. [Speed] Filtering should be reasonably fast

How do I contribute?

See the contributing guidelines.

Is there a Code of Conduct?

Participants in the filter-repo community are expected to adhere to the same standards as for the git project, so the git Code of Conduct applies.

Upstream Improvements

Work on filter-repo and its predecessor has also driven numerous improvements to fast-export and fast-import (and occasionally other commands) in core git, based on things filter-repo needs to do its work: