git filter-repo is a tool for rewriting history, which includes some capabilities I have not found anywhere else. It is most similar to git filter-branch, though it fixes what I perceive to be some glaring deficiencies in that tool and brings a much different taste in usability. Also, being based on fast-export/fast-import, it is orders of magnitude faster (it has speed roughly comparable to BFG repo cleaner, but isn't multi-threaded).

filter-repo is a single-file python script, depending only on the python standard library (and execution of git commands), all of which is designed to make build/installation trivial: just copy it into your $PATH.

Table of Contents

• Background
  • Why create another repo filtering tool?
  • Warnings: Not yet ready for external usage
  • Why not $FAVORITE_COMPETITOR
• Usage

Background

Why git-filter-repo?

None of the existing repository filtering tools do what I want. They're all good in their own way, but come up short for my needs. In no particular order:

1. [Starting report] Provide user an analysis of their repo to help them get started on what to prune or rename, instead of expecting them to guess or find other tools to figure it out. (Triggered, e.g. by running the first time with a special flag, such as --analyze.)

2. [Keep vs. remove] Instead of just providing a way for users to easily remove selected paths, also provide flags for users to only keep certain paths. Sure, users could workaround this by specifying to remove all paths other than the ones they want to keep, but the need to specify all paths that ever existed in any version of the repository could sometimes be quite painful. For filter-branch, using pipelines like git ls-files | grep -v ... | xargs -r git rm might be a reasonable workaround but can get unwieldy and isn't as straightforward for users; plus those commands are often operating-system specific (can you spot the GNUism in the snippet I provided?).

3. [Renaming] It should be easy to rename paths. For example, in addition to allowing one to treat some subdirectory as the root of the repository, also provide options for users to make the root of the repository just become a subdirectory. And more generally allow files and directories to be easily renamed. Provide sanity checks if renaming causes multiple files to exist at the same path. (And add special handling so that if a commit merely renamed oldname->newname, then filtering oldname->newname doesn't trigger the sanity check and die on that commit.)

4. [More intelligent safety] Writing copies of the original refs to a special namespace within the repo does not provide a user-friendly recovery mechanism. Many would struggle to recover using that. Almost everyone I've ever seen do a repository filtering operation has done so with a fresh clone, because wiping out the clone in case of error is a vastly easier recovery mechanism. Strongly encourage that workflow by detecting and bailing if we're not in a fresh clone, unless the user overrides with --force. (Allow the old filter-branch workflow if a special --store-backup flag is provided.)

5. [Auto shrink] Automatically remove old cruft and repack the repository for the user after filtering (unless overridden); this simplifies things for the user, helps avoid mixing old and new history together, and avoids problems where the multi-step process for shrinking the repo documented in the manpage doesn't actually work in some cases. (I'm looking at you, filter-branch.)

6. [Clean separation] Avoid confusing users (and prevent accidental re-pushing of old stuff) due to mixing old repo and rewritten repo together. (This is particularly a problem with filter-branch when using the --tag-name-filter option, and sometimes also an issue when only filtering a subset of branches.)

7. [Commit message consistency] If commit messages refer to other commits by ID (e.g. "this reverts commit 01234567890abcdef", "In commit 0013deadbeef9a..."), those commit messages should be rewritten to refer to the new commit IDs.

8. [Empty pruning] Commits which become empty due to filtering should be pruned. Note that pruning of commits which become empty can potentially cause topology changes, and there are lots of special cases. The most basic is that if the parent of a commit is pruned, the first non-pruned ancestor needs to become the new parent; if no non-pruned ancestor exists, the commit becomes a new root commit. Normally, merge commits are not removed since they are needed to preserve the graph topology, but the pruning of parents and other ancestors can ultimately result in the loss of one or more parents. If a merge commit loses enough parents to become a non-merge commit and it has no file changes, then it too can be pruned. Topology changes are also possible if the entire non-first-parent history is pruned away; rather than having that parent of the merge be rewritten to the merge base, it may (depending on whether the merge also had file changes of its own) instead make sense to just prune that parent. (We do not want to prune away a first parent being rewritten to the merge base since some projects prefer --no-ff merges, though this could be made an option.) Finally, note that we originally talked not about pruning empty commits, but about pruning commits which become empty. Some projects intentionally create empty commits for versioning or publishing reasons, and these should not be removed. Instead, only commits which become empty should be pruned. (As a special case, commits which started empty but originally had a parent and which become a root commit due to the pruning of other commits will also be considered to have "become empty".)

9. [Speed] Filtering should be reasonably fast

Warnings: Not yet ready for external usage

This repository is still under heavy construction. Some caveats:

• It will not work without a specially compiled version of git:
  • git clone --branch fast-export-import-improvements https://github.com/newren/git/
  • Build according to normal git.git build instructions. You can find 'em.
• I have a list of known bugs, conveniently mostly tracked in my head. I'll fix that, but the fact that you're reading this sentence means I haven't yet.
• Actually, there's a couple exceptions to where bugs are tracked mentioned above. In particular, the following bugs are tracked here:
  • Multiple unimplemented placeholder option flags exist. Just because it shows up in --help doesn't mean it does anything.
  • Usage instructions and examples at the end of this document are rather lacking.
  • Random debugging code or extraneous files might be checked in at any given time; I'll probably rewrite history to remove them...eventually.
• I reserve the right to:
  • Rename the tool altogether (filter-repo to be like filter-branch?)
  • Rename or redefine any command line options
  • Rewrite the history of this repository at any time
• and possibly more...but do you really need any more reasons than the above? This isn't ready for widespread use.

Why not $FAVORITE_COMPETITOR?

Here are some of the prominent competitors I know of:

• git_fast_filter.py (Original link dead, use google if you care; this repo is the successor, though.)
• reposurgeon
• BFG repo cleaner
• git filter-branch

Here's why I think these tools don't meet my needs:

• git_fast_filter.py:

  • This was actually the basis for filter-repo, though it required lots of additional work.
  • Was meant as a library more than a tool, and had too high of an activation energy.
  • empty commit pruning was not as thorough as it should have been
  • had no provision for commit message rewriting for commit message consistency.
  • missing lots of little conveniences

• reposurgeon

  • focused on converting repositories between version control systems, and handles all the crazy impedance mismatches inherent in such conversions. I only care about rewriting history that starts in git and ends in git. If you care about converting between version control systems, though, reposurgeon is a much better tool.
  • might be general enough to use for other uses, but can't find any documentation or examples on anything other than huge repository conversions between version control systems.
  • way too much effort for many simple repository rewrites that many users want to perform

• BFG repo cleaner

  • Very focused on just removing crazy big files and sensitive data. Probably the best tool if that's all you want. But lacks the ability to handle anything outside this special (but important!) usecase.
  • Has useful options for helping you remove the N biggest blobs, but nothing to help you know how big N should be.
  • Doesn't prune commits which become empty due to filtering; if you just want to extract a directory added 3 months ago and its history, you'd be stuck with years of commits touching other directories, all empty.
  • The refusal to rewrite HEAD, while it makes sense when trying to remove a few crazy big files and sensitive data (users tend to re-add and re-commit bad files if you didn't manually remove it and have them update), is totally misaligned with more general rewrite cases (e.g. the desire to turn a subdirectory into the root of a repository, or move the root of the repository into a subdirectory for merging into some other bigger repo.)
  • Telling the user how to shrink the repo afterwards seems lame since that was the whole point; just do it for them by default.

• git filter-branch

  • Fundamental design flaw causing it to be orders of magnitude slower than it should be for most repo rewriting jobs. So slow that it becomes a major usability impediment, if not a deal breaker. However, it is extremely versatile.
  • Generally quick for users to invoke (quick one-liners with lots of examples), just missing some useful capabilities like selecting wanted paths (as opposed to unwanted paths) and providing easier path renaming (also, e.g. no --to-subdirectory-filter as the opposite of --subdirectory-filter)
  • Doesn't rewrite commit hashes in commit messages, causing commit messages to refer to phantom commits instead.
  • Mixes old repository information (original tags, unrewritten branches) with new, risking re-pushing the old stuff
  • Lame defaults
    • --prune-empty should be default (although only commits which become empty, not ones which started empty)
    • allows user to mess with repos which aren't a clean clone without an override
    • Makes it very difficult to actually get rid of unwanted objects and shrink repository. Long multi-step instructions in manpage for this, which are incomplete when --tag-name-filter is in use.

Usage

Run git filter-repo --help and figure it out from there. Good luck.