This rewrites history to upload some files to a git-lfs server. It
is similar to BFG's --convert-to-git-lfs, github.com/bozaro/git-lfs-migrate,
and probably some other tools.
FIXME: Need to implement most of this still.
Scripts making use of filter-repo as a library may want to be able to
compare the list of file changes in a commit to the original list of
file changes before other filtering occurred. For example, an
lfs-conversion script might want to know that e.g. --path selection and
--filename-callback said a bunch of files should be removed, so that the
lfs-conversion script can instead load those files up to git lfs.
FIXME: This slows down rails rewriting by over 60%, which is unacceptable;
need to find a different way to introduce this change. Maybe only when
explicitly requested somehow?
Signed-off-by: Elijah Newren <newren@gmail.com>
Make it clearer that absolute paths should not be used for pathnames
within a git repository. Also, fix the comment about how the
insert-beginning script could be implemented as a one-liner; the
example commented-out code should have used bytestrings.
Signed-off-by: Elijah Newren <newren@gmail.com>
The prerequisites and installation docs were not quite detailed enough,
and no code of conduct or contribution guidelines were included. Flesh
out the docs to cover these issues.
Some of the systems I ran on had a 'python3-coverage' and some had a
'coverage3' program. More were of the latter name, but more
importantly, the upstream tarball only creates the latter name;
apparently the former was just added by some distros. So, switch to the
more official name of the program.
Signed-off-by: Elijah Newren <newren@gmail.com>
- correct paths to including missing "Documentation/" prefix
- use fully specified "origin/docs" branch in case the "docs" branch is
not checked out locally
Signed-off-by: Benoit Fouletier <bennews@free.fr>
It appears that in addition to Windows requiring cwd be a string (and
not a bytestring), it also requires the command line arguments to be
unicode strings. This appears to be a python-on-Windows issue at the
surface (attempts to quote things that assumes the arguments are all
strings), but whether it's solely a python-on-Windows issue or there is
also a deeper Windows issue, we can workaround this brain-damage by
extending the SubprocessWrapper slightly. As with the cwd changes, only
apply this on Windows and not elsewhere because there are perfectly
legitimate reasons to pass non-unicode parameters (e.g. filenames that
are not valid unicode).
Signed-off-by: Elijah Newren <newren@gmail.com>
Unfortunately, it appears that Windows does not allow the 'cwd' argument
of various subprocess calls to be a bytestring. That may be functional
on Windows since Windows-related filesystems are allowed to require that
all file and directory names be valid unicode, but not all platforms
enforce such restrictions. As such, I certainly cannot change
cwd=directory
to
cwd=decode(directory)
because that could break on other platforms (and perhaps even on Windows
if someone is trying to read a non-native filesystem). Instead, create
a SubprocessWrapper class that will always call decode on the cwd
argument before passing along to the real subprocess class. Use these
wrappers on Windows, and do not use them elsewhere.
Signed-off-by: Elijah Newren <newren@gmail.com>
During the python3 transition, StringIO was renamed to io -- but the
import wasn't moved to preserve appropriate sorting.
Signed-off-by: Elijah Newren <newren@gmail.com>
Note that this isn't a version *number* or even the more generalized
version string that folks are used to seeing, but a version hash (or
leading portion thereof).
A few import points:
* These version hashes are not strictly monotonically increasing
values. Like I said, these aren't version numbers. If that
bothers you, read on...
* This scheme has incredibly nice semantics satisfying a pair of
properties that most version schemes would assume are mutually
incompatible:
This scheme works even if the user doesn't have a clone of
filter-repo and doesn't require any build step to inject the
version into the program; it works even if people just download
git-filter-repo.py off GitHub without any of the other sources.
And:
This scheme means that a user is running precisely version X of
the code, with the version not easily faked or misrepresented
when third parties edit the code.
Given the wonderful semantics provided by satisfying this pair of
properties that all other versioning schemes seem to miss out on, I
think I should name this scheme. How about "Semantic Versioning"?
(Hehe...)
* The version hash is super easy to use; I just go to my own clone of
filter-repo and run either:
git show $VERSION_HASH
or
git describe $VERSION_HASH
* A human consumable version might suggest to folks that this software
is something they might frequently use and upgrade. This program
should only be used in exceptional cases (because rewriting history
is not for the faint of heart).
* A human consumable version (i.e. a version number or even the
more relaxed version strings in more common use) might suggest to
folks that they can rely on strict backward compatibility. It's
nice to subtly undercut any such assumption.
* Despite all that, I will make releases (downloadable tarballs with
real version numbers in the tarball name; I'm just going to re-use
whatever version git is released with at the time). But those
version numbers won't be used by the --version option; instead the
version hash will.
Signed-off-by: Elijah Newren <newren@gmail.com>
A few changes:
* Include notes about git-2.24.0 changes
* Make it clearer that messing with the first parent could have
negative side-effects if the file_changes aren't also updated.
* Fix wrapping of a line that was too long.
Also, update the README.md:
* Note the upstream improvements made in (not yet released) git-2.24.0
Signed-off-by: Elijah Newren <newren@gmail.com>
Partial history rewrites were possible before with the (previously
hidden) --refs flag, but the defaults were wrong. That could be worked
around with the --source or --target flags, but that disabled --no-data
for fast-export and thus slowed things down, and also would require
overridding --replace-refs. And the defaults for --source and --target
may diverge further from what is wanted/needed for partial history
rewrites in the future.
So, add --partial as a first-class supported option with scary
documentation about how it permits mixing new and old history. Make
--refs imply that flag. Make the behavioral similarities (in regards to
which steps are skipped) between --source, --target, and --partial more
clear. Add relevant documentation to round it out.
Signed-off-by: Elijah Newren <newren@gmail.com>
In order to build the correct tree for a commit, git-fast-import always
takes a list of file changes for a merge commit relative to the first
parent.
When the entire first-parent history of a merge commit is pruned away
and the merge had paths with no difference relative to the first parent
but which differed relative to later parents, then we really need to
generate a new list of file changes in order to have one of those other
parents become the new first parent. An example might help clarify...
Let's say that there is a merge commit, and:
* it resolved differences in pathA between its two parents by taking
the version of pathA from the first parent.
* pathB was added in the history of the second parent (it is not
present in the first parent) and is NOT included in the merge commit
(either being deleted, or via rename treated as deleted and added as
something else)
For this merge commit, neither pathA nor pathB differ from the first
parent, and thus wouldn't appear in the list of file changes shown by
fast-export. However, when our filtering rules determine that the first
parent (and all its parents) should be pruned away, then the second
parent has to become the new first parent of the merge commit. But to
end up with the right files in the merge commit despite using a
different parent, we need a list of file changes that specifies the
changes for both pathA and pathB.
Signed-off-by: Elijah Newren <newren@gmail.com>
git.git wants to move more towards core-only rather than batteries
included, and as such, filter-repo will not be part of the git
distribution. Therefore, due to keeping the projects apart, there will
need to be separate translation files (assuming filter-repo ever gains
any translations) and as such we will need a different textdomain
definition.
Signed-off-by: Elijah Newren <newren@gmail.com>
Allow folks to periodically update the export of a live repo without
re-exporting from the beginning. This is a performance improvement, but
can also be important for collaboration. For example, for sensitivity
reasons, folks might want to export a subset of a repo and update the
export periodically. While this could be done by just re-exporting the
repository anew each time, there is a risk that the paths used to
specify the wanted subset might need to change in the future; making the
user verify that their paths (including globs or regexes) don't also
pick up anything from history that was previously excluded so that they
don't get a divergent history is not very user friendly. Allowing them
to just export stuff that is new since the last export works much better
for them.
Signed-off-by: Elijah Newren <newren@gmail.com>
Since we now have a separate user manual and it does not make sense to
duplicate information in multiple places, restructure the README:
* Refer to the actual manual early on
* Limit the README to mostly be about why I wrote it and why folks
might want to consider it instead of existing tools
* Include a new section on upstream improvements, especially since it
looks like inclusion of git-filter-repo in git.git is unlikely.
Signed-off-by: Elijah Newren <newren@gmail.com>
This is a re-implementation of git-filter-branch that is nearly
perfectly bug compatible (it can replace git-filter-branch and still
pass the git testsuite). It deviates in one minor way that should not
matter to real world usecases, but allows it to run a few times faster
than filter-branch.
Signed-off-by: Elijah Newren <newren@gmail.com>
Being able to find the new commit hash for either an abbreviated commit
hash or a full commit hash is much more useful than only working for a
full commit hash.
Signed-off-by: Elijah Newren <newren@gmail.com>
When I wrote git_fast_filter.py, I was unaware of and did not forsee
libgit2. So, although the license said the project could be used under
whatever license git.git was, there was still a potential barrier for
usage by libgit2. I'm not sure if libgit2 will ever want to use
filter-repo, but I don't want the barrier there and I would like to
avoid a repeat of this problem. (Also, since filter-repo is for the
most part a one-shot usage tool, I doubt that the normal copyleft
provisions could provide much value.)
MIT is widely used, compatible with just about everyting, and is
preferred by Palantir (my current employer) for open source
contributions. So, I contacted all other contributors (Jim is still at
Sandia) and got permission to relicense.
Signed-off-by: Elijah Newren <newren@gmail.com>
Commit 346f2ba891 (filter-repo: make reencoding of commit messages
togglable, 2019-05-11) made reencoding of commit messages togglable but
forgot to add parsing and outputting of the encoding header itself. Add
such ability now.
Signed-off-by: Elijah Newren <newren@gmail.com>
Elijah Newren
Matthisk Heimensen
Benoit Fouletier