We have a good default for pruning of empty commits and degenerate merge
commits: only pruning such commits that didn't start out that way (i.e.
that couldn't intentionally have been empty or degenerate). However,
users may have reasons to want to aggressively prune such commits (maybe
they used BFG repo filter or filter-branch previously and have lots of
cruft commits that they want remoed), and we may as well allow them to
specify that they don't want pruning too, just to be flexible.
Signed-off-by: Elijah Newren <newren@gmail.com>
If a commit was a merge in the original repo, and its ancestors on at
least one side have all been filtered away, and the commit has no
filechanges relative to its remaining parent (if any), then this commit
should be pruned. We had a small logic error preventing such pruning;
fix it by checking len(parents) instead of len(orig_parents).
Signed-off-by: Elijah Newren <newren@gmail.com>
fast-import syntax declares how to specify the parents of a commit with
'from' and possibly 'merge' directives, but it oddly also allows parents
to be implicitly specified via branch name. The documentation is easy
to misread:
"Omitting the from command in the first commit of a new branch will
cause fast-import to create that commit with no ancestor."
Note that the "in the first commit of a new branch" is key here. It is
reinforced later in the document with:
"Omitting the from command on existing branches is usually desired, as
the current commit on that branch is automatically assumed to be the
first ancestor of the new commit."
Desirability of operating this way aside, this raises an interesting
question: what if you only have one branch in some repository, but that
branch has more than one root commit? How does one use the fast-import
format to import such a repository? The fast-import documentation
doesn't state as far as I can tell, but using a 'reset' directive
without providing a 'from' reference for it is the way to go.
Modify filter-repo to understand implicit 'from' commits, and to
appropriately issue 'reset' directives when we need additional root
commits.
Signed-off-by: Elijah Newren <newren@gmail.com>
Notable items:
* We use bytestrings _everywhere_. This is incredibly annoying to
me, as I think users will be tempted to use "normal" strings in
callback functions and get surprised when things compare as
unequal, but I did like 3-4 python3 conversions with different
amounts in bytestrings and regular strings, and I always hit
real world repositories with alternate encodings on user names
and commit messages (despite commit messages not necessarily
having a special 'encoding' field). Further, I was always
risking munging data the user didn't want by trying to 'decode'
the bytestrings into unicode, and I was probably slowing down
performance. So, in the end I gave up and everything must be a
bytestring.
* The performance of the python2 version of filter-repo drifted
slightly over time with additional features and more robust
checking (particularly the become-empty and become-degenerate
pruning), though largely still providing the same performance
as I highlighted in my BFG/filter-branch/filter-repo comparison.
There certainly weren't any factors of 2 difference. A pleasant
surprise was that the python2->python3 conversion appears to have
only made a difference of a couple percent to performance and
some tests were faster and others slower than the python2 version.
So performance seems to be a wash.
* The individual commits on python3-conversion do not work
independently, but rather demonstrate separate aspects of what work
was needed in the large conversion to python3.
Signed-off-by: Elijah Newren <newren@gmail.com>
This is by far the largest python3 change; it consists basically of
* using b'<str>' instead of '<str>' in lots of places
* adding a .encode() if we really do work with a string but need to
get it converted to a bytestring
* replace uses of .format() with interpolation via the '%' operator,
since bytestrings don't have a .format() method.
Signed-off-by: Elijah Newren <newren@gmail.com>
Unlike how str works, if we grab an array index of a bytestr we get an
integer (corresponding to the ASCII value) instead of a bytestr of
length 1. Adjust code accordingly.
Signed-off-by: Elijah Newren <newren@gmail.com>
python3 forces a couple issues for us with the conversion of globs to
regexes:
* fnmatch.translate() will ONLY operate on unicode strings, not
bytestrings. Super lame.
* newer versions of python3 modified the regex style used by
fnmatch.translate() causing us to need extra logic to 'fixup'
the regex into the form we want.
Split the code for translating the glob to a regex out into a separate
function which now houses more complicated logic to handle these extra
conditions.
Signed-off-by: Elijah Newren <newren@gmail.com>
We need a function to transform byte strings into unicode strings for
printing error messages and occasional other uses.
Signed-off-by: Elijah Newren <newren@gmail.com>
Commit ca32c5d9afe2 ("filter-repo: workaround python<2.7.9 exec bug",
2019-04-30) put in a workaround for python versions prior to 2.7.9, but
which was incompatible with python3. Revert it as one step towards
migrating to python3.
Signed-off-by: Elijah Newren <newren@gmail.com>
Python issue 21591 will cause SyntaxError messages to by thrown if using
python versions prior to 2.7.9. Use the workaround identified in the
bug report: use the exec statement instead of the exec function, even if
this will need to be reverted for python3.
Signed-off-by: Elijah Newren <newren@gmail.com>
Extra LFs are permitted in git-fast-import syntax, and they serve to
make it easier to read the stream (from --dry-run or --debug), if they
are so inclined.
Signed-off-by: Elijah Newren <newren@gmail.com>
When we invoked the 'ls' command of fast-import, we just passed the
filename as-is. That will work for most filenames, but some have to
be quoted. Make sure we do so.
Signed-off-by: Elijah Newren <newren@gmail.com>
While the underlying fast-export and fast-import streams explicitly
separate 'from' commit (first parent) and 'merge' commits (all other
parents), foisting that separation into the Commit object for
filter-repo forces additional places in the code to deal with that
distinction. It results in less clear code, and especially does not
make sense to push upon folks who may want to use filter-repo as a
library.
Signed-off-by: Elijah Newren <newren@gmail.com>
Use UTF-8 chars in user names, filenames, branch names, tag names, and
file contents. Also include invalid UTF-8 in file contents; should be
able to handle binary data.
Signed-off-by: Elijah Newren <newren@gmail.com>
The sorting order of entries written to files in the analysis directory
didn't specify a secondary sort, thus making the order dependent on the
random-ish sorting order of dictionaries and making it inconsistent
between python versions. While the secondary order didn't matter much,
having a defined order makes it slightly easier to define a single
testcase that can work across versions.
Signed-off-by: Elijah Newren <newren@gmail.com>
Assuming filter-repo will be merged into git.git, use "git" for the
TEXTDOMAIN, and assume its build system will replace "@@LOCALEDIR@@"
appropriately.
Note that the xgettext command used to grab string translations is
nearly identical to the one for C files in git.git; just use
--language=python instead and add --join-existing to avoid overwriting
the po/git.pot file. In other words, use the command:
xgettext -o../git/po/git.pot --join-existing --force-po \
--add-comments=TRANSLATORS: \
--msgid-bugs-address="Git Mailing List <git@vger.kernel.org>" \
--from-code=UTF-8 --language=python \
--keyword=_ --keyword=N_ --keyword="Q_:1,2" \
git-filter-repo
To create or update the translation, go to git.git/po and run either of:
msginit --locale=XX
msgmerge --add-location --backup=off -U XX.po git.pot
Once you've updated the translation, within git.git just build as
normal. That's all that's needed.
Signed-off-by: Elijah Newren <newren@gmail.com>
Over a decade ago, I added code to deal with splitting and splicing
repositories where you weren't always dealing with first parents and
linear histories, and in particular where the mainline tended to be the
second parent (because there was no integrator or special central
gatekeeper like gerrit or github; instead, everyone pushed directly to
the main repository after locally testing, and integration happened via
everyone running 'git pull'). When attempting to splice repositories
the fact that fast-export always gave changes relative to the first
parent caused some grief with my splitting and splicing efforts.
It has been over a decade, I don't know of a good testcase of this
functionality separate from the live repositories I lost access to over
six years ago, git-subtree was released in the meantime which I'm
certain handled the task better, git-fast-export since gained a
--full-tree option which might have provided a better way to attack the
problem (though with splicing repos you often want work with additive
changes rather than recreating from scratch), and I just don't
quite understand the code anymore anyway. I think it had some
fundamental limitations that I knew my usecase avoided, but I don't
remember the details (and I'm not certain if this is true).
Even though code coverage hits all but one of the lines, I'd rather
rewrite any needed functionality if the usecase arises, and in view of
what facilities exist today rather than what I was working with a decade
ago. So, just nuke this code.
Signed-off-by: Elijah Newren <newren@gmail.com>
The original idea was to add --path-rename-(glob|regex) options, but
I like the general flexibility of --filename-callback better for
special cases and keeping the number of command line options at least
slightly in check.
Signed-off-by: Elijah Newren <newren@gmail.com>
There are several lines equivalent to BUG() calls in git that are
supposed to be unreachable, and which exist just to make debugging the
fundamental system problem or refactoring of the code slightly easier by
trying to give a more immediate notification of a problem. If these
error cases are ever hit and happen to be wrong, then the individual
will at worst get a stacktrace and the program will abort...but that
might arguably be even more helpful. Since there is no harm in avoiding
the work of finding ways to break the system to force these lines to be
covered, simply exclude them from line coverage counting.
Signed-off-by: Elijah Newren <newren@gmail.com>
The AncestryGraph setup assumed we had previously seen all commits which
would be used as parents; that interacted badly with doing an
incremental import. Add a function which can be used to record external
commits, each of which we'll treat like a root commit (i.e. depth 1 and
having no parents of its own). Add a test to prevent regressions.
Signed-off-by: Elijah Newren <newren@gmail.com>
There are a number of things not present in "normal" imports that we
nevertheless support and need to be tested:
* broken timezone adjustment (+051800->+0261; observed in the wild
in real repos, and adjustment prevents fast-import from dying)
* commits missing an author (observed in the wild in a real repo;
just sets author to committer)
* optional additional linefeeds in the input allowed by
git-fast-import but usually not written by git-fast-export
* progress and checkpoint objects
* progress, checkpoint, and 'everything' callbacks
Signed-off-by: Elijah Newren <newren@gmail.com>
While most users of filter-repo will just use it as a tool and
RepoFilter.run() is the final function, filter-repo can be used as a
library with additional work being done after calling that function.
So, simply return from that function when it is done rather than calling
sys.exit.
Signed-off-by: Elijah Newren <newren@gmail.com>
The everything_callback took two arguments, the first being a string
that was the name of the type of the second argument. There was no
point for this argument; someone can just compare type(second) to the
relevant classes. Remove it.
Signed-off-by: Elijah Newren <newren@gmail.com>
The only times this is ever printed is when debugging filter-repo
itself, or when trying to add tests to get to 100% line coverage. But
the printing was broken when objects were skipped (which caused a
mapping from int -> None). Fix the format specifier to handle this
case too.
Signed-off-by: Elijah Newren <newren@gmail.com>
We don't expect to ever get progress or checkpoint directives in normal
operation, but the --stdin flag makes it a possibility. In such a case,
the progress directives could actually break our parsing since
git-fast-import will just print it to its stdout, which is what we read
from to find new commit names so we can do commit message hash updating.
So, pass these along to a progress_callback, but don't dump them by
default. Also, it is not clear checkpoint directives make sense given
that we'll be filtering and only getting a subset of history (and I'm
dubious on checkpoint's utility in general anyway as fast-import is
relatively quick), so pass these along to a callback but don't use them
by default.
Signed-off-by: Elijah Newren <newren@gmail.com>
We don't run fast-export with rename detection, even though we have
code for handling it, because we decided to use a rev-list|diff-tree
pipeline instead. The code was manually tested and determined to be
working and it might be useful in the future so I don't want to just
outright delete it, but since we know we can't trigger it right now,
add a
# pragma: no cover
on these lines so it doesn't show up on coverage reports.
Signed-off-by: Elijah Newren <newren@gmail.com>
