This option walks through the repository history and creates a report
with basic statistics, rename related information, and sizes of objects
and when/if those have been deleted. It primarily looks at unpacked
sizes (i.e. size of object ignoring delta-ing and compression), and
sums the size of each version of the file for each path. Additionally,
it aggregates these sums by extension and by directory, and tracks
whether paths, extensions, and directories have been deleted. This can
be very useful in determining what the big things are, and whether they
might have been considered to have been mistakes to add to the
repository in the first place.
There are numerous caveats with the determination of "deleted" and
"renamed", and can give both false positives and false negatives. But
they are only meant as a helpful heuristic to give others a starting
point for an investigation, and the information provide so far is useful.
I do want to improve the equivalence classes (rename handling), but that
is for a future commit.
Signed-off-by: Elijah Newren <newren@gmail.com>
Commit messages often refer to past commits; while rewriting commits we
would also like to update these commit messages to refer to the new
commit names.
In the case that a commit message references another commit which was
dropped by the filtering process, we have no way to rewrite the commit
message to reference a valid commit hash. Instead of dying, note the
suboptimal commit in the suboptimal-issues file.
Signed-off-by: Elijah Newren <newren@gmail.com>
This will be used later to help with commit message rewriting (so that
commits can continue to refer to other commits in their history, using
the new rewritten hashes for those commits), and perhaps also in
removing blobs by id.
Signed-off-by: Elijah Newren <newren@gmail.com>
If ancient history that pre-dated some subdirectory had a few empty
commits, we would rather those all got pruned as well. Empty commits
from the original repository should only be retained if they have at
least one retained parent.
Signed-off-by: Elijah Newren <newren@gmail.com>
When the pruning of empty commits causes a culling of parents of a merge
commit, so that the merge commit drops to just one parent, the commit
likely becomes misleading since the commit is no longer a merge commit
but the message probably implies it is. (e.g. "Merge branch maint into
master"). There's nothing we can do to automatically fix this, but we
can note it as a suboptimal issue in the filtering process.
Signed-off-by: Elijah Newren <newren@gmail.com>
Our filtering process will rewrite (and drop) commits, causing refs to
also get updated. A useful debugging aid for users is to write metadata
showing the mapping from old commit IDs to new commit IDs, and from the
hash that old refs pointed and the hash that the new ones do.
Signed-off-by: Elijah Newren <newren@gmail.com>
In the previous commit, we detected when an entire line of history back to
a common ancestor of the merge became empty commits, and avoided having a
commit be merged with itself. This commits looks through the changes
specified in the commit, which are always specified relative to the first
parent, so that if the first parent side was the empty one we can still
detect if the merge commit adds no extra changes relative to its remaining
parent.
Signed-off-by: Elijah Newren <newren@gmail.com>
Pruning of empty commits can cause an entire line of history to become
empty and be pruned, resulting in a merge commit that merges some commit
with one of its ancestors. In such a case, we should remove the
unnecessary parent(s) -- which can and will often result in the merge
commit being empty so we can remove it as well.
Currently, if the side that becomes empty is the first parent side, then
we do not detect if the commit becomes empty, due to the way that
fast-export lists changes in a merge commit relative to first parent only.
A subsequent commit will address this.
Note that the callbacks could theoretically insert additional commits or
reparent our commit on top of something else, meaning that the ancestry
graph might need post-callback updates. However, in any extreme case
where that mattered, we would more or less need full updates to the
ancestry graph to be made for all the new commits from the callback as
well, and once we expect the callback to handle any ancestry graph
updates it can handle modifying it for the current commit. However, it
is hard to come up with a case where it matters, since for the most part
we just want to know whether our filtering causes commits to become
empty and knowing the source repo we are exporting from is sufficient
information without knowing any new commits inserted or reparenting that
happens elsewhere.
Signed-off-by: Elijah Newren <newren@gmail.com>
Use the 'feature done' ability to mark when the fast-import stream is
finished, so that an aborted run (due to running into some kind of bug
while filtering, whether a bug in the code, or an error in the repo or
flags specified for the case under consideration) won't cause the repo
to be rewritten.
Signed-off-by: Elijah Newren <newren@gmail.com>
It may be that the only time a reference is shown in the fast-export stream
is for a commit which will become empty due to the filtering. We do not
want such refs to be left out and thus not be updated; we want them to
instead be set to the nearest non-empty ancestor. Only if it has no
non-empty ancestor would we want it to be stripped out.
Signed-off-by: Elijah Newren <newren@gmail.com>
Slightly re-order the code to make input, output, and filtering sections
distinct. Also, avoid running `git fast-import` at all when we're in
--dry-run mode.
Signed-off-by: Elijah Newren <newren@gmail.com>
We always called record_id_rename with handle_transitivity set to True, and
I do not know of a use case that would do otherwise so let's just hardcode
that value.
Signed-off-by: Elijah Newren <newren@gmail.com>
My idea to use --export-marks and --import-marks to avoid the need for the
id_offset was not tested and apparently a bad idea. When splicing together
multiple repositories, the second will croak if we pass it --import-marks
with a file having sha1sums that don't exist in that repository.
I'm afraid this might conflict with the --import-marks stuff used in collab
so I've only enabled it for streams beyond the first. So there might be an
issue using --import-marks on a second or later fast-export output stream,
but I can't think of a use case for that...
Signed-off-by: Elijah Newren <newren@gmail.com>
Filtering input from multiple repositories can still be done; however, to
avoid overloading of mark numbers, one should pass --export-marks=<file>
to the first git fast-export and pass --import-marks=<file> to the second.
Signed-off-by: Elijah Newren <newren@gmail.com>
Elijah Newren
James Foucar