The existance of a header has already been specified in the documentation.
Further adapt it to the real text implemented now.
Signed-off-by: Markus Heidelberg <markus.heidelberg@web.de>
The wording "exact paths" appears to not be clear enough for folks and I
keep repeatedly getting bug reports about filter-repo not following
renames. Make it very explicit.
Signed-off-by: Elijah Newren <newren@gmail.com>
Like --replace-text, add an option --replace-message which replaces text
in commit/tag message bodies, so that users can easily replace text
without constructing a --message-callback.
Signed-off-by: Gwyneth Morgan <gwymor@tilde.club>
Signed-off-by: Elijah Newren <newren@gmail.com>
The "Filtering based on many paths" section includes this code snippet,
```
regex:^.*/.*/[0-9]{4}-[0-9]{2}-[0-9]{2}.txt$
```
and this text
```
files whose name
was of the form YYYY.MM-DD.txt at least two subdirectories deep
```
Update the text to YYYY-MM-DD.txt to correctly match the regex
in the code snippet.
Signed-off-by: Cody Martin <codytylermartin@gmail.com>
Some people don't like htmlpreview.github.io. I once or twice saw a
case where it appeared to be affected by load limits. Since external
sites are making the manual available, and it's unlikely there are too
many changes between the last release and the current manual, just link
to it as an alternative for folks.
Signed-off-by: Elijah Newren <newren@gmail.com>
fast-import gained a new raw-permissive date format explictly for
allowing people to import repositories as-is. Make use of the flag, and
stop rewriting the bogus timezone found in rails.git.
If users do not like these bogus times, they can of course write a
filter to fix them (or even make them bogus in a different way). For
example:
git filter-repo ... --commit-callback '
if commit.author_date.endswith(b"+051800"):
commit.author_date.replace(b"+051800", b"+0261")
'
Signed-off-by: Elijah Newren <newren@gmail.com>
Useful commit and reference mappings are created on every run. These are
helpful in a number of situations, and should be documented so that
end-users and Git hosts can understand how to use the output.
The commit-map is particularly useful for Git hosts to override
retention mechanisms, like hidden refs. This allows end-users to purge
large files and sensitive data.
Signed-off-by: James Ramsay <james@jramsay.com.au>
Online blogs/articles/Q&A as well as direct feedback suggests that
people use the --force flag rather cavalierly. Add words like
"irreversible" and "immediate pruning" to discourage such blithe
application of this flag. I hope this encourages folks to either learn
the ramifications of irreversible full-repository entire history
rewrites first, or to follow the recommendation of only operating on a
fresh clone.
Signed-off-by: Elijah Newren <newren@gmail.com>
People keep trying to read this file, unaware that it is the source code
for generating the documentation, not the generated documentation. Add
a comment at the top that explains this and points people in the right
direction.
Signed-off-by: Elijah Newren <newren@gmail.com>
Allow lines starting with '#' to be treated as a comment and be ignored.
Update the documentation to note that both blank lines and comment lines
are ignored, and mention how filenames starting with '#' can be matched
(namely, the same way that filenames startwith with 'regex:', 'glob:',
or 'literal:' can be -- by prefixing the filename with 'literal:').
Signed-off-by: Elijah Newren <newren@gmail.com>
Flags like --local, --shared, --reference (and --dissociate), and
--origin would all mess up the fresh clone checker. Attempting to
defend against all of them would not only be costly, but make it harder
to draw the line about guesses as to whether a repository is a fresh
clone or not. --origin also has problems in that filter-repo has
special handling for the 'origin' remote that I don't want to apply to
other random remotes.
Flags like --depth, --single-branch, and --no-tags could prevent enough
data from being downloaded to do a full rewrite and result in a
partially rewritten or possibly even corrupt history (no idea how
shallow clones interact; probably badly). --filter would also make the
repo start without enough info though it'd at least be downloaded on
demand; it'd still be a really slow way to do it, though, so it's a bad
idea.
filter-repo doesn't really provide an easy mechanism to rewrite a repo
and its submodule simultaneously, so recursing submodules seems useless
and unhelpful. --shallow-submodules would be bad for at least the same
reasons --depth is for the parent module, assuming we handled
submodules. --remote-submodules just provides a way to make the repo
dirty to start, which is counter-productive. --jobs could be useful, if
recursing submodules was.
--no-checkout might be safe to use and --sparse might also be okay for
as long as it only affects the working tree, but in both cases why not
go --bare or --mirror if you're doing that? Likewise, --no-hardlinks is
useless given that we're already saying people need to use --no-local.
-b would be okay to use, but why wouldn't you just change the default
branch on the server rather than just within this one clone used for
rewriting the history? Whether you push back to the original repository
or to a new repo, you'd have to take a separate step to change it in
that remote repo. And if you really will use this new local repository
as the official source, then you can switch branches at the end of the
rewrite just as easily.
--separate-git-dir and --template might be okay to use, I haven't
tested. If either doesn't work now, or breaks at any point in the
future, I feel much better being able to say, "I told you to only use
these three flags to git clone."
-u only affects the ability to receive the clone; it's fine to use.
Also, -q only affects the console output during the clone operation, so
you could use it.
There will probably be more flags added to git-clone over time. Testing
against all of them is insanity. Recommend people only use --no-local,
--bare, and --mirror, with the first only needed when cloning from a
local filesystem, and the other two never needed but allowed for those
that prefer.
Signed-off-by: Elijah Newren <newren@gmail.com>
The desire to format or lint code throughout history has arisen several
times. It's more natural to do this in filter-branch since it somewhat
forces people to run external commands, but we have an example contrib
demo that shows how to run an external command on each file in history
that I created even before any of these requests came in and yet I still
periodically get requests about it.
Make lint-history ever-so-slightly easier to apply to a subset of
filenames, and include its usage as an extra cheat sheet comparison for
filter-branch-vs-filter-repo commands.
Signed-off-by: Elijah Newren <newren@gmail.com>
I occasionally get people doing special things, or see people
recommending to others to just use --force. Add some explanations
behind the safety checks so that those doing special things know when
it's okay, and to explain why it's a really bad idea to casually or
haphazardly recommend others use --force.
Signed-off-by: Elijah Newren <newren@gmail.com>
Commit 5e04dff097 (filter-repo: add new --no-ff option, 2020-01-01)
added support for a --no-ff option, but only added documentation in the
built-in output, not in the intended-to-be-more-complete manual. Add
documentation to the manual for this option.
Signed-off-by: Elijah Newren <newren@gmail.com>
I had lots of examples of these being horribly mis-used and being used in place
of each other; add some examples with some description of the repository layout
to try to avoid all that confusion.
Signed-off-by: Elijah Newren <newren@gmail.com>
fast-export/fast-import only work with filenames (using full path from
the root of the repository); thus that's all that filter-repo works
with. Full pathnames implicitly include all leading directories as part
of the pathname, which is what allows us to match against directories.
However, it obviously means --use-base-name can't be used to match paths
against directories.
Signed-off-by: Elijah Newren <newren@gmail.com>
Cloning local repos by default makes a bunch of hardlinks, giving you a
non-packed repository, and leading folks to use and suggest --force.
That, of course, bypasses the important fresh clone checks to prevent
people from accidentally and irrecoverably deleting their non-backed-up
data. Let's make it easier for people to avoid (and suggest) that
mistake.
Signed-off-by: Elijah Newren <newren@gmail.com>
You can call bytes.replace() or re.sub(), but you can't call
bytes.sub(). Oops. Fix the example in the documentation.
Reported-by: John Gietzen <john@gietzen.us>
Signed-off-by: Elijah Newren <newren@gmail.com>
git fast-import, in an attempt to be friendly, allows the same file to
be specified multiple times within a commit and just takes the last
version of the file listed. It determines whether files are the same
via fspathncmp, which is defined differently depending on the setting of
core.ignorecase. Unfortunately, this means that if someone is trying to
do filtering of history and using a broken (case-insensitive) filesystem
and the history they are filtering has some paths that differed in name
only, then fast-import will delete whichever of the "colliding" files is
listed first.
Avoid these problems by just turning off core.ignorecase while
fast-import is running. This will prevent silently modifying the repo
in an unexpected way. Users on such filesystems may have difficulty
checking out commits with files which differ in case only, but that is
a separate problem for them to deal with after rewriting history.
Signed-off-by: Elijah Newren <newren@gmail.com>
A few changes:
* Include notes about git-2.24.0 changes
* Make it clearer that messing with the first parent could have
negative side-effects if the file_changes aren't also updated.
* Fix wrapping of a line that was too long.
Also, update the README.md:
* Note the upstream improvements made in (not yet released) git-2.24.0
Signed-off-by: Elijah Newren <newren@gmail.com>
Partial history rewrites were possible before with the (previously
hidden) --refs flag, but the defaults were wrong. That could be worked
around with the --source or --target flags, but that disabled --no-data
for fast-export and thus slowed things down, and also would require
overridding --replace-refs. And the defaults for --source and --target
may diverge further from what is wanted/needed for partial history
rewrites in the future.
So, add --partial as a first-class supported option with scary
documentation about how it permits mixing new and old history. Make
--refs imply that flag. Make the behavioral similarities (in regards to
which steps are skipped) between --source, --target, and --partial more
clear. Add relevant documentation to round it out.
Signed-off-by: Elijah Newren <newren@gmail.com>
Allow folks to periodically update the export of a live repo without
re-exporting from the beginning. This is a performance improvement, but
can also be important for collaboration. For example, for sensitivity
reasons, folks might want to export a subset of a repo and update the
export periodically. While this could be done by just re-exporting the
repository anew each time, there is a risk that the paths used to
specify the wanted subset might need to change in the future; making the
user verify that their paths (including globs or regexes) don't also
pick up anything from history that was previously excluded so that they
don't get a divergent history is not very user friendly. Allowing them
to just export stuff that is new since the last export works much better
for them.
Signed-off-by: Elijah Newren <newren@gmail.com>
Elijah Newren
Jonathan Malmaud
Todd Zullinger
Mo-Gul
Markus Heidelberg
Gwyneth Morgan
Cody Martin
Tom Matthews
James Ramsay
Kate F