eb9677fb13
It's rather pointless to try to get a sensible error management with os calls; just rely on wget -c to download the missing/incomplete files, even though this will cause some additional connections. |
10 years ago | |
|---|---|---|
| batchdownload | 10 years ago | |
| listsofwikis | 10 years ago | |
| research/paper-wikiteam-2014 | 11 years ago | |
| testing | 10 years ago | |
| .gitignore | 10 years ago | |
| LICENSE | 10 years ago | |
| README.md | 10 years ago | |
| commonschecker.py | 10 years ago | |
| commonsdownloader.py | 10 years ago | |
| commonssql.py | 10 years ago | |
| dumpgenerator.py | 10 years ago | |
| gui.py | 10 years ago | |
| requirements.txt | 10 years ago | |
| uploader.py | 10 years ago | |
| wikiadownloader.py | 10 years ago | |
| wikipediadownloader.py | 10 years ago | |
README.md
WikiTeam
We archive wikis, from Wikipedia to tiniest wikis
WikiTeam software is a set of tools for archiving wikis. They work on MediaWiki wikis, but we want to expand to other wiki engines. As of June 2014, WikiTeam has preserved more than 13,000 stand-alone wikis, several wikifarms, regular Wikipedia dumps and 24 TB of Wikimedia Commons images.
There are thousands of wikis in the Internet. Every day some of them are no longer publicly available and, due to lack of backups, lost forever. Millions of people download tons of media files (movies, music, books, etc) from the Internet, serving as a kind of distributed backup. Wikis, most of them under free licenses, disappear from time to time because nobody grabbed a copy of them. That is a shame that we would like to solve.
WikiTeam is the Archive Team (GitHub) subcommittee on wikis. It was founded and originally developed by Emilio J. Rodríguez-Posada, a Wikipedia veteran editor and amateur archivist. Many people have helped by sending suggestions, reporting bugs, writing documentation, providing help in the mailing list and making wiki backups. Thanks to all, especially to: Federico Leva, Alex Buie, Scott Boyd, Hydriz, Platonides, Ian McEwen, Mike Dupont and balr0g.
|
|
|
|
|
Quick guide
This is a very quick guide for the most used features of WikiTeam tools. For further information, read the tutorial and the rest of the documentation. You can also ask in the mailing list.
Requirements
Confirm you satisfy the requirements:
pip install --upgrade -r requirements.txt
or, if you don't have enough permissions for the above,
pip install --user --upgrade -r requirements.txt
Download any wiki
To download any wiki, use one of the following options:
python dumpgenerator.py http://wiki.domain.org --xml --images (complete XML histories and images)
If the script can't find itself the API and/or index.php paths, then you can provide them:
python dumpgenerator.py --api=http://wiki.domain.org/w/api.php --xml --images
python dumpgenerator.py --api=http://wiki.domain.org/w/api.php --index=http://wiki.domain.org/w/index.php --xml --images
If you only want the XML histories, just use --xml. For only the images, just --images. For only the current version of every page, --xml --curonly.
You can resume an aborted download:
python dumpgenerator.py --api=http://wiki.domain.org/w/api.php --xml --images --resume --path=/path/to/incomplete-dump
See more options:
python dumpgenerator.py --help
Download Wikimedia dumps
For downloading Wikimedia XML dumps (Wikipedia, Wikibooks, Wikinews, etc):
python wikipediadownloader.py (download all projects)
See more options:
python wikipediadownloader.py --help
Download Wikimedia Commons images
There is a script for this, but we have uploaded the tarballs to Internet Archive, so it's more useful to reseed their torrents than to re-generate old ones with the script.