Commit Graph

1056 Commits (5986467b12a9eef5731142fadd98ac6ea67b3b85)
 

Author SHA1 Message Date
emijrp aec3a14b7b update spider incomplete results, still running; userwikispacesXY lists instead 6 years ago
emijrp 98386e0b4c codification, wikilicense 6 years ago
emijrp 3ee22b27d0 codification try 6 years ago
emijrp 51ebefa1c4 100,000 wikispaces 6 years ago
emijrp 9fb8d4be0e file check 6 years ago
emijrp 8c30b3a2b9 bug invalid content, redownload 6 years ago
emijrp 7280c89b3b duckduckgo spider 6 years ago
emijrp 83158d4506 70k wikis by spider 6 years ago
emijrp 4b483c695b Merge branch 'master' of https://github.com/WikiTeam/wikiteam 6 years ago
emijrp 60704e3303 searching wikis with duckduckgo 6 years ago
Federico Leva 7c545d05b7 Fix UnboundLocalError and catch RetryError with --xmlrevisions
File "./dumpgenerator.py", line 1212, in generateImageDump
    if not re.search(r'</mediawiki>', xmlfiledesc):

UnboundLocalError: local variable 'xmlfiledesc' referenced before assignment
6 years ago
Federico Leva 952fcc6bcf Up version to 0.4.0-alpha to signify disruption 6 years ago
Federico Leva 33bb1c1f23 Download image description from API when using --xmlrevisions
Fixes https://github.com/WikiTeam/wikiteam/issues/308

Also add --failfast option to sneak in all the hacks I use to run
the bulk downloads, so I can more easily sync the repos.
6 years ago
Federico Leva b8909baa3d Update Wikia list with wikia.py 6 years ago
Federico Leva be5ca12075 Avoid generators in API-only export 6 years ago
Fedora ebc02a3b45 Merge branch 'master' of https://github.com/WikiTeam/wikiteam 6 years ago
Fedora a8cbb357ff First attempt of API-only export 6 years ago
Fedora 142b48cc69 Add timeouts and retries to increase success rate 6 years ago
emijrp 60a0ba2e54 sleep 6 years ago
emijrp 061709d9e6 50,000 wikis, do not use this list, use wikispacesXY instead 6 years ago
emijrp 5002eb723a print 6 years ago
emijrp 30a6dc268b wikispaces lists 6 years ago
emijrp e01b2fb0c3 bug wikitext 6 years ago
emijrp ffff6cf568 sleep 6 years ago
emijrp 24ba4ae0ca originalurl metadata 6 years ago
emijrp cd90d30aaa ia checking 6 years ago
emijrp af680ced4a help, params 6 years ago
emijrp 2fe1c0b6b2 uploader included 6 years ago
emijrp 254486af06 param 6 years ago
emijrp 9ab9c64df2 bug in redirects; script accepts wikilist.txt now 6 years ago
emijrp 0574b5f33a second version, it downloads all, including sitemap and mainpage 6 years ago
emijrp 145b040784 update, 10000 wikis, still more arriving 6 years ago
emijrp 0b2dd6f8f8 Merge branch 'master' of https://github.com/WikiTeam/wikiteam 6 years ago
emijrp 557323d85e index path 6 years ago
emijrp cfb225ea5e first version, wikispaces downloader 6 years ago
nemobis e4a384e721
Merge pull request #303 from nemobis/master
Add alive MediaWikis from the WikiTeam archive.org collection
6 years ago
Federico Leva 293da80da9 Add alive MediaWikis from the WikiTeam acrhive.org collection 6 years ago
nemobis f33f316500
Merge pull request #302 from nemobis/master
Wikia dumps now use 7z, not gz
6 years ago
Federico Leva 6a34bf65ea Wikia dumps now use 7z, not gz
Note that existence doesn't mean the dump is usable.
6 years ago
nemobis 23efbefda8 Merge pull request #298 from mirkosertic/patch-1
http://www.mirkosertic.de is no longer powered by DokuWiki
7 years ago
Mirko Sertic c9fc4d2105 http://www.mirkosertic.de is no longer powered by DokuWiki
Removed http://www.mirkosertic.de from the list.
7 years ago
emijrp 9fccd3f4da disabling travis notifications 7 years ago
emijrp 0e20be9a6e sort 7 years ago
emijrp bbdaf7723b update neoseeker 7 years ago
emijrp fc48c895ae update info 7 years ago
emijrp c7d5f9bb2e update, 2244 wikis 7 years ago
emijrp 75e7628a11 now get ALL wikis, even closed ones 7 years ago
emijrp 10072a20eb . 7 years ago
Hydriz a8270a7769 Update Miraheze wiki farm 7 years ago
Hydriz 9fd6df7a3c Scan for closed wikis as well 7 years ago