Commit Graph

204 Commits (eb97cf1adf8fe36706f7e147e5ccea744288c94d)

Author SHA1 Message Date
nemobis 7c94815e2c Issue 68: Use GET, not POST, to download images; some harm and no? good
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@851 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis e1b34b7f6b Fix whitespace
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@844 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis 57e226c049 Use urllib2 and set user agent in some more places; some webhosts block urllib.
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@842 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
Hydriz 414fb1988f Add a little more singular/plural support.
Along with this commit:
* Changed original %s to %d for number of edits, since it is recognised as an integer.
* Directly defined the number of edits to be 1 when the if condition is true, to optimise performance.


git-svn-id: https://wikiteam.googlecode.com/svn/trunk@841 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis 13ebee4c28 Issue 60: Add authentication cookie support, patch by Fredrik Roubert
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@840 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis b34f01646a Use getUserAgent in one more place, urllib is blocked by some
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@839 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
Hydriz 8b4480be64 Adding plural support for number of edits saved for a page
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@831 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis ed8d174d5a Issue #61: some skins hide that stuff, use meta tag generator
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@829 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
emijrp f1874656ed comments
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@818 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
12 years ago
emijrp af81adebeb comments
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@817 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
12 years ago
emijrp 424588a55c comments
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@816 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
12 years ago
emijrp 360d1c1fa1 fixing createnewdump() and resumepreviousdump()
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@815 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
12 years ago
emijrp 8295990df0 moving code to functions; tiny changes in comments
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@814 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
12 years ago
emijrp 79a310c470 tiny changes in comments, some clarifications
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@813 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
12 years ago
emijrp 0d7d73d420 ignore case
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@812 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
12 years ago
emijrp 5dddc9c502 tiny changes in comments, some clarifications
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@811 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
12 years ago
emijrp 73bdc5a5dc caps
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@810 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
12 years ago
emijrp 5f4d10dd5a 2013
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@809 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
12 years ago
emijrp a78f390557 fixing issue 56
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@806 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
12 years ago
nemobis 0080197b8f WARNING: API support is experimental (faster but broken).
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@805 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
12 years ago
Hydriz 240b9f9dac Add support for Wikidata and Wikivoyage.
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@804 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
12 years ago
emijrp a850194829 rewriting the inline help a bit
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@798 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
12 years ago
emijrp a57a490513 fixing file description bug
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@796 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
12 years ago
emijrp 36ea489313 fixing file description bug
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@795 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
12 years ago
emijrp e40eb9b826 disable check integrity function by now
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@758 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
12 years ago
emijrp c8ca525ff7 just adding documentation and comments
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@743 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
12 years ago
emijrp d616bcc2e6 just adding documentation and comments
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@742 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
12 years ago
Hydriz bbffb24a0c (Issue 34) XML integry check inside the code
An attempt to integrate a XML integrity checker into the script. If
the dump integrity check fails, the script will be redumped from the
start.


git-svn-id: https://wikiteam.googlecode.com/svn/trunk@710 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
12 years ago
nemobis 7e083de24b Partial revert of r675: the script was always downloading only the last revision, as reported by Hydriz... *facepalms*
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@709 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
12 years ago
nemobis 9f58ca6a9b Fix Issue 26: Error loop "XML for ... is wrong"
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@675 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
12 years ago
nemobis a8479f9936 Get namespaces and images info from API. Code badly copied around, we should probably use modules such as http://packages.python.org/simplemediawiki/ to use API. Tested, seems to work.
Fixed some typos.

git-svn-id: https://wikiteam.googlecode.com/svn/trunk@668 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp c73a535ea7 adding &templates param for Special:Export, useful when other software use getXML() function to export wiki pages, e.g.: WikiEvidens
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@610 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
nemobis 721eb8c596 Added link to api.php or index.php being checked, so that if you called the script via another script you can understand something.
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@588 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
nemobis fac6c33846 google, not gmail; upload docs
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@572 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
Hydriz b680b07709 Fixing sentence to avoid having 2 commas too close to each other
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@506 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp 7e0ebdbff4 link to doc
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@461 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp 4a5e1bc880 converting api check and indexphp check to fuctions
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@395 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp a5655e7885 notebook, simple menu
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@364 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp 8e03c6ccf3 years
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@344 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp 0c40b02aac new version caring about long filenames and weird chars in url
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@343 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp 15189ce31a header
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@263 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp 3c43b3cde3 fixing issue #39 wikia images error
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@261 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp d408c06b77 randomtitle = 'Main_Page' #previously AMF5LKE43MNFGHKSDMRTJ
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@225 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp 76897b09b2 retries for image list retrieve;
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@224 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp 31ed2aff33 moving some more GET requests to POST;
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@223 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp 60f7fa88a1 \n
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@222 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp e933c780b5 doc
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@221 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp fb4aa9e9dd doc
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@220 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp d1b2c2cd96 doc
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@219 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp 2caf754ef4 moving getImageFilenamesURL() from GET to POST; some comments;
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@218 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp 0746f49db0 moving some GET requests to POST: index.php check and namespaces get
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@217 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp a865564713 .lower()
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@216 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp e61f1ef974 doc
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@215 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp 6c42144f89 adding --force parameter to skip wiki[mp]edia projects alert
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@214 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp 13dfc9be88 adding empty docstrings;
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@200 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp c489de6533 issue #14
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@197 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp df3bd03336 prints
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@194 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp b25bf51a59 prints
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@193 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp 42640f3781 image limit errors improved
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@192 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp 4a0fb86751 index.php detector improved
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@191 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp 778a8ad7ae adding support to download images on old mediawikis; regexp4;
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@165 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp 5b7674edb7 params in main() to call it from external scripts (using import dumpgenerator; dumpgenerator.main(params=params))
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@161 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp e4d0de09d6 params in main() to call it from external scripts (using import dumpgenerator; dumpgenerator.main(params=params))
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@160 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp da6c4d1468 print error output when splitters error occurs; 5000 -> 500 in image list parser
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@158 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
13 years ago
emijrp c5b6b8a866 md5.new->md5; new uploaded wikis;
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@157 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp 835158b79b more wikis; md5 or hashlib
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@154 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp c7b511ee38 more wikis; new content spliter;
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@152 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp 9a20327939 undoHTMLEntities for titles when scrapped from Allpages; protocol http https;
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@150 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp af64c6b6c5 http or https
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@149 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp e9f379b888 fixing print;
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@147 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp b26a86cdd5 verbose false for imagedump;
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@144 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp 17235cffd9 better comments;
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@140 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp 6e45398878 when full history fails, retrieve only the last version; various server errors handled
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@139 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp e4b233cc37 print verbose; seconds numbers to variables;
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@138 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp 7b85e243b0 :
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@137 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp 3c49a30764 fixing issue #12 and issue #13;
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@136 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp 3388c7e83b print
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@135 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp 101cd62e3d print
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@134 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp 46303d780d fixing issue #11;
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@133 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp 1f91b4c63e fixing issue #11;
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@132 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp e13b4a6428 removing http/https for file prefixes;
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@126 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp 79f961aef5 more wikis;
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@122 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp d6c1a773a0 removing some old #fix comments; moving TODO to Issues section at Google Code;
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@119 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp 88cbc0e871 removing some old #fix comments;
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@118 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp 23c9a06a31 removing some old #fix comments;removing thread option not implemented and better not to be done;
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@117 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp 05d1fb97c4 sorting titles; fixing issue #9;
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@116 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp 200baf406c fixing ampersand & and ' issues while retrieving XML (issue #2)
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@91 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp 727e4dbd7c pause and reload xml when no </siteinfo> is found
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@90 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp 6cb2575b1b empty lines
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@85 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp f7d5519757 issue #4
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@83 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp d580f53e00 file reading performance: f.read() -> for l in f:
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@82 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp ff42a64776 xml file reading performance
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@81 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp bb61e71db2 'all' -> ['all'] list
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@78 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp a618c228db adding support for excluding namespaces
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@77 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp 0bec4c7e69 api.php and index.php checker;
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@76 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp 94aa7b0a40 fixed error with index.php; add a new content splitter
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@73 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp f9179049fe moving filenamelimit into other{}
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@72 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp 0b4ff64501 fixing issue #3
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@71 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp 8812df3ff8 user input improved
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@59 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago
emijrp a38a0922c5 another image regexp;
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@56 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
14 years ago