emijrp
dd0f4a4593
350,000
1 year ago
nemobis
ef03cff447
Merge pull request #454 from yzqzss/patch-6
...
Fix a small syntax error in uploader.py
1 year ago
yzqzss
d7153f4c60
Update uploader.py
1 year ago
nemobis
0c4c54dc9e
Merge pull request #449 from yzqzss/patch-2
...
make `requests.session` to use `--retries` value
1 year ago
yzqzss
90a64c6a22
make `requests.session` to use `--retries` value
...
(default=5)
1 year ago
nemobis
9d614cf8ad
Merge pull request #444 from Pokechu22/wiki-engine-session
...
Use the same requests session for getting the wiki engine and checking API/index
2 years ago
Pokechu22
97146c6f01
Use the same requests session for getting the wiki engine and checking API/index
2 years ago
Pokechu22
6668999658
Update User-Agent to latest Firefox
2 years ago
nemobis
ea5e130517
Merge pull request #442 from Pokechu22/missing-image-description
...
Fix crash when the image description is missing for an image containing non-ascii characters
2 years ago
Pokechu22
cad7260d7c
Fix crash when the image description is missing for an image containing non-ascii characters
...
title is already unicode, so we shouldn't need to decode it (and don't in generateXMLDump).
2 years ago
nemobis
25329be008
Merge pull request #441 from Pokechu22/mwclient-session
...
Pass requests session to mwclient
2 years ago
Pokechu22
5b3fc4ac7b
Pass requests session to mwclient
...
This means it uses our configured user-agent, as well as any cookies.
2 years ago
nemobis
52fe2d89a6
Merge pull request #440 from Pokechu22/xmlrevisions-skip-empty-revision
...
Skip empty revisions when using --xmlrevisions
2 years ago
Pokechu22
1af69ca147
Skip empty revisions when using --xmlrevisions
...
Before, the download would die, and need to be resumed from the start.
2 years ago
nemobis
5d83703d50
Merge pull request #438 from Pokechu22/getXMLHeader-session
...
Use `session.get` instead of `requests.get` in `getXMLHeader`
2 years ago
Pokechu22
4a2cbd4843
Use `session.get` instead of `requests.get` in `getXMLHeader`
...
`session.get` uses our configured User-Agent, while `requests.get` uses the default one.
2 years ago
nemobis
9808279a6a
Merge pull request #436 from Pokechu22/unicode-resume
...
Work around unicode titles not working with resuming and fix truncation when resuming
2 years ago
Pokechu22
9b2c6e40ae
Fix truncation when resuming
...
There already was code that looks like it was supposed to truncate files, but it calculated the index wrong and didn't properly check all lines. It worked out, though, because it didn't actually call the truncate function.
Now, truncation occurs to the last `</page>` tag. If the XML file ends with a `</page>` tag, then nothing gets truncated. The page is added after that; if nothing was truncated, this will result in the same page being listed twice (which already happened with the missing truncation), but if truncation did happen then the file should no longer be invalid.
2 years ago
Pokechu22
43945c467f
Work around unicode titles not working with resuming
...
Before, you would get UnicodeWarning: Unicode unequal comparison failed to convert both arguments to Unicode - interpreting them as being unequal. The %s versus {} change was needed because otherwise you would get UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-5: ordinal not in range(128). There is probably a better way of solving that, but this one does work.
2 years ago
Federico Leva
e33f14fce6
Support GiveUpGitHub
2 years ago
nemobis
269841c909
Merge pull request #431 from simonliu99/updatelists
...
Update mediawiki wikifarm lists
2 years ago
Liu
d9885e0845
Update shoutwiki-spider to remove duplicates
2 years ago
Liu
fcc4080b23
Update neoseeker.com.info instructions
2 years ago
Liu
e7f7266550
Update fandom.com spider and remove duplicates
2 years ago
Liu
9c5c55342d
Update miraheze.org spider and remove duplicates
2 years ago
Liu
a96e482fd3
Add .DS_Store to .gitignore
2 years ago
Liu
4c970e358d
Remove duplicates from wiki-site.com
2 years ago
Liu
74a8e9609f
Update wiki-site.com spider and list
2 years ago
Liu
ba7fab2e96
Add fandom-spider and update metadata and lists
2 years ago
Liu
49e41ee75d
Update neoseeker.com spider and list
2 years ago
Liu
6346fd6553
Update shoutwiki.com spider and list
2 years ago
Liu
f93988e9c6
Update fandom.com to HTTPS
2 years ago
Liu
91faa34529
Update shoutwiki.com list
2 years ago
Liu
d6fe1d9ff8
Update battlestarwiki.org list
2 years ago
Federico Leva
cd6d40d5ac
Merge branch 'simonliu99-updatelists2'
2 years ago
Liu
6f8f160d75
Update fandom.com list
2 years ago
Liu
6b39402ebf
Update miraheze.org list
2 years ago
Liu
f755153de9
Update neoseeker.com list
2 years ago
Federico Leva
10ee80ca3b
Rename wikia list to fandom
2 years ago
nemobis
054397aecb
Merge pull request #425 from simonliu99/append_date
...
Display appended-date IA URL if appended
2 years ago
Liu
3638e6992f
Simplify tracking item identifier
2 years ago
Liu
94eb932b3f
Display appended-date IA URL if appended
2 years ago
nemobis
1f911c0142
Merge pull request #424 from simonliu99/master
...
Add dump date to item identifier
2 years ago
Liu
d947d7571a
Allow date append only if not admin
2 years ago
Liu
9f9df1e0aa
Update logic to only append date if identifier without date exists
2 years ago
Liu
6c137764cb
Move identifier-date option behind flag
2 years ago
Liu
990d5dfb4f
Add dump date to item identifier
2 years ago
nemobis
d7b6924845
Merge pull request #408 from shreyasminocha/fix-resume-images
...
Fix image resuming
2 years ago
nemobis
7f1f9985f6
Merge pull request #419 from timgates42/bugfix_typos
...
docs: Fix a few typos
2 years ago
Tim Gates
ecbcc6118e
docs: Fix a few typos
...
There are small typos in:
- dumpgenerator.py
- wikiteam/mediawiki.py
Fixes:
- Should read `inconsistencies` rather than `inconsistences`.
- Should read `partially` rather than `partialy`.
2 years ago