Archives
/
wikiteam
mirror of https://github.com/WikiTeam/wikiteam
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
1,098 Commits
3 Branches
0 Tags
81 MiB
master
PiRSquared17-patch-1
rewrite
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '25329be008'
${ noResults }
Commit Graph

1098 Commits (25329be0088b4ece8cc2656f3654e85b5e73a9af)
 

Author SHA1 Message Date
nemobis 25329be008
Merge pull request #441 from Pokechu22/mwclient-session 
Pass requests session to mwclient
 2 years ago
Pokechu22 5b3fc4ac7b Pass requests session to mwclient 
This means it uses our configured user-agent, as well as any cookies.
 2 years ago
nemobis 52fe2d89a6
Merge pull request #440 from Pokechu22/xmlrevisions-skip-empty-revision 
Skip empty revisions when using --xmlrevisions
 2 years ago
Pokechu22 1af69ca147 Skip empty revisions when using --xmlrevisions 
Before, the download would die, and need to be resumed from the start.
 2 years ago
nemobis 5d83703d50
Merge pull request #438 from Pokechu22/getXMLHeader-session 
Use `session.get` instead of `requests.get` in `getXMLHeader`
 2 years ago
Pokechu22 4a2cbd4843 Use `session.get` instead of `requests.get` in `getXMLHeader` 
`session.get` uses our configured User-Agent, while `requests.get` uses the default one.
 2 years ago
nemobis 9808279a6a
Merge pull request #436 from Pokechu22/unicode-resume 
Work around unicode titles not working with resuming and fix truncation when resuming
 2 years ago
Pokechu22 9b2c6e40ae Fix truncation when resuming 
There already was code that looks like it was supposed to truncate files, but it calculated the index wrong and didn't properly check all lines. It worked out, though, because it didn't actually call the truncate function.

Now, truncation occurs to the last `</page>` tag. If the XML file ends with a `</page>` tag, then nothing gets truncated. The page is added after that; if nothing was truncated, this will result in the same page being listed twice (which already happened with the missing truncation), but if truncation did happen then the file should no longer be invalid.
 2 years ago
Pokechu22 43945c467f Work around unicode titles not working with resuming 
Before, you would get UnicodeWarning: Unicode unequal comparison failed to convert both arguments to Unicode - interpreting them as being unequal. The %s versus {} change was needed because otherwise you would get UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-5: ordinal not in range(128). There is probably a better way of solving that, but this one does work.
 2 years ago
Federico Leva e33f14fce6 Support GiveUpGitHub 2 years ago
nemobis 269841c909
Merge pull request #431 from simonliu99/updatelists 
Update mediawiki wikifarm lists
 2 years ago
Liu d9885e0845 Update shoutwiki-spider to remove duplicates 2 years ago
Liu fcc4080b23 Update neoseeker.com.info instructions 2 years ago
Liu e7f7266550 Update fandom.com spider and remove duplicates 2 years ago
Liu 9c5c55342d Update miraheze.org spider and remove duplicates 2 years ago
Liu a96e482fd3 Add .DS_Store to .gitignore 2 years ago
Liu 4c970e358d Remove duplicates from wiki-site.com 2 years ago
Liu 74a8e9609f Update wiki-site.com spider and list 2 years ago
Liu ba7fab2e96 Add fandom-spider and update metadata and lists 2 years ago
Liu 49e41ee75d Update neoseeker.com spider and list 2 years ago
Liu 6346fd6553 Update shoutwiki.com spider and list 2 years ago
Liu f93988e9c6 Update fandom.com to HTTPS 2 years ago
Liu 91faa34529 Update shoutwiki.com list 2 years ago
Liu d6fe1d9ff8 Update battlestarwiki.org list 2 years ago
Federico Leva cd6d40d5ac Merge branch 'simonliu99-updatelists2' 2 years ago
Liu 6f8f160d75 Update fandom.com list 2 years ago
Liu 6b39402ebf Update miraheze.org list 2 years ago
Liu f755153de9 Update neoseeker.com list 2 years ago
Federico Leva 10ee80ca3b Rename wikia list to fandom 2 years ago
nemobis 054397aecb
Merge pull request #425 from simonliu99/append_date 
Display appended-date IA URL if appended
 2 years ago
Liu 3638e6992f Simplify tracking item identifier 2 years ago
Liu 94eb932b3f Display appended-date IA URL if appended 2 years ago
nemobis 1f911c0142
Merge pull request #424 from simonliu99/master 
Add dump date to item identifier
 2 years ago
Liu d947d7571a Allow date append only if not admin 2 years ago
Liu 9f9df1e0aa Update logic to only append date if identifier without date exists 2 years ago
Liu 6c137764cb Move identifier-date option behind flag 2 years ago
Liu 990d5dfb4f Add dump date to item identifier 2 years ago
nemobis d7b6924845
Merge pull request #408 from shreyasminocha/fix-resume-images 
Fix image resuming
 2 years ago
nemobis 7f1f9985f6
Merge pull request #419 from timgates42/bugfix_typos 
docs: Fix a few typos
 2 years ago
Tim Gates ecbcc6118e
docs: Fix a few typos 
There are small typos in:
- dumpgenerator.py
- wikiteam/mediawiki.py

Fixes:
- Should read `inconsistencies` rather than `inconsistences`.
- Should read `partially` rather than `partialy`.
 3 years ago
Shreyas Minocha e55de36cb7
Fix image resuming 3 years ago
nemobis 0cfde9e9d1
Merge pull request #394 from nsapa/nico_fix_1 
Nico's fixes (dumping wiki.dystify.com/CI fixes)
 4 years ago
Nicolas SAPA 5986467b12 Cleanup of link rot 
Lot of wiki in test_dumpgenerator.py doesn't exist anymore.
Remove them from the CI.
 4 years ago
Nicolas SAPA b289f86243 Fix getPageTitlesScraper 
Using the API and the Special:Allpages scraper should result in the same number of titles.
Fix the detection of the next subpages on Special:Allpages.
Change the max depth to 100 and implement an anti loop (could fail on non-western wiki).
 4 years ago
Nicolas SAPA 1048bc3275 skilledtests.com doesn't host a MediaWiki anymore 
http://skilledtests.com/wiki/ redirect to https://simcast.com,
something 'Powered by Microsoft News'
 4 years ago
Nicolas SAPA 320115fe5a Try to fix CI by using current URL for archiveteam.org 
In commit 966df37c54, emijrp changed http://archiveteam.org/ to https://www.archiveteam.org/
Today, https://archiveteam.org/index.php?title=Special:Version show a canonical URL of https://archiveteam.org/
So try to fix the CI by doing a s/www.archiveteam.org/archiveteam.org/g
 4 years ago
Nicolas SAPA e4b43927b9 Fixup description grab in generateImageDump 
getXMLPage() yield on "</page>" so xmlfiledesc cannot contains "</mediawiki>".
Change the search to "</page>" and inject "</mediawiki>" if it is missing to fixup the XML
 4 years ago
Nicolas SAPA eacaf08b2f Try to fix a broken HTTP to HTTPS redirect in generateImageDump() 
Some wiki fail to do the HTTP to HTTPs redirect correctly so try it ourself.
 4 years ago
Nicolas SAPA 7675b0d17c Add exception handler for requests.exceptions.ReadTimeout in getXMLPageCore() 
Treat a ReadTimeout the same as a ConnectionError (log the error & retry)
 4 years ago
Nicolas SAPA 4a5eef97da Update the default user-agent 
A ModSecurity rule block the old UA so switch to the current Firefox 78 UA.
 4 years ago