balr0g
|
fd6ea19b4b
|
config['api'] is set but empty; properly handle this
|
10 years ago |
nemobis
|
1ff96238eb
|
Denote as alpha until revamp is tested
Per emijrp who asked not to run dumps with this, at https://github.com/WikiTeam/wikiteam/issues/104#issuecomment-48039143
Currently proposed things to fix or check: https://github.com/WikiTeam/wikiteam/issues?milestone=1&state=open
|
10 years ago |
Emilio J. Rodríguez-Posada
|
89e3c3e462
|
standarize getImage* functions names
|
10 years ago |
Emilio J. Rodríguez-Posada
|
aaa1822759
|
improving image list downloader
|
10 years ago |
Emilio J. Rodríguez-Posada
|
88c9468c0e
|
improving image list downloader
|
10 years ago |
balr0g
|
3929e4eb9c
|
Cleanups and error fixes suggested by flake8 (pep8 + pyflakes)
|
10 years ago |
Emilio J. Rodríguez-Posada
|
c07b527e5d
|
adding session to getWikiEngine()
|
10 years ago |
Emilio J. Rodríguez-Posada
|
30c153ce1f
|
chg: using 'with open' for files
|
10 years ago |
balr0g
|
9aa3c4a0e1
|
Removed all traces of urllib except for encode/decode; more bugs fixed.
|
10 years ago |
balr0g
|
c8e11a949b
|
Initial port to Requests
|
10 years ago |
Emilio J. Rodríguez-Posada
|
9553e3550c
|
adding wiki engine detector
|
10 years ago |
Emilio J. Rodríguez-Posada
|
eb97cf1adf
|
version 0.2.2 and tiny bits in --help
|
10 years ago |
balr0g
|
50b011f90d
|
Initial port to argparse
|
10 years ago |
Emilio J. Rodríguez-Posada
|
568deef081
|
adding comments for clarification
|
10 years ago |
Emilio J. Rodríguez-Posada
|
d4eed1f738
|
fixing #127 and #134 , now works with APIs that returns 'name' field for images and those that don't do it (in this case we unquote over ascii); also fixing bug that re-download image list when it was completed previously
|
10 years ago |
Emilio J. Rodríguez-Posada
|
005de23c1d
|
adding gzip to siteinfo downloader
|
10 years ago |
Emilio J. Rodríguez-Posada
|
d79ea64d41
|
fixing issue #97 pretty siteinfo json saving, indenting 4 chars
|
10 years ago |
Emilio J. Rodríguez-Posada
|
3854a344fe
|
Merge branch 'master' of https://github.com/WikiTeam/wikiteam
|
10 years ago |
Emilio J. Rodríguez-Posada
|
1c1f0dbb86
|
replacing XML with JSON in image downloading
|
10 years ago |
balr0g
|
481323c7f7
|
Don't try to download sites with disabled API
|
10 years ago |
nemobis
|
1933db8a94
|
Merge pull request #124 from balr0g/scraper-unicode-title-fix
Fix scraper for sites with Unicode titles
|
10 years ago |
balr0g
|
62be069026
|
Fix scraper for sites with Unicode titles
|
10 years ago |
nemobis
|
62d961fa97
|
Fix typo, unused variable spotted by balrog
|
10 years ago |
nemobis
|
95bc2dec38
|
Link GitHub issue tracker
|
10 years ago |
balr0g
|
d60e560571
|
Add Content-Encoding: gzip support
|
10 years ago |
Emilio J. Rodríguez-Posada
|
5261811fa4
|
only if api exists
|
10 years ago |
Emilio J. Rodríguez-Posada
|
610764619a
|
add saveSiteInfo() to download meta=siteinfo data from API to a file
|
10 years ago |
Emilio J. Rodríguez-Posada
|
d395433513
|
comments and newlines
|
10 years ago |
Emilio J. Rodríguez-Posada
|
5eff4bd072
|
comments and tabs
|
10 years ago |
Emilio J. Rodríguez-Posada
|
0b0c40f5da
|
adding more user-agents, but keeps the first as default by now
|
10 years ago |
Emilio J. Rodríguez-Posada
|
81468c4a7c
|
using JSON to retrieve namespaces via API
|
10 years ago |
Emilio J. Rodríguez-Posada
|
703eb9011b
|
improving checkAPI() using JSON properly loaded
|
10 years ago |
Emilio J. Rodríguez-Posada
|
44d3fe1e36
|
Merge pull request #117 from nemobis/bug/48
Issue 48: Check that API actually works
|
10 years ago |
Emilio J. Rodríguez-Posada
|
fc80556d8a
|
merging...
|
10 years ago |
Emilio J. Rodríguez-Posada
|
f474deb71f
|
now we use JSON properly in getPageTitlesAPI(), instead of XML; fixing some wrong prints, now support utf-8
|
10 years ago |
Federico Leva
|
997276110c
|
Issue 46: dumpgenerator should follow redirects
Patch by @balr0g from libsonic (GPLv3+).
|
10 years ago |
Federico Leva
|
a8e1575879
|
Issue 48: Check that API actually works
|
10 years ago |
Emilio J. Rodríguez-Posada
|
c9aa165504
|
fixing header with the new year, info and documentation link
|
10 years ago |
nemobis
|
ac4c93c12a
|
Issue 85: more cross-platform shebang on all scripts
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@962 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
|
11 years ago |
nemobis
|
403dc213ef
|
Issue 71: English-only match for an older case
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@942 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
|
11 years ago |
nemobis
|
0ede45b7cf
|
Special:BadTitle works only in English wikis
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@917 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
|
11 years ago |
nemobis
|
034866a32e
|
Handle permissions-errors for wikis requiring login or whatever
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@916 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
|
11 years ago |
nemobis
|
6c69d9800f
|
Followup, delay needs config; should be BC
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@914 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
|
11 years ago |
nemobis
|
55185467e1
|
Add delay to all checking and listing functions, crappy hosts die on them
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@902 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
|
11 years ago |
nemobis
|
6efe406ea5
|
Followup r877, first check most common conditions for shortcut performance
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@882 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
|
11 years ago |
Hydriz
|
611d13f8c5
|
Follow up r877, check the number of revision tags
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@878 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
|
11 years ago |
Hydriz
|
64bd837cab
|
(Issue 34) XML integry check inside the code
This *really* fixes the issue and asks the user whether or not to regenerate a dump.
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@877 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
|
11 years ago |
Hydriz
|
79047a3ded
|
(Issue 71) Use a better check for private wikis
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@873 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
|
11 years ago |
nemobis
|
26873ad495
|
Fix typo, make domain2prefix quiet again
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@869 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
|
11 years ago |
nemobis
|
626118cfab
|
Let's call it 0.2 then, a bump to 1 would require announcements etc. We're not there yet (API support etc.).
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@867 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
|
11 years ago |