nemobis
cbd0905cba
Add 2k more URLs from another crawl
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@950 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
4a5c91d471
Typo in filename
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@947 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
eb60580e91
New URLs from Incola
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@946 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
53236811d9
Also reupload the dump when verified missing
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@945 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
bab70e31c0
Use temporary name for history archive too to avoid conflicts
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@944 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
a1c89623a4
Another intermediate update with results from one more run
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@943 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
403dc213ef
Issue 71: English-only match for an older case
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@942 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
a3fe96b782
Revamp uploader.py for wider usage: move issues to tracker; add options --help, --prune-directories, --prune-wikidump, --admin
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@940 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
b74d6f79ce
Reduce requests for existing items and remove whitespace: tested with wiki-smackdownneoseekercom_w
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@939 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
54f9798be0
Mark staging 7z files as .tmp to avoid uploading them by mistake
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@938 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
2bb3fd7a50
Add neoseeker.com, from mutante's wikistats farm list
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@937 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
a358ffbfe0
Issue 88: Escaped a bit too much, some HTML we really need
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@936 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
c95465ac14
Nice to see curl progress, but only for actual upload
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@935 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
0ef0a5b229
Actually update last-updated-date
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@934 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
Hydriz
16ec333e7d
Adding rewrite code so others can build on top of it
...
This is a partial rewrite of the dumpgenerator.py, and
is largely incomplete. I am no longer working on this
rewrite, so I am releasing it for others to build upon
it and work towards releasing DumpGenerator 2.0.
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@933 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
43074360b7
Experience shows 30 seconds is a more realistic timeout
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@932 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
991b96fbe2
Mark dump uploaded only if confirmed by curl exit code
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@931 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
8cf60a3285
Re-updated pavlo list with 30 s timeout
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@930 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
e2956e60a0
Replace some index.php with api.php where available
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@929 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
scottdb56
57da61aac6
a minor syntax-error fix in checkalive.pl
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@928 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
8da3f15b35
Some manual filtering
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@927 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
84a2f9d6dc
Add index.php discovery and other fixes, update checked lists consequently; bad input makes it spit ugly errors but it keeps going
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@926 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
eba5f2d54e
More gamepedia from their homepage
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@925 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
scottdb56
51ee9e9847
added a user-agent and another search string
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@924 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
009682a037
Sync API check needle with checkalive.pl, </api> is unreliable
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@923 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
ac8ed21b0c
Add Terraria, 270 wikis might be missing but where is the list?
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@922 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
01b177bcf2
Update raw list with scraper run by odder
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@921 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
scottdb56
0ac7e477f5
Re-formatting of readme-checkalive.txt
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@920 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
scottdb56
fc9207291a
readme for checkalive.pl
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@919 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
scottdb56
d9247aa0ba
Lots of changes - improved error handling, progress reporting and other minor changes
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@918 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
0ede45b7cf
Special:BadTitle works only in English wikis
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@917 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
034866a32e
Handle permissions-errors for wikis requiring login or whatever
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@916 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
2bdf8da30c
Add orain and gamepedia lists, might have mistakes
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@915 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
6c69d9800f
Followup, delay needs config; should be BC
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@914 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
7e412d1f17
update with export of http://www.shoutwiki.com/wiki/Category:Flat_list_of_all_wikis
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@911 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
emijrp
2f9985ac6f
instructions to compile LaTeX paper;
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@909 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
emijrp
a88797717c
first draft of paper;
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@908 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
emijrp
95c2228f36
creating directory for paper
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@905 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
emijrp
6912c8bc71
creating directory for research, papers, etc
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@904 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
55185467e1
Add delay to all checking and listing functions, crappy hosts die on them
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@902 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
scottdb56
fb87cd9951
This is the first publicly available version of this Perl script.
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@901 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
91cf5b4d08
Commit skeleton for Scott's use
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@900 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
72d43634d5
Update Wikia list
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@899 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
emijrp
9152b6861b
list of wikis extracted from WikiIndex dump, excluding most wikifarms
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@898 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
emijrp
29fc882754
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@897 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
f7c50f8ee5
Add retroshare
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@896 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
1fb5865166
Remove some obvious false positives
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@895 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
67eeed88e4
Add silly RSD discovery to checkalive.py and update wikis list
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@892 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
b8c90df787
Upload separately some already checked with the script
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@891 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago
nemobis
500e8ef350
Remove some more obvious duplicates including trailing slash
...
git-svn-id: https://wikiteam.googlecode.com/svn/trunk@890 31edc4fc-5e31-b4c4-d58b-c8bc928bcb95
11 years ago