arkiver
|
40063adcaf
|
Use wget-at with ZSTD.
|
4 years ago |
|
Arkiver2
|
831f79f0d9
|
Do not import warcio. Update version to 20200102.03.
|
5 years ago |
|
Arkiver2
|
cf3f6c7af9
|
Skip URL on status code 204. Update version to 20200102.02.
|
5 years ago |
|
Arkiver2
|
ac65b0a818
|
Update version to 20200102.01.
|
5 years ago |
|
Arkiver2
|
0eb4b6205a
|
Fix string joining.
|
5 years ago |
|
Arkiver2
|
ad2cf89404
|
Split off checking if URL was processed. Do not add URL without trailing / already added with trailing /.
|
5 years ago |
|
Arkiver2
|
d4d5c9a93f
|
Skip amp.reddit.com post pages.
|
5 years ago |
|
Arkiver2
|
4cf7bd18f0
|
Version 20190729.01; do not get page requisites from outlinks; do not pip install warcio.
|
5 years ago |
|
Arkiver2
|
8902255c76
|
Version 20190405.01; support www.reddit.com; support videos; support outlinks
|
5 years ago |
|
Arkiver2
|
9d1ea0c688
|
rewrite
|
5 years ago |
Arkiver2
|
c08fd59a29
|
reddit.lua: ignore urls, fixes
|
9 years ago |
|
Arkiver2
|
11aef69a32
|
pipeline.py: cookies!
|
9 years ago |
|
Arkiver2
|
38074381c4
|
cookies
|
9 years ago |
|
Arkiver2
|
e87a2e4a51
|
README.md
|
9 years ago |
|
Arkiver2
|
2dd4e29062
|
reddit.lua
|
9 years ago |
|
Arkiver2
|
9f531c900f
|
pipeline.py: use redd.it to discover comments
|
9 years ago |
|
Arkiver2
|
61dd537f15
|
Update pipeline.py
|
9 years ago |
|
Arkiver2
|
ff1bb532c6
|
pipeline.py
|
9 years ago |
|
Arkiver2
|
2ab52c1b3d
|
first files
|
9 years ago |