43 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
arkiver	7c5ea717a8	Version 20210108.03.	2021-01-08 22:42:48 +01:00
arkiver	5f3958c282	Merge branch 'master' of https://github.com/ArchiveTeam/reddit-grab	2021-01-08 22:42:24 +01:00
arkiver	1924d5217e	Version 20210108.02.	2021-01-08 22:42:03 +01:00
arkiver	16836ba201	Support single comment and post items. Queue outlinks to URLs project.	2021-01-08 22:40:49 +01:00
arkiver	ae57a81baf	Use multi items.	2021-01-08 22:40:09 +01:00
km09	eb945d2470	Use updated grab-base	2020-10-31 10:49:34 +00:00
arkiver	4284d24b47	Version 20201031.01. Support Wget-AT version 1.20.3-at.20201030.01.	2020-10-31 01:20:27 +01:00
arkiver	9ecf9a3a30	Version 20200902.01. Support Wget-AT version 1.20.3-at.20200902.01.	2020-09-02 09:46:17 -04:00
arkiver	99875895b6	Version 20200821.02. Set tracker host to trackerproxy.archiveteam.org.	2020-08-21 16:22:24 -04:00
arkiver	2087174a5c	Version 20200821.01. Ignore comment URL with utm_source param.	2020-08-21 16:12:06 -04:00
arkiver	ace1a4f037	Version 20200805.01. Support Wget-AT version 1.20.3-at.20200804.01.	2020-08-05 10:10:57 -04:00
arkiver	8b40429e95	Use new README template.	2020-07-29 19:16:48 -04:00
arkiver	23bfe8b12c	Version 20200730.01. Support /user/ post better (like /r/).	2020-07-29 18:40:05 -04:00
arkiver	450d4e0413	Version 20200728.01. Ignore non-reddit URLs. Fix extraction of tokens for morecomments.	2020-07-27 19:53:46 -04:00
arkiver	9a6417ecbc	Version 20200727.03. Fix handling video URLs without extension.	2020-07-27 10:37:39 -04:00
arkiver	869cdc4e6e	Remove unused cookies.txt file. Update README.	2020-07-26 20:40:10 -04:00
arkiver	911c675e74	Version 20200727.02. Set TRACKER_ID to reddittest.	2020-07-26 20:11:56 -04:00
arkiver	147c6416ed	Version 20200727.01. Use trackerproxy for dictionaries. Ignore irc: URLs.	2020-07-26 20:04:24 -04:00
arkiver	910687b053	Version 20200726.06. Fix project name for ZSTD dictionary request.	2020-07-26 17:44:05 -04:00
arkiver	496c018eef	Version 20200726.05. Add cookies to access some quarantines subreddits.	2020-07-26 17:20:17 -04:00
arkiver	3d5e7e17f9	Version 20200726.04. Use reddittest tracker for size estimate.	2020-07-26 16:59:20 -04:00
arkiver	23fec56409	Version 20200726.03. Support galleries and comments.	2020-07-26 16:43:41 -04:00
arkiver	2f6a602313	Version 20200726.01. Fully support new and old design for posts.	2020-07-26 16:05:38 -04:00
arkiver	56571306dd	Use default upload concurrent of 2.	2020-06-30 19:12:32 -04:00
arkiver	40063adcaf	Use wget-at with ZSTD.	2020-06-30 19:11:06 -04:00
Arkiver2	831f79f0d9	Do not import warcio. Update version to 20200102.03.	2020-01-02 17:43:53 +01:00
Arkiver2	cf3f6c7af9	Skip URL on status code 204. Update version to 20200102.02.	2020-01-02 17:41:38 +01:00
Arkiver2	ac65b0a818	Update version to 20200102.01.	2020-01-02 17:39:20 +01:00
Arkiver2	0eb4b6205a	Fix string joining.	2020-01-02 17:37:06 +01:00
Arkiver2	ad2cf89404	Split off checking if URL was processed. Do not add URL without trailing / already added with trailing /.	2019-07-30 01:49:46 +02:00
Arkiver2	d4d5c9a93f	Skip amp.reddit.com post pages.	2019-07-30 00:18:23 +02:00
Arkiver2	4cf7bd18f0	Version 20190729.01; do not get page requisites from outlinks; do not pip install warcio.	2019-07-29 22:58:09 +02:00
Arkiver2	8902255c76	Version 20190405.01; support www.reddit.com; support videos; support outlinks	2019-04-05 04:52:06 +02:00
Arkiver2	9d1ea0c688	rewrite	2019-02-22 01:15:18 +01:00
Arkiver2	c08fd59a29	reddit.lua: ignore urls, fixes	2015-07-19 22:58:23 +02:00
Arkiver2	11aef69a32	pipeline.py: cookies!	2015-07-06 18:46:00 +02:00
Arkiver2	38074381c4	cookies	2015-07-06 18:45:42 +02:00
Arkiver2	e87a2e4a51	README.md	2015-07-05 21:50:11 +02:00
Arkiver2	2dd4e29062	reddit.lua	2015-07-05 21:46:15 +02:00
Arkiver2	9f531c900f	pipeline.py: use redd.it to discover comments	2015-07-05 19:27:32 +02:00
Arkiver2	61dd537f15	Update pipeline.py	2015-07-05 17:48:53 +02:00
Arkiver2	ff1bb532c6	pipeline.py	2015-07-05 12:03:02 +02:00
Arkiver2	2ab52c1b3d	first files	2015-07-05 11:43:31 +02:00