You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

230 lines
10 KiB
Markdown

2 years ago
reddit-grab
9 years ago
=============
2 years ago
More information about the archiving project can be found on the ArchiveTeam wiki: [Reddit](http://archiveteam.org/index.php?title=Reddit)
9 years ago
Setup instructions
=========================
Be sure to replace `YOURNICKHERE` with the nickname that you want to be shown as, on the tracker. You don't need to register it, just pick a nickname you like.
In most of the below cases, there will be a web interface running at http://localhost:8001/. If you don't know or care what this is, you can just ignore it—otherwise, it gives you a fancy view of what's going on.
**If anything goes wrong while running the commands below, please scroll down to the bottom of this page. There's troubleshooting information there.**
Running with a warrior
-------------------------
2 years ago
Follow the [instructions on the ArchiveTeam wiki](http://archiveteam.org/index.php?title=Warrior) for installing the Warrior, and select the "Reddit" project in the Warrior interface.
9 years ago
Running with Docker
-------------------------
The recommended way to run these projects is with Docker. The instructions below are a short overview. For more information and detailed explanations of the commands, follow the follow the [Docker instructions on the Archive Team wiki](https://wiki.archiveteam.org/index.php/Running_Archive_Team_Projects_with_Docker).
It is advised to use watchtower to automatically update the project. This requires watchtower:
docker run --name watchtower --restart=unless-stopped -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower --label-enable --cleanup --interval 3600
after which the project can be run:
2 years ago
docker run --name archiveteam --label=com.centurylinklabs.watchtower.enable=true --restart=unless-stopped atdr.meo.ws/archiveteam/reddit-grab --concurrent 1 YOURNICKHERE
Running without a warrior or Docker
9 years ago
-------------------------
To run this outside the warrior, clone this repository, cd into its directory and run:
python3 -m pip install setuptools wheel
python3 -m pip install --upgrade seesaw zstandard requests
9 years ago
./get-wget-lua.sh
then start downloading with:
run-pipeline3 pipeline.py --concurrent 2 YOURNICKHERE
9 years ago
For more options, run:
run-pipeline3 --help
9 years ago
5 years ago
If you don't have root access and/or your version of pip is very old, you can replace "pip install --upgrade seesaw" with:
9 years ago
wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py ; python3 get-pip.py --user ; ~/.local/bin/pip3 install --upgrade --user seesaw
9 years ago
so that pip and seesaw are installed in your home, then run
~/.local/bin/run-pipeline3 pipeline.py --concurrent 2 YOURNICKHERE
9 years ago
Running multiple instances on different IPs
-------------------------------------------
This feature requires seesaw version 0.0.16 or greater. Use `pip install --upgrade seesaw` to upgrade.
Use the `--context-value` argument to pass in `bind_address=123.4.5.6` (replace the IP address with your own).
Example of running 2 threads, no web interface, and Wget binding of IP address:
run-pipeline3 pipeline.py --concurrent 2 YOURNICKHERE --disable-web-server --context-value bind_address=123.4.5.6
9 years ago
Distribution-specific setup
-------------------------
### For Debian/Ubuntu:
Package `libzstd-dev` version 1.4.4 is required which is currently available from `buster-backports`.
9 years ago
adduser --system --group --shell /bin/bash archiveteam
echo deb http://deb.debian.org/debian buster-backports main contrib > /etc/apt/sources.list.d/backports.list
apt-get update \
&& apt-get install -y git-core libgnutls-dev lua5.1 liblua5.1-0 liblua5.1-0-dev screen bzip2 zlib1g-dev flex autoconf autopoint texinfo gperf lua-socket lua-filesystem lua-sec lua-zip rsync automake pkg-config python3-dev python3-pip build-essential poppler-utils \
&& apt-get -t buster-backports install zstd libzstd-dev libzstd1
python3 -m pip install setuptools wheel
python3 -m pip install --upgrade seesaw zstandard requests
2 years ago
su -c "cd /home/archiveteam; git clone https://github.com/ArchiveTeam/reddit-grab.git; cd reddit-grab; ./get-wget-lua.sh" archiveteam
screen su -c "cd /home/archiveteam/reddit-grab/; run-pipeline3 pipeline.py --concurrent 2 --address '127.0.0.1' YOURNICKHERE" archiveteam
9 years ago
[... ctrl+A D to detach ...]
In __Debian Jessie, Ubuntu 18.04 Bionic and above__, the `libgnutls-dev` package was renamed to `libgnutls28-dev`. So, you need to do the following instead:
5 years ago
adduser --system --group --shell /bin/bash archiveteam
echo deb http://deb.debian.org/debian buster-backports main contrib > /etc/apt/sources.list.d/backports.list
apt-get update \
&& apt-get install -y git-core libgnutls28-dev lua5.1 liblua5.1-0 liblua5.1-0-dev screen bzip2 zlib1g-dev flex autoconf autopoint texinfo gperf lua-socket lua-filesystem lua-sec lua-zip rsync automake pkg-config python3-dev python3-pip build-essential poppler-utils \
&& apt-get -t buster-backports install zstd libzstd-dev libzstd1
5 years ago
[... pretty much the same as above ...]
9 years ago
Wget-lua is also available on [ArchiveTeam's PPA](https://launchpad.net/~archiveteam/+archive/wget-lua) for Ubuntu.
### For CentOS:
THESE INSTRUCTIONS ARE OUTDATED
Ensure that you have the CentOS equivalent of bzip2 installed as well. You will need the EPEL repository to be enabled.
9 years ago
yum -y groupinstall "Development Tools"
yum -y install gnutls-devel lua-devel python-pip zlib-devel zstd libzstd-devel git-core gperf lua-socket luarocks texinfo git rsync gettext-devel
5 years ago
pip install --upgrade seesaw
9 years ago
[... pretty much the same as above ...]
Tested with EL7 repositories.
### For Fedora:
The same as CentOS but with "dnf" instead of "yum". Did not successfully test compiling, so far.
9 years ago
### For openSUSE:
THESE INSTRUCTIONS ARE OUTDATES
9 years ago
zypper install liblua5_1 lua51 lua51-devel screen python-pip libgnutls-devel bzip2 python-devel gcc make
5 years ago
pip install --upgrade seesaw
9 years ago
[... pretty much the same as above ...]
### For OS X:
THESE INSTRUCTIONS ARE OUTDATED
9 years ago
You need Homebrew. Ensure that you have the OS X equivalent of bzip2 installed as well.
brew install python lua gnutls
5 years ago
pip install --upgrade seesaw
9 years ago
[... pretty much the same as above ...]
2 years ago
**There is a known issue with some packaged versions of rsync. If you get errors during the upload stage, reddit-grab will not work with your rsync version.**
9 years ago
This supposedly fixes it:
alias rsync=/usr/local/bin/rsync
### For Arch Linux:
Ensure that you have the Arch equivalent of bzip2 installed as well.
1. Make sure you have `python2-pip` installed.
5 years ago
2. Install [the wget-lua package from the AUR](https://aur.archlinux.org/packages/wget-lua/).
3. Run `pip2 install --upgrade seesaw`.
9 years ago
4. Modify the run-pipeline script in seesaw to point at `#!/usr/bin/python2` instead of `#!/usr/bin/python`.
5. `useradd --system --group users --shell /bin/bash --create-home archiveteam`
2 years ago
6. `screen su -c "cd /home/archiveteam/reddit-grab/; run-pipeline pipeline.py --concurrent 2 --address '127.0.0.1' YOURNICKHERE" archiveteam`
9 years ago
5 years ago
### For Alpine Linux:
Install the dependencies for Zstd and Wget-AT:
apk update && apk add lua5.1 lua5.1-socket lua5.1-sec python3 py3-pip git gcc libc-dev \
lua5.1-dev lua5.1-filesystem zlib-dev gnutls-dev automake autoconf make bash \
bzip2 rsync flex gettext gettext-dev xz gperf texinfo wget coreutils ca-certificates \
poppler-utils
Version 1.4.4 of Zstd is required, so this needs to be built from source for compatibility with arguments provided from other Alpine builds:
# Source: https://git.alpinelinux.org/aports/tree/main/zstd/APKBUILD?h=3.15-stable
git clone https://github.com/facebook/zstd.git --depth 1 --branch v1.4.4
cd zstd && export CFLAGS="-O2" && \
make -C lib HAVE_PTHREAD=1 HAVE_ZLIB=0 HAVE_LZMA=0 HAVE_LZ4=0 lib-mt && \
make -C programs HAVE_PTHREAD=1 HAVE_ZLIB=0 HAVE_LZMA=0 HAVE_LZ4=0 && \
make -C contrib/pzstd && \
make PREFIX="/usr" install && \
cd ..
Close the repo and build Wget-AT:
2 years ago
git clone https://github.com/ArchiveTeam/reddit-grab
cd reddit-grab; ./get-wget-lua.sh
Run the project with
# uncomments the next line if you want to use a virtualenv (sh/bash example)
#python3 -m venv --prompt at .venv && source .venv/bin/activate
pip install --upgrade pip setuptools wheel
pip install --upgrade seesaw zstandard requests
run-pipeline3 pipeline.py --concurrent 2 --address '127.0.0.1' YOURNICKHERE
5 years ago
9 years ago
### For FreeBSD:
Honestly, I have no idea. `./get-wget-lua.sh` supposedly doesn't work due to differences in the `tar` that ships with FreeBSD. Another problem is the apparent absence of Lua 5.1 development headers. If you figure this out, please do let us know on IRC (irc.hackint.org #archiveteam).
9 years ago
Troubleshooting
=========================
Broken? These are some of the possible solutions:
### wget-lua was not successfully built
If you get errors about `wget.pod` or something similar, the documentation failed to compile - wget-lua, however, compiled fine. Try this:
cd get-wget-lua.tmp
mv src/wget ../wget-lua
cd ..
The `get-wget-lua.tmp` name may be inaccurate. If you have a folder with a similar but different name, use that instead and please let us know on IRC what folder name you had!
Optionally, if you know what you're doing, you may want to use wgetpod.patch.
### Problem with gnutls or openssl during get-wget-lua
Please ensure that gnutls-dev(el) and openssl-dev(el) are installed.
### ImportError: No module named seesaw
If you're sure that you followed the steps to install `seesaw`, permissions on your module directory may be set incorrectly. Try the following:
chmod o+rX -R /usr/local/lib/python2.7/dist-packages
5 years ago
### run-pipeline: command not found
Install `seesaw` using `pip2` instead of `pip`.
pip2 install seesaw
9 years ago
### Issues in the code
If you notice a bug and want to file a bug report, please use the GitHub issues tracker.
Are you a developer? Help write code for us! Look at our [developer documentation](http://archiveteam.org/index.php?title=Dev) for details.
### Other problems
2 years ago
Have an issue not listed here? Join us on IRC and ask! We can be found at hackint IRC [#shreddit](https://webirc.hackint.org/#irc://irc.hackint.org/#shreddit).
5 years ago