torrent-paradise/README.md

# About
## What is this?

If you don't know what Torrent Paradise is, see the [website](https://torrent-paradise.ml/about.html).

This is a repository of all the tools I use to build and run torrent-paradise.ml. The 'code name' of the project is nextgen (next gen torrent search), so don't be surprised if it comes up somewhere.

## Can you help me?
Maybe, open an issue. Be sure to demonstrate an effort that you tried to solve the problem yourself.

# Setup

Here's what the setup looks like rn:
- VPS, Debian Stretch, 2 GB RAM
  - PostgreSQL 9.6. pg_hba.conf contains this:

    ```
    local   all             all                                      peer
    # IPv4 local connections:
    host    nextgen         nextgen          localhost               md5
    ```
  - IPFS v0.4.18
  - user with username nextgen on the server
- my laptop w/ Linux
  - Go toolchain installed
  - node v10.15 & npm
  - Python 3 (required only for index-generator/fix-metajson.py)

The programs create their own tables in the DB that they need. Database name is "nextgen".

What I did first after getting the server up and running was importing the TPB dump. Download https://thepiratebay.org/static/dump/csv/torrent_dump_full.csv.gz to the import-tpb-dump directory and run `go run`.

There is a complete database dump available in torrentparadise-staticbackup.torrent, so you don't have to do that. This same database dump is available on https://mega.nz/#!ddESlChb!3YBqfxG-a4fwpXzPG3QsXa-C6FeQ9AbNSGXxY7W7xm4. It contains the same data as the torrent, only .xz compressed.

# Usage

## Generate the index

See `update-index.sh`.

## Spider the DHT

Run `go build` in spider/ to compile and scp the binary it to the server. You can use the systemd service file in `spider/spider.service` to start the spider on server boot.

## Scraping trackers for seed/leech data

Run `go build` in seedleech-daemon/ to compile and scp the binary it to the server. You can use the systemd service file in `seedleech-daemon/seedleech.service`.


# Contributing

Before working on something, open an issue to ask if it would be okay. I would love to [KISS](https://en.wikipedia.org/wiki/KISS_principle).
Clean up Before publishing to Github 2019-01-31 12:51:03 +00:00			`# About`
			`## What is this?`

Redo tracker scraping, index generation Seed/leech counts are fetched from all trackers for every torrent and data from the tracker with most seeds for a given torrent is used. I also got rid of SQLite which was previously used for generating the index. It was replaced by a simple CSV file. KISS Some minor bugfixes and tweaks also included (sorry for not breaking them up into more commits). 2019-03-02 20:26:57 +00:00			`If you don't know what Torrent Paradise is, see the [website](https://torrent-paradise.ml/about.html).`

			`This is a repository of all the tools I use to build and run torrent-paradise.ml. The 'code name' of the project is nextgen (next gen torrent search), so don't be surprised if it comes up somewhere.`
Clean up Before publishing to Github 2019-01-31 12:51:03 +00:00
			`## Can you help me?`
			`Maybe, open an issue. Be sure to demonstrate an effort that you tried to solve the problem yourself.`

			`# Setup`

			`Here's what the setup looks like rn:`
			`- VPS, Debian Stretch, 2 GB RAM`
			`- PostgreSQL 9.6. pg_hba.conf contains this:`

			```
			`local all all peer`
			`# IPv4 local connections:`
			`host nextgen nextgen localhost md5`
			```
			`- IPFS v0.4.18`
			`- user with username nextgen on the server`
			`- my laptop w/ Linux`
			`- Go toolchain installed`
Redo tracker scraping, index generation Seed/leech counts are fetched from all trackers for every torrent and data from the tracker with most seeds for a given torrent is used. I also got rid of SQLite which was previously used for generating the index. It was replaced by a simple CSV file. KISS Some minor bugfixes and tweaks also included (sorry for not breaking them up into more commits). 2019-03-02 20:26:57 +00:00			`- node v10.15 & npm`
			`- Python 3 (required only for index-generator/fix-metajson.py)`
Clean up Before publishing to Github 2019-01-31 12:51:03 +00:00
Redo tracker scraping, index generation Seed/leech counts are fetched from all trackers for every torrent and data from the tracker with most seeds for a given torrent is used. I also got rid of SQLite which was previously used for generating the index. It was replaced by a simple CSV file. KISS Some minor bugfixes and tweaks also included (sorry for not breaking them up into more commits). 2019-03-02 20:26:57 +00:00			`The programs create their own tables in the DB that they need. Database name is "nextgen".`
Clean up Before publishing to Github 2019-01-31 12:51:03 +00:00
			What I did first after getting the server up and running was importing the TPB dump. Download https://thepiratebay.org/static/dump/csv/torrent_dump_full.csv.gz to the import-tpb-dump directory and run `go run`.

Add link to MEGA for database dump download 2019-11-24 11:53:39 +00:00			`There is a complete database dump available in torrentparadise-staticbackup.torrent, so you don't have to do that. This same database dump is available on https://mega.nz/#!ddESlChb!3YBqfxG-a4fwpXzPG3QsXa-C6FeQ9AbNSGXxY7W7xm4. It contains the same data as the torrent, only .xz compressed.`
Add link to torrent with database dump 2019-10-22 15:01:33 +00:00
Clean up Before publishing to Github 2019-01-31 12:51:03 +00:00			`# Usage`

			`## Generate the index`

Revamp seedleech count collection Add seedleech-daemon that takes care of keeping seedleech counts updated 2019-03-30 09:38:27 +00:00			See `update-index.sh`.
Clean up Before publishing to Github 2019-01-31 12:51:03 +00:00
			`## Spider the DHT`

Redo tracker scraping, index generation Seed/leech counts are fetched from all trackers for every torrent and data from the tracker with most seeds for a given torrent is used. I also got rid of SQLite which was previously used for generating the index. It was replaced by a simple CSV file. KISS Some minor bugfixes and tweaks also included (sorry for not breaking them up into more commits). 2019-03-02 20:26:57 +00:00			Run `go build` in spider/ to compile and scp the binary it to the server. You can use the systemd service file in `spider/spider.service` to start the spider on server boot.

			`## Scraping trackers for seed/leech data`

Revamp seedleech count collection Add seedleech-daemon that takes care of keeping seedleech counts updated 2019-03-30 09:38:27 +00:00			Run `go build` in seedleech-daemon/ to compile and scp the binary it to the server. You can use the systemd service file in `seedleech-daemon/seedleech.service`.
Clean up Before publishing to Github 2019-01-31 12:51:03 +00:00

			`# Contributing`

Add link to MEGA for database dump download 2019-11-24 11:53:39 +00:00			`Before working on something, open an issue to ask if it would be okay. I would love to [KISS](https://en.wikipedia.org/wiki/KISS_principle).`