Explain how to speed up crawling

This commit is contained in:
Aloïs Micard 2020-09-24 08:19:20 +02:00
parent 8231edf6ce
commit 6876463ba5
No known key found for this signature in database
GPG Key ID: 1A0EB82F071F5EFE
3 changed files with 15 additions and 4 deletions

View File

@ -22,7 +22,7 @@ You can start the crawler in detached mode by passing --detach to it.
Ensure you have at least 3GB of memory as the Elasticsearch stack docker will require 2GB.
# How to start the crawling process
# How to initiate crawling
Since the API is exposed on localhost:15005, one can use it to start the crawling process:
@ -42,6 +42,17 @@ $ docker run creekorful/trandoshanctl --api-uri <uri> schedule https://www.faceb
this will schedule given URL for crawling.
## How to speed up crawling
If one want to speed up the crawling process, he can scale the instance of crawling process in order
to increase performances. This may be done by issuing the following command after the crawler is started:
```sh
$ ./scripts/scale.sh crawler=5
```
this will set the number of crawler instance to 5.
# How to view results
## Using trandoshanctl

View File

@ -1,3 +0,0 @@
#!/bin/bash
./scripts/exec.sh logs "$@"

3
scripts/scale.sh Executable file
View File

@ -0,0 +1,3 @@
#!/bin/bash
./scripts/exec.sh scale "$@"