Aloïs Micard
fa348dca5d
Last cleanups
...
- API: implement pagination for search endpoints
- Crawler: do not save body when code > 302
- Scripts: add stop.sh
4 years ago
Aloïs Micard
e0dfc648b6
Implement search in trandoshanctl
4 years ago
Aloïs Micard
8d9d9524a7
Finalize search endpoint
4 years ago
Aloïs Micard
cacf4f1236
Improve api search endpoint
4 years ago
Aloïs Micard
0e6477dd0a
Now follow redirect
4 years ago
Aloïs Micard
742ccbaa79
Finalize whole implementation
4 years ago
Aloïs Micard
6081a6a7c2
Move url extraction logic to extractor
4 years ago
Aloïs Micard
f2b8984356
Little cleanup
4 years ago
Aloïs Micard
ae5812c566
Make extractor publish found URLs
4 years ago
Aloïs Micard
560d7cb846
Implement extractor
4 years ago
Aloïs Micard
5b220de671
Move messaging into internal package
4 years ago
Aloïs Micard
20f67edd28
Create API client
4 years ago
Aloïs Micard
1c8368704c
Delete pkg/ package and split it
4 years ago
Aloïs Micard
11f04b1ca3
Add missing comments
4 years ago
Aloïs Micard
a0be5160dc
Start implementing new architecture
4 years ago
Aloïs Micard
8eedbdd572
Release 0.3.0
4 years ago
Aloïs Micard
325c6ef175
Migrate to zerolog
...
Closes : #20
4 years ago
Aloïs Micard
42ee930160
Unit test scheduler
4 years ago
Aloïs Micard
c043ad86f7
Release 0.2.0
4 years ago
Aloïs Micard
a635722690
Fix docker image build
4 years ago
Aloïs Micard
82a4a9c527
Add tdsh- prefix to executables.
...
API#searchResources:
- Serialize date
- Do not return body in get
4 years ago
Aloïs Micard
5c739b5809
Release 0.1.0
4 years ago
Aloïs Micard
1413680121
[ #9 ] Prevent from crawling binary, image, etc...
4 years ago
Aloïs Micard
45a9848395
Lint source code
4 years ago
Aloïs Micard
b5b58a8d19
Cleanup code
4 years ago
Aloïs Micard
8ae38445cf
Move ResourceDto to proto package
4 years ago
Aloïs Micard
482dde3e17
[ #8 ] Handle case no ES collection yet
4 years ago
Aloïs Micard
6519672b13
Api#addUrl: Fix sent message
4 years ago
Aloïs Micard
ed7ea4596b
[ #7 ] API should publish to URLFoundSubject
...
this will allow scheduler to approve or not.
4 years ago
Aloïs Micard
945651b93a
[ #7 ] Fix publish URL
4 years ago
Aloïs Micard
05df5c56a4
Name apps, write test
4 years ago
Aloïs Micard
56cb94258f
Crawler: Allow to customize user agent
4 years ago
Aloïs Micard
599e6ef4d3
Fix wrong endpoint being used by scheduler
...
Also b64 encode the URL.
Closes #6
4 years ago
Aloïs Micard
75fa6724c9
Allow to submit new URL trough the API
...
Closes #4
4 years ago
Aloïs Micard
68ddf09aaa
Use logrus everywhere
4 years ago
Aloïs Micard
8a32bbe5fa
Finx lint issues
5 years ago
Aloïs Micard
6b28f074d1
Implement API
...
Now persister process will use API to save resource content.
Scheduler will also use the API to get resource by URL, and will later
determinate if scheduling should be done based on his own algorithm
5 years ago
Aloïs Micard
680eccef96
Fix wrong usage of logrus in trandoshan-api
5 years ago
Aloïs Micard
317a4eabbd
Add api process
5 years ago
Aloïs Micard
5f1dd4bec8
Implement persister
...
Add kibana & elasticsearch dependencies.
5 years ago
Aloïs Micard
7d2e666ba9
Refactor nats logic into natsutil
5 years ago
Aloïs Micard
82250b46ae
Centralize logging initialization
5 years ago
Aloïs Micard
fd9d2e2b9e
Add persister process
5 years ago
Aloïs Micard
4e49e0aca9
Crawler: now publish message with resource body
5 years ago
Aloïs Micard
29da7859b4
Scheduler: normalized received URLs
5 years ago
Aloïs Micard
cf8c2875cb
Run gofmt over the project
5 years ago
Aloïs Micard
28f32042c6
Setup crawler to use tor proxy to reach hidden services
5 years ago
Aloïs Micard
33269f7ffa
Centralize ReadJSON into natsutil
5 years ago
Aloïs Micard
6d349da3a6
Fix lint errors
5 years ago
Aloïs Micard
c6a857f45b
Add basic scheduler implementation
...
- Create separate proto package to store Trandoshan related protocol implementation
5 years ago