Archives/bathyscaphe
2
0
Fork 0
mirror of https://github.com/creekorful/bathyscaphe synced 2024-11-19 15:25:44 +00:00
Code Issues Packages Projects Releases Wiki Activity
70 Commits 2 Branches 15 Tags 1 MiB
e61dc42d3c
Commit Graph

70 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Aloïs Micard
e61dc42d3c
Some cleanup 2020-09-22 17:08:24 +02:00
Aloïs Micard
fa348dca5d
Last cleanups 
- API: implement pagination for search endpoints
- Crawler: do not save body when code > 302
- Scripts: add stop.sh
 2020-09-22 16:07:31 +02:00
Aloïs Micard
e0dfc648b6
Implement search in trandoshanctl 2020-09-22 13:32:40 +02:00
Aloïs Micard
8d9d9524a7
Finalize search endpoint 2020-09-22 13:25:54 +02:00
Aloïs Micard
cacf4f1236
Improve api search endpoint 2020-09-22 13:00:52 +02:00
Aloïs Micard
0e6477dd0a
Now follow redirect 2020-09-22 12:43:52 +02:00
Aloïs Micard
a74003a818
Configure timeout for API 2020-09-22 12:25:47 +02:00
Aloïs Micard
742ccbaa79
Finalize whole implementation 2020-09-22 12:10:04 +02:00
Aloïs Micard
6081a6a7c2
Move url extraction logic to extractor 2020-09-22 07:11:51 +02:00
Aloïs Micard
f2b8984356
Little cleanup 2020-09-22 06:56:26 +02:00
Aloïs Micard
ae5812c566
Make extractor publish found URLs 2020-09-22 06:53:43 +02:00
Aloïs Micard
560d7cb846
Implement extractor 2020-09-22 06:43:05 +02:00
Aloïs Micard
5b220de671
Move messaging into internal package 2020-09-21 17:16:41 +02:00
Aloïs Micard
aa40a11725
Add comments 2020-09-21 17:15:50 +02:00
Aloïs Micard
20f67edd28
Create API client 2020-09-21 17:14:15 +02:00
Aloïs Micard
1c8368704c
Delete pkg/ package and split it 2020-09-21 16:54:00 +02:00
Aloïs Micard
11f04b1ca3
Add missing comments 2020-09-21 16:45:51 +02:00
Aloïs Micard
a0be5160dc
Start implementing new architecture 2020-09-21 16:40:12 +02:00
Aloïs Micard
8eedbdd572
Release 0.3.0 2020-09-21 08:07:21 +02:00
Aloïs Micard
0ee5ce2246
Fix build.sh 2020-09-21 07:36:10 +02:00
Aloïs Micard
325c6ef175
Migrate to zerolog 
Closes: #20
 2020-09-21 07:33:45 +02:00
Aloïs Micard
589f3e2b07
Use dockerhub for images. 
Closes: #22
 2020-09-21 06:40:04 +02:00
Gagan Bhat
3721428ebb
Java options fix in docker-compose.yml. (#18) 2020-09-07 20:21:54 +02:00
Aloïs Micard
42ee930160
Unit test scheduler 2020-09-03 07:07:17 +02:00
Aloïs Micard
230dc550c1
Add badge 2020-08-15 17:17:54 +02:00
Aloïs Micard
c043ad86f7
Release 0.2.0 2020-08-15 17:11:30 +02:00
Aloïs Micard
a635722690
Fix docker image build 2020-08-15 16:27:46 +02:00
Aloïs Micard
82a4a9c527
Add tdsh- prefix to executables. 
API#searchResources:

- Serialize date
- Do not return body in get
 2020-08-15 16:21:39 +02:00
Aloïs Micard
9e80f41df2
Bump go version to 1.15.0 2020-08-15 15:42:46 +02:00
Aloïs Micard
5c739b5809 Release 0.1.0 2020-08-13 11:59:25 +02:00
Aloïs Micard
976500084c
Persist elasticsearch data 2020-08-12 12:10:59 +02:00
Aloïs Micard
52c2bc2ded
Create docker subfolder in build/ & deployments/ 2020-08-12 12:01:43 +02:00
Aloïs Micard
98bd85e0c4 Bump Go version to 1.14.7 2020-08-11 21:08:02 +02:00
Aloïs Micard
1413680121
[#9] Prevent from crawling binary, image, etc... 2020-08-11 12:18:09 +02:00
Aloïs Micard
509268e532
Commit useful scripts 2020-08-11 07:30:18 +02:00
Aloïs Micard
45a9848395
Lint source code 2020-08-10 12:02:42 +02:00
Aloïs Micard
b5b58a8d19
Cleanup code 2020-08-10 08:01:50 +02:00
Aloïs Micard
8ae38445cf
Move ResourceDto to proto package 2020-08-10 08:00:06 +02:00
Aloïs Micard
482dde3e17
[#8] Handle case no ES collection yet 2020-08-10 07:10:28 +02:00
Aloïs Micard
6519672b13
Api#addUrl: Fix sent message 2020-08-08 10:48:42 +02:00
Aloïs Micard
ed7ea4596b
[#7] API should publish to URLFoundSubject 
this will allow scheduler to approve or not.
 2020-08-08 06:57:32 +02:00
Aloïs Micard
945651b93a
[#7] Fix publish URL 2020-08-08 06:56:16 +02:00
Aloïs Micard
05df5c56a4
Name apps, write test 2020-08-07 19:31:18 +02:00
Aloïs Micard
56cb94258f
Crawler: Allow to customize user agent 2020-08-07 17:47:08 +02:00
Aloïs Micard
de231a851a
Update README.md 2020-08-07 17:30:58 +02:00
Aloïs Micard
599e6ef4d3
Fix wrong endpoint being used by scheduler 
Also b64 encode the URL.

Closes #6
 2020-08-07 13:15:33 +02:00
Aloïs Micard
75fa6724c9
Allow to submit new URL trough the API 
Closes #4
 2020-08-07 13:12:17 +02:00
Aloïs Micard
2313c242e8
Add script to delete everything 2020-08-07 12:47:17 +02:00
Aloïs Micard
68ddf09aaa
Use logrus everywhere 2020-08-07 12:44:33 +02:00
Aloïs Micard
a64cd3dff2
Improve build script 2020-08-07 12:24:24 +02:00