69 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Aloïs Micard	fa348dca5d	Last cleanups ... - API: implement pagination for search endpoints - Crawler: do not save body when code > 302 - Scripts: add stop.sh	2020-09-22 16:07:31 +02:00
Aloïs Micard	e0dfc648b6	Implement search in trandoshanctl	2020-09-22 13:32:40 +02:00
Aloïs Micard	8d9d9524a7	Finalize search endpoint	2020-09-22 13:25:54 +02:00
Aloïs Micard	cacf4f1236	Improve api search endpoint	2020-09-22 13:00:52 +02:00
Aloïs Micard	0e6477dd0a	Now follow redirect	2020-09-22 12:43:52 +02:00
Aloïs Micard	a74003a818	Configure timeout for API	2020-09-22 12:25:47 +02:00
Aloïs Micard	742ccbaa79	Finalize whole implementation	2020-09-22 12:10:04 +02:00
Aloïs Micard	6081a6a7c2	Move url extraction logic to extractor	2020-09-22 07:11:51 +02:00
Aloïs Micard	f2b8984356	Little cleanup	2020-09-22 06:56:26 +02:00
Aloïs Micard	ae5812c566	Make extractor publish found URLs	2020-09-22 06:53:43 +02:00
Aloïs Micard	560d7cb846	Implement extractor	2020-09-22 06:43:05 +02:00
Aloïs Micard	5b220de671	Move messaging into internal package	2020-09-21 17:16:41 +02:00
Aloïs Micard	aa40a11725	Add comments	2020-09-21 17:15:50 +02:00
Aloïs Micard	20f67edd28	Create API client	2020-09-21 17:14:15 +02:00
Aloïs Micard	1c8368704c	Delete pkg/ package and split it	2020-09-21 16:54:00 +02:00
Aloïs Micard	11f04b1ca3	Add missing comments	2020-09-21 16:45:51 +02:00
Aloïs Micard	a0be5160dc	Start implementing new architecture	2020-09-21 16:40:12 +02:00
Aloïs Micard	8eedbdd572	Release 0.3.0	2020-09-21 08:07:21 +02:00
Aloïs Micard	0ee5ce2246	Fix build.sh	2020-09-21 07:36:10 +02:00
Aloïs Micard	325c6ef175	Migrate to zerolog ... Closes: #20	2020-09-21 07:33:45 +02:00
Aloïs Micard	589f3e2b07	Use dockerhub for images. ... Closes: #22	2020-09-21 06:40:04 +02:00
Gagan Bhat	3721428ebb	Java options fix in docker-compose.yml. (#18)	2020-09-07 20:21:54 +02:00
Aloïs Micard	42ee930160	Unit test scheduler	2020-09-03 07:07:17 +02:00
Aloïs Micard	230dc550c1	Add badge	2020-08-15 17:17:54 +02:00
Aloïs Micard	c043ad86f7	Release 0.2.0	2020-08-15 17:11:30 +02:00
Aloïs Micard	a635722690	Fix docker image build	2020-08-15 16:27:46 +02:00
Aloïs Micard	82a4a9c527	Add tdsh- prefix to executables. ... API#searchResources: - Serialize date - Do not return body in get	2020-08-15 16:21:39 +02:00
Aloïs Micard	9e80f41df2	Bump go version to 1.15.0	2020-08-15 15:42:46 +02:00
Aloïs Micard	5c739b5809	Release 0.1.0	2020-08-13 11:59:25 +02:00
Aloïs Micard	976500084c	Persist elasticsearch data	2020-08-12 12:10:59 +02:00
Aloïs Micard	52c2bc2ded	Create docker subfolder in build/ & deployments/	2020-08-12 12:01:43 +02:00
Aloïs Micard	98bd85e0c4	Bump Go version to 1.14.7	2020-08-11 21:08:02 +02:00
Aloïs Micard	1413680121	[#9] Prevent from crawling binary, image, etc...	2020-08-11 12:18:09 +02:00
Aloïs Micard	509268e532	Commit useful scripts	2020-08-11 07:30:18 +02:00
Aloïs Micard	45a9848395	Lint source code	2020-08-10 12:02:42 +02:00
Aloïs Micard	b5b58a8d19	Cleanup code	2020-08-10 08:01:50 +02:00
Aloïs Micard	8ae38445cf	Move ResourceDto to proto package	2020-08-10 08:00:06 +02:00
Aloïs Micard	482dde3e17	[#8] Handle case no ES collection yet	2020-08-10 07:10:28 +02:00
Aloïs Micard	6519672b13	Api#addUrl: Fix sent message	2020-08-08 10:48:42 +02:00
Aloïs Micard	ed7ea4596b	[#7] API should publish to URLFoundSubject ... this will allow scheduler to approve or not.	2020-08-08 06:57:32 +02:00
Aloïs Micard	945651b93a	[#7] Fix publish URL	2020-08-08 06:56:16 +02:00
Aloïs Micard	05df5c56a4	Name apps, write test	2020-08-07 19:31:18 +02:00
Aloïs Micard	56cb94258f	Crawler: Allow to customize user agent	2020-08-07 17:47:08 +02:00
Aloïs Micard	de231a851a	Update README.md	2020-08-07 17:30:58 +02:00
Aloïs Micard	599e6ef4d3	Fix wrong endpoint being used by scheduler ... Also b64 encode the URL. Closes #6	2020-08-07 13:15:33 +02:00
Aloïs Micard	75fa6724c9	Allow to submit new URL trough the API ... Closes #4	2020-08-07 13:12:17 +02:00
Aloïs Micard	2313c242e8	Add script to delete everything	2020-08-07 12:47:17 +02:00
Aloïs Micard	68ddf09aaa	Use logrus everywhere	2020-08-07 12:44:33 +02:00
Aloïs Micard	a64cd3dff2	Improve build script	2020-08-07 12:24:24 +02:00
FFrozTT	4160f69d68	Update README.md	2020-08-07 08:00:05 +02:00