Aloïs Micard
|
d66eee8b94
|
Add working files
|
4 years ago |
|
Aloïs Micard
|
72a198cccd
|
Fix README.md
|
4 years ago |
|
Aloïs Micard
|
5b1b973300
|
Organize scripts
|
4 years ago |
Aloïs Micard
|
efc1ec2600
|
Merge pull request #136 from creekorful/add-back-goreleaser
Add back goreleaser
|
4 years ago |
|
Aloïs Micard
|
d7e663fef6
|
Add back continuous delivery
|
4 years ago |
|
Aloïs Micard
|
ddcf5b40a8
|
Add back goreleaser
|
4 years ago |
|
Aloïs Micard
|
fede4ba44d
|
Merge pull request #135 from creekorful/add-process-description
Add process description
|
4 years ago |
|
Aloïs Micard
|
02a0dca6ed
|
Add process description
|
4 years ago |
|
Aloïs Micard
|
e0694b6cef
|
Merge pull request #134 from creekorful/129-find-new-name
Rename project
|
4 years ago |
|
Aloïs Micard
|
96c75a85b1
|
Rename project
|
4 years ago |
|
Aloïs Micard
|
3cc47a071a
|
Merge pull request #133 from creekorful/develop
Release 0.11.0
|
4 years ago |
|
Aloïs Micard
|
b657123d91
|
Bump version
|
4 years ago |
|
Aloïs Micard
|
bbdba2546f
|
Merge pull request #132 from creekorful/120-blacklister-final-tweaks
Blacklister final tweaks
|
4 years ago |
|
Aloïs Micard
|
9b83830d95
|
Merge remote-tracking branch 'origin/develop' into 120-blacklister-final-tweaks
|
4 years ago |
|
Aloïs Micard
|
6c678478a1
|
Implement better blacklist config
|
4 years ago |
|
Aloïs Micard
|
871daf1bcc
|
scheduler: hash url before caching it
|
4 years ago |
|
Aloïs Micard
|
69352f7237
|
indexer: sort headers to have deterministic output
|
4 years ago |
|
Aloïs Micard
|
b84e6d28ac
|
Merge pull request #131 from creekorful/130-optimize-redis-memory
scheduler: hash url before caching it
|
4 years ago |
|
Aloïs Micard
|
e07ed8156e
|
scheduler: hash url before caching it
|
4 years ago |
|
Aloïs Micard
|
cae3bb514f
|
Merge pull request #128 from creekorful/114-fix-tests
indexer: sort headers to have deterministic output
|
4 years ago |
|
Aloïs Micard
|
faee8b48c1
|
indexer: sort headers to have deterministic output
|
4 years ago |
|
Aloïs Micard
|
8297dc7616
|
Merge pull request #127 from creekorful/124-improve-scheduler-speed
scheduler: increase performances
|
4 years ago |
|
Aloïs Micard
|
84a28c5be0
|
scheduler: increase event prefetch
|
4 years ago |
|
Aloïs Micard
|
afed403e6a
|
Remove useless regex
|
4 years ago |
|
Aloïs Micard
|
4e33813b21
|
Merge remote-tracking branch 'origin/develop' into 124-improve-scheduler-speed
|
4 years ago |
|
Aloïs Micard
|
7820820fa9
|
scheduler: add batch support for dialing with cache
|
4 years ago |
|
Aloïs Micard
|
de50ed02e3
|
Merge pull request #126 from creekorful/125-indexer-bulk-indexation
Indexer: implement bulk indexation
|
4 years ago |
|
Aloïs Micard
|
9b46dc205e
|
Indexer: support buffered indexing
|
4 years ago |
|
Aloïs Micard
|
71f82d4aad
|
process: Rework whole flags system
- Turn the flag into Feature system to allow easier configuration.
- Add prefetch flag to event feature
|
4 years ago |
|
Aloïs Micard
|
4075dfc98a
|
Merge pull request #121 from creekorful/develop
Release 0.10.0
|
4 years ago |
|
Aloïs Micard
|
829afcbb6a
|
Release 0.10.0
|
4 years ago |
|
Aloïs Micard
|
ec3357be5d
|
Big improvements
- Reduce debug noise
- Create scripts to blacklist 'famous' legit hostnames
- Make blacklister more resilient
- Merge archiver & indexer together
- Better prefix for cache key
- Rework scheduling process
- Update architecture.png
- Remove trandoshanctl
- Improve testing
|
4 years ago |
|
Aloïs Micard
|
2d7499f7e2
|
Merge pull request #118 from creekorful/106-improve-blacklister
Implement new blacklister
|
4 years ago |
|
Aloïs Micard
|
8da1f29a43
|
little fixes
|
4 years ago |
|
Aloïs Micard
|
d0dffb9928
|
Implement new blacklister
|
4 years ago |
|
Aloïs Micard
|
6c4ecc1a7d
|
Merge pull request #117 from creekorful/develop
Release 0.9.0
|
4 years ago |
|
Aloïs Micard
|
2133a1aeb5
|
bump app versions
|
4 years ago |
|
Aloïs Micard
|
46a7a05e4a
|
Merge pull request #116 from creekorful/110-archiver-new-format
Implement new storage format
|
4 years ago |
|
Aloïs Micard
|
a27092fd13
|
Use new storage format
|
4 years ago |
|
Aloïs Micard
|
1ac5c1e036
|
Merge remote-tracking branch 'origin/develop' into 110-archiver-new-format
|
4 years ago |
|
Aloïs Micard
|
571b1e2628
|
Merge pull request #115 from creekorful/111-prevent-duplicates-urls
Prevent duplicates urls in crawlingQueue
|
4 years ago |
|
Aloïs Micard
|
cc3c0d62d6
|
remove hacky check
|
4 years ago |
|
Aloïs Micard
|
c8352d3299
|
Use url cache to determinate if crawling should be done
|
4 years ago |
|
Aloïs Micard
|
e245e5d79a
|
last fixes
|
4 years ago |
|
Aloïs Micard
|
60a23f7182
|
Fix ttl
|
4 years ago |
|
Aloïs Micard
|
12362e0100
|
Fix tests case
|
4 years ago |
|
Aloïs Micard
|
4a0fbd0b9b
|
add configapi key prefix
|
4 years ago |
|
Aloïs Micard
|
0aba4fa4f9
|
Finalize redis cache impl
|
4 years ago |
|
Aloïs Micard
|
387a93b7b9
|
Create new flags for cache
|
4 years ago |
|
Aloïs Micard
|
d826fe73b6
|
Refactor configapi to use new cache
|
4 years ago |