mirror of
https://git.zx2c4.com/cgit/
synced 2024-11-10 07:10:33 +00:00
23debef621
My dmesg is filled with the oom killer bringing down processes while the Bingbot downloads every snapshot for every commit of the Linux kernel in tar.xz format. Sure, I should be running with memory limits, and now I'm using cgroups, but a more general solution is to prevent crawlers from wasting resources like that in the first place. Suggested-by: Natanael Copa <ncopa@alpinelinux.org> Suggested-by: Julius Plenz <plenz@cis.fu-berlin.de> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
4 lines
47 B
Plaintext
4 lines
47 B
Plaintext
User-agent: *
|
|
Disallow: /*/snapshot/*
|
|
Allow: /
|