←back to thread

Nepenthes is a tarpit to catch AI web crawlers

(zadzmo.org)
646 points blendergeek | 1 comments | 16 Jan 25 13:57 UTC | HN request time: 0.213s | source
Show context
taikahessu ◴[16 Jan 25 15:13 UTC] No.42726337[source]▶
We had our non-profit website drained out of bandwidth and site closed temporarily (!!) from our hosting deal because of Amazon bot aggressively crawling like ?page=21454 ... etc.

Gladly Siteground restored our site without any repercussions as it was not our fault. Added Amazon bot into robots.txt after that one.

Don't like how things are right now. Is a tarpit the solution? Or better laws? Would they stop the chinese bots? Should they even? I don't know.

replies(4): >>42726365 #>>42735381 #>>42740706 #>>42743952 #
jsheard ◴[16 Jan 25 15:15 UTC] No.42726365[source]▶
For the "good" bots which at least respect robots.txt you can use this list to get ahead of them before they pummel your site.

https://github.com/ai-robots-txt/ai.robots.txt

There's no easy solution for bad bots which ignore robots.txt and spoof their UA though.

replies(3): >>42726409 #>>42730796 #>>42731908 #
1. breakingcups ◴[16 Jan 25 20:56 UTC] No.42730796[source]▶
Such as OpenAI, who will ignore robots.txt and change their user agent to evade blocks, apparently[1]

1: https://www.reddit.com/r/selfhosted/comments/1i154h7/openai_...