←back to thread

Nepenthes is a tarpit to catch AI web crawlers

(zadzmo.org)
646 points blendergeek | 1 comments | 16 Jan 25 13:57 UTC | HN request time: 0.212s | source
Show context
Havoc ◴[17 Jan 25 10:51 UTC] No.42736207[source]▶
What blows my mind is that this is functionally a solved problem.

The big search crawlers have been around for years & manage to mostly avoid nuking sites into oblivion. Then AI gang shows up - supposedly smartest guys around - and suddenly we're re-inventing the wheel on crawling and causing carnage in the process.

replies(2): >>42736252 #>>42737200 #
1. marginalia_nu ◴[17 Jan 25 13:17 UTC] No.42737200[source]▶
I think it's largely the mindset of moving fast and breaking things that's at fault. If say ship it at "good enough", it will not behave well.

Building a competent well-behaved crawler is a big effort that requires relatively deep understanding of more or less all web tech, and figuring out a bunch of stuff that is not documented anywhere and not part of any specs.