AI web crawlers are destroying websites in their never-ending content hunger

At the same time, a lot of HN push back for new solutions like Signed Agents by CF

https://news.ycombinator.com/item?id=45066258

Because it's a bad solution. The core problem is that the internet is vulnerable to DDoS attacks and the web has no native sybil resistance mechanism.

Cloudflare's solution to every problem is to allow them to control more of the internet. What happens when they have enough control to do whatever they want? They could charge any price they want.

The idea itself has merit, even if the implementation is questionable.

Giving bots a cryptographic identity would allow good bots to meaningfully have skin in the game and crawl with their reputation at stake. It's not a complete solution, but could be part of one. Though you can likely get the good parts from HTTP request signing alone, Cloudflare's additions to that seem fairly extraneous.

I honestly don't know what is a good solution. The status quo is certainly completely untenable. If we keep going like we are now, there won't be a web left to protect in a few years. It's worth keeping in mind that there's an opportunity cost, and even a bad solution may be preferrable to no solution at all.

... I say operating an independent web crawler.

I think the solution is some sort of PoW gateway like people are setting up now. Or a micropayments system where each page request costs a fraction of a penny.

You could combine that with some sort of IPFS/Bittorrent like system where you allow others to rehost your static content, indexed by the merkle hash of the content. That would allow users to donate bandwidth.

I really don't like the idea that you can get out of this by surveiling user agents more or distinguishing between "good" and "bad" bots which is a massive social problem.