←back to thread

The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

(positiveblue.substack.com)
454 points positiveblue | 1 comments | 29 Aug 25 16:35 UTC | HN request time: 0.235s | source
Show context
matt-p ◴[29 Aug 25 16:50 UTC] No.45066473[source]▶
I have zero issue with Ai Agents, if there's a real user behind there somewhere. I DO have a major issue with my sites being crawled extremely aggressively by offenders including Meta, Perplexity and OpenAI - it's really annoying realising that we're tying up several cpu cores on AI crawling. Less than on real users and google et al.
replies(6): >>45066494 #>>45066689 #>>45066754 #>>45067321 #>>45067530 #>>45068488 #
swed420 ◴[29 Aug 25 18:12 UTC] No.45067530[source]▶
> I DO have a major issue with my sites being crawled extremely aggressively by offenders including Meta, Perplexity and OpenAI

Gee, if only we had, like, one central archive of the internet. We could even call it the internet archive.

Then, all these AI companies could interface directly with that single entity on terms that are agreeable.

replies(2): >>45067816 #>>45074266 #
teitoklien ◴[29 Aug 25 18:38 UTC] No.45067816[source]▶
you think they care about that ? they’d still crawl like this just in case which is why they don’t rate limit atm
replies(1): >>45078306 #
1. swed420 ◴[30 Aug 25 21:51 UTC] No.45078306[source]▶
It would of course need to be legally enforced somehow, with penalties high enough to hurt even the big players.