The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

Are bots using a large number of IP addresses simultaneously, so they look like a DDOS attack? Or are they just making ordinary requests from a small number of addresses. If it's the latter, all you need is some kind of fair queuing so those requests compete with each other for access, not with other users.

Often it is rotating residential proxies. It is virtually impossible to mitigate this behavior from the IP level.

They're using state of the art obfuscation that makes them indistinguishable from malicious botnets. It's an arms race with billion dollar companies vying to consume the most content before it all collapses

the open web is dead and whatever's left will be locked be authentication and paywalls

Bots are probing for access from various servers, eventually falling back to executing requests from residential IP addresses: https://blog.cloudflare.com/perplexity-is-using-stealth-unde...

Cloudflare is dealing with a couple million faked requests every day just from Perplexity users, and Perplexity is far from the worst player in the field.

The problem would be quite easy to solve with basic rate limiting if it weren't for the attempts to bypass access controls.