←back to thread

The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

(positiveblue.substack.com)
454 points positiveblue | 1 comments | 29 Aug 25 16:35 UTC | HN request time: 0.226s | source
Show context
TIPSIO ◴[29 Aug 25 16:57 UTC] No.45066555[source]▶
Everyone loves the dream of a free for all and open web.

But the reality is how can someone small protect their blog or content from AI training bots? E.g.: They just blindly trust someone is sending Agent vs Training bots and super duper respecting robots.txt? Get real...

Or, fine what if they do respect robots.txt, but they buy the data that may or may not have been shielded through liability layers via "licensed data"?

Unless you're reddit, X, Google, or Meta with scary unlimited budget legal teams, you have no power.

Great video: https://www.youtube.com/shorts/M0QyOp7zqcY

replies(37): >>45066600 #>>45066626 #>>45066827 #>>45066906 #>>45066945 #>>45066976 #>>45066979 #>>45067024 #>>45067058 #>>45067180 #>>45067399 #>>45067434 #>>45067570 #>>45067621 #>>45067750 #>>45067890 #>>45067955 #>>45068022 #>>45068044 #>>45068075 #>>45068077 #>>45068166 #>>45068329 #>>45068436 #>>45068551 #>>45068588 #>>45069623 #>>45070279 #>>45070690 #>>45071600 #>>45071816 #>>45075075 #>>45075398 #>>45077464 #>>45077583 #>>45080415 #>>45101938 #
avazhi ◴[29 Aug 25 18:16 UTC] No.45067570[source]▶
Nobody cares about robots.txt, nor should they.

If this is your primary argument against being scraped (viz that your robots.txt said not to) then you’re naive and you’re doing it wrong.

If the internet is open, then data on it is going to be scraped lol. You can’t have it both ways.

replies(1): >>45067634 #
1. verdverm ◴[29 Aug 25 18:20 UTC] No.45067634[source]▶
It seems the Open Internet is idealistic.

If others respected robots.txt, we would not need solutions like what Cloudflare is presenting here. Since abuse is rampant, people are looking for mitigations and this CF offering is an interesting one to consider.