←back to thread

Cloudflare.com's Robots.txt

(www.cloudflare.com)
145 points sans_souse | 3 comments | 17 Nov 24 12:39 UTC | HN request time: 0.411s | source
Show context
yapyap ◴[17 Nov 24 13:28 UTC] No.42164094[source]▶
That’s cool, if any scrapers would still respect the robots.txt that is
replies(4): >>42164168 #>>42165000 #>>42165017 #>>42165663 #
1. bityard ◴[17 Nov 24 13:48 UTC] No.42164168[source]▶
Think of robots.txt as less of a no trespassing sign and more of a, "You can visit but here are the rules to follow if you don't want to get shot" sign.
replies(2): >>42165338 #>>42165715 #
2. iterance ◴[17 Nov 24 17:11 UTC] No.42165338[source]▶
If you do not respect the sign I shall be very cross with you. Very cross indeed. Perhaps I shall have to glare at you, yes, very hard. I think I shall glare at you. Perhaps if you are truly irritating I shall be forced to remove you from the premises for a bit.
3. blacksmith_tb ◴[17 Nov 24 18:06 UTC] No.42165715[source]▶
There's a lot of talk of deregulation in the air, maybe we'll see Gibson-esque Black Ice, where rude crawlers provoke an automated DoS, a new Wild West.