←back to thread

Cloudflare.com's Robots.txt

(www.cloudflare.com)
145 points sans_souse | 1 comments | 17 Nov 24 12:39 UTC | HN request time: 0s | source
Show context
yapyap ◴[17 Nov 24 13:28 UTC] No.42164094[source]▶
That’s cool, if any scrapers would still respect the robots.txt that is
replies(4): >>42164168 #>>42165000 #>>42165017 #>>42165663 #
1. dartos ◴[17 Nov 24 16:24 UTC] No.42165000[source]▶
I was surprised any ever did, honestly