←back to thread

Cloudflare.com's Robots.txt

(www.cloudflare.com)
145 points sans_souse | 1 comments | 17 Nov 24 12:39 UTC | HN request time: 0.208s | source
Show context
orliesaurus ◴[17 Nov 24 15:54 UTC] No.42164810[source]▶
Has anyone worked on anything like this for AI scrapers?
replies(3): >>42165005 #>>42165055 #>>42165872 #