←back to thread

Cloudflare.com's Robots.txt

(www.cloudflare.com)
145 points sans_souse | 4 comments | 17 Nov 24 12:39 UTC | HN request time: 0.695s | source
1. orliesaurus ◴[17 Nov 24 15:54 UTC] No.42164810[source]▶
Has anyone worked on anything like this for AI scrapers?
replies(3): >>42165005 #>>42165055 #>>42165872 #
2. dartos ◴[17 Nov 24 16:25 UTC] No.42165005[source]▶
A robots.txt that asks AI scrapers not to scrape?

There’s a couple services that keep updated lists of known scraper user agents. A quick search reveals a handful.

3. zorked ◴[17 Nov 24 16:34 UTC] No.42165055[source]▶
https://github.com/ai-robots-txt/ai.robots.txt/blob/main/rob...
4. gnaman ◴[17 Nov 24 18:27 UTC] No.42165872[source]▶
https://llmstxt.org/ https://www.answer.ai/posts/2024-09-03-llmstxt.html