(www.cloudflare.com)

145 points sans_souse | 1 comments | 17 Nov 24 12:39 UTC | HN request time: 0.247s | source

Show context

Has anyone worked on anything like this for AI scrapers?

1. dartos ◴[17 Nov 24 16:25 UTC] No.42165005[source]▶

A robots.txt that asks AI scrapers not to scrape?

There’s a couple services that keep updated lists of known scraper user agents. A quick search reveals a handful.

Cloudflare.com's Robots.txt