Cloudflare.com's Robots.txt

(www.cloudflare.com)

145 points sans_souse | 5 comments | 17 Nov 24 12:39 UTC | HN request time: 0.201s | source

Show context

jsheard ◴[17 Nov 24 13:27 UTC] No.42164090[source]▶

This is what happens if your robot isn't nice

  > curl -I -H "User-Agent: Googlebot" https://www.cloudflare.com
  HTTP/2 403

replies(1): >>42164220 #

1. jamesog ◴[17 Nov 24 14:02 UTC] No.42164220[source]▶

That's not from robots.txt, but their Bot Management feature which blocks things calling themselves Googlebot that don't come from known Google IPs.

replies(1): >>42164616 #

2. speedgoose ◴[17 Nov 24 15:21 UTC] No.42164616[source]▶

Are GCP IPs considered Google IPs?

3. crop_rotation ◴[17 Nov 24 15:25 UTC] No.42164648[source]▶

No I am very sure they are not.

4. jgrahamc ◴[17 Nov 24 15:27 UTC] No.42164657[source]▶

No.

5. judge2020 ◴[17 Nov 24 17:58 UTC] No.42165651[source]▶