←back to thread

Cloudflare.com's Robots.txt

(www.cloudflare.com)
145 points sans_souse | 6 comments | 17 Nov 24 12:39 UTC | HN request time: 0.258s | source | bottom
1. jsheard ◴[17 Nov 24 13:27 UTC] No.42164090[source]▶
This is what happens if your robot isn't nice

  > curl -I -H "User-Agent: Googlebot" https://www.cloudflare.com
  HTTP/2 403
replies(1): >>42164220 #
2. jamesog ◴[17 Nov 24 14:02 UTC] No.42164220[source]▶
That's not from robots.txt, but their Bot Management feature which blocks things calling themselves Googlebot that don't come from known Google IPs.
replies(1): >>42164616 #
3. speedgoose ◴[17 Nov 24 15:21 UTC] No.42164616[source]▶
Are GCP IPs considered Google IPs?
replies(3): >>42164648 #>>42164657 #>>42165651 #
4. crop_rotation ◴[17 Nov 24 15:25 UTC] No.42164648{3}[source]▶
No I am very sure they are not.
5. jgrahamc ◴[17 Nov 24 15:27 UTC] No.42164657{3}[source]▶
No.
6. judge2020 ◴[17 Nov 24 17:58 UTC] No.42165651{3}[source]▶
For reference https://developers.google.com/search/docs/crawling-indexing/...