(www.cloudflare.com)

145 points sans_souse | 1 comments | 17 Nov 24 12:39 UTC | HN request time: 0.335s | source

Show context

jsheard ◴[17 Nov 24 13:27 UTC] No.42164090[source]▶

This is what happens if your robot isn't nice

  > curl -I -H "User-Agent: Googlebot" https://www.cloudflare.com
  HTTP/2 403

replies(1): >>42164220 #

jamesog ◴[17 Nov 24 14:02 UTC] No.42164220[source]▶

That's not from robots.txt, but their Bot Management feature which blocks things calling themselves Googlebot that don't come from known Google IPs.

replies(1): >>42164616 #

speedgoose ◴[17 Nov 24 15:21 UTC] No.42164616[source]▶

Are GCP IPs considered Google IPs?

1. jgrahamc ◴[17 Nov 24 15:27 UTC] No.42164657[source]▶

No.

Cloudflare.com's Robots.txt