(www.cloudflare.com)

145 points sans_souse | 1 comments | 17 Nov 24 12:39 UTC | HN request time: 0.001s | source

Show context

CodesInChaos ◴[17 Nov 24 14:18 UTC] No.42164291[source]▶

What's the purpose of "User-Agent: DemandbaseWebsitePreview/0.1"? I couldn't find anything about that agent, but I assume it's somehow related to demandbase.com?

But why are it and twitter the only whitelisted entries? Google and bing missing is a bit surprising, but I assume they're whitelisted through a different mechanism (like a google webmaster account)?

replies(2): >>42164338 #>>42164695 #

1. Maken ◴[17 Nov 24 15:34 UTC] No.42164695[source]▶

>>42164291 #

My guess is that the Twitter one is for previews when you link to a web in Twitter.

↑

Cloudflare.com's Robots.txt