←back to thread

Cloudflare.com's Robots.txt

(www.cloudflare.com)
145 points sans_souse | 3 comments | 17 Nov 24 12:39 UTC | HN request time: 1.175s | source
1. CodesInChaos ◴[17 Nov 24 14:18 UTC] No.42164291[source]▶
What's the purpose of "User-Agent: DemandbaseWebsitePreview/0.1"? I couldn't find anything about that agent, but I assume it's somehow related to demandbase.com?

But why are it and twitter the only whitelisted entries? Google and bing missing is a bit surprising, but I assume they're whitelisted through a different mechanism (like a google webmaster account)?

replies(2): >>42164338 #>>42164695 #
2. saddist0 ◴[17 Nov 24 14:26 UTC] No.42164338[source]▶
It is one of the service they use. As per the cookie policy page [1]:

> DemandBase - Enables us to identify companies who intend to purchase our products and solutions and deliver more relevant messages and offers to our Website visitors.

[1]: https://www.cloudflare.com/en-in/cookie-policy/

3. Maken ◴[17 Nov 24 15:34 UTC] No.42164695[source]▶
My guess is that the Twitter one is for previews when you link to a web in Twitter.