←back to thread

Cloudflare.com's Robots.txt

(www.cloudflare.com)
145 points sans_souse | 2 comments | 17 Nov 24 12:39 UTC | HN request time: 0.417s | source
1. chrisweekly ◴[17 Nov 24 17:06 UTC] No.42165313[source]▶
One nice thing about CF's robots.txt is its inclusion of a sitemap:

https://www.cloudflare.com/sitemap.xml

which contains links to educational materials like

https://www.cloudflare.com/learning/ddos/layer-3-ddos-attack...

Potentially interesting to see their flattened IA....

replies(1): >>42165519 #
2. palsecam ◴[17 Nov 24 17:38 UTC] No.42165519[source]▶
Little-known fact: a syndication feed (RSS or Atom) can be used as a sitemap.

Quoting https://www.sitemaps.org/protocol.html#otherformats:

> The Sitemap protocol enables you to provide details about your pages to search engines, […] in addition to the XML protocol, we support RSS feeds and text files, which provide more limited information.

> You can provide an RSS (Real Simple Syndication) 2.0 or Atom 0.3 or 1.0 feed. Generally, you would use this format only if your site already has a syndication feed.