←back to thread

556 points campuscodi | 1 comments | | HN request time: 0.223s | source
Show context
wenbin ◴[] No.41870201[source]
At Listen Notes, we rely heavily on Cloudflare to manage and protect our services, which cater to both human users and scripts/bots.

One particularly effective strategy we've implemented is using separate subdomains for services designed for different types of traffic, allowing us to apply customized firewall and page rules to each subdomain.

For example:

- www. listennotes.com is dedicated to human users. E.g., https://www.listennotes.com/podcast-realtime/

- feeds. listennotes.com is tailored for bots, providing access to RSS feeds. Eg., https://feeds.listennotes.com/listen/wenbin-fangs-podcast-pl...

- audio. listennotes.com serves both humans and bots, handling audio URL proxies. E.g., https://audio.listennotes.com/e/p/1a0b2d081cae4d6d9889c49651...

This subdomain-based approach enables us to fine-tune security and performance settings for each type of traffic, ensuring optimal service delivery.

replies(1): >>41871246 #
kevindamm ◴[] No.41871246[source]
Where do you put your sitemap (or its equivalent)? Looking at the site, I don't notice one in the metadata but I do see a "site index" on the www subdomain, though possibly that's intended for humans not bots? I think the usual recommendation is to have a sitemap per subdomain and not mix them, but clearly they're meant for bots not humans...
replies(1): >>41871391 #
1. wenbin ◴[] No.41871391[source]
Great question.

We only need to provide the sitemap (with custom paths, not publicly available) in a few specific places, like Google Search Console. This means the rules for managing sitemaps are quite manageable. It’s not a perfect setup, but once we configure it, we can usually leave it untouched for a long time.