←back to thread

556 points campuscodi | 1 comments | | HN request time: 0.193s | source
1. pentagrama ◴[] No.41871183[source]
Can you whitelists urls to be readead by bot on Cloudflare? Maybe this is a good solution, and there you can put your RSS feeds, sitemaps, and other content for bots. Also Cloudflare can make a dedicated fields to whitelists RSS and Sitemaps on the admin panel so users can discover more easily that they may don't want block those bots.

Can you whitelist URLs to be read by bots on Cloudflare? Maybe this is a good solution, where you as a site mantainer can include your RSS feeds, sitemaps, and other content for bots.

Also, Cloudflare could ship a feature by creating a dedicated section in the admin panel to let the user add and whitelist RSS feeds and sitemaps, making it easier (and educate) users to avoid blocking those bots who aren't a threat to your site, of course sill considering rules to avoid DDOS on this urls, like massive requests or stuff that common bots from RSS readers don't do.