←back to thread

556 points campuscodi | 2 comments | | HN request time: 0.41s | source
Show context
kevincox ◴[] No.41865353[source]
I dislike advice of whitelisting specific readers by user-agent. Not only is this endless manual work that will only solve the problem for a subset of users but it also is easy to bypass by malicious actors. My recommendation would be to create a page rule that disables bot blocking for your feeds. This will fix the problem for all readers with no ongoing maintenance.

If you are worried about DoS attacks that may hammer on your feeds then you can use the same configuration rule to ignore the query string for cache keys (if your feed doesn't use query strings) and overriding the caching settings if your server doesn't set the proper headers. This way Cloudflare will cache your feed and you can serve any number of visitors without putting load onto your origin.

As for Cloudflare fixing the defaults, it seems unlikely to happen. It has been broken for years, Cloudflare's own blog is affected. They have been "actively working" on fixing it for at least 2 years according to their VP of product: https://news.ycombinator.com/item?id=33675847

replies(3): >>41867168 #>>41868163 #>>41869223 #
vaylian ◴[] No.41867168[source]
I don't know if cloudflare offers it, but whitelisting the URL of the RSS feed would be much more effective than filtering user agents.
replies(3): >>41867185 #>>41868217 #>>41869916 #
derkades ◴[] No.41867185[source]
Yes it supports it, and I think that's what the parent comment was all about
replies(1): >>41867257 #
BiteCode_dev ◴[] No.41867257[source]
Specifically, whitelisting the URL for the bot protection, but not the cache, so that you are still somewhat protected against adversarial use.
replies(1): >>41868789 #
londons_explore ◴[] No.41868789[source]
An adversary can easily send no-cache headers to bust the cache.
replies(1): >>41868869 #
acdha ◴[] No.41868869[source]
The CDN can choose whether to honor those. That hasn’t been an effective adversarial technique since the turn of the century.
replies(1): >>41870197 #
1. londons_explore ◴[] No.41870197[source]
does cloudflare give such an option? Even for non-paid accounts?
replies(1): >>41878921 #
2. acdha ◴[] No.41878921[source]
They ignore request cache control headers, I believe unconditionally so you’d have to disable caching for the endpoints which clients are allowed to request uncached.