←back to thread

Using Cloudflare on your website could be blocking RSS users

(openrss.org)
556 points campuscodi | 2 comments | 16 Oct 24 22:46 UTC | HN request time: 0.536s | source
1. wraptile ◴[17 Oct 24 11:32 UTC] No.41868597[source]▶
Cloudflare has been the bane of my web existance on Thai IP and a Linux Firefox fingerprint. I wonder how much traffic is lost because of Cloudflare and of course none of that is reported to the web admins so everyone continues with their jolly ignorance.

I wrote my own RSS bridge that scrapes websites using Scrapfly web scraping API that bypasses all that because it's so annoying that I can't even scrape some company's /blog that they are literally buying ads for but somehow have an anti-bot enabled that blocks all RSS readers.

Modern web is so anti social that the web 2.0 guys should be rolling in their "everything will be connected with APIs" graves by now.

replies(1): >>41870021 #
2. vundercind ◴[17 Oct 24 14:29 UTC] No.41870021[source]▶
The late '90s-'00s solution was to blackhole address blocks associated with entire countries or continents. It was easily worth it for many US sites that weren't super-huge to lose the the 0.1% of legitimate requests they'd get from, say, China or Thailand or Russia, to cut the speed their logs scrolled at by 99%.

The state of the art isn't much better today, it seems. Similar outcome with more steps.