←back to thread

AI web crawlers are destroying websites in their never-ending content hunger

(www.theregister.com)
211 points CrankyBear | 4 comments | 02 Sep 25 16:24 UTC | HN request time: 0.623s | source
Show context
bdefore ◴[02 Sep 25 18:26 UTC] No.45107084[source]▶
I created and maintain ProtonDB, a popular Linux gaming resource. I don't do ads, just pay the bills from some Patreon donations.

It's a statically generated React site I deploy on Netlify. About ten days ago I started incurring 30GB of data per day from user agents indicating they're using Prerender. At this pace almost all of that will push me past the 1TB allotted for my plan, so I'm looking at an extra ~$500USD a month for the extra bandwdith boosters.

I'm gonna try the robots.txt options, but I'm doubtful this will be effective in the long run. Many other options aren't available if I want to continue using a SaaS like Netlify.

My initial thoughts are to either move to Cloudflare Pages/Workers where bandwidth is unlimited, or make an edge function that parses the user agent and hope it's effective enough. That'd be about $60 in edge function invocations.

I've got so many better things to do than play whack-a-mole on user agents and, when failing, pay this scraping ransom.

Can I just say fuck all y'all AI harvesters? This is a popular free service that helps get people off of their Microsoft dependency and live their lives on a libre operating system. You wanna leech on that? Fine, download the data dumps I already offer on an ODbL license instead of making me wonder why I fucking bother.

replies(9): >>45107161 #>>45107210 #>>45107270 #>>45107274 #>>45107832 #>>45107839 #>>45108629 #>>45108648 #>>45134051 #
gjsman-1000 ◴[02 Sep 25 18:32 UTC] No.45107161[source]▶
Your mistake is openly suggesting on HN that you're going to use Cloudflare, increasing the centralization of the internet and contributing to their attestation schemes, while society forces you to be a victim of the tragedy of the commons.
replies(1): >>45107171 #
1. bdefore ◴[02 Sep 25 18:33 UTC] No.45107171[source]▶
Please believe me that it is not a step I want to take.
replies(2): >>45107200 #>>45107418 #
2. gjsman-1000 ◴[02 Sep 25 18:35 UTC] No.45107200[source]▶
You don't need to apologize - HN needs to get their heads out of the sand that not everything is a tragedy of the commons, there's a reason why centralization exists, and the decentralized internet as it is now comes with serious drawbacks. We're never going to overcome the popularity of big tech if we can't be honest with the problems they solve.

Also, sue me, the cathedral has defeated the bazaar. This was predictable, as the bazaar is a bunch of stonecutters competing with each other to sell the best stone for building the cathedral with. We reinvented the farmer's market, and thought that if all the farmers united, they could take down Walmart. It's never happening.

replies(1): >>45108597 #
3. azdle ◴[02 Sep 25 18:52 UTC] No.45107418[source]▶
Another option that wouldn't contribute to more centralization might be neocities. They give you 3 TB for $5/month. That seems to be _the_ limit though. The dude runs his own CDN just for neocities, so it's not just reselling cloudflare or something.

P.S. Thank you for ProtonDB, it has been so incredibly helpful for getting some older games running.

4. hombre_fatal ◴[02 Sep 25 20:30 UTC] No.45108597[source]▶
In this context, the farmers are trying to deal with rampant abuse that is inconceivable to handle on an individual level.

It's not clear to me what taking down Cloudflare/Walmart means in this context. Nor how banding together wouldn't just incur the very centralization that is presumably so bad it must be taken down.