←back to thread

AI companies cause most of traffic on forums

(pod.geraspora.de)
770 points ta988 | 3 comments | 30 Dec 24 14:37 UTC | HN request time: 0.018s | source
Show context
markerz ◴[30 Dec 24 17:07 UTC] No.42551173[source]▶
One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-...

It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again.

My solution was to add a Cloudflare rule to block requests from their User-Agent. I also added more nofollow rules to links and a robots.txt but those are just suggestions and some bots seem to ignore them.

Cloudflare also has a feature to block known AI bots and even suspected AI bots: https://blog.cloudflare.com/declaring-your-aindependence-blo... As much as I dislike Cloudflare centralization, this was a super convenient feature.

replies(14): >>42551260 #>>42551410 #>>42551412 #>>42551513 #>>42551649 #>>42551742 #>>42552017 #>>42552046 #>>42552437 #>>42552763 #>>42555123 #>>42562686 #>>42565119 #>>42572754 #
TuringNYC ◴[30 Dec 24 18:22 UTC] No.42552017[source]▶
>> One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-...

Are they not respecting robots.txt?

replies(1): >>42552085 #
1. eesmith ◴[30 Dec 24 18:29 UTC] No.42552085[source]▶
Quoting the top-level link to geraspora.de:

> Oh, and of course, they don’t just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not. They also don’t give a single flying fuck about robots.txt, because why should they. And the best thing of all: they crawl the stupidest pages possible. Recently, both ChatGPT and Amazon were - at the same time - crawling the entire edit history of the wiki.

replies(1): >>42562600 #
2. vasco ◴[31 Dec 24 23:18 UTC] No.42562600[source]▶
Edit history of a wiki sounds much more interesting than the current snapshot if you want to train a model.
replies(1): >>42564834 #
3. eesmith ◴[01 Jan 25 08:55 UTC] No.42564834[source]▶
Does that information improve or worsen the training?

Does it justify the resource demands?

Who pays for those resources and who benefits?