←back to thread

AI companies cause most of traffic on forums

(pod.geraspora.de)
770 points ta988 | 3 comments | 30 Dec 24 14:37 UTC | HN request time: 0s | source
Show context
markerz ◴[30 Dec 24 17:07 UTC] No.42551173[source]▶
One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-...

It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again.

My solution was to add a Cloudflare rule to block requests from their User-Agent. I also added more nofollow rules to links and a robots.txt but those are just suggestions and some bots seem to ignore them.

Cloudflare also has a feature to block known AI bots and even suspected AI bots: https://blog.cloudflare.com/declaring-your-aindependence-blo... As much as I dislike Cloudflare centralization, this was a super convenient feature.

replies(14): >>42551260 #>>42551410 #>>42551412 #>>42551513 #>>42551649 #>>42551742 #>>42552017 #>>42552046 #>>42552437 #>>42552763 #>>42555123 #>>42562686 #>>42565119 #>>42572754 #
CoastalCoder ◴[30 Dec 24 17:30 UTC] No.42551410[source]▶
I wonder if it would work to send Meta's legal department a notice that they are not permitted to access your website.

Would that make subsequent accesses be violations of the U.S.'s Computer Fraud and Abuse Act?

replies(3): >>42551475 #>>42551557 #>>42551847 #
betaby ◴[30 Dec 24 17:37 UTC] No.42551475[source]▶
Crashing wasn't the intent. And scraping is legal, as I remember per Linkedin case.
replies(3): >>42551556 #>>42551790 #>>42551791 #
1. franga2000 ◴[30 Dec 24 18:02 UTC] No.42551791[source]▶
If I make a physical robot and it runs someone over, I'm still liable, even though it was a delivery robot, not a running over people robot.

If a bot sends so many requests that a site completely collapses, the owner is liable, even though it was a scraping bot and not a denial of service bot.

replies(1): >>42552206 #
2. stackghost ◴[30 Dec 24 18:40 UTC] No.42552206[source]▶
The law doesn't work by analogy.
replies(1): >>42563460 #
3. maximinus_thrax ◴[01 Jan 25 02:14 UTC] No.42563460[source]▶
Except when it does https://en.wikipedia.org/wiki/Analogy_(law)