←back to thread

AI companies cause most of traffic on forums

(pod.geraspora.de)
770 points ta988 | 1 comments | 30 Dec 24 14:37 UTC | HN request time: 0.001s | source
Show context
markerz ◴[30 Dec 24 17:07 UTC] No.42551173[source]▶
One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-...

It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again.

My solution was to add a Cloudflare rule to block requests from their User-Agent. I also added more nofollow rules to links and a robots.txt but those are just suggestions and some bots seem to ignore them.

Cloudflare also has a feature to block known AI bots and even suspected AI bots: https://blog.cloudflare.com/declaring-your-aindependence-blo... As much as I dislike Cloudflare centralization, this was a super convenient feature.

replies(14): >>42551260 #>>42551410 #>>42551412 #>>42551513 #>>42551649 #>>42551742 #>>42552017 #>>42552046 #>>42552437 #>>42552763 #>>42555123 #>>42562686 #>>42565119 #>>42572754 #
bodantogat ◴[30 Dec 24 17:50 UTC] No.42551649[source]▶
I see a lot of traffic I can tell are bots based on the URL patterns they access. They do not include the "bot" user agent, and often use residential IP pools. I haven't found an easy way to block them. They nearly took out my site a few days ago too.
replies(5): >>42551680 #>>42551803 #>>42556117 #>>42558781 #>>42574346 #
echelon ◴[30 Dec 24 18:03 UTC] No.42551803[source]▶
You could run all of your content through an LLM to create a twisted and purposely factually incorrect rendition of your data. Forward all AI bots to the junk copy.

Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills.

Maybe you don't even need a full LLM. Just a simple transformer that inverts negative and positive statements, changes nouns such as locations, and subtly nudges the content into an erroneous state.

replies(5): >>42551837 #>>42551968 #>>42552052 #>>42553499 #>>42553755 #
endofreach ◴[30 Dec 24 21:04 UTC] No.42553499[source]▶
> Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills

Or just wait for after the AI flood has peaked & most easily scrapable content has been AI generated (or at least modified).

We should seriously start discussing the future of the public web & how to not leave it to big tech before it's too late. It's a small part of something i am working on, but not central. So i haven't spend enough time to have great answers. If anyone reading this seriously cares, i am waiting desperately to exchange thoughts & approaches on this.

replies(1): >>42563943 #
1. jorvi ◴[01 Jan 25 04:25 UTC] No.42563943{3}[source]▶
Very tangential but you should check out the old game “Hacker BS Replay”.

It’s basically about how in 2012, with the original internet overrun by spam, porn and malware, all the large corporations and governments got together and created a new, tightly-controlled clean internet. Basically how modern Apple & Disneyland would envision the internet. On this internet you cannot choose your software, host your own homepage or have your own e-mail server. Everyone is linked to a government ID.

We’re not that far off:

- SaaS

- Gmail blocking self-hosted mailservers

- hosting your own site becoming increasingly cumbersome, and before that MySpace and then Meta gobbled up the idea of a home page a la GeoCities.

- Secure Boot (if Microsoft locked it down and Apple locked theirs, we would have been screwed before ARM).

- Government ID-controlled access is already commonplace in Korea and China, where for example gaming is limited per day.

In the Hacker game, as a response to the new corporate internet, hackers started using the infrastructure of the old internet (“old copper lines”) and set something up called the SwitchNet, with bridges to the new internet.