←back to thread

AI companies cause most of traffic on forums

(pod.geraspora.de)
770 points ta988 | 3 comments | 30 Dec 24 14:37 UTC | HN request time: 0.448s | source
Show context
mentalgear ◴[30 Dec 24 17:42 UTC] No.42551541[source]▶
Note-worthy from the article (as some commentators suggested blocking them).

"If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet."

replies(5): >>42551717 #>>42551976 #>>42552122 #>>42552700 #>>42552885 #
1. IanKerr ◴[30 Dec 24 18:18 UTC] No.42551976[source]▶
This is the beginning of the end of the public internet, imo. Websites that aren't able to manage the bandwidth consumption of AI scrapers and the endless spam that will take over from LLMs writing comments on forums are going to go under. The only things left after AI has its way will be walled gardens with whitelisted entrants or communities on large websites like Facebook. Niche, public sites are going to become unsustainable.
replies(2): >>42552793 #>>42553743 #
2. raphman ◴[30 Dec 24 19:56 UTC] No.42552793[source]▶
Yeah. Our research group has a wiki with (among other stuff) a list of open, completed, and ongoing bachelor's/master's theses. Until recently, the list was openly available. But AI bots caused significant load by crawling each page hundreds of times, following all links to tags (which are implemented as dynamic searches), prior revisions, etc. Since a few weeks, the pages are only available to authenticated users.
3. oblio ◴[30 Dec 24 21:26 UTC] No.42553743[source]▶
Classic spam all but killed small email hosts, AI spam will kill off the web.

Super sad.