AI companies cause most of traffic on forums

Note-worthy from the article (as some commentators suggested blocking them).

"If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet."

I'd kind of like to see that claim substantiated a little more. Is it all crawlers that switch to a non-bot UA, or how are they determining it's the same bot? What non-bot UA do they claim?

I would take anything the author said with a grain of salt. They straight up lied about the configuration of the robots.txt file.

https://news.ycombinator.com/item?id=42551628

What is causing you to be so unnecessarily aggressive?

Liars should be called out, necessarily. Intellectual dishonesty is cancer. I could be more aggressive if it were something that really mattered.

Lying requires intent to deceive. How have you determined their intent?

When someone says:

> Oh, and of course, they don't just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not. They also don't give a single flying fuck about robots.txt, because why should they.

Their self righteous indignation and specificity of the pretend subject of that indignation precludes any doubt about intent.

This guy made a whole public statement that is verifiably false. And then tried to toddler logic it away when he got called out.

That may all be true. That still doesn’t mean they intentionally lied.

What is the criteria of an intentional lie, then? Admission?

The author responded:

>denschub 2 days ago [–]

>the robots.txt on the wiki is no longer what it was when the bot accessed it. primarily because I clean up my stuff afterwards, and the history is now completely inaccessible to non-authenticated users, so there's no need to maintain my custom robots.txt

Which is verifiably untrue:

HTTP/1.1 200 server: nginx/1.27.2 date: Tue, 10 Dec 2024 13:37:20 GMT content-type: text/plain last-modified: Fri, 13 Sep 2024 18:52:00 GMT etag: W/"1c-62204b7e88e25" alt-svc: h3=":443", h2=":443" X-Crawler-content-encoding: gzip Content-Length: 28

User-agent: * Disallow: /w/