AI companies cause most of traffic on forums

Note-worthy from the article (as some commentators suggested blocking them).

"If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet."

Ban evasion for me, but not for thee.

This is the beginning of the end of the public internet, imo. Websites that aren't able to manage the bandwidth consumption of AI scrapers and the endless spam that will take over from LLMs writing comments on forums are going to go under. The only things left after AI has its way will be walled gardens with whitelisted entrants or communities on large websites like Facebook. Niche, public sites are going to become unsustainable.

I'd kind of like to see that claim substantiated a little more. Is it all crawlers that switch to a non-bot UA, or how are they determining it's the same bot? What non-bot UA do they claim?

> Is it all crawlers that switch to a non-bot UA

I've observed only one of them do this with high confidence.

> how are they determining it's the same bot?

it's fairly easy to determine that it's the same bot, because as soon as I blocked the "official" one, a bunch of AWS IPs started crawling the same URL patterns - in this case, mediawiki's diff view (`/wiki/index.php?title=[page]&diff=[new-id]&oldid=[old-id]`), that absolutely no bot ever crawled before.

> What non-bot UA do they claim?

Latest Chrome on Windows.

Presumably they switch UA to Mozilla/something but tell on themselves by still using the same IP range or ASN. Unfortunately this has become common practice for feed readers as well.

I instigated `user-agent`-based rate limiting for exactly this reason, exactly this case.

These bots were crushing our search infrastructure (which is tightly coupled to our front end).

Yeah. Our research group has a wiki with (among other stuff) a list of open, completed, and ongoing bachelor's/master's theses. Until recently, the list was openly available. But AI bots caused significant load by crawling each page hundreds of times, following all links to tags (which are implemented as dynamic searches), prior revisions, etc. Since a few weeks, the pages are only available to authenticated users.

So you get all the IPs by rate limiting them?

Classic spam all but killed small email hosts, AI spam will kill off the web.

Super sad.

Thanks.

I would take anything the author said with a grain of salt. They straight up lied about the configuration of the robots.txt file.

https://news.ycombinator.com/item?id=42551628

What is causing you to be so unnecessarily aggressive?

Liars should be called out, necessarily. Intellectual dishonesty is cancer. I could be more aggressive if it were something that really mattered.

Lying requires intent to deceive. How have you determined their intent?

> Lying requires intent to deceive

Since when do we ask people to guess other people's intent when they have better things to show, which is called evidence?

Surely we should talk about things with substantiated matter?

When someone says:

> Oh, and of course, they don't just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not. They also don't give a single flying fuck about robots.txt, because why should they.

Their self righteous indignation and specificity of the pretend subject of that indignation precludes any doubt about intent.

This guy made a whole public statement that is verifiably false. And then tried to toddler logic it away when he got called out.

That may all be true. That still doesn’t mean they intentionally lied.

Because there’s a meaningful difference between being wrong and lying.

There’s evidence the statement was false, no evidence it was a lie.

How do you know what the contextual configuration of their robots.txt is/was?

Your accusation was directly addressed by the author in a comment on the original post, IIRC

i find your attitude as expressed here to be problematic in many ways

CommonCrawl archives robots.txt

For convenience, you can view the extracted data here:

https://pastebin.com/VSHMTThJ

You are welcome to verify for yourself by searching for “wiki.diasporafoundation.org/robots.txt” in the CommonCrawl index here:

https://index.commoncrawl.org/

The index contains a file name that you can append to the CommonCrawl url to download the archive and view.

More detailed information on downloading archives here:

https://commoncrawl.org/get-started

From September to December, the robots.txt at wiki.diasporafoundation.org contained this, and only this:

>User-agent: * >Disallow: /w/

Apologies for my attitude, I find defenders of the dishonest in the face of clear evidence even more problematic.

What is the criteria of an intentional lie, then? Admission?

The author responded:

>denschub 2 days ago [–]

>the robots.txt on the wiki is no longer what it was when the bot accessed it. primarily because I clean up my stuff afterwards, and the history is now completely inaccessible to non-authenticated users, so there's no need to maintain my custom robots.txt

Which is verifiably untrue:

HTTP/1.1 200 server: nginx/1.27.2 date: Tue, 10 Dec 2024 13:37:20 GMT content-type: text/plain last-modified: Fri, 13 Sep 2024 18:52:00 GMT etag: W/"1c-62204b7e88e25" alt-svc: h3=":443", h2=":443" X-Crawler-content-encoding: gzip Content-Length: 28

User-agent: * Disallow: /w/

Your attitude is inappropriate and violates the sitewide guidelines for discussion.

There are currently two references to “Mangion-ing” OpenAI board members in this thread, several more from Reddit, based on the falsehoods being perpetrated by the author. Is this really someone you want to conspire with? Is calling this out more egregious than the witch hunt being organized here?

> intentional lie

There are no “intentional” lies, because there are no “unintentional” lies.

All lies are intentional. An “unintentional lie” is better known as “being wrong”.

Being wrong isn’t always lying. What’s so hard about this? An example:

My wife once asked me if I had taken the trash out to the curb, and I said I had. This was demonstrably false, anyone could see I had not. Yet for whatever reason, I mistakenly believed that I had done it. I did not lie to her. I really believed I had done it. I was wrong.

"conspire" and "witch hunt", are not terms of productive discourse.

If you are legitimately trying to correct misinformation, your attitude, tone and language are counter productive. You would be much better seved by taking that energy and crafting an actually persuasive argument. You come across as unreasonable and unwilling to listen, not someone with a good grasp of the technical specifics.

I don't have a horse in the race. I'm fairly technical, but I did not find your arguments persuasive. This doesn't mean they are wrong, but it does mean that you didn't do a good job of explaining them.

No worries, I understand. The author admitted to me that he was lying via DM, that he often does this for attention.