"If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet."
> Oh, and of course, they don't just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not. They also don't give a single flying fuck about robots.txt, because why should they.
Their self righteous indignation and specificity of the pretend subject of that indignation precludes any doubt about intent.
This guy made a whole public statement that is verifiably false. And then tried to toddler logic it away when he got called out.
The author responded:
>denschub 2 days ago [–]
>the robots.txt on the wiki is no longer what it was when the bot accessed it. primarily because I clean up my stuff afterwards, and the history is now completely inaccessible to non-authenticated users, so there's no need to maintain my custom robots.txt
Which is verifiably untrue:
HTTP/1.1 200 server: nginx/1.27.2 date: Tue, 10 Dec 2024 13:37:20 GMT content-type: text/plain last-modified: Fri, 13 Sep 2024 18:52:00 GMT etag: W/"1c-62204b7e88e25" alt-svc: h3=":443", h2=":443" X-Crawler-content-encoding: gzip Content-Length: 28
User-agent: * Disallow: /w/
There are no “intentional” lies, because there are no “unintentional” lies.
All lies are intentional. An “unintentional lie” is better known as “being wrong”.
Being wrong isn’t always lying. What’s so hard about this? An example:
My wife once asked me if I had taken the trash out to the curb, and I said I had. This was demonstrably false, anyone could see I had not. Yet for whatever reason, I mistakenly believed that I had done it. I did not lie to her. I really believed I had done it. I was wrong.