AI companies cause most of traffic on forums

It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again.

My solution was to add a Cloudflare rule to block requests from their User-Agent. I also added more nofollow rules to links and a robots.txt but those are just suggestions and some bots seem to ignore them.

Cloudflare also has a feature to block known AI bots and even suspected AI bots: https://blog.cloudflare.com/declaring-your-aindependence-blo... As much as I dislike Cloudflare centralization, this was a super convenient feature.

In addition to other crushing internet risks, add wrongly blacklisted as a bot to the list.

[0] Previously Soviet-aligned countries; i.e. Russia and eastern Europe.

After doing so, all of our logs, like ssh auth etc, are almost completely free and empty of malicious traffic. It’s actually shocking how well a blanket ban worked for us.