My cousin manages a dozens of mid-sized informational websites and communities, his former hosting provider kicked him out because he refused to pay the insane bills as a result of literally AI bots DDoS-ing his sites...
He unfortunately had no choice to put most of the content behind a login-wall (you can only see parts of the articles/forum posts when logged out) but he is strongly considering just hard pay-walling some content at that point... We're talking about someone who in good faith provided partial data dumps of content freely available for these companies to download, but, caching / etags? none of these AI companies, hiring "the best and the brightest" have ever heard of that, rate limiting? LOL what is that?
This is nuts, these AI companies are ruining the web.