> Anubis uses a proof-of-work challenge to ensure that clients are using a modern browser and are able to calculate SHA-256 checksums
https://anubis.techaro.lol/docs/design/how-anubis-works
This is pretty cool, I have a project or two that might benefit from it.
But I find that when it comes to simple serving of content, human vs. bot is not usually what you’re trying to filter or block on. As long as a given client is not abusing your systems, then why do you care if the client is a human?
Well, that's the rub. The bots are abusing the systems. The bots are accessing the contents at rates thousands of times faster and more often than humans. The bots also have access patterns unlike your expected human audience (downloading gigabytes or terabytes of data multiples times, over and over).
And these bots aren't some being with rights. They're tools unleashed by humans. It's humans abusing the systems. These are anti-abuse measures.
And if that doesn't happen, you go to their ISP's ISP and get their ISP booted off the Internet.
Actual ISPs and hosting providers take abuse reports extremely seriously, mostly because they're terrified of getting kicked off by their ISP. And there's no end to that - just a chain of ISPs from them to you and you might end with convincing your ISP or some intermediary to block traffic from them. However, as we've seen recently, rules don't apply if enough money is involved. But I'm not sure if these shitty interim solutions come from ISPs ignoring abuse when money is involved, or from not knowing that abuse reporting is taken seriously to begin with.
Anyone know if it's legal to return a never-ending stream of /dev/urandom based on the user-agent?
You will be surprised on how many ISPs will not respond. Sure, Hetzner will respond, but these abusers are not using Hetzner at all. If you actually studied the actual problem, these are residential ISPs in various countries (including in US and Europe, mind you). At best the ISP will respond one-by-one to their customers and scan their computers (and at this point the abusers have already switched to another IP block) and at worst the ISP literally has no capability to control this because they cannot trace their CGNATted connections (short of blocking connections to your site, which is definitely nuclear).
> And if that doesn't happen, you go to their ISP's ISP and get their ISP booted off the Internet.
Again, the IP blocks are rotated, so by the time that they would respond you need to do the whole reporting rigomarole again. Additionally, these ISPs would instead suggest to blackhole these requests or to utilize a commercial solution (aka using Cloudflare or something else), because at the end of the day the residential ISPs are national entites that would quite literally trigger geopolitcal concerns if you disconnected them.
They're certainly positioning themselves for providing scraping servers for AI training. What will they do when I say that one of their customers just hit my server with 1000 requests per second? Ban the customer?
Let's be rational. They'll laugh at that mail and delete it. Bigger players use "home proxying" services which use residental blocks for egress, and make one request per host. Some people are cutting whole countries off with firewalls.
Playing by old rules won't get you anywhere, because all these gentlemen took their computers and work elsewhere. Now we all have are people who think they need no permission because what they do is awesome, anyway (which is not).
At the minimum they're very likely to have a talk with their customer "keep this shit up and you're outta here"
And those residential proxy services cost their customer around $0.50/GB up to $20/GB. Do with that knowledge what you will.
Good luck with that. Have you ever tried? AWS and Google have abuse mails. Do you think they read them? Do you think they care? It is basically impossible to get AWS to shutdown a customers systems, regardless of how much you try.
I believe ARIN has an abuse email registered for a Google subnet, with the comment that they believe it's correct, but no one answer last time they tried it, three years ago.
The hierarchy is: abuse contact of netblock. If ignored: abuse contact of AS. If ignored: Local internet registry (LIR) managing the AS. If ignored: Internet Registry like ARIN.
I see a possibility of automation here.
Also, report to DNSBL providers like Spamhaus. They rely on reports to blacklist single IPs, escalate to whole blocks and then the next larger subnet, until enough customers are affected.
Yup, I'm assuming that immibis thinks that the ones using Anubis are those ones with high legal budgets, but this is not necessarily the case here.
Assume that you are in the shoes of Anubis users. Do you have a reasonable legal budget? No? From experience, most ISPs would not really respond unless either their network has become unstable as a consequence, or if legal advised them to cooperate. Realistically, at the time that they read your plea the activity has already died off (on their network), and the best that they can do is to give you the netflows to do your investigation.
> You think they wouldn't cut off customers who DDoS?
This is not your typical DDoS where the stability of the network links are affected (this is at the ISP level, not specifically your server), this is a very asymmetrical one where it seemingly blends out as normal browsing. Unless you have a reasonable legal budget, they would suggest to use RTBH (https://www.cisco.com/c/dam/en_us/about/security/intelligenc...) or a commercial filtering solution if need be. This even assumes that they're symphatetic to your pleas, at worst case you're dealing with state-backed ISPs that are known not to respond at all.