A New Future for Icanhazip

> There were many times where I saw a big traffic jump and I realized the traffic was coming from the same ASN, and likely from the same company. I tried reaching out to these companies when I saw it but they rarely ever replied. Some even became extremely hostile to my emails.

A hostile reply from a netblock operator seems like a perfectly valid reason to block their traffic.

The problem is that you don't know what the source of the traffic is. It could be an incompetent network operator/sysadmin, but it could just as well be something like an IP camera that people bought in good faith. If you block the CGNAT system of an operator that has a hundred million subscribers because it all seems to come from a single IP range you know nothing about, you could be hurting innocent users with the block.

That being said, a service like this doesn't come with any guarantees and if it'd disappear from the net tomorrow, I wouldn't blame the author. Blocking is a perfectly valid solution to this problem, but assuming malice isn't always the right answer.

Were I in this situation, I'd rate limit networks per /24 (maybe even /16?) as much as I could, and work together with antivirus companies to help identify infections of malware known to use the service to discourage criminals from abusing the system. I wouldn't even bother hosting the site on IPv6 since those addresses are supposed to be public anyway. The author clearly has more patience than I do.

Looking at the icanhazip.com site, I wonder how much any kind of rate-limiting per address/block would even help.

At the HTTP level it's probably cheaper to just return the HTTP 200 response. I suppose if you're doing TLS handshakes then a packet-level rate-limit would help significantly, but at the same time I'd be wary of triggering any kind of retry-behavior.

Worst-case scenario for a service like this would be having an error response/timeout trigger some kind of unlimited retry flood.

In some sense, it might not matter. If an ASN/company admin responds to emails in a hostile fashion, does it matter if they bought their devices in good faith? They're still assholes.

This isn’t “one person bought a bad camera”, it was certain ASNs accounting for a huge portion of the traffic. If the operators are unresponsive to the abuse request (making them incompetent network operators), then you absolutely block them. At that point the fallout is the fault of the network operators for operating an abuse friendly network.

This is how cloudflare handles it for normal web services. If you’re coming from trash IPs there is no chance a curl request is going to make it through to a backend without an onerous captcha.

As a network operator, you would be surprised how many AS operators are hostile or simply don’t respond. It’s unfortunately very common, even Tier-1’s are hostile.

Out of the last month, I sent out 191 abuse reports, of which 10 got replied to, 2 were resolved 6 were “no f** off” style, and 2 were told “can’t fix / won’t fix / don’t know how to fix”.

I’m not just referring to Chinese ASNs either, some US Telco’s, German, Australia even.

The block route I'd go with is blackholing the entire range into nothing through BGP or similar so the servers wouldn't have to deal with the traffic, similar to how anti DDOS tools often work. Might even redirect the DNS for that subnet to the IP of the people running the network, let them deal with the abuse. That'd be a very offensive approach, though.

I probably wouldn't bother with TLS either, just a plain HTTP 0.1 response with minimum information should be enough.

Hostility can often come from a place of ignorance or misunderstanding. I can't say much for the former, but the latter can easily go wrong with the cultural and linguistic barrier between operators.

The guy operating the NOC may be a dick, but is taking down the IoT networks for all of their customers unknowingly relying on your services really the right way?

Personally, I'd say yes, it'd help. However, there's an argument to be made that the hostile ASN operator doesn't represent the people behind the network in the slightest. I can understand that someone may give such an asshat the benefit of doubt and drop it despite their abuse.

I wouldn't expect one person with one camera to cause such a load, but popular, cheap internet cameras pull this crap all the time. I remember reading a story here about one company that hardcoded a particular IP address for their NTP bootstrapping in their firmware, with thousands of devices all across the world and no way to easily update them. Such a thing can easily happen with consumer routers and other networking equipment, generating a publicly accessible link for their user's convenience.

If I saw the Time Warner ASN send too many requests, my first thought wouldn't be to just block a huge ISP. Who knows what mihjt be causing these issues and what you could be breaking by interrupting service.

The Time Warner NOC wouldn't be able to completely fix the problem if the source of the issue is the firmware of a certain shitty IoT device. If someone emailed their NOC about some weird IP cams installed by their customers causing load on their servers, they could feel like that's a problem between icanhazip and the camera manufacturer, not something they can fix.

The author is quite tolerant of the obviously malicious behaviour others are attacking his servers with. I'd have taken more aggressive measures instead of scaling up capacities myself. Because the problem is volume and not necessarily anything complex, I'd wager that even a simple block could be quite expensive because that traffic and the associated retries will be going somewhere. Directing the traffic towards the last router in their ASN through DNS would be something I'd consider, making it the problem of the network operators.

We're entering an era where the content providers and the major clouds are >50% of demand on new subsea capacity. Feels like this blurs 'Tier 1' and the role of backbones because major eyeball networks now interconnect directly with e.g. Facebook or Google.

Anyway, on abuse@ response rates, my probably unpopular but realistic take based on looking at tens of thousands of such complaints over the years and having worked for ASNs which have received millions, I'd hazard everyone has an SNR and ROI problem with handling these. There's just too many of them and most aren't actionable.

Some examples, "I saw a failed SSH login attempt from 1.2.3.4 and OMG that's a huge issue, you have been compromised, and you must solve this immediately!". OK, well, the subscriber might have: a) Typoed your IP address, b) Been running nmap/zmap over a wide range of IPs for research purposes; c) You're on an IP with a provider who recycled it to you, subscriber has outdated DNS records.

What do you expect a 'Tier 1' to do with your report?

Many ASNs are now just looking at the pattern of reports per IP address or subscriber, are automating scanning for e.g. open mail relays when whatever processes abuse@ determines the person is complaining about spam, or automating looking for anomalies in flows for DDoS complaints, a human may not even see the ticket unless the automation was able to confirm a problem may exist, and the human will probably only engage the subscriber and won't respond to the 1-1000 things received to abuse@ related to the issue.

In Major's case with icanhazip.com it looks like pretty bad behavior from the Chinese ASNs mentioned, but could just be IOT configured to fetch its IP every minute instead of every 60 minutes of 24 hours because someone misunderstood cron. Unfortunate that nobody responded but 30B a day is ~350kRPS (which isn't a lot, in the grand scheme of the internet). I'm sure 30B requests per day is nothing at Cloudflare's scale and they have options to cure these ASNs behavior should they choose, including stuff like IP-based or ASN-based ratelimiting, or even IP/ASN restrictions.

I'm sure Cloudflare will learn some interesting things about both the accidental contributors (e.g. cron) and intentional contributors (e.g. botnets) from analyzing the sources generating the requests, and I'm ultimately glad it is them picking this up, their other initiatives like 1.1.1.1 have had been positive for the internet (IMHO).

This is not an unimportant or victimless problem, however said problem is the network operator's entire job. Making them deal with this is not uncalled-for.

This raises something I've wondered for a while: is there a service or database that can give an indication of how many humans are behind a particular IP address? e.g. with CGNAT, there might be many thousands of people sharing a single IP. For some residential services, it might be 1-2 people.

It feels like this sort of data (even if only providing order of magnitude estimates) would help greatly with deciding on appropriate rate limits for small operators who don't have the time to research all the traffic they're receiving.