This story is kind of sad. I wonder why the operator didn't blacklist certain netblocks/ASNs who were abusing the service.
replies(2):
A hostile reply from a netblock operator seems like a perfectly valid reason to block their traffic.
I dropped my jaw when I read it was getting 30B req/day.
Thank you for running this site for so long, and thank you for keeping it up for free, and deciding to not monetize it.
We are already doing this with trade. The amount of leeway and free lunch China has gotten from the west is insane. I don’t blame China, I blame the west and the rest of the world for not preventing it. Asymmetrical policies are often exploited by capitalism and governments have been caught off guard.
I’m not an Anti-China lunatic. It’s just common sense.
That being said, a service like this doesn't come with any guarantees and if it'd disappear from the net tomorrow, I wouldn't blame the author. Blocking is a perfectly valid solution to this problem, but assuming malice isn't always the right answer.
Were I in this situation, I'd rate limit networks per /24 (maybe even /16?) as much as I could, and work together with antivirus companies to help identify infections of malware known to use the service to discourage criminals from abusing the system. I wouldn't even bother hosting the site on IPv6 since those addresses are supposed to be public anyway. The author clearly has more patience than I do.
Small feature request: back in the day {ipv4,ipv6}.{icanhazptr,icanhazip}.com only had A / AAAA records, but now it seems they have both and thus a simple "curl ipv4.icanhzptr.com" can also give me a v6 address (of course, "curl -4" works). Would Cloudfare be OK with separating them again?
At the HTTP level it's probably cheaper to just return the HTTP 200 response. I suppose if you're doing TLS handshakes then a packet-level rate-limit would help significantly, but at the same time I'd be wary of triggering any kind of retry-behavior.
Worst-case scenario for a service like this would be having an error response/timeout trigger some kind of unlimited retry flood.
In regards to trade war, HN has discussed this ad-nauseum, I think we should restrict the discussion to internet traffic even though I brought it up as an analogy about asymmetric response from the west in general: https://hn.algolia.com/?q=trade+war
Glad it will live on!
There are a plethora of unauthenticated STUN servers around, and while there's still room for abuse, the protocol is a bit more lightweight than full-blown http requests, and faster, too!
I've dabbled with doing this on my own, but I've found `myip` to do the job nicely and without hassle:
location = /ip {
default_type text/plain;
return 200 '$remote_addr';
}
I'm curious as to what other overused utilities can be trivially done with pure server configs.
This is how cloudflare handles it for normal web services. If you’re coming from trash IPs there is no chance a curl request is going to make it through to a backend without an onerous captcha.
I'm somewhat curious what fixed things, as I've not had to use neverssl.com for some time.
Also i feel little bad you didn't get any money out of it whether the site was designed to make money or not. It would have been a wonderful end to the story if you got something back for all the years of hardwork you put into running it. You do have my appreciation if that means anything though.
P.S. this story is very similar to rawgit which was a wonderful site but also fell prey to malware aholes.
Took me 5 minutes of work and exactly one line of code.
Out of the last month, I sent out 191 abuse reports, of which 10 got replied to, 2 were resolved 6 were “no f** off” style, and 2 were told “can’t fix / won’t fix / don’t know how to fix”.
I’m not just referring to Chinese ASNs either, some US Telco’s, German, Australia even.
It's time to put this service to the first place.
I probably wouldn't bother with TLS either, just a plain HTTP 0.1 response with minimum information should be enough.
This is absolutely mind blowing.
nslookup myip.opendns.com resolver1.opendns.com
Which might not be a bad thing overall, but it's sure not gonna make any transnational corporation's bottom line happy over the next few quarters, so they'll be waving a lot of money at politicians to make this not happen.
The guy operating the NOC may be a dick, but is taking down the IoT networks for all of their customers unknowingly relying on your services really the right way?
Personally, I'd say yes, it'd help. However, there's an argument to be made that the hostile ASN operator doesn't represent the people behind the network in the slightest. I can understand that someone may give such an asshat the benefit of doubt and drop it despite their abuse.
If I saw the Time Warner ASN send too many requests, my first thought wouldn't be to just block a huge ISP. Who knows what mihjt be causing these issues and what you could be breaking by interrupting service.
The Time Warner NOC wouldn't be able to completely fix the problem if the source of the issue is the firmware of a certain shitty IoT device. If someone emailed their NOC about some weird IP cams installed by their customers causing load on their servers, they could feel like that's a problem between icanhazip and the camera manufacturer, not something they can fix.
The author is quite tolerant of the obviously malicious behaviour others are attacking his servers with. I'd have taken more aggressive measures instead of scaling up capacities myself. Because the problem is volume and not necessarily anything complex, I'd wager that even a simple block could be quite expensive because that traffic and the associated retries will be going somewhere. Directing the traffic towards the last router in their ASN through DNS would be something I'd consider, making it the problem of the network operators.
It very much echoes the problems of intellectual property theft in China.
The benefits of gobalization and the spread of democracy (or even just alternative governance models) via exposure to other cultures cannot be understated
Anyway, on abuse@ response rates, my probably unpopular but realistic take based on looking at tens of thousands of such complaints over the years and having worked for ASNs which have received millions, I'd hazard everyone has an SNR and ROI problem with handling these. There's just too many of them and most aren't actionable.
Some examples, "I saw a failed SSH login attempt from 1.2.3.4 and OMG that's a huge issue, you have been compromised, and you must solve this immediately!". OK, well, the subscriber might have: a) Typoed your IP address, b) Been running nmap/zmap over a wide range of IPs for research purposes; c) You're on an IP with a provider who recycled it to you, subscriber has outdated DNS records.
What do you expect a 'Tier 1' to do with your report?
Many ASNs are now just looking at the pattern of reports per IP address or subscriber, are automating scanning for e.g. open mail relays when whatever processes abuse@ determines the person is complaining about spam, or automating looking for anomalies in flows for DDoS complaints, a human may not even see the ticket unless the automation was able to confirm a problem may exist, and the human will probably only engage the subscriber and won't respond to the 1-1000 things received to abuse@ related to the issue.
In Major's case with icanhazip.com it looks like pretty bad behavior from the Chinese ASNs mentioned, but could just be IOT configured to fetch its IP every minute instead of every 60 minutes of 24 hours because someone misunderstood cron. Unfortunate that nobody responded but 30B a day is ~350kRPS (which isn't a lot, in the grand scheme of the internet). I'm sure 30B requests per day is nothing at Cloudflare's scale and they have options to cure these ASNs behavior should they choose, including stuff like IP-based or ASN-based ratelimiting, or even IP/ASN restrictions.
I'm sure Cloudflare will learn some interesting things about both the accidental contributors (e.g. cron) and intentional contributors (e.g. botnets) from analyzing the sources generating the requests, and I'm ultimately glad it is them picking this up, their other initiatives like 1.1.1.1 have had been positive for the internet (IMHO).
Blocking the entire country will do little to hurt the government (who can employ state resources to get whatever information they want) and do quite a bit more to harm the Chinese people by reducing whatever level of information independence they still have.
If there is going to be significant change in China, it will have to come from the Chinese people. Cutting them off from the Internet vindictively does not advance that goal.
There are specific people in China doing specific bad things with specific computing resources. It would be far better for the U.S. government to dedicate more resources to finding and partnering with orgs and projects (like icanhazip or Cloudflare) to find the info they need to apply targeted mitigations.
“China does it, so we should do it too” only makes sense as a strategy if our goal is to become exactly like China is today. I don’t think that should be our goal.
It's not really a problem anyway. If some capitalists in the US and Europe don't get to skim off a slice of profit from another country's manufacturing output, then so what?
Most likely it got them a much higher paying job than they would have otherwise gotten. Walking in and saying you single handedly run a site with billions of requests per day and petabytes of traffic will get you noticed.
I very strongly disagree. An eye for an eye is exactly what needs to be done and should have been done from the beginning. Unfortunately, it is too late. 1989 massacre should have been condemned more solidly and trade restrictions should have been placed in the 90's. The bet that western alliances made is that China would open up in the 2000s leading into 2010s. That has gone horribly wrong.
The west is finally waking up: https://en.wikipedia.org/wiki/Inter-Parliamentary_Alliance_o...
My use case was needing to set the result of `hostname -f` in /etc/hosts in an automated fashion if a VPS provider didn't already add a line for the public Internet address in that file. You need to do this so that sendmail doesn't fail on `apt install` when it attempts to read your FQDN. So I couldn't use the NGINX example posted elsewhere here.
It seems like https://checkip.amazonaws.com/ is much more "reliable" in that it is publicly documented at https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/s....
To anyone who needs to read this: please don't use "services" like icanhazip for your provisioning. Even my examples above are bad.
It does strike me as weird that there is seemingly no POSIX-compliant way to get your public Internet address, from my readings.
Edit: Oh goodness... even Amazon's documentation recommends using Google's undocumented DNS query.[1]
[1]: https://aws.amazon.com/premiumsupport/knowledge-center/route...
As long as the truth doesn't match what the preferred narrative is we'll continue to suffer the consequences, which is true of so many things beyond just attitudes towards China.
Being able to get in front of a hiring manager who is offering a high paying job and convincing them to hire you is the only thing. And having a very popular website with a ton of traffic is more likely to get you in front of a hiring manager with a good job than actually being good.
Also, I'm generally bothered by comments like this one that stir up the general human tendency toward xenophobia. We should be fighting that tendency within ourselves, not fighting the out group. Whichever group of people we want to demonize, we should remember that they're people just like us. We shouldn't punish the majority of them for what a minority are doing to us.
I agree you don’t want to cut them off, but on the other hand, I don’t want 90% of all global malicious traffic to originate from a specific country.
Is that actually true? I guess I'm inclined to believe that claims like that are more likely to be propaganda from western governments and/or western-owned companies.
If it is true, I wonder why their government isn't stopping it. They must realize that it's giving them a bad reputation in the wider world.
That would be one's own culture, whichever one that is.
These queries could load balanced across the whole internet, the code is small enough that you could just whitelist the code by checksum and then compare results from multiple underlying hosts. The fact that you need a giant corporation to be able to practically run the backend for what is essentially urn:asker.public.ip is absurd beyond belief.
The big players aren't going to fix this because the broken protocols give them a major competitive advantage.
I was reading from neverssl maintainer that they get a _lot_ of traffic, questionable ones more than it is not. Its DNS runs on AWS IIRC, and we all know Route53 isn't the cheapest.
``` <!--#echo var="REMOTE_ADDR" --> ```
Cool, I didn't know about that one.
It is not possible to know your public IP address, except by fetching the information from a known entity on the public network.
And in some scenarios, your public IP will change frequently. There is no guarantee that it will be consistent across multiple requests.
This sounds like a couple of people I've met, who have a philosophy of "treating people the way they treat me". And what if the other person/side also "believes in reciprocity", what happens then? This seems to rely on other people being nice first, and then always treating them how they treated you, imitating their behaviour, like Tit-for-tat[0]—except Tit-for-tat begins by being nice. It's not easy to put my finger on what seems fishy about that strategy, but it doesn't at all seem the easy solution to being fair and just (or whatever word you most prefer here) its proponents seem to think it is.
[0] https://en.wikipedia.org/wiki/Tit_for_tat (See particularly "Problems" and the next section)
Maybe that's an acceptable price to pay for not being the ones to take the next step toward war. If war is a game in which "the only winning move is not to play", then maybe it's also true that when it comes to doing the peaceful thing, the only winning move is to keep on playing, even if it hurts us.
It's not racist in China to say the truth, why is it where you are ? You probably live in an oppressive political regime with a biparty dictating what you can think ? :P
If you run a small-medium sized business in the US blocking all of countries you can't do business with anyway will save a ton of trouble.
That network guy (classic long hair "security" guy) was a lazy asshole for doing it then and the internet needs to have the technology to deal with bad actors beyond AS/geo-level blocking now.
No of course no reason to switch away from us! We provide many more details than icanhazip and others (geo, asn, anycast etc). We handle 40 billion requests a month,and plan to be around forever!
I'm just glad that icanhazip is staying up - it sucks when useful services go offline, and negatively impact all their users.
Even if just one port is left open, people will be curious enough to find it and use it. Chinese people are humans too :D
There is no singular thing called a "public Internet address". Imagine you're writing paper letters to someone. You write a letter, you put your own From address, you drop it in the slot. When the mailperson comes to collect the letter, they replace your mailing address with a special other codeword. And when they receive mail, they replace that codeword back with your original address. You would never know it was intercepted unless you asked around. There's no official protocol to ask for your codeword, it's just a trick the mail service does on your behalf.
Your home router does exactly this; it's known as "Network Address Translation", or NAT. It's not an official part of IPv4, and there's no protocol to ask what it is. Your computer thinks its local IP address (typically some variety of 192.168.0.1) is its real, public address, and your router does the swap behind your back.
Are you sure you're doing your part of the bilateral exchange? It cant just be China changing, the US must learn too to adapt and accept a larger, more powerful country, with a widely different model.
Living in China, I can tell you the american model is known, and not particularly impressive to them. They care a lot less about freedom of speech, maybe because they never had it, than they care qbout order, unity and crime rate for instance. And what I always hear is that throwing themselves at the communists in revolt to get the same shitty system as the US is not so seductive.
Maybe become a role model and people will beg to ressemble you ? I have a hard time convincing them voting for their government is gonna work better because "if even idiots can vote, look at who they elect" :s
Rackspace:
- Principal Architect, OpenStack - Cloud Architect, Cloud Servers
- Chief Security Architect, Director
There sort of is, it just doesn't help answer the question.
Both types of addresses have blocks explicitly carved out which are not unicast addresses to be routed over the public Internet. If you have one of those addresses, such as 192.168.0.1, that definitely isn't your "public Internet address" because people can't route stuff to it.
When do we accept the fact that Xi's ambitions extend far beyond the borders of Mainland China and pose a threat to the very idea of human dignity?
Is it when he invades Taiwan? Floods the Uyghur camps with gas? Bombs Japan? Lands an army at the port of Darwin?
It's also a good reason for CI providers to mirror package repositories.
That cert is no joke.
RHCA(dminstrator) is a great cert to get (not super difficult but not super easy, but like all RH certs you need to actually know the material).
RHCE is awesome to have, and a good bit harder than RHCAdministrator. But when I grow up I want RHCArchitect...
Once you have been allocated an IP address, there should be a way to fetch said address. That's the whole mechanism behind forging UDP packets. If I didn't know what my source IP address was, it would be OK to send 0.0.0.0 out into the world all the time.
Practically speaking, today, it is, considering most, but not all, ISPs won't let you do this and they will rewrite UDP packets that attempt to forge their source.
Edit: OK, since I'm not being clear, let me be explicit: There should be a mechanism in DHCP which allows for the querying of your public Internet address when receiving a packet from a client in one of the Private-Use Networks as defined in RFC 6890. This query should be exposed as a feature of ifconfig or ip, lest a user be forced to manually write such a packet to receive the data.
I hate certs, but if you want in on certain projects, it doesn't hurt to have the desired certs, especially if you can get your employer to foot the bill.
Anyways, knowing him personally, I can attest he's a really good dev and wholesome person. Sad to see this site get bogged down with malicious activity, because like downforeveryoneorjustme, I kind of have it committed to muscle memory. Hope CloudFlare can keep it going strong!
Well, there’s assumption 1 that isn’t true, often, you aren’t allocated a public IP address…at least in IPv4 world
Sure you could attempt to generate a bunch of keys and cycle them but it would be easier to just cache your gems.
location /ip {
add_header Content-Type "application/json";
return 200 '{"host":"$server_name","ip":"$remote_addr","port":"$remote_port","server_ip":"$server_addr","server_port":"$server_port"}\n';
}
Anyways, I agree for 1 off things, whatever, use icanhazip or whatever you want, it doesn't matter if you make 1 request a day or 1 request a month... But if you are doing anything in code that uses it, you should just host it yourself and be a good net citizen.
Upnp allows you to do that. But that relies on a lot of assumptions. Your address when connecting to X and Y could be different. It could be different depending on the port. It could be load-balanced and come randomly from a pool. It could vary depending on time of day. And many many other cases.
Because traditionally if you're doing things right, you're not using NAT, which is against IP specs and a nonstandard kludge. So you just take your socket and query its local endpoint address using getsockname and voila.
location /remote_addr {
default_type text/plain;
return 200 "$remote_addr\n";
}
dig... | tr... works fine without it.
also, `dig -4 ... ` to get your IPv4 address, for us dual-stacked folks. Otherwise it returns your V6 address by default.
Particularly in these days of IPv6, just give everything a notionally publicly routable address and then every device can know its real address (of course this doesn't have to mean you actually route public traffic to every address if you don't want to).
If this trust is repeatedly broken, peering networks may be forced to depeer the AS as a result, like what happened to McColo when they were depeered.
I sure hope that "just medicine" extends to Po Chai pills. So welcome for treating diarrhea symptoms (despite my initial skepticism) when loperamide wasn't available while I was traveling in China!
Also do keep in mind that an OS instance can have multiple Ethernet interfaces, and that each interface can have multiple IPs, and that not all DHCP gives out private addresses(University Wi-Fi gave me 133.xx back in the days ... behind NAT!), and that double NAT exists.
1: https://www.iana.org/assignments/bootp-dhcp-parameters/bootp...
It feels like this sort of data (even if only providing order of magnitude estimates) would help greatly with deciding on appropriate rate limits for small operators who don't have the time to research all the traffic they're receiving.
Sigh.
Damn, I'm almost mad at you for not hustling. Could have left with a few million easily over that entire time period, including charging Cloudfare a small fortune for the site! But hats off to you for being a true altruist. Too many paper altruists these days.
And :
"If you’re curious, Cloudflare did pay me for the site. We made a deal for them to pay me $8.03; the cost of the domain registration. The goal was never to make money from the site (although I did get about $75 in total donations from 2009 to 2021). The goal was to provide a service to the internet. Cloudflare has helped me do that and they will continue to do it as the new owners and operators of icanhazip.com."
BTW, speaking as a nerd, he has the best formatted resume that I have ever seen !
Genuinely asking because I've always used the query
dig +short myip.opendns.com @resolver1.opendns.com
I use Route53 and I either completely missed the checkip link or they simply don't mention it
Most Chinese internet users would not miss Western internet for a second, a fact you would be aware of if you actually had any insight into Chinese culture.
This attitude that you cannot give consequences to abuse because THINK OF THE POOR CHINESE is so utterly laughable.
http://ipinfo.io/ip https://test-ipv6.com/ checkip.amazon.com checkip.dyndns.com http://whatismyip.akamai.com/ ifconfig.me ip.zfx.fr // I even have mine ^^ etc...
When the host has a non-private-use address, there might still be NAT, or there might be no Internet access. I have servers in 128...* without Internet access.
The "public" address might differ depending on the destination or anything else.
"ip ad" shows the address(es) the network interfaces has, nothing more.
Most countries cooperate internationally in getting bad actors from hackers over pirates to pedos booted off the Internet and into jail.
The exceptions are China and Russia who won't do anything against any bad actor and India which is a big base for phone scams (as is Turkey for the European Union, but even Erdogan's regime is cooperating with EU police in taking down scammers).
I agree, the line between demanding at least some sort of common decency standards and xenophobia is thin in these days, but we have to get everyone on board to protect everyone else from rampant abuse.
In live networks, you don't always know which device presents your last-hop routable address to the public network, so it is not clear who can authoritatively answer the whatismyip question.
You can get an instantaneously-correct answer (i.e. correct in the instant, not necessarily quickly-answered) from an empirical test, but you need to choose an known (and trusted) entity on the public network to query. Querying multiple different entities might very well get you multiple different answers, depending on how your packets are routed and manipulated before they reach the public network.
And in all cases, there is no guarantee that your public address will be stable over time or location.
My reasoning for these kinds of stats is usually: Of course it makes sense to attack targets in jurisdictions which can't catch you or equally hide in a country which won't extradite you. (But I never looked into it to any depth, so it's baseless reasoning.)
Thanks for the blind assumption. I'm Chinese myself.
It's probably totally an option if you want to work for one of those corporations, too.
* /ip responds with the src ip of the request
* /req responds with the full headers and body of the request
As long as it is continuing working, OK. (I can verify that the returned data is correct without too much difficulty)