A story about bypassing air Canada's in-flight network restrictions

If a ping to a specific IP times out, I wouldn't say the IP is blocked. It could be that ICMP specifically is blocked, following some network rules on the firewall. This is pretty common in entreprise networks to not allow endpoint discovery. I could be missing something and happy to be corrected here, but I was surprised to read that.

Yeah, ICMP tunnelling is also a common bypass method for captive networks, so simply blocking all ICMP seems logical.

Yes, you need to test the exact protocol you want to use. This means tcping/curl, TLS with proper certificates and SNI domains, etc.

However, just as you make sure that the power supply actually supplies power before dismantling something that refuses to work down to the last washer, repairing network problems should start with the basics. Simple test that does not work, or shows something nonsensical, is a great hint that you forgot something, or should start digging elsewhere.

I find it's important to remember, too, that a failed PING tells you nothing other than your echo request did not receive a response. If the remote host received your request, and if it responded, are both things a failed PING can't tell you, because both of those things could be true but you still end up with a failed PING.

I've seen technicians get tripped up in troubleshooting thinking that a failed PING tells them more than it does. When the possibility of asymmetric return paths is involved it's always important to remember how little a failed PING actually tells you.

I’ve had to explain this over and over throughout my career. The only way to know if something is accessible is to try the exact endpoint and protocol. Even application-aware firewalls will mess with things at times.

And that can be a lot more subtle than you might think. I've had a persistent very hard to debug false alarm triggered on pings sometimes not making it and most of the time they did. But very rarely that would happen three times in a row and that was the threshold for raising an alarm. We spent days on this. Finally, the root cause was tracked down to a BNC 'T' connector at the back of a media adapter that filtered out the header of some percentage of ICMP packets. It is one of the weirdest IT problems I've ever encountered and it makes me wonder how much of what we rely on is actually marginal.

> It is one of the weirdest IT problems I've ever encountered and it makes me wonder how much of what we rely on is actually marginal.

Vernor Vinge had a character who was a "Programmer-Archeologist" on a relativistic starship. Feels more and more prescient as time goes on.

I thought that too up until this GenAI moment, and now I wonder if needing to be an archaeologist will be so valuable if one can get your needs met by a quickly GenAI-written script/program.

I'm a SRE and encountered this recently. To prevent DDoS, there is a buffer setting on the kernel that will limit the number of pings (a few settings actually). So if you have a group of machines that all ping a single destination at once, it's very possible to have some that fail to get a reply.

I've yet to have my needs met by a GenAI-written script/program. Archaeologists tend to be a lot more precise in their statements, especially about what is speculation and what is not.

Oh, that's nasty. How long did it take you to troubleshoot that?

In my old company it was the oposite. Ping worked allways, even when you where blocked on to a specific VLAN.

Relatively speaking, it wasn't that bad. It took a few weeks of getting trouble tickets with no root cause, and a bit of googling. But management wasn't okay with fixing the root cause, instead they just increased the timeout/retry window.

I've worked in gigs that wanted that. They were all about segmentation, but wanted ICMP echo / response available throughout.

Edit: I wonder if any "enterprise" firewalls do ICMP echo proxying. Having the firewall replace the payload would remove some of the tunneling capability (thought I assume you could still finagle a side channel by just timing the packets) but would also eliminate some of the utility (since being able to craft the payload provides a way to test for specific bit patterns in packets causing problems).

I had an experience recently setting up a third-party VPN where the echo responses were being delivered to the correct (host,interface) but with the wrong destination address (not the same as made the request)

I mean, if you're willing to accept AI slop, that's fine. But if you're willing to accept AI slop, you'd probably be willing to accept human slop (at least if it claims to be AI) too, and then the job gets a lot easier.

Every time I've had to fight with path MTU discovery not working I've cursed the people who block all ICMP, though. If ICMP echo / echo-reply is the problem just block that. At the very least, allow destination unreachable / fragmentation needed thru (type 3, code 4).

Wow. That's a classic. We were quite motivated because we were the ones that got the automated alerts. I still see them in my nightmares: "chopper is down". The machine was called chopper, I'll never forget, it's been close to 30 years. My buddy Jasper and me spent multiple nights trying to track it and when we finally found it we still couldn't believe that that was it. But a simple swap was proof.

It's for reasons like this that ping is one of the worst protocols to use for aliveness.

Even worse is I've had completely dead Linux boxes that will gladly respond to ping and nothing else.

Most of the people blocking ICMP have no clue that ICMP codes/types even exist.

> I thought that too up until this GenAI moment, and now I wonder if needing to be an archaeologist will be so valuable if one can get your needs met by a quickly GenAI-written script/program.

I never have actually read those books (though I read some summaries about them, interesting concepts). My understanding is the "programmer-archeologists" basically had an archive of massive quantities of very high-quality software that did pretty much anything you'd want software to do. So it made more sense to find the software you need and glue it together than write from scratch.

And given GenAI doesn't write high quality software (at least not yet, and hopefully never), I don't think that "GenAI-written script/program" would be a good replacement (though an AI archeologist might make more sense, with such an archive).