Mitigating the Hetzner/Linode XMPP.ru MitM interception incident

* The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt for jabber.ru and xmpp.ru domains since 18 Apr 2023

* The Man-in-the-Middle attack for jabber.ru/xmpp.ru client XMPP traffic decryption confirmed to be in place since at least 21 July 2023 for up to 19 Oct 2023, possibly (not confirmed) since 18 Apr 2023, affected 100% of the connections to XMPP STARTTLS port 5222 (not 5223)

* The attacker failed to reissue TLS certificate and MiTM proxy started to serve expired certificate on port 5222 for jabber.ru domain (Hetzner)

* The MiTM attack stopped shortly after we begun our investigation and network tests on 18 Oct 2023, along with tickets to Hetzner and Linode support team, however passive wiretapping (additional routing hop) is still in place at least on a single Linode server

* Neither servers appear to be hacked

* Both Hetzner and Linode network appear to be reconfigured specifically for this kind of attack for the XMPP service IP addresses

---

Neither that page, nor the page linked from here, mention certificate pinning, maybe because XMPP doesn't support it (I don't know), but if it did, wouldn't that have prevented this kind of attack?

My understanding is that certificate pinning is only possible if you control the clients, in which case you can embed which certificates are allowed directly in the client and bypass the whole web PKI.

In a situation with general-purpose clients connecting, how would they know which certificates are meant to be allowed? That's what the web PKI is used for.

Of course, if you do provide your own clients, this just moves the problem further up the chain - in this case the place where customers would download the custom client software would be compromised and a malicious client served instead.

Well you cannot. If you were paranoid, you would perhaps supply a hash through some out-of-band mechanism, which would require manually updating for each new cert.

Obviously most people wouldn't ever want to do that.

XMPP clients could implement such a mechanism, and if any certificate/domain along the path changes, the users in a conversation would be notified.

This suggests a compromise of Hetzner and Linode network management.

This implies that the attacker was able to validate domain ownership of jabber.ru and xmpp.ru by either a) being able to respond to the ACME protocol on domain(s) hosted http OR b) being able to write to a DNS TXT file for the domain(s). This implies that the victim already lost control of their process server and/or their nameserver.

Same can happen to you.

This is gold.

Why did they only target the STARTTLS port? On a related note, I would never use the STARTTLS port (opportunistic encryption) if I knew that the server had a regular TLS port...

No it doesn't. It suggests that they both complied with a request from law enforcement.

The MiTM attacker can pass through a command stream without STARTTLS. If they intercepted 5223 they would have to do their own client-side TLS handshake with the attacked server, which would look really obvious to anybody doing TLS fingerprinting on the server: all of a sudden, 100% of their clients have the exact same TLS fingerprint.

Stop outsourcing your PKI to ICANN, folks. Domains are not public keys.

Use them.

Stop using domains. Stop.

The next level of mitigating that sort of a thing is to have keys not be on the hosts at all. Enter HSM - Hardware Security Module. A wildly complex topic I cannot hope to cover in an HN comment, but fundamentally, the private keys are not on the same HW as the server software which needs them.

A fundamental property of an HSM is that you, the HSM user, don't actually see the private key. You can ask the HSM to generate one. Derive things from it (in the cryptographic sense of derive). Even prove the provenance of such derived data. But the HSM should not reveal the actual private key.

In the cloud world, the HSM equivalent is known as KMS (Key Management Service), and the Good cloud providers all let you manage your own (with the downside being that you now need to manage your own keys).

1. https://dns.coffee/nameservers/ns1.jabber.ru

As I understand the techniques applied; this could have also been done at another place on the network route to the targeted server.

Ie. by the telecommunications company delivering traffic into Hetzner.

Them expanding into mtm https wiretapping is a new for me but maybe to be expected …

Pinning is problematic these days, because certificates are short-lived and renewed frequently. Users would be constantly asked to accept new certificates on a monthly basis, and they wouldn't think twice about clicking 'Accept' on an attacker's certificate then.

Channel binding works regardless of the certificate, and ensures the TLS stream is terminated by the entity you exepect.

The focus on (lack of) CT/SCT support in XMPP clients puzzles me, because it would not have detected this attack at all, which used valid CT-logged certs from Let's Encrypt. SCT verification would help detect theoretical issuance from another CA if they had a strict CAA record in place (they did not). But even then, channel binding is more privacy-friendly and does not depend on a third party.

Also you can phase in a new cert with pinning.

Since this affected only one client and it's more likely than not a lawful intercept, it's not a compromise.

> they would have to do their own client-side TLS handshake

By intercepting the STARTTLS port the attacker can merely decrypt -- rather than, as you wrote, decrypting and re-encrypting.

> Traffic dump on port 5222, the connection is hijacked on application level (L7), the server receives replaced ClientHello message from the client.

  --reuse-key           When renewing, use the same private key as the
                        existing certificate. (default: False)

Not just. The disk itself may be encrypted, the keys may be encrypted. Doing this unnoticed and without rebooting would require modifications to the hardware.

In hacking, compromised means hacked.

In intelligence, compromised means you have someone doing something on your behalf that they feel forced to do.

You seem to understand this so I'm not sure where the confusion lies.

That is what XMPP clients tend to do...

These days XMPP servers tend to default to requiring TLS on both 5222 and 5223 (Let's Encrypt has changed everything). Prosody does this for example. It doesn't even support port 5223 by default anymore. Port 5223 was never an official port assignment.

So it is very possible that the MiTM was only done on port 5222 because that was the only port that clients were using.

Also, I've never understood why they've moved 5223 (regular TLS) into deprecated. It was pretty useful to enable SSL on AWS ELB on this port. Which is not possible with 5222, because it does XML stuff before switching to TLS.

I would speculate it is to not keep 2 ports open or something, was a reason to move it to deprecated. + you can do some cleartext communication before switching to TLS (not like it is that useful).

There is the plain incompetence explanation: the hosting provider gave control of the operation to the government entity. The underpaid and indifferent government employee did the best they could with their level motivation and skill level.