Getting ready to issue IP address certificates

I recall one of the main arguments against Encrypted server name indication (ESNI) is that it would only be effective for the giant https proxy services like Cloudflare, that the idea of IP certs was floated as a solution but dismissed as a pipe dream. With IP address certificates, now every server can participate in ESNI, not just the giants. If it becomes common enough for clients to assume that all web servers have an IP cert and attempt to use ESNI on every connection, it could be a boon for privacy across the internet.

1. Want to connect to https://www.secret.com.

2. Resolve using DNS, get 1.2.3.4

3. Connect to 1.2.3.4, validate cert

4. Send ESNI, get separate cert for www.secret.com, validate that

... and the threat you're mitigating is presumably that you don't want to disclose the name "www.secret.com" unless you're convinced you're talking to the legitimate 1.2.3.4, so that some adversary can't spoof the IP traffic to and from 1.2.3.4, and thereby learn who's making connections to www.secret.com. Is that correct?

But the DNS resolution step is still unprotected. So, two broad cases:

1. Your adversary can subvert DNS. In this case IP certificates add no value, because they can just point you to 5.6.7.8, and you'll happily disclose "www.secret.com" to them. And you have no other path to communicate any information about what keys to trust.

2. Your adversary cannot subvert DNS. But if you can rely on DNS, then you can use it as a channel for key information; you include a key to encrypt the ESNI for "www.secret.com" in a DNS record. Even if the adversary can spoof the actual IP traffic to and from 1.2.3.4, it won't do any good because it won't have the private key corresponding to that ESNI key in the DNS. And those keys are already standardized.

So what adversary is stopped by IP certificates who isn't already stopped by the ESNI key records in the DNS?