Usually you can just import the leaf self signed cert as a CA in your OS trust store and the problem goes away (assuming it has an IP SAN). Slightly tedious but you can issue the certs with long validity
Let me rephrase that: How is the CA supposed to know they didn't handshake with an attacker? All they have is the IP, there's no identity to check like with DNS.