There was a prior concern in the history of Let's Encrypt about hosting providers that have multiple customers on the same server. In fact, phenomena related to that led to the deprecation of one challenge method and the modification of another one, because it's important that one customer not be able to pass CA challenges on behalf of another customer just because the two are hosted on the same server.
But there was no conclusion that customers on the same server can't get certificates just because they're on the same server, or that whoever legitimately controls the default server for an IP address can't get them.
This would be a problem if clients would somehow treat https://example.com/ and https://96.7.128.175/ as the same identifier or security context just because example.com resolves to 96.7.128.175, but I'm not aware of clients that ever do so. Are you?
If clients don't confuse these things in some automated security sense, I don't see how IP address certs are worse than (or different from) certs for different customers who are hosted on the same IP address.