Cross-Process Spectre Exploitation

Genuinely asking, regarding Spectre (including ARM), does it really push the argument towards running sensitive software outside the cloud, even when it's resource-wise convenient? Sure, owning your hardware gives you control, but the key to mitigating Spectre is isolation. If your cloud provider can guarantee that your VMs aren't sharing physical resources with other customers (or that those resources are strongly isolated), then the Spectre risk is arguably comparable to running on your own hardware. Top cloud providers have more resources and expertise to dedicate to security updates and mitigations than a smaller operation. Maybe the real question isn't 'cloud vs. on-premise', but rather 'how well-isolated am I from other tenants in any environment?

Yes, the "cloud vs on-prem" is not really interesting. Because one project must be well isolated from other projects, regardless of their owners : as such, inside a single compagny, we must isolate all projects from each others.

Do not fall into the "I put production on one side and preproduction on the other"

Cloud providers allows great isolation, even if many people fail to implement this (for instance, by using VPC-peering / network hub / shared VPC / whatever).

Indeed, one could implement this "on-prem": vxlan and friends are there for you. It does require some skills, tho.

I believe the backbone of infrastructure security lies in two pieces: first, the ability to deploy stuff easily, quickly, autonomously. Then, the ability to deploy stuff with no cost overhead (no "price per project" or whatever).