If you use rootless Podman on a Redhat-derived distribution (which means Selinux), along with a non-root user in your container itself, you're in for a world of pain.
Either the machine is a single security domain, in which case running as root is no issue, or it's not and you need actual isolation in which case run VMs with Firecracker/Kata containers/etc.
Rootless is indeed a world of pain for dubious security promises.
If I just want to run a random Docker container, I'm grateful I can get at least "some security" without paying as much in setup/debugging/performance.
Of course, ideally I wouldn't have to choose and the thing that runs the container would be able to run it perfectly securely without me having to know that. But I appreciate any movement in that direction, even if it's not perfect.