Why did Windows 95 setup use three operating systems?

Because, when they did it right, in Windows NT 3.51, the users with legacy 16 bit applications screamed. There was a 16-bit DOS compatibility box, but it wasn't bug-compatible with DOS.

Microsoft underestimated the inertia of the applications market. NT 3.51 was fine if you used it as a pure 32-bit operating system. You could even configure it without DOS compatibility. Few did.

Something the Unix world can certainly learn from.

Sun used to take binary compatibility very seriously. Solaris 8 (and perhaps later releases) still had a compatibility layer for SunOS 4.x binaries. Solaris 11 can still run Solaris 2.6 binaries.

Linux is another matter entirely, if your binaries run at all from one distribution release to the next you're doing well.

To be clear it is not binary formats themselves that change and cause incompatibility. ELF is ELF and hasn't appreciably changed. And it isn't even really kernel syscalls (though that isn't etched in stone, I don't get the impress it's changed that much). The problem is the libc or other shared libraries.

Seems like the way that this is "fixed" is by using containers. But it feels so...bloated.

The solution that Windows provides for the problem -- WinSxS -- is, ultimately, no less bloated.