Why did Windows 95 setup use three operating systems?

Because, when they did it right, in Windows NT 3.51, the users with legacy 16 bit applications screamed. There was a 16-bit DOS compatibility box, but it wasn't bug-compatible with DOS.

Microsoft underestimated the inertia of the applications market. NT 3.51 was fine if you used it as a pure 32-bit operating system. You could even configure it without DOS compatibility. Few did.

Something the Unix world can certainly learn from.

Sun used to take binary compatibility very seriously. Solaris 8 (and perhaps later releases) still had a compatibility layer for SunOS 4.x binaries. Solaris 11 can still run Solaris 2.6 binaries.

Linux is another matter entirely, if your binaries run at all from one distribution release to the next you're doing well.

Linux doesn't need binary compatibility as much as Windows, lot of source packages will compile right away with a vast array of different operating systems, typically excluding Windows but including Linux, and Linux is a few clicks away from running a fair number of MS-DOS and Windows applications, probably more than any single Windows version. Linux is king in compatibility.

That doesn't help you at all when you don't have the source and even then, compiler changes break source compatibility all the time.

Not just Linux, Mac too (people forget that they run a certified BSD kernel).