Answers I've received before is "it can call the native API", but any program can call the native API. So, what's in a subsystem?
Answers I've received before is "it can call the native API", but any program can call the native API. So, what's in a subsystem?
The Native API is mostly implemented by the kernel itself, ntoskrnl.exe, which has a system call table (KiServiceTable). Most of the native API is exposed to userspace via ntdll.dll, which calls through that system call table. This API is available from very early boot.
The Win32 subsystem is mostly exposed by win32k.sys, which is a kernel mode driver. It has a second system call table (W32pServiceTable) which is then consumed by user32.dll and friends. Some parts of the Win32 user-mode API are implemented by calling the Native API via ntdll.dll, though.
Processes which exist in the Win32 subsystem have additional kernel data associated with them (the private state of win32k.sys for that process), had additional win32k.sys code ran during initialization, and can therefore make use of the services of that driver.
If you create a Native process with RtlCreateUserProcess, it won't be able to make Win32 API calls. But it can run in environments where a Win32 process can't -- for example, autochk.exe, which is the version of chkdsk which runs as part of boot before the graphics driver has loaded, is a Native executable.
There's also support in the PE executable format to indicate which subsystem to use, so the loader knows which type of process to create. Obviously, this won't be relevant for running Linux ELF executables.
The layering isn't perfect, especially since while NT was designed to support multiple subsystems (OS/2, Win32, Posix), Win32 is the only one which Microsoft have historically focused on.