On the other hand, all of the input, graphics and "other crap" being mandatory means that as a developer you can pretty much guarantee that they are there and if you care about having your stuff work everywhere (well, everywhere X is available anyway) you can simply rely on that without having to support a bunch of different APIs that may or may not be there (see the audio situation a few years ago in Linux).
(also FWIW a lot of people use the X primitives, not everyone uses Gtk or Qt)