At least for certain types of OSes, it should be relatively easy to get most of Linux's hardware support by porting LKL (
https://github.com/lkl/linux) and adding appropriate hooks to access hardware.
Of course, your custom kernel will still have to have some of its own code to support core platform/chipset devices, but LKL should pretty much cover just about all I/O devices (and you also get stuff like disk filesystems and a network stack along with the device drivers).
Also, it probably wouldn't work so well for typical monolithic kernels, but it should work decently on something that has user-mode driver support.