Despite some confusion in terminology, modern interfaces also operate on frame buffers, but with an API for switching them out atomically. Likewise, your display server (at least modern ones) just take your client window as a completed frame buffer, and mainly serve to let multiple applications all show their frame buffers without having to worry about what other applications are doing. Frame buffers all the way down, just with support for DMA, atomicity, fancy formats and color spaces, etc.
You can use the appropriate modern APIs to make a single, exclusive graphical application with all the benefits of modern display hardware easily enough. That is, after all, what your display server is. You just don't gain much - just a little bit of saved sideband IPC.
X11 makes it seem like a display server is a complicated thing that has to support drawing and what not, but with alternatives like Wayland, a full screen client buffer is handed off zero-copy from the client application tot he hardware without being looked at.