It is not that hard, DOM model is amazing, that's why there are so many frontend frameworks in the first place.
However, you'll inevitably end up with your own after some time. Even if you try your best to stay away from it, you'll need abstractions for:
- components and lifecycle
- some sort of templating
- connecting your app's data and components
But you can have minimal layers for that.