Switching zoom levels takes a long time to load the assets. The bunny has some temporary displacement relative to the background. Panning results in visible lag.
I don't think vanilla is a magic performance pill. Everything compiles down to vanilla, after all, and in this case this app's performance seems quite a bit slower to me than Leaflet, OpenLayers, OpenSeadragon, Konva, Pixi.js, etc. I haven't looked at your code, but maybe it could benefit from tile-based rendering and asset virtualization?
I also think if this is the extent of your desired usage (rendering graphics on a zooming panning canvas) you're reinventing a wheel that's already been invented, abandoned, reinvented, improved, forked, improved, rewritten a thousand times. Why do you want to? Just for fun? To make an even better one? To make a custom game or mapping engine? Those are all fine, it's just a lot of work with uncertain payoff vs using a ready made library (not necessarily framework).
Just my opinion. I'm a lazy dev though.