Shells are slow (particularly bash), sure, but I'm not sure how the rest of your comment follows from that. The shell is not involved in interpreting terminal escape sequences in any way, and modern terminals are quite fast - I can render animations in a 350-column terminal and they are as smooth as can be, given the constraints.
Besides, the whole premise of this post is that bash is the wrong language for raycasting, kind of like writing bubblesort in CSS.
>put the other 10% into some compat mode
There is nothing preventing that, just test if the string contains sane characters only and use a fast path for that. The problem is that there is no actual fast path for software text rendering, you still need to process ligatures, etc.