I'm not saying having LLMs narrate the entire situation. I'm saying have the LLM sit between gamestate and the player. The LLM is the UI.
Essentially the LLM can see the current game state and possible moves and it's the LLMs job to change the game state and report the current game state to the user (via a well written narrative).
That keeps the world consistent and structured, but the LLM adds enough dynamism to keep it flowing well. You can even make the underlying game state complex as well. Like you can have enemy AI's that actually move through the world too (independent of the LLM).