I replaced Animal Crossing's dialogue with a live LLM by hacking GameCube memory

This is awesome. LLM-powered NPCs is one thing I'm most excited about in the future of gaming. Characters repeating the same scripted dialog over and over again is one of the biggest immersion breakers.

There's a number of little games and a major mod for Skyrim that implements this. I've spent a bit of time playing with the Skyrim mod.

There are some significant issues with it at the moment. One is that you have to train on vast swathes of text to get an LLM, and it's difficult after the fact to remove things after the fact. If you cooperate with the AI and stay "in Skyrim" with what you say to them it works out OK, but if you don't cooperate it becomes clear that Skyrim NPCs know something about Taylor Swift and Fox News, just to name two examples. LLMs in their current form basically can't solve this.

The LLMs are also prone to writing checks the game can't cash. It's neat that the NPCs started talking about a perfectly plausible dungeon adventure they went on in a location that doesn't exist, but "felt" perfectly Skyrim-esque, but there's clearly some non-optimal aspects about that too. And again, this is basically not solvable with LLMs as they are currently constituted.

Really slick experiences with this I think will require a generational change in AI technology. The Mantella mod is fun and all but it would be hard to sell that at scale right now as a gaming experience.

This is something fine tuning should be able to improve. The main caveats are the same as always: dataset collection, labeling, and training/experimentation time.