Calypso: LLMs as Dungeon Masters' Assistants [pdf]

So story time.

I've been hosting a DnD campaign with a group of college friends for almost a decade at this point. Since we've all moved away, we use Tabletop Simulator to play weekly.

In one of my adventures, the players met a creature named Dorian with the same backstory as the classic tale The Picture of Dorian Gray. Later, the players discovered the secret to the creature's immortality was an indestructible painting of the creature.

Nearly a month before encountering this creature, my players had explored a random dungeon I’d made where one of the rooms had a huge, ornate mirror on the wall. It made everything in the reflection appear older, covered in a layer of cobwebs and dust.

Although this was not the primary solution to defeating Dorian, I laid subtle clues to see if anyone would remember that far back. Well, one of them did in fact remember the dungeon. They stole the painting and brought it back to the dungeon along with a secondary mirror. They then placed the mirrors across from each other along with the painting to create a sort of “hall of mirrors” effect on the painting itself, which caused the painting of Dorian to accelerate in age—creating a feedback loop that infinitely aged Dorian (think drinking from the wrong Holy Grail).

It was amazingly satisfying to all my players, and it's also the exact sort of thing that I don't think the current batch of SOTA LLMs (even with the temperature cranked to 1) would ever think of in a MILLION years.

> It was amazingly satisfying to all my players, and it's also the exact sort of thing that I don't think the current batch of SOTA LLMs (even with the temperature cranked to 1) would ever think of in a MILLION years.

That actually seems like it'd be a pretty easy one for an LLM to get to. I tried

    You are a highly intelligent and creative D&D campaign planner. Imagine a creature that is immortal with mechanics like The Picture of Dorian Gray - its soul is protected in a painting and that gives its body immortality. Please brainstorm some interesting ways to defeat this antagonist that might connect to other aspects of a campaign, suggesting new ideas.

and on the 2nd run on a small local LLM got "*Reflections and Mirrors*: The creature's soul is reflected in mirrors and reflective surfaces, making them vulnerable to attacks. The party must use mirrors to their advantage, luring the creature into a trap or using them to distract it while they attack." and "* *The Corruption of Beauty*: The immortal creature is a symbol of beauty and corruption. To defeat it, the party must find a way to strip it of its beauty and corruption, using it against itself. This could involve using a magical artifact that amplifies the creature's own corruption, or finding a way to expose its true nature and render it powerless.".

It seems quite reasonable that a large modern LLM could come up with exactly that idea in far less than a million years. Art, mirrors and defacing art are all pretty standard sounding themes for handling that sort of monster.

I disagree - a generic suggestion to use a "reflection to defeat Dorian" isn't at all the same thing as standing up two mirrors to create a hall of mirror effect on a magic mirror that "ages" the world around it. The key difference is in the level of specificity.

Furthermore, the true test of the LLM would have been to devise both deciding to use a "Dorian Gray" type creature along with an appropriate mechanism for its demise.

IMHO, you've guided it by posing that this monster exists within the world. I'd like to see an entire campaign devised by an LLM - no guidance necessary. I don't know what small model you are using (mistral, vicuna, etc) but try asking it to create a list of monster based encounter/puzzles.

This is why I think the current batch of LLMs aren't really capable of being more than an assistant at best for writing. Interestingly I actually think AI dungeon released back in 2019 was far more capable in this regard and IIRC used a significantly older model - GPT 2.0.

I added "Be really specific" to the prompt and on the second go got:

11. *The Art of Decay*: The Portrait's soul is tied to the painting, but what if the players could use the painting's power to accelerate the decay of The Portrait's physical form? This could involve using a powerful magical effect or finding a way to imbue the painting with a magical property that would accelerate the decay process.

in the brainstorm. I don't think it'd take a million years; it's got the concepts and it is trying to put them together with only 5 attempts so far; and that is only 2Gb of weights on a relatively cheap GPU.

As for coming up with an entire campaign that copys an idea from Wilde, well it depends and I'd agree that is a task that would want a human in the loop. It does seem difficult to do in one prompt - but I think a million years is wildly overestimating the novelty of the campaign you're describing. It is actually damn hard to come up with good, novel ideas and the odds are the LLMs have literally seen variants of the Dorian Grey idea 100 times.

It's just describing the fact that the players need to find a way to age the portrait. At the risk of being a bit glib, I mean, duh. And its solution is the rather pedestrian "powerful magical effect or finding a magical way" jazz hands pile of claptrap.

I will admit that a million years was a bit of hyperbole.