Exactly. I've been working on something like this for a while using finite state machines to control the prompts. The biggest struggle I've had is creating memory. It's one thing to save a list of events and feed it into the prompts, but there are always issues interpreting it and making sure the memories are detailed enough but not an entire page.
For example, you "conquered the dungeon in level 1". If this gets saved as "conquered the dungeon" the next time you get to a new dungeon, it may think you already beat it and won't generate NPC monsters, that kind of thing.
I am also working on something similar. I have written a grammar system for controlling the llm that is not context free.
One of the main challenges is to have a database of what is reality, id -> object, and then have the llm return ids for objects referenced by the user. I am doing a system where something that has no id has not been created yet.
I use story templates with labeled segments to create characters, items, location as well as events. Example: "Once upon a time there was a <role>cowardly knight</role> named <name>Billy Bonkers</name>. The named qualities become the data structure associated with the id of the new character. It can be hilarious to prompt the llm to do nerdy space humor!
I seems to be very possible to create a "narrator driven" game happening in a "real" world.
I plan on using a multimodal llm and all the stuff that is created will have an image. That combined with id-less objects existing but not having a detailed description will make qualities and objects visible in the created images also part of the world.
For example, you "conquered the dungeon in level 1". If this gets saved as "conquered the dungeon" the next time you get to a new dungeon, it may think you already beat it and won't generate NPC monsters, that kind of thing.