Or if training the model on many FPS games? Surviving in one nightmare that morphs into another, into another, into another ...

What you're pointing at mirrors the same kind of limitation in using LLMs for role-play/interactive fictions.

Maybe a hybrid approach would work. Certain things like inventory being stored as variables, lists etc.

Wouldn't be as pure though.

Give it state by having a rendered-but-offscreen pixel area that's fed back in as byte data for the next frame.

Huh.

Fun variant: give it hidden state by doing the offscreen scratch pixel buffer thing, but not grading its content in training. Train the model as before, grading on the "onscreen" output, and let it keep the side channel to do what it wants with. It'd be interesting to see what way it would use it, what data it would store, and how it would be encoded.