Analysing Historical Distortion in Large Language Model Roleplay

This paper examines how large language models perform as game masters in a text-based historical roleplay set in 1936 Vienna, centered on the murder of philosopher Moritz Schlick. The authors designed an interactive installation in which visitors, guided by a constrained four-option interface, could select among five models, including GPT-4o, GPT-4o-mini, Mistral, DeepSeek-Chat, and the locally run Llama3.1. The work is framed around how LLMs curate collective memory and whether such systems could be suitable for history education, with the stated aim of probing both narrative engagement and historical accuracy.

The study analyzed 206 exhibition game logs, of which 79 contained five or more interactions sufficient to form a story. Sentiment analysis using VADER showed a highly significant effect of model choice (ANOVA p<0.001), with GPT-4o and GPT-4o-mini producing slightly positive average scores while the other models trended negative. For factuality, proper names were extracted with a local LLM and then manually checked. Of 37 resulting names, only eight of the 24 fully named historical figures could plausibly have been in Vienna in July 1936, and several named individuals had no verifiable historical basis, indicating substantial inaccuracy in model-generated history. Llama3.1 introduced the most names per session on average, while GPT-4o introduced the fewest.

To expand the data, the authors ran automated sessions in which a second LLM instance played the human role, generating 70 complete ten-turn logs across seven models, five of them run locally. The extraction of 89 proper names from these logs produced results similar to the human-interaction data. The authors conclude that LLMs are promising as roleplay game masters, both for engaging play and for evaluating knowledge representation, but that all tested models have serious historical accuracy weaknesses. They argue these weaknesses could be turned into a basis for critical media and history education, and they propose design principles for educational history games built on constrained interaction.