Skill Check: Some Considerations on the Evaluation of Gamemastering Models for Role-playing Games

This analysis explores the challenges of modeling Game Masters (GMs) for tabletop role-playing games (TTRPGs) using natural language processing and interactive storytelling frameworks. The primary objective is to establish a standardized, objective evaluation methodology for AI-driven GM systems, moving beyond subjective human judgment. The research identifies four core technical challenges for automated GMs: world and story design, semantic extraction of player actions, commonsense reasoning, and persistent game state tracking.

To address these challenges, the study proposes three specific unit test categories: GM-P-GM (evaluating feasibility and commonsense reasoning), item tracking (assessing state management), and map design (testing spatial coherence). These tests were applied to three large language models—ChatGPT, Bard, and OpenAssistant—using a sample of 90 total unit tests conducted in both English and Spanish. The methodology focuses on objective pass/fail criteria to minimize the evaluation noise and bias inherent in creative system assessment.

The findings indicate that while ChatGPT and Bard provide a compelling, naturalistic gaming experience, they struggle significantly with commonsense reasoning and maintaining long-term narrative consistency. OpenAssistant demonstrated limited capability, often failing to maintain the GM role or producing nonsensical outputs. Quantitative results revealed that while models performed reasonably well in map design and item tracking, they consistently failed in commonsense reasoning tasks. The study concludes that current LLMs lack the necessary control mechanisms for reliable gamemastering, suggesting that future research should prioritize neuro-symbolic approaches to improve controllability, explainability, and the ability to manage complex, persistent game states.