LLMs May Not Be Human-Level Players, But They Can Be Testers: Measuring Game Difficulty with LLM Agents

Large Language Models function as scalable, autonomous agents capable of measuring game difficulty, offering a cost-effective alternative to traditional human-based playtesting. By leveraging Chain-of-Thought reasoning and expert-informed strategies, these models approximate human decision-making processes, allowing developers to evaluate relative difficulty levels during the production cycle. While these agents do not consistently match the peak performance of specialized algorithmic solvers or highly skilled human players, their gameplay outcomes—such as win rates and resource management—demonstrate a statistically significant correlation with human-perceived difficulty metrics.

The research focuses on logic-heavy and strategy-based titles, specifically utilizing Wordle and Slay the Spire to validate the framework. By representing game states clearly and calibrating performance through techniques like deck strengthening or adjusted constraints, models such as GPT-4 successfully capture nuanced difficulty levels that often elude standard heuristic-based AI. These findings indicate that LLMs are particularly effective at mirroring the human experience of problem-solving, providing designers with actionable data to balance game mechanics without the logistical burden of extensive human testing cohorts.

Although current applications are primarily limited to isolated challenges, the integration of LLMs into development workflows represents a significant shift toward automated quality assurance. Future iterations of this methodology hold the potential to model more complex phenomena, including cumulative gameplay effects, player learning curves, and broader metrics of game quality. By serving as reliable proxies for human testers, these models provide a generalized, efficient toolset that enhances the precision of difficulty tuning across diverse game segments.