The Procedural Content Generation Benchmark: An Open-source Testbed for Generative Challenges in Games

The Procedural Content Generation (PCG) Benchmark establishes a standardized, open-source framework to address the historical lack of unified evaluation protocols in game artificial intelligence. By providing a consistent testbed across 12 distinct game-related tasks—ranging from maze generation to platformers and puzzle games—the initiative enables researchers to quantify the quality, diversity, and controllability of generative algorithms. This framework serves as a critical tool for comparing diverse methodologies, including search-based, evolutionary, and machine learning approaches, in a reproducible manner.

Experimental evaluations reveal that performance varies significantly depending on the specific game environment and fitness targets. While traditional search-based methods like Genetic Algorithms (GA) demonstrate superior reliability in finding feasible solutions, Evolution Strategies (ES) often achieve a more effective balance between feasibility, uniqueness, and controllability. Furthermore, the inclusion of modern generative technologies, such as few-shot Large Language Models, highlights a clear performance gap; while these models show potential, they currently struggle to maintain feasibility in complex, constrained tasks compared to established evolutionary and constructive algorithms.

Ultimately, the benchmark functions as both a rigorous research instrument and an educational resource for advancing procedural generation. It underscores that no single algorithm dominates across all problem types, as constructive methods remain highly effective for specific constrained environments like Sokoban. By facilitating the direct comparison of traditional and emerging generative technologies, the benchmark provides a necessary foundation for future developments in game AI, emphasizing the ongoing challenge of balancing automated content generation with strict design requirements.