The evaluation of procedural content generation (PCG) systems currently suffers from a lack of standardized methodology, which hinders the ability to identify genuine advancements in the field. By analyzing 137 academic publications, it becomes clear that while nearly all research includes some form of assessment, these evaluations often prioritize internal metrics over comparative analysis. This trend suggests that researchers frequently focus on meeting publication expectations rather than establishing a clear, cumulative progression of knowledge. The reliance on redundant, custom-built domains and unvalidated fitness heuristics further complicates cross-study comparisons, creating a fragmented landscape that limits the practical utility of academic findings for the broader game industry.
To address these systemic weaknesses, the field requires a shift toward more transparent and collaborative research practices. A new taxonomy based on data extraction, calculation methods, and domain context provides a framework for better categorizing and evaluating PCG systems. By prioritizing the reuse of established benchmarks and frameworks, researchers can move away from the current reliance on isolated, custom-built environments. Furthermore, the industry should reconsider the mandatory pressure for quantitative metrics in every study, allowing for qualitative demonstrations of novel systems when appropriate.
Ultimately, adopting these standardized evaluation frameworks will improve the validity of PCG research and foster a more cohesive academic environment. By moving toward diverse, shared research paradigms, the community can better align its output with the practical needs of game development. This transition is essential for transforming PCG from a collection of disparate, self-contained experiments into a rigorous discipline capable of delivering consistent, measurable improvements in level generation technology.
