Statistical Significance Testing at CHI PLAY: Challenges and Opportunities for More Transparency

Research within the CHI PLAY community faces a critical crisis of reproducibility and statistical integrity. A systematic review of 119 papers reveals that current reporting practices are frequently flawed, characterized by a pervasive misuse of Null Hypothesis Significance Testing (NHST) and a lack of transparency. Only 5.9% of examined studies provide a justification for sample sizes, while 28.6% contain internal inconsistencies in reported statistical values. These shortcomings are compounded by a failure to distinguish between exploratory and confirmatory research, which significantly elevates the risk of false-positive findings and the adoption of questionable research practices.

The prevailing reliance on NHST without adequate rigor—such as omitting assumption tests or failing to report essential statistical details—hinders the scientific advancement of game research. Because researchers rarely share raw data or analysis scripts, the field remains vulnerable to p-hacking and other biases that undermine the validity of published results. The absence of standardized reporting protocols creates an environment where exploratory findings are often presented with the same weight as confirmatory evidence, further obscuring the reliability of the conclusions drawn.

To address these systemic issues, the field must transition toward a more robust framework rooted in Open Science principles. Adopting practices such as pre-registration, transparent data sharing, and the utilization of alternative statistical methods, including Bayesian or estimation-based approaches, is essential for improving research quality. By implementing a structured template for statistical reporting and clearly defining research objectives, the community can enhance the reproducibility of its findings. Strengthening these methodological standards is necessary to ensure that game research maintains academic rigor and contributes meaningfully to the broader scientific discourse.