A Framework for Usability Evaluation of Mobile Mixed Reality Games

This research establishes a decision-making framework designed to assist usability experts in selecting appropriate evaluation methods for Mobile Mixed Reality Games (MMRGs). Given the diverse nature of these games—which integrate virtual elements into real-world environments via mobile devices—the study posits that a one-size-fits-all evaluation approach is insufficient. The framework, titled Augmented Reality Usability Evaluation Method ElectioN (ARGUEMENT), guides practitioners through a series of diagnostic questions regarding game length, player count, session duration, and the level of physical activity required by the gameplay.

To validate the framework, the study compared several evaluation techniques, including heuristic evaluation, diary studies, interaction logs, audio diaries, and retrospective think-aloud sessions. These methods were tested against three distinct game titles: Parallel Kingdom, Portal Hunt, and Tidy City. The effectiveness of each method was measured by the number of usability issues identified and their respective severity ratings, which were analyzed using Mann-Whitney and chi-square statistical tests.

The findings indicate that while most tested methods perform statistically comparably to heuristic evaluation, they offer varying levels of qualitative efficiency. Specifically, the combination of interaction logs and audio diaries is identified as a superior alternative to retrospective think-aloud sessions, as it yields a higher volume of identified issues with less intensive effort. The study acknowledges limitations, such as the reliance on a single expert for multiple evaluations and a small participant sample, noting that further validation is necessary to ensure broader reliability. Ultimately, the framework provides a structured, context-sensitive approach to improving the usability of location-based mobile games as they transition from research prototypes to consumer-facing products.