1 free read left. Create a free account for unlimited reading + AI research.

WhitepaperAssociation for Computing Machinery15 Oct 2021

Not Very Effective: Validity Issues of the Effectance in Games Scale

Key insights

6 takeaways

01
The 'Effectance in Games Scale' is unsuitable for current use because it fails to demonstrate a valid unidimensional factor structure and lacks sufficient discriminant validity from existing measures of mastery and enjoyment.See it on page 6
02
In a study of 467 participants, the scale showed poor statistical fit (CFI = .831, RMSEA = .118) and failed to clearly distinguish the construct of effectance from competence or general enjoyment.See it on page 6
03
The scale's validity is compromised by the inclusion of reverse-coded items, which formed their own separate factor during analysis, and a distinct 'input lag' factor that measures system responsiveness rather than the psychological experience of effectance.See it on page 6
04
The current 11-item scale appears to measure antecedent system features like interface legibility and responsiveness rather than the intended psychological experience of causing effects in a game world.See it on page 6
05
Researchers advise against using the scale in its present form and recommend that future work focus on developing and validating an entirely new instrument to measure effectance.See it on page 6
06
The study utilized three distinct browser-based games—'Getting Ogre It', 'Fireboy and Watergirl', and 'Jewels FRVR'—to ensure variance in gameplay experiences during the validation process.See it on page 7

Summary

The Effectance in Games Scale, a tool frequently employed to quantify the enjoyment players derive from exerting influence within digital environments, fails to meet rigorous psychometric standards. An evaluation of the 11-item instrument reveals that it lacks a consistent unidimensional factor structure, rendering it unreliable for academic or industry research. The scale suffers from significant issues regarding discriminant validity, as it fails to effectively distinguish the specific construct of effectance from broader, established measures of player competence and general enjoyment.

Primary technical flaws contributing to these validity issues include the presence of reverse-coded items and an unintended secondary factor related to input lag. These elements introduce multidimensionality that obscures the intended measurement of player agency. Based on data collected from 467 participants, the current iteration of the scale is psychometrically unsuitable for assessing player motivation or experience. The findings suggest that the existing instrument conflates distinct psychological drivers, thereby undermining the accuracy of any data derived from its use.

To address these shortcomings, researchers should discontinue the use of the full 11-item scale. A refined, four-item version is proposed as a more robust alternative for future investigations into game-based motivation. Moving forward, the development of more precise instruments is essential to capture the nuanced relationship between player actions and psychological satisfaction. By moving away from flawed metrics, the field can better align its quantitative methods with established psychological theories regarding self-efficacy and mastery in interactive media.

Not Very Effective: Validity Issues of the Effectance in Games Scale

Not Very Effective: Validity Issues of the Effectance in Games Scale

Key insights

Summary

Next steps

More from Association for Computing Machinery

Mapperoni: A Tool for Teaching the Basics of Level Design

‘That’s What Girls Have to Think About All the Time’: Board Games for Civic Engagement and Perspective-Taking

Exploring Player Experience Factors for Designing Persuasive Recruitment Games

“It’s Not My Fault”: Exploring Attributional Patterns in eSports Player Toxicity

Towards Custom Quest Tutorials: Identifying and Addressing Players' Cognitive and Performative Struggles in Games

A Design Toolkit of Ludic Contentious Politics: The Case of Protest in Videogames

Serious Gaming Meets Audiology: Developing a Video Game for Auditory Testing

Graveyard Symphony: A Study Investigating the Effectiveness of a Prototype Music Game

Towards Democratisation of Games User Research: Exploring Playtesting Challenges of Indie Video Game Developers

Storytelling Games for General Anxiety: Clinician Perspectives on Walking Simulator Games as Intervention

Behavioral Design in Video Games

Jumping Out from the Pressure of Work and into the Game: Curating Immersive Digital Game Experiences for Post-Work Recovery

More on Game Studies

Hexed by Pressure: How Action-State Orientation Explains Propensity to Choke in Super Hexagon

Computer-based Deceptive Game Design in Commercial Virtual Reality Games: A Preliminary Investigation

Conversations with Pigeons: Capturing Players' Lived Experience of Perspective Challenging Games

I Want To See How Smart This AI Really Is: Player Mental Model Development of an Adversarial AI Player

Examining Immersion in a Game-Based Experiment to Study Extreme Behavior

Human-Agent Trust Relationships in a Real-Time Collaborative Game

Measuring Fun: A Case Study in Adapting to the Evolving Metrics of Player Experience

Power-Ups in Digital Games: The Rewarding Effect of Phantom Game Elements on Player Experience

Horror, Guilt and Shame: Uncomfortable Experiences in Digital Games

Exploring Video Games That Invoke Curiosity

The Players' Experience of Immersion in Persuasive Games: A Study of My Life as a Refugee and PeaceMaker

Player Experience and Deceptive Expectations of Difficulty Adaptation in Digital Games