Are Crowdsourcing Platforms Reliable for Video Game-related Research?: A Case Study on Amazon Mechanical Turk

Crowdsourcing platforms like Amazon Mechanical Turk are increasingly unreliable for video game-related research, as standard validation mechanisms frequently fail to filter out non-genuine participants. An analysis of 302 responses across 14 popular titles reveals that the vast majority of submissions are invalid, with validity rates ranging from as low as 5% for titles like World of Warcraft to a maximum of 28% for PUBG. In some instances, researchers encountered near-total failure, with 229 out of 231 responses proving illegitimate despite the implementation of various filtering constraints.

The prevalence of low-quality data suggests that many respondents are not authentic gamers but are instead utilizing automated tools or generative artificial intelligence to complete tasks. Even when researchers mandate the submission of verifiable, publicly accessible gamertags, the average validity rate remains stagnant at approximately 13%. These findings indicate that traditional crowdsourcing safeguards are insufficient to maintain data integrity in the gaming domain, where specialized knowledge and verifiable history are essential for accurate analysis.

Given these systemic issues, future research must move beyond standard crowdsourcing protocols to ensure the quality of participant data. The current landscape necessitates the implementation of rigorous, domain-specific verification mechanisms or the development of dedicated platforms tailored specifically to the gaming community. Without such advancements, reliance on general-purpose crowdsourcing platforms poses a significant risk to the validity of studies concerning player psychology, social behavior, and virtual environments. Researchers are advised to exercise extreme caution and prioritize stringent authentication methods to avoid the inclusion of noisy or fabricated data in their findings.