University of Liechtenstein

Are Crowdsourcing Platforms Reliable for Video Game-related Research?: A Case Study on Amazon Mechanical Turk

Crowdsourcing platforms like Amazon Mechanical Turk are increasingly unreliable for video game-related research, as standard validation mechanisms frequently fail to filter out non-genuine participants.

• General-purpose crowdsourcing platforms like Amazon Mechanical Turk are highly unreliable for video game research, with validity rates for participant responses ranging from as low as 5% for World of Warcraft to a maximum of 28% for PUBG.
• Standard validation mechanisms are failing to filter out non-genuine participants, as evidenced by a study where 229 out of 231 responses were found to be illegitimate despite the use of filtering constraints.
• The average validity rate for research participants remains stagnant at approximately 13%, even when researchers mandate the submission of verifiable, publicly accessible gamertags.