University of Würzburg

News Ninja: Gamified Annotation of Linguistic Bias in Online News

News Ninja functions as a Game with a Purpose (GWAP) designed to address the dual challenge of identifying linguistic bias in online news media while crowdsourcing high-quality annotation data for natural language processing.

• News Ninja is a gamified web-based application designed to crowdsource the annotation of linguistic bias in online news, addressing the high costs associated with expert-only data labeling.
• In a validation study using 370 sentences from the BABE dataset, the game achieved a 78.11% agreement rate with expert labels, demonstrating the viability of using non-experts for complex linguistic tasks.
• The game utilizes five distinct modes—Context, Publish, Quick Words, Co-Op, and Critique—to maintain engagement and provide a structured progression for players to learn bias detection without explicit scientific terminology.