Player Rating Systems for Balancing Human Computation Games: Testing the Effect of Bipartiteness

This study examines whether standard multiplayer rating systems can reliably estimate player skill and task difficulty in human computation games, where players and tasks form bipartite graphs with no direct player-player or task-task matches. The central question is whether this bipartite structure, along with imbalance in the match graph, degrades the predictive accuracy of Elo, Glicko-2, and TrueSkill. The authors motivate the work by arguing that such structural properties could impair the flow of rating information and thereby undermine efforts to select and sequence tasks for difficulty balancing.

The evidence draws on two datasets. On FIDE chess data, Elo trained on bipartite or non-bipartite subgraphs generally outperformed a naive always-predict-white baseline by up to 14 percent in median error, with bipartiteness showing no detrimental effect. Unbalanced bipartite training graphs in fact performed better on bipartite test sets, plausibly because a small number of high-degree players contribute many matches. On the HCG Paradox game dataset, all three rating systems beat the white-win baseline by up to 25 percent in median error, and Glicko-2 and TrueSkill both outperformed Elo.

The authors conclude that Elo, Glicko-2, and TrueSkill can plausibly estimate user skill and task difficulty in human computation games, and that bipartite user-task graphs do not harm prediction accuracy. On the Paradox data, TrueSkill and Glicko-2 beat Elo, with Wilcoxon p-values below .001 for most comparisons. The findings are subject to stated limitations: reliance on prior solution data, binary win/loss outcomes, unclear data requirements for rapid learning, and the need for validation of engagement effects across a wider range of human computation game types.