Comparing the Performance of Off-Policy Deep Reinforcement Learning and Neuroevolution in Stealth Game Problems

This study compares off-policy deep reinforcement learning (DRL) and neuroevolution (NE) methods in stealth game environments, a genre the authors note is underrepresented in deep learning research. Using three stealth scenarios built in Unity, the researchers implemented 18 algorithms with hyperparameter tuning and evaluated them on learning stability, adaptability, and task completion. The work addresses a gap in prior literature, which includes only two earlier deep learning studies focused on stealth games.

Among the DRL methods, Rainbow DQN, which combines several prior DQN extensions, proved the most consistent, while Genetic Algorithms (GA) emerged as the strongest neuroevolution approach. Across the three levels, all algorithms learned to complete the first two, but performance diverged on the more demanding third level. There, only N-step DQN, Categorical DQN, and Rainbow DQN achieved positive average returns among the DRL methods, whereas GA and GA-NS (a Novelty Search variant) performed best overall, and CMA-ES failed to reliably reach the goal. The authors attribute CMA-ES's shortcomings to an update rule that inadvertently rewards experiences of being caught, suggesting the reward structure may need revision.

Statistical testing using the Wilcoxon signed-rank test indicated significant differences across most comparisons. The analysis suggests DRL agents learned the underlying dynamics of each level, while NE agents relied more on broad search. NE was substantially less sample-efficient, requiring roughly 93 times more interactions than Rainbow DQN when population size is considered, though the parallelizable, low-cost nature of these environments limited the practical consequences. The authors caution that these findings may not generalize beyond their particular reward designs and level structures.