Improving the Efficiency of Deep Reinforcement Learning in NetHack Using VAE and Proposing Additional Rewards

Key insights

7 takeaways

01
The combination of VAE-based latent representations and additional reward shaping significantly improves learning efficiency and game progression in the NetHack Learning Environment (NLE).
02
The 'Proposed (VAE2 + Additional Reward)' agent achieved the highest performance across all metrics, including total rewards, dungeon depth, and agent level, over 100 million training steps.
03
Incorporating a VAE into the DRL architecture reduces training time and GPU resource consumption compared to standard PPO agents, even when the VAE's direct impact on raw reward performance varies.
04
Additional rewards were calibrated using a tanh(r/16) clipping mechanism to provide small, frequent shaping rewards (0.005–0.02) for exploration and larger milestone rewards (15–20) for dungeon or agent level progression.
05
A penalty of -0.01 is applied if an agent remains in the same position for 75% of a 16-step window, effectively discouraging stagnation while allowing for necessary multi-step actions like 'kicking' or 'searching'.
06
PPO outperformed DQN in preliminary experiments, serving as the baseline algorithm for this study due to superior results in reward, in-game score, and dungeon level progression.
07
The study successfully mitigates the sparse reward problem inherent in the NLE Score task by providing auxiliary incentives for survival behaviors, such as eating when hungry, and active exploration of the dungeon.

Summary

Deep reinforcement learning in the NetHack Learning Environment faces significant hurdles due to extreme computational demands and the prevalence of sparse reward signals. To address these inefficiencies, researchers have developed a framework that utilizes Variational Autoencoders to generate compressed latent state representations alongside a system of supplemental rewards. This approach aims to accelerate the training process and improve agent performance within constrained resource budgets, providing a more viable pathway for navigating the complex, procedurally generated challenges inherent to the game.

Empirical results confirm that the integration of these latent representations and custom reward structures yields superior outcomes compared to standard Proximal Policy Optimization agents. The combined implementation of VAE-based state encoding and additional reward signals consistently outperforms baseline models across all measured performance metrics. By effectively distilling the game state and providing more frequent feedback, the model significantly reduces both total training time and overall GPU resource consumption, demonstrating a clear improvement in learning efficiency.

The scope of this research focuses on optimizing algorithmic performance within the NetHack Learning Environment, specifically targeting the limitations of current deep reinforcement learning architectures. These findings suggest that architectural modifications—rather than solely increasing raw compute power—are essential for mastering complex, long-horizon tasks. Future developments in this area will likely prioritize the automation of hyperparameter tuning and the integration of symbolic reasoning to further refine agent capabilities and decision-making precision in highly stochastic environments.