Strategies for Using Proximal Policy Optimization in Mobile Puzzle Games

Automating play-testing in mobile puzzle games requires overcoming significant technical hurdles associated with reinforcement learning, particularly the instability and overfitting inherent in Proximal Policy Optimization (PPO). By applying targeted training strategies to the match-three mechanics of titles like Lily’s Garden, developers can transition PPO from a theoretical model to a reliable production tool. The primary objective is to create agents capable of navigating complex, procedurally generated levels while maintaining performance stability and avoiding catastrophic learning failures where agents repeatedly select invalid moves.

The most effective approach involves a combination of environment resets, color shuffling, and soft action masking. While traditional hard action masking often leads to algorithmic collapse, integrating masks directly into the observation space as a partial forward model significantly enhances training speed and reliability. Among the tested configurations, the CSResetMaskV2 framework emerged as the most robust, yielding the highest completion rates and demonstrating gameplay patterns that closely mirror human behavior. These techniques effectively mitigate the risk of agents becoming trapped in suboptimal loops, allowing for superior generalization across previously unseen game levels.

Future advancements in this field will likely focus on refining the human-like quality of these agents through the integration of imitation learning and automated curriculum learning. By moving beyond basic completion metrics, these methodologies aim to provide developers with more nuanced insights into player experience and level difficulty. These findings establish a scalable foundation for utilizing reinforcement learning to streamline quality assurance processes, ultimately reducing the manual labor required to balance and test high-volume mobile puzzle content.