Game State Learning via Game Scene Augmentation

GameCLR is a novel self-supervised learning technique designed to improve the extraction of game state information from raw pixel input. By leveraging direct access to a game engine, the method synthesizes highly controlled image augmentations to guide contrastive learning. Unlike traditional approaches that rely on simple pixel-level modifications like rotation or brightness adjustments, this technique uses the game engine to generate semantically similar "positive" pairs and semantically dissimilar "negative" pairs, allowing the model to learn which visual features are critical to the game state and which are merely aesthetic.

The methodology was tested within the CARLA urban driving simulator, utilizing a ResNet18 encoder trained on 50,000 anchor images. The study compared the performance of GameCLR against the standard SimCLR baseline. Researchers evaluated the effectiveness of the learned representations through linear probing, measuring the correlation between the model’s latent embeddings and six specific traffic-related variables, such as the distance and direction of surrounding vehicles.

Findings indicate that GameCLR significantly outperforms the baseline, achieving an 11% average improvement in capturing critical traffic-related features compared to SimCLR. While both methods improved upon a randomly initialized model, the inclusion of engine-specific hard negatives in the GameCLR framework provided a more informative learning signal. The results demonstrate that by controlling the synthesis of training data, researchers can effectively train vision models to identify complex game states directly from screen footage. This approach offers a robust alternative for applications in artificial intelligence and games research, including deep reinforcement learning, procedural content generation, and player modeling, where accurate state representation is essential.