Towards Situation Awareness and Attention Guidance in a Multiplayer Environment using Augmented Reality and Carcassonne

This research introduces a framework for developing a Situation Awareness and Attention Guidance (SAAG) system within a multiplayer augmented reality (AR) environment. By utilizing a digital, AR-based version of the tabletop game Carcassonne, the project aims to create a computational pipeline capable of identifying a player’s intended actions and providing subtle guidance to improve strategic decision-making. The study focuses on three core components of situation awareness: perception, situation recognition, and intention recognition.

The methodology involves the development of a custom AR application, cARcassonne, designed for the HoloLens 2. To address the lack of existing gameplay datasets, the researchers created a gameplay AI using the Unity ML-Agents framework. This AI serves two purposes: generating large-scale training data for the situation model and acting as an opponent for human players. The system employs two distinct game state encodings—a bit-based matrix for convolutional neural network processing and a graph-based representation for logical game engine operations. The situation model itself is built upon a graph neural network (GNN) that predicts the likelihood of specific tile placements based on the current board state.

Initial findings indicate that while adversarial self-play was initially ineffective, training the AI in a single-player environment successfully taught it to prioritize feature completion and strategic meeple management. The research also incorporates gaze tracking via the HoloLens 2, with plans to integrate heatmaps and gaze plots into the SAAG pipeline to better predict user intent. Future work will focus on expanding the dataset through human-participant studies and utilizing imitation learning to refine the AI’s strategic capabilities, ultimately aiming to provide a robust, real-time system for enhancing human performance in complex, multi-user environments.