Deep Unsupervised Multi-View Detection of Video Game Stream Highlights

This research introduces an unsupervised deep learning methodology for automatically detecting highlights in video game streams. The primary thesis posits that game highlights are not solely defined by in-game events, such as score changes or combat, but are significantly influenced by the streamer’s social signals, including facial expressions and verbal reactions. By integrating these social cues with game footage, the system provides a more comprehensive approach to identifying salient moments that resonate with viewers.

The methodology employs a multi-view architecture that utilizes convolutional autoencoders to analyze game footage and player facial expressions, alongside spectral analysis and principal component analysis for audio processing. These modalities are fused using recurrent layers, specifically Long Short-Term Memory (LSTM) networks, which perform temporal smoothing and novelty detection. The system identifies highlights by measuring reconstruction error; frames with the highest prediction error are classified as apex points for potential highlight clips.

The study evaluated this approach using over five hours of footage from the game PlayerUnknown's Battlegrounds (PUBG) sourced from Twitch.tv. The dataset comprised 11 video segments featuring two streamers. Results indicate that the fused model, which incorporates face, game, and audio data, outperforms single-view models. Notably, the system successfully categorized highlights into "funny," "action," and "social-interaction" events. The findings reveal that action-oriented highlights are more prevalent toward the end of game sessions, likely due to game mechanics that force player interaction, while social and humorous highlights are better captured by analyzing the streamer's facial and audio cues. This research demonstrates that multi-modal analysis is essential for accurately identifying the diverse range of events that constitute a compelling game stream.