Emotional-Driven Gameplay: Exploring Gesture-Based Interactions in XR Environments

This study explores how gesture-based interactions in virtual reality can evoke emotional responses and foster empathy, with the aim of making non-verbal communication in digital environments more natural and intuitive. The authors built a VR prototype on the Meta Quest 3 headset, using its built-in hand tracking alongside a Rokoko motion capture suit to drive a non-human, faceless character in Unity. Gestures were mapped to emotional states, with a high five signaling happiness, a fist conveying anger, a thumbs down expressing disapproval, and a pointing finger suggesting confusion, and environmental feedback such as lighting and audio was adjusted in response to recognized gestures.

The methodology combines embedded, explanatory sequential, and exploratory sequential mixed-method designs, integrating technical validation of gesture recognition with qualitative observation of user behavior. The recognizer was limited to static poses, so dynamic gestures such as swiping were excluded. Two environments were tested, a living room and an open park, but the open park was abandoned because its rendering demands were unmanageable on the headset, leaving the enclosed room as the main setting for development.

Reported results are observational and qualitative. Participants' reactions were documented across gestures, with the fist eliciting empathetic responses despite being read as hostile, the thumbs up and wave encouraging positive engagement, and the thumbs down producing mixed reactions. The authors conclude that enclosed, distraction-free spaces produced stronger emotional connection than open ones, and they propose applications in nonverbal communication training, interactive storytelling, education, and therapy. The paper does not report participant counts, statistical measures, or validated emotion instruments, so its conclusions should be regarded as preliminary.