Viewpoints AI: Procedurally Representing and Reasoning About Gestures

Viewpoints AI is a prototype co-creative system that pairs a human performer with an artificial co-performer in gesture-based, open-ended interaction. Its central thesis is that a theatrical movement framework, the Viewpoints method, can serve as a procedural basis for AI systems that interpret and respond to human movement without relying on pre-authored motion libraries. Rather than privileging either the human or the machine, the work places both on equal footing, positioning itself as a contribution toward natural, gesture-driven communication between players and agents in games and interactive media.

The architecture integrates several components in a processing pipeline. A Kinect sensor perceives human gestures and converts them into symbolic aesthetic predicates covering tempo, duration, shape, energy, facing, and spatial relationship. A Soar-based rule system then selects a response mode, such as longitudinal reflection of the user's gesture, and applies a library of domain-independent functional transforms to modify it. The response is rendered as a firefly-swarm human silhouette, with predicates mapped to visual parameters like color and speed, and projected onto a semi-opaque muslin screen so the human's rear-projected shadow and the AI performer co-create movement-based proto-narratives.

The contribution is presented as a proof-of-concept architecture and a working on-stage installation, with an emphasis on the perception, reasoning, and rendering models. The authors acknowledge that several predicates, including kinesthetic response and the birth, life, and death of a gesture, remain unimplemented, and that the system relies on random past-gesture selection and limited pattern-following. No quantitative evaluation is reported; the work is a qualitative system description supported by a single worked example, and its claims about future natural-interface games remain prospective.