Give MEANinGS to Robots with Kitchen Clash: A VR Human Computation Serious Game

The MEANinGS framework addresses the challenge of providing autonomous robots with human-level world knowledge, which is often underspecified in natural language instructions and difficult to aggregate manually. By integrating natural language processing, scene generation, and physical simulation, the framework utilizes a virtual reality (VR) serious game, Kitchen Clash, to extract actionable knowledge from human players. This approach aims to bridge the reasoning gap in robotic AI by capturing both symbolic task sequences and subsymbolic movement trajectories in everyday household environments.

The effectiveness of this approach was evaluated through an exploratory pilot study involving 26 participants, comparing the VR-based human computation method against a manual, text-based knowledge accumulation control group. The study focused on three complex household tasks: setting a table, preparing a cucumber salad, and cooking a steak. Findings indicate that the VR group significantly outperformed the control group in efficiency, with faster task completion times across all categories. Furthermore, participants in the VR group reported higher levels of perceived competence and interest-enjoyment, demonstrating greater motivation compared to those providing manual descriptions.

The research concludes that serious games serve as a viable, scalable, and engaging mechanism for accumulating world knowledge. While the current implementation faces challenges regarding the collection of unnecessary data and inconsistent player decision-making, the framework successfully demonstrates that human computation can resolve underspecified information more effectively than traditional manual methods. Future iterations will focus on refining data collection through deep player behavior models and implementing a knockout system for object alternatives to improve the quality and precision of the aggregated robotic knowledge.