Designing Compelling AI Character Interactions in VR Using Body Language

[Music] my name is vivian tan i'm the co-founder and ceo of beast inc and i just want to apologize in advance if i need to call for my voices cracking because i'm getting over a fluke i'll try try to keep that down so i want to start off by telling a little you a little more about our company and what we do we are a san francisco-based startup and we make ai driven virtual pets we are the creators of beast pets which is a vr sandbox experience and pet dragon simulator it's currently out in early access on steam it's this friendly little world where you can go into and play with five baby dragons and they each have their own unique personalities and our goal for these pets is to create your best virtual pet friend for vr and ar so

over the past few years my team and i have put a lot of work into creating these lovable believable little creatures and in this process we discovered that body language plays a huge role in creating very believable and compelling ai characters um and and um you don't really need a lot of hyper realism to create a believable character but you do need to look at a lot of nuanced movements and and presence so what do i mean by body language we're i'm here i'm not really just talking about animations or what we traditionally think of as body language i'm using it in a more holistic sense especially in a very spatial medium like virtual reality so body language in this particular context

could mean physical presence or spatial interactions eye contact or even the player's gestures not just the ai characters gestures so why is body language important this is something we i admit right off the bat we didn't realize that body language would play such a huge role in creating our characters when we first starting started putting together this wishlist for making awesome ai characters we looked at all this the sexy tech basically like we're like okay we're gonna have this very sophisticated audio ver uh communication layer with dynamic dialogue chat bots voice recognition and then we're gonna add an even more sophisticated animation layer was procedural physics based animations and mocap and on top of that hey why not computer vision and machine learning and all this for creating contextual awareness and then we just go right in like not

really knowing what we're doing except like all this text sounds really cool and then once we started building we realize that we are missing something much more fundamental body language body language as i mentioned before not just animations but how this character shares this physical space with the player how do these physical interactions play out and lots of other small things and in this process we also realized that you know this wasn't immediately obvious to us because if you think about our real lives on a day-to-day level we engage in body language all the time right i'm moving my hands to communicate i stand a certain way where i stand a certain distance from other people or animals but a lot of this is subconscious we don't really actively think about what we are doing with our postures with our hands not not for the most part but when you're building this creature

or this humanoid this character from scratch you have to build that in and then we realize that and we kind of use that as a check now it's like okay if we're creating a character that can talk or do something more advanced but something still feels off we kind of dial back a lot we look at the foundation layer it's like is there something about the character's body language that's sending the wrong signals on the subconscious level because usually those subtle little things is probably something that can make or break the believability of a character and as we spent more time on creating these foundational interactions we discovered two more things one these actions are very much native to vr you now have the space in which you share with vr character that wasn't previously available if you were designing games or interactions even in 3d if it's available if it's you're viewing it on a 2d screen that

was not nearly as obvious as when you're literally sharing the same physical space do this is great because and that means all of these interactions are available to pretty much every vr designer out there right you don't need to know necessarily anything about voice recognition or machine learning and all the extra layers but you can by building in vr you can still take advantage of body language so we double down on focusing on body language for our pet dragon characters and we also discovered pleasantly that you can convey a lot of emotion or or context relationships even without going into the other layers so that's why we decided to focus more on body language as a foundation and that can serve as a baseline and a framework for more advanced ai layers so through body language we can

provide a lot of context for the player you can establish the power dynamics between the player and the character you can explain the context of experience without even using words you can convey the relationship between them and you can convey the intent of the ai characters you can prompt players to perform desired actions without explicitly telling them and very importantly you can minimize reliance on dialogue subtitles and menus and on top of body language you can build a foundation for more complex ai behaviors like we were just talking about so i want to give you some tools and examples that you can implement in your own vr experiences this is not intended to be a comprehensive exhaustive list of everything you can do with body language

because we're still learning and you know discovering a lot of things about how we can use vr and build characters nor do you have to include everything i just mentioned but this is kind of food for thought and come bring up more prominent examples and that you can hopefully integrate so first off scale and power dynamics scale is one of the most powerful things you can use in vr to convey whether the player or the ai character has more power and more control so on the two and two extreme ends of this is you have a very large player and very small characters or vice versa so if you're in a space with tiny characters and a big player you are automatically conveying that the player has control and the player's roles could be a variety of things you can build you can guide the characters you

can protect them or you can manipulate the characters or their environment and the relationship here could be benevolent or middle of it it really depends on the context of the game but the size differential already conveys a lot of information an example of this is moss in the world of moss you the player are the guardian you are this benevolent being and your role here is to guide and protect a little mouse named quill so quill's very cute she has these really cute animations so all those are cues that you know you have a friendly relationship you're supposed to protect her but right off the bat even before you interact with her you already know the power dynamic based on the size differential the other extreme are giant characters and tiny players in this world you tend to have very little control

and your role is typically to observe to avoid or to learn an example of this is the big whale in the blue so the whale's not really an ai character it's more of an npc but i think it's just a good way to illustrate how the size differential sets the context and the power dynamic in this world so you're in the water underwater world but you're kind of a tourist you don't have a whole lot of control or influence over the environment and um nothing demonstrates that more than when a big whale swims past you and you realize that you're really just a very small part of this world another example is the large life-size dragon in skyrim so in this case you aren't just an observer you have to fight the dragon but the size differential tells you that you're not the boss here you have to fight the boss

so in between these two extremes we have normal size characters kind of uh they don't necessarily have to be the same size of the human but they're kind of a more realistic scale and this is where you have a lot of nuance so here on screen you can see the representations right the ai character red designed by the little robot icon can be slightly smaller or slightly bigger than the player and on a 2d screen it doesn't make too much of a difference but in vr that small size differential could convey volumes about your relationship with the ai character and what your roles are so an example that i really like that i came across a couple of weeks ago when i was at oc 6 and i sat in on a talk from the oculus creators that created first steps so first steps is a really short intro app on the oculus quest headset

where you encounter a robot and he she it walks you through a tutorial to teach you how to use the controllers and how to interact with the environment so this is a friendly relationship in which you're supposed to be learning from this robot and the oculus developers that created this they discovered that initially they had like a six foot tall robot but players come in all shapes and sizes so when they were testing this with players that are about my size and i'm fly feet tall right this robot is towering it is sticking its hands in your face like use these controllers and you're just like dude get away from me so that is not the relationship you want to create when you you want to have a friendly interaction so i just thought that was a really good example of like you know when you are creating friendly characters you typically want to err on the side

of having the friendly ai characters be slightly smaller or shorter than the player now if the relationship is supposed to be antagonistic then you have a lot more freedom right the enemy can be larger or smaller and if you want to like dial up how threatening the enemy is you can just make them increasing a lot larger and tell them the size of the dragon and skyrim um one tool that you can use is to dynamically scale the character height to match the player so for instance you measure where the average hmd height is on the player and then you can scale the character down or adjust their height placement to match that of the player it's not always available but it's one tool that you can consider and also these are not hard and fast rules and i also believe that rules are meant to be broken especially as we're trying to discover more and more things in vr but it creates a good baseline on which you can kind of

measure you can set expectations and measure and test the results another tool that's at your disposal in vr is vertical space where you put ai characters conveys a lot about how much threat they're posing to the player right if they're coming in high above you they tend to pose more threat if they're far below eye level and we're on the ground they present low threat oh sorry i think i have that backwards when they're coming in high high threat coming in low low threat and about eye level and chest level they present a neutral level of threat so you can use other cues to represent if they're supposed to be friendlies or unfriendlies if they're in the neutral zone some example of this so all of these characters and entities on the screen are roughly one cubic feet

but they're starting in their default positions say a lot about the threat level they present so up high you have the space pirate trainer robots right when you start playing in space pirate trainer you are on a platform and a robot is hovering high above you eventually the robots will lower to about your height and they say zoom in towards you but the starting position already conveys a lot about their their intent and their relationship to you on the low threat level you have the little robot dog in the valve it's small it's cute but it's also on the ground so it's not really presenting a lot of threat in the neutral zone is where we're putting our pet dragons typically um and i have the beat saber blocks in there i know they're not area characters but i think they're just kind of like the most recognizable example of entities you can react with and they're typically placed in the neutral zone and then the designers use other cues

such as the fact that you're holding sabers to convey to the player what you're supposed to be doing with them so in the neutral zone you have a lot of leeway and what's funny is that um we discovered through trial and error that the positioning conveys so much about threat levels that even if your character design is that of a friendly being if they're placed high above they're they become almost automatically threatening so initially we did not limit how high our dragons can fly we're like okay well they're the size of house cats they're meant to be cute and cuddly we have all of these contextual cues telling you that these are friendly creatures but then we also noticed that because we designed them to fly into the player's field of view if players start looking up the dragons fly higher the players look up even more and pretty soon you would have the swarm of dragons staring down at you and it doesn't matter if you're cute

when you have five dragons just hovering above you it becomes really intense and that's when we learned like okay we need to put a limit on how high they can fly so right now we're aiming to keep them at between like chest level and eye level so they can stay between the neutral to friendly zone proximity such a huge role in context and it's one of those things in retrospect it's like oh yeah obviously right proximity but it's like as we when we started breaking this down we noticed a lot of nuance and a lot of things that we can play with with only proximity so one you can indicate which characters are active or passive based on how close they are to the player so one of my favorite examples is gorham so in gorn you play a gladiator and you're inside this gladiator arena

fighting other gladiators the arena is divided into two sections there's the pit in which you're actively fighting these ai gladiators and then there's the spectator area that i like to call the bleachers that's elevated above the immediate play area so there are multiple contextual cues going on here so you have the players that you're immediately fighting and then you have these spectators that are up there and they're leering down at you they're not friendlies right hence they're in the high threat zone but they're also far away they're not in the immediate player area so you know by proximity that um okay those guys are not an immediate threat but they're also not friendly another observation here um is that the other gladiators you fight have limbs so they can attack here because you know they have the ability to run towards you they have the ability to wield weapons whereas the spectators are just talking

heads they have no bodies so at first glance you know that those guys aren't just going to start jumping into the pit and attacking you because they physically cannot so i just thought this is such a great example of multiple layers of body cues at play conveying the context the antenna who your enemy is who's a friendly or who's just a passive npc you can also use proximity to prompt player movement right a character that's immediately within the player's reach encourages the player to be stationary because you don't have to move a character that just slightly out of reach encourages the player to physically take steps and walk towards the character and a character that is further away encourages the player to explore teleportations or other virtual types of locomotion in order to get to that part of the play area

eye contact is also a very useful tool especially in vr instead of just having the character stare out to the screen kind of like breaking the fourth wall as you would in 2d game they can make direct eye contact with the player and that conveys a lot of information and you can use that to accomplish a number of things and when you use eye contact in conjunction with field of view you can do a lot to direct the player's attention so when we first started building this we're like oh okay well eye contact is so cool we never had this before on a 2d screen let's that let's make sure the dragons acknowledge you looking into the eye and we kind of over did it a little bit so they wouldn't break eye contact with the player and that became really really intense so like i said all these little cues

could undo your character design a set design because we had people cringing at the dragons we're like reach out pet them they're cute they're like they're staring so we're like okay all right okay we gotta break eye contact how do we do that naturally so now we kind of have the dragons just zone out a little bit the stairs in space they look at each other they look at toys so they're no longer just staring at the player um and another thing we did was that we wanted to give these little guys like a life of their own so even if you're not playing with them they can be off plane by themselves or doing their own thing the problem is the players don't know that there are dragons behind them and this is not just us vr developers complain all the time about you know you want to encourage the players to turn around and explore their play space but if you're new to vr you just go in there and you stare straight ahead so this is where you can use a combination of eye contact and field of view to direct the player so what we do is we

have the dragons occasionally check for the player and just kind of fly into their periphery they don't have to straight on stare but just kind of come into the side of field of view maybe flit out a little bit and just little actions letting the player know it's like hey look to your left they are interesting things over here so it's a useful tool for getting players to follow or look around so another thing i'd like to talk about is using the aforementioned tools to in combination to apply attention pressure to the player so we talked about proximity we talked about direct and indirect eye contact so these are two variables you can play with but you can also adjust the number of ai characters and by combining these three variables and adjusting them you can accomplish a lot of non-verbal cues to get the players to take different actions

so for example if we dial up all three variables right lots of ai's close proximity to the player making direct eye contact you're creating high attention pressure and therefore high urgency this could mean that you know there's you want the players to fight them or that there's something really important going on and you want the player to pay attention conversely low pressure would put the player more at ease and encourage exploration so in the examples i mentioned with us accidentally putting too much pressure on the player with the dragons we were kind of just making them cower and not really encourage them to go out and explore this friendly world we had built for them so we learned that by easing up on the attention pressure right kind of have the dragons just stay a safe distance away we're telling the players like you can

navigate this world at your own pace and we're inviting curiosity we're encouraging exploration you can also switch it up right if you increase attention pressure and decrease it for instance having a character come towards the player make eye contact and then run away and by alternating that you can create curiosity the player will be like hey this ai is trying to grab my attention maybe i should follow so yeah so by combining all of these variables and you know adjusting the parameters you can invite the player to take a lot of actions without ever expressly telling them to lastly i want to talk a bit about hand gestures so this is yet another tool that's available in vr that we didn't quite have before when we were just creating for uh like a 2d screen play space

so you can use hand gestures to reinforce the player's roles and also the character's roles one thing we learned is to treat hands as tools and i'll elaborate on that a little more in a second um and two hand gestures comply not just to the ai characters but to the players themselves so this is where you can have a lot more dynamic interaction um one of the things we learned is to only allow the hand gestures you want the players to use and this ties back to using hands as tools right if you don't want them to attack the ai then don't give them a fist because we learned the hard way too like if you give them a fist they will use it so a couple of examples if you give them an open palm that signifies

a friend friendliness right because they can wave they can high five close fist typically signifies hostility because they can punch two examples of this in alchemy labs vacation simulator you have these cartoony hands but you can wave at the robot characters and that gets their attention and they know that you need help so they come to you for help um i don't have it on here but in rec room you can high five other players they're not ai characters but it's also just an example of how you'd be using hand gestures in boxing games like creed uh you're you have fists you have boxing gloves so you know that you have an antagonistic relationship but as we learned in creating our dragons that even if you don't have gloves just the fact that they can form a fist people will use it so again kind of the incongruousness of the

body language as compared to the context you're trying to the tone you're trying to set like in our game we wanted a friendly relationship but when we put people in there they had fists they were punching the dragons we're like okay well time to take those fists away um because that was not the relationship we wanted to reinforce and i should also mention that when we defined hand gestures this was using the htc vive controllers or the oculus touch controllers so you could limit hand gestures now that oculus has introduced hand tracking you pretty much have a full range of motion we have not yet looked into that what that means because now people can form fists that they want to so that will be a really interesting area to explore as we move forward with this last but not least so how do you know that any of these tools tips and tricks actually work it really just comes back to doing a lot of play testing

um i feel like as developers we're kind of primed to think about things a certain way already so what we ended up doing was just testing across lots of different people people who are not in our genre people have never played in vr so we can get their immediate visceral reaction right we're talking about body language being operating on such a subconscious level it actually works really well to test with first-time people because that's the level they're reacting on and you can get a lot of useful feedback so that it wraps it up for today and uh if you'd like to get in touch with me that's my email and my twitter and we're opening up to questions [Music] you know with the hand tracking um i wonder if it'd be possible to game it so to speak not to make too bad a pun so that no matter what you did no matter what the sensor was picking up

you would it would never show a fist couldn't you do that somehow that's a program it that way that's a really really good observation um i'm sorry let me repeat the question in case other people didn't hear it the question is can you limit the hand motion range even with hand tracking so like even if you're forming a closed fist in real life right all you see with this this yeah that that's really interesting because um one of the reasons we didn't look into hand tracking yet is because we wanted the haptic feedback but then i've been told by a lot of developers that you get this sort of ghost sensation even though you don't have haptics and so i wonder how much incongruity that would create like if if your real hands don't match your virtual hands it would be like forcing people to be friendly and it would be interesting to observe what effect it has on them if your anger can never find the muscular expression how will that affect you ultimately it could be an interesting psychological tool as part of the game that is that is really interesting we

would when we get into hand tracking would definitely test that thank you any other questions anybody okay so you indicated that you tried to get the dragons to play around with toys did you give any audio cues the dragons were playing around interacting that the player could listen to and try to direct their attention to the audio that's a really good question i'm just gonna repeat it um so you're asking if we looked into using audio cues to indicate what the dragons are doing what direction they're coming from okay yeah so that's something we tried a little bit and i didn't i thought about including the presentation i didn't so thank you for brewing that up um we have it didn't work

as well as we thought it would one your 3d spatial audio right in vr right now is not entirely reliable um it's it's still very subtle so visual cues are at this right point are more powerful i heard i have heard of like really good audio designers creating entire experiences using only audio cues as guides but i think to accomplish that you need one really good audio designer because it's much more subtle than visual design and two um it's it's still the player perception is still not as consistent like we kind of we tried out audio cues on me because i have trouble picking up where sound is coming from like even in real life like if an explosion goes off that way i i can't really tell like i'm probably an extreme example but i'm just not as sensitive audio cues as i am visual cues

so we do to answer your question that was long-winded we do use audio cues but we don't find them nearly as reliable as visual cues yeah thank you thank you hi i'm eric um so i recently have been doing a lot of play testing for the tutorial of my game and as a just as a design principal i challenged myself to have zero text and no you know like panels or anything like that and you know throughout the different iterations no matter how streamlined and how many guides there were and only like one event at a time uh i too like to play tests with people who are completely new to vr and even if it's just the most obvious single visual slash audio cue right in front of them they might miss it because they're so overwhelmed with the environment i was just wondering if you could share any sort of best practices of onboarding or tutorial uh techniques without you know overt text to guide them or or voice over

yeah that's a really good question so to repeat it you want to know if there are some best practices for onboarding new players without relying on text or voice over or voice over okay um yeah so we have some insight into this in our initial product we actually built it for vr arcades so we were expecting a lot of new players and we try to minimize the amount of text full disclosure here unfortunately we had to add a menu and things later on so what you described is something we're still struggling is with trying to minimize amount of text and voice-overs entirely but we still haven't but um some of the things that we discover that really work well is to kind of control the environment your players are in so we when we first started we would drop players in this very lush green environment with birds chirping lots of things going on dragons flying

around and we're trying to get them to pay attention to the tutorial and they're not they're like there's a dragon i want to go over here they're doing all sorts of things so what we do is we drop them not now off in a starting zone no teleportation virtually nothing to interact with other than the thing we want to pay them to pay attention to so one way you can think of it is kind of like stage lighting right if you black out the rest of the stage and just put the spotlight on the thing you want the audience to watch you can recreate something similar in vr using that like it doesn't mean you need to have a blank background but um you know the proximity we're talking about right in this stage since you think of scenery right out of reach fading into the darkness or something like that and just focus the players on the thing you want them to interact with oh oh another thing and i didn't mention this in the talk is when we were talking about coming into the field of view um even in that situation your player will pop they'll probably still look around

it's funny like they don't look when you want them to look and when you don't want them to look they're looking all over the place so have whatever you want them to interact with just inch its way into their field of view so we not only do that with the dragons we also do that with a menu sometimes just to have it come within their field of view okay thank you you're welcome okay cool thank you very much