Beyond Gamification: Embodied Cognition for VR Games

[Music] i'm claudio i i work with interaction design in ai but today i'm going to talk to you about a vr game i worked on last year and we're going to talk about ux i for this game i cover the role of principal ux designer just for clarity is not going to be about ui ux but it's more like the type of experience that's around how the user how the player is going to use uh air hands around the environment and playing the game so it's a concept which is more key into what some people start like like calling special computing and things like this and um so like uh also going to talk about how to use some theories from cognitive science to actually make these interactions and those experiences more engaging and more compelling so the game is called in death

it's a vr game it's an icelandic production it's been produced in iceland by a studio called sulfur and we like with the help of rvx special effect studio we all based in iceland and it's been released across the major vr platforms like rift vive and playstation vr it's a danger crawler a classical concept it's based in the afterlife so you are in this environment there's like a lot of creatures that try to get rid of you and you you've been like equipped with a weapon and a shield uh to make your like your way through it there's no specific goal you just have to survive and you have to go through the game until like you reach the final boss and eventually like you win it and then you start the game again but this time it's more difficult so you have to survive it the concert is like pretty classical what makes it different what makes it unique is the fact that it's immersive it's using vr so you are immersed into this action

what that's there means it means for example that you don't have a button for strafing and dodging prototypes you literally have to lean left or right with your body to avoid incoming incoming threats or you can use like part of the environment to hide yourself get cover and like you know gain some tactical advantage it also means that you don't move the camera smoothly like we rely on teleportation like many other like vr titles uh for comfort reasons uh what we did here the overlap for this game was to try to fit the teleportation into the narrative of the game and uh like giving the player a special arrow that you can shoot with the bow that like it's gonna teleport you to a certain destination which is the point where the harder landed the game is designed to be uh very hard so there was a conscious choice of the design which is good because allows a certain amount of replayability of the game if the player really gets into the challenge but most importantly i think it's pushing the player to actually achieve

mastery of the weapons mastery of like marksmanship into using the bow and become really good at handling the weapon and becoming better and better since like the game is so hard the only way to make it through is to become like really accurate with the shooting but also understanding architecture and i understanding like um like what what could be a way to actually gain a tactical advantage by leveraging on like the shape of buildings and and how things are laid down in the environment uh also like by using teleportation like in a smart way so uh like this idea of of uh of mastery of the weapons requires an attentive design of the interaction of how the person how the player interacts with this like with this object so as i said at the beginning i worked on like designing the hand interactions for this vr game which in a game like this mostly means uh the way in which the player handles the weapons because this is a type of like end manipulation that the person's gonna go through uh very often uh

like on a second to second basis uh especially in combat situations the action becomes really frantic and has to actually be able to control those weapons and handle those weapons accurately and a sense of control so definitely like the functionalities of these weapons have to be there you have to have like the player like we wanted to have the player uh experience a sense of becoming better and better using those objects what was it so important to nail like a certain kind of font so an engagement like designing the interaction so they were like engaging and they were like an intense experience of of like combat like uh visual experiences uh designing weapons for vr is different than designing weapons for uh for games and it's important to understand why it's in order to understand where really lies the difference here so if you go on youtube and you search for a video of a game or a vr game you hardly see the differences between the two if you're looking at what's happening in the

virtual world they can almost look like the same but the big difference really like lies when you actually look at the people what people do when they actually play these experiences it's completely different in games the posture is like quite static and usually exceeded but what's important about it is that like you're holding like a device with buttons and like what you actually move is just your thumbs and that like the beginning of your interaction something meaningful happens when you move your thumbs on the button and like whatever is meaningful it's a feedback that gets like bounced back to you through the screen so the person keeps looking at the screen continuously so the head is fixed in vr instead um there's like a way larger set of movement that like a person do when hasn't when it's like inside these experiences uh people can be standing uh or like moving around but most importantly there is no screen uh the display is attached in your head and you can literally move your head

around that like plays a big difference in the interactive experience and also like you uh like move your hands for uh like performing certain interactions uh in environment around you so there's a big like big difference in terms of experience here um so like like like what's what's like really different uh what's really different about it too if we think about it is the motion space that we allowed uh to the person so like in games which are interactive experiences uh we have a very small motion space that's allowed to play it's like you move the times over the buttons and things are happening compared to other type of like interactive experiences for example like you know face to face social interactions another kind of social interactions we do on facebook but face to face uh it's still an interactive experience it's still like something interactive that we do with others which is meaningful but it requires the use of our entire body i think your body language for example you can rely on uh

like an entire spectrum of movements that your body allows to actually um to actually like uh like make meanings with others what does realize uh in this picture is like something in the middle so the the like the movement space is like larger compared to the one of games but you know is this is this actually like lending itself to the same experiences like having having having like uh like the player being able to move more uh doesn't mean that we can actually think of this game in vr in the same way in which we think games in the screen so like the practices and the guidelines are really the same your games tend to be more kinesthetic because of this and in like um like the fact that we have a larger box uh is not obviously like a better or an increment in experience it need to be leveraged in the proper way like back to the weapons like if we now

we have those type of um uh like larger space of body movements which allow the player can manipulate uh things with like with their hands um like uh how are we going to design those weapons like off like obviously the guidelines that we use for games might not actually quite fit here because we have like you know the player moving hands and manipulating and handling those weapons correctly so like like how can we do that we need to think differently we need to think about uh affordances uh we need to think about like how those objects are shaped and what kind of forms they have so that they can actually inform a person how to use them properly so affordances are important like good design is invisible but bad design is like pretty obvious when the affordance is not working for us you can you can see it right away so like affordances are important to actually need the functionalities of those objects for us other weapons that we focused on but like um how can we

connect this to the fun and that's where it came to quantity sites like central to the design process that i full on uh was the idea of embodied condition so it's typical to think that the mind affects the body with the body condition we allow ourselves to think that like body movements affects our thinking so for example we can look at uh what happens with dancers like dancers they like they learn choreographies to actually perform and like make make make a performance like learning a choreography takes a long time and it's like quite complex in terms of like you know motor coordination and everything but they actually develop a trick they have a technique called marking that they used to learn choreographies much faster so marking uh it's not doing the choreography it's kind of like sketching the choreography like performing movements that kind of mix the choreography without really

putting the full energy into it and and that's like the way in which they learn choreography is like much faster they they by moving their body they actually influence their learning in fact we we can actually see some research done by uh david kirsch that's been looking at the way in which dancers are performing those markings and like performing those live movements for facilitating their learning and that's exactly what i found out that by kind of mimicking the choreography uh they were able to actually learn it much faster than doing the choreography like full out full force or even just like thinking about the choreography so like what now that was helping them was like mimicking the movements at them like to actually build a temporal model or a mental image of of what what um what the choreography would have been like kind of like lay itself out in time and space and how to coordinate with others so the idea here that we had was like to leverage the same like the same concept

and think about like what is the choreography that the player is doing when he's actually moving her hands around those weapons uh what is the performance that like the player is actually doing and the idea here is that like the performance is a role model it's a hero character um and like um so it's not only so the players not only just like operating those weapons but it's also kind of like as it does that it like it moves in ways if you design like the weapons in a certain way you make like the player moves in ways that it's kind of like enacting a character and acting like a certain a certain like uh hero role model and which is like an idea akin to this like marking the dancers do and in like you know role playing it's fun so central to our design process really was like how can we make uh role playing out of end movements how can we design or think of like the way in which the

player moves his ends around those objects in such a way that that becomes like an actual role playing so overall the picture was something like this design affordances uh for those objects the mixed move like the makes like you know the like a person operate those those uh those um uh like weapons in the best way and like the interaction with the weapons elite like certain movements the movements is acting out uh it's like similar to acting out like the hero little characters with the same time reinforces the idea of like performing really well or performing like on top of certain um like out of the ordinary like like through those weapons um so like making the affordances um for the objects like it was it was like you know really important we need to make those affordances really well and like the first like the first approach the first temptation would be to actually copy the forensics from the real objects and translate them into the virtual world

that doesn't work out of the box like the reason why it doesn't work out of the box is because we all kind of understood this like you know the vision of vr this idea of a vivid dreaming um like some kind of like manufacturing reality where we impart the senses uh but like the reality like the visually we have now the essence we have now that allow also to do like a minor reality a minor version of this vision so the question really becomes like how do you project like real objects into this minor reality that we have at our disposal but because objects and affordances are really meant for the human body the deeper question becomes how do you see the human body projected into this minor reality because we need to understand that in order to make a forensics for this and that's the way in which we look that's the way in which the human body looks inside um like manual reality it's a vr monkeys with a head and two ears and not even

eyes they're actually kind of binocular holes it's quite different and we don't even have ends we have like some kind of a punching gloves in there and that's like makes a big difference in terms of florence's because not having fingers and having palms it uses a lot of our ability to explore our surroundings so the real question becomes when you think about like making anything that's like any interactions in vr how do you design an object for this creature so because if you think about like objects that way in real life those are designed for our body but now we have an embodiment which is different how do you design an object for this creature how do you find out but the only way for us or at least like in our approach um the way that we actually move forward uh was surprise prototypes you are just have to fast prototypes as fast as you can um like make as many prototypes as you can to actually get a sense of what responds what works for these vr molecules but also very important to use user testing as soon as possible like prototypes alone is like tells you only up to a point you need to see other persons like

jumping on board with those prototypes and see how far they go with that it's important to prototype like i i i experienced it it's important to prototype as soon as possible not too early like you know bad ideas it's like but we adjust them so our approach was like to kind of like user test internally and then like open up to other user testers but like fast prototyping it's been instrumental it's been important for us also to think of these weapons like props um like the same way in which like you know stage designers or like people that are working together things about props for actors objects that are not really meant to work in certain ways but like to kind of like suggest or release it a certain way to act around them and what also we find out was uh tremendously valuable the prototypes became a communication tool inside the team it was the best way to talk about vr interactions for us like sometimes it's kind of like complicated to talk

about like what will happen in vr and like you might write down you might have pictures of animations sometimes nothing really works you just have to have actual prototypes within the actual vr space where you see how things are unfolding so let's look at the bow and let's look how the real bow translate into a bow for the vr molecules how does the object change so real bow it's uh it's an elegant minimal design but it's also quite sophisticated as a number of parts which are important for us to actually handle it properly there is a grip there is um uh how to rest which is like where you place the arrow more importantly there is like a serving that that area that you see over there if you can see pointer but there's like an area there where where you can actually like grab like the string put it back um like that part which is like actually quite thin in reality in visual reality became thick and really really long as long as the string so it became kind of like barked up

instead of having like a grip we have to replace that with a touch point something that gets attached to your hand and something really important because the vr monkeys doesn't have really eyes they have like two these like binocular holes uh we have to cut short and like make the arrow appear as soon as you grab like the knocking like the knocking area because that was a thing like the person to have a straight line of sight because we wanted to actually have the person to actually achieve this idea of like you know becoming better and becoming a master or shooting like accurately so like that that like that really like helped to like to keep this like straight line of sight as much as was helping um the other idea that we had to do it was like constraining certain worst orientation like like there are ways of twisting your like your hands that like have been disabled when you actually control the bow so that like the hair will get placed into line of sight which is constant so you constantly get reminded of what's

like your line of sight you can actually get into the flow of using this bow quite fast and quite quickly and the final result was something that lends itself to be used in combat situations uh also like when there is kind of like a frantic moment and like uh the player can actually shoot accurately can get a sense of like becoming better a bit more accurate but it can also do like stuff that you wouldn't do with a real ball for example shooting horizontally that's something that like it's connected this idea of like enacting the use of an object so that you think like you are an hero and uh that was like pretty much what we did like for the bow was important also to kind of like keep a consistent uh interaction and i could like like design the object so that like the player could keep a consistent rhythm of of the interaction and that's like pretty much what we did like for the bro uh in short uh for the crossbow uh it's a little bit more like more complicated because the crossbow is

like more recent as an object there like many different variants so we went into archives of uh like museums and some other connections i have in italy uh to actually study like different variants different crossbows and understand like the various mechanics and like like what was the advantage of others compared to others uh like in general like the other crossbow was like you know be able to load a bolt like fast so fast firing was like like like you know the reasons why they were invented um like um in my research i also find something quite interesting i discovered that leonardo da vinci uh like prototyped a crossbow but just like on paper it was never realized but it was like an automatic uh like kind of like you know something you could load quickly so there's this artisan in italy who actually made it uh this like following the blueprint of the vg and it's a crossbow that is like cut in alpha you can open it and it loads automatically so the idea was really

like rapid firing what we did like for the crossbow was like to think about like how to fuse all these characteristics of different like crossbows from different like areas and like try to fuse into one object that we thought uh it could be like a representative even different from the goal um in the way to use it so we went through a number of prototypes like moderate prototypes that we build quite quickly out of gray boxing and like like moderate parts that you can interact with uh we started by adding like an angle on top and then looking at people and how they were responding and how much they were getting into the flow and then we moved the handles to the side just to see like what's the difference it was like better for aiming or not and then we even went as far as like putting some kind of like uh lever uh type of interaction here and like kept looking at how people were moving and what we're doing and like the final version uh is uh this one over here uh where you can uh grab the string and pull it backwards and like load immediately so if you come really good at doing this movement you can actually shoot and like reload it very fast and then you can also like

grab it with two hands and do some more like precision shooting so that's what concern looking at those objects in the virtual world but it's interesting to look as i said at the beginning to look at how people responded in terms of movement and this is actually the marking of like the people mimicking like a certain character through the use of that object so design the object so that people could move in certain ways when they were actually using it and it's important here to actually think about fatigue profiles like like since you design an object that gets used like kind of like uh you know make people move in certain ways you need to think about like what kind of muscles get involved in there and like try to actually make fatigue profiles which are orthogonal between these two different objects in in you can also see how how um like the movements are not necessarily the same movements that we would do with the real objects they're like sketched

but nevertheless they are enough to actually build a mental model or what's happening in in inside the virtual world and what's happening when they are enacting into those objects another thing that we did was very important was like to parameterize the forensics uh that was like key for us because we didn't have much time to do user testing or like a formal user testing so we had to go fast uh try to iterate as fast as possible so like like what i did was like to organize the forensics into a number of parameters that we could actually tweak and change right after every user testing so user testing was an occasion to observe people but also like to mold those values around like what we found out was like more or less comfortable more or less easy uh to actually get like working for the players when they were like trying this also once again like those parameters helped a lot as a communication tool because it helped us to have like a meeting where we like focused on certain ideas and certain concepts so they really worked as a form of like information

architecture like provide us a vocabulary to talk about like what we saw during this uh like user experiences and like during lewis like user testing um something that was also very important uh that sometimes get overlooked was active feedback the law of effort into try to actually uh get it right why it's important because optic feedback is a way to like highlight uh like rhythmic pattern of the interaction also like vr temps tends to be very loaded uh like on the vision it tends to be like visually very loaded so to actually provide feedbacks in a subtle side way through vibrations is actually like a good thing uh like reduces the quantity load uh on the person uh but for example let's let's look at the bow at the act of shooting the bow and and and how the the like the active feedback comes into play uh to actually define or reinforces the diabola tempo of the interaction

uh like reinforces um like reinforcing the idea of a certain rhythm and like getting into arithmetic pattern as using this object and like uh that's achieved by providing an active feedback in key moment of the interaction uh like when the player uh touch the string first you get like a first impulse then you get like longer one will grab the strings then you get like a progressive one when the string get pushed backward and then you get a shake so all of those up to feedback are there to actually mark a tempo in time that in turn and gets the person into a repetitive flow of movement and get into like the repetition of those movements like in a in a form of aesthetic of interaction something we didn't realize early on uh when we started working on uh on on on on the two weapons especially because the cross boss and idea came further down into the game was that like the two like these two weapons like in games when you add a new weapon it's it's like

you know adding more to what you really have um like uh but like here it was really about like having an extra object and also like demanding the person to actually relearn a new set of movements and because like the handling of the two objects is different like the bow is something that let's say you all with your left hand and you operate with your right end the crossbar is the other way around you all the way right and you operate with your left hand um so like when you tend to use one object you build up the muscle memory of that object and when you switch to another weapon the master memory doesn't work anymore you have to relearn it and and that's you know it's bit different like from like from games where like ultimately all the weapons really like you know it boils down to pressing buttons like at least like a sensory motor level but here is different and so um like every new weapons like require new conditioning and effort something has to be considered that we did not

consider first but then we kind of like crashed onto it and we had to work around it and but the thing like the good news was that we found a way to interpret these differences instead of looking at them as two different weapons we started looking at them as two different classes two different way or place the game so they were like two different like gameplay styles and when we actually started looking in their way everything became like uh in a way like more productive in a sense that we could see how we could imagine two different communities of players forming out there one that prefers the bow another one prefers the like the crossbow and start kind of like comparing each other out in performances around the idea of like designing like designing uh objects like in in a different way so um here's was like where we have to go through a very intense user testing session that was done like remotely because at that point the game was already nearly accessed so we could we got like a a few very dedicated

players on board they really wanted to test the new additions one of which was like the new weapon at the beginning they hated it because you know they were like super good at playing the bow and then suddenly they have to relearn the things from scratch and they like it uh but there wasn't really a way to be able to operate the crossbow the same way in which you operate it's like they're two different objects you can't really manipulate them in the same way so by doing some something i never done before like some uh like remote user testing through slack uh we were able to actually like work really well they were they were like incredibly passionate about the game so they start like commenting and talking to each other how what works what not and that was like very good for me to keep a track of all those conversations uh and also like uh what was really important was like asking them to send me a video recording of their play sessions because it was really hard to understand or it was really hard to read through what they were actually experiencing it was important to see what was happening really in the game

so that was like sandy i didn't expect to walk out like could have worked out uh but it worked out pretty well so just to conclude um uh forget any rules about like practical practical like uh intrigues or practical like rural times here technology is changing as the technology is changing that we are homunculus we look more and more like a person so the affordances will be different so here like like what what's really a takeaway is not about how to do things like in the in the details but uh to keep looking at people and keep looking at people move so like not only focus on what's happening on the screen what's up individual but look what people do because that's where part of the fun is coming part of the fun is coming from like making an experience which is like kinesthetic and finding a kinesthetic way so in a sense like practice dancing in a sense like reconnect to your body like like rediscover what's fun to do with your body in a sense because that like allows you to connect this idea of like making beautiful experiences

through body movement and that's concludes my talk and an open for questions thanks [Applause] yeah um i have a lot of trouble using haptics in vr and especially when doing them the up and down modulation that you showed in a graph can you share your technique with making that happen technique yes so like the graph i showed it's like a very crude representation because uptake is like you know you have frequencies and amplifications um but what i did uh was basically having like uh a similar approach to um like motion design so i i made like a way to actually modulate the optics through a couple of

parameters and and then i tweaked them until i i i was getting like a response that i thought was actually good and then i also create like different categories there is something that i call like the optic click that it kind of like mimics like a click or it's kind of like a soft like rhythmic kind of like a um feedback and then another one which is like thicker and longer i don't really know a better way to describe it it's a bit like describing sounds yeah yeah like thicker and longer and and then like uh that i remember something worked really well uh it was like the shake on the bow like it's an active fit but just like shakes like this like it kind of like gives you the idea of of like uh like like energy that gets released into the object but the main concept really i wanted to show was really try to place it into like key moments of the introduction like try to imagine the interaction almost like a dramaturgy and i try to imagine like what could be the point

where you want to make an emphasis in the tempo and that was that's at least what i found out was like the most effective way to do it and then of course the user testing yeah like sometimes i overdid it i put too much of it and it was getting confusing so as a little time like music it's better to just like be conservative and not too much on it thank you very much oh um how closely did the design of of the ux elements that you were just demonstrating time with the ui elements for instance the the little bar that you can see on the bow on the other hand did you guys experiment with using ui elements that were um i guess more attached to the camera was it or was it an immediate thought to we're just gonna have diegetic ui design and have that as part of the props themselves yeah uh thanks that's uh uh how to recap on this this uh it's been like we we didn't want to have uh uis on the

camera at the beginning even though i was one of those pushing for it like to have some of this information on on you know attached to the camera so the health specifically ended up to be like in the bow because we kind of wanted to have it visible as you were shooting other type of uis maybe uh that i am trying to go over here was how you actually select the arrows because i didn't mention that you can have like power up arrows and all these kind of things and like you can carry like those arrows like with you in some form on inventory and and that was kind of like a more challenging ux ui type of thing because like uh at some point we tried to work it out in a way that was uh um user friendly let's put it that way try to have it like with smooth animations and everything but then it really like broke against the fast-pacedness of the game like when it was game like really fast-paced and when it was getting like really just uh

intense uh that was just like in the way so we strip it down to something much more basic uh that literally appears and disappears but what we did was to play with time and actually with the time dilation because the action can be so frantic and you might feel actually so overwhelmed but everything that happens around you which is kind of threatening um like we we basically slow down time every time you open any of those you know ui for selecting like for selecting hairs it was used to be it was like thought out to be a way to like help like you know the people to actually select the arrows properly but then it became like a trick the players like exploded to actually do better you know the game so you know it turned out like in those ways so there wasn't like just a recap like we didn't really did like a very intense thinking about like where to place like the uis we knew that we wanted to minimize it as much as possible and really just make it about

like the action and and in fact we like the only kind of like ui augmentation we did uh were with the one that you see there like the health plus uh like the selection of the of the hours thank you thanks i have a follow-up questions about the happiness um so did you design them as audio waveforms that are parameterized so that they change procedurally or did you do something else for that uh no i wanted to design them as sound waves there was no time for that so i i designed them as uh kind of animation curves that that they were just like bound between zero and one so they were like normalized but then you could add parameters to them to kind of change them in terms of like length amplitude so those are the kind of parameters that i was changing so um probably there was like a better way to make it like more refined in terms of active feedback i'm pretty sure like if you talk to someone which is very expert a lot of feedback can tell you like a much better way to do it

uh like for me from the point of view of the interaction design and the general ux was important to nail that concept of like rhythm like when do we want to have them at what point of the interaction we want to have and you know it's like what types of active feedback we want to have the how long it is how intense it is as i said before like to classify them into in into things but you know i was like really keen to actually try the sound waves and see what will come out of it i don't know i don't even know if the like controllers that we have like at the moment are actually uh like responding very deeply to this kind of like you know sideway uptick yeah so i was actually wondering about that too because i've been struggling with the same kinds of things in politics right now um and uh oculus controllers versus like the um uh the vive controllers have like a very different haptic profile yeah like did you like do custom curves for both of those and like do you actually have like amplitude up and down kinds of stuff in your animation curves because if you just drive a single animation curve it

doesn't really feel like anything at least in the experience i've done we we did we did like mostly they are the same like like optic curves let's call them that way mostly they are the same but in a few occasions i remember we have to actually make them different for platforms so because like the vibe is very different the oculus was much better from this point of view it'll give you a much better response also like in terms of kind of like identifying a different type of optic you know in terms of like uh you know how long was the optic click or the intensity of it uh that really was better um but yeah all in all we actually kind of like moved kind of carefully around it because we didn't want to overdo it but also like we didn't know how far we could go with the platforms we are then so we like we took it like little by little um adding like you know little by little uh every now and then and see like if it was improving thank you so much thank you hi thank you so much for your talk by the

way oh yeah yeah all right um so i found it really interesting you know you mentioned the remote user research i found it really interesting how you're able to get your users to collaborate within you know the the you know their group can you talk to how you were able to set that up and how you found the users how you facilitated the the channel yeah uh about the collaboration i had to say we were lucky they were just talking they're out of passion we're just talking out about themselves we set it up as internal beta testers officially they were internal beta testers and but then i recruit them into like the user testing of the crossbow the way we did it was like simply by releasing the prototype or the crossbow into the game and and tell them like okay that's a new thing like just go like go white with it uh we did not anticipate anything really i wanted to get like the most kind of like pure in a sense environment possible and um

and then what we did uh was like we we didn't really think that true because i wasn't even thinking that was like a good idea to do it remotely but like what we did was like just to have a channel on slack and they were like talking they were like in the channel and they were like talking out each other like in the channel and they allow me to listen a lot they were like talking so much they were like typing so much how many basically many people yeah we got like uh i think was like four or five beta testers oh wow uh on board and and they were like no by the time they got into into the beta testing they already had like hundreds of hours on the shoulder so they were really really like like they really had built like a lot of muscle memory for the bow so the beginning was like like we did something crazy like we switched a teleportation like device from one end to another one and they couldn't play the game anymore uh so that was like the thing that we did but then i had to say very very important was to actually get the video playback from them like like the video recording

to send them on slack and just like to send them on slack and then like kind of like organize as a forum like free to comment on it and that was giving me a lot of in like information in the backline awesome thank you thanks next