A Game Designer's Overview of the Neuroscience of VR

[Music] okay everybody looks like we're ready to get started and thank you nice to see I got a full house here I don't see I see few standing up and actually usually there are empty spaces but I think we've we've got everything full so welcome this is a game designers view of neuroscience of VR and I will be talking about quite a range here but I want to start out with a disclaimer right at the beginning that I am a game designer I've actually been making games professionally since 1980 so a long time in this field been interested in science as a lot of game designers are all that time and I was working you know for big companies I was one of the first employees at places like Lucas arts and DreamWorks interactive but then from 96

or so up until about four years ago I was a freelance designer and producer and as I did that freelance work I got more and more interested in what was for awhile called serious games and then specifically working with games on the medical side and games that either work as treatment or therapy for medical conditions or games that are designed to Train doctors and caregivers and that was an area that got really interesting to me then four years ago I got an offer to come to Google and work as their chief game designer and couldn't pass that up and spent the last four years doing that but wasn't able to really work on the kinds of projects that that get out in people's hands and meanwhile I felt that neuroscience in particular was making wonderful strides and coming together with games and I just wanted to be part of that again so I've dived back into that end of it and I'm hoping to inspire some of you just out of curiosity how many people here have already worked with medical neuroscience

projects on your side of things okay not too many how many of you are professional game developers okay so about the ratio I expected mostly on the game side and I've a lot of hands it didn't go up of all the different niches of Technology and video and media and everything that we've got going on here so anyway I will be speaking about this from a game designers point of view and I I think I'm reasonably well-informed but any of you that are actual neuroscientists can talk rings around me so my apologies on that level if I don't have everything together for you but I'll be talking today about why I think neuroscience matters why it's such an exciting field at the time I'll be going into three areas specifically motion in VR and how you move through spaces with VR and how that relates to neuroscience the whole thing about immersion and presence and you know I'm not going to dive too deeply into any one of these just for

time issues but a lot of people bandy about words like you know VR gives you a sense of immersion and presence without ever really talking about what does that mean and and why does it give us that kind of immersion and finally in the area that I'm most interested in is what does this mean for emotions and feelings that are able to be evoked by VR and how is that actually affecting our brains and what can we do as developers to learn and take advantage and then the final thing will be to sort of wrap that up to talk a little bit about some of the future both of some entertainment possibilities and also these health applications that I think are really interesting crossover so in this process I'm going to try to you know give you a sense of what I know is really a fact what is being inferred from facts and what is pure speculation I may not always get that right so when we do go to the QA please feel free to ask you

know when you said X did you really mean that this has been proven and tested or is that just your wild ass speculation and sadly in some cases it will be the third but hopefully that'll be inspiring to my view of all this in fact having a long history in the games industry I think is relevant for a number of reasons that I've seen new technologies come and go I've seen new evolutions the first games that I played on a computer we're tech space in fact the very first ones were paper-based before they're even video screens and I've seen the evolution first two video screens where we use them to display more text but you didn't have to actually worry about whether you were running out of ink - of course graphics and animation and then we had revolutions like cd-rom and suddenly a hundredfold increase in storage than things like video and being able to have full what we called full motion video the idea that you could actually do you know amazing you know

about five frame a second video clips that looked absolutely terrible but all the different things we've gone through to get the latest revolutions in virtual reality which are both new and actually very old as I'll get into but an analogy that I found useful that I like to look at is that each time you get one of these new technologies it's like you're going through a mountain pass and there's a sense of excitement is this gonna open out is it going to be just a little cul-de-sac and you're gonna have to backtrack is it going to be a whole new continent and all you really don't know until you get out there and you start experimenting with it one of the things that everyone is nervous about is that both of those might be right and that you you're going out there and you end up getting stuck on this little precipice with no way to get into this huge area that you see opening up and all of your competitors are out there and they said oh yeah we figured out exactly how to use VR and monetize it and we're making lots of money I don't think anybody's quite at that point yet but all of us are waiting to see who's

going to find their way down to these open valleys potentially entire new continents to explore and I really think personally I'm a strong believer I think VR a are mixed reality whatever you want to call it all of these things are already really exciting and are going to get much bigger in the near future and I'll hopefully be able to tell you some of the reasons why but along that way it is easy to get stuck and part of why I think getting down to science I think neuroscience is critical with VR I've often used evolution and an understanding of what we as humans are evolved to perceive and to to appreciate to guide me into new forms of entertainment because I think if you get down to the fundamentals of who we are as human beings and what makes us excited what makes us engaged that's the basis of entertainment it's the basis of learning it's the basis of a lot of what these technologies are intended to do so hopefully neuroscience can help us find

this right path and get down from these little precipice and out into the the rich farmlands below it's in some ways kind of like a compass it doesn't always tell us exactly how to get somewhere but it can point the right direction give you some tantalizing hints and I'm not going to be giving you a lot of turnkey solutions to things but in some cases hopefully I will be intriguing you with areas that you can look into on your own more and get into a little bit more depth because there's very wide range of things to cover so I don't have the time to get into a lot of depth for any one thing so just to give you a sense though I'm not going to spend too much time on this you know particular aspect of it but I would like to give you a sort of historical sense of where VR fits in the sort of larger scale of human evolution and technology so I love this this cave painting this is a cave painting from a

believ around 20,000 years ago but there are cave paintings going back as far as 40,000 years and of course as people explore they may find even older ones I find it amazing that this is a twenty thousand year old piece of art and try as I might I couldn't draw anything with the best tools that was even close to as realistic a bison as that is it's hard to see in this particular flat picture here but the artist who was clearly doing this by torchlight with pigments that had been you know made Lord knows how actually use some of the contours of the rock to evoke almost a 3d sense of this bison at one point I knew the actual name of its it sanest an extinct relative of the Oryx I think that were roaming Europe at the time that this was made so it's amazing for at least 40 thousand years our ancestors have been struggling with this problem of I saw something and I want to

describe it to you in fact no I want you to be able to see what I saw and maybe it's not even that I saw something but I had a vision I had an something I imagined is something that came to me while I was sleeping I would like to share that with you and we've had this quest now for tens of thousands of years of how to improve our technology to do that some of the stuff going back thousands of years the camera obscura the the Romans were into this of using a sort of pinhole camera to be able to get this beautiful view of essentially a live-action camera what was going on around you in the Middle Ages they really started to get into shadow plays and projecting a light onto a wall and using you know your hands to make shadows and do animation there's been an interest in doing that for a long time in fact the next couple of slides I'm going to show you are the pictures that I took at a museum in Girona Spain near

Barcelona museum of cinema curious has anyone been to the museum of cinema in Girona just just one or two hands I would highly recommend it there are game conferences in Barcelona which is how I got out there if you have a reason to get out there it was great because I thought it was going to be just about moviemaking but they got into all this information about the predecessors including things like the Magic Lantern that you see here that was from roughly two hundred to a hundred years ago where they would start with originally a very bright flame and then they they used electric lights and was essentially some of the first slide projectors but it was a way to show images and project them against the wall and like many other technologies it was first something that you would you know is expensive and difficult to do so you would come to a public performance and then wealthy people were able to afford these very beautifully made versions of it at home and then they started to make less expensive maybe less fancy but more accessible versions for the home it was a huge

thing around the late 1800s and moved on of course to the early film cameras same kind of evolution of stuff we have to go to the theater and then wealthy people were able to afford their own cameras and eventually it became something where people could afford them and of course now we carry around a whole film studio in our pocket i I love the scene in the Back to the Future movie where he comes in with his 1985 camera and you Doc Brown is amazed that he's got this whole television studio on his shoulder and now we have something much more sophisticated that we stick in our pocket you know and give it another 30 years and it will be either on our glasses or possibly embedded in our skulls at that point but that kind of process of always wanting to have better ways of showing a higher resolution more realistic images and make it more accessible so that we could have it every day has been going on for a long time going into the 20th century

same kind of thing VR started to get involved earlier than a lot of people have seen have been wonderful talks at other VR conferences about some of the early stuff from the going back to the 1950s and even earlier I love this sensorama a thing you know just really trying to engage all the different senses and the military with heads-up displays has been working on this for quite a while I saw my first virtual reality demo in the mid 80s I think around 1985 when Jaron Lanier who actually was popularizing the term virtual reality came to give us a demo when I was working at LucasArts was actually Lucasfilm games at the time and he had a company called VPL which stood for visual programming language and it was really excited because they had invented this visual language where you could use images to program and almost as an afterthought he had created something called the data glove that you could put on and had sensors and you

could manipulate the images on the screen by gesturing with your hand and one of the things that kept happening is that he would be showing us the screen and moving stuff and he was so excited about this visual programming language and we were all focused on the glove and this idea that you're you're reaching into the screen and you can actually kind of engage with what was then even at that point being called cyberspace and in fact David Fox who just walked in here was at that very same demo but these were early stages like a lot of things you know it was promised to be this wonderful new thing that was going to be around the corner in just a few years and took a lot longer than people thought the Virtual Boy that Nintendo did was yet another precipice out there that was really an interesting experiment but it was just not good enough and way too cumbersome to be mass-market in the way that it could be successful but a lot of what we can do I think is a common thread and

all of these things is what's going on with our eyes with our brains how do we perceive these things you know what is it about these images going back to those earliest cave paintings they're still things you know we think of that as a picture of a bison but ultimately what you're seeing are photons from an image that's projected from my computer of a picture that was taken of a painting that was done of this bison there are so many levels between the reality of what that person saw 20,000 years ago and what you just saw now and yet coming I'm getting chills just thinking about this this you know that wasn't just a thought that they had that was a real animal that was alive 20,000 years ago that through this long chain of Technology and humanity has been brought to the present and you are perceiving it in your brain in many ways through a whole bunch of tricks to trick your eye into thinking that that image you see those photons are like the real

photons you get off of that animal itself so if we can understand how our eyes and brains work we can also understand all the shortcuts that our nervous system does because it can't afford to take in everything that's around us and process it all at once we're dealing with this now when we've got you know the the phones in particular are just going flat out trying to good do a great 3d display much less being able to scan the world around them you were just at the edges of them being able to start to do that a little bit so we have to get selective and understanding what's going on in people's visual systems their brains their vestibular systems and balance is really helpful because we don't need to give all that information we can give just the pieces that are necessary so the first of these areas on and getting get into is motion and one of the first things that comes up with with VR of course is that idea of a potential motion sickness this is something that's

been around for quite a while as well I've got the picture of the boat and a storm out there because of course there's a connection to seasickness car sickness and again thinking of it from an evolutionary point of view if you are moving around in a world and you see things in that world there's a one-to-one correspondence with how your head is moving and what your eyes are seeing and over history if there was a problem if there was a mismatch let's sort of get into the vestibular system that's able to sense how your head is moving and and therefore keep a sense of your orientation if there was a mismatch between what the semicircular canals in your your inner ear were sensing that you know it's really interesting that I'm sure many of you have seen this but there are those three semicircular canals that are oriented like the the three mutually perpendicular areas of pitch yaw and roll so that they can sense within your

head as you move your head rotated in any direction and if that was not matching up with what you saw for millennia for for millions of years really that usually meant that you had maybe eaten a poison mushroom or maybe that leftover haunch from the the kill from a week ago had kind of turned a little green and you're hungry and you ate it anyway and poison in your system sometimes interfered with what you saw compared to what you're actually sensing and the best response to realizing you were poisoned was to try and purge your system of those poisons which is why we get nauseous when we have things like motion sickness you know from whether it's seasickness car sickness or now VR sickness so it's useful understanding that it doesn't do much for the people that actually experienced it but we have learned quite a bit of how to minimize those things you know and sadly just as with seasickness car sickness there are some people that are really sensitive and can never you know

read a book and be in a car for example I think we're always and this is just my conjecture here but I think we're always going to have some people at one end of that spectrum who just have trouble with that and are gonna have trouble with a lot of VR if they they move too swiftly but there are things that we can do to help I'm going to talk about how the brain tries to match the motion of your head with the visual system get into some more of the details of that later but the bottom line is that if there's a mismatch there's a problem and unfortunately there are dozens if not thousands of ways that there can be mismatches so we're working on that and you know it you don't want that sense that you're being poisoned now those of you working in developing VR something that's happened universally with every team I've talked to is pretty quickly everyone finds the one person on their team that is most sensitive to VR motion sickness and that poor person becomes the guinea pig for all of the

new demos and it's a really simple system I frankly don't think we're gonna have a better system than that for a long time to come so those of you that are anybody here who is that person on your team oh yeah seasoned some some poor poor schlubs there that are doing that and happily are committed enough to keep coming here so more power to you and hopefully there will be some some reason for hope as we can figure out ways to minimize that but this will be an issue that we're going to have to deal with as we move forward so how can we minimize it well turns out obviously one of the biggest mismatches I say obviously but we only found this out empirically through trying stuff so having a really high frame rate is is critically important 90 frames a second is being held out it turns out that it you can certainly get up to 120 frames a second before it starts to get to the point where I don't think there's a whole lot of necessity to go much faster than that the the ability of us to proceed

stuff happening faster than 120 frames a second is it is there but it's it's now we're getting down to a pretty tiny percentage of the population and yet this is really an interesting point because frame rate of course is governed heavily by how much you're doing in each frame the more processing you do the harder it is to keep up you know the faster a processor you need to be able to do that and so almost everything else that we have in VR is fighting against that frame rate problem there's this constant temptation to say well we could make the visual field bigger we can come into higher resolution we could do you know better quality graphics we if we could just live with a little bit of a less of a frame rate and sometimes that's actually a reasonable trade-off if you're not moving people through that space you can get away with a little bit slower frame rate but in general that's going to be a critical thing and closely coupled with that is sensing where your head is and what your vision is is is looking at at any given time it's been

thrown around many times that twenty milliseconds motion to photon is kind of the minimum that you want or rather the maximum that you want it's faster than that is even better and what that would mean is that you need less than twenty milliseconds between the time you're turning your head and the time you're actually seeing the visual system respond 20 milliseconds is actually pushing it at just that's the area where it's comfortable for a lot of people but not everyone and it's really hard again to get much faster than that particularly with the kinds of systems that we have that were not designed for this in the first place meaning that the kind of visuals we have on our phones and TVs and everything we're not originally set up for VR they were set up for watching a screen passively and not having a screen that's moving so getting all those visual cues right is really difficult you don't need to get every single visual cue right of course and I'll talk a little bit about some of the shortcuts how am i doing on time

yeah but won't have much time for that minimizing acceleration has been really critical because of course your inner ear tells acceleration if you're moving at a constant speed it has trouble telling that pilots and planes go through a process where they're shown that if they are you know very gradually moved into you know different planes they'd actually don't sense it and they can even be hanging upside down and not realize it if they don't have the visual cues to help them understand that but there are some creative solutions I'll talk about one particular one that I think is a great example of using understanding of the brain to help use some shortcuts so that we don't have to do quite as much processing and we can make it more comfortable for a lot of people so some of the stuff that we found having a constant speed certainly helps it's interesting Chris Pruett who was just in here earlier in the room gave a talk a couple years ago where he talked about how

we've had as game developers to unlearn certain things that smooth acceleration into it to a certain speed and deceleration was the the gold standard for a long time and then we hit VR and it turns out that no going from a dead stop to a constant speed and back to a dead stop is much better because nobody really does that in real life so it doesn't set off that sense that there's something wrong in your brain that wait a minute something's weird because of course it's weird just like the other solution of teleportation we aren't used to what teleporting is like if if an alien race that can teleport showed up came in and and saw our VR they might be horrified because we get teleportation all wrong and doesn't feel that way but luckily we're not one of those aliens so we're okay there but those are things that work the other thing that I mentioned is curves this is from a game that we did at Google called Zushi and yes Google actually does make some games internally this was a game a VR game where you're on a raft moving down a stream and we did that on purpose

because it's slow and it lets you see where you're about to turn and curve and it did have acceleration as you move around those curves you're getting you know acceleration around the curve but we found for most people as long as the speed was slow and they could see where they were going and anticipate that they'd have that curve it wasn't so bad much like somebody in a car if you're driving you're much less likely to get carsick than if you're a passenger and if you're looking at a book or something and you're not seeing the world around you or with seasickness if you're in the hold and you're not able to see the horizon it's you don't get the right visual accused you know the cues are that nothing is moving and yet your inner ear is telling you you are moving so matching those things up really can help now back in the movie industry you know I'm sure most of you are familiar with this little bit I have you know very strong memories of being in a theater in 1977 and seeing that first Star Wars movie and this moment where it zooms down and bounces up you could hear the audience go whoa because there's

enough of a sense with this huge screen that you're actually kind of on a rollercoaster ride and they did it again in the Battle of Hoth and and Empire Strikes Back where there's a quick moment where the snow speeders do a similar kind of maneuver but in general people don't get movie sick you know there there are some people actually shakey cams and having that that move around can can really disturb some people but a lot of this doesn't bother people and it's a really interesting clue you know when you see well why is it that we haven't had so much of a problem with films and you look for the cases where it is a problem so you know one of the areas where it does come to be a problem is in IMAX films suddenly people get motion sickness again the interesting thing there is understanding about the way your visual system works you've got what's called the the fovea area that is about two degrees left and right up and down the very center so when you're looking at something if you're focusing your eyes are focusing the image on a

particular part of your retina that essentially is for high-resolution viewing the rest of your retina has a lot of rods and cones that can pick up visual images but that's what we call our peripheral vision and even though our brain tricks us into thinking it's a nice continuum it's not those two areas work quite differently and if you can take advantage of that you can actually shortcut a lot of these things and can be very helpful so what happens is in an IMAX film or other cases where you sit really close to a screen your peripheral vision actually sees all of this motion and it's starting you know it turns out that your peripheral vision is connected to a lot of your systems that you a sense of orientation partly because if you are focused on for example you're stalking an antelope you may have your foveal vision aimed right at that antelope but as you're moving along your peripheral vision is telling you the way your body is moving and matching that up with what your inner

ear is saying so if you have motion in your peripheral vision your your brain is processing that so one of the things that has come up is if you can you know there's some really interesting experiments of processing an image that you see only in the fovea area that you're looking at and having the peripheral vision you know not being as high-resolution not requiring as much processing power it's necessary to have eye tracking and I'll talk about that in a bit but one of the shortcuts that people have found is that if you blur or even black out your peripheral vision and still move the foveal vision you get an interesting effect and this is why something like Star Wars was okay but IMAX can can be a problem to people and what this is called is is tunneling locomotion and actually coincidentally at this very moment there there's some Google Earth people and one of the other talks and you're going to get at least a little sense of Google Earth this is a

brief clip of tunneling in Google Earth what they do is in order to move you they bring in this grid into your peripheral vision and you still see a moving image in the center and you're gonna have to take my word for it because you really need to be in VR to get that full sense of this but oddly enough even though this looks really strange in VR it actually feels quite comfortable if you're looking and for one thing is that as the rest of the peripheral vision goes away you naturally are drawn to look directly at the area in your fovea view there that's directly ahead of you on the screen and it turns out that this eliminates a lot of the kind of motion sickness people get from that acceleration and deceleration because that is apparently being picked up by your peripheral vision and when it's not there to disconcert your peripheral vision it's okay and you can still keep people oriented with the motion that they're going through here's a game that I have to say I haven't played I just found this online call sublevel zero that does a similar thing and there are other VR demos and games

out there that use this tunneling locomotion I bring it up not to say that everybody needs to do this but just to point out that it's a really interesting clever way that not only deals with the problem of motion sickness and VR sickness here but it also means you don't have to process all that stuff you you're just graying it out and that gives you a lot more cycles to work on the area that you're showing in the center of the screen which can be very handy you can keep up a very high frame rate in the areas that you do show this is likely to be coming to mobile in the future and we right racking as I said is really interesting right now the fovea view is done by sort of brute-forcing saying the only thing to look at is this one thing right in front of you but wouldn't it be great if we get to the point where wherever you happen to look that's the only area we render in high resolution and the rest of it we don't have to bother and in fact with eye tracking rigs that are out there they'll probably be demos that I haven't had a chance to see

what's being done on here today but I've seen it a lot of other VR conferences there are eye trackers that providentially you need to be really close to somebody's eye and of course with VR headsets and AR headsets for that matter we now have a reason to mount a camera that can look at somebody's eye and see the motions of their pupil it's not perfect yet but it works really well already and as these things become more commercially available I'm expecting that it's going to become built into all systems certainly on head mounted VR but even on your phone there's a some interesting stuff going on and the announcements about the new iPhones and facial recognition I've read some interesting stuff about using those forward-facing cameras to actually track people's eyes there too so hopefully even with mobile we'll be seeing this in the near future I don't have that speculation again on my part but it's nice not having to work for Google and and worry about having it

be front page news if I even hint at something like that see some of my fellow Googlers are smiling ruefully at that one so the trick is you you know I can only cover this one area here but if you learn about the ways that the eye and the brain interact you can actually get little cues little clues really that you can use to them ferret out these systems and find ways that you can take shortcuts that can make the VR quality better so that's all I'm going to say about the motion side but motion is an important part of the sense of immersion that we get in VR that word immersion gives you this sense of it's sort of this nice warm bath and and certainly VR is pretty amazing when it gets everything right of how how you feel that you're actually in the moment that you know it's just it's almost indescribable and the nice thing is this is one of the few areas where everyone has had that experience and knows what it's like but the trick is

there's a whole lot that you have to get right there are entire talks given to even just single elements of this certainly this whole range of things I've listed some of the more common areas that you need to get right I've talked about getting the frame rate in the head tracking right that's only a beginning of the the puzzle there are so many other things that you need to do to try and recreate what we see with our eyes that it frankly at this point with the processors with the screens and everything we have it is impossible to get really close to what our eyes and brains are capable of seeing happily though it feels that we've gotten just good enough in the last few years that it starts to be a convincing simulation of that one of the things that isn't even on this list that I've seen some interesting first steps on is the fact that your eye works a bit like a camera and that you focus with muscles in your eye changing the shape of your eye and and you know of your your the lens in your eye to focus as a camera does close and far away so if you hold your finger

close to your eye you focus on it and everything else in the background is blurred and likewise if you look in the distance you can hold your finger up and that's kind of blurry it's because the lens in your eye is actually stretching and changing shape in current systems that are out there now they can't account for that but there will be systems coming in the fairly near future that will be able to actually even with one eye close be able give you a sense of depth but right now one of the miscues that we're getting is that things don't actually change depth of field when you're looking at them through a screen you're looking at this screen that's right in front of your eyes and you can kind of fake it with some cases but it actually doesn't quite do a full job there so some part of your brain is saying there's something not right here and if you're one of those unlucky people who raised your hand it's probably making you feel a little bit ill so there's a lot to get right if anybody here worked on this don't raise your hand I'm sorry I brought this up because I found this online I thought it

was a great example of what not to do and I don't want to hit these people too hard because it's a language teaching program and they really weren't working so much on VR visuals but some of those things I had in that list before are just broken here for example in the foreground you've got a kind of blurred table in a crystal clear sharp bell that just really irritates me because it feels like that you know if that why is that so clear when the rest of it is is blurry and the blurriness is consistent it should actually be sharper closer up and a little blurry or and the distance or vice versa depending on where the depth of field is you the worst thing about this and in my opinion though is that the interface that they have is floating therein in space in front of you and it turns out that this is a bad thing for a lot of reasons you know for one reason your eyes are focusing on it and they're converging on this image and even though that may be shown left and right in slightly different places so

that your eyes can say yes this is about you know two feet in front of me it is almost certainly being done without any perspective change in the two eyes and if it were a real thing you'd have a different view from your right eye in your left eye particularly something that's only a few feet away you see a little bit more of the edges of something on the right side with your right eye left side left left eye and in a lot of shortcuts people use like this for interface they don't do that so it's really a mistake to sort of violate that sense of presence by breaking any of these rules and die jet ik interface is the the turns brought up to actually put it into the world so in a case like this you could just as easily put that interface on what looked like some plaques that are part of the 3d world and that are situated in that 3d world that your eye converges on properly and you know it makes it a lot easier for someone to perceive it's harder is a lot more processing to actually put those up correctly but this is one of the things

that you I think should not take too many shortcuts with or it's really going to be irritating to the people and again study the way the visual system works there's a lot more than I can go into in just this hour but another thing another great lesson that turns out to be true of VR as it was of other things is that less is more you don't actually have to get everything right you can actually cut a lot of corners as long as you're very minimal about what you do and I'll talk about some examples but first I want to remind everybody when you do get it right it's pretty amazing that this is one of the first VR images I saw on the modern you know VR world that really convinced me I that's how you have time for the anecdote I'll keep it brief I saw one of the early valve tag rooms before they even announced what the vibe was and it was just a life-changing experience for me in particular I remember when I was told ok pick up the controller and I said well I can't see

the controller so I'll sure you can look over your shoulder and they're on a virtual table was a virtual controller and I remember reaching over thinking I thought I saw a table when I came into the room not really believing it and feeling an actual controller exactly where this virtual I told me it would be and feeling that in my hand was one of the most amazing you know moments I've had when VR gets this right it's amazing and it wasn't very high resolution it was really just a simple trick of combining that that haptic feedback of what I could feel in my hand with what my visual system told me and it was exactly where it was supposed to be that's really a part of the critical thing and part of what I think is the magic of VR and what we're really aiming for here so go for that feeling a lot of people say yeah VR isn't ready yet you know we need another 20 years no I think you have that kind of experience and how many people here experiencing VR

had at some point that amazing whoa this is way more real than I thought it would be kind of see hands yeah look at how many hands and virtually everybody in the room is here because they have that moment believe in it folks you can fool some of the people some of the time you know or even all the time but there's a lot of us here and I've had that experience with every person I brought in to VR with some very very few exceptions and often it turns out that they have a wandering eye or some other problem that they can't focus and get that full sense of VR okay next thing I want to cover though what I think is the most important thing is sense of emotion anybody want to call out what this movie is Creature from the Black Lagoon a classic was really interesting about this is that it was one of the first movies that was done in 3d on a large scale with those awful little red and green things back in the 1950s I believe and what they did is what a whole bunch of people with modern

VR doing which are saying hey we can scare people with this stuff and it turns out that VR is really good at scaring people I don't know if you've seen maybe first hand or maybe the videos of people who were you know put into some kind of horror game and took their thousand dollar headset and threw it against the wall before they realized oh gee sorry I was just so scared I had to get it off my head as quickly as possible so probably not a great thing to do but cinema is with some of these other things I've mentioned has been experimenting with these things they're great as a place to learn from well my friend Hal Barwood who is a film director and writer before he got into the games industry had all sorts of interesting stories like this including one about 3d sound and how when they started to first do surround sound they found that people were about three times as likely to jump from the exact same sound played behind them as in front of them and again from an evolutionary standpoint makes perfect sense why we

are hard-wired to be a lot more scared of stuff that's creeping up behind and stuff that we can see in front of us and with the 3d stuff of course all the way up even to frustratingly movies that are coming out this very week their stuff being thrown at your face to kind of oh no and you know that may work the very first time you've seen it but please you know don't keep doing that it gets very thin very quickly you end up with something that will look you know as dated as our little monster here does very quickly so really what I'm saying is that that VR and emotion in VR is a good servant and a bad master you know that you can go from that that a warm sense of immersion I was showing you two really scary horror very quickly and this is actually I if you have that ability to do you focus your eyes or cross them to make a 3d image when this girl sort of pops out in

3d even even just looking at the screen it's a little bit disturbing but why is it so disturbing why is horror in VR so strong well this is one of those areas where there's some very clear brain science behind it at the very bottom center of this there's a the amygdala called out and the amygdala is a very old part of the brain it's present in in fish and reptile brains in fact it's much larger in proportion to the entire brain in a lot of our ancestors and our relatives like that our cerebral cortex all that big gray matter where you do a lot of your critical thinking is a relatively recent addition compared to the amygdala and what the amygdala can handle some very basic functions fear is one of the primary things among them it also handles anger and aggression and those two together give you the the famous fight-or-flight you know that you need when you're under a threat to be able to decide almost instantly should I

run away from this thing should I fight it and that is so much built into our DNA that you know it's the first thing that comes up before we do any kind of extra processing on that but one of the things that the amygdala also controls that is also and just as important for propagation and an evolution is arousal and intimacy this is also part of our brain that gets activated in romance and when somebody is close to us physically or emotionally it a lot of that is is mediated by the amygdala now in videogames the fight-or-flight stuff aggression fear we're really good at that sort of thing the sort of softer emotions like empathy and love we're not quite so good at that you know we're still working on that that piece and I'm really excited because for a long time I've worked for several companies LucasArts and

DreamWorks in particular where there are a lot of movie people and they were sometimes even a little smug about the fact that movies are so much better at empathy at love and romance and that sort of thing this is something that's gone back a long time but one of the reasons they're good at it is because we feel that sense of intimacy in fact the very word brings forth the idea of being close to somebody physically actually close to them this goes way back again to film days about a hundred years ago the director DW Griffith was pitching to some people he had Mary Pickford was a one of the first it girls one of the the first actresses that was was famous because she was so pretty and had that sense of presence on screen that came through and he was going to do something called close-ups where her her face could be 50 feet tall on the screen and try kiya he didn't put it this way of course but the idea is to trick your brain into thinking not that she's 50

feet tall but that she's normal-sized and she's really close to you and it turns out that movie executives haven't changed at all in the last hundred years and that what he was told is you know mr. Griffith you know I'm horrified at this when we hired miss Pickford we paid for the entire actress I want to see the entire actress on that screen but he was able to convince them that close-ups actually are a good thing and of course they're one of the ways that movies have been able to give us that sense of intimacy and leapfrog well we don't really get in games I mean having even a full-screen game with a face close-up doesn't quite have that same sense as when you're in a movie theater but with VR you can be every bit as close and in fact even closer and it's even better than a movie screen or even a 3d projection because it's tricking more of your senses and as you turn your head you can tell that this

character or this person or this Hedgehog is right there next to you and some of the oculus people a few years ago gave a great talk about how they actually ran into trouble with this because I'm sure a lot of people here have seen Henry the oculus movie that they did and he's a sad little Hedgehog it's his birthday and all of his friends of abandonment doesn't have many friends because he's so spiny and and and you know you really feel sorry for the guy and part of that I'm convinced and now it's conjecture but I think it's based on good inference from the way that that your brain works is because you are physically right there in the room with him and he's right next to you you get that sense of empathy and caring about him partly because of that and also partly because of something else he looks you right in the eye and he can tell when you look at him not so much from where your eyes are but just from the way your whole head is pointing in in the camera and they found that people responded really well when you look at him directly and he would smile and wave at you and his eyes would get big so they put a lot of that in there they

didn't stop to think that the story they did is about him being sad and lonely because he has no friends hi how are you doing yeah it's so awful everyone I care about you know hates me hi how are you doing it's like oh wow I guess I'm a real pariah here and he doesn't like me for some reason so I actually had a bit of a Miss connect between their script and the emotional connection of both that proximity and gaze and gaze is it we haven't experimented with a lot in VR as we get into eye tracking expect we'll do more in film it's been done all over the place you know you actors are famous for their smoldering looks and it really affects you emotionally and again you look at this close-up when Leonardo DiCaprio yeah kate winslet are right there in front of you you can see them close up and it affects you emotionally well this works really well in VR and in terms of shortcuts you don't need an actor you don't need full resolution google has something called spotlight stories that

I'm still very fond of and when I was working for them I would try and promote it because a lot of people weren't that familiar with them how many people here have seen spotlight stories and played around with them well it's not bad to see I've up to about a third of the room I think two years ago I would get one or two hands that would go up strongly recommend I'm going to mention a few more of them in the next few minutes this particular one is called pearl and it was the first movie made for VR that was nominated for an Oscar unfortunately didn't win this year but it did win an Emmy this year I'd really recommend you check it out on whatever VR system you have it's available on quite a wide range and certainly on daydream and the less expensive ones as well as on vibe and it's really effective part of the reason it's effective it's a very short it's like a 5 minute almost like the Pixar shorts you see before they're their full features and as you can see not very high resolution there's no dialogue in it so quite simple but it's shot almost entirely from the passenger

seat of a car so you guys this is a full 360 view and of course when it's actually in VR you're seeing a much narrower field of view so everyone seems much closer and you get that sense of physical intimacy which I believe is triggering your amygdala is making you feel like these people are right there in the car with you you could reach out and touch them and vice versa and that I think is triggering a sense of empathy I found myself watching this in VR it's extremely emotionally engaging and when I've seen it on YouTube on a flat-screen doesn't have that same effect and then I go back into VR and I actually find myself moved to tears again so I would recommend that you check that out of course your mileage may vary about what really hits you so for a motion I would recommend watch out with fear and aggression yes you can get really strong with VR very easily but you can overdo it very easily let's not be the the laughingstock of the future and overdo it I really think there's a

lot of room for emotion and empathy and even romance in games that we can do in VR films now I'll get into some speculation on that in a moment but realize that VR is building on very old biology this is talking about stuff that in some cases as many millions of years old the amygdala was formed you know as far as scientists can tell hundreds and millions of years ago by our very very distant ancestors and you can use this to explore some really new approaches that we've never been able to do before what are some of those things well let me speculate on a few of them in terms of movies again I'll hit some of the spotlight stories there are two that I would also strongly recommend you look at special delivery and the Simpsons did a six hundredth episode in VR and what's interesting about both of these is that both of these are comedy special delivery was done by aardman studios who do the Wallace & Gromit films they both use a 360 view all the spotlight stories

do to give you the ability to look anywhere around you but they've built a bunch of Easter eggs and little in stories that pop up in an inspector in fact it's about a night watchman who thinks there's a burglar and it turns out it's Christmas Eve and it's Santa Claus sorry for the spoiler there but the the point is is that he's tracking Santa all through this courtyard and you're in the center of the courtyard rotating around looking into the rear windows it was actually kind of Hitchcock inspired in a couple of ways of all these different apartments but if you look away from the night watchmen they basically go into a pause loop and there are other little stories that you can uncover just by looking at them without needing any gaze but just seeing which way your head is turned and some of those play out over several episodes so that even in the couple of minutes that this story runs if you go back and look at a woman she's feeding some pigeons and that's great you go away you look at her again there are so many pigeons that you get scared and runs inside her house the third time you look at her the pigeons have formed a little pyramid and the one on top is trying to

open the doorknob and she's inside peeping out you know in horror at this because they want to get all the breadcrumbs right away so that kind of thing of building in Easter eggs I is is really great and the fact that it's up to you that there's interactivity and that you get to choose where to look but another thing that's come up is sort of a negative lesson people don't like to stand up and rotate for very long periods of time so I really think that we're going to be seeing a lot of 180 degrees seated VR where you sit down and you can turn your head and see stuff happening all around you it seems but you don't need to be constantly looking behind you it really is is it's distracting it's exhausting to try and do it too much and that in turn will make it easier we won't have to render quite as much it'll be much easier to shoot it when you don't have to worry about hiding the cameras completely because of course with regular film you can do that because they're only pointing one way here's a way of compromising between that wide wide field of view of VR but not 360 so

I think we're gonna see quite a bit of that it's easier to shoot it it's easier to light it that way and the interactivity can come through the gaze you can tell which people which way people are looking at least at first with the way their heads are pointed and soon with their actual eyes with eye tracking and this could be a shared reality where we're all seeing the same thing or it could be essentially single-player perhaps you know you're your own particular preferences are either for Kate Winslet or for Leonardo DiCaprio you could see it only from the point of view that you want to see and not have to as we do now and movies see both points of view of that or you can have something that's meant to be essentially a single-player experience that everyone gets a different experience each time that they they view it but in general there's going to be strong replay value so you can shoot these things once and have people come back and watch them over and over again to catch all the bits they missed with with that Simpsons episode there's one point where I look up and I realize that they've done a little gag on the Sistine Chapel and painted on the roof above me

is a little Easter Egg it's also the monetization side that if you imagine these kinds of VR movies happening you can do ads or product placement in real time so that in the world around you there can be actual real time ads that are changed every time somebody sees it okay games is medicine this is an area that I've been working a lot in I know not a lot of people are in it at the and I'm gonna talk about some stuff that is not vr for a moment just in the last week there have been a couple of announcements pear therapeutics got FDA approval for an app that will help treat substance addiction it's used in conjunction with therapy and there's another company called Akili that I'll mention in a moment but what's exciting about them is that they're in the process of trying to get FDA approval for the first time of actually being able to prescribe the game on the same level as pharmaceuticals which opens up a 300 billion annual market where you

can actually have a doctor say yes you need to play this game and here's a prescription for it really exciting area they're hoping to get approval it's in double-blind testing so they won't know until later this year play to prevent was a project I worked on very briefly with some people at Yale it's a game a 2-d game you see it pictured here are shells jessie shells shell games studio did the work on that and it's a great game to teach kids teenagers about risky behavior and has actually now been found in their own study to actually lower the amount of risky behavior by teaching him through the gameplay of that it's not such a great idea to get drunk and you know go out and have sexual encounters for example neuro racer was another project i got involved with that was done about started about 10 years ago here at UCSF that was to treat aging adults who are less able to pay attention to signs as they're driving as they get older and they were fined if they found that through a game therapy

they were able to bring them back to the level about it a 30 year old even if there were 70 when they start this process in terms of being able to perceive signs and still control a car and process this and that became the the basis of this Evo game that is the one that I mentioned that's currently being reviewed by the FDA I really think we're going to be seeing a lot more of this if they get the approval and it opens up this part of this 300 billion dollar a year market they'll obviously be a pretty quick gold rush to this it's tough though because in order to get approval you need to make a game high-quality game as you can see the graphics and this are great there's a lot of ex Lucas it's people who are behind this to begin with so they they drew heavily on the game side but they also do just as heavily on neuro scientists and doctors to make sure that the medicine worked and what you're going to see though is that you're going to need to put a team together make a game and then freeze it and test it for sometimes a year or two years and then be able to release it so

all those people you hired have to stick around for those couple of years before you even know whether you're going to get the approval to release it it's not an easy path but it can be a really lucrative one it can be one where games are actually helping people in a really direct obvious scientifically provable way so I'm very excited about that a lot of possibilities with that in VR some of them have already started to happen we've seen phobia treatment in VR since some of the very early headsets women on top there is an arachnophobia and seeing VR spiders was just as scary to her as seeing real spiders but much safer fear of flying for example also they used to take people into airplanes and then get them off but after 9/11 it became much harder to bring people into an airport and get on an airplane and get off you can do that with virtual planes and if somebody gets nervous or has a panic attack in the midst of a takeoff bang they shut the simulation down you don't have to worry about the fact that you're actually in a plane and you can't stop

post-traumatic stress disorder has been treated successfully with VR acute pain remediation as you're seeing in the bottom slide there turns out that VR sense of immersion can take people out of their bodies and take them away from that feeling of pain that they're there in the midst of it's great for training doctors and caregivers there's a company in Chicago that's teaching people to care for people with macular degeneration where you get this sort of black I talked about the fovea they're phobias are obscured by this this essentially great cloud as it breaks down in the training you actually have to go through having a dinner with some people and wherever you look there's this big gray cloud so you get to experience very much like what the patients that you're going to be dealing with have to deal with every day and you suddenly understand how much harder it is to be able to just get stuff out of the corner why mirror therapy after a stroke is being done where if someone's right side is functional on the left side is not if

in VR they see their left side functioning turns out that it grows some neural connections and can regrow some of the connections that are broken by the stroke Asperger's if you look at it-- racking people who have are on the autism spectrum often have trouble knowing where to look what cues to get what to what emotion somebody may be experiencing now you can tell whether they're looking at somebody's face or not or even whether they're looking at the right part of somebody's face and there's some interesting training going on there there's even room there there now more speculation but I've seen some early suggestions that VR may both diagnose and possibly even treat some aspects of depression Parkinson's Alzheimer's so really exciting stuff but really the future is in your hands a lot of that stuff that last stuff is very heavily on the speculative side and yet it is inferred because of some very promising early research that I've seen so I'm urging you to get out there try these things whether you are looking for

neuroscience to inform your games and make your entertainment better or use your entertainment to make neuroscience more effective in treating people and helping people both of those are really exciting areas and I'm gonna leave it at that I've got time for just maybe one or two questions so if people will line up at the microphones and also don't forget when you get your your responses your requests for responses to do a rating of the talk and I'll leave it at that thank you very much this I have a neuroscience question for you which has always puzzled me you know the expression looks can kill so we know that we can give out a certain kind of energy from our eyes I mean without changing any other part of our face I can look at you in a way that is devastating and they can look at you in a place that is loving do we know what's actually happening physically inside the eye when those changes are happening you know I point out I am NOT

a neuroscientist so there's very layman's understanding of this the one thing about that that I can say is that it is easy for us to think that we're getting it just from someone's eyes but there are a lot of interesting studies that suggest we may be reading body language and a lot of other things subconsciously so I would question that assumption there was a famous again comes to the film side a famous thing where a I believe is a Russian filmmaker cut between people's close up staring eyes and yeah I think so so and that was a case where depending on how you were primed you read a totally different expression into that exact same image so I my guess would be that it may actually be more complex than you know we focus on people's eyes for many reasons in particular because it's really critical to know what that other person is looking at on you but I'm guessing and this is again a layman's sense that it's probably if I'm looking at you like this or like that then my body language may

be telling you a lot more than my eyes actually are and yet I don't know it may be micro muscles around the eyes it may be the way that your pupils grow or they your pupils get bigger the more attentive you are so if I'm looking at you casually it may actually not be my pupils are very small if my pupils get bigger you can pick up on the fact that I'm interested in what's going on it's actually a reason why romantic settings are off often dim because your people's get bigger when it's darker and it makes you feel like you're more you actually have more attention of the other person so that's I was just thinking in terms of replicating or evoking an emotion in VR that that could be an important thing to me was an interesting thing to explore and that's partly why I think eye tracking is going to be so critical but we'll have to experiment and one more question I think I'll go out in the hall people have more stuff to follow up on yeah I had a question so I run a immersive audio company and last year we

had a we worked on a horror VR game and our audio programmer actually had PTSD from having to work inside us Wow he couldn't yet eventually he said no no more changes it is how it is and so we run a we're out of Tempe Arizona and we run a VR for good organization now and so I'm just wondering what is the hahaha trying to make up for what you've done exactly yes yes we're in repentance right now but I just wanted to know what is the what is the discussion that's happening like we're talking about this right now but what about developers I know just reading the brain by David David Eagleman he talks about how our brain really functions almost is like a peripheral device that can take outside information and start to interpret it what is my big question is what is happening what is gonna happen once we start taking this incredibly immersive

tool hooking it up to our brain and feeding it content that that in the beginning it's it's it we kind of have this friendly like innocent approach so like let's try to scare somebody but what kind of effect is that gonna have on people and are we having this discussion with amongst developers yeah so the good news is I have heard that discussion going on in a number of areas the a bit scary part is the fact that it is a little bit of Wild West out there and there's nothing to stop people from trying things that may be too scary or you know too jarring for the public but of course that's always been the case with books movies a lot of other areas and I I'm not forgiving it but I also don't think it should be banned or regulated because I think we need to let people be able to experience that and make their own decisions I often use an example of pastry chefs in that if you

do nothing but eat its was it desserts it's really terrible for your health but the the solution is not to ban desserts it's to use it in moderation and to use common sense and to be responsible you're both as creators and as consumers and that's unfortunately the best I can I can say about it at the moment but part of what I've actually been drawn to this partly because I do want to see what positive things what undeniably scientifically proven positive results we can get from this because that does help counterbalance the fact that like any tool I have that you know good servant bad master thing you have to really be careful about how you use it and I think we need to be responsible I know I'm not the only one bringing this up I've heard at a number of other VR conferences people talking about responsibly use of that emotional connection in VR but beyond that I think you know there's so much we're still experimenting with if we will have to try our way through that and I'm encouraged by the fact that there are a lot of reputable scientists and universities that are exploring a

lot of this stuff and the faster we can get that information the better so is there a resource that because I totally agree with you is a resource that we can go to in order to learn more about things that we should be thinking of you know I don't know of any central resource I actually have people dead at another conference somebody came up in questions and said maybe we should start a central repository and somebody else said well yes but who do you write how do you regulate what gets into it and verify whether it's some crackpot thing that you know it's chemtrails that are you know seeping into our VR systems and that's a tough problem and it's not really my area of expertise so I'm gonna dodge that for now but I would happily support efforts along those lines and I hope that we will have some some better ways of being able to measure that in the near future okay thank you thank you and I'm going to go ahead outside now so they can clear the room and I think we've got lunch coming up next but thank you very much you