Believable Humans (In Video Game Animation)

[Music] this talk is called believable humans and I'm Mike mumar I'm the director of PlayStation visual arts service group and this is my colleague Dr Mark sear director of laboratory for animate Technologies I'm going to start with a interesting thing when I was putting together this talk I there's a couple quotes in here but this one's particularly interesting to me um you know you can connect the dots looking can't connect the dots looking forward you can only connect and looking backwards so you have to trust that the dots will somehow connect in your future and this talk really is about my experience in the last 12 years in kind of the digital human space and markx had even more experience with that but my experience starts at um oh it's not the Talk's not on hang on hang on we didn't actually put the that'll be better for you if it actually shows there we go at least we didn't get too far down there's that

okay so again my my story starts um at least in digital character space around 2003 um and that's when I first met Mark SAR I was working on the Polar Express and he was working on Spider-Man um I joined PlayStation in 2007 and that's when I started working on on character stuff there cinematic stuff since I've been at PlayStation we've worked on about 45 different titles um and we do art and cinematic support for worldwide Studios focusing on digital characters faces and animation so uh I'm going to show you here if you can see so since I joined PlayStation we've done roughly about uh 14 animated feature with the equivalent of about 14 animated feature films in six years but it's been in the video game Space

um going back to Mark just so that he so when we do this talk we're going to transition I want to do the introduction so he doesn't have to do that um he's a PhD in engineering and specializes in performance facial capture uh and advanced rendering techniques and also facial motion capture he's got two um Academy Awards one for his work on at image works on Spider-Man 2 and the other four his work at weda on uh King Kong and Avatar so we're going to start talking now about um some of the stuff from PlayStation and some of the story of kind of where we were how we got there and then kind of where we're going and I want to start with the last of us we're going to talk about it really briefly um how many of you guys went to Damon Shelton from Naughty dog's uh map talk okay cool I'm not going to talk a lot about it but one one of the important things from for me in this talk especially in terms of

of uh believable characters I think this is a defining moment for video games and this is a scene that we did in um in in San Diego with my team we uh helped create the characters and we did um a lot of actually this entire scene of animation for last of us we did about 80% of the Cinematic animation for the title so hopefully this will play is there audio hang on how many you guys know this scene oh this won't work without audio David or whatever from the sorry y'all let's see if this works now but yes sir

somebody we've just been through hell okay we just need [ __ ] please don't oh no Sarah move your hands baby I know baby I know God listen to me I know this room you're going to be okay baby stay with me I want to pick you up I know baby I know Hur come on baby please I know baby I know Sarah baby don't do this to me babe don't do this to me babe come on

[Music] please please please please all right we're back so uh sorry to start starting such a downer um but this is a pretty important um moment for video games I think um you know Empire called the gaming Citizen Cane moment we've heard a couple of things like that before but this really has uh a lot of emotional impact I've watched this scene we worked on it and I have a hard time watching it and you know I've been I've grown up with video games since uh you know I was a kid five five years old so really grown up with them and maybe it's because I have two daughters of my own that I look at this and it's really hard to watch but I think it's more than that I think there's empathy I think we can all relate to the characters here there we can all relate to loss and and I

wondered that um if we looked at this 20 years ago if we did the same thing 20 years ago and it looked you know in the old 8bit kind of graphic stuff could we get the same level of empathy so it's the same scene right it's the same writing and and that's how we would have done it and that's the same music but I'm not crying why right it's it's just it's an interesting thing to look at because it's not just a context it really is something else so these games now are

getting incredibly complicated and they're very challenging just like making anything great right it's about the quality of it quality is never an accident it's always a result of high intention sincere effort intelligent Direction and skillful execution represents the wise choice of many Alternatives and this particular title is a great example of of skillful execution right at every level it's got relatable characters outstanding casting Direction and pacing and that's really important doesn't matter what kind of entertainment we're talking about it's using performance capture which is really really important they've made smart visual choices some of the choices are related to the the power of the machine and what can be achieved but some of it really is about um the choice just in terms of the style that they want to show there's Amazing Music and Sound design and the truth is that naughty dog is one of the most advanced uh developers in the planet in terms of

the way they think about collaboration they handed us one of their best tionally impactful scenes to deliver for them and we had the same OT emotional investment in the game as they did so game and movies at least in terms of digital characters in my opinion fall into three different categories realistic hyperreal and animated and these are kind of you know some game examples that that are falling into these categories this is not an opinion of of where success lies it's just an opinion of what they're striving to achieve um and and the category in which they um kind of fall into and for the films that really tried to do digital characters well um this is where they fell into as well so I told you I started image Works in 2003 one of the last films I worked there uh was baywolf baywolf has a significant

place in in in digital character history it's one of the few films that you know attempt to make a realistic believable character human character um in in a completely digital world but it struggled to find an audience because it struggled to find a reason to exist if you could shoot beao physically why did you need it digitally and the other issue is this right The Uncanny Valley I haven't heard this term in a little while since I'm out of film and it was a real big deal in 2003 till 2007 um but for some reason we don't really care about that much in games she's still ha of theol so we both know that but answer me did you kill would I have been able to escape her had I not

visually this this movie still has a lot of relevance there's a lot of digital character stuff we would like to do Sticky Lips for an instance um the shaders and the lighting a lot of stuff is still pretty Advanced even by today's standards the issues are that you've got a fabricated digital character bolf who's not based on anyone acting up against realistic digital double characters like Anthony Hopkins and you know Anthony Hopkins and so when you see Anthony Hopkins as a digital character you know know something's wrong if it's not exactly right and what's interesting to me is if this movie had been done completely with with characters that you didn't know were based on real life characters would you have accepted it more would it have been easier to watch maybe I still go back and watch the movie and even though I'm too close to the movie I still think it's easier and I can I can still watch bolf and don't see as creepy you know I don't feel that creepiness but I look at Anthony Hopkins and I still get the same feeling everybody else does and

uh we also were you know this digital character thing was real high focus at image works at that time and it was actually interesting because Mark was working on the image-based rendering stuff for Spider-Man which ended up being a demo for uh Playstation 3 if you guys yeah PlayStation 3 if you remember during the transition so the timing of the film it was released in November 16th in 2007 and the commentary you know obviously the reviews were mixed and the commentary is kind of um pretty clear you can go but I'm not going to read all the stuff we're already behind schedule but I'm going to tell you one down at the bottom it says Bol is Solly gorgeous at times borderline stalled piece of tolken with a joystick mythology that's a fascinating commentary joystick mythology it's referencing video games it's it's it's relating the these critics are relating this to games even though games at that time couldn't touch this visually in November 19th 3 Days Later un charted one came out much better

reception right this was a game that used a lot of similar ideas performance capture no it was using motion capture with animated faces but the idea of doing digital characters was there was there and there was a clear change a clear sort of chapter shift I started in on in PlayStation this coincidentally kind of the dots backwards theory on November 19th 2007 and that was the release day of Uncharted one but also 3 days after the release of bolf so for me personally this was a Closing one chapter right and then opening another and this was the first thing that that I personally worked on with my team a lot of who came down from image works with me hopefully I can two person job no no no three actually this is Uncharted to speak of the devil here she comes now what

Fraser Nate Drake Nathan Drake hello Harry Chloe here one of the best drivers in the business she'll take good care of us so Uncharted 2 had a very different reception right this is one of the most [Music] um uh highest rated games on PlayStation 3 and is one of the best examples of cinematic games and you can see even in the commentary again they're even saying you don't need to make a movie of this there's a game it's better than you're going to see in any movie coincidentally that same year this came out and it's interesting Metacritic score and this is not about Metacritic and I'm not a fan of Metacritic but it's a good reflection of this sort of view of the time but it's interesting to see that Avatar billion dollar franchise probably two billion dollars at this point it's an 83 Metacritic score that's a B minus isn't it fascinating but it's it's it's incredibly it's it's an incredible um

it's a landmark film visually and they made very smart choices to to avoid digital humans because again they had moved on at that point and they had found better ways to do I mean the techniques were still very similar to ones we used but the output was very different so the looking at the evolution of the digital character it's kind of funny we kind of um gave up in film we we solved the problem of the monkeys and the fantasy characters and the aliens but we kind of stopped here and we we we moved on but that's the starting point for games that's where games really picked up so around that time we started doing more Advanced Techniques in around 2007 to 2009 doing more advanced scanning techniques at PlayStation and we scanned um Dennis Hopper um which should show up here hopefully

no here you go this uh was a title that never came out but we were in 2007 2008 starting to look at this digital double technique the game was canceled not for any reason for having to do with digital doubles but it certainly was um an interesting time to even be exploring that and now we're seeing even more of that with Kevin spacy and the advanced Warfare but again we're still the verdicts out it's interesting because the whole uncanny valley thing isn't really being discussed people aren't too freaked out about it in games but the the the the technology we were using for scanning went into titles uh like Socom 4 and when we released Uncharted 3 it set a new Quality Bar for our titles and the expectation for all of our titles that needed digital characters need to be at that level and frankly Socom 4 when I was there what the developers were doing was not at that level what you see on the left is what they had completed and on the right

was when we entered in we completely revamped their characters with no topology changes no higher vertex counts same texture resolution it was just a technique change honestly um and it was a challenge to kind of convince them at the time that this was worth it but the reality was that it PlayStation 3 toward the end of that life cycle the the guy on the left just would wouldn't have wouldn't have held up so not only do we we we we have to kind of do scans but we we ended up doing we started doing fax based blend shapes uh so because their engine could support blend shapes we're able to get 60 blend shapes which is pretty good for fax although we weren't doing facial capture and we still use the same animation techniques and ended up getting you know reasonably believable characters for what would be digital doubles but not digital doubles that anybody would recognize

we started seeing neck bulge and we were able to get wrinkle maps in and we started using more advanced cinematic techniques um we also started characterizing started pushing away from doing exact scans toward kind of hyper stylist stylistically moving to get to more uh palatable visual characters we we we also played at this time LA Noir had come out played with some of the ideas of the video based scanning which uh yeah secretary looked like it was going to be very disruptive at the time this is also a title that didn't come out we did some testing the eyeline stuff really doesn't work and it works in a sense if you have a viewer and you're using a fourth wall where it's coming talking directly to you it works because the ey line is at you but beyond that it really doesn't work but it was interesting just to see some of the tests we had done some tests to see how it would hold up with characters like Kratos could we put Kratos in a game this was

not for a game it was just testing to see could it be done this is not anything other than an actor in makeup but it was interesting to see what it would do and again this is PlayStation 3 so at the end of PlayStation 3 we knew that PlayStation 4 was going to be a higher fidelia expectation we were going to have to do a lot more and we need a lot better technology and the costs were rising and we need to create efficiency so I I contacted mark saying we need better facial capture Solutions in what we have uh and everybody all the studios are talking about building one or using off the-shelf stuff which doesn't seem to give us the results we're looking for so I called him you know he was at WETA at the time at least I thought when I contacted him he he had actually left WETA a week earlier and started a laboratory for digital characters working on AI and um we talked about building a uh commercial partnership immediately one week after he started we started talking doing a commercial partnership in working on uh the

technology advancements all the stuff we'd learned over the years building the um technology and character advancements the text the techniques all the stuff that we'd learned over the years using the performance capture techniques and the scanning techniques to build a system which we call parrot and this was literally our first test out of this system this is only going to take all beginning incision at upper abdomen exposed internal organs seem to produce an UltraViolet glow this was again not for a title it was just testing to get the tech to to see how far we could go with it when I asked Marone or something whatching to in your opinion is the next gen in facial he's been doing this since 1997 so if anybody knows about this Mark knows about it so I said what are you looking at next where everybody else is

looking at is a lot of stuff we talked about a decade ago and honestly did on movies a decade ago so if I'm going to build something and we want to spend a lot of money and time on it we want to do something that's more advanced and can honestly break some of the barriers that we we hit and interestingly enough he told me that at that time he said I'm actually working on stuff for games I'm more interested in the game Space we kind of solved King Kong and we did Avatar and that was really cool but I'm really interested in actually building stuff behind the face and using AI um to to actually drive facial animation so that's where we're going to lead to really quickly though most recently the stuff that we built we revamped all of our photog gometric scanners to build some incredibly High detailed scans some of the highest you can possibly get out of photo photo photometric scans for facial right now and we're also working on bodies and it's it was was doing done with a partnership with Sony electronics

because we have cameras we have a really good camera division that we didn't have access to at the time and we had to build custom software which is kind of a pain in pain in the ass but we did that and it paid off and we were able to immediately put that work into the order and so the order is a culmination of a lot of techniques some of the of all of the years of these things in one place using the best performance capture techniques using the best scanning techniques um and we still had stuff to learn but I think if you look at what ready at dawn had achieved coming from a PSP developer to a PS4 developer with no generation knowledge uh play with whatever or fancy but do it's remarkable and they're actually setting the bar in the industry terms of visuals they're not getting recognized I believe as fairly as they should and there's

certainly an interesting commentary going on that's kind of a little bit of a rebound effect they're getting too close to film and now games are sort of pushing back saying that's too film likee and it's sort of ironic because games have been a safe haven for Innovation especially in the digital character Space games like LA Noir even though the flaw there were flaws and the disconnect between the body and the eyes were you know frankly very very difficult to look at but people really were okay and the critics were thrilled and the issue is that we are getting too close to film and for some reason now we're getting pushed back it's interesting so now I'm moving on and then we're going to transition to Mars I'm going to close out with our work on Uncharted 4 I don't have a lot to show you here because we're still working on it um but when we go back to the types of techniques we were using on bolf this was very interesting we did a a base digital scan of Ray Winston who played

bolf that couldn't be really utilized in any way it was only the scan was only used for reference and only reference for you know how his face moved and and ultimately map drove the poses that we would actually model from so it's a remarked difference from today for where we are with Uncharted 4 We're actually scanning an actor that's got similar facial structure to the character we want to look at it's Nathan it's not an actor it's not based on an actor but these facial shapes have resemblances to um the character we want and we're using these blend shapes to to influence the rig directly which thereby make the animation a lot more um much higher Fidelity and a lot higher accuracy and also um just in in you know much better overall here's an interesting example this is the blend shapes we used about 70 on Socom and this is about 180 on the

order and we're about a th000 on Uncharted Four so you can see the Fidelity now how all those blend shapes they won't all be used in the engine they're going to drive a rig but we're using what we call Micro Expressions to Define all these really really core shapes that are really important to get the kind of fidelity we need and you know the visual Target this is Bea wolf in 2007 you know and this is Uncharted 4 in real time in 2015 and this is the rig just showing you kind of a range of motion I can't really show you any of the sols or animations we are using all of the techniques the scanning techniques the motion capture techniques all the animation techniques that we've learned along the way

and finally right now our biggest challenge in this area is a lack of domain experts for me you know I've worked in the industry between games and film for um for over a decade and I know very few people that can build these kinds of systems marks one of the one of the few um the scal and complexity is enormous and and the artists and Tech artists are highly specialized they have to be you know completely understand how the face moves and there's no training for that really and there's no schools that are focusing on that so it's really hard to find blend shaped artists it's really hard to find animators with that Focus we need better technology and automation otherwise it's going to be impossible to scale so that's where our Focus are you know um and we just need more face experts so I leave you with a quote the reasonable man adapts himself to the world unreasonable one persists in trying to adapt the world to himself

thereby all progress depends on the unreasonable man I'm going to now pass it on to the most unreasonable man I know Mark sear who has some really incredible technology to show you in terms of again this is not being utilized right now but this is stuff that is going to be coming and how when we start figuring out how to use this stuff it's going to change things so I'm going to drop a few emails here um that's mine feel free to contact me I'm sorry to and you know wasn't highly technical but if you have any technical questions I'll happy to answer them along with K michod who is the one of the most important influencers on on the art side if not the most important art you know influencer on titles like the order and Uncharted 4 and Tom Bland who's been the animation director on our side for these Titles since Uncharted too so these are the guys to hit up and ask questions about thank you I'm going to pass it on to Mark okay you [Applause]

I'll just uh check the audio works now so it's coming through this this Jack here yeah great thank you okay um so thank you very much uh Mike thank you for the unreasonable um characterization that's fine yeah um so this is I'm going to take that to an extreme now so um so one of the uh one of the things which interested me about digital characters was um so this is going to be a slightly different tack on on how to on the whole creating human aspect this is really a bit more about creating Behavior so how do you create the inner life of a character now the the thing is which Mike's talked about in in a lot of detail is the face and the face is our primary means of communication expression so it's critical that you get the Expressions right because that communicates the information that's telling you what the character's mental state is and the Face's identity has to be right so that's like the um an Hopkins example that faces it's your

signature it's very important to get right so this is a film that uh we made in 1999 a short film it was in the cigar electronic theater and for this I wanted to digitally age an old woman this is 10 years before Benjamin Button and what we're doing here is is sort of creating a character it's a digital character that we want to basically make you think about what she's thinking so she's just supposed to be remembering things with trying to make you think about the mental world of a character that doesn't exist and that really got me thinking about you know how that's done and so a lot of the films that I worked on like especially King Kong and Avatar were about how do you make a digital character appear like a conscious thinking being and you know that's basically you have to pay a lot of attention to what's going on in the face the most subtle things like characters like King Tong don't speak so the emotional weight of that is coming through most subtle movements and that

really got me thinking about okay well what are the Symphonies of our facial movement how they composed because that's the music of the face which is coming across and our Behavior what's expressed on the face is actually the result of very complex multiscale system it depends on how you feel at the time whether you had lunch what your thoughts are what your history is how you EXP Express that and then somebody else's reaction so uh the whole believability of a character in a scene depends on the context is does everything tie together so um to be unreasonable thought can we actually model this so can we actually create models which integrate these different elements and so what we're doing in my lab is uh creating autonomous animation so we're basically simulating faces and simulating brains and putting these together and we're trying to do it in real time and but the

the motivation there is that for a computer to really simulate a human face Behavior you have to simulate its brain and body too so here's an overview of of our architecture so we're basically creating a system to build artificial nervous systems so what it does is it interconnects sensors and effectors with um expressive displays but through a brain and so the brain will do multimodal Association it evaluates things and Robotics this would create mve M we're really spe working on how does the computer convey how it feels how does it express itself and then how does that Connect into a loop to people that it's interacting with so we're basically I set up this lab just when Mike had called to do exactly this because like this is where I I wanted to go I really ultimately I want to create a conscious computer might be impossible but it's a really fun challenge so really um so that's the unreasonable bit so so we're um what we're doing is we're combining bioengineering neuroscience Ing and computer graphics and putting these things together to basically

embody models of the brain with faces that you can interact with there's two reasons for this one of it to basic explore the retical models of brain function the other thing is is a human computer interface technology and so games fits somewhere in between that and so the way we look at the faces is as an instrument and so what we're doing is really building a synthesizer and um so we want to look at what's the information that we're conveying you know how do how does the facial information get across if you're really boiling it down and you know ultimately it's muscle Activation so those are the notes the musical notes of the face if you like so um Paulman in the' 70s basically came out for a system to sort of encode these different muscle groups so that's been used a lot in in film and games and so one of the things though is it's really how to generate these shapes so this is a particular thing I was looking at is is um creating a biomechanical model at the face so that we could order medically generate these blend shapes so what you're looking at now is a physics based

simulation of a human face it's a finite element model so it's my face because I was a guinea pig but basically we now digitally activate these muscles and it's creating the shapes these aren't being sculpted they're being actually created by a physics simulation now what we're going to do is just start activating all the muscles and randomly and the physics simulation is doing all the flesh movement it's doing all the Collision detection and this only works if you get the aty really in a lot of detail and really right with all the multiple complex material properties and sliding properties of skin and so forth so it's uh it's not an easy job but this was this was sort of the stuff that I was doing in the past and what I really wanted to do was to create the control system for this so what we're really doing in my lab is essentially creating nerve animation so how do how do we create animation which animates the nerves which Drive the face so if we do this and I believe that it's ultimately going to be necessary to to do this because this is what conveys the subtlety of what a face can really do so

your PE people have strokes for example half a face will be paralyzed but they'll smile naturally because the emotional system driving the face is different from the cognitive system so there's a whole bunch of things like that so the meat of the stuff that we're doing is is what we call brain-based Computing so it's neurobehavioral models so how do you create a digital character to really want to do something it has to have its own inner life so for example so what would doing is essentially neurobehavioral animation so if my blood sugar level is low I'll get hungry and then I'll start seeking food and that motivates my behavior if I get blocked in getting my food I'll start getting angry and so forth so these are core models it's like we're trying to model what motivates a human and actually put that in a computer and at the same time we're also doing models of the emotional systems attention systems and also learning how do these systems learn because we want them to learn in the same way that people do and also how do they interact socially

so we build all kinds of different models like for example this is a model of um basil ganglia which is a crocodile as a giant basil ganglia so this basically does habitual behavior and learning reinforcement learning things like that and we also want to look at what's actually driving the emotion so we actually been building models of the emotional systems now they're very um controversial but we look at the physiology so we build physiology models of these emotional systems and then we start looking into like how how does the brain learn and you know this is a sort of simple example of the brain self organizes a lot you just expose it to new information it just learns new stuff so I hit a Go Button here's a bunch of random colors it's just self organizing that could be words it could be colors that you're seeing part of your visual system learning or it could be words that you're learning for the first time and that sort of stuff and you buil hierarchies of these types of things and you start getting essentially Nature's equivalent of some of the things which like Google we doing with things like deep learning and so forth but it's all the same type of stuff we're taking a biolog iCal approach so we also look at

memory you know how does a computer remember its experiences so we're building basically models of the memory system so this is just a an old test which is basically the computer imagining a sequence of numbers that knows what's going to happen next so so this is like a computer imagining something so then we start you know like looking around so there's been lots of interesting computational neuroscience and cognitive science models out there what happens if we put them together and actually embody them and how do you even do that and so to do that we've basically built essentially a Lego system it's where it's called it's really a brain Lego system but because of trademark stuff we've basically called it brain language and what it does is it combines neural networks sensors Graphics all kinds of stuff so you can just plug it all together and build very complex systems so I'll show you a really simple system so what we're looking at here is a very simple circuit down the right this is like the knee reflexor circuit so basically we'll sort of connect a few things together whack the knee and so we're simulating the spinal cord circuit which gives you a

knee reflex and then you can kind of see all the parameters so what our system does is it's a continuously integrating system so you can basically see what each neuron is doing you know live and you can investigate that while the model is running now basically to give you an example of you know how to put all these things together we've um we've started off as simple as we can on a human which is a baby so this is called Baby X and um so baby x here a real baby so this is kind of like a baby capture now here's a real baby for the very first having solid food for the very first time so the baby's having a fundamentally new experience and it gets a wo moment now we want the computer to do that too so anyway you start basically Building Up 3D models of the baby had to do that when my wife was asleep so for the scanner cuz I would have got in trouble and um so then we start building a biomechanical model of the baby and activate the biomechanical babies

muscles because you can't get real babies to do what you want but you can simulate them and then we start putting in a bunch of the different types of brain systems which are driving the various aspects of the baby so I just um C out of the presentation for a second and fire up a a quick demo here um now what is what's actually happening now is the is the um is the uh the brain is compiling itself it's literally compiling at runtime so it's kind of like a python or something like that but it's basically building all the connections and things like that so uh baby here she is okay so she can see in here so if I make a loud noise she'll get a fright we can kind of um zoom in let's see if I do this we just change all the view Keys we can move the light around like that so you know she's watching me and reacting to me we can kind of take a look at what she can actually see so you can see me here so I'm looking at her and I smile at her

hey s now she's not copying my expression she's actually getting feel-good chemicals and then smiling now she's constantly learning so she's learning online so if I do like you've heard of Pavlov's dog so if I do I can and I'll see if we can hear this from the microphone can everybody hear that so okay that's that's called a conditioned stimulus so now I'm going to appear it and I'm going to go okay let's see and hey sweetheart hey you like the Bell yeah the Bell is awesome okay so now I've kind of paired that and see what happens now when I do the Bell she's sort of learning her context so she's basically learning online all the time okay so then let's jump through to a few other things and so you can see and what what we're looking at up here is basically a bunch of different neurom modulators so she's got oxytocin she's got dopamine she's got all these different types of things driving here now if I abandon here I'm just going to sort of be a bit craw here and sort of

like hide from here now she's going to sort of start wondering where I've gone she knows I've gone off somewhere to the to the uh left but if you start watching where the arrow pointer is she's going to start getting stressed out we're actually sitting off a stress system so by the way all of our animations are completely generated by neural network circuits and those are driving the particular muscle so she's getting actually upset now we've got cortisol's gone up as she's stressed out hey swe it's okay and she's listening to my voice and everything so it's hey don't cry it's okay okay so now we're going to be a bit more cruel and actually remove her face so um and so you can see what's actually driving behind it so literally what you're looking at now are the circuits actually driving the muscles driving the face so you can see the eyes you can see the neural networks you can see the nuclei and the brain stem which are driving the eye muscles and as I'm moving around it's just kind of looking so now I'm going to do is kind of like uh whoops um change all my view keys so

so we've got this whole model of the brain you can go around with put the things in the appropriate anatomical place so the idea here is we put our neural networks in the place where they really are and now we're going to sort of we can zoom into the brain stem do stuff we can even do other sorts of things here so I because the models based on these computational Neuroscience models we can do all kinds of things like if I do this I'm actually gonna I don't know if you can see this green light pumping up there but what I'm doing is I'm actually like overriding the sort of there's a there's a whole thing called a phma cortical Loop which is a cortex sends stuff down to the phalus and that's a feedback loop and if that builds up you actually do something and if it shuts down you don't so what we're kind of simulating now is actually Huntington's disease so this is a case where people basically all the muscles are moving too much we can do the opposite and kill off those cells and now it shuts down this feedback loop this is Parkinson's Disease people want to move but they can't but we can build these things into the model and now we come out of the model and so we can even I'll even show you the effect externally so I'm now going to bump up the dopamine

so watch your pupils so this is essentially like putting the baby on amphetamines so it's like you can do all these types of things because but it actually drives the behavior the model it changes its learning abilities and all kinds of things now let's just jump to another thing and we've been teaching her to read so I'm going to show you this is her little first words book so um I'm from New Zealand so I'm going to show you a sheep okay right okay sweethart what's this what's this she good girl okay and we see if she can read a word okay what's this what's this say what's that what's that say hey come on what's that hey look at me pay attention pay attention it might be the lighting yeah that could I'll show her some some other word okay hey what's this what's

this CL good girl okay so she's hearing me and getting rewards and all kinds of things so she's basically can hey what's this look good girl okay now I'll show you some things which I can't sort of which take longer to train which are these are things which might take an hour or something to train so this is learning to imitate a face so what What's Happening Here is the baby is now going to just do random facial movement and I'm copying here this is what parents do to kids so we're basically trying to build a computer which you can teach like a child now as time passes the baby starts learning those associations and now she's copying this good girl good girl yeah Clipper okay let's do that again okay you ready so we're comb all the all the expressions are generated through neural nks there's no it's everything synthetic now this is what we've done is we've connected her brain to the G bat and the game pong she

doesn't know this this is just like a baby sees its arm for the first time type thing flowing around but she gets a reward every time she hits the bat so she's actually going to teach herself to play Pong right so over time this takes about an hour now she's got mental control over the bat so that's a sort of example of the of the baby so it's basically creating a character which is embedded in the world which is learning from the world which is reacting to it so if we kind of um move forward a little hang on yeah so this car so I'll just show you an adult model that we're working on so we're also we don't just build babies we're also building adults so this is called Zea so she's a working progress so I'm just going to fire her up um now let's See's this okay so Zea's basically first job is going to be in the airport as a um it's not doesn't require too much

intelligence but she's going to be like cheick you in and stuff so um but anyway uh so but she's just uh she's just being built at the moment so I'll just I'll just show you a bit of bit of stuff so you see where we're at with it so this she is I don't know if you can see it yep so um and then we can uh let's see s of zoom in on there let's see wrong button so she's if we look at the types of things she's got all kinds of controls now I'm just going to find my mouse it's gone oh [ __ ] I've lost my mouse oh there it is it's over there this is weird Okay this oh yeah I bet uh there and then I was trying to get this control Pan the mouse I can't see on the um there we go okay so she can do various behaviors so we just um get her saying something hello how can I help

you thank you so all the time she's reac she's actually watching and interacting her these are sort of canned things that she says I understand yes no now as she's doing all this sort of stuff what we can do is we can start like messing around with it you know you can change your expression you know changes the whole sort of feel of what she's doing if I you're a bit disgusted you know sort of so we can do all these things live now this is just literally changing sort of expressions and stuff like that one of the other things that we can do is we can now start activating her own sort of neural emotional neural network so if we start doing things like this if I start say something I'll just wait for her to finish you're welcome okay I don't [Music] know okay there we go so now we now start see something sort of disappointing happens to her so these

are all being generated in real time and they'll never be the same and basically you know these things still still so she starts feeling bad about it until she's just completely shuts down he's completely depressed and now so and now we can do other sorts of things like um you know say something's happening she's not sure how she feels about that and then you know starts getting sort of irritated by it so these these are all sorts of things which were being generated live now we can kind of O overlay everything hello you know so she might not be how can I help you you know and all these different sorts of States so um and because we're in San Francisco at the game developers conference we got some cool glasses for her so she looks like a game developer so uh there she is and so she can hear me and do various sorts of things so anyway um pretty much uh what I will then so basically that's a sort of

system overview and you know it's thanks to a fantastic team um that have got that we can put all these things together so we're working on lots of different Technologies uh which are all going to start coming together this year so we've got a a lot of goodies to to show probably in about a year so we some of this stuff is going to be um out in the public in a few months so uh yes thanks very much and um yeah [Music] I think somehow we miraculously made it on top I went through giv all the hiccups sorry about the hiccups guys thank you guys for coming um I guess we have like three minutes you guys anybody have any questions no awesome okay thanks [Applause] okay