Beyond Bots: Making Machine Learning Accessible and Useful

[Music] [Applause] thank you in 1986 fred brooks proclaimed that there would be no single development that would deliver one order of magnitude improvement in a decade in software engineering fast forward 30 years this slide was presented by chelsea finn at the dprl boot camp at uc berkeley last summer and what this slide is showing is that every generation of machine learning algorithm is a 10x improvement on the last that means that a problem that was taken 100 million steps just a few years ago is now take it's now been able to be learned in real time on a robot in about 20 minutes and we're seeing these improvements not every 10 years but every 6 to 12 months so my hypothesis is if reinforcement

learning along with reward functions if we can apply that to software development that will result in a multi-magnitude improvement in productivity reliability and in simplicity and to use video game ai as proof so today i want to take you through a couple of stages i want to talk about kind of a brief give you a brief taste of reinforcement learning and then i want to share the results of the research that i've been working on into this space first a little bit about my background i started in the 1980s on the commodore 64. i'm dyslexic dysgraphic i also have adhd and i really struggled at school and traditional learning but i had this aptitude to be able to understand a computer and i had this passion for video games and so while i was failing at school i was winning contracts to make video games and i left school at 15 and to join the industry over the next

25 years i worked my way up and grew with as the industry grew working in multi-disciplines so in engineering production and creative direction and at the end i was an executive producer with xbox up in microsoft over the last couple of years i've been working in a couple of different areas so on the on the one hand i'm working with neuroscientists and people at the bleeding edge of academic research on like projects that take three to five years to bring to market from neuroscience and i also work with a a guy called nils liar who is um he is one of the fathers of video on the internet um and he's also a vc so i work with him building his companies and both of those kind of pull into this research that i do into reinforcement learning which i started about three years ago

so there's a problem for me with learning reinforcement learning if you look at this equation at the bottom here i literally cannot read that and to make to make matters worse my big hack for reading complex materials is to use voice to text software but math notation breaks that software so i've i've i have a couple of kind of tricks that i've used so the first thing and most important thing for me is building a mental model and the second thing i would say is get hands-on and get hands-on as quickly as you can and i say don't waste your time with all these online courses that um that you see everywhere because you're going to be writing gradient descent algorithms in python and that really doesn't offer you a lot of advantage other things if you're dyslexic i recommend doing podcast videos i also have learned to skim read papers

the one online course place that i did find is machine learningmastery.com that is built for engineers who want to learn machine learning so it's high on the engineering and low on the math background so if we think about this model here this is the classic reinforcement model that was created by sutton umbalto in the 70s and 80s um and the picture on the the left there is from the atari deep mined paper um so to give you a little kind of instinct or try and build your mental model of how this is working so the environment in this case are raw pixels coming in and they go into a convolutional neural network and if you think of that as compressing the pictures so the problem space becomes much less than having to work on the ro the raw pixels and then there's an action space so in the case of breakout you can you can go left you can go right

or you can just do nothing the third step is delayed discounting so when it gets rewarded the end of the game or a reward of a point it steps back in time and gives a discount of that to those previous steps and then each of those each individual frame and that reward and the step that it took the action that it took they go into something called an experience replay buffer and this is uh there's a million uh this is a million big right and uh it randomly samples that and that is really the core of the deep qn algorithm it does no planning it's not thinking about anything in the future but it is able to find this advanced strategy of working its way around the the side of the breakout one thing just to talk about is the exploration versus exploitation

trade-off so exploration is just randomly searching inside the environment exploitation is taking the the the feed the signal that has the highest highest future reward and there's a trade-off there and and algorithms have to work in different ways to to explore the space in an efficient way for the deep qn paper they had a very naive approach they just for the first million steps were totally random and then they reduce that so that over time two percent of the choices would be random but i wanted to step back to dyslexia and neuroscience and automaticity because it turns out that us as humans have this trade-off as well and different different like sets of people will explore problems in a different way so there's a researcher in the uk called rod nicholson and he researched dyslexics and he found that for them to

learn things through rote memorization and automaticity it took n times the square root of n so if it takes 100 times 100 repetitions for a normal student to learn the math times table it would take a thousand repetitions but the trade-off there is that the the dyslexic will find we'll be looking at more different solutions in life so i recommend you do keep them on your team in terms of the landscape so if you want to get your hands dirty and start playing with reinforcement learning today these are three of the many players but i think these are the top ones to look at so first deep mind they write some really great papers and do a lot of research that you're probably familiar with um the frameworks that they have starcraft 2 if you're a starcraft player that's maybe a good place to start and deepmind lab but they're not so good at publishing algorithms

they did publish a one version of the dqn openai that's where they excel so their baselines has nine world-class algorithms that you can use it's all open source and they have open ai open air gym framework that has classic control at atari mojoco and they also built universe on top of that that allows you to play games like gta or flash games or to learn against those games and the the new kid on the block is unity and danny's going to talk about this and so they launched ml agents last year and so they already have a couple of algorithm algorithms there and they have at least 10 different sample environments that you can use the first thing that i did with machine learning and this is going back two or three years was i i was playing around with a soccer some soccer ai and i took a high cost function and i sampled you know this over 24 hours this this

this function i put it into a linear regression um so i had a mod i learned this and i put the model back into the game and i was saying that this was choosing the same outcome as the function 90 to 95 of the time and so that gave me enough kind of confidence to carry on looking at this space and today i've kind of built up my own system i call it analog i started before unity had um created their system and what my system does is it bridges between open ai's baselines and unity and it gives me this nice ui so that it kind of like being adhd i remember what i was working on when i go back and just finish the training run so next i want to talk about the research that i've been doing and share some of the results the the first area i'm going to share here is using reinforcement learning for locomotion so if you think why we might want to do this well locomotion is very hard to get right and and it's very

expensive is a big part of our budgets and it'd be good to have an extra tool set to be able to do that so the video here is from deepmind this is a character that's learnt completely from scratch with the reinforcement algorithm and the one in the bottom left from the same paper and this one is from the open ai where they took that algorithm and they improved on it using something called parameter noise and the photo in the bottom left there is um this is from a talk from a couple of gdcs ago from ben fodie he was talking about physics and making against his physics and he came up with this idea to use a midi controller to tune all the parameters that you need to do just to do a 2d game that he was doing um and so be good if we could um you know put that to rest and give give people the ability to to learn those things so in terms of the

methodology um i'm building the architecture on top of my analog i'm using a couple of algorithms actor and ddpg and i went through some stages so initially i wanted to use my naivety of not really knowing unity not really knowing the physics system inside unity and i'm taking the baselines of the mojoko which is the world-class gold standard physics engine and with majoko it's the the simulations are so strong that if you train a model on majoka you can put it onto a real-life robot and it will reproduce that and then i went through stages of progressively kind of looking more into the research to improve my outcomes so initially things worked very well i was able to get this little worm moving and i was able to create recreate the the and from the joko um but to give you a sense of the the time that took and in

terms of the experimentation so there's 44 different experiments just for the ants and that's when over 86 million training steps um so to get things you know when i plugged it in some other characters um the hopper or the humanoid um i started to see that i'm getting some some results but they're they're kind of strange right they don't look like you're seeing in the papers um and so to address that i dug into the research some more and one of the most important things i found was that a lot of the finesse comes from tuning the reward functions and also the termination function so if you see at the top we've got the hopper and it will terminate the hopper so say it's failed if the angle goes over you know more than 20 degrees either way it also has things like a cost function so it's kind of penalizing the the the ai for um over-exertion um and the one on

the bottom is the open uh sorry the deep mind reward function for the for the walker um and that has things that we wanted for being upright and and the the height so there i was able to get uh reproduce the the benchmarks the the ones we're seeing on the left are on the top that's about a million steps and the one on the right is about three million um and then the final stage i went through is comparing the results with the mojoko and i was able to see that the training the training time was within about five percent and the steps were within about 10 percent it does train differently which is it was kind of nice because the algorithm gets around problems that you may have in the physics implementation so in conclusion i've been able to reproduce the hopper and walker inside the commercial game engine i

expect that those deep mind results so having more elaborate obstacles will be reproducible and this is definitely worthy of more study in terms of future work there's a lot of new algorithms that i'd like to look at as well as mix in motion capture data with reward functions in terms of the total scope i added this up last night so everything i did in that space was 380 experiments and that's about 750 million steps so the second area that i'm going to talk about is learning and advanced control in 99 training steps so the the reason to to do this is that often the problems that we we want to solve are pretty simple right they may not have the same problem space as learning locomotion and we might may want to retrain this very quickly so this was an exercise to see how how

far um how how much i could optimize it and what were the steps i need to do to be able to optimize it the problem space i created was teaching the ai to hold fire so it's got two actions it can uh it can have fire or not hol you know not not pressed fire but it's got a whole fire over a number an appropriate number of frames to hit the target an appropriate distance and so we know from usability that like people like my wife have struggled learning this control um so i thought it'd be fun to train an ai um the the way to get this down is i i use something called a grid search so this is taking the number of layers in the neural network and the layer size and just running iteration over iteration until i'm seeing a trend or a convergence on one combination and then taking in another variable so in this case the number the number of stacks to the number of observations um previous observations that it takes in a row

and then doing a search to break that down into 99 steps so i was very surprised i was able to get it to train this quickly one caveat is that there the the solution to the problem is probably correlated to the design of the network and there isn't a lot of resign a lot of research over that so we know that this there's a correlation but we don't necessarily know okay for this problem what network should we do the third area i want to present today is the case for real time server of real reinforcement learning so let's say you want to publish a game today you have your python backend that's doing your reinforcement learning could you publish that using some of the technology that we use in multiplayer games here so my inspiration was um for the visual reaction delay so on average we have a 250 millisecond

delay between the visual stimulus and being able to react my method was to train a friend 20 frames a second with a 10 step delay so it's 200 milliseconds delay and what i found is that this slowed down learning by about 30 percent so it's not a silver bullet but it is worth looking at if you want to launch the day and get a bring a product to market okay so that concludes my talk and i'm going to hand over to wolf and please come find me i love talking about this stuff thank you very much all right that's been my talk that's been my talk it's been great talking to you all we'll move on uh hi uh i'm wolf uh and i'm from google and uh the reason why you're listening to me is because i am the technical lead for developer relations for tensorflow uh tensorflow is a giant framework that does machine learning and a bunch of other stuff it's built by the brain team at google and i'm not going to talk to you about

tensorflow much at all because honestly there's no way i'm going to teach it to you all in 20 minutes um but um we are at this sort of uh oh and i should go back and say that i'm also a game developer i've been a game developer i actually worked on applying machine learning to aaa games uh like a decade ago um it's tricky it's interesting and what i wanted to share with you today here oh well sorry uh and as we alluded to before you know we're at this sort of magical time in machine learning where things that we that a decade ago that we thought we were going to be able to do we just can't uh or sorry we finally can do things like self-driving cars and uh obviously you know our friends in london uh having success playing games we have we're beginning to apply machine learning and deep learning to medical data which is pretty amazing and actually getting quite uh interesting results uh that's uh looking at retina scans and of course making little stompy robots which you know everybody likes stompy robots

that's he's in our lab but in this talk uh as game developer i want to take you on a tour of a bunch of things that i find interesting in machine learning ways to make your the process of making games more interesting more exciting more efficient but i'm not really going to talk about npcs because npcs are cool but you know it's a different problem so let's start with part one let's talk about statistics so your players are all out there creating data and in that data you are finding out about the kinds of things they like and they don't like to do and you should be taking advantage of that you should be ah there we go you should be taking advantage of that so you know is your player going to quit today is your player going to spend today these are all sort of time series models you can use uh you can learn to do anomaly detection with cheating you can learn about uh check if they're grieving you can find out if they bought x are they going to buy why these are all sorts of things you should be doing and you know talking about sentiment analysis

you know this is a this is a well studied problem learning you know learning about your community and learning the kinds of things that they're saying uh and trying to figure out if they're saying the right things or things that maybe you don't want so much this can make your game a more interesting experience and more uh pleasant experience for your users you should be totally doing this there's open source models where you can try this stuff out um it's a lot of fun and you know you can set up a service to do things like sentiment analysis or or time series prediction and make your game more efficient but i want to talk about things that would affect kind of the look and the feel of the game directly that the player interacts with rather than the sort of subtle things uh how many of you know about style transfer yeah some of you uh so style transfer is uh the idea that you take the semantic structure of an image like this picture of a food dog i took in japan and uh uh a model trained on like a famous uh piece of art like hokusai's

wave and you put them together and you get sort of hocus-sized food dog this is kind of a neat trick and it comes out of the research we did into finding out how convolutional networks actually do their work so you find out that the beginning level uh uh layers on a network can find you know like uh small features like lines and circles and things and then as you go down you see textures and eventually it's detecting or activating around groups of of of features like dog faces and things like that and with that you can do things like style transfer you can do deep dream and i'm not going to show you pictures of deep dream because they're really trippy but you can look it up on the internet that's where you like imagine dogs on everything there's actually a really great paper we published recently uh on distill.pub about how to understand what's going on inside convolutional networks it's totally fun i recommend checking it out style transfer is the kind of thing that you can actually do with a good gpu the frame rate um in this example we're actually blending between multiple

kinds of uh styles at the same time and you can imagine as a game developer using style transfer to uh like bring up that emotional punch that you need in your game you know by by beginning to add one of these effects based on art that maybe your artist drew uh to get this so we're blending between like pastelly and wooden i mean think about the ways you can do this it's pretty neat and of course there's other image transformations like cyclogan where this is from the berkeley ai research this isn't uh and there's you can go try it out and here you can do transformations between uh things like you know horses to zebras and apply other kinds of style the neat thing about this is it doesn't need paired images to do it so uh we had a student at the jeju ml camp in korea that we did last year she was doing uh image simplification so what she's really doing is converting like pencils to inks without actually having an entire corpus of inks to train on she just had some inked things and some

some pencils and then she was able using a variant of cycle again to do an okay job of kind of inking in this picture it's nothing that your artist would probably be completely happy with but it's something you can do in a hurry if you want to try something out so let's talk about generators speaking of generated images text generators are fun um they're just an example of the kinds of things you can generate um rnn generated text it's been around for a while it's kind of fun you predict sequences of text andre carpathi has a just wonderful blog post about the unreasonable effectiveness of recurrent neural networks um uh there's a guy some folks uh got together uh it's called robo-rosewater and they began they took all the cards in magic the gathering and poured them into an rnn and then began predicting more uh cards and on a per character basis not on a per word basis so sometimes it's sort of nonsense but as you train the network it becomes more and more plausible and with

the thousands upon thousands of magic the gathering cards you eventually get into kind of fun stuff uh there's they actually made an entire draft deck out of uh rnn generated things only about 40 of the cards are playable according to this blog post but nonetheless everybody's really excited to try out these things um and i actually just for fun i did this how many of you played dominion i did this with dominion dominion doesn't have nearly as many cards so you run the risk of overfitting pretty badly but i ended up generating this totally ludicrous night card uh or dame card where uh you reveal the top three cards of your deck you look through your discard pile then you trash a car there's not a treasure and then you look through your discard pile again because the network is just in this little like loop and then eventually you trash a card in the trash costing three to six and then you gain one of them so this is a very silly card but the thing to think about is you know if you have a lot of content pouring it into a giant hopper and sort of shaking it around and having it sort of generate things back to you can give you perspective on ideas that you haven't had before you know the fact

they can make an entire draft deck out of completely uh novel magic the gathering cards is pretty playable um with hard work and editing means that you know this is a tool this is a little thing generating a way you as an artist are still doing all kinds of work but uh uh you know you have this this this helper that's making you uh efficient and creative so let's talk about image transformation so this is a task called super resolution um and it's it's a an area of study and basically you start with the uh high resolution photos on the left you down sample them and then you learn the mapping between the down sample pixelated images to the high resolution images and then at any point you can hand a low resolution photo and it will generate a high resolution photo for you um which is crazy but it totally works and uh not all of them are perfectly convincing but uh you can imagine handing this off to your pixel artists and them having

fun you can imagine heading it to your players and having them sort of poke around with some pixels and generating art that's in style into your game things like this uh neural editing is a really interesting task and there's a lot of different kinds of ways of doing it and once you have one of these models you're starting to do this adversarial uh training stuff you can start generating things so this is uh there's a japanese uh engineer uh started took a giant database full of japanese pop star faces and then just began generating more pop star phases which again can be another way of sort of shuffling your data around and trying to find something new and interesting and and uh give you ideas and now we're getting into tasks that are a little bit more like what uh your artist might do this is infill painting so in infill painting you basically give part of an image to a trained network and it tries to fill in what's missing uh without any context uh in this uh thing we have in the upper right hand

corner that's a human trying to fill in the the painting lower left hand corner we're using kind of an average loss on a network and you get kind of an average result um uh and you know if there were a car in there sometimes cars come out kind of brown because that's the average color of a car no car very few cars are brown but if you average them all together but if you use adversarial loss uh where you have another piece of your network that's saying this picture is not true and and it makes that as you know this is not what cars look like you eventually end up with these kind of much more crisp things and again this is this is you know an artist can totally do this work but this might be a way for artists to be faster or more efficient or get ideas okay now let's talk about latent spaces uh and here i'm going to be talking about uh project magenta uh most of their work project magenta is a team at google that works on the intersection

between art and machine learning and i just really love their stuff variational auto encoders what are these well you take an image and you encode it into some latent vector z and then you train another decoder that takes this sort of compressed format this just a series of floating point numbers and you expand it out to make the image back again and what you're trying to do is reduce the loss between the picture on the left and the picture on the right and you also have another kind of loss in the middle in the late latent uh vector where you're trying to make the latent vector of these variables contain the most information they can it's complicated to explain but that's basically what you're up to and when you're done you end up with the vector space that can describe a lot of things and a whole bunch of things that the network has never seen before and with the you know with the the special loss you end up with a a smooth

space of describable things that doesn't have like a lot of holes in it like a lot of weird jumps or nonsense in it um so you want to train on a big data set uh and we wanted to work on sketch data so we asked our friends at the google creative lab to make a game and in that game we asked people randomly on the internet to draw objects that they know about like drawbear and then the ai would try to guess whether or not they had drawn a bear and it was a fun game but it also generated a huge data set of sketches of a common object so going back to this picture imagine these aren't pixels imagine these are now vector drawings so now we have a vector drawing of a cat and then we pour it into this fancy encoder which is now no longer convolutional but instead the sort of lstm thing goes into the vector and back out again uh and so we can actually reconstruct the drawing of the cat and you can see it's kind of generalizing because if you draw a sideways cat you

and you draw a human input and then the output is another sideways cat and if i draw a cat face with five whiskers i get a cat waist cat face with six whiskers because most cats have balanced whiskers in the data set if i draw a three-eyed cat i get a two-eyed cat for obvious reasons and if i pass in a toothbrush well the model hasn't been trained on toothbrushes so i get toothbrush cat and we can do it with pigs and as well if the model trained on pigs pig face pig face sideways pig sideways pig eight legged pig four-legged pig and truck you get truck pig um and because this is a vector description of this and this is a giant vector space you can begin to do vector math inside this space so if we pick any two points in this in this space we can interpolate linearly between these things and then we can ask the decoder to uh generate a picture in each one of those things even though it hasn't actually seen necessarily this

particular point in space before and so in this case we're interpolating between a cat face and a pig and you can see as we get closer and closer to pig it begins taking on both the sidewaysness of the pig and also the little curly tail eventually pops up in the snout because that's the thing that distinguishes the concept of a pig and a cat really when you think about it on drawings of the scale and we can do it as well interpolating between cats and trucks um i just the little the cat that's just to the right of the middle one is like my favorite cat i love that guy and with a model like this as well you can begin to do image completion you can start drawing something and then it will try like oh i've seen this before let me let me see if i can help and with a little bit of variance you can draw all kinds of different kinds of of say mosquitoes if you have a model chained on mosquitoes and again thinking about the kind of work that you do from day to day you may not actually want you would obviously not necessarily take any of this art but you might try it just to see it try it on for size see

what it looks like and you can do it with fonts too this is here we're interpolating between the uh sans serif fonts and serif fonts in the upper left and the upper right and we can sort of and the results are in the lower right there kind of going back and forth between serifs and boldy and less bold and more seraphi and again you can imagine trying this out in fact this is a little toy you can play with if you follow that link it's totally neat um and lastly with these latent spaces we can also describe music so uh here with this was just published last thursday music vae is uh basically has a giant space of melodies and then you can interpolate between any two places in this melody space so let's let's take a listen whoop that didn't work let's let's try actually pressing the button and take a listen so this is the first melody

[Music] [Applause] [Music] okay [Music] [Music] [Applause] [Music] [Applause] [Music]

pretty nice [Applause] [Applause] [Music] so it's really cool and it's open source so we have blog posts and more uh uh ways to play with this and other ways to think about it so i'm running out of time so let me wrap up um i know this from being a game developer myself making games as hard and when i think about all the things i spent time i spent making games a huge amount of it was trying things and throwing them away over and over and over again trying to find in the latent space of games what the fun thing actually is and what the beautiful thing is actually

is and i think tools like this and thinking about these problems like this uh might be a really interesting way to become more creative and more efficient although i will warn you machine learning is uh can be a lot of work up front for uh good results later so by no means am i promising you you will get things done faster but you might try a lot more things so going back to the very beginning right now you know there's tools right away where machine learning can make you uh make your game more efficient or uh more uh more easy to manage but later on i think you know uh and even now with some of these tools that we're bringing out and people are building in the community deep learning can help you generate you know sort of unique interesting and compelling content but we will we always need artists and programmers and designers because machines know what they've been taught and you're the people out there

being creative and you know these things are here to help you and uh uh if you want to know more about any of this stuff tensorflow.org is great place to go obviously a lot of the research i talked about here although by no means all of it because we talked about other stuff not by google research.googleblog.com and magenta has its own link there that's it thank you very much okay um this is gonna be a tough act to follow because i don't even have audio in my presentation so anyway um i'm i'm danny lang i'm running the ai and machine learning team at unity and for those of you who don't know me because this is my first time at at the ai summit here i have been leading the machine learning efforts at microsoft

and amazon and uber at both amazon and uber i rolled out a machine learning platforms that basically been used in every corner of both companies and coming to unity i decided to to deal with with one of the harder questions in this world which is what is ai and i was a bit involved in some of the alexa work people ask me all the time is alexandria is that ai i'm like nah i don't think so it's sort of really hardwired though there's a lot of machine learning in there um on the speech recognition part and the text to speech part so there's a lot of small elements here and there but they're not really ai yeah and then of course there's all the recommendations work i did some of that at amazon you know netflix all these recommendations is that ai

no it's not really it's machine learning but again there's some very very good elements in there there's actually some really incredible stuff going on say in the work on recommendations some of the latent vectors that the previous speaker mentioned that's really advanced stuff that goes on in there thought detection for your credit card that's another fantastic area of detecting uh strange patterns in there and uh and and and and catch the bad guys uh when they tinker with your credit card yeah equity trading a lot of machine learning there where you're getting you know tens of thousands of new sources and you try to trade stock facebook feed you know get all the best fake news for you and and then of course one thing that ai is really uh that is in every other job title on linkedin i think in my title i had sort of doubts that you know people gonna talk about me behind my back uh but now everybody has it so i feel like

like an everyday joe now um so what about real intelligence yeah and uh if you i hate people doing this showing definitions from the dictionary but it's something about about you know your ability to acquire and apolog apply uh knowledge and skills uh so it doesn't really that that's sort of too abstract for me too so i thought a lot about it and when you really think about intelligence it's in its inaction around us it's basically the senses and processes that nature gave all living things so that they can run around or grow by consuming energy yeah so they have to eat uh to survive yeah they have to to grow uh they have to consume energy they also have to avoid getting eaten um or at least you know delay the process of becoming energy themselves yeah they have to multiply we're not going to go deeply into that today

and they have to be very careful with physics in particular you know with inertia and gravity you know when you climb after that apple and you fall down and break you break something and kill yourself etc so this kind of intelligence in action uh is is really everywhere around us yeah it's from bacteria which are not particularly intelligent but they have some stuff built into them that make them a little smarter bees are a little smarter than that and humans we are so sometimes sometimes very smart sometimes not so smart yeah um so it's really all that intelligence is made from infrastructure yeah so it's really the the evolution created a complex apparatus for that yeah chemical mechanisms cellular structure all this stuff was created so that nature the version of nature we live in survives and grows yeah there may be another version of nature that failed because it forgot some of these things here here and and we see that organized organisms

have messaging systems that's what neuroscience is about and we had developed sensors you know like a vision came around 450 million years ago that suddenly led to uh the creatures at that time uh suddenly having a very different life and life and evolution took them in a very different direction yeah so so if you think about the game engine yeah it's pretty close to what i talked about it it's a 3d environment where there's a physics engine so so things have inertia there's gravity here and it's a controlled self sufficient ecosystem that sort of mimics the real world yeah so uh looking at that that's actually a an awesome environment for really working with your private ai biodome yeah so at unity we launched ml agents because it adds one last thing to that equation

of the three environment physics engine and that's that's the mechanisms the neural science mechanisms yeah that all those things in that game engine can now start exhibiting some kind of intelligence surviving in that 3d world with a physics engine yeah so what is unity ml agents it's uh essentially a it's open source it's an ai toolkit it uses environments which are worlds which are unity-based so you can create your own uh your own biodome yeah do you remember the biodomes you can create that in unity and then you can add npcs in there you can add agents and they essentially game objects that are attached to an agent the agent can observe

and that's actually the the census yeah you you can have a camera in the agent so it you can see visually or you can have some ray tracing going on you can have collision you can basically mimic everything that nature created yeah uh let's see how many sensors is it that we have some people say five um i see that's you know we can taste and smell and see feel and hear yeah but there's something about we also feel heat cold stuff we can also feel the balance and we can feel movement we have to have many sensors yeah all those sensors you can actually build into into your into your game objects think about it you can actually invent new sensors uh we actually had a guy he did a lidar you know the laser thing that robots and self-driving cars have boom add that as an asset into unity and now you have another sensor

so so observation observations of these game objects or the agent is really about sensors they can take actions they can move around and when they accomplish something they can sort of send a signal back and say success or they maybe fall over and die and they failed to have success yeah that's the reward signals here and then we use reinforcement learning um we talked a bit about first speaker spoke a bit about reinforcement learning today so it's essentially the the idea of running a simulation learning from that simulation which is basically uh coming up let the system come up the policy uh that is essential and optimal always think about a policy as an optimal mapping between what you see and what you do yeah so there's always the random policy which is you know you throw a coin to make decisions yeah but that's not that's not an optimal policy so we try to let the system learn the optimal policy

and then when you have that policy you can now move into the inference phase which is you can let the system play out in the diagram very briefly this is what it looks like you have the learning environment that's unity the big gray thing there's a python site to it the python the python part of it it provides you with the mechanisms and the ability to hook algorithms up outside unity so we're doing all of our work on tensorflow but you you can put hook it up with your favorite machine learning system but if you go to to the ml agent github uh there will be uh this of how to how to download uh and install and get tensorflow hooked up with this yeah so you have that training taking site outside unity then inside unity there's the academy that embeds a communicator that you don't have to care much about but that's the thing that sort of takes things from

insanity in your in your unity environment out to the to the for training outside yeah so in in in the academy you have basically agents that's game objects you have game objects attached to agents and they can do stuff as i said observe and have actions and send rewards and there are one or more brains the brains is sort of what you try to build here and that's what i call that's the new science part of our biodome training scenarios so we already heard about some of them but uh this this is actually why i'm so excited and why we're doing him all agents because by making basically democratizing ai so that everybody not just the big corporations can play with this yeah it is really bringing it out and what i hope you you take home from this is that there's so much to be done

you're just scratching the surface and what we're trying to do with the ml agents at unity is to open the door for you to take this and run with it yeah so i'm going to give you some examples and this is you can just imagine how how many opportunities there are here when i when i go through this is like as an example you have a single agent with a single brain doing something yeah it's a very very simple combination that can be you know a game object trying to get from a to b or learn to get from a to b with some obstacles in between yeah single agent single brain uh fairly straightforward but then you can also have examples of uh multiple agents uh with a single brain yeah and and and what that does is that it sort of gives you an opportunity to have a lot of action going on a lot of reward signals coming back in but you you you accumulate all that knowledge in the single brain there's also adversarial self-play you

have two agents have a single brain single set of skills but for each agent uh the reward signals are inversion i try to beat this guy and this guy tries to beat me and we feed all women into a single brain so we get two guys who are actually really really good at playing each playing against each other yeah and they make each other better and better because all the experience goes into the same brain yeah so you can see a lot of this computer science stuff is really sounds more like what we do every day together by learning and exchanging information and playing games and i'm not even computer games but maybe even you know soccer games or things like that where we we we do things together so it brings me on to cooperative and competitive multi-agent scenarios where we have multiple agents multiple brains they may have shared or inverse rewards here so hopefully you can understand that the opportunities here of working

and combining these brains and agents and the rewards are are basically endless um i'm gonna tell you about comments so so the training environments that's basically the the the setting uh in which your game objects are doing things yeah and uh what we've been doing is that we've provided you with with a bunch of training environments that you can just grab and start playing with which means you can actually go home and do all these things i show you right away but you can also take these training environments and change them yeah or you can clone them and then change them like crazy yeah there's no limits here so a lot of the work going on in reinforcement learning and other machine learning and ai areas ends up being a lot of papers that you read and then you're kind of oh i wish i could try this

this is what we we're giving you here is this ability to actually uh stop maybe even writing all those papers and actually do some of these things and share them um i'm going to show you a few examples uh here's the 3d balancing ball it's one brain with 12 agents and i'm going to show you the video here so it's uh it's about balancing a ball and a pad and it's 12 agents and the observation of for these a 12 agents is actually the the the location and velocity of the ball so where where's the ball rolling based on that input yeah when we run this with with 12 agents they learn to they learn from scratch uh what we call tabula rasa yeah from a clean slate they learn to balance that ball in 30 seconds yeah in this case by adding 12 agents in

there it's it's it's a way of just generating more data yeah and i just said something very important there because what we use the unity engine to here is to generate the data and we need tons of data to train these systems yeah i'll show you another one here it's it's basically two brains it's it's it's it's two-person talk or four people two in each team playing soccer yeah so it's really two brains and four agents so before i run it i just have to explain that there's a there's a there's a goalie brain and a striker brain yeah the two goalies they both share brain because that job is the same and the two strikers share brain because their job is the same yeah but they're in two different teams so they are in the rewards are inversed because they're trying to beat the other guys yeah um so you see uh in this case the

observation is actor raycast yeah so that you will see the little square agents i apologize for the graphics here but it's sort of to keep things simple has a little eye yeah they are one-eyed and they uh look at the strikers by the way have you noticed the strikers are looking sideways all the time yeah but the goalies are looking forward yeah because the goalies need to look at the ball coming against them but actually when you're a striker with a 180 degrees view point of view then if you're on the side looking at the field from the side you see the most if we would now end that view of field they will probably start turning a lot more around to to scan the space here so that's one thing that's important to remember here is that you have opportunity to to basically put a camera

on the agent or put a camera over the entire world yeah so in this case in soccer here the agents actually don't see everything they only see their point of view they don't see the entire world you could in other cases have cameras that show the entire world let's do some uh oh that was a joke that was not supposed to get in here uh but it's it's wall jumping um where we basically give a agent uh uh two two brains because it needs to do two different things yeah it needs to to learn to to push another block next to the wall so it can step up on the block and get over the wall yeah so look there is cheating it could jump on the low wall here but in this case it it it has to to basically learn two skills yeah so one one is pushing the block and i want to tell you how it learns that because this is again so fascinating

yeah because it learns that through curriculum learning so what we do and we support this directly in in ml agents we basically give you a chance to teach the agent the simplest case where the wall is actually just a tiny tiny tiny and then we make the wall taller and taller as the agent reach some objective some some some threshold with the low wall then make the wall higher yeah and what's so fascinating is that you get a better model out of curriculum learning rather than just let the agent try out to scale the tall wall right away actually well you can see the orange one is its ability to scale the tall wall just never seen the the low wall yeah so it's a bit like when you go to school and go to first grade and second grade third grade you sort of learn things step wise yeah instead of just

jumping right into it isn't it weird that these systems in with all their primitive technology from tensorflow wherever it comes from exhibit that same behavior yeah it's the same behavior that that that building up building upon a previous learned skill we have seen other examples of that where if you want to build a robot hand that really grabs stuff you teach it a lot of primitive things it can do and then it gets much better at learning the hard task so that that's an important message because it's really uh it really tells everybody that you need to sort of take this out of pure research and really put it in the hands of a lot of people because they would actually do you know be able to explore many more aspects of this i want to mention another one which is essentially a sweet little one using lstm which is is basically called long short-term memory it's the ability to remember something and use it later

on so what this little blue uh uh cube does is that it goes and look at the big cube and then if the big cube is red and goes to that side and if it's orange goes to the other side but it has to do two things again yeah it has to use raycast identify the color of the the first cube and then decide where to go and find its way over there and end up in the right corner so again two two different things here so it works goes to the on side this means goes to the red side go to the orange side looks at that and then go to the one side yeah so it is an example of the use of specific of memory yeah i think that's fascinating too with the unity engine we have high fidelity graphics we have physics we have rich assets i mentioned the radar lighter stuff like that as sensors and uh you can create scalable

simulations where you and in theory you can run you know 100 000 instances of unity and gather data and uh oh sorry i went too fast here i just want to show this uh if it runs is it running yeah here we go um we are working with with uh microsoft and uh and and the city of belgium and and the idea here is to to actually train a machine learning system to uh detect dangerous uh traffic situations uh through cameras yeah but the traditional way of doing this would be to record hours and hours of video and have a thousand people hand labeling the video etc yeah to train the machine learning system and what we're doing here is we're showing that you can actually create these uh scenarios that in unity that will basically kind of fool the machine learning system i think this is the real world here and train it yeah and on on a very sad

day like today uh when there was a pedestrian killed by an uber self-driving vehicle this is kind of a way to to train vehicles train machine learning systems uh in a in an artificial environment before you actually bring them out in the real world yeah so both the cell driving cars for robots and also for systems that actually deal with what we call real world situations it's a fantastic environment for doing that and this is sort of when you look at ml agents and then you kind of put them on steroids okay that was all i had for today and i appreciate your time you