Machine Learning Summit: Successfully Use Deep Reinforcement Learning in Testing and NPC Development

[Music] hi everyone thank you guys for joining us for our presentation obviously we're very sad that we won't be able to be everyone in person but still very excited to deliver our presentation today my name is Jeff I'm a lead product manager here at AI unity I'm joined here with dr. Dan who's a researcher also at unity and Robin Linda Nielsen is a co-founder in Kerry castle today we're gonna be talking about successfully using deep reinforcement learning and testing and NPC development so you may be asking why unity at the GDC element here unity we see a variety of games whether it's the 2d puzzle games all the way the first-person shooters so for us we see a lot of the studio's trying to implement deep reinforcement learning using in machine learning in their games and so one of the things we want to provide a day some of the aspects with and learnings that we see at Unity so

first we want to start a fire so many stairs trying to eat enforcement learning well first they want to shift games faster let's code what makes them spend less time on testing and of course maintain the quality so it's one of those situations where you know we want to have a free lunch we have our cake media - and then and you sort of insert your favorite idiom and this is really the core about why as long as these are trying this approach so there's a lot of poems applying deep reinforcement learning in gaming it can actually solve quite a bit of challenges but but we should be mindful about some of the difficulties and really that's kind of a hard way we want to talk about today a lot of the challenges that studio space will try to implement deep reinforcement learning so how our studio is using B for student learning well the first is from the player perspective now this is creating some sort of player bot or virtual player and this is usually for a game testing game balancing the second is from the non player so these are unlabeled characters or objects in the game and these are enemies can be

companions would be passerby characters and the last we would call the invisible scenario these are the scene itself or other experience and not seen by player so this is where the reinforcement learning is is trying different things for the player and this is good and they're under generation difficulty tuning and player engagement so really today we're going to focus on the easy to use cases this is really predominantly where we see quite a bit of studios trying in the game testing and creating enemies area and game testing side one of the things that we do constantly see that students are trying to use deep reinforcement when they're testing levels so this is just some Surgenor is a shove a player bought but to test you levels or content the second is run enemies now this is the natural looking enemies now these think about like making the enemies you look and feel real without having to write a lot of vertically so really that's what we're going to focus on today you know testing new levels you know in content URL and then obviously natural-looking

enemies they're also obviously very many more scenarios and a lot more use cases but today we're going to focus on these so which I'm going to pass it over to urban so we've talked about reinforcement learning thanks Jeff so to recap what is the reinforcement learning so many of you are probably already familiar with this but if you're not think of a traditional machine learning right it's taking an input maybe it's an image of a cat or a dog and learning how to map that to an output and this output could be like a label right like is this picture cat or is this picture a dog so reinforcement learning isn't too different let's imagine you have you want to train a behavior for an agent in a game you collect a bunch of observations in your game this could be things like the agents location its velocity how close the enemies are and you and then the agent basically learns the policy what we call a policy that

takes those observations and maps them to actions so just like the classifier would take an image and map it to a label it maps observations to actions and these actions could be for instance moving around in the game shooting the enemies and doing other actions right but how does it learn how to map these observations to actions and this is where reinforcement learning is a bit different than traditional machining so in reinforcement learning or RL the environment which in this case is your game gives a reward to the agent when it does something desirable so through the agents playing the game exploring and trying different actions it learns how to maximize this reward it receives during a playthrough of the game and we call this playthrough like an episode so this is really good for creating behaviors and games because you don't have to explicitly specify how the agent

exceeds achieves what you want you just give it a reward when it does and it kind of through exploration figures out a winning strategy so let's say you wanted to use reinforcement learning to create a bot for playtesting right so you want a bot that you can use to test and balance your game and reinforcement learning sounds great because you don't have to hard code the behavior so what is this bot has to be able to do it has to first and foremost obviously be able to play the game and win the game just like a human player but the bottle so needs to do a couple of other things first it needs to be able to deal with changes in your game so if you're tweaking your level it needs to be able to deal with these changes and adapt it needs to be able to play new unseen levels to evaluate them things it hasn't seen in training otherwise like why are

you testing right and it also needs to solve the levels in a human-like way and not find some exploit to get the reward but not actually play the game the way you wanted to so to give a concrete example let's take the game Snoopy pop from jam city so Snoopy pup is a bubble shooter you can see on on the slide there's a screenshot from the game and you see that Snoopy is floating our red belt and the objective of the game is to shoot the bubble at some angle up to the top of the screen and if you get a match in color it'll clear those bubbles off the screen and you win the game by clearing enough bubbles and enough of the special bubbles with the yellow a bird inside of it so what gem city needed from a testing bot was a bot that could play all of the existing levels of Snoopy pop with similar performance to human players and

be able to play newly design levels and determine a if they're solvable and be how challenging the level would be to a human player would the player need to use power-ups would they get frustrated and quit these metrics would be really invaluable to the level designer while tweaking the level so this seems simple enough but what makes this game especially difficult to train about for well first of all there are over a thousand levels and they all have different layouts and different ways of solving them furthermore each level is randomly generated so that the position of the bubbles and the color of the bubbles can change from play through to play through that so that means if you scripted a bot or if you your RL but simply memorize the the actions it took

to solve a particular level it would not be able to and to make things even worse every couple of levels a new element is introduced so these can be different obstacles they could be different power-ups they can be bubbles that explode in different ways and your agent would have to learn how to deal with all of these things so how are some ways that you might try to approach this problem so we could take a naive approach and we could say hey each level is a separate game we'll just train a different agent to to play each level and this this will work but because now you have thousands of levels you have thousands of training runs and thousands of neural networks that you're creating a fish training run takes several hours to a day this could get really expensive really fast not to

mention having to retrain a bot for each new level that your level designers create makes them all separate and you can't really directly compare the performance of one bot to another because they're different BOTS secondly we could also train the agent the way a human would play the game starting from level one then once it masters level one train on level two then one it's masters that three and so forth if we do this though the agent is likely to forget the things that it saw in the earlier levels and in machine learning we call this problem catastrophic forgetting and basically neural networks have this issue where what is learning something new it tends to quickly forget things that it has seen in the past humans don't really have this problem but we do have to worry about this it reinforcements light finally if you imagine that for each of the levels that you've created you probably have at least a couple human

playthroughs of the game you could train a bot to directly imitate these human playthroughs and hopefully be able to solve the level the problem is as you remember the levels were somewhat procedurally generated there were some random bubble positions there were some random bubble colors and it's impossible to have human demonstrations for every single possible combination the agent could see and so you can't just imitate what you see in the demonstrations because you won't have demonstrations for all scenarios so it's clear that no one of these approaches would work by itself but what could work now we can't say we've solved all of the thousands of levels of Snoopy pop but these approaches did prove effective in this game so if we want one agent that can play all of the levels and we don't want it to forget earlier game mechanics we will have to rely on domain randomization to

keep the agent robust to variations in their part in the Snoopy fraud case that this means being robust to new levels and different bubble configurations furthermore while pure imitation learning might not be a good strategy having those demonstrations having those human demonstrations can really help especially if you combine it with reinforcement learning it can guide the agent towards a good solution but not prevent it from learning how to solve situations that were not in the demonstrations okay so let's talk first talk a little bit about domain randomization so it's a common technique used in reinforcement learning to make sure your trained agent can generalize how to solve a problem rather than just memorize very specific instances of your problem so the idea is that you find aspects of your domain so in this case your game levels that can vary for instance lighting textures layout of the

level type of enemy and so forth and during training you randomly vary these aspects so that the agent sees all of the variations then the agents will learn to cope with or ignore these variations so I have three examples of the main randomization here on the slide on the left is a classic example from opening eye where they had a virtual robot hand trying to manipulate a Rubik's Cube and I was learning how to do that and during training they varied the background color as you can see the lighting and the physics of the actual robot and manipulating the cube and since the trained agent saw all of these variations and learns how to ignore the ones that weren't important and deal with the ones that mattered it was able to eventually manipulate a cube in real life so in your in your in a game there's no real-life test but you're

this real-life test is now instead training on a new level right and it's kind of the same idea so in a game you can do domain randomization in a couple of ways for instance if you have thousands of levels like you did in Snoopy pop you could randomly sample the different levels and treat that as variations in your domain likewise if you have levels that are procedurally generated you can use procedural generation to create an infinite number of variations of that level so if you want to read a bit more about how to use domain randomization to Train robust agents I'll leave this links here and they'll also be available after the presentation okay so let's talk a bit more about what did I mean by we can combine demonstrations with reinforcement learning to improve

performance well so when you're training a reinforcement learning agents the first thing you have to do in your game is define a reward function that describes what you want the agent to do now the easiest types of rewards to define our sparse for instance you can reward the agent when it wins the game right and give it a negative reward when it loses the game and this gives the agent the freedom to find the best solution towards that goal but sparse rewards are much harder to solve as the agent needs to through Lux stumble upon them so reinforcement learning is essentially guided random search the agent tries different actions and it sees what kind of rewards it gets and it's very unlikely that the agent will randomly find a way to win a complex game however if you give small incremental rewards you may spend a lot of time tweaking these rewards and they may lead to unintended non human-like

behavior for your player bot so what we can do to solve these this issue then is to mix reinforcement learning with human demonstration data so rather than purely using the reward functions trainer agent we can use some of the demonstrate is provided from a human player to guide the agents learning this can not only speed up exploration towards those sparse rewards but also result in the agents solving your level in a more human-like manner as it shows to follow that path while exploring so here's some two examples on the left on the right is Snoopy pop which we've seen that on the left is an environment that we've created where the agents needs to hit a switch and then grab a green block before it even sees any reward and it gets you know plus one for finding touching the green block now it's very unlikely though possible to find this by

random search but it's it's pretty obvious to human and just providing a couple of demonstrations around ten we can cut the training time in more than half and this is because it aids the it kind of pushes the agent to get the switch and to find the green block so this is a sparse rewards even with dense rewards like Snoopy pop where we do give a reward every time bubbles are cleared on the screen that makes sense your score goes up demonstration still can help you reduce your training time so a similar number of demonstrations brought the training time down around 25% so why would this be worth it well in some cases a few minutes of human playtime could save hours in training and in many cases you may already have this data maybe you as a designer have already played through the level with uppal times or maybe there were human testers that so you

could do a bit more with demonstrations you can also solve problems that were not solvable using pure reinforcement learning a good case study of this was during the unity obstacle Tower challenge where the competitors had to create agents that could play the platformer game shown on the screen now you can see that the agent basically has to solve this fairly complex block puzzle to pass this level would push the purple block onto the switch and then go through the right this is pretty much impossible to find through random luck like the agent would never figure out how to jump on that thing and then push the lock down and then go through the door so furthermore the levels procedurally generated so every time you play it's a slightly different layout and the blocks in a different place and so forth so I can't just memorize the actions it took to be able to solve level now our winning competitor Alex Nichol used playthroughs of the game to guide the agents learning leading the agent to

discover the winning solution so after enough iterations the agent learn how to solve even the randomly generated levels that were not in the demonstrations now a little caveat here it took way more than ten playthroughs to solve this level in total Alex sees more than two days of human playtime to be able to solve the levels of obstacle tower that he did you can read a lot more about his approach which i think is really cool and check out his code at the blog post link on slide and again they'll be available so if you want to read more about how you can use demonstrations to help reinforcement learning agents learn in a timely manner I'll leave these links here on the slide and again they'll be available so something we've hinted on a little bit is that RL takes a lot of time often it requires the

agent to take millions of actions in a game to learn anything useful and this would be okay if it wasn't for the fact that modern games are fairly complex and they have simulated physics they may have complex 3d scenes and running a million actions in a game isn't free right tech techniques such as the main randomization would make the agent required to like even see even more samples because it not has to deal with all of the variations of the environment so how are some ways we can train these play testing BOTS more quickly well any sort of speed-up is gonna come one of two forms it's either gonna be through sample throughput which is the you know how many actions we can take in our game per unit time or sample efficiency which is how many samples our algorithm takes to train right so some

ways to increase sample throughput you can speed up your game you could increase the interval between physics calculations so you could increase friend skip decrease rendering quality and so forth it's nice because you can use the same hardware that you're running on before but it's limited and the first sense if you speed the game - if you compute physics - infrequently you may have bugs that you didn't intend huh stuff may start tunneling through each other and so forth so if you don't want to do that then you can run many copies of your game this doesn't require any modifications doesn't change the dynamic dynamics of the game but it does proportionally increase your computation cost depending on how many games you're running that's pretty obvious now okay so that's sample throughput and for sample efficiency we've already talked a

bit about using demonstrations to reduce the number of samples needed you do have to produce your demonstrations but something else you can also do that we found effective especially in the Snoopy pop case is making sure that you're using a sample efficient deep reinforcement learning algorithms there's kind of two main categories of deep reinforcement learning algorithms I won't go too much into detail today it's a bit out of scope but basically we have on policy and off policy and off falls the algorithms tend to require far fewer samples to learn behavior but they tend to require larger neural network models and more computations and unter training so this trade-off could could be beneficial depending on how quickly you can take the actions in your game and in the Snoopy pop case which was the game itself was in speed up because we couldn't like

increase the physics steps too much otherwise the bubbles would tunnel through each other using a sample efficient algorithm a policy algorithm we saw more than 5 times training speed-up versus a non policy ogre all right so to quickly recap using our elf for testing so if you want agents that are robust across levels and could be able to play your new levels you can use a some form of domain randomization to make sure your BOTS can deal with all the different scenarios now if the problem in your game is too hard to solve by pure random exploration you may want to use demonstrations to guide your learning this can reduce training time as well as produce more human-like behavior and finally we did talk a

little bit about how we could reduce long training time so you could leverage parallel computing you could use demonstrations or you could try sample efficient reinforcement learning algorithms to reduce how long it takes you train your agents so that's that's reinforcement learning for playtesting and I'll pass it on to Robin to talk about reinforcement learning for non-player characters thank you very much hi everyone my name is Robin I represent Kara castle a small Swedish game company started together by me and Pat Fernanda we are a small team but we love technology and we use it to make a great content for our games right now we are working on a game called source of madness source of madness is an action road light as an acolyte from the cult of

knowledge you have to close the box of Pandora and end the madness you travel through a rich dynamic world but every time you die you start over and the whole world changes the world is full of horrible monsters from the void and these monsters are different every time you play so we asked ourselves how can we make millions of monsters and keep them interesting even after playing for many hours we are only three people know these were our challenges we wanted to make the monsters move by controlling their muscle physics - looks unique and super creepy we have millions of different procedurally generated monsters and it needs to look natural this would be almost impossible to cure by hand especially with only three

people so we turned to reinforcement learning as a solution to train the monsters we use the following setup in this slide the AI gets in the data about the environment and its body and it learns to control its muscles they can also grab the ground monsters are rewarded for moving towards the player but we also use some additional rewards to get different behaviours and I will show you an example of that now in both of these videos the main goal was to walk right towards the player but we also gave them a smaller extra reward so on the top as you see they got rewarded if they stayed close to the ground so they got this tumbling movement and the bottom they got more reward by staying above the ground and they learned to

jump so as you see in this way we can get much more variation without spending too much time the enjoy extends well to different body structures as well the neural network will learn how to swap with its body for example a quad pad as you see on the left side or a spider on the right side many of you are also maybe using machine learning or want to try it out so I will now share some things we learn during this project and hopefully it will be of some help to you so our main challenge with machine learning is to understand why when it doesn't work if your training doesn't learn there are many potential reasons so when it doesn't work you want to find out what to change and you want to avoid start guessing randomly because that takes a lot of time so from our

experience if you're working with machine learning in real time like for a game the biggest pitfalls are probably their reward function and the handling of actions because these are done by your own code and as we all know it's easy to make mistakes our icky takeaway is to always make sure that these are working as intended whenever you change your code just check them again and make sure that this bill works as you want it to work mmm-hmm otherwise you may be waste hours on a failed training and it takes time to find out what went wrong or even worse they I might work fairly well but not optimal you might not even notice at first that was the case with our monsters recently we have not focused on the AI for awhile and we just started to notice that they didn't work as good that they as they

used to and finally we found that a lot of the inputs were always zero and it had been like that for two months and once we fix that the monsters got a lot better again to avoid this kind of mistake it is important to make good visualizations of your data so that you can easily spot if something is wrong this is an in-game gizmo wood that we made early in the project as we can see in the engine it shows the reward in real-time so when this little monster is walking right towards the player we should see that the bar goes more green and if it didn't then we know that something hour or a code must be wrong either the reward function or the training itself could be wrong done so this will save you a lot of time if you can just see what is going on and not just guess randomly let's not consider

these monsters again as I said we have a main reward and several other smaller rewards and we added them together here comes a problem if one reward is very big and the others are small then the small rewards would have almost no effect so we need to balance them properly and how do we do that so to help with that we modified the reward gizmo you see this on the right this is the same gizmo as before for the reward but it shows each kind of reward as a separate bar so this monster is getting a reward for moving that's a green a big bar on the left and it's getting a punishment for being upside down that's a red bar and then if any bar is too big or too small we can adjust the reward function and avoid training in a long time to realize that it didn't learn what we want to do as urban talked about

that it could get a naughty human-like behavior if you're training a human agent you can balance a reward function better if you see what's going on the pair is another gizmo that we made to see when a limp is grabbing you see this little blue balls with this gizmo we can easily see if an agent learned to use the grabbing or not it also helped us to find out quickly when the agents were not driving at all beautiful code mistake that people seem to not able to grab so it wouldn't help if we trained it longer that we realize very fast in this way so I hope this advice will be useful and inspiring it's a really fun topic to work with machine learning and if you're interested to know more about this game check out source of madness calm and finally thank you all for watching if

you point your camera at this QR code you can find more information related to this talk and also the links that are bein a post about so thank you all for watching and have a nice day thank you everyone