'Little Learning Machines' Postmortem: A Game About Training Neural Networks

[Music] thank you so much for coming it means a lot to us um so this is a talk called Little Learning machines a game about training neural networks um fundamentally this talk is about reinforcement learning in games um and why training agents is actually fun and we made a small game about it and a lot of the second half of the talk is how to make training fast which is a very important part of this um but first off we need to kind of like recognize some prior work um black and white the 2001 classic I hope you all know about it creatures 1996 which is even bigger influence I hope everyone knows what that is um both of these games uh use forms of reinforcement learning actually while you play um and then more more

like more recently modern machine learning has been used in games such as project Nero while true learn artbot which uh was made or we found out just as uh we entered Early Access incredible game um and then light bbot which doesn't use machine learning but focus on teaching kind of like simplified models of programming um but actually our main influence when going into making little learning machines was a reinforcement learning uh short film we made uh called agents and it was directed by Petro gallano who is in the crowd and he's the best um and both of these projects were created by transitional forms um trans transitional forms was founded with a goal of kind of like human machine empathy and getting machines to understand humans and humans to understand machines um I'm Nick uh my background is in animation and illustration like primarily focusing on kind of like general science education

and I helped co-direct this game and we're co-directing this talk so I'm Dante I'm a I was technical director on this game and uh my background is in research and emerging Technologies uh one of which at the time was reinforcement learning specifically deep reinforcement learning uh quick overview of reinforcement learning don't want to get too far into it this is the AI Summit after all but you have an agent you have an environment they need to do things together usually in the form of the agent doing taking Tak some sort of action that changes the environment and then that environment changing and those changes being reflected in what the agent can observe so the agent has some information about the environment that it observes and the way that the agent uh changes its own behavior is through reinforcement uh reinforcement happens through the provision of rewards that can either be positive or negative you tell an agent to you give an agent positive rewards for doing the things that that you wanted to do and negative rewards for doing the things that you don't wanted to do over time the agent

optimizes and then learns how to develops a policy or a behavior that U allows it to optimize this this amount of reward um cool so um this really kind of one of the places I'd like to start is with this uh and this was one of the primary Inspirations for our director Petro when uh he was uh coming up with what we saw in like RL as like being inspirational this idea that within just these pixels you can see that the and the red pixel are collecting these green like apples and uh they're shooting yellow lasers at each other but even within this like little 2D simulation there's conflict and it's kind of like interesting you can root for one blue is my favorite like by far um and this this is a pattern that repeats during like reinforcement learning right you can see uh this is an example of clean RL where it's trying to learn how to walk in the mojoko environment and yeah there's like Strife there there's some sort of the the only reward function for this one was go right and ourl doesn't have the

intuitions we have so it just does what it can and this is we thought that this was very cute um and then finally we also saw RL at the time kind of exceed our expectations again this is an old paper but opening I trained a team of five players to play DotA which is if you've played DOTA you know that's like really hard to collaborate at at DOTA so um uh yeah RL kind of created a model that played better than we did so that was like amazing so all this kind of came to a head when in 201 19 2020 2019 uh we released this film called agents yes and uh so agents was like I said like the main inspiration going into little learning machines it's a short film about uh reinforcement learning agents um as you can kind of see in the film it's uh kind of interactive you can kind of move these kind of creatures around this planet um try to like balance it out um and agents live on this place it's weird it's spinning it's kind kind of

like um being influenced by your interaction and the weight but actually gravity goes down which is weird but also that means the agents can fall off so it's quite a dangerous place that they're trying to kind of like coexist uh cooperate but also kind of like create a bit of conflict around this um and what's special about agents is that the characters uh the actors were actually trained through reinforcement learning um and this was back in 2019 like we just said uh they're just neural networks they're just neural networks and more importantly users could actually interact with the pre-trained models in VR or literally on their uh smartphone um and it was spe it was special for me at the time because I had no idea what any of this was and uh uh it kind of like inspired kind of I guess the next six years after um so the process was very classic um in this video you can kind of see agents training uh there's a simplified

environment um all running in parallel and basically uh developers at the time Dante and Alex would set up these trainings and leave the agents training overnight and hoping for some sort of success um uh often that took 16 18 hours plus with our small team and for a long time we didn't see anything um but one one day we all came into the studio like in the morning and we saw the runner um might not be that impressive to any like AI enthusiasts in here but to us this was like a big deal um and we love the runner he's kind of like turned into the meme of the office um and it was really the first time that the team like really started feeling a connection uh and kind of like a love for this kind of like digital being if you can see what he just did he he tricked the other four agents to fall off by going to the other side and then just continued to run infinitely what was its reward function just to stay alive and and it did it it

did it well um and so after like this kind of like success with the runner we're like okay we can do this and we continue to train like more behaviors like this one we call like the flocking or the dancers and stuff um and the more behaviors we trained uh the more we kind of like developed a lot of like love and pride and attachment for these like little beings but more than anything we found that watching intelligence and behavior kind of emerge out of the randomness they start with is in itself a uniquely rewarding experience and unlike anything we kind of like previously worked on or kind of experienced in games so we started to think about uh how we could use this in a in a project in the future yeah so uh now as like we were a small team and uh once we started kind of coming together and seeing these things like happen like the mornings were just cool it was like oh what did they do last night one time they actually learned like we gave them a reward to get as close to the middle of the planet as possible and they learn how to clip through the geometry which

was crazy uh so but the the problem we were having is like okay we show up and we showed those simulations and then the whole team has ideas like uh like Nick had ideas Petra had ideas but like like everyone like I think at one time our marketing manager Michelle came and she's like oh you want them to fight more just like increase the scarcity or like have less flowers and then we did that and it worked and it film became more interesting so the whole idea is like we we Alex and I were gatekeeping kind of any from running whatever simulations they wanted and that would that wouldn't work so we wanted to share we wanted to make it like so that they can run their own simulations but we had some problems um so the first problem is that it was slow right uh 16 hours is a lot it's a long time I can't even like wait for my crockpot to go that long I I usually just eat raw me you know what don't do it um uh and then the other thing was a hardware right like Alex and I had like big thread rippers like uh back in the day those were beefy and we ran them on that and we had to run sorry sorry we had to run like parallel

simulations and that was expensive so uh we either had to pay for cloud compute or pay for expensive hardware and it it just it wasn't something we could easily do or provision at least with a small team uh the other thing is like this may not seem like a big deal but coding is a big barrier of Entry um especially for something that we saw as like as accessible as reinforcement learning so having to just code your own reward functions or change the physics of the world or whatever was a big barrier of Entry uh it was also non-visual which is uh difficult if you have like uh disabilities that prevent you from like engaging with textual formats or or it is it is sometimes difficult to work in a non-visual format and then finally it was just very complex this is like reinforcement learning after all it's it's deep reinforcement learning there's a lot of like alment equations all that stuff so like having an understanding of um of kind of optimiz ability was a bit difficult to kind of uh convey well like oh that won't optimize it won't converge so uh just to give you an example of how complex things were when we wanted to help someone set things up it's like okay well you have to install Unity install python set up your virtual

environment okay not that one um that other virtual environment set up tensorflow no not that version of tensorflow the the the one that's not make sure it supports Cuda but not for that video card uh okay make get your environment running do your reward functions whatever run the simul oh you have a bug ah yeah you might want to try running it you know what I hope you like graphs because that's what you're going to be looking at just graphs and oh and command lines and more graphs and more command lines shout out to ml agents they're they're they're tooling was great um uh but uh we wanted to make it way more accessible and by way more accessible meaning really accessible so we we did it uh we made this game uh we we made a whole game out of it uh so yeah the rest of the talk is going to tell you about little learning machines where we oh sorry wait we made this game Sorry I'm looking at the wrong slides um yeah we made this uh this game called little ring machines which we're going to talk about um and in this game um we implemented something called real-time

reinforcement learning which is not a new AI concept but rather a game mechanic one that you can hopefully put into your game if you want your your players to experience it so let me just quickly run you through a live demo it was supposed to be a live demo but Nick doesn't trust me with the computer um this is an agent that's been trained to water flowers you can see it's kind of walking around it's it's kind of walking around looking at like flowers that it wants to all right Nick Nick moved it put it up here we can see that it it kind of navigates the area looks for flowers to water but what we're going to do is we're going to reset it just so you know that it's not pre-trained and um that's what it looks like when you reset your brain I know firsthand um so this agent is now reset it's brand new even though it previously was uh watering flowers the same network is now completely randomized so the actions that it takes are kind of at random anything um and what it's going what we're going to do now is we're going to give it something completely new to do um and uh we're I want to show you how

quick and easy it is to kind of uh give it a new thing um to to focus on so uh in order to do that we have to go to this uh space called the training cloud and in the training Cloud we can set up an environment for our agent to learn um the first thing we need is obviously our agent this one is affectionately named chungus so it gets put right in the middle and we're going to remove its old rewards doesn't need them anymore and uh we're going to give it something to do so the easiest thing that I can show people in in a talk is just collect crystals let's put down I don't know 14 crystals and um and see can it collect them and we don't have to code any sort of AAR and there's no Pro this this agent is not pre-programmed okay this is this is like completely onty we could have trained it to I don't know chop dogs with axes but instead we're telling it that it loves collecting crystals and like that's put in there we set up the end conditions so that it doesn't go forever we saw what happens when you let agents go forever and boom that's it we're running a training it's uh

reinforcement learning happening on your computer now that doesn't sound like super impressive but to us at the moment this was like oh yeah this was this is especially since in undergrad I did a lot of this in Labs it's like this is the the smoothest it's gone in a long time and uh as it trains you can see right now it's starting to take uh random actions but little by little you can see right now it didn't want that Crystal but little by little it um develops a desire for crystals it can develop a desire for blood too if you train it that way um but yeah you can see that there's a little graph on the left there's still graphs we couldn't get rid of the graphs at least not yet but it's a little bit more intuitive it uh you can visually see the behavior that's happening you can get a sense of how the agent is improving and within the span of two and a half minutes we have an agent oh well the other thing is you can't really see the behavior that well it develop in real time unless you kind of speed it up collecting crystals is like a macro Behavior so if you speed it up you're going to get a better sense of kind of the macro decision that your agent is

doing which is why we added that little speed up there but yeah t a trained agent in a matter of minutes now obviously this is a very simple monotonic uh problem but um let's recap was it slow no it was pretty fast it trained in like the setup is what took the most part that took like a minute and a half but the whole training happened in a matter of like a minute or so um did we need a special Hardware no this was trained on an M1 Mac but if you have a Surface Pro 3 from 2019 it'll work just fine um did we have to do any coding no all we did was select the reward functions like visually uh which clearly the whole process was a visual one and finally uh was it complex well there we have a bit of a caveat and we'll talk about the for the rest of the talk we'll talk about some of the complexity that arises in trying to set up uh a world and and behaviors like this so I'll let Nick introduce that yeah so first of all uh our approach requires kind of like four steps um the first being designing an

actual world for uh realtime reinforcement learning the second being designing the agents the third designing the training experience itself and what kind of like training Loop the players will go through and then fourth like packaging it all into a game um but before we start that I just need a few key learnings from Agents that really kind of like influenced early decisions on how we kind of like set those systems up um first off there's in agents there was a fund m al mismatch between how our team um experienced training the actual reinforcement learning agents seeing them learn and seeing their behaviors kind of emerge out of nothing versus audiences who like viewed these pre-trained models for the first time and may have not actually understood anything about machine learning so it was kind of like it's difficult to understand machine learning in general or what a pre-trained model is what they see what they're doing um and fundamental how they're trained um it's a lot easier to understand them when

you're actually there in the room experiencing and watching them learn um second um agents had this really beautiful kind of story with a lot of different story states which each story State kind of required different behaviors from the agents themselves so agents were expected to kind of like hit specific kind of behaviors at specific different story beats which made it kind of really difficult to craft them um and it didn't necessarily allow them to FL flourish on their own uh and on top of that as you probably noticed the environmental conditions we place the agents in in agents were extremely difficult and very complicated um any action they took could easily like lead to their death um on top of that even their their actual actions they could take to survive were quite limited they could move they could um they could kind of like consume the flour and they could also headbutt each other all which kind of combined to making it quite difficult

to find really beautiful kind of uh Behavior out of them and that kind of led to the main thought which is like the agents can only be as smart as their environment lets them be um so that's the main thing going into the designing of world for little learning machines um so when designing this world like we wanted to First of off first off like prioritize potential diversity of behavior um by letting like the agents uh develop personality and actually show that they're learning um second the environment that is we wanted to create an environment that was more of a playground and like a bit less of a death trap and um which meant that like it needed to be safe but not too safe um it needs to be full of opportunities and it needed to react to the decisions of the agents um and all the decisions we made like in this design Pro process were kind of like based on playfulness um we

want agents to interact with each other whether it be through competition or cooperation and so because of that the world itself needed to be able to support many agents running at the same time we ended up shipping with nine very diverse kind of brains in the experience but the system actually allows up to a 100 agents um operating in real time at a time um everyone who starts a game like always wants to make it MMO it needs to support everything yeah sorry um so we also realiz realized that these agents would be like prone to accidents and prone to silliness right off the uh right off the bat like inherently they are pretty dumb in fact they start off we're only taking random actions um so the environment need to needed to be simple enough for a small brain to understand it and diverse enough that it's not easily optimizable but most importantly we want to encourage more generalized Behavior over were like over optimized

zombies uh so how do you encourage General Behavior how do you create a dynamic world the same way you do it for your players uh you you add the Dy dynamism to kind of a component of the world and then you you stretch that out so what we ended up doing is we we stole a a page from the book of Minecraft and uh we created items uh which are these kind of uh uh things that can exist on one of the blocks within the world and the they allow the agent to perform Dynamic Behavior without the agent having to be dynamic itself the you can have the same agent perform different types of behavior depending on the item that it's holding um yeah here's a couple examples there's like a fat an axe whatever that is at the bottom there's a dog these are all items within the world of little learning machines and then the world itself is supposed to be editable by the user the main uh user exper uh like the user journey is to kind of create these environments in which your agents learn and we wanted to make the environment as editable uh by the player as possible so we we took that concept of like boxal and items and

stretched it to as much as we could uh but there was another reason we did this and this is probably one of the trickiest parts of the talk is because making the world this way lets you train really fast um the the world is actually not 3D it's a 2d type Soo bound tile type grid um the little modifications in that there's like one item per cell and then the agents kind of move around within those uh within those 2D cells there's height but the height mostly is used for navigation and uh there's no tunnels allowed and the reason we did this is so that we could a discretize the space and discrete observations for neural networks are much faster to train than uh than analog ones and then the other thing is that uh or continuous ones the other thing is that um we it allowed us to um kind of reduce the dimensionality of the space um instead of having to store 3D information and having to do uh a lot of the reasoning of the agent having to do with like 3D information uh it allowed the agent to make decisions on a 2D uh space which is

uh again trains a lot faster these are and there was no kind of end problem to the user the user still sees a 3D world but the agents and the AI train a lot faster because they're able to work on reduced Dimensions so uh the representation drove the design uh but not only did we discretize space we also discretize time um having an agent make a decision every like every couple of frames was um not possible for the kind of scale again don't make an MMO the kind of scale that we were hoping to reach with this kind of game and also uh having it run on the players computers and uh the the more we kind of discretize time the faster it trained um so what we ended up doing is we ended up dividing uh actions into turns so every time uh anything in the world has an effect on other parts of the world uh any item has an effect on an item or an agent has an effect on an item uh these all happen uh within turns and that lets the um that's the evaluation of the the simulation happened much much faster um

yeah but not only that but the the key kind of focus on kind of making training fast was also the design of the agents themselves so the design of the agents um the agents in Little Learning machines uh we call them Animo um and we wanted to create like a Goofy character that could demonstrate kind of the the transition from non-intelligence to intelligence semi intelligence um uh their desire the design is like directly influenced by the networks themselves with their actions representing their body which you can kind of like you know like put some gloves shoes on or some eggshell um their network is kind of like their Crystal brain um and their observations are represented by their kind of like see-through bulbous head it was really important in the design that we like made sure you could see the brain at all times because it's kind of like the the kind of like core consistency of them um but the main goal was con con converting the discretized 2D space and the discretized kind of time space uh into a 3D representation that the

audience could kind of like understand and learn from um for a long time we didn't actually know how smart these agents could become and that was a huge barrier and a huge struggle so we created a style that kind of showed that they're fallible machines and quite dumb at times um they need to be cute silly and most of all forgivable because they're going to spend a lot of time running into walls and leaving the heavy lifting basically portraying their intelligence through the animation system um basically filling the gaps uh of the 2D occupant style brains and their turn-based decisions like filling in those gaps with kind of like animations that could communicate the intent behind those decisions um so that players could actually uh observe what they're doing and learn from what they're doing cool so I'm going to break down uh kind of some of the agents architecture U just uh really quickly um it's going to get a little technical but hopefully we can get through it in some some quickness um

the first thing is the vision as I showed you earlier the agents can see kind of an area around them some agents can actually see infinitely far away but they can only see a certain number of items uh and the way that kind of gets processed is that each of these observations um is kind of this stack of of kind of uh specific features that they can observe so uh for example this agent that you see on the top left can observe a range of 5x5 around them and within that range can observe maybe the terrain whether like the medium that kind of that uh that that cell has any items any creatures and any things that the creatures themselves are holding uh and then the each one of these features um turns into uh a set of uh of Boolean values uh there is there are still some kind of um uh floating Point values that get passed in but uh we found that the the more we stuck to again um Boolean values the the easier it was to train the networks uh originally we had a one hot encoding for every single item in

the game but that quickly blew up as the number of observations needed to grow with everything that was in the game So eventually we landed on kind of this like 20 Questions model which are these handcrafted embeddings where items that are similar can share similar uh embeddings and this uh this also actually sped up training again uh because it allowed uh certain uh it allowed an overlap in the types of behaviors that certain agents were able to develop so if you want want an agent to not step on flowers and not step on on uh fire then they can generalize that behavior more easily through finding commonalities in the features uh these so all of these kind of like uh bits get stacked into a big stack and they get passed into the network now we we we put we these diagrams are in the game we try to explain this to the users to some degree to the point where they can be confidently curious about some of the agents that they train with but uh we really don't think that this is that important really it the the point of adding these

different kinds of like learning to the game is so that the players get a sense of a different Taste of the kind of uh like representations of the of these different approaches at uh at kind of collecting observations for an agent and uh that's kind of how the whole game uh plays out different agents have uh differences that kind of just make them feel a little bit different uh as I said all of these observations get like tucked onto uh a network uh and this network is just a standard neural network um it's very small you'd be surprised at the at the level of intelligence that uh these kind of little networks are able to show uh our smallest networks uh for some of the first agents we really want to prioritize training there's two metrics that we want to prioritize the first metric is time to First intelligence like the first like as you saw when the agent was collecting crystals it happened really really quickly and we need that to hit fast so that players are confident in the fact that training that they're actually learning uh but then the second metric that we try to balance against is maximum intelligence so smaller networks are not going to be

able to do some of the more complex tasks in the game for which you'll have to use some of the more complex networks uh so the our agents that we shipped uh uh vary from 128 uh like it's full two fully connected layers of 128 neurons and then some of the later agents uh just have 512 neurons and four layers and again fully connected there's nothing too special but just with this alone they're able to do some fairly complex behavior um if they're trained appropriately uh one interesting thing we do is that every these agents are trained using proximal policy optimization we actually use the ml agents trainer we kind of stripped it out and reused it for our own purposes uh so uh one of the things that we uh that we do is we reset the critic every time we restart the training you would think that this would result in slower trainings but actually as long as the policy is accurate enough the samples kind of converge fast uh fast enough that you can you don't notice a drop that much and it allows you to kind of fine-tune your a your policy towards uh the direction which you want to to to take more easily uh lot of tech words

but finally after we take all of those observations we put them into this kind of like Network um that you you can either reset or train continuously they result in actions which is what your uh character actually does so in uh in kind of uh deep neural sorry deep reinforcement learning uh agents have or in reinforcement learning in general agents have a set of actions um in this case we've divided it to uh one a single action per turn and this again allows us to uh normalize kind of the the uh the set of things that we're training for in agents for example we had multiple outputs but here we're actually just looking for one and the way I like to explain this is you kind of have a controller and uh you give it to to your little sibling where you're playing with them and instead of like holding the controller with both hands they press one button at a time with their finger that's basically what an agent is doing it's pressing one button at a time uh so every turn it's taking one of these uh seven actions as what what it wants to do and here's where another kind of

trick and optimization happened that let us uh speed up the training which was um converging actions in to have different meanings depending on semantics originally we had more actions but we found that the the that was really one of the big limiters on the kinds of behavior that we can train and we found that by having actions perform similar uh types of uh have similar results depending on contextual um contextual situ ation it allowed us to put the weight of of training not on the action decisions but on the recognizing the different situations which meant that um we could reuse the latter parts of the network to optimize for specific actions sorry to to reuse specific actions like for example the same kind of part of the network can be used for like watering a flower than it is for chopping down a tree and it allows you to train an agent to do multiple things and uh the movement actions were also something where we found quite a bit of success by using these kind of tank controls and instead of kind of uh like uh what's

called cardinal directions um it allowed us to come uh exploit symmetries in a grid so you only need to train in One Direction and then all actions are kind of relative to that direction which again was exploiting another symmetry for uh so that you instead of it actually sped up training by like by a factor of three uh which was uh pretty pretty useful uh and it um it let it let agents kind of develop egocentric views of the world and it's a little bit harder to debug because you when you're looking at kind of the agents observations you have to take into account which direction they're looking at but um it it does result in better um like faster training times um yeah so those were all kind of little tricks in designing your agent so that can train faster but now that we have this kind of uh world and we have this kind of agent um collecting crystals is going to do it what's kind of like some of the things you can expect out of an agent that's built like this and uh I think Nick can cover the first time we saw that um so this is sneaky he's my all-time favorite agent this was trained

during the Prototype um I love him a lot but he sadly no longer exists um he's an agent that was trained like originally to use multiple items um to solve a puzzle as you can kind of see um what he's doing is he his goal is to collect all of those flags um and he has access to two um tools at once he has a block maker to make bridges and a axe that can uh break down down those blocks um and if you notice it I think he doesn't make a single mistake he does it in the most optimized way and this is why I love them um but sadly when I put sneaky into a different environment with the same puzzle uh he does nothing and he's basically overwhelmed with indecisiveness or something um at the time I had no idea kind of like what had gone wrong like what did I do wrong but obviously it became clear that he had over optimized um for that previous

environment and so rather than understanding anything about the environment understanding kind of like the IDS of the the different kind of items and stuff uh sneaky had purely um basically remembered a set of actions in a row basically kind of like a cheat code from a Sega game or something um so obviously we would have to address this and make this clear in the design of the training experience itself um so onto the designing of the training experience um our main goal was to kind of like Faithfully recreate training uh reinforcement learning as best as we could um and we wanted players to have as much control over the simulations as possible as we could let them um so we based it kind of like just like how in real machine learning and Robotics there's a simulation space and then there's a real world application we designed the experience so that we would have a training space which is the cloud that we showed separate from basically the ACT actual World they live in which is uh the sandbox main island and the

quest islands that you kind of put them in um and this meant that we would have a training set versus an evaluation set and this was inspired directly by what we saw with sneaky is that um I was training sneaky um in a very specific environment for like a long time and at no point did I take sneaky out and just test him somewhere else at no point did I evaluate his behavior um so we basically created a training from this um we kind of like base it on a process of tweaking and observing and tweaking and observing um starting with tweaking which is creating a simulation space setting up an environment setting the rewards setting the end conditions starting the training and then watching the training happen like watch the uh ano uh kind of collect its rewards but most importantly watch Its Behavior see its intentions is it doing something different is it having problems even getting any rewards at all um is it

bumping into walls or is it literally just running off a cliff um which happens a lot um and so we found that actually watching an agent train lets you train them better and that even if you don't know much about the theory of training RL like like me I'm very new to this um through a process of tweaking and observing training becomes actually quite intuitive um fun we we didn't know that this was the fun part of the game until very late into production exact yeah exactly um so in this slide I can uh just talk about like basically setting rewards a lot of this experience and a lot of part of the tweaking is setting appropriate rewards for the appropriate environment so basically we um created this kind of emoji like system where it's kind of easy to kind of set different very specific rewards based on the kind of items that exist in the simulation space we kept the um the rewards to five at a time because these are small networks and as soon as you have more than five Rewards it can very quickly confuse them if an agent is

gaining five rewards that are kind of like opposite at the same time they're going to break they're just going to get confused obviously um and that's not actually a big deal because you can you know always play it observe it remove a reward um their networks keep their shape so you can constantly change them while you're tweaking and observing and also fundamentally you can get some of the most complex Behavior with a single reward in a really well-made environment um and yeah so that's setting rewards and while you're actually observing we wanted to make it important that you could see the rewards being kind of accumulated at the right time in this case this little guy is receiving a lot of rewards because he's receiving just one for holding on to the block maker to make sure he doesn't drop it um and then a lot when he's kind of uh using the block maker to make a block at the very specific height to gain crystals or to collect crystals so you can kind of see and debug The Experience just by

watching them and seeing them gain these rewards um and it allows you kind of like figure out if you need to change anything and then so after that you gain these checkpoints so all the simulations are kind of being combined and kind of crushed in the background we got lots of running in the background and so we're averaging out kind of like all the different rewards and we're sending them back in so that you can see kind of like the average growth and progress of your uh of your ano at the same time um so in this video you can kind of like see me like just scrolling over this is a lot of checkpoints collected and you can see the averages of the um the rewards being collected changing over time and most importantly you can kind of like see the ratio of the rewards changing over time which actually became kind of like that's like top tier players looking at the ratio change um but uh on top of that just going over end conditions really quickly again we really needed to kind of like allow players to control

how the simulations were being reset um basically the first way to do that is how many turns in a single simulation you can run you can run it at 50 turns or up to 200 turns and this really depends on the kind of behavior you want out of it um and two we wanted to end a simulation if a specific item that of you're choosing no longer exists and this is basically to waste no training time training time is precious when we're doing this and we also didn't want you uh to have a chance or the uh the ano to have a chance to collect kind of like straggler rewards when it already completed its task and that's a no-o as well that can that can lead to awful things and that's kind of like based on back in the day when we had this Runner we didn't set the end conditions properly it was able to trick the others to fall off and no they could no longer collect rewards he could just hog them all for himself and we don't want that we want them to cooperate we want them to play um and then finally one more thing um if you remember sneaky and the

optimization problem that's still potentially a problem here like fundamentally you can kind of start a training and move an acts a bit further or tweak the environments to add a bit of noise on your on your own but by starting and observing and stopping and starting that actually process becomes like quite like quite a lot quite quickly um so we designed kind of the ability to kind of create a bit of noise in between resets um sadly we call this wiggle and we can't ever turn back um but basically players can control how much they want if they want their environment to change in between resets if they want the uh agent or the items to kind of like change place change places just to add a bit of noise or uh how much height they want to change if you can kind of see here I'm just like restarting the simulation and everything is just slightly moving away so it's not able to memorize that I need to walk forward pick up this block maker and then walk forward again uh and use it so just really trying to get them to operate in the training space but then

operate in the real world with actual like Integrity so um and uh just to kind of break down what's happening while you're you're seeing these like kind of uh uh Loops happen uh even though you're seeing kind of the training happen in front of you the training uh doesn't run in unity itself uh we actually started by doing that uh but there was a lot of problems the least of which was we had to write our own trainer and that that that's harder than you may think um the other thing is like sharing resources within the same process can often get in the way of the game itself and we wanted to keep the game kind of running smooth while the training happened so we ditched the whole thing and sent it to another process um so this uh what what happens when you hit play it actually starts a python process in the background which actually does the training uh and originally we were as I said we were using the ml agance Unity trainer but turns out running Unity is really expensive so so we got rid of unity too um well we didn't get rid of the unity that you're seeing when you're playing

the game but we got rid of the uh actually Unity for running the simulation uh what we ended up doing is we ended up taking all of the logic of the game and shipping it to python so that the logic can run in Python completely uninterrupted making use of all of those optimizations we we've been mentioning throughout the talk um this made it so that the the the training runs in like blazingly fast when you're watching a single like Animo do a motion it's performing hundreds and hundreds of steps uh it takes like 100 steps for every step it takes in in in game and that varies by your computer but even on the slowest computers we're able to get a significant speed up by doing this this actually extremely laborious and I go into detail uh later on but uh in the in the paper you'll see at the end of the talk but it it took a lot of kind of engineering to embed uh C into Python and be able to run the same logic that we used to run the game within the the python uh like kind of back end and the other the other thing that this does too is that if we ever wanted to kind of Shi to mobile it wouldn't be tooo difficult to kind of take that architecture and

just send it to the cloud and then just you know have a mobile game where the training happens in the actual Cloud not just the the training Cloud so um yeah these are all like uh this kind of architecture let us really speed up training and uh and I I should have put a metric slide but I'm just going to tell you uh that sneaky behavior that you saw that that Nick trained it took 30 minutes to train that like to get someone to build a block maker and then like use an axe to chop down kind of either trees or obstacles um and that that wasn't even a generalized Behavior Uh in the launch version of Little Learning machines you can get that kind of behavior in like 5 minutes um or less and all the time you're kind of watching your agent kind of get stuck you have to change a couple things and you have to run it again and it it creates a lot more of an interactive experience which actually makes it viable as a game mechanic in in our opinion so just to kind of summarize all of the tricks the big bag of tricks on how to speed up your RL so you can put it in your game and have your players experience it while you're playing uh number one discretize your space it's much easier to make decisions about a discrete space

for example if this was a standing room would you know where to sit or stand it's much easier if there's chairs that are discretized you can make the decision for yourself um number two reduce your Dimensions as much as possible if uh you can flatten it do so because uh making choices on multiple Dimensions increases the uh the the arity of the computation you have to do which uses more uh more neurons uh discretize your time uh the less decisions you have to make the better discretize your actions um using uh kind of leaving some of the intelligence to the animation and navigation systems whether they're using a nav mesh or a grid um it kind of really reduces the kind of uh like work that your network has to do exploit symmetries in your world or design a world that has symmetries so that uh your agents can uh reuse learning that they've done when facing north and they don't have to relearn it when facing south this is especially useful if you have visual observations that use lighting data um um next uh exploit overlapping actions

make if for example this is like contextual actions in video games just like just like players networks have a limited amount of verbs uh so if you can use those very wisely it re really reduces your uh training time and finally just optimize your code uh that's not really something that I can give you advice for but Steve here has a lot of a lot of good advice for you if you need that and then uh finally a whole deck of cards uh but uh yeah and uh yeah that's that's all all the that's all the tips and tricks on how to make a game that has RL happening in real time in front of the players um the last part of the talk it's about making it into a game this is just a little design Indulgence where we talk about kind of some of the reasons we made it into the game and why we have this little training experience that's very like kind of packaged novel but what we were really kind of like struggling with especially in early prototype is uh giving players a sense of well why should I train or what should I train so we kind of decided to go both take two approaches on this uh and kind of we were divided at the beginning but we decided on doing both

the first one is kind of like a player driven uh in uh intrinsic motivation where we give the players a bunch of items and tools and we create interesting interactions between them and we leave it up to the players to explore those interactions out of curiosity or a desire to see their machines grow but then also to kind of challenge the players and kind of give them a sense of hey you can do this with an agent we've tried we we've been able to uh we give them these things called quests so we kind of divide the game into two um so both of these are provided through islands and the it kind of creates an extension to the training Loop we mentioned earlier um there's this idea of a player progression where a player reaches a new island they get new items new Animo maybe new rewards to train with and then they kind of get issued a challenge whether that's intrinsic or extrinsic intrinsically they're like oh there's a skull here I wonder how this got here or hey this thing makes fire what can I light on fire um and then to be able to use that item the player can't directly interact with the world they have to train their agents uh this is similar to what black

and white did for like some of creatures that they're the only ways in which you can achieve certain elements in the game um and then but then also we kind of provided these quests which is like hey do this thing which we you we know you can do and that gives players kind of the motivation they need to kind of not get um the like uh be like oh I I don't think I don't think the the networks are smart enough to learn this uh by guaranteeing basically that they are um and then this kind of ties into training so the whole thing is like the player goes into an island figures something out they're like oh I want to do that and then they go into the training Loop and then they're like oh I got to tweak observe is my agent doing the thing I wanted to tweak it more it's ready bring it down to the island and have a have a go at it and then unlock more things and it kind of goes into this Loop and the islands are kind of made to kind of increase like sorry unlock increasingly complex items but what this is actually this is actually illusion because what we're actually to do is show players different reinforcement learning skills so if you're trying to

do reinforcement learning in an academic setting there's like a set of like things that you have to understand that aren't necessarily immediately intuitive from how you would for example train a dog like you have to realize that oh your agent can kind of get stuck doing a certain thing or you have to teach it to do something before you teach it to do something else some of these are kind of can measure it with training like an infant or a dog but there are specific things that we think are like unique to neural networks for example uh this is the first Quest and and we we don't want to overwhelm players we don't want to introduce randomization right away so we introduce the dog and the dog kind of has randomization built into it for free you pet the dog and the dog runs away he's like come chase me come pet me over here and that's basically recreating the same problem for your agent and it's doing the randomization for you so you don't have to modify the environment you don't get stuck in the same trap that sneaky got stuck in and then the kind of the the dog does that like that that that skill for you and you don't have to learn it as a player later

on um what's interesting is the second Quest you have to pet the dog in those cacti over there and the first thing you notice is that your Animo that you have just trained to pet dogs is starting to pet cacti guess what it can't distinguish between either of them because the cacti were not in the training data you didn't train an animal that can pet dogs you trained an animal that can pet anything in front of it so it's gonna start petting the cacti and being like I can't do anything with this uh so as you can see uh this kind of the these little like skills allow players to become more familiar with uh with training but also with kind of some of the intricacies of of AI such as like for example bias in data sets or or being able to kind of shake up things a little so in conclusion um basically what we've learned of what we've been trying to tell you is something you might already know that watching something learn is an amazing amazing experience like whether it's like I don't know watching you guys learn or uh

like a family member or a pet rat or something or an ml agent it's it's fun to watch something learn especially when it's happening in front of you and that by using careful design uh to train you can actually train agents in real time as a gameplay mechanic um and we believe audiences are capable of this uh it doesn't have to be a niche thing um it can be understood like any other game mechanic in time it's kind of like what we always hope Pokemon kind of like promise us when we were children tamagi something that could actually live watch your thing grow watch your thing learn it actually learns it actually grows um but we have a few Reflections as well like what did we learn from making this game um first off learning should have not been just real time but it should have been real time and continuous um we wanted to mimic The Experience generally used in reinforcement learning that we talked about the simulation space in the real world space um but in doing so we limited them to kind of perform TKS of the users's desire um rather than having

the agents actually potentially discover behaviors on their own um in the future it would be nice to like aim towards letting the agents teach us as much as we were inflicting our like our agents on them uh but and and fundamentally one of the problems that was in agents was that we talked about um was you were just watching these pre-trained models and in fact we kind of like repeated this mistake in the learning machine Mach to some extent because all the action really happens when you're training them in the cloud and that's where you really find that like that lovely space and when you're actually showing them in the quest islands and stuff you're just watching pre-trained models and watching pre-trained models is not nearly as fun as like watching them learn in front of your eyes if we did it again they'd learn all the time yeah they would learn all the time they would be continuous um and that's possible also it's okay to use smaller in older models like bigger isn't always better um these these days when people talk about Ai Ai and like I'm sure lots of people are talking about AI here but

they're often talking about large language models with billions trillions of different neurons and parameters and but we believe that understanding like the foundations of AI can actually happen with a lot of different scales of models and we believe that there is like kind of a beauty to like the small micro examination of it and that you don't need to use the newest tools you can actually just use uh the best the the tool that works best for you and just go deep into it um and then finally like while working on little little learning machines Dante heard me talk about this a lot it was always kind of like the idea of like the fruitfly and if any of you have taken like a biology course in univ University or anything you know that we study the biology of a fruit fly or like a small worm and we kind of like look at the neurons and look at all the factors of it in order to kind of project into a larger scale and teach us kind of like the fundamentals of you know what some pretty amazing cool science and in this way I think that like playing with AI at these smaller scales actually creates a literacy and

intuition about um AI systems that are kind of like becoming ubiquitous um all-encompassing and somewhat scary um yeah like last week I was talking to someone about embeddings and it was so much easier to say this is how an Animo sees a flower and like this is how like your large language model see your words they're just like bits exactly and that like using real-time learning as a game mechanic like makes this sort of understanding and experience a lot more accessible and fundamentally like I think we believe that this base level of understanding is kind of more important than ever now that we're in this kind of like AI world or um so thank you thank you for coming thank you for listening to us and going on this like weird thank you for staying till 6:30 yeah thank you for staying for 6:30 I'm sure there's lots of fun cool things going on and I hope to see you out there um also thanks to all the people that are listed here um we couldn't have done it without any one of them uh and uh um yeah and to the organizers at the AI Summit the good first day I would say very interesting

talk so yeah great stuff so I think I know everyone wants to just get out of here but uh do we have time for questions maybe yeah five minutes maybe oh six seven minutes but that's what we're aiming for um and if there are none that's okay too go get a drink uh I think we have one over there oh awesome s thank you for the wonderful talk um when you're doing your simulations are you under the hood are you doing many worlds in parallel or are you just doing one single world so we started with doing many worlds in parallel and we tell the players we do many worlds in parallel it's much better for cash locality if you do one world at a time um i' recommend you check out the Min World simulation paper at sigraph 2023 from my labmate um Brennan shacklett um there's some work on high performance GPU simulation of games using ECS systems yes so when when we

first designed the system the reason why it's grid based is because we wanted to do a GPU based simulation but man that's engineering is hard man but well let's talk let's talk cool cool cool thank you uh yeah can yeah yeah let's alternate sure uh really wonderful talk I think this is wonderful work and um maybe one of the largest repositories of RL Behavior or trained RL models in the world so almost well I actually think it's true and it's it's beautiful um you mentioned old techniques um actually about 20 years ago was a game called uh Nero um it was you oh you guys are aware yeah so oh it's on the slides it was on the first SL oh I missed the first slide sorry about that um was that you guys no no no uh my adviser Ken Stanley so but yeah yeah um so I was I was going to tell you Julian's also done a lot of work in evolution I'm wondering if you tried something like that because you know 512 by4 even you

know fully connected layers that large uh neuro Evolution techniques kind of go from the groundup kind of approach where you take as few nodes as possible and grow them um you know many ways to skin and Earl network but so I'm going to try to get ahead of a couple questions here often when I talk about this game people ask hey why not tabular RL why not neur Evolution and uh so we one of the things that we really wanted to emphasize is uh not captur the whole problem like within the network itself you're and this allows for the network to develop more complex Behavior but more specialized behavior that is not tuned to the whole problem uh tabular RL grows in Dimensions really fast and then neuro Evolution has the problem where you kind of has have to start from like basic principles you have to teach it how to navigate first whereas uh neural networks just kind of learn everything at once which uh well at least in in our in our experience and it turned out to be the the the best mation for us to kind of follow in this um hopefully that answers your question but hopefully it

matches your research as well if you've done it um like like black and white um you've got the game play I'm sorry RL is part of the game play um and have you uh uh your QA team might want to talk to you about other people want to put learning in games please check with your QA team about learning after ship and make sure that it's the player that controls it and not the game so your game doesn't go stupid or they accuse it of cheating because it got too smart because it figured them out um any comments on that at all um I'm not sure if I fully understood the question sorry um it it's the decision of if you're learning in the field yeah is it a gameplay element or is it hidden and here you have it exposed and it's under player control MH um uh which is a good thing for the technology because then your QA team won't leave you in the parking lot one night when it you know goes stupid in the field otherwise because a player did it and it's the player's problem if they

teach it stupid or if they teach it too smart super I see I see you want to talk about like the problems Qing this game and yeah this that like I I think you're like bang on and and uh like creating a game like this is extremely difficult to QA and we like like we were quea it a lot ourselves before we could even pass it on but fundamentally that so much can go wrong and it does go wrong that you kind of can't catch it and I think like there's like a lot more work to do in this space in order to kind of like find kind of like where these behaviors actually kind of like break and how to more than just QA it communicating properly why they broke and at what time and I think like we tried our best in some elements of this but you we could have made spent five more years doing that one funny story is that one time we for a month the game agents could not see like deep water so the agents kind of disabled when they fall into deep water and they

couldn't see where deep water was and they still worked because they were able to kind of memorize a lot of the maps that you train them on uh so that was a moment where we were telling QA like oh no they should learn you're just training them wrong and q's like no they're they're really stupid and then we go in there and we look at them and they're like oh yeah they're they're broken sorry they can't see water they can't see water our bad just but uh but yeah no good call I think one more question and I think we got that side next nice hey guys um great talk great work and I first of all there's lots of like you know particular technical things I'd like to comment on from having done some very similar stuff myself but I'm going to refrain from that there's we can take a whole long discussion over the overfitting problems and so on and and and uh GPU paralyzation but first point is like not a question I think it's awesome that this happens and I it must happen because this is the kind of work if we're ever going to use machine learning

in runtime in games in an interesting manner we need to design around it we need to leave old paradigms that are designed for the lack of AI and we need to leave the thinking that we can ever QA it we need to be able to step out into the Bold new future and do things that are built on abilities not on lack of abilities of AI so I think it's great work um the question here is like shouldn't this be much more of a social game and why should and and the question is and and the reasoning for this is that there's always going to be a tension between um wanting things to learn fast and that learning something that is interesting is always going to take a long time um at least in interactions and isn't this something that should happen over kind of a social time scale between players I I love that question cuz I think that's just a great idea and like kind of like what we why we want to kind

of give this talk um we we kind of like framed a lot of this game over kind of like teaching fundamentals because um and maybe we did our best at teaching that and I think if there is more of a base level understanding of reinforcement learning you can kind of like quickly get to kind of games and experience like social games that you're talking about because I I totally agree I think that is kind of the best use scenario for this kind of uh project um and I got to like end but thank you so much and I totally agree with you and thank you [Applause] [Music]