Session: Balancing Player Experience Without Players At Supercell: Case mo.co - Markus Ojala.

[Music] [Music] Heat. Heat. Here we go. Marcus Oila from uh Supercell. Balancing player experience without players at Supercell with the case study of Moco.
Thank you. Really happy to be here. So going to talk about that exactly balancing player experience without players and that's the about our new game Moo. This is actually mainly work done by my colleagues. So, but I'm really happy to present their learnings about the game. So, I'm Marcus. I work as a data scientist in Supercell. Now,

just about a year in Supercell, but I've been doing machine learning on large scale in the industry for last 15 years or so. So, in smartly IO and Unity before Supercell. So, I guess most of you know about Supercell. But what what what is superell mission? Uh it's actually to u create games that as many people as possible play for years and that are remembered forever. So that's what we are here for doing. And if you look our games, so we were founded 2010. So we have now seven live games. And what are we going to talk today is about our latest game that was launched uh earlier this year and it was kind of an invite only launch. It's still kind of an like we are experimenting what's the game about. So it's a new game and if you think about our title so it's about optimizing the balancing the game

without the players. So we have been doing this work actually before the game was launched of of course applying those learnings even now. So how can you balance your game when your game is not even on launch? So for of course for our other games we have a lot of players but for Moo that wasn't the case when we started doing this things. So what is Mo? It's a multiplayer like players versus environment type of game. So you are whacking monsters with your friends. You're collecting loot loot. You're collecting points and you you have maybe your friends or randos there with whom you are playing. So I have actually quite a lot of technical stuff also in this presentation. So we are using reinforcement learning for doing this balancing. So what is reinforcement learning? Why do we need this? So our use case is game balancing. So if you think about the balancing so we have different items, different skills or

different things and uh like when we are developing the game our developers would like to know that hey is this item working? What is the right way to do? Of course, they can try to play the game game themselves, but without actual players, it's really really hard to start doing that. So, and if you don't have human players, we don't really have a way to learn how they would use all of those like gears that we have in the game. So, what is reinforcement learning? So, that's the actually technical way to uh solve this problem when you don't know anything about the game. So the beauty of reinforcement learning is that you don't need to uh give any rules for the players. So you could think about the dog even that like you are training a dog, you are giving a reward for the dog if if the dog does something good and this is the same idea when we are training bots with reinforcement learning. We like bots are basically randomly just doing something and if they do the right thing for example gil gill a boss or something we give a reward and say that do more of

this by utilizing reinforcement learning we can train these type of bots and this actually enables us to do that massive scale testing. So we can train thousands of AI bots and put those bots to play our like new game and then just observe how are they playing but what happens with all of those items that we have been doing. We can possibly if we are introducing new item see that hey all of the bots suddenly start to use that item and it's too good. So maybe we need to come back to the game design start balancing the game with with these insights that we get from the bots. And the beauty is in that that we don't need any data. So of course let's say that if we would have like human player data about the game, we could also use utilize other approaches. There's this imitation learning where we can learn how humans are playing. So that's other approach when you actually know how your players are behaving. But of course, if you introduce something new in the game, then you don't have that that thing. So you anyway need reinforcement learning

type of solutions when you don't have the data about your players. So, so our main use case is actually about that balancing. So, as said, we don't use the bots in Moco for actually opponents. We use them just when we are uh developing the game and introducing new items in the game. So how it actually worked during the Moco like development cycle was that hey developers were like introducing new ideas maybe every one or two weeks and then our B developers basically trained a new model uh with these bots and then saw how did these bots start utilizing all of those new skills then giving back the results of the developers. So it could be for example what was the play time, how fast they succeeded the level, did they use these items that were planned also visual kind of like just uh play out of the game and sharing those

back to developers and then the uh design team actually adjust this possibly gears and then iteration continued on. So here we can see one one example. So this is one fight. So we have basically four bots now trained that are actually pretty nicely playing in the game and we can see what they are doing. So they are going to whack the monster and collect points. So they are actually behaving pretty nicely. Maybe this is something that was intended to happen in in this game. This is pretty long video so we don't need to wait for that. But then the other one so the next one is going to be pretty short. So it's an exploitation use case. What means that we have a single uh like bot there who had actually like very good selection of gears and it took like was it 1 second to back kill the boss basically and that was not intended. So like the bot was able to find a combination of gears in

which like just one second he can kill the monster and complete the level. So of course like feedback about this B is that hey there is something off with our gears that please redesign what happened. So we didn't want that you can complete the level in one second. So technically how are we doing this? So we are building an AI simulator. So what is an AI simulator? So of course we have our game. So game consist of the front end that or the client facing stuff and then we have a server back end back end stuff but we are doing a separate build for this AI simulator use case. So it's basically the headless game client. So we are not rendering the UI what there is happening but we are still kind of simulating all of the actions that the player could do in the in the UI. So it's kind of basically moving and hitting a button and then we know exactly the state where the player is and then we can simulate what happens.

So if you move right we know what would be our new state and this is kind of the simulator that we have and with this simulator available we can actually run about 1,000 parallel simulators in a single GPU machine. So we can like parallelize training this and also simulating different bots what they are doing and the parallelism is actually very essential when you're doing the reinforcement learning. So you need to be able to run thousands of these spots simultaneously and then start training and okay the maybe challenge part of this is that it's pretty tight integration with the game. So we go actually the C++ code to like put our observations from the game and it's a very tight integration of that. So the downside is that if some developer actually changes the code our simulator might broke as well and there of course other ways that people might just take the visual thing but that that's then

the usually much slower approach. So now we are doing tighter integration with the game. But the downside is that it is the break because the AI simulator is not something yet that our like developers needs to take care on. And technically we are like doing some Python binding from the C++ so that we can do that simulation in Python and then how do we train? So it's says so we have that AI simulator that we can call in Python. So in theory it's pretty simple. So we have compiled our AI simulator which is that headless client in the Python wheel. We can just import that and what what does the AI simulator take in is just an action. So action is something that move right move up like hit the monster. Those are the actions possible actions and then the simulator basically just takes this action and returns the next observations. Observation is basically

everything about we know your state that what is your location what are your item balances the status but where are you and the reward is that we can give you some reward so if you completed the round you get a lot of reward if you maybe hit hit the monster you get some more reward and then we have that agent is the bot basically optimizing what should be the next action based on your current state agent returns that hey what should the bot be doing next and Then how have we built this reinforcement learning? So of course reinforcement learning is nowadays pretty standard solution but it's still complicated. So we have an in-house solution built on Pythons and that we are using. There are of course other available solutions that you can use gym environments for example. We also looking now the array a lip library that provides reinforcement learning solutions out of the box. So on the top right corner you see like that's a map that yellow yellow is a area where you can move and pink is kind of borders

where you can then we have some features about players. So more technical details how you start doing reinforcement learning. Of course you can take some of those libraries but all of the libraries what they usually nowadays implement is some form of proximal policy optimization PO. BPO is kind of a simple idea but then lot of technical details to make it actually working. So there is kind of this deep learning model in written in Python. So you have your observation basically observation is that the bottom right corner. So we have features we know about where the player is. Then we do some kind of encoding about those features. Then we have your time series previous features and then we are trying to predict your uh basically what is the optimal action and what is the value from this observation. And how the proximal policy optimization learns is

that you take some action. If you got a if if it was a good action so you got a good reward, we say that hey try to do more this. So it's kind of the idea that it does something randomly. If it was a good thing we say that do more of this. If it's a was a bad thing we say do less. And then the model kind of automatically starts learning. So it starts from scratch doing a lot of random stiff stuff. Sometimes doing good things we say that do more. And the model learns to put those like weights so that it starts to doing more of those good good things. Uh this bit technical but then maybe uh how are we doing then it like utilizing this model. So of course if if we are training this all the time from scratch it taking a lot of time. So when when there is a small iteration so usually we have some model tuned like trained on all of the data sets but then comes a small modifications to the gear. So we

can just like re uh retrain the model using like some small modifications and this saves us a lot of time and then when we have this model we have for example those gears. So then we are separately optimizing what gears this B is now utilizing. That can be done by simple genetic algorithms. That's basically just choose what items I'm or gears I'm now putting it for these spots and then we can start to see for example if suddenly all of the bots would like to use certain gear maybe it's too good. So this is kind of one outcome that we can show the developers that what is the gear like allocation that we are now getting out of these bots. Then if you think about the reinforcement learning so I mentioned about this rewards what are the rewards so this is maybe the most challenging part about uh reinforcement learning in general that what what is your goal what are you actually going to what are we re rewarding of course you can say that hey

let's reward only if you successfully complete the dungeon but that's usually too sparse so like what it means that if the bot randomly starts moving there it's really hard to complete the dungeon if you don't know get any feedback while you're doing it. So the simplest way to fix this is that you kind of start giving some small reward for doing something that looks okay. So for example the simplest idea that hey you stay alive like every time like like half second if you're alive we give a small reward. So basically to stay alive you might start start hitting some opponents and staying alive. Okay, that's the second. So, hitting monsters, you get a bit more reward. Killing monsters, a bit more reward, hitting a boss, killing a boss. And then finally, biggest reward for completely done. So, this is basically a possible way to solve this problem, but it's very tedious kind of manually tuning all of those rewards and trying to make it work.

So one for example challenge with this type of reward hacking is that okay what we observe we had this squid balance gear which basically turns the player invisible if they haven't attacked recently. So of course if you have trained the bots to like get some reward you that you hit hit even all kind of monsters they they will start to hit but the optimal human strategy is to put this invisible thing on and then just walk past as many things and this is kind of for the bots like bots if they get this intermediate rewards it's really hard to learn. So this is like then if we start moving a bit more complex rewards. So then of course the uh we can add some kind of exploration maybe give just reward for hitting the final boss like doing this. This is more kind of that we actually also exploring how to solve this but now with current things that we have done. So the bots start from something they and now they

enable that invisibility. So they are actually able to utilize that invisibility and just walk past the monsters without kind of hitting. So it's something that like the better you design the rewards and better me to utilize for that reward optimization. For example, just saying that hey it's important that you start moving. So encouraging bots to move as far as possible basically or explore the areas. So giving more rewards on that. So it's not it's not direct reward for kind of hitting but more reward of kind of going somewhere more interesting areas. So there are a lot of these techniques how you can encourage exploration for example adding entropy bonuses and or adding pseudo accounts but that's kind of also an active research in the industry how to do these things. So to summarize, so what have we been doing? So we used indeed this reinforcement learning for the Moco

game. So Moo is very new game. We didn't have actual players before we started doing this. Of course, now we have players and we could do something more. But then if you actually tune the gears, it's really hard to do it with real players. So we can really fast iterate on the ideas and do this game balancing. But if you look technically about reinforcement learning, it's still one of those things in the machine learning side that are not really solved. There are libraries, but in practice, you need still experts who can actually implement this. And I think it's very interesting problem as well. So our exploration on on this reinforcement learning journey continues. So thank you. You
Marcus, good stuff. Questions? Yes. anyone.
All right. So, thank you for the speech and u uh I'm I'm interested in hearing

that uh how do you address situations where the bot kind of learns a mechanic that works well but it's for humans really hard to execute mechanically for for example. So, how do you address those issues?
That's a very good question. So one thing is that okay mechanically you could think that you need to do something too fast. So one way we limit that is that we basically don't allow bots to do decisions kind of microsconds. So it's more kind of that you they can take actions every half a second. So that limits also like that they the combination of actions that the bots can take are something that the humans humans could take as well. It still doesn't necessarily prevent that there are ways that the bots could learn to do more. So then it might come also that when we kind of train the butt, we look what did they actually learn visually and then see that hey if this this is this something that the humans would like to do. If it's totally something that we don't want the players

to do, we might come back to the game design. But I think the main idea is to still limiting the like frequency of the actions that the bots can do.
Great time for one more question. Anyone? Thank you for the talk. Um there was this uh point that you reward for staying alive. Is it have you found that it's beneficial to reward staying alive instead of penalizing when you die for example?
Very good question. So what I mentioned so initially we started off giving a reward for staying alive. I think it's actually the same as like giving a penalty for dying because you can think that it's it's the same thing but at the moment we are not using either of those. So as as said so kind of we want to move towards the idea that we need kind of give a reward only for completing goal

and then we give reward from some of those exploration stuff. So getting away of this very shortterm goals that like for example didn't work this with this invisibility gear.
Great. Thank you Marcus.
Thank you. [Applause] [Music] Heat. [Music] Hey, Heat.