Engaging Bots for Multiplayer Games

[Music] [Music] so with our first speaker we have christopher holmgard of model.ai with a session on engaging bots for multiplayer games christopher i did see you where you going there you are lovely stuff you're all good to go yeah we're good to go all right take it away thank you very much oh there we go those are the slides um hi ron it's uh exciting to be somewhere in person today and to have the opportunity to speak to you around engaging bots for multiplayers and for multiplayer games my name is christopher i started a company along with other folks called

model ai and what we're doing in model ai what our business is about is essentially we're building what we're calling an ai engine it is essentially a system for delivering ai bots for all parts of the game development launch and live operations process and in the talk that i'm what i'm giving today is about one of the many use cases that you can do with with artificial intelligence and an ai engine specifically today i'll be talking about the little third graphic on the on the slide here player bots so what it means to deploy and implement player bots that play against players in your in your multiplayer game specifically i'll be i'll be talking about how we did that i'll give you a case study um for a game called fortune fight this is a game that came out on steam in in 2020 and we work with developers implement some multiplayer bots that when you sign on to the game you can you can play against other players or you can play against b spots inside a fortune fight

and i'll speak to you specifically about uh the needs that the developers were facing how we addressed this um i'll talk about sort of like how we identified what kind of box to build and how we went about it i'll go into a little bit of technical detail as well i'll talk a little bit about how we balanced engineering and uh and machine learning so good old-fashioned classic ai and uh and then some some modern machine learning stuff put in there and i'll talk a little bit about what kind of an impact this had for the game in terms of engagement and retention and then sort of like building on jumping off from that i'm going to be talking a little bit about where we sort of like see multiplayer bots headed in this space that sits between classic ai and machine learning in the future and uh and hopefully um you'll have some questions we can have a discussion about that so introducing fortune fight uh if you're not familiar with it it is a team-based game it's a four versus four um player game um so you set up in different kinds of teams and you are deployed into these sort of wacky arenas

where you you face off against another team so like classic action team-based gameplay uh but there is a little bit of a twist in fortune fight or actually a couple the the first twist is that you are fighting both um with melee dog weapons and you're also doing rage ranged combat at the same time and you have to switch between those two things the other thing is that these these melee weapons that you're using you actually build them yourself so what they the way that the game works is that over time you collect different kinds of cards that will if you give you different kinds of components for your weapons and when you put that together and and that becomes a weapon that you're bringing into the next round so so the way that the game really feels to play is that you're sort of like starting out with a basic weapon and then over time you build it up to crazier and crazier weapons for your own designs and all sort of like unexpected and crazy things happen in the game so what you're seeing here is sort of like a very typical way of playing the game this is what a typical session would look would look like you have some players here they're

trying to to capture you know a point here on the level and everybody sort of like converges on this and you have these big wonderful scrums of people engaged engaging in in malay combat and uh and you have some people supporting from afar as well that's that's what the game's about it was built and launched by flame bait games and what we did for them was that we we put in the bots for for the game so when you're playing the game and you don't have enough human players around or you might not have somebody at your particular skill level well you'll be playing against one of the bots that we built for the game a little bit about the fortune fight i talked about the gameplay just a second ago so it has three different game modes you can play capture the flag you can play elimination where it's just about killing everyone else from the opposing team or you can play push for card which you're probably familiar with from from other online team based action games we have the card based weapon building and i'll talk a little bit more about that in a second and and the game is made in unity but it is built with quantum from from photon

and if you're not familiar uh photon engine and quantum has a bunch of like interesting characteristics from an ai perspective and as a bot developer the four things that are really interesting to me is that the the engine is deterministic so if you record something that happened inside of a gameplay session you can actually replay it with 100 fidelity which is absolutely excellent for training bots for games if you need to run game simulations you can run it without running the unity game engine which is pretty interesting as well if you're into bot training um it saves a perfect replays which is something that'll be that cover really comes in handy when you need to do bot training i'll show you that in just a sec and it is very very fast which is something that you like if you use machine learning for game development as well so that's that's sort of like the stack that the game sits on and and building from that we had a conversation with the devs over at flame bait we asked them you know okay so you're building a real-time skill play skill based multiplayer game what do you really need from the boss what's the thing that you would like as a game designer what's the thing that you think would make a delta to your

game what would make this more exciting to your players and the first thing that came up of course was that they said well listen we're building uh we're launching a multiplayer game it's team based and we're gonna need to fill up these teams but game the game is actually only you know super great when you have maybe four versus four free versus three that's when the game really shines so we want to make sure that you always have players available to play against you and can you help us with bots for that and and we said of course and these spots will also need to be uh working in absolutely team-based fashion you can't just have them going rogue you can't just have them pursuing their own objectives they uh they need to to to work with uh with the other team members even if you're just like one human on your team and looking at ads are like what does that mean for user experience in broader terms of course what what they're looking for and what what they were looking for was in particular getting a really positive first time user experience because when you jump into a multiplayer game like this obviously if you're not having fun in the first like two to five minutes and you're just trying out the title you're not meeting any other players um you're probably not gonna

stick around um so really sort of like the two the two things that they were driving at the two things that they were looking for from the bots was was the right game feel to give you that uh positive to tui and making sure that you always had someone to play against um so so after we took a look at that and we had a look at the gameplay here's another example of what one of the maps looked what that looks like and what robots look like uh we sort of like went ahead and had a conversation with the game developers and looked at the tech stack and talked about how to implement this um we started by by having a look at the at the core loop of the game and there are really two parts of the core loop uh the first part is where you craft your weapon so this is where you collect sort of like all the interesting components that you're going to be putting onto your weapon you're seeing i'm i'm building something here which is it has some sort of like smider going on and maybe also a flamethrower and maybe it needs like another blade on the side um and so when you're a player that comes into this game you'll be building these weapons and putting them into fortune fight for you to play with

now interestingly the bots need to do the same thing and so um what we were really interested in in this case was figuring out a way where we could have the bots construct their own weapons and doing it doing it in a way that seems really plausible so we wanted them to build weapons the same way that players would build weapons on top of that they needed to build weapons that would work in the lane range combat and they needed to use some collaborative collaboratively so the question was of course you know how do you get your bots to do these things um and the approach that we took with this was that we actually taught the bots how to do this by looking at how the human players did it so rather than trying to come up with just you know our own algorithm or trying to build like a static system we built a system that would ingest the data of how the current players the soft launch players were doing it in fortune fight and then would learn from that how to build weapons of their own that were sort of like plausible reasonable and would work in a team-based combat setting the way that we went ahead and did this that was essentially that we took the

different parts of the weapons so you know for instance a tomahawk might go well with the spring which might go well with some kind of like buff that really increases the damage of your tomahawk weapon and we'd look at that and say like that's sort of like a plausible set we take these different kinds of weapon parts we put them together and then we say these are sort of like this is a selection that you can have as a bot when we would pull from the player data and we would learn sort of like what would a human do if you gave them these elements what is sort of like the likelihood that they would put the tomahawk on top of a spring but it would go here all these different likes were like likelihoods of where you put the different parts of the weapons that's what we learned from the replays of the player data because as i mentioned before this was already sort of like a feature of the engine but we could pull this out from every single game match that was happening or we could take that data and use it for the learning purpose so eventually we ended up being pretty happy with this and that was like the first challenge of building the bots for this game making sure that they were entering the playing field with something that felt right and looked right and that meant that but we could move on to the bigger part of the challenge which is all about

the fighting in fortune fight so here's a small example of how the fighting can play out and in particular if you have two players against each other it actually gets a little bit intricate um before you were maybe seeing you had this like big scrum and you know you had eight players just whacking away each other at the same the same part of the level but once you have these like individual faceoffs in fortune fight there's a little more to it so what you can see here is that the red player is kind of like keeping keeping um the blue player away using the flamethrower and then trying to sort of like move in and trying to uh to whack them uh once we get close enough so doing this sort of like keep away approach hit go back keep away and that's exactly the kind of behavior that we were interested in getting into the bots but sort of like it's contextual you need to take sort of like the surroundings into account you need to understand what's going on and at the same time you need to like you know balance your actions between am i using the gun am i using the malay weapon am i aggressive am i defensive and all of this sort of like behavior of

course you can implement that using classic ai techniques uh probably you could do something like this with behavior trees as well but what we opted to do was try and capture this kind of behavior from what players were doing in the game and then teach the bots to do it from the examples that we saw from the players and essentially the paradigm that we implemented here was that we said okay we're going to collect player player data directly from what's going on in the game then we're going to train all the little nitty-gritty parts of the gameplay and the reaction and the combat we're going to train that directly off what the players did and then the top level parts we're not really that interested in getting the machine learning to handle that and that works better for two different reasons first of all it can be difficult to get right i don't know if you saw the news on uh sony recently did some very interesting work on gran turismo with the with the drivers where uh obviously there's been work in you know stuff like uh the starcraft ai or the stuff that open ai did in dota but what is israelite characteristic about these projects is that they require quite a lot of tweaking and a really really

large amount of training so the approach that we took here was sort of like it was was thinking well what if we take these sort of like high level decision making and all the strategy and we implement that in a classic ai fashion in this case we implemented it using a finite state machine and then we let the machine learning learn you know everything about like the combat dance and you know the small moment-to-moment choices that you were seeing in the video before so that's what we did and it turned out to work pretty well and we ended up with a system that was sort of like a mix between this classical ai and machine learning specifically the architecture that we had for those who are interested in that uh was sort of like a finite state machine sitting at the top and that's sort of like classic ai making all the decisions about should i go for a waypoint should i try and be defensive should i try and really like get into the mix and then at the lower level we put in the machine learning parts everywhere where it made sense essentially um the specific machine learning technique that we used and if you want to chat

about that later i'm absolutely available but i won't go into too much detail but we're using something called imitation learning or behavioral cloning it's essentially a way of teaching a neural network using machine learning to sort of like look at the situation look at the actions that the player is taking and then user like learn well next time i'm in a situation that looks kind of like this what would this player do what would a group of players do based on what i've seen before what would be the next natural thing to do that's sort of like the easiest way of explaining how behavioral cloning or imitation work imitation learning works so so we did a bunch a bunch of training there and implemented that we collected the data for that uh using frame by frame data out of the out of the multiplayer engine uh here's a small example of what this looks what it looks like when you collect data sets like this you can see for every match that happened in the game we'd be able to say you know exactly where did the player go second by second what were they doing uh what kind of weapons were they using what kind of actions were they taking and so we put that together and we had a look at it and we said like

how much data do you really need to get this like hybrid approach working because again you've probably heard that other ai techniques where it can take you know tens of thousands of hours to train and you need tons of computers it's very expensive but it turns out if you take this like hybrid approach to making these bots you can get a lot of these benefits with a lot less with a much smaller investment specifically what we saw was that for the behavior that i'm showing you today and all the behavior i've been showing you on screen where actually our bots except for the main player character which was me but all the other characters but you're seeing up until now they were bots implemented using the system well we could get this kind of behavior out of the box with less than 10 hours of gameplay data um and that's where like across a few players but you don't even need fastens uh with just like you know tens of thousands of frames of actions and training a network um in most cases it only took like one hour but up you know i don't over promise i said you know with less than five hours of training time you'd be able to build out these kind of machine learning models for these kinds of bots and what that really means is that this

is not necessarily a static implementation of bot behavior you can actually take new data in and you can start updating these on demand because if you don't need a ton of data you don't need a ton of training time this becomes more like sort of like a living system rather than a one-time implementation that you put into your game so if you put in new game features new weapons if you change your audience or if you have new groups of players that might be interesting well you can just take this approach sample a little bit of data from your player base put it through the training pipeline and redeploy the models and you'll get that represented in the behavior that you're seeing inside of your game and that means you can keep up with the player meta and keep the robot behavior relevant uh just giving you a small example of how it sort of looks side by side again you know this is this is more of a thing that you need to feel when you're playing the game um but on the right hand side here you sort of have more dynamic interaction between the bots you'll see a little bit of circling you'll see more intelligent use of the melee weapons whereas in the standard scripted bots and of course this is just you know you could script many different bots but this is the one

that our system replaced we were getting like less organic and more mechanical behavior out of us so always interesting to see these things side by side so the impact on fortune fight uh the developers were very happy with it they focused a lot on the fact that we seemed to give the bots personality uh they felt that the game feel got a lot better uh but perhaps most importantly after they put these bots into the game and shipped an update with them they saw a 50 increase in their day one retention for for new players coming into the game and so that's kind of like what we're also building this argument off that but having this kind of like human-like behavior in your game can really make a difference to how people experience it so a quick takeaway from from what we learned from this case you can definitely use machine learning even if you're building a smaller game and you can obviously also use it if you're building a bigger game um deploying these kinds of bots um at least you know based on this case study can directly improve your your game metrics uh and we think that it's it's a good idea to be smart about how you

combine classic ai techniques you'll probably already have these competencies on your team with uh with machine learning techniques uh which will like build in there for a hybrid approach and you can sort like trade your resources between how much of your your time do you want to put into engineering and how much you want to put into machine learning and then you need to find right balance there um thinking about this certainly a little bit toward the future uh what else could you do this with this well you're obviously not constrained to having one kind of bot for for your game um if you decided to train the bot on different parts of your player data maybe the novice players with medium players and be elite players well obviously you could build blocks that look like those different parts of your player base and if you're really ambitious maybe you split your player base into many different groups and you start serving those bot ups bots up to players of different skill levels in order to give an even better experience in the game um very long term i think it's it's also realistic but you can take the same approach because you only need 10 hours of gameplay and you could probably also train them to represent an individual player and obviously some game games do

this already uh you'll probably know that forza motorsport have something called drivatars you know killer instinct they have something called shadow play i believe but this is something that i think is you know essentially any multiplayer game would be able to do this using this kind of an approach um yeah so that's that's just for a quick recap we saw this impact on the player experience on the um on the two of the players uh resulted in better retention and the machine learning really provided better sort of like real feeling experience and there are some some update um opportunities that you get by implementing a system like this thank you very much thank you to to photon for having this track thank you to flamepaint games for for letting us present and thank you for listening great stuff christopher that was really interesting um we do have a little bit of time for uh questions we could probably take one or two so does anyone have any questions no no maybe yes

no okay no apparently not but um if you have any questions you can obviously contact christopher um either on his email or he'll i'm sure pretty much you'll be around i'll be around the rest of the day yeah or twitter brilliant all right thank you so much [Music] [Music] you