Designing AI for Killer Instinct

[Music] all right everyone hello and welcome to designing AI for competitive games thank you all for coming out we really appreciate it so first I'd like to introduce the two of us my name is Derrick Neill I'm currently the executive producer for maximum games but in a previous life I was working for iron galaxy on killer instinct and the shadow system among other things with me as Bruce Hales he's a software engineer for iron galaxy and the architect of the system so for those of you that don't know Killer Instinct is a fighting game kind of in the veins of traditions set up by Street Fighter and things like that published by Microsoft and iron galaxy has taken over development of the title ever since the season to launch so one of the things that we've noticed I have a long history in fighting games I worked for Capcom before this I've been in the competitive fighting game scene and one of the things that I've noticed just going through this is that the AI that in games often doesn't do a lot to prepare users for online competition so

I want to tell you guys a story this is Bob Bob is a new user one of these competitive games in our example a fighting game he goes into the game he starts playing the single-player mode and he loves it the AI is fun and interesting it's challenging he learned strategies and tactics to beat it gets better at the game and by the time he's done with that experience he feels like he's really good at this game after all he can trounce the AI he gets through it he's like I'm ready I'm gonna go online I'm gonna play some real people now but when Bob goes online and he starts playing against those real people he quickly finds that it's a totally different world out there the kind of game that the human players are playing is just dramatically different from the kind of game that the AI has been playing and the strategies and tactics that he's learned that are effective against the AI aren't effective against those human players as a result he winds up getting destroyed he feels frustrated and he quits the game I'm sure you guys have seen this sort of thing happened before it's happened to tons of my friends when I've tried to get them into fighting games and so this got us wondering is this a problem that the AI could help to resolve are there strategies or tactics

that we could use in order to make it better and so we kind of looked at what are some of the common problems what are some of the common reasons why this happened and by and far the biggest one is that the AI is fundamental playing a different kind of game than players our players are guessing and forming strategies and coordinating with each other but by and large the AI has perfect information about the world and is just making decisions in a very very different way so the kinds of strategies that it employs just don't reflect the same kinds of strategies that human players have so we've kind of distilled all of these observations down to a handful of topics that we'd like to cover today so first of all the fact that AI fighters traditionally don't have the same limitations players do things like reaction times and fidelity of input they also make decisions in a very different way than players make decisions and they don't tend to strategize or socialize the way that players do another thing is that even an AI that's really good really challenging really fun at the time that the game

ships over the course of the game's lifecycle players develop new strategies they get better at the game they come up with new tips and tricks find bugs and exploits and as they do that the way the game is played changes but traditional AI doesn't traditional AI remains static so even something that was a great a AI at the time that the game launched might quickly become outmoded or obsolete as players learn more about the game so what was our solution to this our solution to this was what we called the shadow system and what we're basically doing is crowdsourcing the AI development to our player base so we're looking at all the players all of the things that we're doing and we're learning behaviors from the players and letting them train the AI that's in the game now the cool thing about these shadow fighters is that they're not superhuman they have all the same limitations and restrictions they make the same mistakes that people do they're super easy to train you can make a functional AI that will do whatever you want it to do in literally minutes they learn strategies tactics social behaviors even they'll disrespect each other they'll you know be cautious or

aggressive they have personalities that they learn from players and the other thing is that because they're learning from the player base all of these behaviors they also change as players change when players learn new combos or develop new strategies those strategies then get added to the ai's repertoire but rather than just you know talking about this ourselves we wanted to show you guys the reaction of some of the players that first got it chance to play with the system and when they actually started playing against a shadow that had been trained off of their own their own performance so I'm gonna go ahead and play this video here so essentially what shadows is is a it's a system built into the game that I like to compare it to Skynet it's essentially terminator in real life okay I gotta be defensive with him

that was easy for him story and the craziest part about it is as I went further and the computer got smarter it did exactly what I was going to do at that exact moment ah lucky guess Oh oh [ __ ] damn it no yeah no get the [ __ ] out of there oh man heart isn't really [ __ ] close it's his mark this is the very own oh [ __ ] oh [ __ ] Oh like one hit away come on holy jesus holy [ __ ] like I had to focus

like I'm I was literate like I had the fight stick and I was like like I didn't blink I was gonna go next door to the Gunners booth and ask them to give me a pair because my eyes are hurting because I wasn't blinking I was staring and waiting because I had that the [ __ ] up part about all of it was I had to think about what I was going to think about before I was going to think about it like that's that's what it would felt like I had to think twice [Music] damn you stop being so defensive oh my god I don't think there's a reason for anyone to feel like they are bad at this game I think iron galaxy and Microsoft are giving everyone the tools that they need to learn this game the shuttle map has helped me improve my characters to some

sense because I'm well versed on this place I've been doing that I don't be noticed and will not be playing they fight me you know it is very aggressive and this whole thing works I love the way it is fighting other to the shadows from around the world it is going to change how we play killer instinct it's going to change how fighting games are developed it's going to change how fighting games grow as a community it is the coolest thing I've ever seen in my life alright so obviously that guy was pretty excited about the system but after we released it we did do some surveys and things to try and get some ideas of how people were using the system we wanted to share some of that data with you so first of all as you can see from the pie chart most of the people that responded to our survey were more hardcore users but amongst that demographic almost 80% of them reported using this mode and almost 50% of them said they use this mode regularly that

compares to only 46% of them who say they actually play the game online so more people were actually reported using this mode than playing the game online the other thing is that the responses from the survey were pretty much all give us more we want to see more integration of this we want to see more ways to interact with it more features that use it it was we were inundated by feature requests as opposed to you know complaints that the AI is terrible or whatever so with that said we'd actually like to show you guys real quick what it takes to train one of these by training a shadow live in front of you right here right now and showing you the process so for that I'm gonna hand control over to Bruce as I pick up the controller the shadow is going to be learning from me alright so I'm gonna go ahead and set us up so we're in game [Music] so what we're going to do is we're going to train a shadow for you in real time

so throughout this process we're gonna incremental e build-up a shadow that's able to handle itself in a variety of different situations as we record this I want you to keep in mind that Derek is playing the character in blue so we're going to be looking at the character in blue to basically train the shadow and then we're gonna see the actions playback on the character in red that that's going to show us what the shadow has picked up or learned so what I'm going to do is I'm going to go ahead and reload the match and Derek is going to go ahead and give his shadow examples of being aggressive so the first thing that Derek is going to do is give his shadow examples of how to be aggressive how to do basic attacks how to do combos how to approach the enemy and how to move around the screen so one thing that's important to keep in mind is that we need to give the shadow data from multiple ranges we need to show the shadow how we attack when we're up close how we attack when we're far away and we

need to show it examples of kind of how to jump in and do attacks so now what we want to do is we want to give it examples of how to basically attack against opponent that's being a bit more defensive so I'm going to go ahead and take control and start blocking the IMP it's dead so now that we've got some offensive data basically some examples of being aggressive we're going to go ahead and commit that and see if the shadow is able to pick up on what we were teaching it so we can see that the shadow is able to pick up on derricks basic locomotion it's able to pick up on some of this combos and jumping and attacking it's even picking up on some of this standing

around when we were kind of messing up the demo bed so that's cool but what happens if we do something that the shadow hasn't seen before like what happens if we jump in and attack it with a kick the show has never seen this situation so it doesn't know what to do and it's not really a problem all we have to do is kind of given an example of what to do in that situation so I'm going to go ahead and reload the match and give Derek an example a chance to show a shadow what to do in that situation so just going to do anti errors when the shadow jumps in of course we have to wait for the shadow to cooperate with us because sometimes it's just going to want to attack us so now that we've got a couple examples of an tires in there we want to go ahead and save that out and see again if the shadow was able to pick up I'm over here

showing it so we can see that it's able to understand things like an tires and able to pick up on that behavior but what happens if we show it another situation that hasn't seen for example what if we try to do projectile attacks so we're gonna do is we're gonna reload this and they give Derrick chance to show his shadow how to throw projectiles so now that his shadow has a couple examples of throwing projectiles we're gonna go ahead and reload it and give Derrick a chance so this is gonna give Derrick a chance to basically show his shadow how to react to projectiles how to jump over projectiles how to counter projectiles

so now that we have some of that data we're gonna go ahead and commit that and see if the shadow is able to pick up on that as well so now what we have is we have a shadow that basically knows how to defend against projectiles knows how to defend against an tires and also knows had a attack attack still so basically kind of in just a short amount of time we've been able to create a shadow that kind of we could kind of send out and players could fight against a bed so what's what's also cool about this is not just that we've created something that's that's functional but we've created something that was also able to capture Derrick style playing so we have something that's able to handle itself in a variety of situations but also was able to capture Derrick style which is really powerful because that

would work for any other player right every player has a unique style and this is going to organically pick up on that style without anybody having to come in and sort of ad-hoc program that style into it so now we're going to go ahead and dig a bit into how this works so I'm gonna give a quick overview of the system so I think a really powerful way oh yeah I think I think a really powerful way to think about this is the thing about like replay data to think about replay systems that we already have so with replay systems we kind of right now just use them to kind of share experiences with our friends so I did something really cool and I want to send a replay for my to my friend so my friend can

check it out but what if what if we took that replay data that we had and instead of just playing it back playing back all the replay data what if we looked at it and we tried to play it back in the right situation so for example in that replay data we could have information recorded about how much health somebody had or what distance they were from the enemy at a certain point in time we could use that information to basically jump around in the replay and find the right piece of the replay data to play back at run time to produce behavior at run time so what we're seeing here is kind of a visual a visualization of that process we have basically a bank of a bunch of different replay segments and the idea is that the shadow is always looking for the best replay segments to play back in context at run time so in this we're just finding that replay segment and playing it back at run time and then the behavior is whatever we get when we play back that part of the replay at run time so a big part of this

big part of this that actually makes it work is that sequential actions are very important so even though we are jumping around and the replay or basically we want to keep playing parts of the replay back that will look right so we don't want to just jump around arbitrarily or sort of just at the world state changes just a tiny bit like if the health changes a tiny bit we don't just want to jump around we want to basically continue playing back the current part of the replay until something changes dramatically something changes dramatically then we'll consider jumping around to a different part of the replay this is kind of it's kind of crazy but a lot of stuff is captured just by playing back parts of the replay in context so lots of what we think about is very complicated or high level behavior like planning that a player might be doing can be sort of captured indirectly by just playing back things in context at the right time so we can kind of see an example that in this next

slide this is an example of player fighting his own shadow and it's not really an example of what I would say is like a high level player and just a player that kind of plays very stylistically right he jumps around like he doesn't really know what he's doing and he's it's like he's not even trying to kill the enemy but what's really cool is you can kind of clearly see that those kind of unique locomotions they're happening on both on both the shadow and the player at the same time and this this is captured just by playing back those rebilling back those pieces of replay in context at the right point in time and making sure to maintain the sequences you would expect maybe that we would see sort of jarring shifts and behavior or something like that but it actually in practice if you do it right it turns out to look seamless so I want to talk about human limitations so lots of times we tend to ignore human

limitations when we're designing traditional AI in particular in fighting games one thing that we tend to ignore is reaction time so the graph up here is basically just a distribution of reaction time the data was collected from basically a very simple task where we would show basically a flashing green icon and users had to press a button as fast as possible when the icon flash green so how was the response sign that it took them to hit the button after it flash green so hundreds of thousands of users did this and then we have their data aggregated here what this graph is saying is that the vast majority of people have a reaction time somewhere between 200 and 300 20 milliseconds so why this might be important for a fighting game and in particular killer instinct is because the vast majority of actions and kill arresting the vast majority of things like attacking an opponent or hitting an opponent actually hit in less than that amount of time so

what that means is that a big part of the game almost half of the game is not and yet when people play the game it seems like it makes sense people feel like they're doing something it feels compelling they're playing it and they feel like they're in control but if the stuff isn't reactive or how is that possible well it turns out that for non reactive situations like the situation on the right the player is basically playing a guessing game so you can see the enemy basically doing a pressure string a string of attacks while you're blocking and you have to basically guess when the enemy is going to stop doing the if the enemy is going to stop doing the pressure screen or if he's going to continue doing the pressure spring and that guess makes lets you know when you should do tiger fury in the middle of the pressure string the thing is that you can't react to the enemy stopping the pressure string so you have to guess based on the history of fighting this opponent in the match how aggressive is

a how long is he going to continue the pressure string this is kind of what people are talking about when they talk about mind games in a fighting game and this kind of mind game this kind of thing that's happening at the margin of human reaction time it doesn't just happen in fighting games it happens in other competitive games and a Counter Strike there's lots of you know head shots Ben seemed impossible basically but then when you look at what's really going on it's basically high-level players pre aiming and pre shooting because they know that their enemy is going to end up in a certain blow in a certain location at a certain point in time so basically keeping in mind reaction time is very important in designing these kinds of a eyes otherwise you'll miss out basically and all this interesting behavior that players have at those margins of reaction time so because of the way the shadow system works because we're just capturing replay data and playing it

back in context we are capturing when players react and when players don't react based on the state of the world so that means that if a player doesn't react when it doesn't react in certain situation respond to an attack in certain situations we don't have any data for it so we never get basically you know the instant hit headshot or something like that because we just have what the player did in context we get this human limitation just as a consequence of how the system works and records data so this is kind of an example from counter-strike that's sort of what happens when you kind of ignore human limitations just kind of the traditional example of like an aimbot or something so somebody just you know walks gets head shot instantly it's super frustrating the way that we traditionally deal with this right is we make it less frustrating by adjusting the aim accuracy or randomly dropping

shots occasionally or something like that and that can work great for difficulty adjustment but it always does it but it doesn't capture sort of human limitations in the same way that humans have those limitations so like we were saying with pre aiming it's possible in counter-strike for players to get an impossible shot I mean a shot that's below reaction time but if we just do something like have a difficulty slider it doesn't capture that that aspect of sort of human behavior so what we want to talk about now is kind of decision-making basically what is it like when you build an AI with the shadow system versus building an AI with basically traditional methods so with the traditional approach we kind of decide on some stuff that we want to create we want it we say okay let's make a aggressive AI let's make a defensive AI and then we either sort of use some ad hoc solution or we use a behavior tree or we use a state machine or something to try to explicitly encode

those behaviors that were interested in creating so this stuff works great obviously it's what we always do we can get kind of like a lot of you know range in difficulty and everything one problem with it right is that sort of we're always limited by developer resources there's only so much we can do we have to sort of decide kind of what things we want to model and then somebody has to sit down and do them and like we talked about earlier in the introduction obviously the game continues to evolve after being released and if we don't have some solution that's sort of automatic we're not going to be able to keep up with a damp game after we've release it this is a pose as opposed to the shadow system where instead basically explicitly creating these behaviors explicitly creating a offensive or defensive AI the designer or the developer takes a higher-level approach and designs things called similarity functions weights and heuristics and we'll dig into what each

of those are in a second so first of all there's this idea of similarity when I was talking about the concept of picking the right part of the replay I was talking about we need to find something from our bank of replays that is similar to the current situation so what similar means is kind of defined by the designer so you can imagine kind of a simple Euclidean idea of similarity where you have two dimensions of Health and distance and each of those dots in the graph is basically you can imagine a replay segment so for example the one that's the one that has a hundred health and point five distance corresponds to a part of the replay where you know the enemy's world state was that so the thing to think of the think of the dot on the far right as being the current as being the current runtime situation basically so what we want to do is we want to find the dot the replay segment that's closest to that runtime situation

which happens to be the one that is you know closest to that that's immediately to the left of it so basically it's up to the designer to say what what that means in this case it was just the Euclidean distance but it could be just absolute value or it could be some crazy thing that a designer developer comes up with you just need to come up with some idea of similarity so once you've created your similarity metrics for Health distance or whatever you're interested in and whatever you're interested in using to compare two different situations you need to combine all those similarity metrics to basically get an aggregate score so you can imagine that you have things like health distance timer meter ammo now that you have all these different similarity metrics how do they matter relative to each other is health more important than ammo is timer more important than distance so another role of a designer developer when using this system is to basically adjust all of these weights so that they have the

right relative impact so this is something that kind of requires you know domain expertise that requires kind of an intuition for how the system works you might just have knowledge because of how well you know the game that distance is more important than health for example so it's worthwhile to think about like how you know this kind of approach could be used in different kind of kind of games so like if we think about something like counter-strike instead of using the traditional sort of difficulty sliders of okay maybe all failed to hit the enemy every once in a while or maybe all failed to get a head shot every once in a while what if we track things like visibility like maybe there's including objects on the screen or how far away the pone is or how far are what the relative angle is to the opponents if we have all that information being tracked for each of the players and then what they did when the world state what they did in those different world states then we can capture things like some players being less sensitive to the angle of opponents

and players not just having tunnel vision basically we can capture that part organically so this this gives us something that's a bit different than the difficulty slider right it gets a sort of way that humans fail at this particular task some humans have tunnel vision and they only focus on what's happening right in the center of focus and if we record this kind of data that would allow us to capture that kind of human failing so the other piece of the puzzle is heuristics so basically one thing that we want to be able to do if a system like this is allow as a lot of the shadows to adapt during a match so the one of the ways that we do we accomplish this as we basically try to identify trends in a match for example how often is a player how often is my enemy blocking Lovis blocking hi so you can imagine that we just look at that create a number that represents the relative ratio of attacking versus attacking hi and in addition to other parts of the world state like

health and everything we record that and then what that means is that when we go through the process of finding the best thing to do in the current situation we can use this heuristic value as well as the other world state value to select it correctly so if that value is updating at run time we can see okay what did I do in a situation where my enemy was blocking low a lot and it also works obviously for the cases when a player doesn't do anything or isn't even sensitive to that kind of adaptation so for example if a player ignores the fact that his enemy is blocking low burst blocking hi it'll just record that he does whatever in both situations and it won't really change the outcome at all so the the last last thing that we want to talk about is sort of we think of as human strategy and and killer instinct what this means is like high level behaviors like rush down or footsies and zoning and mix-ups in other games like

counter-strike it might mean things like coordinating with your friends like around an objective or something like that like if you're trying to defuse a bomb you want some user friends to basically defend you or something so the shadow system is able to organically capture a lot of or in fighting games at least it's been able to capture basically everything that we think of there's a high level behavior I like this video a lot because it shows us an example of of a basically a tournament level player fighting against a shadow so this is somebody that's at the top of this game he basically every every sort of part of the game that there is to know and exploit inos hears all these kind of crazy things in his arsenal and it's really exciting to see that his shadow basically is able to capture all that and play it back like when I look at this I can't tell the difference really between the shadow and the player even if like you guys aren't

that familiar of fighting games it's kind of compelling to look at just the locomotions that are going on just like it looks kind of like a player is playing so I think this is like a pretty exciting match so what I want to do is just let the video play through and let you guys see what happens [Music]

[Music] [Music] [Music] so in this case the shadow actually lost but I mean it's a pretty it was a pretty close by it right like the reason why I was able to put up that close to the fight is that the shadow really did pick up on basically like everything that that tournament level player was capable

of doing so I think it's pretty compelling evidence that basically shadows are able of learning these behaviors that we kind of think of as being exclusive to players these higher level behaviors like rush down and zoning and footsies in the context of a fighting game this is another video I like a lot it kind of shows how shadows can capture sort of social behaviors in this example kind of taunt then so this is another video about player fighting his own shadow and it's kind of funny because sometimes players can be [ __ ] and fighting games and this guy is about to get kind of a dose of his own medicine so it's worthwhile to think about kind of like how this stuff might apply it to like counter-strike or a game besides a fighting game so this is just a video of like a bot match in counter-strike and the thing that kind of stands out to me

with this is that it's almost like all these AIS are sort of playing playing like a different game basically they're not really aware of each other they're not coordinating than getting in front of each other sometimes so like is there something the shadow system could do to maybe help with this obviously we can do we can do a lot of stuff explicitly to make the coordination that or we can handle offer AI is to kind of make this situation better but how could the shadow system help this kind of a situation so kind of like the the aiming example we can think about things that we would track to basically help the shadow be able to identify situations that make sense for coordination so you can imagine a situation where in the past you you were kind of at a bomb site or something like that gardening your friend your friend was defusing the bomb or something and enemies were about to rush in if we look at things like if we

record things like the status of the objective the fact that you know your friends of the bomb site your friends defusing the bomb if you record the fact that you're close to your friend that's diffusing the bomb and we record the fact that an enemy is in there just like all the other stuff that we've seen we can be able to identify at run time when we're in a similar situation and pull out the correct part of the replay that basically will allow us to respond appropriately in that situation so in this way just by basically crafting putting kind of looking at the right factors in the world state we can kind of get this coordination not for free but kind of cheaply because we don't necessarily have to explicitly design the coordination we can just sort of capture it and directly so finally it's worth thinking about the whole this whole meta game aspect so when we at the start of the game despite despite how much how much time we spend creating and handcrafting AI

and everything like that a couple months after a release everything is going to be a lot different than it was initially because players figure out different strategies they they figure out things in the game that basically we can never figure out because you know we have you know 200 testers they have hundreds of thousands or millions of testers and what that means is that there's sort of behaviors and AI behaviors that would make sense for AI that we never thought of and we don't get a chance to come back to that usually right it's sort of just too late at that point we designed the AI for release now a couple months after release it's too late to fix anything or update anything so one thing that's nice about something like the shadow system is that it's live it continues to change and update as players played a game and as a result of that it's able to sort of stay current and make sense as the minigame evolves so if that I'm going to hand it back off to Derek and he's going to kind of talk about some sort of future some future directions for systems like this

hello again everyone so as Bruce mentioned I want to go ahead and talk about some of the possible applications for this so as you guys saw iteration times for a system like this are really short once you have this kind of system in place it's possible for a designer or content creator to just sit down and record an example of someone throwing a bunch of fireballs in order to create an AI opponent that will run around and throw a bunch of fireballs and this can be done in minutes this opens a lot of doors to do some pretty interesting things so here are some of the ideas we have for things we'd like to inspire you guys to go out and maybe create some things like this so first of all it's really helpful for training right you can pull down shadows of competent experienced teams highly rated shadows from good players that sort of thing and use them for a training gauntlet you can also kind of have a very gradual difficulty ramp going up through those where you have weaker shadows that's like playing against a weaker player and then the

player goes through the gauntlet and starts playing against better and better players in order to prepare themselves for online competition content generation for tutorials very much the same way if you want to teach a player how to deal with projectiles one of the best ways to do that is to make an opponent that throws a bunch of projectiles at him and let him learn how to defeat that opponent and you can do that just by having a designer play the game to create those AIS that rush down or throw projectiles or run away that are very aggressive or very defensive or you know have any number of behaviors that we can basically imagine it can also help with single player difficulty right you can start people out in the campaign mode with fighting against opponents who not only play like humans but have an appropriate difficulty level for beginner players and then slowly gret gradually ramp that up and you could even pull down from you know the crowd from the cloud from the crowd-sourced AI you could have actual players data being pulled into your single player mode forming the bosses and opponents and enemies that now the

player is learning to fight against another thing that's really cool with this is self reflection giving players the ability to play against themselves identify what their own weaknesses might be also if you're online and there's no one else to play against there's gaps in the matchmaking pool you can pull down shadows and have players play against them while they're waiting on players to show up you can even have players drop in and drop out of matches like that so if you I see that you know my friend is playing against my shadow I could jump in and take over for it and then you know beat him up and similarly if I might connection drops or I have to quit the game my power goes out whatever my opponents experience doesn't have to be interrupted because the shadow my shadow could take over for me and could seamlessly pick up and start resuming the fight finally some stuff that's interesting and kind of theoretical about this nothing is stopping us from combining replay banks from different players so you can kind of create a Frankenstein monster that has the head of one player and the legs of another player by combining data from their matches so if you have a bunch of top

players in the community and people want to see you know what if we had someone with the execution of Justin Wang and the aggressive tendencies of Alex VA or whoever your famous players are in your communities you could combine those people together and let people play against those kind of mixes and then finally there's potential to have AI only tournaments and these can be running 24/7 players could bet on them wager on them or you know train up their own shadows in order to enter them into these tournaments for prizes so these are just some ideas that we had of things that could be done with this system and we're really excited to see what ideas you guys come up with to do with something like this so of course another thing that I'd like to mention is that it wasn't just Bruce and I that was working on this there was a great team at iron galaxy supporting and helping us develop this and we want to give a shout out to all of those guys many of whom are in the front row up here right now and just say thank you so much for all of your help and support we couldn't have done it without you

[Applause] so with that or we're happy to take questions if anyone has any hi how did you decide how long how many actions to keep track of for each I guess data sample are you unifying them or is it a certain amount of time or a number of actions for each data sample for each data so you know that the player performs some saw some situation then they they did a couple of moves they did three four moves or they only did one move they just jumped and and how how do you decide how much of that to store and replay back for that particular scenario yeah so we didn't decide so basically it seems kind of terrible but you can have 90% of the data could be locomotions or something like that for example we aren't doing anything where we basically look at the fact that you have a bunch

of strings of locomotion so we start to compress it by throwing out a bunch of them or anything like that one of the reasons that we don't do that is it starts making it so that we have to do more explicit modeling right so in the case of locomotions it seems like it might be simple but lots of times if you're doing a locomotion are you engaging in some sort of high-level play play or plan or something like that are you sort of kind of staying carefully out of range playing footsies with the enemy so because we didn't want to do that explicit modeling it sort of just captures everything another reason for this is kind of just the whole idea of just playing back whole parts of a replay because things like the exact frame timing of things are so important for a correct execution of combos and stuff like that compressing together multiple sequences into the same sequences poses certain challenges not that it's impossible or anything like that but there were worries that we had they made it so that we didn't do it real quick when do you stop things because you don't play back a whole mash if let's say you use you play back five

moves and when do you switch to some other situation that's okay I see what you're saying I see what you're saying so you can imagine that there's a certain weight given to the current situation so we're calculating a score basically of kind of the current situation for the current plan compared to all the other things that you could be doing at the moment obviously there's lots of there's lots of there's gay in our specific game there's lots of situations we just don't want to interrupt anything but we're we're you're in a situation where you might want to interrupt kind of your current plan it's just sort of a way comparison against the score of the different possibilities that you might be interested in yeah I think so hey are you aware of salty bets okay and the do you do you envision not having to go ever go back to zero now that you have all of this data that is

on this title but basically you now have your Encyclopedia for the next title and what points do you think that you'd actually have to hard reset in order to clear the slate and how much is that colouring is is that positive or negative that you're going to have to worry about for future titles yeah we kind of are worrying about this right now just because we have a game that kind of has lots of game balance patches so the way that we are handling it is maybe not ideal but basically shadows kind of prefer data that they've done recently so if we've throw out another patch and some stuff is kind of gameplay breaking so some setups and some combos don't work anymore it's almost like your shadow will sort of stop using them because you'll still be playing the game and showing your exam with your shadow examples of how to do things the right way so overtime sort of the old bad data will get pushed out and the new fresh data will get pushed in

there's weaknesses with that approach obviously but it's what we're working with right now Peyman um how do you handle asymmetry so for example like you know all the examples you had was a character training against itself so how do you train against different yeah so that's that's a really good question so with what the waving system is currently set up all the data is record specifically per matchup so that means that the data that you have and Jago versus Jago situation is specific to the Jago versus Jago situation and the data that you have in a Jago versus orchid situation is specific for that situation so that doesn't mean that data doesn't translate at all it's just something that we've done to kind of be able to better capture sort of kind of very matchup specific behaviors that players do that being said we have an onboarding

experience where you fight basically against dojo master Jago and he only has data against fighting against another Jago player and players when they're when they're starting out a shadow of any type play against him and he more or less works he's still able to play competently and everything like that it's just that he doesn't have sort of specific strategies against all the different matchups hey can you talk a little bit about the size of data that represents a shadow yeah so probably from each match you can get around up to say like 800 different events or something like that and we probably an aggregate have somewhere between 20,000 to 40,000 events that we're considering her shadow when it's running live so in

the grand scheme of things maybe not that large thank you so you talked a little bit about how it kind of learns and it scales up as you get better and it kind of like keeps you getting better but as a player what if you are for example testing out a new shadow and it already knows how to fight that shadow but like let's say your friend is playing and they're not very good is it able to scale down as well yeah so if you do something like give the controller to your mom or something like that or put the controller down and walk away you're gonna hurt your shadow a bit basically so it will even pick up things like standing around or just jumping around or just doing crazy things basically like you can just go on on live and encounter a shadow that just does nothing but run away from you and jump so there's a question there of show weave detect these bad shadows and throw them away but right now we're not doing any of

that and we're not trying to say oh is this good data or do we want to throw out this data basically the players control over the process at the end of a match a player can say okay I want to keep this data or I don't want to keep this data so it's sort of a player's decision if they want to save the game when their mom played it or something like that okay so basically it you get the choice of whether or not you keep what you currently tested with and fought the shadow with or you can like choose something else later so that way if you wanted to keep your mom's you know your play session against the shadow you could keep that simpler AI versus something maybe where you're more skilled Yeah right just to be clear about the process right now it's only additive so you can't go back and then just pick these three replays in the past even though that's definitely possible the interface doesn't allow for that it just after you record a match you can say do I want to append this to my shadow data isn't okay okay thank you what's the performance hit doing all

those I guess it's lookups or whatever the mechanism is versus a traditional behavior tree or something else it's probably going to be like very dependent on how harsh that behavior tree and everything is but in this particular case for our system that's running and under a millisecond and Xbox one even when there are say like a fully trained shadow like amateur shadow that has 20 to 40,000 or whatever different things that's considering at a time so maybe not something that's possible when you have 40 shadows running unless it's very highly optimized but I think it's definitely possible to if it's something that you really if you really want to use this with a bunch of characters I think you can get it to run fast I think it's cheaper than doing like animation sampling for example so I'm

gonna get a little mathy um so I was curious if the choice of the best move or blur was the best move or was it probabilistic so you add a little bit of randomness what was your choice yeah so basically like you calculate all these scores for all the kind of replay segments that you have and then you know you can kind of you can imagine like you have a sorted list of scores right so there's a question of if I have something where I say have a situation where I have two scores that have the value 0 or I have a bunch of scores that are just sort of different by very very small amounts do I want to just sort of pick the one that maybe is kind of by accident the smallest score or do I want to sort of randomly select between them so what we do basically is we we basically find the thing find like the smallest score the best score and then we sort of pick randomly from a

neighborhood around that score so you could imagine that like everything that's within sort of 10% of that best score is kind of fair game and then as an aside the score itself I think you said it wasn't like including distance you just added up all the individual scores is there a reason why you went with that like a traditional you know what I'm saying the occluding Iseman for that that uh just thinking about it it's like Euclidean distance but there's no there's no like reason behind that the reason why we're doing this stuff this way is kind of just borrowing like if you want to look up more about the people that sort of did this stuff for real life search for like instance based learning or case based reasoning there said that people have been doing for 40 years but just hasn't really I guess found its way that much in the games but yeah if you dig into that then you can actually see what like the technical definition of similarity is and stuff like that and whether or not it's a real metric and

yeah just a in machine learning you usually end up taking just the squared distance instead because yeah typically an l2 thing yeah yeah cool thanks hi for games of sort of a different nature say a card game where your game state is less continuous and it's more discrete but with a lot more different actions would you suggest using the shadow system to develop an AI or is your data gonna be too sparse because of the number of unique scenarios to get into and would you rather do would you suggest something more like a simple min max ting over optimal game states yeah I guess I don't really have maybe Derek as a better intuition for the yeah so I definitely think this approach could be used to do a card game but because in most card games the number of options that are available to the player is relatively small it's not too hard to simulate forward like a large number of

turns or something and could then compare those so because the space that you're dealing with is pretty small I think it's probably unnecessary but you could definitely get something like this working in a card game if there were traits about personality that your card game might capture so where it would be most useful is if different players had different options that would show them as being aggressively bluffing or something like that or being very conservative or very defensive then a system like this could help capture those behaviors that might be hard to you otherwise have to do some explicit modeling to try and create different types of a eyes that play that way but if all you're interested in is in getting an opponent in a card game that plays kind of well I think it's probably good enough to just do a search tree and find a good resulting board state and then go with that line of play Thanks thank you for the talk it's very informative it's kind of two-part question the first one is you mentioned significant situations or kind of that

get captured and then you capture the behavior at that point how do you define what a significant changes is are there like numeric sliders for each metric like angle view or distance or is it some of more specifically programmatic process so so it's a it's a fully manual process where kind of you have to rely on sort of the intuition somebody that's really familiar with the game like Derek Derek is basically like a X tournament player and he was kind of sort of design design guy on this so he knew oh well we should probably be tracking distance or something like that were yes so what we wound up doing was actually setting values for each of the different states that constituted a big divergence right so for example not everything that we track is a numerical metric so for example we have you know the current players state is he standing up or is he knocked down right and so the weight

that's associated with those is very high and then we buy us the way we continue playing back a sequence of actions is we add a bias to the next action that's in sequence and so when the deviation in those other metrics based on their weights is bigger than that bias then it will naturally break and cause it to select the different action ok thank you and the other one do you do you think there's like a market for gamers of all levels amateur intermediate pro super expert to become kind of pseudo developers and that they'll play the game and the game AI when it's launched will just mimic the behavior of them so it's like it's a real human yeah not only that I think there's potential for players to use tools like this to create community tutorials to create community training content or challenges things like that where they could you know deliberately train an AI that has certain behavior and then that could go out as a challenge for other players to play so I think there is a lot of potential for the community to develop content using

systems like this yeah so writing it right now obviously it's kind of just a process where we're sort of doing it almost in the background for them but it could be a more sort of driven process where we could prompt players to do specific activities at a certain point in time and basically do things that would help us curate the shadows a lot better thank you uh-huh how do you handle strategies at different time scales so like you might decide for the next man I really need to focus on attacking this one guy but then during that minute you're going to switch tactics like depending on what they're doing right right so most of those captured by heuristics so for example somebody that has a lot of insight into the game like Derek and say okay something that's important for my shadow to keep in mind throughout the course of a match is how often my opponent is blocking high versus low or how often he's doing reversals on wakeup so you come up with this heuristic and basically just a way of tracking

fluctuating value throughout a match and recording it in addition to all the other worlds they do okay so you're never like combining different replays so so when you switch to a new replay it doesn't remember what it was replaying before right so it doesn't remember what it was replaying before but it remembers the heuristic summaries of what's been going on in the match so that's a huge chunk of the high event which includes what it's been doing before so it knows how many fireballs it's thrown because it's keeping track of basically a fireball count as part of the world state and it knows that when the fireball count is high you tend to do these kinds of actions or when the fireball count is low you tend to do these kinds of actions so if it's done a lot of actions in the past that are fireballs and the player has a habit where he tends to throw a bunch and then rush in it'll know that history of it's already thrown a lot of fireballs and so it will tend to pick actions that were done in that state like running forward and rushing in okay so you kind of have to know what likely things that people are going to want to keep track of so you can claim that well state with a

large enough data set it might be possible to identify some of those things programmatically like you might be able to have some kind of algorithm that is able to identify those trends rather than having a designer specify them but for this specific implementation I specified those those heuristics basically all right thank you do you think of something inherent to fighting games that makes a system like this viable like presently but would make it not viable for something with a crazy possibilities based like RTS moe etc I think that elements of it could be used sort of without great difficulty kind of in any kind of game - I don't think so Derek and I might I think Derek and I actually disagree on this but I think it would be very challenging if not impossible just take this thing and kind of just get it working in an RTS without a ton of effort that being said I do think you could do something like use this maybe just for the high level

strategy selection or use it for maybe just sort of reactive movements and a mova or some kind of mobile or something like that at least if you were going to go ahead and try and trying to do it like with game that has to ship that's like a MOBA or something like that I think that I have to be on the safe side would try to sort of target small pieces of the game like that instead of trying to just use it for everything you know like Bruce said I don't actually agree I think that it's a comparable task personally to implementing it in a MOBA or an RTS or something like that so but as Bruce said use it your own risk thank you um so if I understood correctly you're actually not doing any kind of utility function for this you're just taking replays and you're not real

you you're not really caring too much about the outcome of those options when evaluating the choices available to shadow right utility function basically it just comparing the result and seeing okay so these competion and they're a lot more damage therefore it's useful and or it led to a knock out so it's gonna leave me in a good spot things that yeah you're correct we're not we're not doing anything like that I think it's definitely once you have this data you definitely could do things like that you definitely could say okay we want to make shadows that are you know like the player but we're gonna throw out a lot of the stupid things that players do just so it's like a better version of the player or something like that and I can definitely see you know adding in sort of things too you know oh that combo it does really low damage so we're just gonna throw it out or once you do this certain sequence of actions it always leads to your shadow getting hit so we throw it that subin's of actions I think that's definitely something that you could do if you have to stay that's just not something that

we're doing at the moment yeah I'd also just like to point out that it's a very common technique in AI development to kind of Ford simulate look at the ending state and then assign it a score and say was this a good result or a bad result right and so do I want to do this one of the reasons we really don't want to do that is because we're not interested at least in this case in creating perfect play right we don't want the best result we want the most human result which is kind of by definition whatever a human did in this circumstance right so at least for our implementation here I think it's better for us to just play back what the people did rather than trying to figure out what would have been a better thing than what they did to do in this situation all right all right thank you alright thanks thanks for coming