Triage on the Front Line: Improving Mafia III AI in a Live Product

[Music] um hello my name is kate johnson i'm a senior designer at hangar 13 and this is sergio otio he is our ai lead at hangar 13 and we'd like to welcome you to our talk triage on the front line uh improving mafia 3 an ai and a live product um so over the next 30 minutes we're going to start talking about our approach to the post-launch improvement of mafia 3's ai specifically a patch we are going to start by sort of going over what the ai was in mafia 3 sort of talk about our game play we're going to talk a little bit about why we decided to patch and what our goals were for this patch we're going to go into the patch process and we're going to specifically focus on how we um on triage and and how we identified uh problems to solve and then we're going to sort of wrap everything up with a few examples to hopefully get you guys

seeing how our process sort of worked and there will also be some cautionary tales in there for sure so mafia 3 released in october of 2016 and it is an open world crime drama set in the city of new bordeaux in the turbulent 1960s the gameplay in mafia 3 consisted of a combination of stealth we had driving and then we had a whole bunch of combat and to support this gameplay and also to sort of realize the world of mafia 3 we populated we populated the city with a mix of enemies allies police and civilian npcs and all of these npcs have a bunch of different ways that the player can interact with them the the way that they were implemented was through both systemic and and scripted uh uh encounters the scripted stuff was focused more on the on the pre-combat we wanted to set up

all these really cool narrative vignettes um so that the player would really feel this living world that we were building in new bordeaux and then um as far as systemic behaviors the most of our our search combat and wander ai was all fully systemic um so as an ai team on on on this project we really had this goal of supporting you know the best possible quality in both of those areas um and by the time we shipped we felt like we had actually achieved a lot of this stuff we had we had gone a long way towards achieving our our our list of goals however you know between the upcoming dlc needs um the sort of wish list that we had uh just of things that we had wanted to improve over the course of the project but but couldn't for one reason or another um and then uh finally uh player feedback we paid a lot of attention to forum uh comments and things like that and sort of we're seeing how players were receiving the game um we we still had a lot of work to do

um so the first thing that we decided after we shipped was that we were going to sit down and figure out how to make a patch and i'm going to hand off to sergio yeah so so we are all game developers right and we spend big parts of our lives working on these things like we spend weeks we spend months years working on this thing and you know we are very passionate about what we do and we always want to do things as best as we can and we are always trying to improve and we are never happy with what we do right so that's where we were at the end of mafia we were happy but not happy at the same time right we wanted to do things better and at the same at the same time well around that time is when i joined the studio in the studio two months before shipping mafia and when i joined we were also looking at the future what's the future going to be look uh what's the future going to be for the studio in terms of ai so we decided to put both things

together and you know start applying some of the new ideas and new things to to mafia because we had a good test bed in there for us right uh for so the first thing that we did was just assemble a strike team instructing was a group of people from different departments we had engineers an animator a designer we had qa and we were fully dedicated to this process we during two three months we were just every day just looking at the game and thinking you know what can we do better for the game to make it look even better uh so with this in mind we set our goals and basically we had two the first one was to make it better obviously uh that was the main the main thing right we wanted to to go and fix all those things that we were not super happy about all the things that players were not having so much fun with and we wanted to try and do it for as many people as possible because we wanted this thing to really change the perception of the ai

in the game right but the second objective we had was equally important to the first one it's do no harm so we had so many systems in the in the game and they were interconnected everywhere and we wanted to make sure that whatever we changed we never broke anything so that was that was key for us like do things better but don't do things worse you know always go forward never regress and now kate is going to talk about a little bit about the process so um once we had formed the strike team and identified that we wanted to to start building this patch we sort of came up with a daily uh process that we went through and this was we would get together well actually first we would all play the game at our desk and we'd play the latest build we would look at whatever had gone in the previous day and we all had sort of an idea of what the changes were that we had implemented before we would be evaluating each um individually to make sure that the

changes we had implemented were in and doing what they were supposed to do and we were also looking to make sure that we didn't introduce any new issues um once we had done this and and and sort of gotten an idea of the the the overall health of the game we would all get together and sit down and we go over this master task list that we had of all of the things that we wanted to fix and like improve in mafia 3 and i don't know um if you guys can imagine but like this was a massive list for us just just of all the things that that we had wanted to do or thought would be cool or cool ways to improve the game uh in addition to like that that player feedback and and feedback from the directors that we had compiled so one of the first things that we had to do once we once we all sat down and looked at this list every day was sort of figure out how to prioritize everything and when i'm talking about prioritization um the way we measured it was to look at impact complexity and cost

um and when i talk about impact it's pretty straightforward it's it's how big of an improvement what's the delta uh from the starting point uh to to what we expect the change to be like once we've finished implementing is this going to go from being something that's okay to something that's great or is it just going to be a little bit of a change and then the other side of this is how many people is this going to affect um so we didn't have a lot of time to work on this patch it was a patch cycle so we had about three months and while an issue that affected a small percentage of the the playing population wouldn't necessarily fall off our radar with this prioritization process we definitely put emphasis on looking at issues that were going to affect the most players and that most people were going to get a benefit from so the next thing i'm going to talk about a little bit is complexity um so if you take a system like our

police dispatch system which was sort of one of our global systems that the player interacted with in in the game world um this is something where a change could be a really high impact change because it would affect a lot of players but if you look at all of the things that changing this police dispatch system could potentially infect or affect you could end up seeing a lot of of of potential for unintended consequences and knock-on effects so when we looked at complexity we were talking about what is the potential for this having unintended consequences and we wanted to try and avoid that as much as possible and then the third item that we talked about was cost and cost is is again straightforward it's time and resources we had about three months to put this patch together so we didn't have a lot of time which meant that we couldn't look at fixes or improvements that were going to take weeks and weeks of effort we needed to be able to

get as much in in into this patch as possible and as far as resources there's the human element so the studio was super supportive of us and and they they offered us all the help that we needed but at the same time we needed to be respectful of the fact that they were working on dlc um so we needed to try and avoid pulling in uh you know animators or artists off of what they were doing so so things that included like uh new barks or new animations or or new art assets were generally off the table for us so um to sort of bring it all into to focus on a specific item i'm going to talk about one of the the things that sort of came up for us at the very uh right after we shipped essentially one of the one of the first things that we we tried to tackle and basically we discovered that we had an issue with the way that enemies were reacting to uh gunshot events so the player firing a weapon um so the way this uh the way our enemies

worked is that they had an event-based reaction system we would trigger events based on player actions they would either be local to the player position or based on wherever a thing was happening in the world for example a bullet impact and then we would use those events to trigger a reaction which would then trigger a state change and or or a behavior change in in our ai and the gunshot was uh was was tuned to be about 40 meters and that's pretty short um and i i like if you think of 40 meters for for how far away somebody can hear a gunshot in in a in a big game like ours it's not a lot of distance and we did this for a reason we did this because we wanted to make sure that the uh the player wasn't aggroing enemies that were outside of their area of focus and we had a really densely packed encounter um layout in our in our open world and you know there were enemies around every corner we didn't want the player to get suddenly surprised by a group of guys

running around because they had started a a war with with another group in a backyard so this all worked fairly well we also had an event by the way that was happening outside of this 40 meter distance for bullet impact like i said before and also for enemies dying but this would not trigger combat it would trigger a search behavior so overall you can kind of see how this might play out and i'm going to show an example of of actually where it broke down so this is our get enzo mission and um right now we have the player with a sniper rifle and he's outside of that 40 meter range so the enemies are doing exactly what we designed them to do they are going to search based on their buddies dying

and while this is hilarious and i kind of love it for what it is um it's not how we wanted this encounter to play out this was supposed to be a combat encounter and it wasn't really working and the way that the way that it was set up it was on it was on the golden path um it was one of the main missions that the player had to play through and we put that sniper rifle like right there so we were kind of asking for it um so it was high impact for us and we really wanted to fix this but we also needed to make sure that we were keeping this fix relatively low cost um so the first thing that we did was we tried to sort of figure out what the problem was that we were trying to solve and the first thing that that or the the initial problem that was presented to us was that enemies feel stupid because they don't seem to care that the player is killing them um which is true they really didn't look like they care very much or they subdued like a subdued caring um so uh so we're gonna talk about some potential fixes for for this particular

issue um so we had an infinite time and resources version where they were gonna have this really awesome behavior where they dove into cover and then like peeked out and then sent a buddy up and we're like watching we really wanted to do this but it required a lot of signs and feedback that we weren't able to provide as far as animations and barks and other things so this was pretty much off the table right away um another option that we discussed that was a really inexpensive option for us would have been to actually fix it in this mission directly and just say anytime the player fires a weapon in the get enzo mission everybody just goes into combat and it would have solved this specific problem but there are two reasons why we didn't end up going for that and one is that it wouldn't have solved it anywhere else this is actually a systemic problem with the way enemies react to gunshots outside of 40 meters and it could be reproduced anywhere in the world and then the second side of it is that it was really important to us not to take these sort of tools away from the player and the player had

silenced weapons and had the ability to play through this mission in stealth and we didn't want to remove that so that was again not something that that we felt comfortable doing the third option for us was to extend that 40 meter distance to 60 meters and that would have sort of taken care of of what we saw right there and it would have it would have worked and it would have worked all over the game um but if you remember when i talked about the sort of density of encounters that we had packed into the open world it could have it had the potential to to break a lot of other things um and also there's no reason that the player couldn't move to like 61 meters and caused the same issue so um we we sort of looked at this original problem and tried to to rework it into something that was a little bit more specific and solvable for us and what we came up with was that the enemies feel stupid because they react calmly to the loud shooting death of an ally in close proximity

and um rephrasing the problem this way and looking at the specific actions and the specific thing that the enemies were doing that made them feel broken sort of helped us figure out what the fix could be and we had this reaction system that i described before really all we really needed to do was add a new reaction um to a new event uh that was was an event based on loud shooting that would escalate them to combat instead of search and so it was a super easy fix for us we went ahead and tried it out and this was the result so this is much closer to sort of how we wanted the enemies to behave and we had we had we had solved this issue pretty well we felt and we patted ourselves on the back and we were quite quite pleased which brings me to my next slide um

so while we were all congratulating ourselves and breaking open the champagne for job well done um qa was finding an issue and um basically what we had done was we had forgotten that the pedestrians in our game used a completely separate reaction table from the enemies and we had sort of forgotten to incorporate this event into their reaction table so when we removed it for loud weapons all of a sudden you had a bunch of enemies that were standing around looking like goons whenever the player opened fire it was not great um we were able to fix it pretty quickly and the fix still still ultimately worked but this is sort of an example of where testing really becomes important so um my first line in there is that you are the best person to test your work and the reason i have that is because it's something i always really believe in for myself that as the person who is implementing content i have a responsibility to

understand all of the things that that change is going to impact all of the different systems and everything else that it could potentially touch which means that i should know all of the potential weak points and pay close attention to that however i am fallible so i'm gonna change it a little bit um you are one of the best people to test your work a lot of us here are fortunate to work with a very talented team of uh uh of qa um qa folks and uh we we definitely were we had a great team at hangar 13. and one of the ways that we really tried to empower them was by being vocal about our changes and making sure that anything that we introduced they understood the ramifications of and they understood how it worked what the change was was what all the systems were and what the potential weak points were for it and that allowed them to

test more effectively which made this turn around a lot faster so trust your qa team they're probably not out to get you and with that i'm going to pass you off to sergio all right so now i'm going to show you a couple of exact examples we want to illustrate the process a little bit better so we want you to really understand what it was like for us during during that period of time right so as kate said before the first thing that we did every day was basically play the game and play the game looking at you know trying to find things that were broken in there or that didn't feel that good right uh so this is what what we're going to do now like all together let's look at a video from mafia 3 i try to try to see what doesn't feel right to you so in here basically the the player is whistling at that group of people and one of them is going to come over and the player just takes him down

okay uplift okay yeah we didn't have body on there um so what did not feel right to you so to me it was this part right you were taking down a guy that just came from that group of people those guys were still looking at him and we had that big window in there and they were not reacting to it they were just completely ignoring the fact that his friend was being killed right so you know finding the problem in this situation is pretty simple right the problem is that animals enemies feel unresponsive because they don't react to a visible takedown and the solution for this is also fairly straightforward right it's you you have to make make it visible for them you have to be careful and have some radio so you know they don't see it like

in a super long distance but that's kind of like the normal vision anyway right so that is what we did so i'm just going to show you so now you can see that the other two guys like reacted and freaked out and went into combat and the way they were supposed to do at the beginning right so let's take a look at the video again but i'm going to play it for a little bit a little bit longer now uh so again this is a video from before the patch so it's exactly the same thing so the first part is

[Music] [Applause] so you can imagine what's going to happen to that or die in there right so what did not feel right in here so you know when we were looking at the prong we thought that the the problem was that whistle was op you know problem is that you have this tool that is so powerful that people can exploit it so so easily so you know when someone tells you this thing is super overpowered what you do is like try to make it less powerful right and that's what we tried to do so the first thing that we tried to do was like okay so what if guys after the in subsequent whistles don't come to to investigate they just ignore the fact and that but that felt pretty broken right because it was only one guy coming and then the rest were like you know completely unresponsive to that thing so

so we didn't like that solution uh second solution was to you know having rather than having one guy come to investigate why don't we send two guys because that way you know if you kill the one the other will alert the rest of the of the scenario right but you know that thing was breaking stealth for us because this this was actually a design decision right we wanted we wanted the player to be able to play with these tools and to feel powerful and to be able to you know to in the hurt and to you know be able to to do all these things um so that that solution didn't work for us either and the third solution that we tried was to basically escalate response based on number of whistles so first one would be an investigation but if you whistle again maybe they go into search if we sell the engine they go into combat but it felt pretty weird pretty weird that you know if you whistle at someone from 20 meters they immediately know that you're

there and they just want to kill you because you know someone whistled from there so so we didn't like it either although this this last solution i have to say that we kept it for the game for some things like if you whistle that guy that was right next to you while he was searching that position because you had already whistled at him then he would go into combat because he was pissed off at that point right but you know i think that the the actual problem was more this like you know whistle was op right but you know and this comic is awesome we found it when we were working on the on the on the talk and it reflects the thing perfectly but the actual problem is this the actual problem is that we still feel so be because the enemies keeps coming to keep coming to investigate the same location over and over um so you know and how do we solve that thing so we knew that we wanted to keep the same sort of behavior because that was

the way stealth was supposed to work in our game but we needed to find a way to improve the behavior of the guys so so it didn't feel as stupid as it felt right now and you know this is something that happens in many games but we wanted to be better than that right so what we decided to do is basically to have people that receive a whistle like all those people that were with him in the video we want them to remember that they had already received a whistle and that there was a guy from the group that went and investigated investigated that thing and then when they receive another whistle from the same location they say like okay so i know that somebody went there i know that that guy is dead so so there's something going on here so we wanted them to call out the the fact they wanted we wanted them to tell you you know we know this is wrong so now we're going to escalate the behavior so that's what we did so and this video is a bit weird because i have to move around because it's after the patch so they will see me taking him down so i don't want him to

see me do that i have to wait for him outside of the room [Music] this is what happens when you whistle at the second guy where'd you go frey i'll bring his domains back is also escalating into something else right so we know in general the behavior of the ai was much more appropriate for the for the situation right so now that i've solved the examples i just want to to talk about a couple of things that you should watch out for if if you are working on this sort of

problem so the first thing is for sure that you have to identify the right problem we've been talking about this twice already in this in this talk right like we had an issue with the sniper rifle problem that we didn't know exactly what the problem was at the beginning so we were not fixing the right thing and we have the same thing during whistle so identifying exactly what you're trying to solve that's that's the key second thing is being over causes we had this problem in a couple of situations that we haven't talked about in this in this presentation but you know if you know that basically we knew that that we needed to fix something going the right way and the right way was scary for us so we decided to try and do like little steps just to see if that was enough but in the end that was not enough and we had to end up doing the scary thing anyway so you know identifying the thing from the beginning is better and the last thing is knowing when to stop uh we also had situations like this where we were trying to

improve something that was not really broken but you know we just wanted to polish things a little bit more right uh but in particular we wanted to to improve the surprise reaction like what happened when when someone saw you uh and did someone was completely unaware of you and you just jumped him from in front of him because that right now it was going through normal systems it's a normal perception so it was all good but the reaction was that they were into they went into combat right away and we wanted some sort of behavior first like some sort of reaction and we couldn't make that thing work because of the amount of animations that we needed for it we needed animations from different uh directions for different weapons like a lot of things so just to recap the process is basically that you have to identify a problem you have to find all the potential improvements implement the best one based on impact and cost then test it and you know keep iterating and keep doing this thing until you run out of time basically so that was the end of the talk so thank

you very much for coming [Applause] just