Behavior is Brittle: Testing Game AI

[Music] okay let the records show that I'm the very wide one thanks Mike yeah yeah couple more minutes last session of the day we have kind of gotten into the pattern over the recent years of ending one or of the days with a panel just to kind of it because let's face it we've gotten an earful so far during the day but the it isn't just a panel we always try to make sure that we're giving it something that makes you walk out here and thinking okay yeah you know inspirational or or something you can get and wrap your head around and say this is something I can take back to my team that isn't necessarily technical so that's kind of where we tend to head with this so this came up on the guild list of something I wanted to talk about

so I won't you know blow all the secrets here so but that's that's why we're here and it's anyway what would who was here for Dave Churchill's thing the last last bit complexity classes kneel before him so where is he I want to see if he's blushing no he's probably all thank God I'm done oh there you are so good first day all right I do miss Jorge I have to admit that was like we have one regular who's not here this year there is something that he was the most enthusiastic AI Summit attendee ever and he definitely could bring the decibels that's for sure but I have to check on him hope he's all right so we got quite the slate tomorrow too so you know hopefully we're not up against any serious competition from the other summits in tutorial oh no there are no other summits and tutorials this is the only game in town right that doesn't

work is it so okay well I'm gonna we're still getting some people scanned in here so we've got about a minute go ahead and because we do have some people still coming in if you have some empty seats to the in the middle of your section go ahead and scoot in if you could please yeah thank you and once again my near ubiquitous reminder of please make sure you're scanned into the sessions but make sure you fill out those those evals even if you just score us that's great but more importantly please write some comments some feedback for the speakers on the some of as a whole and we have been pretty shameless in past years about saying writing things like the AI summit is the reason that I come to GDC don't lie but if that if this is like really really important to you write that in there because then the GDC people read that oh we've got to keep this AI summit coming back so not that we're at a risk I don't think but

please put put that kind of stuff in their feedback for all of us it's it's really important for us to to hear that so alright they were good this door is closing so we're gonna go ahead and get things kicked off again my name is Dave mark your summit advisor and more importantly for right now the the moderator of this panel my a job in general is going to be direct traffic up here because the real talent is on either side of me that the lot of experience on this subject and again this kind of came from a discussion on the guild list that as game behaviors get more complex it's a real pain in the butt to test them and so the consensus was yeah we should have a panel on this I can speak to that I could speak to that etc so that this is who we had ending up in here I will let them introduce themselves first Sergio Theo Bach where'd you go Sergio is right here so go ahead so I'm an

engineer a hanger third thing I joined the team when we were finishing mafia free oh yes I've been in the industry for a long time so you know it's very interesting and talking and we have to test it a lot of things elmo's johansson yeah I'm a contractor so I do other people's games and most of that time it's specialized in AI and for the past nine years I've been doing my own middleware on that okay Mike Mike Lewis on the lead gameplay programmer on Guild Wars 2 at ArenaNet and I've been doing AI for most of my game programming career in some form or another and Mike Robbins Mike Robbins I'm the senior software engineer at hollow spark I've worked on games like planetary annihilation spring commander - I got interested in AI as a mater in the community and then transitioned my way into the industry which is like the coolest way to get into the I was just

modding stuff what you want to hire me cool and again my name is Dave mark and the president and lead designer of intrinsic algorithm I do AI consulting for Studios all over the place including I worked with Michael ArenaNet on Guild Wars 2 as well as EverQuest next what else have I done whether the pixel mage Heroes song here recently so I've done a number of different things but that is not important right now what important is what we're covering here briefly to give an overview you know there are a number of common problems that we have with testing we're going to be talking about some of those as well as then best practices of how to approach doing our programming doing our tests etc and then also tools development you know what are some best practices for having tools available to make all of this easier so we're going to touch on some common problem things first just to

kind of set the stage of what it is that we're trying to address and I'm gonna let each of the panelists kind of you know chat about what they view and believe as being the important things on each of these and so the first one is that just identifying problems to start with you know is this broken or not and so what is you know a way of kind of encapsulating you know a mindset if how you have you know just even realize this what how big of a problem is just this one thing well I think a lot of it depends on the actual goal that you're trying to hit with your AI because in some cases you do have a specific thing that you want to have happen and play out in a certain order but in a lot of cases it really doesn't matter like you can totally have an agent do his own thing and come up with its own interesting plan for how to do it and then you get this emergent result that the player thinks is pretty kick-ass and you know you're done that's that's how AI is successful but when you have those

cases that are absolutely black and white you have to know in advance that we we know we really needed that guy to go through the door right then because otherwise the whole plot of the game doesn't unfold or whatever so sometimes the the issue is easy and sometimes it's a little bit more nebulous and hand-wavy so okay what are some other issues surrounding there the whole idea of identifying problems I suppose in one he said like it depends on the problem that you're trying to solve but many times you don't really know if something's working or not and you know it's very hard it's easy for you if you're working on the thing but it's very hard for all the people to know like QA to know if the thing is broken or not so just trying to come up with some rules or some way to tell people what means what broken means for your game or what you know and that's something that is important as he was saying

in games like mafia 3 where narrative is very important things have to happen in a very specific way sometimes but then you have other other games like I don't know like maybe GTA or something like that where anything can happen and as long as it looks ok or it looks impossible and you know not too crazy and people understand what's going on I think it's fine I think one of the things that was mentioned earlier today might actually apply to this too is I came here who was it might have been Brian who talked about how because AI is so central sometimes something is broken but is it the AI or is it you know an animation bug or a scripting error you know by our data error or you know something in the environment or whatever I mean so even just determining is it the is it AI that is broken is sometimes part of our job too rather than is it

and some other system related anything else on this resuming I mean just establishing expectations and common language to will touch on a lot of these things you know as we go here Mike did you have anything or well the next thing then under common problems that we kind of came up with as a group is identifying the causes of course yeah and I just touched on this briefly myself you know why is this broken how would you rate this is that as a challenge on you know to address yeah it can be broken because as you said like it can be broken because the code is broken or you can be working because something else in the setup is broken those things are hard harder to find I suppose sometimes you have promised me things that are scripted and you don't know if it's the AI running or if it's something descriptive so right you know you always have this all these are all these types of problems I guess which lets do a lot of finger-pointing

so anybody else's you know I mean it's it's I'm trying to determine whether or not it's broken because they has broken or because some other system is influencing AI and causing it to behave wrong is is difficult especially when you're AI swapping back and forth between you know doing the AI based behavior or doing something scripted it's like you're trying to determine whether or not it's the script broken or is it the AI behavior that's broken or even just you know some of the stimuli in the world the AI believes that it is perfectly doing the the thing it's supposed to do based on the inputs and but we just kind of sit there go well all the inputs are always correct right so they couldn't possibly be then okay if there's third common problem is the the scope of the testing as games get bigger this is kind of a thing what are some of the scope issues that you've run into just from a problem standpoint let's see we had on the order of several

hundred unique combinations of archetypes for characters so everything from you know your for your forest wildlife - you know pirates - ninjas you know whatever else might be roaming around the game world and then each one doing on the order of dozens of decisions that they could choose with on the order of dozens of things that they're thinking about for each potential decision the number of things that you have to test is ludicrous or and to put this into context this is in in Guild Wars - this is a killer - so you're in an MMO with hundreds of different types of characters right at which you know span it isn't like well I have lots of characters but they're all human or something like that so we just have a variation in all of the different you know base you know behavior you know brain behave you know how they acted so that that type of scope what's another scope issue and there there are massive pitfalls in in over and under testing both manually and automated testing like

again having a good analysis of what are the key areas and what's what's at stake it's I mean that's mostly important for for manual testing for having that agreement and for my automated testing making it a useful task without like while still having sufficient coverage so you don't like go overboard in automated test there's a prioritization combined with at some point enough is enough yeah okay I mean otherwise it becomes this massive big task and people start ignoring it a bit though you know it becomes hassle right okay well yeah so there's a psychological factor internally in the team of you know okay I gotta test absolutely everything and does this really matter yeah okay yeah I mean personally I think that you never really finished testing stuff I give him when the game is released after release you keep playing

the game and you can see in all these problems and it's hopefully in the patch you can fix them or something but normally you don't have to add ability to keep releasing like fixes for your fans so I don't know it's right you know I I coined a phrase by accident during a GDC week when a discussion was with a colleague where I I said something to the effect of asymptotically approaching infinity and he paused he looked at me he goes did you really mean to say that but it that's actually in this case it's one of those things is yes we're getting ever closer to something that's constantly moving away from us it's so yeah you can't win at some point you have to just say we're done well let's get on to the solutions then as far as the best practices for just testing to begin with of you know approaching you know like we talked about the common problem is something broken what would you say that your general approach is to testing AI either you or if you were QA or as a designer so just just a general broad approach of how do

you come at the problem so to me the most important thing when you have to do when you test AI is that you are not trying to test that the thing that you worked on works but you're trying to see if it can break so you're all the time trying to break your stuff basically because you know you basically just want to avoid having problems that people are going to put on duty and you know like your work for like years is going to be like in this one video that is like doing bad stuff about about your eyes you know trying to avoid those things it's always kids nobody ever youtubes they really cool hey there I was doing exactly what it's supposed to I'm gonna put that on YouTube that's not fun normally people say that when the AI in a game is good it is because it's when people don't it doesn't talk about it right yeah we don't notice because it looks natural yeah what are the general approach to I like testing iteratively

you know as you finish a feature test the feature start another feature test that feature so you know lower the scope of what possibly went wrong okay so yeah I've done this and if I test it now if something radically changed it's probably what I just did rather than here I just added 18 different things I think it's time to test oh yeah I have no idea where this broke because I've been working for three weeks on this and right I was also also breaking in before they I have done this in in it's not done before I've already started testing a lot like making integrating testing as part of the development process like using an asset and extra like Intel and it telethons like tools or an important part of your productivity set up so if you leverage the tools you have for testing while you're developing it before you mark it done for so yeah it

is probably a case to be made for it's not done and it'll you've tested it you know because or at least you don't know if it is done is it a different way of looking at it certain or you might guide us um yeah just to kind of riff on that a little bit I think the the real solid approach is to start by assuming that everything you build can break and try to find the way to do that as early as possible and continue doing that until as late as possible if that's after ship and you still can ship patches you should still be testing your stuff even if you think it's bulletproof okay so a good approach now as far as the okay I've you know written my code and I'm running the game what is in your head at that moment you know so you've decided to iteratively test stuff like that using some of your tools early on and everything what's in your head okay go what are you looking for I mean that sounds kind of lame does it what I'm

looking for ship that's wrong yeah that's the easy answer but you know I will you touched a little bit on you know I think the first step would be is it doing what it's supposed to yes oh thank god yes yeah but normally you tend to work in small iterations I guess something news like this tiny thing so testing that tiny thing takes us like you know a few minutes or something but then like testing like all the possible side effects of that type tiny thing or how that new thing interacts with other things and what else is broken based on that and how how can you find like edge cases to to break your new things I think it's well that's the challenge unit of itself is what are the edge cases because in a lot of times they're not obvious yeah and it all depends on what weapon you're working on I guess sure but I guess that's something that you learn from experience like they stone you know once you've been working on something for so long you you know what to look for I guess right always talking with all the people that

have experience or you know trying to learn that thing from other people I got used to have a late Martin Walsh I've turned that that was super good at breaking things it's like I learned all that from him so you know every time I try to test something I just remember handing my stuff over like him the controller over to him and seeing him break my stuff so it's just you know keep learning from other people all the time cool yeah I mean as as I've been iteratively developing or testing and during the production of whatever bit of functionality that's kind of been happening in some sort of lab cleanroom kind of scenario so before adding the done label trying to kind of force myself to put it into as as weird unexpected scenarios as possible to kind of challenge the environment it lives in

to try and wrestle out some unexpected behaviors and complicated environmental interactions stuff like that sure usually when I go through I'm trying to think of okay well if the AI does this how is it going to change the game state and then how is it a I going to react to that because you know you could put it a new feature that breaks something else somewhere else because it's changing game state now that you weren't prepared for so at that point that's again where the iterative thing comes in it seems is that yes I added the one feature because we're doing this iteratively and I can test that one feature but we're also not just just trying to break that one feature we're saying all the other stuff that I thought was perfectly bulletproof is it now being broken just because of this one thing where you have that give-and-take back and forth between them so yeah that's just so as we keep adding features it seems like it's a you know combinatorial explosion of

that could go wrong so which is where we get back to oh and there's hundreds of different you know agents to test so um how does this whole mentality change with doing your own testing rather than other people testing your stuff are you testing other people's stuff is there is there a different approach that you would have to have mentality when you bring in other people into this process I think the approach fundamentally stays the same but the benefit of bringing other people into the the testing situation is that they're going to have different blind spots than you do like they will think of different things that they could try they will think of different combinations of things to put together and see what happens they will have a perspective that you don't by nature of being a different person and that gives them a lot of power to find the weaknesses in the

design that you're so used to looking at that you can't see like oh hey there's this ah be short coming right here yeah I think as AI designers and AI programmers we tend to baby our AI is a little bit we don't want to see them breaking or broken or so yeah bringing other people in that don't do that and really you know put the hammer to the AI is very useful is there those are advantages to bringing in other people or disadvantages to having someone else testing something that you've completed for example I think the biggest problem I've had is the QA team not knowing what's done what's not so they find a broken behavior and the answer is well I haven't programmed that yet so yeah of course it's going to break and it's not going to do what it's supposed to do you ever run into the situation where you just almost subconsciously know well

this is supposed to you know exhibit it this way that way this way and so you kind of you know just if it feels right to you but to somebody else they don't even know what they're looking for on a more subtle level and how we solve that well I think it was brought up earlier like having this shared language of you know what are we actually discussing here because I could look at something and see an AI bug that someone else wouldn't notice and vice-versa because we have different opinions of what we're looking at and what we're actually expecting to see in those situations so knowing you know just that you're on the same page as far as hey yeah this thing is supposed to act like a predator or act like you know a pirate you know those baseline expectations are super useful for knowing that when you do start picking apart the details of what's going on on screen you know you're you're talking about the same set of issues and not something completely

divergent like you know the animation was broken and well I know the animation is broken kind of comes from just what is our game what are the characters you know so and we can have all sorts of great discussions about well who's responsible for that vision but it's almost a chicken-egg thing is that if it's not communicated to you from whoever a designer or creative director whatever of this is the way a character is supposed to be then you're probably possibly going to be programming it wrong anywhere if it's a data-driven system then designers might be putting it you know so now you've run into this whole well know it's wrong but I can't tell you why I can't I can't explain it well you this is what you told me to do and so I mean we've all been there where there's this okay we need to back it up never mind the game character what is this supposed to be what are we trying to emulate and everything so that's that common language that I think you've mentioned there's also an interesting balance between making sure that like

managing expectations but also trying to not go too far so you add because you risk basically sporting your own blinders and blind spots to the other people and you've like basically share people missing the same kind of things and striking that bombs is interesting it's interesting very political you I think it's also important to have a multiple district multidisciplinary team or QA is embedded into your team and they are not just someone that it's not just a person that goes and tests your staff is a person of this helping you build that thing so that way it's not that you have to just write a document saying like this is what respect from the AI it's something that you can talk with them about it every day and if you implement something new you can go and speak with this person and say like you know I've changed this thing so you can go and have a look at these potential things

that I think can happen but I'm not sure and I'm certain you'll find more than I did and we can continue talking about it it's I think as Roxanne said this morning like the sort of team structure they had in what stocks for the civilian AI it's the sort of team structure that everyone should have for AI like having like designers and QA and programmers and everyone like animators everyone like part of the same thing we are all in this together building something together I think just having the embedded QA and just in different teams depending on what the different disciplines are at the game if it cuts down a number of steps or you know just somebody can say you know hey Sergio is this supposed to be happening and they're right there rather than you know having you know - am I even terrified of walking into the QA area or vice-versa them coming over it's so much he said when you sit right next

to each other you know you can just go and talk and rather than having to go to the other side of the office or something you know and it almost inherently reduces their responsibility they're not testing everything in the game they're working it with you know I you know Roxanne said you know there's the civilian team or versus this type of team or or people who are building new characters for the game compared to level designers that have their own QA yeah you basically need experts for okay yeah that's good anybody else had experiences with that yeah on the QA thing I mean you know having somebody you know embedded into the team that you can discuss things with and letting know beforehand like hey this is what I added this is what we're expecting like this is the behavior that should happen and if this is not happening then you know let me know you know setting expectations for the QA team is huge rather than hey there's a new build see if anything's screwed up right great so um at the risk of starting Wow there's

there's two potential you know religious wars in in this panel here how do you go about approaching we touched on this a little bit scripted AI versus emergent AI what are the different challenges and how do you would you approach it differently in those those situations we mentioned a little bit of this earlier in our problem section about how would you approach those two different sort of things I think the main goal is to have something like this you know possible it's something that looks like it could happen that way so either for scripted AI or for emergently AI in the end it's kind of like the same thing everything has to look like they are intelligent and that sort of thing what they are trying to do is yes it's a lot of smoke and mirrors all the time right so it has to be like yeah

looks like they know what's going on they know what they're doing so to me you have to know how to break both things and it's in the end you end up using the same tricks the other thing maybe with scripted AI is that as we were talking about before that you don't know when things are scripted sometimes and it's like like knowing if it's actually the code that is broken or if it's some script that is not it's not built properly and it's not showing behaviors that are coherent with the rest of the AI or that's what the hell a little bit of a retrospect for some of you may remember we had a panel here a few years ago where the concept of the Green Dot which was famous in the in the guild circles and after that lecture were they actually during debug stuff they would put a green dot over the head of characters that were under scripted

control and the first thing that that did was immediately route any complaints of well this is acting weird it's not right well did it have the green dot over if it is it was scripted talked to that level designer who wrote that script for that particular situation and so even just that helps to mitigate that it will is it scripted or is an emergent you know just chopping that problem space in half of who it might what system might be responsible for it is is huge so I very much recommend going back on the vault and watching that I don't remember what the name of the panel was but it was few years back so yeah building on master juris stuff I mean the the point where this splits between scripted and emergent is in the resolution step like identifying the problem it's the same approach and then then okay now we need there's a problem we need to fix it okay no but now we can figure out it's scripted so we diverge it over here and of course in an

automated testing but other than that's the general approach should be the same Mike you've done a lot of work with where systems where you had both scripted and emerging AI and what was your you know was there a mentality shift of how to approach that with you well I think the the hard question is really in the moment is this thing scripted or not and that's easy to solve with tricks like the green dot which are super useful the the emergent end of the spectrum I think is is actually where it gets tricky because you have to ask your designer or whoever's owning the vision of your a I like how attached are you to this exact playbook because if my AI doesn't do a and then B and then C are we gonna have a problem or is it cool that he decided to do some random arbitrary thing that just was flashy in the moment so knowing again it comes back to the expectations thing you know knowing what you're you're looking at

and what counts is broken is is really important but then having that again comes back to the key of a my scripted right now you have to know which direction to go down that fork of am I taking the solution that's going to be kind of hard coded and scripted or am I taking a solution that has to be more systemic in the actual AI implementation right how just dealing only with emergent AI how do you tell if it's broken I mean that's a highly subjective question by nature I think the the best thing you can do is is say does this look plausible does this look like something that is in character for the agent in question and if it is then I mean really who are we to judge our agents for being themselves and being free well that got kind of weird powers of the AI you know I don't and Mike you actually you had some of the same you know situations when you're working with

planetary annihilation where you were like well I didn't tell it to do exactly that but damn it it looked cool and everyone was fine with it yeah is it similar sort of if the designers are happy with the behavior and the behavior looks reasonable and not broken then it's it's not broken as long as it's not doing something that's completely derp stupid it's like no if he's not running in circles or running into walls at me it's fine if he decided to do some random off-the-wall thing but it looked awesome and the designers are fine with it then run with it I think one way I phrased it in the past is if the player says oh I did not expect that I can see why he did that it was a cool move but man I wasn't expecting it that's a cool moment you know so okay but yeah he said that's where you get into the subjectivity of is it broken or not and that's a different difficult space so all right moving on best practices of actually then writing code what can we as programmers do to write

code that's that's easier to test easier to debug etc like that what's the oh boy animals like I'm ready for this one the separation of concerns and do not allow any crossing of that boundary what do you what do you mean by separation because like clearly clearly separating out sensor systems like data storage systems black boards that sort of thing from the various decision logic code and action code that sort of thing have that clearly separated in in in a modular form so you can easily mark that and and basically run unit tests on every level so you're not just left with some massive integration test and yeah helps you to just be able to compartmentalize you get harsh down your what what could be wrong and yeah I mean mocking stuff up like quickly replacing the sensory

inputs and saying okay so this scenario right here given given this world state but with these tweaks let's see what the AI does and and make sure it's it adheres to its personality stays in character what else what I think I think you should try to keep solutions as simple as possible and half code that is readable and that people don't try to do like very fancy stuff yes because we are engineers and we look like fancy you know like super architect ofstuff I don't think that's the right way to go for for AI for character area at least it's probably better to have a simpler solution that then if you have problems with it's going to be easy to understand easy to debug not only by you but but by anyone else in the team as soon as you start using like a lot of templates and stuff like that is like knows like it becomes

more complicated and you know try to keep it simple I would say behaviors are complex enough we don't don't make the situation worse by writing all sorts of crazy stuff and especially modularity is part of that too so when you're writing that for for testing you just you gain so much for what you can reuse in the development of it afterwards it's Mike you have anything either Mike clear ownership I think is important you know what what module homes what what what module is responsible for the lifetime of these objects because if you're if you're passing them around I mean it should be clearly in the code like okay I have a platoon manager it creates platoons those platoons go out and do behaviors the platoon manager owns the platoons it doesn't matter if something else needs a pointer to that or whatever they could have a weak pointer it doesn't matter the lifetime has to be owned by a single object i when you first said you know clear ownership my thing jumps in my

head too is clear ownership from a programmer standpoint you know because there's a lot of times there's you know we're work finally at the point where a lot of teams have more than one AI programmer and or but even you know gameplay server programmers and stuff like that I'm just saying this is the stuff that I'm responsible for yeah you can tap into and here's your interfaces and stuff like that but that was the first thing it jumped into my head but yeah I liked where you're talking about about the ownership of different systems you know in the AI for the record it's always been Mike Robbins fault so I mean I listen to this this conversation and I can't help but go back to kind of my roots in computer engineering and you know what would someone who has been doing this outside of games for 20 years think of the conversations we're having now in the game industry about how to write testable code I think it's kind of

interesting we'll go with that word because there's not a lot of there's not a lot of awareness I think in the game industry or at least people act like they don't have time to go do all these things that make your code testable and this is stuff that's been like common wisdom body of knowledge in the in the industry mainstream for over like two decades yeah and we're just now starting to get up to this idea that hey maybe if we can test our code it'll suck less we're growing up that's an excellent point so moving on to Nixon here's our second potential religious war what architectures you know of the the primary types like your behavior tree state machines utility you know more of the black box scene neural network type stuff what architectures lend themselves easier to testing and debugging and maybe a little bit of why some of the pros and cons of things you would run into with that yeah that's a good start

yeah why is this broken I don't know anything that you can visualize so behavior tweaks you can always build tools to be able to see like what the tree is doing or steam machines you can know like we are in this state now or with you know that sort of thing anything that you can build tools for and then kind of like have a more visual representation of what's happening happening inside well it's just more easy to wrap your head around the fact that it is in this state or it is in this branch even without visualization you can mentally know and so therefore this is the only stuff that should be available for yeah ok no non planner stuff non planar stuff white because it I mean stuff where you have easy clear expectations like a state machine or behavior tree yeah when this kind of goes back to the to the scripted versus emergent thing to

is if a you know who was it was in the the last session was talking about well you can you know backflip and throw the grenade off of four walls and and yeah it figured out all that stuff out there yeah yeah but you're going was that supposed to happen so but of course you can trump that with but it looked cool so therefore it's fine so now we're back into this old well which is right and which is wrong so what else any other I think it really depends an awful lot on the people you're working with much more than the technology you're working with because certain teams have reflections of how they think in their projects and I think every team does that to some extent and the more you have one mentality about how your AI is supposed to function the harder it is to to get away from that and decouple from that and look at your own blind spots and say okay now we got to go find out what we forgot you know there's a lot of impact between the way you organize your

company and your team and you know even the the sub teams within that group and the actual technology that you produce and so if you're building something that's fundamentally incompatible with the way your team thinks you're just gonna have a bad time like there's no two ways around it it's it's going to end up being awkward and uncomfortable so if you can identify like just paradigms or you know even you know general modes of thought that your team spends a lot of time with that can be incredibly useful for going back and saying okay now we're gonna we're gonna try this architecture that happens to align very well with the way that we think mm-hm and so it's an interesting balance consideration in between you know choosing the architecture that matches the needs of the game you know and say yeah there there there can be a little bit of friction there as well you guys tend to think like this however the game system kind of requires us to have an AI like that you know this is goes beyond testing this is this is like little shoots you have to have some bit of a

transition and Retraining and stuff like that of getting the team to think and I think ultimately you know if our goal is to serve the game serve the purpose as a game we might have to change you know the way we approach things whether it be testing or just the architecture to begin with so yeah that's an interesting you know you never think about well I think in terms of behavior trees or you know scoring everything in my life is utility or something you know not many of us think in terms of planners I don't believe so yeah that that's that's an interesting consideration there so anything else on just in general on best practices for coding and programming and our work disciplines as programmers well we brought up modularizing the AI as much as possible that also lends itself to easier time testing and easier time debugging because if if your problem is in one part of a a I well most likely your problem is in that specific module

not somewhere in this entire mess of AI code is there a way that you would then like be able to firewall off okay well I'm going to just continue to work on the premise that these modules are perfectly okay I can't trust everything coming out of them so that I can focus on something in this particular module how would you go about doing that just in a general sense we've already mentioned like iteratively testing as you're putting things together so that you can already say well I have already tested this this particular subset of code so we know that that's good and the inputs outputs you know all of that stuff is what I expect and hasn't changed in the last three months so it's probably not that so using your example like the tactical manager for example and let's say it's giving orders to individual units you know the tactical manager is spitting out you know it's formulating good plans for tactics and it's contribu know sending good but you see units running all over the damn place in places you're not expecting could you you could assume know the plan the tactical plan is and the orders coming

out of it is fine so it must be something right in the execution maybe you know something in your navigation system got worked up or in passing the the tokens of here's your command yeah or the information that's being passed in like maybe your inflam at influenced map is wrong you know maybe that that code is wrong maybe some like you were talking about earlier so on the other side of the tag Tuukka manager okay yeah so it's it's just doing wellness to acknowledge okay yeah but so at that point man I hate to use the term because we've already like you know maligned it you've created a black box that you know are you're comfortable with what's happening in that based on the inputs and the outputs and you've kind of sealed that off is this is okay so is there a danger to that well yeah because the book could be there if you assume that it's not there you could spend hours trying to track it down somewhere else only to find out that oh well I mean this code was not as after all it was yeah okay excellent anybody else these general principles of being an AI programmer to make it easier

on yourself or others ultimately you're not alone so you are in a team with other people and things like getting code reviews and half of the people look at your code before before you check in or we have two possible options and you're not sure about one or the other you can go to other people in your team and try to get like ideas for other people so you know what I found is interesting about that is it doesn't have to be a lead or somebody above you or somebody any more experienced a lot of times even just getting another pair of eyeballs even if it's a junior engineer or something makes it makes a huge difference okay ya know everyone like even if you are super senior you can always learn something from from anyone in the team like even from someone on just you in your team and it's super junior you can still learn a lot from him okay so as a practical example for behavior tree setup I would

generally unit test actions and integration test trees until the point where a certain sub tree is reused in a lot of different systems and a lot of different other trees then I would have a set of unit tests for that as well so it's kind of cornered off as a safe safe thing and it becomes another safe with an asterisk a it seems safe until it's refer for various values of safe it becomes a blind spot let's move on then what what tool could we provide ourselves as programmers you know for right now just for us for you kind of touched a little bit on some of them hearing but let's put a finer point on it what are some things that you use either for yourself or that are off the shelf I mean one of my primary priorities is to make sure that the authoring environment and the

debug insight environment is as much as possible the same so you're looking when you're looking at the execution of the state of running AI you're using the same visualization and it's same window whatever as you're authoring environment says an easy one to one switch over there in some cases that works in in other cases that may actually be difficult you know to have it in game rather than in the authoring environment well you did a lot of work obviously with unity where some of the some of those things are a lot easier because you can have stuff in the inspector window right there and being able to write your custom things in the inspector so it's finding the granularity and the visualization is an interesting challenge sure what else what are some tools to me the most important things is debug graphics yes having demographics for everything that you work on like every time you add something new yes app demographic graphics to it that can help you understand what your system is doing

internally and you can also show it to other people and other people can understand what you were trying to do with that and also you know tools is always getting tool support is complicated sometimes because you know it's not only tools for AI it's calls for mostly content creators so if you need a tools team to be able to work on tools for you maybe you won't have a lot of you want to get a lot of support and if you want to do it yourself we are AI programmers not to programmers sometimes you know like if you if you have to build this thing that interacts with us with the rest of the pipeline or whatever like it's probably too complicated for us already it's going to take us too too long today that's amazing how the this something is simple is drawing a line of you know what's my what's my line of sight who's my current target and if you have no familiarity with the rendering engine where's the code to draw a yellow line on the screen

please you know and it worries somebody who's you know you know in the in the the graphics engine people the rendering people well yeah that's right here right you know oh thank you what you mean you're not gonna do it for me so so what you're saying is rather than hey I've coded a bunch of stuff and it's almost doing it iteratively along with that is I wrote a feature that I'm now paying you know doing the path finding just something simple I want to be able to draw it in the world to make sure that that path is legit rather than an easy walking where he's supposed to be walking I want to draw it in the world so I'll give you an example so we had a sniper and Splinter Cell blacklist and basically the sniper was going to use his sniper rifle if you were far enough and if you got closer to him he would switch to the to a pistol and become like a regular guy and then if

you got away again like he would switch back to the thing right oh I see where this is headed so yes just drawing those little like circles around him so people understood like why he was changing the the weapon that that was super helpful and please tell me that that wasn't the exact same range that you could sit there and just do this back in the fourth and he would keep switching weapons right yes I knew you would be H by by doing this but if you can see the two ranges understand once he gets outside of that or inside of this other one yeah and then let's say that you had something else on top of the of the range right like it's not when he has the pistol it's not only you are far but it's also that he's not engaged with you in combat or something okay so you can add something else like a little ball on top of him that says like now I'm in combat with you so I'm fighting like you it's what we get to the iterative thing where and now that we've added this extra rule I need a way to be able to see that yes so you know that it's clearly you know the the radius is

showing me that I'm far so why is he not not switching the weapon but now I know that he has something else on his head so maybe that's that's why I said I didn't a lot of that is very helpful I'm like well again I think my favorite trick for for debugging code in general is just a state capture I want a way to be able to at a moment in time just dump out everything that the program has in it in terms of what it believes about the world and you can learn a lot about your code by comparing a state capture and your expectations with the way that the debugger actually steps through the code you'll find that it takes different branches and you might have expected it to or loops a little longer than you thought it should or whatever and this is a great way to discover your own blind spots and brand kind of do some of that iterative testing as you're going dump thump to like a log file for an employer I mean you can get as fancy as you want it really doesn't matter I think sure what you have available at your studio is probably fine and if you

can stomach using it then just do it and if not go write something better maybe I'll say good aid for tools for us anything my I think my favorite trick that I stole from our navigation engineer was the ability to tag an agent so that you could set a breakpoint in code so that when that specific agent entered that code path the breakpoint would hit oh if you're in a world where you got you know did dozens or hundreds of characters and say okay so break when this value is greater than two not him neither or not you're sitting there just looking at one box if wait until that good ends in four eight three two oh yeah so now I like that because then that narrows so how do you how do you tag a characters in code or can click on somebody or in our game it was a key bind you selected an agent he would hit a key that would set a flag on that agent and then basically all you're doing is you're setting a breakpoint inside of an if statements like hey if

the agent has this tag set to true well that's love that's so lately simple dude now you've burned it so you can't use it in the simplest trick in the in the book thing next year that's fantastic hey did you have any I usually have a debug gun so I find a character and shoot with that gun and that then starts the session recording on that character and what one thing on top of the the breakpoint setup I also really like to have the ability to overwrite values in the debugger yeah so so we're like saying this is the current game environment we're playing and I've just added this new behavior but I haven't actually implemented the action yet so I'll just go to the the default handler but I just want to test it out so I'm just gonna set set it to always return this value or just basically the augmenting the reality of the agent and going even though you have like full ammo or whatever I want to see how you can respond to not having any ammo and

so this beats the heck out of going and commenting out the real code and saying nope in this case ammo equals four always and because then you always forget to go back and change it right you got a weird idea of fun okay so then turning it around what tools can we provide you know designers in QA and some of them actually apply things for ourselves like just the visualization or the the debug gun which I love to see that in action I'd love to use that on real people actually so what are some of the things that that are good to provide to designers to QA to other non programmer types I think God it's more I said it's also very useful for us but a lot of in-game things like being able to as they were saying select the guy and maybe half a command to kill everyone else in the level except that guy or being able to spawn some people around him to try and

generate a specific scenario where you need like you know you need a sniper and something else so you can go them by using debug commands as you spawn those guys in there without having to go and create a whole map for that you can have a lot of camera controls like fire flying camera so you can like look up things from different angles or you can move the player somewhere else or you can make the turn the player invisible for people and all those things that you want to do you want to kind of fake to try an stress test your your III while you're playing that was a lot of anybody writing these down that's a cool yeah just basically I want to be able to completely screw with the environment in any way in a non you know it's something that doesn't violate the AI but so I can set up my own scenarios visualize view these scenarios to just recreate a very specific

situation is what you're getting at there was have a cool feature and strangers hell blacklist so well for those of you who have played the game basically the lkp the last-known position and were it was basically being drawn and people knew that you had been there or whatever so they thought you were there and I saw you yes so to stress test that thing I just remember we had very quickly if you pressed both [Music] sticks in the controller you started you were able to kind of like fly around so so you started doing that and kind of like flying on the other side and then flying on the other side and I'm kind of like like messing with them all the time to see if they were updating the lkb properly if they were what they were doing when they were not seeing you in there anymore so it's kind of like the invisibility thing like toggling it like on and off or like doing all those things like it's you know it's a lot of tricks but II you know to be able to

speed up the process of getting to the thing that you're trying to test is well anybody else have or some tools for designers and do it I don't know if this is applicable for every game but for planetary annihilation we had the ability of just basically spawning AI on AI games and just having them run as fast as possible no no visuals no nothing just for the purpose of finding crashes you know that's that's an easy way where you can find those weird like you know you know stack errors finding overflow problems finding you know poor interiors where they're not supposed to be fine that sort of stuff well and that's also you know a common thing for doing balance stuff too in things that inherently have well it may be even things that don't necessarily have AI put players but you have created you know AI is little played against the strategy games and and and I know civilization would do this quite a bit where they would just crank up the sim speed and say okay well this dude keeps

winning something's up with his balance that's not a I everything but the point still I kind of like what Sergio was talking about I want to quickly get to test this situation you're going the other extreme of I want to quickly test all situations sure cool so like selective session recording kind of like kind of inspired by the how the new consoles have that share button they can hit the share button and you share like a couple minutes before and after the situation you currently are in having the the designers and the QA people give them basically like a panic button like something's wrong on the screen right now and the session is just on a loop recording the next laughs whatever minutes and and some remaining time afterwards and then you go at that point inject an event a marker this is where something the button was pressed and

this is the context I have to look at rather than here's like 20 minutes a session and around the time where this guy is on the screen and yeah yeah well it's very similar to the rolling 30-second buffer and fraps for example but what you're saying is now I also have a whole lot of game context that's been saved with that yeah instead of just video I mean I like having it as just a custom stream so you can also go oh I'm gonna combine this with the video stream or I've got some other debug systems and just go here's the same section of gameplay so that's great for the designers in QA but then now you have a deliverable Dave here this is something went wrong I have viewed this and here's a shitload of data so okay cool yeah that's have anything real quick yeah suddenly do uh really extensively in Guild Wars 2 is just a little cheat commands that you literally type into the chat box and they range from things like you know

boost me to max level to you know kill everything in a 5,000 mile radius around me or you know whatever a thousand miles wow you can type whatever number you want in there it's pretty open-ended like the the cool thing about it is that those are available to anyone on our development branches so anyone in QA or design can go at any time spin up their own map and go say okay what happens if I just arbitrarily kill these three guards but not that one and then they can play it and find out what happens and then they can go back and pick another guard to not kill the next time you know or whatever they want to do and it's all done through these just little literal literal text strings that you type into the chat window and you can put parameters on them and get real fancy if you're you know so inclined but the the basic power of changing the game state to do whatever you want it's what search is all there yeah it's exactly that mentality of like you should be able to do that at any time all right well I wanted we've got about three four minutes left and I told the guys I said

if you were to just evangelize one or two sentences oh what's the if you could tell your team tell every team on the planet this is one thing to remember what would it be and you know so they thought about it we talked about to see yesterday afternoon and so unless you see what you guys came up with who has something that that's this is the one thing I want everyone to know about making life easier for testing political first the developer breaks separation of concerns get to sit in the corner we built in punishment to this wasn't expecting that then that's that that separation of concerns you said right okay yeah so it gets to sit in the corner what if I like sitting in the corner go ahead anybody who's got one you know just keeping in contact with your QA team I I found that to be incredibly helpful you know keeping them apprised letting know what expectations there are and if you have any ideas on

you know what could possibly break the I mean share it with them you know if you don't have time to test something because you're working on a feature let them know so just talk to the relevant people and listen to them Brian Schwab mentioned this earlier when it's actually just listening to what you know yeah okay cool Mike yeah I think the the main thing that I would advertise is just you know you need to make a tough call about how attached you are to the the specific set of outcomes that you have in your imagination because as we've talked about here you know a lot of times you'll get into a situation where things don't go as planned but they still look okay and it's probably time for a lot of you know designers and AI programmers alike to come to terms with the fact that if the players having fun we've done our job and we don't need to be holding ourselves to any other metric besides that okay cool yeah I like that one surgeon I'm just going to repeat myself basically testing that your stuff works

it's pretty simple but testing if it can break or not is more complicated right that's what you have to do try and break it and when you think you can't break it anymore go and find someone else to break it for you like martin moss yeah i can't wait to see him again you break things i heard you're really good at this so X alone I appreciate that it is 629 we don't have time for questions and I don't know sort of debugging your your games for you I which I don't think we could do in a minute but please help me thank my panelists is a lot of good information there some of it seems obvious but we always forget it and so that's why I really enjoyed having you know having you guys up here so please help me thank Mike and Mike and MO and Sergio [Applause]