Winding Road Ahead: Designing Utility AI with Curvature

[Music] [Applause] hey so like dave mentioned uh this is not actually a talk about utility ai from the content creation standpoint this is a talk about utility ai from the content creators standpoint and where this all came from was about a year ago after gdc i had this inkling of wanting to make a tool that could turn utility ai into something that was intuitive and accessible for the average ai developer um this was partially selfishly motivated i had a lot of designers at arenanet that were working with utility ai projects that dave and i had done there and other things like that and there were a lot of gaps in the education there there's a lot of other ai developers especially in places like the ai summit at gdc where i had conversations with people about how they had difficulty with educating designers or even programmers about how utility ai is supposed to work

so there's a specific challenge that keeps coming up in the guild wars 2 franchise which is probably easiest to summarize as why the hell does my ai do that thing he's supposed to do this thing so the fact that we keep facing this question over and over signals to me that our tools need to be better uh we need to be better about creating tools for rapid iteration and prototyping but we also need to be better about creating tools for debugging so curvature is a standalone tool and that might sound like a strange decision uh since we already have a game engine with a bunch of utility ai technology built into it but my thinking here is pretty straightforward coupling to a particular game engine is a constraint it carries a lot of baggage there's things that i have to do if i'm going to work on a utility ai experiment like set up the rendering and make sure that all my shaders are right so that i can actually see my ai agents running around and maybe if i want to have some physics involved i have to get all that debugged there's an awful lot of extra stuff that just has to happen for a ai experiment to come to life

another problem that i run into a lot personally is that a lot of engines out there are not great at real-time editing and what i mean is they have to do data pipeline builds and this means that every time i need to change something i have to run a process that bakes my data out does some processing of some kind and it generates something that can actually run in the game this is murder on iteration time so instant feedback is actually the secret to building a system that's easy to understand as a result curvature is 100 self-contained my hope is that it's really easy to set up and play with and there's no baggage from any other engines or technologies involved it's literally one exe file written in c sharp you can drop it on your desktop run it and go the last point that i really want to hammer on this is that this is a fully real-time editing experience everything you play with in this tool is reflected immediately back to you so utility ai is an architectural

category of approaches to making behavioral decisions with an ai system the basic principle of utility theory is in a nutshell we're going to score all the possible things that i might want to do as an ai agent and then i'm just going to pick the best one for some value of best this is specifically built on the infinite access utility system that dave mark introduced here at the ai summit several years ago i don't have time to go into a full treatment of how the architecture works but the basic pieces are pretty easy the first thing that we want to do is get a whole bunch of input data about the world so we've had talks earlier in the last couple of days about knowledge representation and understanding the context of what our agents are doing in the world around them this is exactly the kind of thing that i'm talking about now taking data from the simulation and processing it in our case we do this through what we call response curves which are basically just small little compact formulas and those formulas help us decide that for each possible behavior

what is the score should i continue talking right now should i take a drink of water should i blow my nose whatever what i have a lot of options of things i could do right now what's the winner so the recipe for this whole system is really simple uh it's input data plus response curve equals score and then you just pick the best scoring option so before i get into the the bulk of this there's a couple of quick things that i just wanted to point out curvature is a spare time project it's not sponsored by arenanet it's not a public thing it's actually open source just me on my own time uh there's room for improvement but i think that's actually very exciting because that leads me to a challenge that i would like to issue to everyone here specifically i would like you to leave this session thinking about how you can improve ai tooling because i firmly believe that all of us can and all of us should so uh step one of this whole process is knowledge representation uh i'm gonna quickly define some

requirements that i impose on this system uh it's gonna be a lot simpler than most games would work with but it's just enough to get the the basics of a simulation running so that we can play with it so the first thing i wanted to do is identify properties of agents so an agent is basically just a character in npc or whatever running around in the world they have properties some of those are user defined and some of them are calculated from the simulation but regardless of where this information comes from everything needs to have a name this is a personal line in the sand of my own where i want to be able to refer by name every human readable string to every single piece of my ai data the next part of this that's important is validation i want to be able to have an expression of what numbers make sense for this particular type of data so we can do numeric ranges and we can also encourage or do user-defined tags if you were here for res graham's talk earlier this should sound awfully

familiar so let's look at the ui real quick this is what the ui for knowledge representation curvature looks like the idea here is that we have a series of records and each record is a piece of data about the world and it has a number of attributes the first one that i want to point out is name so we have these names over in these boxes here that describe what this data is what is its semantic significance the next thing we want to look at is where does this data come from in this case we have two choices it can be a property of an agent or it can be something derived using a prefab equation i'll describe prefabs a little bit in a second another piece of information that's very useful is the validation does this number have to be in a configurable range or is it a predefined value like a tag or an enumeration so you can see here there's just a couple of different pieces of ui that will dynamically spawn into the window as you choose whether or

not this is a configurable range or a predefined value you'll get ui that automatically helps you flesh out the validation for that data point then down here we have this little use distance prefab drop down which is a nice little transition to why we do prefabs at all so the idea is there are a lot of things that we can compute about the environment we can compute distances we can compute line of sight we can do all kinds of great stuff and we don't want to have to represent that as static data that a designer enters this is stuff the simulation should be able to provide to us for free or small computational cost and then the ai can reason about that information so we could use this mechanism to hook up a bunch of things we could do pathfinding we could do raycasting if you're here just before lunch dave gave an excellent talk on influence maps which would have a lot of potential in a system like this right now in curvature specifically i've been deliberately minimalistic because a it's less work for me and i'm lazy

but b it's actually easier for the user of the tool to not have to keep all of these possible inputs in mind the main challenge that i've run into with explaining ai systems to anyone is that when you get into a certain level of complexity there is just so much going on that is virtually impossible to keep it all in your head so we have a whole lot of data about the world the next thing that we want to ask is what kinds of things can we do to influence that data how can we feed back into the simulation and do things so again i'll start with some quick requirements that i imposed on this project i wanted to have a set of actions that the ai could choose from and these would be things like move to a place or start doing this new set of behaviors or so on and so forth i wanted to basically have a behavior be an actual effect in the simulation so anytime an agent chooses to do something i should be able to measure that response

in the world so again i could move my agent to a place i could change a property of an agent uh and all kinds of different little things that help make a simulation actually move forward through time there's another little element of this that's very interesting as you get into more sophisticated simulations which is targeting um the number of choices that we have to score grows in sometimes complex and difficult ways with the number of targets that i have to consider times the number of decisions that might interact with each of those targets so basically we want to limit the number of times that we do redundant calculations on parallel targets especially because some things just don't make sense to have a target right like if i want to yawn i don't do that to somebody that is a thing that happens to me personally and that is contained within my agent so again let's go to some ui here um i want to look real quickly at the core of what are four behaviors that i've

defined for this particular demo um you can see here we've got a couple of pursuit and a couple of evade and there's variants for the blue team and the red team so each one of these has a specific drop down that tells the curvature system what happens when an ai chooses to do this behavior what does it mean in the simulation as an active event for this behavior to be executed and then of course there's also this little ability down here to change the targeting mode which lets you control what uh context makes sense for me to do this action in so like i said before if i'm yawning i can say no that can't target others and uncheck that box so this is where the ai starts to happen um the real question that we have is why you remember if i said at the beginning uh the problem that we kept running into was why does my ai do this and not that so why would the ai do this and not that the answer to that question is considerations and again going

through the requirements definition um it's really just all about the why why would i prefer to do one thing versus something else the goal here is to account for real world relevant data as much as possible but we also need to contextualize that data because it's not the same for me to say i want to buy a plane ticket how far is too far to fly and i want to walk to lunch how far is too far to walk those are both distance questions but they have very different contexts so ultimately all of the different input factors that we're gaining from the simulation scraping out from this knowledge base system they all have different contributions to the score of a particular behavior and considerations are how we reflect that so this is a little bit wallet texty but i'll kind of point out some of the most important aspects of it a consideration has an input uh this little highlight here kind of shows that we have currently selected to work with the distance to

target input there's also validation which we can do from this view and then there's this interesting little mess down here for a response curve now i've seen enough demos of utility ai to know that as soon as these little uh two axis plots with the blue curve come out a lot of people start feeling a little freaked out that's fine because there is an escape hatching curvature which is right here the presets button so this is the fun part um i am a firm believer in the power of natural language this is sometimes at odds with my desire to make video games because they don't often overlap very much but i think sometimes it's true that being able to express something in a few concise words is much more impactful than spending a bunch of time trying to play with all these possible sliders and ui elements and you know rolling numbers and all this to try and lock in an exact shape that you want so this is the curve preset generator dialog these two drop downs here are basically

highlighting the two controls that we have this system is starting with a list of over a dozen different preset curves with no inputs it will let you choose from amongst all of them but as we refine our request for a curve it's going to use the natural language description of what i put in these boxes to filter down the list of suggestions and only show me the curves that match the exact description that i provide so here's how that works the top input is as input values grow towards one or as input values get bigger basically the consideration score should now we have some options here we can become more appealing we can become less appealing we can stay the same because we don't care or maybe we can fluctuate a bit because we're being random so let's look at what happens if i plug in steadily less appealing scores as inputs grow to the right and you can see that graph there

and i've chosen a constant or linear rate of change in the second input the net result is that we get exactly what we asked for we have this graph that does a perfect linear decrease from yes to no across the input range but we don't have to stop there what if we decided that we wanted to have a curve that started decreasing quickly and then slowly decreased out towards the top end of the input range this is something that dave and i would refer to as a four poly or a quadrat curve so we can see here i plug that input into the preset generator and that's what i get so there are other possibilities in those drop downs uh what if i decide that i just am not content with that and what i really wanted was something that is slow at either end and fast in the middle and you should already have a mental image of what this looks like but the curve is really easy to visualize this starts slow accelerates in the

middle and then slows down again this is an inverse logistic curve and then just for the hell of it i figured i'd do a consideration whose score fluctuates dynamically and there's a sine wave i didn't bother animating the arrow because that was a pain but you can kind of get the idea of where this is going so i wanted to pause here to to kind of cover a couple of thoughts the natural language feature has gotten a lot of great feedback a lot of people really enjoy being able to work with it being able to specify things in what feels like a much more intuitive manner it's super easy to implement it's literally just a tag on each preset that says these are the different properties this function has it increases quickly at the beginning slowly at the end or whatever the curve might be and then the drop downs are just a filtering mechanism so it's super easy there's not actually any natural language ai i'm sorry if i disappointed anyone there's no real genuine difficult work going on here but this is

low-hanging fruit for everyone who doesn't want to have to think about the equation form of all those response curves so this is a very different type of intuition that people can develop about how utility ai works without having to think about it in in more math or formal terms now i think we can go further than this uh you'll notice that in the screenshots there was this notion of as the input grows towards one uh i'm not totally happy with that i would like the ui to say something more uh relevant like as the input which actually is distance grows towards the upper end of the range which is 75 meters then i should get something that says as distance to target grows towards 75 meters the score should and then the designer fills in the blank so that's kind of an idea of where i'm going with all this i wanted to jump back to the consideration view real quick because there's another escape hatch that's worth calling out uh real quickly and that is this wizard button up here in the top right

uh the consideration wizard is genuinely a wall of text but the idea here is that if you've got a moment and you're not entirely sure what you're trying to accomplish you should be able to sit down and read through these instructions and get a perfect idea of what you're trying to do and why so each of these panes each of these tabs in the wizard goes through a different part of building a consideration the first one is input selection and this one here you can see is chosen distance to target i then can do the parameterization which tells me what are my relevant ranges i care about everything between 0 and 75 meters and then of course there's the actual response curve system which has the preset escape hatch again so that you can go look at the natural language version if that's what you prefer there's a fun little twist on the end of this one which is that naming things is actually as i said before very important to me and one of the ways that i've emphasized that is by providing this link which goes to the curvature wiki and

actually explains in a very brief and digestible way how to do good consideration names so this one for example is prioritize closer targets this kind of gives you a real quick and three words idea of what the consideration is doing what it's trying to accomplish for the ai so this is probably my favorite phrase right now is building intuition through play i really like this idea of experimenting and messing around and just getting your hands dirty and by that process alone coming to comprehend how a system functions and how to control it so this is an example of a behavior with some considerations on it there's a set of inputs right up here that we can look at that correspond directly to the inputs that are in those considerations so you remember things like distance or is this target on a particular team i can also see the response curves here

which show that for the current values of the inputs these are the scores that the considerations will have now as i play with these specific input drop-downs we should be able to see these response curves moving in response to what i'm doing so i will change the distance and you can watch that blue dot on the bottom right box slide back and forth that is a very visceral kind of instant feedback mechanism for someone who doesn't want to think in terms of equations but does want to think in terms of geometric shapes so you can play with these and then if you want the numbers you can go over here on the left and look at the actual breakdown of the scores of what the considerations produced as they calculated their response curve values and what the score of the behavior was at the end of that process so this is useful if you're looking at considerations within a single behavior

it doesn't do you much good if what you really care about is how different behaviors compete with each other so there is another tab in curvature which actually targets that specific use case you can see in the center column there are the four behaviors that i introduced at the beginning to pursue and two evade right now i have the variance for the red team selected so i'm dealing with behaviors that only an agent on the red team would actually try to execute and the same input controls are here as well as we had in the consideration system with a twist if i have two decisions that are both based on distance i only get one control in the ui that controls distance it will intelligently collapse all of the different inputs that are the same basic input data so that you don't have the problem of feeding considerations for this decision with a distance of 30 and considerations for this decision with a distance of 50 and now you get broken behavior as a

result so the last piece of this is there's a little chart and this is a lot more uh impressive with larger more complex demos but this is enough to get the point across um you can see here there's these two behaviors that are being scored against each other and the winner of them is highlighted down at the bottom so in the situation that i've configured with these sliders and everything up on the top uh the particular decision that my ai will choose to pursue is actually to go chase down an enemy of the red team or probably a blue guy and we can see that the score for that is 0.171 so this is all different ways of gaining intuition of a running system but it's not quite enough i wanted to put in a last twist on the experimental uh intuition through play concept and do a complete sandbox simulation so again starting with the requirements for what i wanted this to be i wanted to visualize what the ai would do so if i actually took and built

all the data for an ai system and plugged it into this tool i want to watch them actually behave the other thing that i really found really important i alluded to this earlier is just debugging i want a way to go back and look at this stuff and understand where it went off the rails and why so this real quickly is the scenario mode of curvature um you can see up in the top left there's some red guys kind of hanging around a little base and there's some blue guys down in the bottom right doing the same thing i'm going to go ahead and play this forward and what you'll see is a whole bunch of these agents swarming towards each other and then collapsing as they run into each other in the middle of the field and what's happening is they're doing exactly the two behaviors that they were given earlier in the behaviors section which is to pursue their enemies and evade their allies that is the only thing these guys are going to do

you'll notice as they move towards each other they have this tendency to swing out and that is them trying not to run into each other as they go and then the fun part happens and they realize that i've run into my enemy i don't have a reason to pursue him anymore i have no reason to exist and so they stall and this is actually an event that you can get in the curvature debug logs i stalled because i ran out of stuff that do so the debug logs themselves are a little more on the programmer friendly side than the designer friendly side at the moment but this is something that is ripe for exploration i think down the left side is a tree the tree contains a list of every tick that has occurred in the simulation underneath each tick there's a list of every agent that did a thinking operation in that tick

for every agent in that layer there's a list of all the behaviors that they considered trying to do then for each behavior that they thought about doing there's a list of targets so relevant things that they could do that behavior with in the world the winning option happens to be marked uh in this case there's a small blue rectangle on one of the items in the tree on the left side this is just kind of a visual indicator of this is the thing that in that tick this particular agent actually wound up doing so the key here is actually the text on the right and that has all the details that we could possibly want um i'm not going to go through it line by line but basically from the top down there's more generic to more specific information you can look at things like the position of the agent in the world you can look at what their decision was that they actually wound up doing you can look at the comparison between that decision and other competing decisions or even competing targets for the same

decision and then you can finally at the lowest level of granularity you can see there's uh numbers there that represent the actual input values the actual normalized input values and the response curve scores for every single consideration that we processed in that tick so just to kind of bring this around to a little bit more open thinking the debug experience is kind of a work in progress there's a lot of stuff that could be done with it there's a lot of exciting potential there in my opinion uh one of the things that would personally make me very happy is to be able to rewind the simulation to a specific point i would love to be able to scrub forward and backward on a timeline and say this is where things went wrong i'm going to stop there change the data manipulate the ai design in some way and then run the game forward the simulation forward from that exact moment in time and see if it fixes the problem that i noticed another thing that would be

really cool is if you could modify the properties of an agent while you were running through the simulation so i want to i'm in the moment i'm watching these guys have a giant fight i want that guy to have 10 more hit points i want to be able to click on them and go 10 more hit points and everything just works um that kind of thing would be very exciting not just for me but for a lot of the designers that i know work with these systems um the the current limitation of curvature is you can manipulate the properties of agents but you have to pause the simulation to do it so uh you can inspect objects in the world those little uh triangular circular agents that were flying around in the simulation mode earlier you can click on them individually and get a breakdown of what they've been doing and why they're thinking about what they're thinking about it's opportunity for improvement here because there's a lot of information that could be there and it's very difficult to predict in uh kind of ahead of the time what information you might want that display to populate with

so i might know all kinds of things about this agent what he's been up to but the designer only cares about one tiny slice of that and this is a ux challenge to figure out how do i pipe that slice of things that the designer needs to know through that particular interface of inspection so i got a couple of basic ideas that i really want to leave you with um if you get nothing else this is the good stuff we talk a lot about tools for producing stuff we do a lot of work to make tools so that we can ship games uh this is a great thing i think tools are very important i think tools are a critical piece of how we get our jobs done but we don't spend a lot of time about talking about tools that just provide learning and exposure to a concept we don't spend a lot of time on educational tools that exist solely to help people build intuition through play one of the things that i think is very critical when we're talking about that type of tool is there has to be multiple ways to

understand what's going on and that's why there are so many different modes and angles for approaching data in curvature because what i want is for anyone to be able to find a route to that information that they're comfortable traveling on i want everyone to be able to sit down with that tool and feel that they understand utility ai because they can own the process of understanding what's going on under the hood and that means we have to give alternatives so iterating on the tool is just as important as iteration while you're using the tool a lot of tools talks spend time talking about the user's iteration time and how you need to make sure that they can save and reload and you know do all this stuff really quickly that's all very important but actually evolving the tool past version one is also extremely important and so that's something that i would like to reinforce here is that when we build these systems we shouldn't just let them sit once we're sort of happy with them we need to tend them and grow them and evolve them with ourselves sandboxes are a lot of fun this is just kind of a bonus point

i love the idea of doing one outside of an engine because it means that it's very lightweight it's very flexible and it gives you the ability to experiment very quickly most of all i want everyone here again think about how you can improve ai tools and what you can do along these lines to make the experience of learning and using your ai systems even better curvature is open source you can find it on github at this url i'll leave this up and this is also my email address and that's what i got thank you very [Applause] much you