Faerie Dust and Sparkles: Bringing Your AI Characters to Life

Thank you. So, the title was inspired remembering back to the Wizard of Oz books, not the first one, but I think the first one after that was the Marvelous Land of Oz. I mean, I had these characters that have stuck with me ever since. Uh this is Jack Pumpkinhead and Sawhorse. Um and the premise is that you build something out of everyday objects, right? Uh Jack is just a scarecrow and Sawhorse is really just a sawhorse. Um and then you sprinkle the magic dust of life over them and they come to life. Um it's interesting to note that this book was written in 1904, which is 4 years before the Model T Ford. So, you know, there were automobiles and electric lights and things like that, but they were brand new. All right, it was a time sort of like now when everything was changing. Um but these characters, although they would come to life, they were always a little bit awkward, right? There were things about them that were not quite right, not quite lifelike. So, Jack, for instance, can't change his expression

because it's carved into his pumpkin. So, if he, you know, something sad happens and he doesn't want to be smiling, he actually has to take his hand and put it over his mouth to block his expression. Right, Sawhorse moves in sort of awkward ways. Eventually, they make this character whose name's Gump, uh who's made out of two sofas and some palm fronds. Um and they use Gump to fly away and get away from the bad guy, um but they forget to make any way for Gump to be able to turn, so he can only go in straight lines. They have to kind of aim them the way they want to go and then they can fly off in that direction. And that seems like a really good analogy to me for the struggles that we have when we bring our characters to life and we really are trying not to have those awkward moments when they don't move quite right or they don't quite have the capability they need, um and so they don't feel alive. So, I'm going to talk about character AI today. Um this is not really going to be a technical talk. There might be a few tidbits. Um but, you know, just thinking about the types of characters I've done, right? Uh animals for for water shows in Zoo Tycoon 2: Marine Mania. Uh, this

boxy dog character was made out of nine boxes. We used them for a tech demo, um, for what eventually turned into World of Zoo. We were making our next generation animal AI at Blue Fang. Um, characters in Red Dead going about their everyday lives. Uh, Angry Granny is a mixed reality character. She's actually projected on the wall. Uh, these characters are the latest game I've been working on. It's an educational game. These are Mars rovers that have crash landed on Mars and they have to build a base for humans who are on the way. Um, so there's big differences between all these characters in terms of the types of technology, right? In Zoo Tycoon 2, we didn't even have blending, right? All of our animations had to link together with transition animation because we had no way to automate transmit transitions. And then you look at Red Dead, right? Huge budget, lots and lots of assets. You can throw in everything in the kitchen sink. Um, and then, you know, boxy dog, the other extreme. But the great thing about boxy dog, it was a tech demo. Our animators could turn out 15, 20 animations a day for boxy dog. So we could iterate really, really quickly. Um, so depending on what you've got, your technology's going to be different. But

the mindset, the philosophy, the craft that you bring to making your characters is the same. So that's the stuff I really want to talk about today. Um, and I'm really interested in non-combat behavior. So I, you know, some of this stuff will apply to combat behavior, too. But combat behavior is comparatively easy. I hate to even use the word easy in reference to game AI. But the non-combat behavior is much harder to get right. But that's really where I feel like the industry's pushing forward, getting more story, deeper experiences, more emotional experiences by having characters that do more than just shoot at you. So, I want to start out just with a little bit of uh, video that we used at Blue Fang. Oh, where am I? Here we go. So, this is just some reference video of a tiger. Can I make it big? And we just wanted to get a tiger laying still, not doing much. So, you notice you the tiger's constantly looking around. You can not quite sure what he's looking at. It It almost doesn't matter, right? But, it's clear that it's paying attention to the environment. Tongue

motion in the breathing, you get some twitches in the flanks here, this big characteristic oval-shaped yawn. Um there are some twitches. You not any tail movement at the moment, but the tail can also move. Here's the yawn again. I recommend not getting this close to the tiger's yawn, by the way. That's a bad idea. Um So, what did we just see there? The character's constantly moving, constantly attending, right? The characters are never completely still, they never freeze. Yeah, I think we know that by now, but the attending is really interesting. Looking around at the world around them is Dave drilled on this a little bit this morning, is really really important. Um no looping or repetitive animations anywhere in that tiger because it's not animated. Um we're not going to get away from looping and repetitive animations, but again, as Jillian pointed out, humans are really good pattern matchers. Um so, those looping and repetitive animations start to be spotted, so we need to find ways to disguise them. No straight lines, no crisp turns, right? When we're building AI, we use math, and math does really nice things

with straight lines and moving along the circumference of a circle and splines, and none of those things look really natural. Um so, anything you can do to fuzzify them helps make the character feel more realistic, except when the fuzzification makes them look less realistic, which I guess is part of the craft, right? And at the end of the day, the character's believable, organic, grounded in and reacted to the world around him. And this is the stuff we want to achieve. And if we're going to be photorealistic, you can push on graphics all day long, but if your behavior isn't realistic, then the it's never going to feel real, no matter how photorealistic you get it. So, the first point I want to bring out is the idea of an AI artist. You hear a lot of talk about AI designers, it's been sort of hip for the last 5 10 years to talk about AI designers and how close AI sits with design. Um and that's absolutely true, but I haven't heard a lot of talk about AI and art. And my my argument here isn't that we all need to go start making our own assets. That's exactly wrong, but we need to learn to think like artists. Most of us were

trained in technical fields with a technical skillset. Artists are trained with a whole different skillset and we need to be able to bring that skillset to our games if we want to make believable and compelling characters. Um so become an observer of life, right? Become a people watcher. Look at the world around you. Um watch different people in different situations and how they stand, interact, gesture. When we were working on Boxy Dog, our animator, we we all had dogs at home and we'd bring in video of our dogs playing. Um and the animator would take the video and he'd cut it apart into like almost second by second. And he'd say, "See this motion? This is a play bow. See how his head's down and his tail's up and his butt's wagging and he's super excited? That means he wants to play with you. And see this where he comes over and he sits down and he looks up at you like this? This is a solicit close. This means he wants something and you need to figure out what he wants. And see how he goes over to the door and he scratches at the door or he looks at you and he kind of whines? That's a solicit far. And that you know, and every motion that the animal did, the animator would pick apart and say what that motion means, right? We need to get

good at that because that's what we then turn around and build our behavior out of. Um so you can watch people in different situations. I don't have to pick, you know, dig into every one of these. Uh how people move down the hallway. I even, you know, simple stuff like that is a good thing. I've been noticing I moved up to the mountains as I've told a lot of people the last couple days. Um and I go skiing a lot, which is part of the reason I moved to the mountains. The way people hold doors for each other at ski mountains is very different than the way they hold doors in the office place because you've got a lot of little four-year-olds and six-year-olds trying to walk around in ski boots just about falling over. And people have a lot more patience for holding the door for that harried mother trying to get her kids dressed and out the door. You know, when office you hand the door to the person behind you and you blow through. Um so, you know, the way you hold the door says something about your mind state, the fact that you're on vacation, the fact you're more relaxed, where you are, the little smile on your face as you see the cute little 6-year-olds and they remind you of when your kid was 6, right? All of that kind of stuff. Another thing artists are very, very

good at is constructive criticism. You really need to really crave criticism, right? Get your ego out of the way, which is hard to do, right? It's easy to say, it's really hard to do. And get people to tell you what's wrong with your work. What's the worst thing about it? Um a lot of times I'll get stuck somebody to look at it and they'll want to be polite and they'll say, "Oh, wow, that was awesome. I really love this and that and this was really cool." And you know, that's great, thank you. Um but but what was wrong, right? What could I do better? What did you see that you wasn't quite right? Or what what didn't you like? People don't want to tell you the negative stuff, but that's the stuff you really need. Um lots of eyes on it, of course. Spouse and friends are great people to go to. Um players, of course. Although, players are biased. Any of these, you have to be able to filter what they say and get the real meaning out of it. Players especially. Players just want to be super powerful. Um so, you have to filter that a little harder. Getting the artists and animators to look at it is a really good place to start. Although, the artists and animators have some of the same problem as you do. Um which is that they're too close to it. They built it and so, like

you, they're going to see what they think they put there. Um and that may not be what other people will see. And if you show it to a lot of people, they may see different things, which is why you need to get a lot of eyes on it. QA, of course, but again, QA is too close to it and they're going to look at it way too much. Um so, more and more the more different eyes you can get, the more fresh sets of eyes you can get as it changes over time, the better. When I'm building AI, I really like to to try and build it bottom up. Um so, everybody's probably familiar with the idea top down and sort of game requirements come down, the game is going to be this, the story is going to be that, right? And bottom up is I'm building pieces of behavior up into larger systems, into story. Um as I was looking for to steal graphics on the web, I found this blog that had this graphics graphic which I stole, but the blog is actually a pretty good description of these ideas. Um so thinking about the tiger, right? Start building the little stuff, the breathing, the fidgets, yawning, stretching, tail motions, those twitches in the flanks, right? Get the attention right. Get the character looking around.

Um and then from that build performances, sitting performances, standing, laying down, walking, running. And these performances should have as many knobs on them as possible, right? I don't just want to get the character walking. I want to get the character walking when he's tired, when he's sad, when he's excited, right? These walks are going to be different. Um so if I can put knobs on them to change the performances, then I can build those up into my larger behaviors, playing, eating, sleeping, fighting, working, whatever your game has in it. With those knobs tweaking the behavior to make it appropriate to the context. Uh I mentioned the animation loop, breaking the animation loop. So this is Angry Granny. This is the a character that I worked on at um Lockheed Martin. Not the level of graphics that we're used to. This was another very fast demo. I had about 3 months to build this character from scratch. Um but one of the things that we wanted to do was was work on breaking the animation loop, so the character would sort of stay alive for 10, 15 minutes, even though we had a relatively small amount of assets.

One of the great tricks for that is to take your looping animations, make sure they're different lengths, and then use different animations on different parts of the body. So the lower body has one set of sort of pacing animations. She's got to stay she's projected on the wall, so she can't move very far. Um the upper body has gestures that, you know, and other animations, but because they're asynchronous, when I see the same upper body animation happening, the lower body's doing something different. So it looks different. It looks like I'm doing it again, not like I'm playing the same animation a second time. Um and you can split out breathing, attention, expression on the tiger. You can do the tail. Uh Um, also, anything you can split out and do asynchronously is going to help to break out break up your looping animations. The other things to think about, where's the next part? There we go. Um, is avoiding repetition. So, aside from looping animations, gestures for instance, right? She has a lot of these big, you know, this gesture, right? And this gesture, this gesture, right? And when you see those gestures again, you want it to look like she's doing the

same thing. Ideally, you don't see them more than once. If you do see them more than once, um, you want it to look like she's doing the same gesture a second time, not like she's playing the same animation more than once. Um, so, if you're Red Dead Redemption, right? And you can afford hundreds of millions of dollars worth of assets or some, I don't know what the number is, some insane number of animations, right? It's not so bad cuz you can just avoid playing the same animation again. Um, but if you haven't got Red Dead's budget, then the more you can do to vary those animations, um, maybe change the speed you play it at. So, a great animation to think about is that sort of oval-shaped yawning animation that the tiger had. Um, change the speed. So, you you know, he he yawns faster, he yawns slower, I don't know if it's a a he or she. Um, or even vary the speed over the course of the animation, right? So, it sort of starts fast and then it stretches out to the side. Or maybe it starts slow, but then it accelerates as it goes, right? Have parameters that tell you how much you can tweak those those values and then tweak them at random at runtime to

get different animations out of that one asset. Um, especially good if you're working on a Wii game, which is where the tiger was headed, World of Zoo, where you have very limited numbers of animations that you can afford in memory. Um, but changing the speed of playback isn't actually that big a deal. Um, changing blend weights, getting into more experimental, like I haven't actually tried it, but it would be interesting stuff, right? You could try actually tweaking bone positions to make the the yawn taller, make the yawn wider, um, add in some Perlin noise to get a little bit of wiggle on top of your animations. Um, if you don't know what perlin noise is, go look it up. Um, and beyond just animation, um, voiceover is another great place for this, right? You all of your speech, all of your dialogue. So, with Granny, uh, we had a preve- professional actress who recorded all of our dialogue, um, which was in Pashto, which is the language that they speak in Afghanistan. Um, more about that tomorrow. And the actress recorded three or four takes of every line. So, rather than just keeping my favorite take of each line, I

kept all four, right? And then when she says says the same line again, um, I would just play a different take. So, it sounds like she says the line again. Um, some things like when they question her, um, she might say yes or no repeatedly. So, I had 20 takes of yes and 20 takes of no. And that way every time she says yes, it's a little bit different. And I even sort of built it so the intensity builds up over time a little bit. So, she'd start to sound more exasperated. Um, would have been nicer to have a model for that, but there there wasn't time. Attention, as I said, is really really important. Um, the simplest way is to build some head motion into your animation, right? Which is what I think of as neutral attention, because it's just the head motion that's built into the animation itself. That's expensive to do. And it also means that it's unless you know exactly where the character is going to be, you can't have the attention be directed, right? It's going to be wandering, which is great when you're looking at a room full of people, for instance, giving a talk, cuz nobody knows what I'm looking

at. Um, but if the player starts Now, I'm looking at Dave. If the player starts trying to figure out what they're looking at, right? It's going to fall flat. Um, again, you could just pick random head look targets and get that same sort of wandering gaze like the tiger is doing, if you're not too worried about what to look at. What we did in Red Dead, um, was we had characters pick random entities to look at. And there was a little bit of utility going on to weight the entities. So, if I look at somebody over here for a couple of seconds, then when I pick somebody else, I don't want to pick the person next to them, right? Cuz that's not enough head motion to read to the player. I want to overact. So, humans might really do that, but I'm going to look at somebody over here next. And then I'm going to look at somebody back there. So, how far away my targets are is one of the weights that goes into it. Um, as the player becomes more infamous, um, you they get a little bit more weight, but you want to be careful of that. Um, in Bully, for instance, which was one of Rockstar's earlier games before Red Dead, towards the end of the game as the player became really infamous, everybody started staring at

them. And so, I, you know, I'd be riding my bike around town, right? On my way to paint some graffiti on a wall or something. And everybody's staring at me as I go by, like, "What? What did I do? I didn't do anything. I'm just a good kid going to school here. WHY ARE YOU STARING AT ME?" LOOK OVER THERE. UM, SO, that gets a little creepy. Um, another great example was EverQuest II was one of the first games they got attention in. And all of my friends who'd played the original EverQuest and then played EQ II, commented on how much it sold the characters that they looked at you. But at the same time, if you're playing a great big troll, and you go up to a little gnome vendor, and you open up the vendor screen, then they're going to look up at you like this, right? And then if you go and get lunch, uh, and come back half an hour later, and the vendor screen is still open, they will still be standing here like this staring at you with a crick in their neck. Um, so, one of the important things about attention is thinking about duration. How long should you look at something? If you're just, like the tiger, looking around, you notice the tiger's gaze shifts pretty quickly every couple of

seconds. Um, so, you can play with tweaking those parameters. I think we ended up in Red Dead, I don't remember, but I want to say it was like 15 to 25 seconds. Pick a random number somewhere in there, and that was your duration to keep the attention moving. You probably want to move the attention a little bit more than a real human would. Because you want to show the attention moving. The attention being fixed doesn't sell the fact that they're paying attention right that they're grounded in the world around them that they're aware of the world around them and paying attention to it. All it does is you know it looks like part of the animation. The next step now that you've gotten an attention model is to build reactions off of that. So players should not react in point two seconds or point four seconds until after they see something. So if there's an event that goes on over there, right? I might be oblivious. I might be looking over here. And I might be completely unaware of that event and that's fine, right? Everybody shouldn't whip their head around and stare when an event happens. That's a little bit creepy and unrealistic, too. But I might

notice these guys all looking over there and then I might go oh oh wow, look at that, right? Um one of Disney's 12 principles of animation and I don't know the other 11 um is anticipation, which is really powerful and something that we as AI programmers can take advantage of. So it started with the notion that you know before you jump you have to bend your knees, right? Before you swing a golf club you have to have a back swing. Um but in cartoons in general because this idea is exploded before any big reaction any big moment there's a little bit of anticipation that tells you something's about to happen. Animators can probably describe this way better than I can. But get that notion into your AI. If everybody people remember Damian Isla's talk from a bunch of years ago now when he had the little triangle trying to find the player and it was searching around and it would look here and when it didn't see the player it would freeze for a moment like it was surprised. That moment of anticipation like oh it can't find the player and then it would go look somewhere else. And that

moment completely sold the character even though the character was a triangle. Um so get that moment of anticipation eyes widening whatever and then the response to the event that the character just saw. I'm actually doing really well on time this time. Last slide. Um so and this I don't have a lot here other than just the thought. Puppies don't use plat path planners. They weave all over the place, right? And really humans don't use perfect straight lines and crisp turns either unless they're in the military and they're marching and working at it. Um so whatever you can do to fuzzy that path, um we've used flow fields to try and get fuzzy paths with the with the puppies. Flow fields have all sorts of other problems. I don't know that I recommend that solution, but it did get us a lot of wander. Um I really like the way parkour is looking. One of the great things about parkour is the way that it blends

different types of motion together. So they can go from running to jumping to rolling to crawling to swimming to climbing, you know, sort of whatever smooth motion blending, but parkour is typically very linear, very sort of aggressive, right? I want to get all of those blended animation types as you move naturally through the world, um but with much more free motion. So that's all I've got. I've got way more time for questions than I expected. Dave says 4 minutes. Remember, please use the microphones up here if you do have a question so that it ends up on the tape. Notice that we still say tape. How awkward is that? I'm not being picked up by the tape, I'm sure. Instead of having a bunch of animations for the tiger, could we use AI to build the inside of the tiger and then those

animations might come out of that? So procedural animation is really hard. Uh if you've got a system that'll do it, that's awesome, right? And it'll buy you a lot, but um I haven't seen a lot of procedural animation systems that are completely there. There's stuff like natural motion, right? Which which Red Dead used, which buys you a lot, but is really expensive. So, it depends on what you're building. It's part of the reason I wanted to stay away from technical, right? If you've got those sorts of capabilities, absolutely use them to make your characters. Um if you haven't got them, then there are traditional techniques that work well, too. All right. So, this may be a little out of scope for this, but um one of the things that you didn't talk about was conversations and dialogue. Is there anything that you can say about extending some of these principles to how AI handles conversations? So, I'll rant a little bit about what went wrong with conversations in Red Dead.

Um I've had thoughts about conversation, and really what I I think I'd like to try, which I didn't get to do on Red Dead, um is to tag up the So, you're going to have a whole library of lines, of course, and you're going to have, you know, I play a statement, you respond to my statement, I play another statement, you respond to it. Some sort of flow like that. But to be able to tag those up so that you can get those flows to feel more natural, I think would do a good job of taking us sort of to the next step of the random conversations we see in games like Red Dead or Skyrim or, you know, any of lots of games. Uh Karen, based on your other experience, um could you offer some advice on how to integrate AI into uh game education? For instance, uh in a program that would involve a minor in game development. Uh that's a longer conversation, I

think. Um I mean, there are good books on game AI. There are You certainly game AI class is a great thing to have. Um I made the argument earlier um today that game AI is something we should be teaching because it teaches So, this is aside from this talk. Um but game AI teaches decision-making, right? How to get computers to do things that look right, right? Not necessarily that aren't intelligent, but that look right, that look intelligent. And that's a set of techniques that are wide more widely applicable. Um so, you know, maybe we we should be pushing harder to get the rest of the world to be attending this summit and listening to us talk. I don't know. Anybody else? Um if if I could real quick. One of the things that One of the things that you had mentioned is talking about um attention shifting. Of don't just continuously stare at one target. And that was For those of you who were with us this morning um for me and Mike, one of the things we did is we had a a maximum run time on a behavior.

We actually used it with the look at. Where within about 5 seconds, no, dude, move on. Do something else. We would also say, don't come back to looking at the same person for, you know, 5 10 seconds. And so, it forced the character, I'll look at you for a little bit. I'll shift around. I'll look at different things. But enforcing that, so rather than you're very important, so I'm going to stare at you as you ride your bike down the street. You know, no, you mix it up a little bit. And And it's amazing how just doing that gives you that thing, he's aware of his world. Right. Yeah, if you want the character to feel like it's in the You think about animation, right? And it needs to have weight to it, feels physically motivated. If you want your character to feel like it's really in the world, it needs to have it needs to be aware of the world around it. And the way you show that is you show the character looking at the world around it. Or you show the attention shifting and and show them responding to things they see. It's It's once you get the animation

problem of getting the head to look right, which is a little tricky, um but but is an animation problem. Then getting it the character looking around isn't actually that hard. It's, you know, a week or two of work, I think, to get a decent at least a decent baseline. Everybody, let's thank Kevin. Good stuff.