Session: AI Characters in Video Games, Dayan Paul, Masquerade Media Inc

[Music] thank you [Music] our last speaker is Diane po talking about AI characters in games so we're gonna go from voice to characters give a more welcome to to Diane so I'm Dan Paul of mass grade media here and I'm going to talk about generating AI characters for games so um AI characters is this something that's kind of come up recently right you know how do you generate characters for a game um and so I'll talk about the different aspects of generating a character um and so so I'm not going to take too

long of a trip down memory lane but um you know classically um AI is not really new to video games right what's new is really more um machine learning um what we've been using AI for in video games is is not really necessarily character generation but you know it's been in production aspects uh player analytics um you know non-player characters things like that we've been using for a while um but in a lot of ways they're perceived so you know we had perceived character intelligence even though it's you know it's sort of an auto response or a program response so that perception of the character um being being uh sort of AI is there um but now we have some new tools so now we have you know machine learning which is not so much been um really a big factor in generating characters but in uh we found it useful in other aspects of game production right um so you see here a picture of uh you know like uh if anybody's watched the

next generation of Star Trek the the the computer needed to generate a character that could compete that would allow that would be challenging for um data who's a computer right to to solve to to kind of compete with him because you'd walk into the Holodeck right he'd play this cerlock home thing he'd look at all the facts analyze them and then just you know solve the crime so um but in a real situation right it would be much harder to come up with a challenging player for a real person because this is a computer versus a computer right but how can the computer generate a challenge for a real player right that's far more complicated than a computer generating a problem for another computer unless you're just talking about puzzles or encryptions right um so image creation tools so so this is not really um you know this is sort of the visual aspect of creating a character right you've all done this in

a game right you've all generated a character where you can set up the parameters you can choose what they look like you can choose their clothes and things like that um so how do you auto-generate that um and that's not so much maybe the Core Essence of building a character right The Core Essence is going to be the personality but a lot of times we craft our story around how our characters look gonna look in a game so there's a lot of tools now that will allow you to generate right imagery right you've seen lots of these right and so and so some of the challenges for doing it in a game like if you're going to create an avatar if you're going to create a non-player character um you you could try to use those and so you know this is always that shortcut like well I don't have any artistic skills so let me just go make something right but sometimes that could be a lot that's a lot longer hurdle like to to get the computer to generate what you want is a lot harder than just say hey I hired this artist I like his style he's just going to do this so

um so what are the strengths of it though um so you can create a lot of characters without drawing right you can create many different characters right without drawing and you can create them in many different styles so what are the weaknesses with these so one of the weaknesses is that you know if you try to create a character set right they'll be very inconsistent and look right so you might have one character in this style and then the other and that takes some like you know this goes to the third Point here of too many levers right you're like pulling all these levers and trying to generate these things and um and so you know so so what would be easier than that right um so when there's too many levers sometimes it's just go go back to the easiest thing it's like pick up a pencil and sketch so in my middle point here that goes to that so when you um when you see people using these you've probably seen tons of YouTube videos of people that generate these things and some are generating from the games but the ones that are most successful the ones that are like okay I like this look

and then they're sketching something and then it takes that sketch and it generates another one another one so in a way there is still there's still some artistic skill needed and so even at the sketch level so even even if somebody's a non artist might come out with a sketch and then the artist comes with a more refined sketch um that that sort of in between step is a lot more important so even though AI can generate really nice images um it's not it's not a mind reader right and then you know it's all based on sampling you'll see more and more and more all this stuff when you generate a character or a storyline or a um or an image it's all based on sampling right you even saw from the voices right the better the better the voices are the more they're sampled the less automated they are so the more sampling them more data the better the output is um and so these aren't necessarily all weaknesses but they're things that get improved on by sampling so you know that that the more data that AI has the

better the results are but you know my my thoughts on that when I say just pick up a pencil it doesn't mean that you don't use these tools at all when I started out as a an artist years ago I was kind kind of frustrated that you know when I did 3D I'm like oh well that's no fun it renders all the lighting and already I want to control the lighting and all this stuff but then I realized you know I'm a painter but I can't paint 30 images per second right that's just not possible right I can't do that many things so you have to use the tools as an assist right you want to use these tools and assist and that's what a lot of artists are doing right they don't necessarily say like oh I'm not going to draw anymore they use this as an assistant I'm going to do I have a refined version of this character I'm going to sketch it now I do the all the other views let it automate it right and so that's that's kind of one of the things that generating characters can do is you know you're using the computer to assist somehow and machine learning will help with that in the long run I'm not

saying there won't be tools that aren't useful and then you know there's going to be a lot of misuse of it but for us the best use as artists or creators is to be able to use these tools to assist us rather than replace us right so um the next slide here and so for Content creation so you know the core of the character is it's going to be in the writing right so um you know there's a few tools out there to keep your eyes out on they're not necessarily generating games but you've probably seen like text-to-video avatars right um and again this is from sampling right you'll have a full character narrating um you take this text you type it in and then that character starts talking right and so it's automated automatically generating that character uh for you to narrow it um not not so much useful for games but maybe for marketing games right maybe for uh Vlogs or different things if you're marketing your game story Generation Um you know and so when we start generating story trees and things like that in our game they really you know it

really it's real a balance between the writers and the engineers and the artists but you are seeing some story generating tools that will allow you to kind of simulate right they already have 3D characters in them you put up a storyline um you pick which characters say which things and then they generate not necessarily a game but a fun little exercise if you've seen people play those um and now that now of course now you have chat GTP and some other plugins that will allow you to use a sort of a chat environment in a game so uh sorry GTP chat GTP yeah all right so uh strikes uh so let's talk about like the the easiest way where do you start right the easiest way to is to start in that chatter environment right so you saw like um with the plugins and things like that right you can you can generate a text game and you can generate Auto responses um and you can relate those to the game

tree um but what you want to do is you know when you whenever you work in these in these environments you want to limit the data set you don't want something that looks at all the different data in the internet right you wanna you want this character to live in a certain world you want that data set to be extremely small um so one of the advantages to doing a text game right if you have a you know a text game where you're texting and then you have an interaction from a character that's also a response to text you're taking away a lot of um different complexities there which I'll get into in a minute but um so what what can you do with the text data you can generate Auto responses from the character and that could be a pre-programmed list um it could be multiple choice or you can make something more complicated but you want to start with something like that so you want to have emotions in the character right you want to be able to express emotions in the character and you can do that more visually but you can also do

it in the style and in the response of the way the character talks so that's why I said that character has to have a very small data set and live in a specific World in order for it to respond to you and and fit within that world of that Realm uh and so one of the other advantages to doing a text um a text only game is that it's easier to edit right it's easier to edit um things in a text format than it is when you start getting to voice and other interactions it's also mobile friendly right so mobile friendly doesn't take a lot of resources to do stuff when you have a character that responds with text uh so what's the what are the weaknesses um for it well it's harder for a player to interact right if I could just talk to the player uh the non-player character and they're going to respond to me right um it's a lot easier for them to do that than it is for me to type something in right um so when it comes to consoles PC games

right the expectations are higher right we want a higher expectation we want more Innovation we don't want the same experience we have on our phone um uh primitive gameplay right is a sort of nostalgic too um it's okay on the phone when you get into uh consoles and things like that it does seem a little primitive even though a lot of games still do use it in some capacity um and um and the lack of the um the actors influence so you saw with you know with the last presenter right we had some some voice interaction so when you have a voice actor right they give all these inflections and they bring all these other things to the character you can't necessarily do in text so those are some of the weaknesses for doing the text game um so voice interactions so voice interaction you just saw a great presentation on that so um voice interactions are great because uh the player can interact directly with the dialogue and get a response in voice um it's a more immersive experience

right if I can if I can talk to the character and it can talk back to me with my voice that's far more immersive than just kind of texting and texting back um also console and PC friendly right even though it does take a lot more resources it still meets that expectation of what you're going to have in a console game um and so what are the weaknesses um so you saw it like you know in the last presentation right some how quickly that evolved um so when I say um uh monotony or a monotonous voice right I'm not necessarily talking about the voice itself you can cure a lot of that with sampling but it still goes back to you know how the AI handles a response that is the content and then the content can sound very monotonous even though the voice might be fairly good um a voice from players is um is not always understood so how many people have ever screamed at their apple carplay and trying to get directions or

or cursed at Siri or anything like that right pretty much everybody right unless unless maybe it's just me maybe it just doesn't understand my voice right so but I have that experience all the time um uh player can um so you know voice data also takes a lot more resources right so it can be slower um you know and that this depends on the device you're using in the data set you're handling right and so that goes back to either limiting the data or you know using a device that can handle more data processing um so what's needed you know what's really needed in the future what are we going to need to create an AI so we already can create kind of visuals and that's you know even though there's shortcomings there it's probably going to come along pretty fast um with 3D you know I talk about 2D generation I didn't really talk much about 3D generation but you know with 3D characters um you know for for the most part which all the create characters I've been creating it comes down to like okay

we've got more you know the more um you know the more presets we have the more examples we have the better the 3D models got and the better the tools got and the better you know the more sampling we did of real world for texturings and things like that that improved a lot um in how we can build characters but what you could do what we really need is more of an assist right so if I'm building a character and I need to put a bunch of bones in it and animate it and things like that right that's still a very much a hand process there's some scripts yeah for Mayan things like that will automatically again some of them might have used fuse or other products like that but they always require some cleanup or um or the the handles aren't there right you might be doing a quadruped everything's based on a human ik right so there's all those things that could probably with time if somebody focused on it they could cure all that problem of rigging technical you know 3D 3 animation has come a long way but it hasn't really gotten any easier it's

only gotten there's just more tools right now you got to use you know like three or four or five different tools rather than um one so um so if automation can handle that it's going to make character generation a lot easier not just on a 2d side but on the 3D side so we want to be able to train a character to have a unique voice right and that doesn't mean just invoice in you know in audio but in the way it writes right so um does it speak with an accent does it does it speak in vernacular does it have a certain style um in which it interacts with you right so the more the more we can sample that the more human it becomes right so it's not like one size fits all there's many different voices there's many different ways people speak and that not only relates to like like I said the the actual audio part but the inflections that we're going to have in those characters um and so what you also need is you know for tools for game developers you need to be able to change the character to react emotionally right so um this again comes from sampling you

can't really you can't really sample um emotions but you can sample emotional reactions right so how do you want the character to act you want to be able to as a writer right you want to be able to train the AI in a few tests like okay if I type this and I'm I'm giving an inflection that's sad what is that response to that sadness or if I'm happy how does it respond to different things so in that training of the AI there needs to be again a limited data set in The Limited world and a way to do that um and again you know the behavior of that character needs to live within that world right so how does that what is the behavior of that character is it a medieval warrior world is it a spaceship you know what are the parameters um and the more you can limit it to that world the better now you see I have a picture of uh Comer back there from the the Turing um so it doesn't need to pass a Turing test right it just it just needs to be believable right you need to believe that character lives in that world they

don't have to be a real character but they have to be believable in in other words I need to believe they're a real character in that world um so avatars and NPCs uh you know so um you know they need to be diverse um so when I say that you know no haters it's room for every kind of uh player there and every kind of um uh every type of ethnicity so you need large data set of characters um to choose from as a player um and this goes for really just games in general right these are the same kind of things we want today right we want language options we want personalities we want a diverse set of characters want customizable configurations what do they wear what do they look like all those different things as tools we want all those things available when we generate AI characters um and that's that's pretty much all I have is first the ingredients for making

a character and um in AI so uh any questions yeah I saw a demo where someone took essentially chat gbt and 11's Labs voices and they put it in like a like a Skyrim character where they were talking using their voice to a character in the game and then it was doing presumably some type of API call to chat GPT and then running it through 11's laps do you see that in like kind of as a use case in RPG games where you're able to sort of give these care NPCs like parameters on how much they can role play as these characters from the games and then you can just talk to it get Quest information from it do you see it heading that way right and I think you know that all comes from making the tools to where you can train that character right you need to be able to train the character and one of the things that you know that's hard about um about machine learning machine learning is great at looking at data and coming up with a solution but what it's

what it's not great at is being taught like being taught by a person so that that part of teaching AI um is going to be key component into like how do you want that character and you just want to run tons of tests as a as a QA right how do you QA all that stuff and come back and make sure that you craft that character um to specifics and that the tools actually do what you want them to do it all comes back to like the control right as the designers right we want to be able to control these things we can make all the tools in the world but if we can't control them um they're really pretty useless as far as game design right any other questions one more and then thank you um in world aad does the same thing you just described um so I'm kind of curious about your response to that because you talk about the parameters in control of the character creation isn't one of the fundamental aspects of generative AI that many systems are black boxed and there is not full control over the

system right well that that could be problematic for it's it's really from the designer's percept perspective right so if you give if you the more control you give the AI and less control you give the designer probably the less you know the less desirable the outcomes are going to be right so um you know it's it's sort of like um even if you let's say you gave let's say you gave your engineering team full control over the design and then you don't have any designers right so you know you have a similar problem not that the engineers are are you know like uh you know robots but but they're trained in different things right and so that and that's the whole the whole generative AI is is more from that world it's more from a world of engineering and machine learning than it is from the creative side so thank you for that perspective um one short follow-up based on that and the QA aspect of it will we ever be able to have a system where you can talk to an AI in a game and 100 know it's not going to say something offensive or racist or

horrible yeah I mean I think those those flags are sort of easier to set up it's it's really more on the admin side right like how many flags do you set up to to find those things and with time those would be easier and easier to identify right and uh you know and then you know a number of protections and security for games it's it's really up to the player too like how many how many things they want to open themselves up to to right you gotta you gotta have the player choice the player might want more offensive things right or or less offensive things so so you have to be careful about you know like how do you balance that as a as an admin right thank you very much Diane [Applause] [Music]

[Applause] foreign