Voice AI For Game Developers: Optimizing the Narrative Process

[Music] hello everyone and welcome to gdc 2021 this talk is voice ai for game developers and it's all about optimizing the narrative process my name is trace i'm co-founder and ceo at replica studios for those of you who haven't heard about replica yet we're an ai company we make ai voices for games and movies and so i'm really excited to be here today and talk to you all and tell you a little bit about this new technology that's coming your way and can massively complement all of the wonderful work you guys are doing so to start with some of you probably wondering what is an ai voice what is voice ai um so we've got a couple of demos for you right up front and then over the course of this presentation we'll be diving deeper and and giving you more practical examples but check it out hello stranger needing a new set of armor area i mainly specialize in plated

armor but on the odd occasion i've worked with leather as well that'll be two gold pieces uh and here's another one how dare you if you speak to me in that filthy tongue again i will cut yours out now leave me while i still allow it no please don't kill me i can help you regain the throne but only if you let me live okay so what you heard there are two of replicas ai voices and so all of that spoken dialogue was synthesized by a neural net by a computer uh which is pretty cool and so hopefully by the end of this presentation you'll go away with a bunch of ideas your head will be buzzing about cool ways you can use ai voices in your game or movie uh within the context of game narrative so to start out i thought we'd cover a brief overview of all the topics we're about to cover in this presentation so

we'll start with a quick who am i a bit of background about me uh then we'll jump right into game narrative and storytelling uh why is it important why is it great we'll then look at some of the challenges with regards to game narrative and storytelling and how voice ai could help overcome some of these challenges then we'll look at a bunch of really cool voice demos uh there will also be visual parts to this voice demo so you guys are going to love it definitely stick around for this um we'll then break down the process so what did you actually witness and hear and how does that fit into the the game narrative theme uh and finally key takeaways because we're covering so much stuff in 30 minutes uh we'll just summarize before the end of this presentation so in toto a lot of stuff in 30 minutes um thankfully this is a pre-recorded session and we'll be around to answer any questions you may have uh and you can also go back and watch parts again that maybe was too quick the first time

you saw it so with that let's get started um my name is trace uh i started my career as an aerospace engineer uh so yes i'm a bit of a nerd um i've been doing startups hardware and software for about 10 years about four years ago i got into deep learning uh specifically voice ai uh about the same time i met my co-founders we started replica uh the rest is history uh but really at the time we four years ago we saw the enormous potential that voice ai as a technology has for creative people and we thought we would set out and build photoshop for voice and outside of work i do have some hobbies i like playing drums i like playing all sorts of games but specifically i like racing games because i also like to race in real life so while i'm an expert on voice ai i'm not actually an expert on game narrative which is also

part of this talk today so the most popular google search result says narrative and game consists of plot sounds music atmosphere dialogues player choices and of course gameplay and it creates the overall impression of the game and allows the player to feel a part of the story uh so that sort of makes sense to me uh i've also spent some time asking experts in the field and while they more or less agree with the most popular google search result um there are slight variations to this some feel more uh that it has more to do with the actual writing aspect uh some prefer uh the ability for player choices but overall i think we can all agree that game narrative is like good storytelling with the addition of player driven choices so when i think back to lord of the rings when it was still a book i had to imagine what all of these characters looked like what they sounded like and tolkien was amazing because he could do this he could paint pictures and movies with words um but when the movies

came out obviously uh it was fantastic and so we think that game designers and narrative experts should really be using all of the tools and technologies that are available today to really help you tell your story in the most compelling way possible and if we think about how storytelling has evolved the visual side of storytelling has really come a long way from our cave painting ancestors where we're now using we've evolved through books and we now have video games thanks mostly to you know advances in 3d graphics a company like nvidia is doing a lot in the space epic games is doing a lot with their unreal engine uh unity roblox these are the ways in which stories are being told today unfortunately though here's the problem voice acting is still too hard and this is despite how powerful the human voice is and how crucial it is to storytelling um from the previous slide back in the days sitting around campfires we would

still tell stories um and so it's it's a shame that the voice acting component is often an afterthought and left to the final stages of any creative production for those of you who have worked with voice actors you know that they're pretty expensive and the process can also be quite time consuming and so what happens is most of you are using this guy get to the chopper and that's a shame because generic tts will break immersion it is not going to give you the superpowers that you need to really sell your story and so we think this is a challenge but we think voice ai is the solution to this because voice ai does give you super powers because it can sound like this one step closer demon filth and your general dies right it's pretty cool um so voice ai is much more than just plain old text-to-speech we're about to be overrun fall back to the fortress now

right it has the ability to inflict emotion tone and stylistic control so even though you can still use text to go to speech using voice ai it is fundamentally different to the generic text-to-speech that you've probably been working with so far in this presentation we're not going to go into the technical details of how this works uh we thought the best way to prove to you what it does is through a practical example so let's play a little game um this game is we're gonna pitch you an idea for a video game very meta and you the audience are going to help us fund it um this game is called the quest it's got two lead characters male and female like you can see on the screen and um it's loosely inspired by the witcher it's inspired by the lord of the rings um so this game this adventure is set in magical mystical fantasy land but also our two main characters they're warriors and they're hard fighting

people so we thought uh the next step now that we have the general idea for what the quest is uh we would progress to a rough visual storyboard because that is the best way to try and pitch your idea so we've put together some scenes i'll be taking you through it we've also collected some sound effects and foley to really add to the atmosphere so in this scene we've got our two main characters they're on horseback they're riding through the woods and they're having a conversation things are fine no problem so far then things are changing they dismount from their steeds draw their swords because the enemy is advancing and then finally there's a fight scene and in the end they ride away on horseback

okay so so far so good we've only spent a couple of minutes mocking up a visual storyboard uh to convey this idea we've collected some sound effects and fully and i guess the next step is let's put some voices into this right remind me again what the plan is kill there's too many we must make for the village all right so um well to take nothing away from generic text to speech it's really not compelling um there's no acting going on and so this breaks immersion and it's not a very convincing pitch let's contrast this with using replica voices right that's the whole point of this little game that we're doing so this is how we're going to do it

we jump into replica studios which is a desktop app that works on your windows or your mac and as you can see we have a growing library of ai voices that you can choose from so in this menu you can browse through voices you can audition them listen to samples you can also drill down into individual voice profiles for example let's look at colt over here so colt's voice is inspired by your stereotypical sort of cowboy kind of guy as you can see there he's got a a few different moods or styles so he can be charming uh he can be intimidating and he also has a battle style so he can shout in in battle and uh what we're gonna try and do is find a good fit for our two lead characters uh so this is what cult sounds like there's too many we must make for the village cool so he sounds like what you'd expect him to sound like and that's not a good fit for the lord of the rings which are

inspired theme that we're going for but it's a good it's a cool voice anyway um similar to colt we have vinnie here and vinnie is inspired by your stereotypical italian mob boss guy uh so this is what he sounds like there's too many we must make for the village all right so just like colt and vinnie uh surprise surprise we also have a voice that's perfect for this for the quest uh and uh we're calling him decard this is what deckard sounds like there's too many we must make for the village all right so hopefully you've understood by now replica lets you audition and search for a number of voices so similar to finding decade we've gone ahead and auditioned our female lead character and now that we have our two characters let's go to this interface which is the main replica interface where we can start adding in the lines that our characters are going to be saying in these scenes and if you look there you can also see there is a column for the

default style that we're applying so you can see that initially things are quite light-hearted and casual so they're having a casual conversation then things turn serious uh the enemy is upon them and finally there's like battle scenes so there's battle shouts and all that stuff going on so this interface is really cool because it lets you experiment very quickly you just type in text or we're also building a feature where you can import all of your dialogue and lines from a csv and pre-populate all of these things in replica so you can go from text to voice in an instant um so this is a great way for you to get an overview of everything that's happening in your project or your scene but you can also go one level deeper so for a given line you can drill down and you can fine tune the performance and you can do this in a number of ways you can highlight bits of text and then apply some transformations using sliders and dials and knobs that we provide some of these screenshots that i'm showing

you in the presentation are in beta so you can download replica studios right now but you may not see exactly what we've got here because we're still testing this with a smaller subset of users but this is coming very soon to everyone on the product so in this scene uh we're gonna listen to a couple of variations of the same line uh one with emphasis on the word anything and one without have a listen kill anything that comes our way kill anything that comes our way so that was just to illustrate the ability of fine-tuning that you could do uh with this interface all right so now that we've spent some time populating all of our lines we've even spent the time drilling down individually for lines that we wanted to fine-tune and tweak and so now we're ready to go back to our rough storyboard and listen to what this sounds like with replica voices here we go do i hear a thank you for saving my life

coming that's not what i'm saying whatever you're trying to say that's what it sounded like to me pretty good next scene speaking of smells do you smell that it smells like ghouls the girl they must have her scent okay and the last one kill anything that comes our way freya behind you so that's not bad right it's not bad especially when you compare it to remind me again what the plan is kill anything that comes our way okay so hopefully that illustrates the point in a nutshell um and uh hopefully at this point you the audience actually like what you've been watching um well we kind of made the decision for you uh we think you really loved it and we think you want to green light this

you want to fund this project so we can go out and actually make it uh so let's do that we we've gone now to unreal engine we're going to the marketplace where we can go and find assets so that we can put them into a scene add some cameras add some lights and really start blocking out some shots for the visual aspect of storytelling and that's when we ran into a really interesting challenge because it turns out if you've got high fidelity characters on the screen and they're speaking their lip movements really need to match what's being said otherwise it breaks immersion check this out then all of a sudden you somehow elbow deep with your obsidian blade in its jugular do i hear a thank you for saving my life coming that's not what i'm saying right so even even though this looks really good it's got high fidelity humans way better compared to the rough visual storyboard

that we had a couple of slides back it's actually worse because it breaks immersion because all of your focus is on the lips and the fact that they're not saying or they don't seem to be saying what's actually being heard so uh we decided that this was a big enough problem that we wanted to fix it and we started working on trying to fix this earlier in the year and so um i'm going to show you a quick update on the progress we've made but before then i'll just say quickly we've got these plugins coming out that can do lip sync that work with unreal engine and also works with re-illusion iclone and we're going to be working to add integrations and support with unity and roblox so that you have not only do you have voice acting with ai voices you also have compelling lip sync as an automated solution so uh this is our latest progress then all of a sudden you somehow our elbow deep with your obsidian blade in its jugular do i hear a thank you for

saving my life coming that's not what i'm saying okay so that was a sneak peek don't worry we'll show you the full video in a moment but um it's a big improvement from the other video that i showed a moment ago so this is not bad given we've only been working on it for a few months and so we're fairly confident that with continued effort we'll get to something that looks really really good um not long from now and so this is a demonstration of how we're solving lip sync for unreal engine meta humans so i'll play this video for you in a moment but let me just explain what you're going to see we've got our developer here and he's using an iphone camera which is running an app that scans his face and based on the changes in his face and lip movements that drives the metahuman visually in real time so that's really cool thank you unreal engine but the cool part here is at the same time that he's performing the line and doing the face acting

replica can take the audio input from your own voice and use that to direct the performance of the ai voice which can then be applied onto the metahuman and so that's what you're about to see have a look do i hear a thank you for saving my life coming thank you for saving my life coming do i hear a thank you for saving my life coming so i should explain that the visual side of things where you're driving the metahuman through the face capture that is in real time thank you unreal engine uh however the part where your voice is being used to direct the performance of the ai voice that isn't quite real time just yet it's going to get there currently depending on the length of the line that you're trying to direct it could take anywhere from 40 seconds to a little bit more than a minute per line before moving on i thought i would play that demo for you again so you can

just see that again do i hear a thank you for saving my life coming thank you for saving my life coming do i hear a thank you for saving my life coming okay so that feature that lets you direct the performance with your own voice this is coming soon uh it's not part of the app yet um but if you think back to the slides that i showed you of replicas product and ui most of the controls for fine tuning now involve selecting bits of text and then applying transformations by changing a value on a slider or a dial to get the desired output this new feature is another way that you could get the desired output where you don't have to use the the sliders and dials you could just use your own voice to perform the line and replica can transfer the energy cadence and tone from your own performance onto any of our ai voices to further demonstrate how this works so you start out with a default

ai voice output that sounds like this there's no way you can copy how i speak you then use your own voice to coach the performance there's no way you can copy how i speak and finally there's no way you can copy how i speak so what i've just played for you there is the default ai voice output for decard then we had gavin who's a replicas designer sit there and record his version of how he wanted the performance to come out and then you have the final output that combines gavin's attributes in terms of his cadence and his tone and his energy and applies that through an ai voice this can be extended to any of our voices so here's a version with a female ai voice there's no way you can copy how i speak there's no way you can copy how i speak there's no way you can copy how i speak

so all in all really cool functionality this is still in development but we're excited about this feature because this is how we think at least for high fidelity characters you can solve lip sync or it's one of the ways in which you can solve lip sync so finally now that we've shown you all of the tools available with replica uh we've actually gone ahead since you were so kind to fund our project we've gone ahead and created an entire cutscene uh for the quest and so we're really proud to present the quest remember the time we fought the dragonos of course i remember i remember a dragon swooping down and almost lighting the whole field on fire then all of a sudden you somehow were elbowed deep with your obsidian blade in its jugular do i hear a thank you for saving my life coming that's not what i'm saying whatever you're trying to say

that's what it sounded like to me at any rate i smelt like dragon fuel for weeks after that speaking of smells do you smell that it smells like ghouls the girl they must have her scent the village is only an hour's ride if we hurry we may be able to warn them there's no time get ready [Music] remind me again what the plan is kill anything that comes our way freya behind you [Applause] there's too many we must make for the village nope you get to the village i'll leave them away find the girl she is the key no i won't leave you i've got this

now come on you belt are you hungry you want the fresh well we're hoping you really enjoyed that it's not bad given the fact that this entire cutscene was put together by one talented cg artist working for about two and a half weeks and so from an indie developer perspective voice ai definitely gives you super powers and this is also quite novel i should mention because it's one of the first early use cases of using ai voices to drive digital humans especially in this case realistic unreal engine metahumans uh to make a cutscene so as the technology evolves all of this will start to look and sound much better but it's really not that bad given where we

are right now in fact one could go as far to say that it's kind of like we're comparing toy story one with toy story four and see how far they've come in a matter of several years the good news is i don't think audio technology and ai voice is going to need that much more time to get to very acceptable levels even for big budget studios and we're definitely working on it and so we're finding that a lot of the biggest studios use replica in the pre-production cycles and during the early creative process and if you think about it from the context of game narrative this is also a great chance to really help tell a compelling story without having to resort to voice actors which traditionally happens much later on so you've actually got all this extra free time to experiment with story ideas and dialogue and that's what we mean by voice ai literally gives you creative superpowers because this was not possible before so it's really hard to

compare what was possible before voice ai with what's possible now because they're two entirely different things you could use voice ai right now as a big studio in pre-production but as a smaller studio or an indie developer well the voices are good enough for you to ship in the final game so you've gone from having no voices potentially or just text on the screen to actual voice acting pretty compelling in your game and doing this in an affordable way and in a way that doesn't take you too much time so all in all voice ai gives you super powers we hope that this will help game designers and narrative experts make bigger and more compelling games if they have tools like this early on in their cycles so here's an example of how a pretty big studio midwinter entertainment used replica in pre-production in their title scavengers and this is how they did it so they had writer game designer creative producers and a bunch

of folks all around the world being able to iterate on big parts of story and dialogue and get those changes in front of their 5 5000 beta testers on a almost daily or weekly basis again this is an example of something that you could never have done without this kind of technology and obviously scavengers then brought real voice actors in to record the final lines once they had ironed out all the pieces of what they actually wanted to say so this helped them in a massive way it's important to say here that voice actors are a really big part of what replica does we're working closely with a growing pool of really talented voice actors that help create all of these voices that are available in our library in fact some of the voice actors we've worked with have made three or four different character voices that are all on our library and we're developing profit sharing models so that as these ai voices are used by millions of studios around the world the voice actors who helped create them can

earn a passive income stream alongside their regular day-to-day job you know so this is really a win-win scenario it's great for talent and it's really great for creative studios because now they can do things that traditionally were impossible so we're really happy about this um and that brings us to our mission replicas trying to build the world's greatest library of ai voices for games and movies and as the technology improves uh we will expand our pool of talent to include you know premium actors famous voices celebrities and work with the unions so we're really excited about all of that finally is the future bleak this is a joke title guys uh the future of storytelling is not ai and npcs alone but we do think ai and npcs have a part to play in the future of storytelling and so this video that i'm about to show you was is actually a game created by a user of replica so the game is called modbox it's on steam and so the developer here has done this

really smart thing where he's combined open ai's gpt-3 with replica voices so in this game which is a vr experience you can walk around and have believable conversations with npcs and they will respond to you in almost real time check it out hello bobby what do you want uh where are you going i have to go to work i'm already late and my boss will punish me if i show up even later please leave i can't help you right now uh where do you work i work at city hall it's a government building in the center of town okay so i paused it there to explain to you what's actually going on again as you can see

the developer who's actually made this video asks a question to the npc after which point there is a bit of a time delay or a lag it combines a whole bunch of different ais strung together so there's a one ai that listens to the developer or the player speaking and converts or transcribes his speech into text so that that's part one that then sends the text transcript to gpth3 gpt3 takes the text input and then comes out with a response based on that text input that takes a bit of time and then finally when gpd3 has its text response that gets sent to replica and replica converts that into bobby's voice and that takes a bit of time so all together you've got three separate ai systems that all add up in terms of how much time they take but we think it's really awesome you guys should def definitely check out modbox and steam because it's a fun game to play we've covered a lot in this video so hopefully you'll also go back and

re-watch sections and if you have any questions or any thoughts or any feedback that's our email so we'd love to hear from you we also have an active discord community where loads of replica users have discussions this is where we make early announcements where you can potentially sign up for the beta and get access to some of those features that are in early access and it's also a great place to share the kinds of work that folks are making with replica and how they're using that to make games and movies so we hope you'll join and check it out but thank you again for tuning in it's been a pleasure speaking here at gdc sres from replica signing out thanks