The Frontier of AI NPCs with Convai's CEO | Purnendu Mukherjee

hello everyone and welcome I'm here today with pendu mukui who is the CEO of convey and convey is a very interesting company taking AI technology and applying it to gaming at least as a start and so pendu I thought we could just start by having you explain what is it exactly that you're working on what is convey what is the service yeah so convey enables humanik uh abilities for characters in Virtual Worlds so now you know non-player characters or AI characters in Virtual Worlds can understand what you're saying respond back intelligently and Carry Out actions in the world perceive the environment effectively have their own lives and and engage with you in various manners can role play anything and can uh be used in various kinds of applications not just games a NPC

basically AI NPCs yes that's yeah AI NPCs AI characters they are very various terminologies yeah like uh yeah and can you tell us a little bit more about you as well in terms of your background and how did you get to wanting to start convey and working on this problem of AI NPCs yeah so many things came together for me to um uh start convey I would say there are like four or so reasons uh the primary one I would say is is I had this major bent towards understanding how the human mind works or my brain works effectively uh you know where I would analyze myself as to how I learned this how I why did I react the way I reacted and that naturally led me into uh you know learning about AI in a in a very early age like when I was in my high school undergrad but I was quite disappointed about the State of Affairs uh with with with AI back then which was 2009 0 and uh the mainly because it was

very rule based at least the books that I learned from and uh and uh you know like there was no uh data oriented learning back then and and uh and I realized like we humans don't learn like that we come with a blank mind and you know no matter if a baby's born in China and you let them grow in in in India they will pretty much speak Hindi like uh like like they they were always born there so like they come with a blank mind and they learn BAS on data around them right and um they uh so like that was not how I was seeing the State of Affairs in at uh back then uh I after my undergrad back in India I was uh uh working at a research lab and uh that's where I got introduced to deep learning back in 2015 which struck a card in me such that whoa this is exactly how I've been thinking about it and I went very deep into deep learning and got me to move to uh us for my grad school where I

studied all about deep learning national language processing and uh uh and did my thesis track on question answering systems and shortly after graduating I started at uh at at Nvidia where I spent the next four years primarily focused on uh you know national language processing conversational AI which is where I also found the and I had a bent towards the kind of things that I wanted to do based on my interest and that got me into this this research group that uh worked heavily into conversational AI one of the projects that I did uh that got a ton of traction internally and and basically uh Jensen showcased it back in 2020 in GTC um where uh it was about bringing this uh it was nvidia's first 3D avatar chatbot and I got a chance to lead that project from from inception uh and that is where I learned the pain from a developer standpoint of what are the different set of tech technologies

that you need to bring together to make that happen and that also connected really closely with how I have been thinking about the human mind and such uh so I found the problem you know uh that uh that I that was really close to my heart that I wanted to solve something that I'm happy to spend the next 105 years and um at the same time I had over the years I've been learning about entrepreneurship I have uh done you know like uh projects before that for with heart and soul that uh I wanted to um uh that that got me to learn a lot about business and I wanted to apply that and I was a big time G competitive gamer back in undergrad right so like all of those pieces came together I was and and I realized that you know this is something that I would be very happy to work on that got me to jump for for convey uh about two years ago and just to be clear on the progress of this technology where are we today and if you were to extrapolate into the far future like 20 years from now what would be the

difference between what's possible today in terms of AI versus what's possible in the future what's possible today with AI in general or like AI PCS oh AI PCS okay like so more specifically what are the current limitations such that if I'm talking to an AI NPC I would not be able to distinguish the NPC from another real person for example right right right okay so the main limitation as of as absolutely today both uh the current state of the art as well as in our platform today um primarily would be voices so so what are the different levels uh voices animations Fidelity of characters these are kind of like the main main aspects that will tell you that oh yeah it's not fully human but uh so like for example if you do text in text out it it is going to take you a while before you realize this is not a real human uh you know if you adjust the typing speed and everything you do that uh and you talk to the latest large

language model uh crafted properly with the right prompts it will be hard for you to tell if this is not human or not the moment you add a voice layer on top of it you know now it's voice in voice out excuse me um then it becomes um um you know uh a little more obvious because the voice may not have the right pauses the inflections the um uh you know the the natural uh inflections that we make the emotive inflections uh there is a lot of progress happening over the last few months and last one year which is getting there but it's still still a way to go to solve that fully and then especially when you do the embodiment aspect of it meaning you give it a face and a body um then it has to have the like the human mind or or any an animal nature or living beings have been trained to observe the facial changes in a very very very minute manner right and those things need to be done right if we are shooting for a high fidelity

embodied character and um those those are areas that we need to improve effectively to cross the uncanny valley right that people people notice and the reason people notice uncanny valley is because our minds are trained to capture those micro expressions and uh and changes and you know so uh those are the two main things to um to to start with you know like within 3D characters and avatars but then there are more things like you know as you go deeper and deeper into it like is this character aware of its surrounding is this um AI character able to move around in the world naturally just like a human W and things like various other things basically and when I tried your demo during GDC which by the way I thought was very interesting and cool I did think that the one thing for me that seemed a little bit of a barrier in terms of really buying into the AI NPC character was just the that kind of like two to 3 second delay

like I would say hey how are you doing and then the a imbc would be like one two three and then there'd be a response what is the cause of that delay and is it possible to speed up yeah I think and what do you think is required to overcome yeah so I would say those are those could be related to more setup scenarios and voice qualities and whatnot uh if you see some of our demos not not all of the demos the um for the for example the one we did with Nvidia that uh you know uh Jensen showcased as well back in 19 months ago the delay was very very minimal you know and and even today there are a bunch of our demos the the difference happens if we are using a higher quality voice model higher quality facial expressions and those uh additions add up right in terms of the the latency and then there is there could be many factors regarding the the location the internet capacity the uh the latency the of of the internet

itself um and and various other things however uh assuming those are um um you know and and also like how complex the character is in terms of how much does it know in the knowledge how much does it have a narrative structure or Guard railing and things like that uh the more you add add to it the latency increases but in in theory we could have a very Snappy response time basically uh for most cases if we optimize for it so um I would say the latency problem is not fully solved but it's can be addressed to a large extent and I want to now ask you more about specific application areas and how AI could potentially more generally be applied to different applications in gaming so many years ago I think it was around 2015 or so I actually met Ray kurtwell at an AI conference and I asked him what is the first application area that AI is going to have a major impact

in and he actually kind of got upset he like looked at me with this kind of intense stare and he said everything it's going to impact everything and then he just turned around stormed off but if we were to speculate on specific near-term application areas and it does seem for whatever reason that there does seem to be a fairly significant f focus of AI technology being targeted towards gaming and even for your specific application area let's call it NPC's just is a shorthand why do you think that if we look at funding activities there's you guys also enrolled but as far as the amount of funding going into this specific application area why do you think people are more excited about this space that you guys are going after relative to the other ones and then maybe you can even talk about some of the other application areas that you think think might gain more early

traction yeah so I I think it has to do with uh traction quite a bit you know like uh uh like I think I think your your question BS down to like what what distinguishes uh us you know versus like new new players versus is is that is that what you're shooting for or well what I'm saying is that when I look at the AI space generally right so I see a lot of AI technology being targeted to many kinds of applications but certainly in gaming as well as a focus which is great and then within gaming you see AI applied to different things like generating 3D models based upon 2D sketches or generating character animations based upon AI or even animations based upon text or speech prompts and then there's all sorts of different applications in gaming you could apply AI to right but when I look

at funding it seems like your area is one of the areas at least from an investor perspective there's a lot of excitement around so why this AI NPC application versus other application areas that AI potentially could be applied to sure so okay I see I see I think the the what it has to have is probably the potential impact and thereby the market size so what um convey is doing for example is not just solving for games I mean it is our Target domain right but if we solve the problem of these embodied AI character in Virtual Worlds it solves or effectively uh almost like everything that a human can do in a in a in a virtual world so not only is it all kinds of games right role playing games or you know like uh MMOs or take take

take any kind of games where either today it's a scripted AI or even a multiplayer humanik World those kind of uh uh places this this AI NPC can definitely uh play there right or act there but it also opens up everything else uh when it comes to um you know servicing uh uh like brand agents or learning and training or education characters that can be your Mentor can teach you in all kinds of sectors all kinds of Industries and um and even real world cases where if you have this very realistic High Fidelity character you know in in in 3D that could uh affect and that could that could have a conversation communicate uh it could uh be applied in real world use cases as well all the way take like any information agent kind of scenario whether it's uh talking about uh certain of product inside stores or giving you

general information Direction in malls in um you know in airports or take take anything right so so all of these cases um these uh AI characters can or embodied AI characters can have a uh have a huge impact so it's the market size I would say that that that allows uh probably for higher funding in this in this space versus uh versus developer tools where you have developed it and that's it that was the lifetime of the uh of the use uh for a particular game or asset and whatnot so that would purely be a developer Tool uh in our case while it is a developer tool it is focused on developers but when developers go to production with it it's it's it, X's the usage for our service right like if it's it's one or two developers developing that game when it's played and it's going to production thousands of end users play it and they are calling our services right so so basically that that is uh an area where

the life cycle of our usage for that particular developer for that game doesn't end during development process it goes into the the the runtime process as well um which kind of separates us from most developer tools uh and and and thereby the the possibilities of what that can happen all the way to things like West world and and free guy kind of scenario is possible with our tools to to like real world cases like learning and training brand agents and various other things it kind of sounds like there's optionality and the potential if you kind of get this right in terms of gaming as the initial Beach head you'd then be able to kind of like go from there no I would say it's kind of like opposite actually so uh the the reason is like we we we target gaming because that is the the super set of all features that is necessary and also gaming has the most um discriminative of

audiences it's the most hardest one as well you know where it comes to like it has to be perfect from a quality standpoint has to be low cost enough you know like the the constraints are much higher uh within gaming and also the the the aspect of innovation of things that you can do uh and and the amount of uh usage it can have is also higher but if you solve for that solving for Education application solving for brand agents is much easier it's much easier yeah yeah and if I were to think about and certainly if you were to look at the Players entering the space right now I've seen you I also recorded a podcast with kylin from inworld but if you were to think about the players coming into the space and if as a developer trying to understand the differences in terms of approach or business model or things of that nature how should we be thinking about convey or enal or any other player in the market and you don't have to talk

about that specifically but could you talk about whether you are all trying to do the same thing with a similar approach or what are the different ways in which you could think about the differences in terms of the basis of competition or the way in which you guys are approaching this problem differently or maybe it is pretty similar in approach um I cannot talk to the the plans of of competitors we primarily listen to Listen to uh our uh uh users developers or and audiences and get feedback um and I think um uh in terms of what is our Focus I can tell you that so so for example the the number one Focus for us is developers we are we want to be known as the uh Earth's most uh developer Centric uh you know company so uh you know like taking inspiration

from Jeff pizos so uh so basically the um uh so that that's what it is you know in terms of having great developer experience you know like consistently listening to them and a trading so that's number one uh number two is a deep focus for uh 3D so uh everything that we do everything we integrate we look from the lens of uh you know how this will apply in 3D worlds right so the aspects of uh embodiment is very important from a 3D avatar aspect of it uh but also at the same time how uh how would the this particular AI character effectively have a life of its own you know like they they will in interact with other AI characters they will interact with the environment around them they will uh you know like uh engage uh in um actions seen they will perceive the environment and Carry Out actions based on their

conversations with um with uh with the end user or another AI so uh maybe they will plan their day they will um uh they will get they will need to change their plan in the middle of the day based on interactions and eventually their personalities will shape based on their interactions with users uh or relationship with users and um they will have memories and things like that so uh so those are some of the things that we do uh that involves being in a 3D environment uh very deeply and it connects with what I was saying about my you know like uh how why I started convey in the first place where you know like it's important to have that 3D grounding uh very importantly so uh those you could say are are kind of like our all the features that we do how we do it uh how we craft it um has a bent towards these uh actions and uh perception of the 3D environment um of course making the platform number one

for developers cool and maybe just for our audience because we do have a number of game developers that are part of our audience could you walk us through so let's say I wanted to make a game like cyberpunk with an NPC character may be very similar to the Nvidia demonstration that Jensen had demonstrated could you walk us through the process of integrating your technology what are the specific steps required from step one through whatever one through five one through 10 what does a developer do how do they work with you and I would assume most people in gaming are probably using Unity or unreal so how do they get that into the engine and what does that look like so basically there are three steps and then we are going toward the fourth now but the three steps are this you craft the mind of the character character right where you decide what is

their backstory how do they sound like what kind of personality they have what kind of knowledge they should have you know like uh what are the areas they can or cannot talk about do they need to follow a certain narrative you decide all of that when crafting the character's mind craft is this is this a text document or just like a description or is it something else yeah it's it's basically our our playground uh you know like uh where you effectively write the backstory you choose the personality with the different knobs that is there you provide example sentences of how they will say you craft the narrative is kind of like a state machine with that that flows uh you know from from node to node uh so so basically these are these are all the all the tools that and you don't need to uh H have any prior knowledge for for doing this right so you you craft the mind of the character now that you have this particular Mind through which you can have a voice in voice out conversation now you need to take that mind and embody that within an

avatar right of of your choice we support most popular Avatar systems and we ourselves are Avatar agnostic uh you know so we connect you can connect it with a metahuman Ready Player me reallusion character or a custom avatar any kind that you want and um and and now our apis drive the facial expression the lip sync the hand gures all of th those while talking right like while it's listening idle animations and it's responding with with with all of that animations playing at the same time driven by our apis so so basically that's why I'm calling it in an embodiment aspect so once that is done when once this embodiment is done the third step is giving this Avatar the ability to perceive the environment around them and Carry Out actions so now if you remember that the one that we did with with Nvidia the uh the the second version of it it there there were characters this J and NOA Char you could tell them uh you know like can you get

me a Ramen or can you you know turn down the light or bring me a give me your favorite drink or things like that or can you take me to your lab or something they will do those things not only just will they say respond but they will carry out those actions so our apis are automatically able to connect with uh all of your available actions or available animations that you have and apply them uh intelligently right based on on the response of the of the character and execute them right so so all of those things are um are applied you know like uh uh automatically so like that's kind of like giving these characters the ability to not only respond in interesting ways but actually take action like the what you say and do has consequences right uh uh from from a conversation standpoint like if you ideally enrage them or you know they may attack you or if you're nice to them

they will uh show you a certain location or maybe they are programmed to be your escort so they will take you from location to location and give you a tour or escort you out of a situation so we we created a couple of the demos that had the these functionalities um so so you could do all of those things next now where we are going is effectively having many many characters in the environment each of them having their own plans and executions and interacting with each other effectively like a simulation kind of scenario and being able to uh be in a scenario where there are many characters talking and engaging and the AI character has to pick and choose what and when they should uh respond basically that is a that is like a kind of like a group discussion Dynamic is uh is is a new an interesting problem that we are uh helping solve and so that's great and when you think about doing that today because NPCs in games exists today and it's

generally executed through some type of approach using a dialogue tree of some kind right and so if you were to kind of characterize the effort cost complexity of using a dialogue tree to try and develop an NPC character versus using the AI approach how would you compare those two approaches so those there are two things there one is the the dialog tree aspect of it and then there is the behavior tree aspect of it right so traditional traditional dialogue trees are uh you know you approach a character the character may say something um and you have like three or four options on the screen that you can choose of what what to say and you choose that and the character will say something else and then you'll be given another set of three or four responses that you can do so effectively it's like there is a node which has four options that you can choose and each of those no nodes has another set of options it's kind of like a tree there right that you are

navigating the whole flow um which uh which which is fun and nice but like it still doesn't give you the ability to explore alternative scenarios of than the options that are presented right so these these open-ended conversations especially what we did with narrative design can still like the the good thing about the dialogue trees is that it can have a certain set of outcomes of where the conversation can end or lead to no matter which options you choose uh the problem with open-ended conversation is that if you're not going into these tracks or outcomes it can uh you know go on forever without coming to a certain outcome uh what we can we can effectively with with this new feature that we did called narrative design you can combine the control of a dialogue Tree in terms of the outcome with the open ended nature of uh these large language models in terms of uh being able to go deep deeper into the uh Mind

Of The Narrative writer right so uh and and understand the character even more deeply so so there is that aspect of it that you can combine those and the character will be able to go to certain extent into certain topics uh or L of the world and such that you are interested in but it can if it has an objective and an agenda it will pull you back to to what it needs to do where you need to go you know so it'll get you back on topic and um so that that's kind of like the the interesting part the other aspect that you mentioned is behavior trees now Behavior trees are more important for uh the actions and animations that the character may take when uh you said something you know or hit did a certain action the character needs to react in a certain way so each of these effectively let's say the dialogue tree that you had which is a very equivalent to our narrative tree can um uh can can effectively have their own

behavior trees effectively so so basically the the character can respond to what you are saying through actions and animations and and um uh you know doing certain thing in the world as well so so basically it's kind of like a a state machine the the the dialog tree or dialog graph or narrative graph in our case could be effectively like a state machine that each of those nodes are having their own behavior trees right so so uh so it it it allows for open-ended conversation and yet connects with the animations and uh actions that the character can take right and so it sounds like on the dialogue side a dialog tree is very fixed and kind of limited in terms of the interactions you can have with the NPC but with your Approach at convey it'd be more open-ended and using your new tool you can actually drive towards certain outcomes or like that and on the

dialogue Tree Side it sounds like well I would say that for some games like Call of Duty for example you've got full-time dedicated AI people doing these dialogue and behavior trees like dedicated people but maybe with your technology if you just kind of set your knobs and put in the context of the backgrounds of characters and possib desired range of actions maybe the effort is potentially dramatically less is that a fair characterization yes depends on the kind of character that you're making if it's a lead character you know main character you still want to ensure that it's still staying within the bounds of of what you're doing and it's carrying out the objectives uh for a for effort to impact ratio if you are shooting for that you know the amount of effort that you need to have like a oneliner NPC to having a

open-ended NPC so the effort is pretty much the same that has doesn't have much impact to the story but you can chitchat with this side character the effort is pretty much the same uh yet the impact is significantly higher in for the main main characters it's uh I would say you would anyway need to today put a lot of effort and you still need to put a lot of effort in terms of their animations in terms of their narrative flow and whatnot but but the experience is just like next level you feel like this character is alive uh in terms of um its inter its intelligence and interactivity when you are giving it curve balls after curve balls in terms of conversation reactions and it's understanding what you're doing and and uh you know responding if um in an intelligent manner so depends on what you're trying to do uh I would say it may it may keep the effort level more or less the same you know in terms of uh the development

process but the impact is basically 10 to 100x higher from a gameplay perspective yeah I guess it would be difficult to get even with AI to get the kind of behaviors that would be specifically appropriate for a character in terms of the specific kind of personality that you have in mind and specific kinds of actions and for the actions not to be bad where actions and di dialogue are safe like right yeah yes yeah yeah yeah yeah exactly exact yeah so and and basically the uh the reason is that it's uh uh you know like the amount of animations the character would need you still need to connect it with the uh the you know like the system and and the most important reason the reason is not less effort is because uh you have to test it quite thoroughly here you only had so many uh limited set of things

that you knew that it would do and thereby test here there is effectively infinite things it can do so we created this test framework actually that can help uh help you as the developer or designer of the game G gain the necessary confidence that yes it is responding correctly no matter what I do or what crazy places I take it it still stays with its character still stays within the lore says the right things has the correct knowledge does the right animations so you can test it within this test framework and gain the necessary confidence to ship it to production so because of this requirement of extra testing uh I would would say the effort is not less today but the impact that it can have in the uh in the during the uh game playay time is it's just at another level basically I want to now talk about the economics in the business model because as a developer are working in a mobile freeo play cost can be a major factor and it

can oftentimes make or break a game in many situations so there could be a number of ways given that you guys have a new technology that you could potentially charge your customers so I was wondering if you could speak to the economics today in in the business model of how you plan to charge and I think game developers would be interested in terms of the two models freeo play where you've got an ongoing live service and how you would charge for that and then for a premium game if I were to make a game like cyberpunk with a one-time single purchase do you think of different pricing models for those two or what do you do so so you as the developer are in control of it right based on your uh your economics uh in terms of the the underlying costs of what it takes to uh

run such a system right so effectively we are running somewhere between uh five to six gpus minimum for all of the pieces to pieces to work so like there's the large language model there's that could take anywhere between one to four gpus right um there is the um or you could say like you you could call an external service like open AIS uh models like assuming you have fine tuned them enough to stay in character and do all the things that you need to do um and then there is the voice input service which is relatively pretty less cost the speech to text then there's the text to speech service like if you want really high quality voices from an emotive standpoint and then there is all the uh animations it needs to do all the way from lip sync facial expression hand gestures and all of them needs to be kind of almost real time generated because of the possible variations of it right so there needs to be some models deep learning models that are executing for that and potentially um

you know and this goes into the llm side of things as well which is seen perception and actions which may require a larger context input you know in terms of the number of tokens so um effectively if you uh and and where we keep our pricing is pretty much at cost in terms of how much uh if you if you combine all of these things okay how much does it cost to have a uh you know 4,000 tokens for let's say opening I's fine tune model how much does it take for generating a speech whether it's Azure 11 labs and whatnot you combine all of them and that's kind of like how we came about to the to the price basically price point uh and uh we we charge it by interaction meaning you say something or ask something to the character and the character responds back voice in voice out with animations out and all of that right so uh and and why I say it can work with almost all business models is uh if you're a free topl for example

in your case you could potentially unlock this interactivity uh you could give that to based on your economics you know how much you are making per user and whatnot uh and how much you can afford to spend per user you could let's say budget uh 10 interactions that I can give to every player no matter what and then beyond that if you want to unlock more engagement more more interactions they can buy something you know like uh offer the character address or something and you know like uh or buy them a gift or some kind of um you know some kind of purchase element basically and that will unlock some interactions with them so you can design around this from a free-to-play standpoint um subscription based games or battle pass kind of scenario works out perfectly fine like if you can budget a few dollars or percent out of that uh pass like let's say somebody's buying $10 per month you know you can say Okay effectively we can spend $3 per person uh average out some will be power users some will barely use it uh and and you

could provide them uh with with these interactive characters um yeah and then uh onetime purchase games can also budget it out like let's say it's a $30 game you could budget $5 for for that particular user where they can not only finish the game but they can play the games like two times over or something right and beyond that they can purchase more interactions uh but what where I see it best serve is persistent worlds uh multiplayer worlds you know which are effectively always always on kind of scenario right there is usually a uh persistent singular World kind of um or or a particular server those are those are places where um uh these uh a inpcs can not only just be there serving many people but but also evolve and you know like you can see the effects of that character evolving over time and um yeah like we did that recent deploy

deployment with second life where you know like uh they they went live on a small scale in a in a certain region currently um and uh these characters are there all the time you can go and talk to them and as we add more and more features there they they will start remembering you and uh you know like remember your conver conversation potentially have friendship with you and things like that so so yeah these are these are you know like the economics works and and and it's today like with this set of quality it's today and the uh either the price will remain same and the characters will just become absolutely insane in terms of the conversation quality the um the lifelike aspect of it uh or if you are fine with today's quality the price will go down right so so basically um the that's where that's where we are headed and the and the prices are going down drastically like two years ago when um

you know when I started convey uh the price for uh what what is gp4 today was uh the price for gpt3 base model like Da Vinci was basically um uh more than what gp4 charges right or or similar basically right then uh in 2 years that became so old it's it's almost deprecated it's worse than 3.5 right and 3.5 is on10th the cost of of uh of gbd4 right so um so basically the cost is reducing by more than 10x uh in I would say in a year year and a half right so so if you're fine with today's quality it's only going to be it's a matter of one year to two years that the price will drop by 10x if you're fine with today's quality right um and even even then like you know if you know your game is going to have this much audience we can potentially deploy you know like

fine-tuned open source models dedicated for your case that reduces the cost drastically right that can um one thing is development cost and another thing could be production cost which we could bring down in a in a drastic manner for your your game for example can you give me an example of That So when you say you can bring down the cost dramatically how would that happen how would you do that yeah so like if you if you know you know for example that in a month you would probably have uh let's say 100,000 active users you know like uh or concurrent users let's say overall you will serve um 1 million users uh you know throughout the throughout the month uh and then each will have at least uh 20 interactions so it's like 20 million interactions per month right now we will know that at any given time when it comes to if you have let's say 100,000

concurrent users not all of them are speaking at the same time probably 5 to 10% are speaking at any given time with the characters so we need a concurrency we need a concurrency of like 5,000 right 5,000 people that can speak at any given time uh we now know that how many gpus we need to deploy right so to achieve the to maintain that level of concurrency and that can serve the entire um uh you know like user based throughout that that 20 million interactions for that entire month now you can do the math right like how much can you make with 20 to 20 million interactions or based on your economics and and business model and what is the cost of deploying those and you see the difference right so so basically our goal right now is to make make this feasible and prove that this is a uh order of magnitude better in terms of uh experience for the end user and it's engaging its fun and um the costs are only going to go

down from here and I'm here in India and we just started a second game project and I've talked to you about this pendu but this second game we're trying to serve the Indian audience and certainly for the Indian audience getting support for lower-end devices is critical but getting payers and higher payments in India is also very difficult and so how feasible is more of a client or oriented Solutions such that you're not paying per transaction over the cloud versus the current model and for our audience I want to explain that we want to create a kind of like an AI girlfriend that you're dating virtually and so we want players to be able to interface with this AI products and so even if we were to try and get like seven to 10 Rupees from our players like how low can we bring the cost cost down or could we make it even client based so

that the costs are going to be based on that per transaction model how do you think about that yeah which is where where which is what I were alluding to in my previous calculation where uh at the end of the day what it'll come down to is concurrency right it'll come down to um uh the concurrent people live for your game that uh which will which will give us the ceiling for the amount of deployment that we'll have to do for your certain game right and uh if we could project that well you know we could uh deploy that many Hardwares like gpus effectively That Could set the the base costs effectively right so um and it like if we really want that future to happen for uh you know such Ai and PC based environments and games that is how uh it it will come down to basically we

will have our uh you know license or whatever margin on top of the hardware hardware and that will be much much lower cost from a for interaction basis and also you are not capped by interaction at that point right but it will H so happen that since we know this is your target concurrency or you know like we have deployed that GP so we have a base level cost that we will have to support or or deploy accordingly as you you come so basically if whether you have uh 15 million interactions or 20 million interactions the cost from our side is probably the same but it it's so much worthwhile for you because it's it's much better than a per interaction model I think that's where where it'll go to in terms of um when it comes to massive usage across many many such games so um where we are at today we just go with the Baseline economics but we know the uh but those are more like developer cost kind of scenario when you go to production and if you're having

millions of uh players on a regular basis uh the way it will effectively be and I think the cost to impact ratio will more than justify itself meaning um whatever uh the end user is paying for whether they're paying uh effectively let's say in Indian rupees equivalent it it is like $100 a month or $500 a month or whatever it is right so so that that they are paying with the experience that they're having daily they are playing for a couple of hours let's say is just outsize that they are very happy to pay a hundred or $200 rupees um you know so per month the um yeah so that's that's kind of like the the thesis and it it is well Justified uh from your margin standpoint as well as um overall economic standpoint where we can get this to and I think I forgot to ask when we were talking about applications but I also want to ask you

about what you're seeing today in terms of what you are seeing from your customers who you're speaking to uh you had mentioned Second Life For example but for AI NPCs what are the specific areas that are starting to be used the most or where there's the most Traction in terms of whether it's the genre of game or specific ways in which the technology is being deployed what are you seeing today and then maybe you can also kind of speculate about where you think you might be headed in the future as far as where you think the traction is going to be yeah so um we we see a so like I said like it's it's it's very early for gaming in the sense that um it still has this uncanny value to cross the voices to be better um the the integration to be more more robust and well tested and thoroughly tried out there is yet to have a major breakout

level game you know that's played by millions of people that's yet to happen uh so I would say gaming is still early however from our user based standpoint or I would say interest standpoint I would say 50% is still gaming the other 50% is non-gaming applications like but still pretty much like games those are like you know like Virtual Worlds with characters that are uh that are like learning and training applications or uh you know like brand Avatar use cases or uh you know real world deployment scenarios so like digital humans so so basically um uh where we see the traction I would say within games it would be MMO worlds like second life right like uh you know uh multiplayer worlds like second life that can Prov that can help solve the solve various aspects of onboarding uh solve aspects like um you

know uh taking being a tour guide like a uh you know character that can show you around be a companion character uh those are definitely use cases that we are seeing um uh in in games and Virtual Worlds within the like social virtual world games um and uh very similar use cases in learning and training applications and and brand Avatar use cases so so yeah like it's kind of like evenly split overall uh we uh but we also see you know developers who are developing uh single player games and you know like um uh both multiplayer games that are leveraging like we have a big Indie developer use use uh base basically that is uh actively developing a lot of these projects um uh and plans to plans to ship to production um by later this year okay the other thing that I think about

when it comes to Ai and maybe not necessarily conversational or AI NPCs as we've been speaking about would be something new because we have a new capability and I would assume some new kind of game or genre of game will be enabled whether it's because of Technology like your convey technology or other kinds of AI capabilities that are brought to bear against something in gaming and if you were to think about that what do you think would become possible because of a AI in terms of new kind of game or new kind of game experience yeah so I mean anytime you have this open-ended aspect of conversation is just new right like uh in terms of the end user experience now that has to be applied not just for the novelty of it but where it makes sense

from a overall game design aspect right so uh a clear fit is any kind of role playing game right where um you are a certain character that is inhabiting this world along with many other inhabitants you have to um you know kind of like uh achieve certain objectives based on what you are playing uh in the game and then um there may be set of uh and usually today it's done by various different kind of people that are playing different personalities right like real people but there may be a certain set of uh uh characters that uh that nobody's role playing so AI could fulfill those right and um uh not only that but let's say yours is uh uh either a PVP game or maybe a collaborative game you don't have enough players to uh form the team so you could have these AI NPCs that you can partner with collaborate

with so so these are these are definitely areas that people are trying and looking into for application of these AI NPCs um and this and where we we see and which is what I was referring before is uh these game worlds being inhabited persistently by these AI NPCs where it's kind of like the movie free Guy where um you you know this um this character you can interact with you know like a almost like a human player right and uh they can evolve based on their experiences in the game and their story can evolve their planning can change and things like that so that's where we are going like almost like a simulation right like each instance of the server that is hosted let's say it h it consists of 200 people in in a particular server uh that 50 of those could be NPCs and 150 could be real people and those 50 no whether anybody's

there or not always inhabit that uh server right uh and live their life effectively so um so I think that's definitely where where uh like uh which is which is what I saw when I started convey you know uh bringing Full Circle back but these characters can live in a simulation and uh you know like uh effectively evolve there kind of like Westworld but more appropriately freeway so when I interviewed kyin from inworld I asked him this question about whether is it possible to do Dungeons and Dragons with this kind of conversational AI technology today and he seemed to think you could would you agree with that is it possible for there to be Dungeons and Dragons today with existing technology yeah so so the the main challenge is this right um the the fun

part about such a game like like Dungeons and Dragons is that you could totally create something entirely new right like or come up with an concept and even have a dialogue or mechanic around that that whole new invention that that came out of nowhere right the challenge of that is uh about having that a game is basically uh the the uh totally New Creation aspect is um uh like how do you make that real convincingly especially if you're going for High Fidelity worlds right like now you also need that particular object that you talked about but not only that you may need to interact with that object you may need to animate generate animations into it uh in terms of like if you if you create a world that is uh dreamy enough or I would say crazy

enough that you could use all kinds of real-time generation generative aspects of create a new objects create animations on the Fly uh and all of that you could do that but today as it stands I mean even like paper and pencils Dungeons and Dragons is that possible oh paper yeah oh yeah that that is totally possible I think some startups are already doing that um I was I was thinking more like a realistic triaa 3D World kind of scenario that would be challenging but but but yeah paper and pencil with a with a you know scope you can totally do it especially leveraging things like you know DOL mid journey and and uh you know like conversation these multimodal uh like Gemini uh Pro and and and gp4 vision tur turbo uh you could totally do a lot lot using that paper pencil 100% uh the uh when it comes to 3D worlds there is there's a lot more but what I was wanted

to go there is that you can do a subset of that today meaning um if you know that you have these let's say 1,000 assets or objects that you can only use and you know you have let's say 2,000 different animations you could use and this is the scope of your large scope of your map now you can think of all the different possibilities right maybe maybe 50 characters are there so it's uh the possibilities is are still exponential number of characters multiplied by you know number of animations and actions they can take multiplied by number of objects that they can interact with multiplied by the different locations or Grid in the map they can move to or take these objects to so that effectively this could become an exponential uh possibilities of things that they can do and interact and and engage in you know like so um it is uh you could say a limited set from that Dungeons and Dagon scenario uh that you

could still have in a very 3D realistic World okay great and maybe the last couple of questions I have and just to shift the focus a little bit but I'd really love to ask you more questions about your entrepreneurial journey and by the way it's not lost on me that you left Nvidia at a time when Nvidia had this massive runup and so certainly the opportunity cost for you to do what you're doing has been very high but the first question I have is in terms of your entrepreneurial Journey what have you learned you mentioned that when you were in college you didn't spend so much time studying but you were focused on projects and there were a bunch of things that you learned this time around what are the key lessons that you've learned since starting convey yeah so it it it is I mean I love my work at NV and I love the people there and I still am in close touch with them and uh it was quite a jump but it

has paid off so much more than I than I expected and and imagined and I uh and I'm super glad and I'm uh that I made that jump you know like it it it's by far the most transformational last two years like insane you know like nothing compares so um uh so so to give you the brief yeah like in experiencing life in in general in terms of uh like understanding how you know that you could create a product that thousands tens of thousands of people are using and dependent on building their business on it super excited and grateful and you know like when they they when you I receive these emails every day like tens of them that people are so grateful and you know like and so many people like that these are quite transform you know and and people that you're engage like I who who would have thought

like you know less like a little over one and a half years into my startup uh I got a chance to Showcase with with Jensen like which is like I'm super grateful to Nvidia for that right so um and and um and recently unity's leadership showcased that and we are working with the largest Studio game studios and their uh Executives and and whatnot and um uh and and and not only just these large and big names but how it changes like whether it's a high school student through uh you know someone in in other side of the world in every part of the world that is creating such transformational applications um for their uh for their Community for their like uh in especially in training and education and you know like many other places it's just um it feels amazing you know that uh and also learning about the VC world the finance world how you just coming up with an and and and basically I have realized that I have

lot more Powers than I gave myself credit for right both in terms of in terms of mainly creativity and pushing through making something happen so it's a it's a uh journey of not just uh experiencing the external world of of how how societ even like how Society Works in a sense like you know all the way from the finance world the VC world how companies are created how that changes the behaviors of people around the world right how it changes civilization in general so uh not only just that part but but a self transformation self-discovery I would say you know like of uh of like realizing whoa I am pretty good at this kind of kind of scenario like some things I of course and also and also a lot of learning in terms of like I was never a manager you know like uh at at Nvidia so I was mainly an independent contributor um but uh here I became I I

mean I don't call myself a manager but like these are such exceptional people like they can they just need to know uh why we need we are doing what we are doing what needs to be done and they can figure it out how to do it like I don't have to Ma and and that's how we choose people in our company but um the uh the but effectively from a responsibility standpoint now there are like almost 40 people at convey and so so I became like initially managers of like or or like um you know like of of about 10 or so people in the first year and then over the last one year it's been like another big increase so now it's like there's manager of matters there like people in the company I don't get to engage with uh week over week or or month over month even right so um and I'm really grateful how how they work super hard with their heart and soul and and they're hitting it off the park uh with uh with the impact that that convey is causing so truly grateful to the to the team and

convey that all sounds super positive for me and the second question I have is like when I started my company three and a half years ago it was it has it was and has been and is very very difficult and just kind of even bringing this back to Invidia Jensen actually had this talk where he kind of became popular popular for talking about through his entrepreneurial Journey how it's been a lot of pain and suffering so I wanted to ask for you has it all been very positive or has there been some amount of pain and suffering big time but the pain and suffering has been very meaningful you know like it's also the most brutal time that we are all of us goes through but like every waking up like since waking up through um uh you know going to sleep it's non-stop right uh it's non-stop

like always on like low sleep effectively try to get get sleep but it's not there isn't enough time um and uh yeah so so basically the um it is it is very tough not going to lie uh it is really really hard but it's a meaningful suffering you know like the the suffering is is something and we are suffering towards we are we are suffering towards something that we believe will have Major Impact in the world so so effectively um yeah you're choosing your battles we chose one of the hardest battles you know like of of doing a startup and trying to uh just survive and like make make it worth worth our while basically so the uh just uh but but we know that as we are doing it it's changing people's lives in a very meaningful manner you know changing people's uh day-to-day in a very meaningful manner so the uh so it's it's

worth it you know like it's very painful the pain in it's there it's a I would say I have never by by uh have worked this hard at at such a long stretch you know especially over the last one and a half years after coming out of stealth and whatnot it's uh it's pretty darn crazy uh but um uh yeah so like but it's worth you have to you have to you know um yeah okay pendu those are all the questions that I have now is there anything that I might have missed or any last message that you have for our audience um I think you you ask pretty much covered everything the message for the audience uh I would say you know we got a lot of I would say a uh constructive criticism uh over our last release at GDC and um and uh especially

those things like we have been developing for developers primarily of the things that they need this was one of those times where we got a chance to be in the players shoes or end audience shoes and and being beside them where we are showcasing in our demo room and you know so lots of journalists tried it out lots of uh studio exx and developers tried it out and we got to hear from them directly as a player not as a developer as a player so um uh and we understood the bottlenecks and pain points the what what I would say is the things that let's say uh like the journalists are criticizing is a matter is is a matter of when not if right so like that's uh uh we know those are solved problems X months from now right so so basically and one thing that I'd like to tell your audience is that we take art and artists very very seriously

you know like I wouldn't be doing this if there wasn't an aspect of creating great art right so we we we uh admire and uh treasure amazing art and uh we will ensure that we uh deliver at the highest of qualities when it comes to uh artist expectations great all right pandu thank you so much for your time and insights and all the information that you've given us and with that to our audience we'll catch you next time thank you bye-bye [Music] [Music]