NPCs Have Feelings Too: Verbal Interactions with Emotional Character AI

[Music] hello everyone so i'm kotia boida i'm working at square enix in the advanced technology division as an air engineer and today we'll talk about verbal interaction with emotional character ai so at first before going further i would like to say that all the content and assets that has been created for the demo i will show later has been created only for conferences and studies purposes so it's not a new iep at all not a new game it's just experimental and this asset has been created by the people in the bottom row so mostly technical artists and technical animators and the top pro we have two ai engineers and two product managers so thank you to all of them and also thank you to all the people that helped me on this project okay so let's move to the motivations and what are we trying actually to improve here so when we started to experiment in virtual reality with inaugural

characters we were amazed by how we were able to feel their presence in this world giving us a real feeling of being part of it but we were with our own body so we wanted to interact with them in a natural way but we were limited to buttons other classic mechanisms that did not apply well to vr so we wanted to do more on the ai side it was looking at us as if we were a ghost just looking through us not really caring about what we are doing so we wanted to also improve this aspect so how can we do it by bringing more natural interactions through voice also through body language and on the ai side by creating a more aware expressive and heavy agent that can interact with the player appropriately through actions emotion and reactions but also to enter their own needs so at first uh before the wall explanation i would like to show you a demonstration of what we have done so far and we'll be

able to see that the agent corbin will express a lot of different emotions and reaction to what he is experiencing in the world and we will he will also react to my voice and my speech and also what i am pointing in the world so let's check it hello can you plug the tank in the hole so here you understood what i was talking about but the canister has a lot of smoke emitting from it so it's quite poisonous so he's flying away right now we have to fix that can you plug the nozzle in the white thing so here i am trying to fix the white thing which is a canister in this case great yeah repaired and can you plug this in this hole please and here i was using my hands to point

at the object in the world and the engine was able to understand great take the orange cape here again with another predicate orange and you understood that it was a clip i was a bit bad about my eye at the end he's not very happy about it but he didn't know that the electricity was a something bad for him but he learned about it and so he flew away from it so what's on the menu today so we'll talk about our speech recognition pipeline and then the decision making with the emotional components and some factual statement at the end so let's start with the speech recognition and pipeline so the first block is our speech reconnection engine taking the voice and translating translating it into a sentence or a list of predicated words with their part of speech and we are sending this data to our game experience through the voice pipeline and the first block is a grammar parser

and here the goal is to translate this language-based grammar into our ontology-based grammar so you can see some new types such as predicates and objects which is much more easy to easily understood by our ai and then we have a word abstraction step where the goal is to translate these words into a language independent representation and here it's the code block you can see here so why do we want to do that well we want to support multiple languages without limiting the player set of vocabulary so if later i want to add french or another language i don't want to do all the bindings one by one manually it's too much work and i want a better way to do that a method that is maybe automated the main problem here is that words are language based so they don't have binding one-to-one between languages so we need to abstract them and the idea here wants to create the dna of a word

in this case thinking about what could be the genes so if i say take an apple or take a break they still take has different meaning so you could actually have your word take and target with all the different meaning it has and one meaning could be one gene and then the dna will be just a list of meanings so with that idea we decided to use wordnet a database of set of creative synonyms named synthet it's expressing a distinct concept and it support multiple languages which is what we want so let's take an example let's say that we want to create a concept of big in our experience as in a big apple so an adjective here at first we search in our database what are the different meaning of big we have maybe 20 but it's too much so which big meaning are we actually interested in so we said an adjective so let's remove

the adverbs and then we're going to take all the meaning one by one and select the one that we are interested in in this case we are interested in above average in size or number or quantity or extent and also significant and these two meanings will make our dna for the concept of big we wanted to create if we have some doubt about one of this meaning we can check warned database in detail and understand to which words they are actually bound to in the different languages it supports and we also have synthesis examples at the bottom so it's very easy to understand if this meaning is the one you want okay so now our statement is totally uh language independent and we are supporting multiple languages thanks to wordnet and also now that on the previous one yes this one so now

the goal will be to try to understand this abstracted world uh into the concept that we're actually supporting in our game world for instance pickup and enormous are not part of our game but we have tech and we have big as well and because they are close to pickup and enormous our grounding step will be able to transact them into taken big so here we are going to use a utility-based scoring method so here is a simple one we have the word enormous with its dna and we have a list of predicates that we are supporting in our game world so big small red with their own dna and we're just going to compare their dna and check which dna is closer to the dna of enormous in this case big is going to be the closest so enormous is going to become big so now if i say for instance

take a giant apple giant will also be translated into big so we are actually now extending the player set of vocabulary and we are also supporting multiple languages so now we don't even know actually how many words we are supporting now however here we are still outside of our ai we're just in our voice pipeline so we cannot ground everything objects particularly relies on the knowledge of each agent for instance on the left agent a known about a ball a banana in a table so if i say take a fridge it will try to understand the dna of fruit against a bowl or a banana or a table it doesn't know yet what is a fridge so we'll do it later okay so now our statement is ready so we just throw it in our statement memory and we can move to the next step which is the decision making so our

decision making system is very simple we have a goal manager and a planner and the goal manager is utt-based so we have a list of goals and we are going to computer score for each of them and we are going to execute the highest scored one in this case it's going to be listen to player and the plan that we are going to have is listen which will just copy the statement from the statement memory into the ai memory and then the agent can decide if you want to execute or not this order in this case it does want to so we are going to have a plan that will be look for this object and then go near this item and take it and that's where we are actually going to do our grounding for the object so they look for action we try to understand uh to which object it is the closest to inside the ai memory and to do so we are going to use again a ut based system it's going to be the infinite access

utility system so what i didn't say earlier is that goals can actually have targets so you can have multiple goals with a different target and computer score for each of them and that's what we are going to do for the look for action we are going to have one goal find suitable objects and plenty of targets and these targets are going to be the items that my ai knows and we are going to computer score for each of them so now we are going to try to understand uh how close or similar is this target against the object that we have in our statement which was our enormous apple so it will depend on different axis the first one is object type so we are going to compare how close the type of target and this object dna is and then the size and a lot of different axis such as color location closeness and confidence we are going to have a weight a score for each of them

and combining the score and we get the score for target so let's move into our object type axis and understand how we can compare the type of this apple for instance against our object dna what we can do is just just compute the dna of apple and compare both dna together in this case we have the same dna so we are going to have a score of one and for benign table the dna is totally different so the score will be zero i don't know if you see it now but we are looking for anonymous apple which is an edible fruit nevertheless the score of banana and table are the same very far away but bananer is a fruit as well an edible fruit so it would be better to have a better score than zero but maybe not one because it's a utt system so we have actually a better way to do so warnet organized all these thinset or meanings into a hierarchy and our object apple

is actually going to be located right there below edible fruit which is also the case of banana but not table which is located quite high in zeriaki so we can see how we can get a better score here so at first we are going to check where our object dna is located inside this hierarchy and set the node to one and just interpolate until the root and all the nodes below because they are also apples are going to be set to one and now we can compute a score for all the the different object we have so apple is going to be one but banana is going to be this time 0.86 and table is going to be 0.14 so now we have a better score and for instance if in my game there is no apple and i say pick up anonymous apple maybe the agent will actually take a banana instead but it's still close enough and acceptable it's better than a table right

okay so let's take an example this time with our apple a and try to score to score it so the type is the same so we are going to have a score of one and then for the size we are going to check our ai memory how big is this apple against the thing i know it's average in size so we are going to have a score of 0.5 and we do the same for the other axis and we get the score of 0.91 for apple a we do this for all the different items and the winner in this case is apple b our biggest apple inside the memory which is what we wanted okay so the look for action has found that the closest item inside the heia memory is apple b so now we are just going nearby this item and take it and that's what i'm going to to show you inside this video but with different objects

it's going to be an orange clip and because from now on we're going to talk about emotions and you will also see some new things such as emotional reactions with liking disliking behaviors take the orange clip sorry but please take it i don't want to take it right i will remove electricity and now can you take the orange clip great you're ready to take it this time can you plug it as a red ball please cool so the goal of this experience was to fix the rocket and by plugging the cable to the red pole we fixed half of it

that's why the agent was happy at the end but earlier he has been shocked by the cable so he was not really liking it at all but at the end he was maybe starting to like it so i will explain this in the next part but before let me introduce our emotional component so it's composed of three different modules the first one is emotion so a very short-term feeling that evolves quickly over time such as joy distress and fear and we have also a moon module which is a much more longer long-term feeling that evolves slowly and smoothly over time like exuberant depressed and afraid and they're going to influence each other and on top of this new module we have a personality module that will define the agent that has no real evolution over time and it's things such as curiosity shyness and laziness so let's go in our emotion module at first so this is an inspired occ model it's not the original

one but it may still look a bit complex so i will explain it so it's taking at the top a valence reaction or an event and we just assess from our ai point of view if it's a positive thing or a negative thing and then depending on the type of event like an aspect of object if i saw something in the world for instance i may like it or dislike it therefore i will generate a love or hate emotion if this event is a consequence of something that happen in the world or a consequence of our actions we may express joy or distress and if this event is an action of an agent if it's my own actions i'm going to express pride or shame and if it's the action of the player for instance i may express admiration or reproach towards the player and also if there is a related consequence to these actions

if it's a related consequence to my own actions i'm going to express and be gratification or remorse and if it's a related consequence to the action of the player i may express gratitude or anger towards him so that's the model it may even still be complex so let's take an example so earlier in this video the agent has been shocked by the cable and this cable was activated by the player yes press the button right so this event is negative for our agent i think right it was a consequence of the event of the action of the player activation of the button so we are going to express our distress emotion we are not really happy that we have been struck by the cable but we are also going to express hanger towards the player because it's his fault that has been shocked by this cable and that's how we are going to generate our emotions so now how can we like or dislike the

thing in the walls well earlier when the agent has been shocked by the cable we have generated our distressed emotions but we can do one more thing at this step we can also add a negative effect to the cable object but also to the electrified predicate and an effect will just be an intensity with a memorable duration so that it can forget over time and now thanks to this effect the agent will not like anything that is a cable or anything that is electrified so if there is another identified something over there it will not get close to it so great the engine does not like the rectified cable anymore so what if we tell him to take it again well the plan is still the same we're still going to look for our cable we are going to find it and then we are going to try to go nearby this cable but because he's hating it he will

refuse and the plan will be concealed okay next the mood so it is based on on the pid model so pleasure arose all dominance and the p mean how pleasant is an emotion a is about how intense is an emotion and d a bit more complex it's about how much control and influence the agent has over situations and inside this cube we'll have all of our emotions and this group can be separated into eight octanes for instance here we have the mood a thread and at the top right where joy is it's going to be the mood exuberant so we have eight of them and inside this cube we have our default mood which is currently at the position zero zero zero and this mood is going to move smoothly over time depending on what emotions are

going to be generated so let's see the update now so let's say we have a joy emotion with a certain intensity for five seconds depending on the intensity the speed at which the dot will move towards the joy emotion will be different and when the emotion has been expressed there is no emotional activity anymore so we will slowly go back to our default mood in a more complex example it can be something like that so here we are exuberant and even more exuberant right now we can see that he's very much very smiley now and now a bit of eight so we are just in the hostile mood just at the beginning and we can see a sort of vicious smile nami is quite not happy at all and now due to the same emotion he's very sad a bit afraid and now a very strong hate emotion and

we can see that he's very not happy so thanks to this pid model now we are able to blend our different animations for the different moods and get some new things such as the vicious smile the vicious smile it's very powerful tool and lastly we have our personality module which is the simplest one it's just utter parameters such as laziness curiosity honesty hobbies and i want to introduce at first how we have done the laziness one so in this video at first yes laziness is 10 so not very lazy so let's check take the purple clip please so here yes he's just going to take the purple clip nothing special take the purple tip please but now he's very lazy he does not care anymore about the color or any other

thing you just want to take the clip the closest one because it's very lazy so how did we do that so personality lazyness parameter is going to affect our look for action will favor closer object of a further object if you remember earlier i introduced our fine suitable object action with a certain axis name closeness and you can see that the weight is 0.1 so very small so maybe why is it small maybe that's the first question well we just wanted just in case we have two items with the same score we wanted to favorite the item that is a bit closer to the agent but just a little bit but in this case the lazier the agent is the more important the closeness axis should be so instead of having a static weight we are going to have actually a way that will depend on this laziness

axis and we can add a range if we want and thanks to this we are now able to express our laziness personality parameter in general the personality parameters can affect a lot of different things in the decision making they can affect your goal manager goal score or the set of actions you will use for the planner they can affect also the tolerance on liking disliking or the way the ex the emotions are going to be expressed for instance with sinus so it's a very powerful tool despite being very simple and it allows for a great variation of place and if we have multiple npcs they will all feel a bit different to each other okay so lastly i would like to talk about a factual statement which is about informing the agent about the world maybe there is another important rocket

can you take the nozzle so here is looking around in front of the rocket and you found the nozzle very happy maybe there is another in front of the rocket can you take the nozzle so hope you see here there is no nozzle but he's still going to look for it in front of the rocket but yes this time there's another right there is none other so he's not really happy about it we lied to him so he's not very happy so how did we do that well at first the player is just going to say a statement such as there is an apple on the table so we're going to listen to this statement and we're going to create a factual apple in our memory we don't know yet if it's actually existing and then the player is asking the agent to take an apple so our look for action we find that the

best object is this factual object the spectral apple but we don't know yet if it's actually existing in the world so we are going to try to ground it so another goal will take over is going to be the ground flag goal and the goal will be to go on the table and just look around in this case there is an apple so we're just going to replace our factual apple with this apple so when they take apple action will take over we'll find the correct apple and the agent will be happy about it so what if there is no apple on the table well the ground factor and still gold still happen we still are still going to go on the table but we are going to find no apples so we just remove the practical object and then when the look for action restarts we find just nothing so the agent is not pleased about that so he's quite angry about it we lied to

him so what did we achieve so far i think we bought more natural interactions through at first the speed recognition pipeline but also we created a more aware and expensive and lively agent thanks to different emotional reactions the moods but also the and this agent has have great variations thanks to the personality module and this agent also environment aware they can like dislike and react appropriately with this knowledge they can for instance refuse to do an action that is involving an object that they are hating and the npcs can also react to truths and lies regarding the environment and what can we do from here well the cool thing would be to develop a relationship between the player and the agent for instance if the player is polite maybe it's going to be okay but if we're

hungry towards the agent maybe he should be away or be be scared this kind of thing we could also have multiple agents especially in vr it's a big world but we are just alone with this little creature it will be even better with a lot of different agents that interact with each other through emotions but also with the player and also this is from play testing we're working with speech recognition so it's important to have a lot of diverse feedback from our ai agent the most easy easiest one is certainly i did not understand your speech but something such as okay i understood but i don't have the ability to execute your order this is some kind of feedback that you need to actually create so that the user understands what's happening so i think that's it today for me and we have two minutes four questions three two three

yes in one of the videos the little character um it um when you told it to go get something it turned and knew exactly what you were talking about yes but then when you lied to it it also turned into exactly what you where to go so what's confusing it seems that it it knows that there are things that are there in the first case but when you lied to it why wouldn't it know that there is not an apple there and i know that's a double negative i think i did not understand the first example so it when you first spoke to the to the creature yes it knew that what it behaved as if it knew everything that was around but when you lied to it it it behaved as if you were surprised that you didn't know that seems contradictory so if at the beginning

you're speaking about the factual statement where at first i am teaching the ai life for instance there is an apple in front of the rockets and then i am like they don't actually check and there is nothing so it's quite angry so this little difference right well my mic there is that but i just it seems that at the beginning because it didn't discover with its eyes what was in its world right it knew what was in its world but when you lied to it why didn't it know what was not in its world because they cannot see everything we have a perception model here so what you can see will be inside this memory but maybe i am able to see something that is in front of the rocket that he was not able to see and in this case that's why he was still checking and then be surprised that actually lied to him so there's a camera view so it sees that that it's different from yours so yeah the agent has a perception module okay so just a cone or something in

front of him and just adds this knowledge to its memory it's not what the player is seeing it's what he is seeing himself okay okay thank you sorry okay if you have other questions we can go to the practice room i don't know which one it is and just discuss after no problem thank you