Full-Body Animation Generation for Expressive NPCs

[Music] thank you hello everyone I'm euden I work at netease fushi AI lab I'm so happy to introduce our work on generating for body animation for talking NBC firstly let me introduce what the program means first let's look into the existing methods and their limitations traditionally animation trajectories are created manually by experienced artists a 10 second animation often takes two days or even one week recently with the development of motion capture techniques making a 10 second animation takes half a day including

data capturing and correction both manually creation and motion capture are time consuming hiring experienced artists and building mock-up systems are both quite expensive moreover they created animations can be only applied to planned scenarios traditionally we often reuse the 100 craft or motion capture data by concatenating the existing animation Clips these methods result in a repetitive animations and the last variability of animations another problem is how to select a fitting integration function to bridge two animation clicks the integration function May perform well on a few of Animation Clips but fear of others

moreover the interpretation function damages and misses the subtle animations observing these existing method and their limitations we think whether it is possible to automatically generate the full body animations foreign specifically our work attempts to develop an animation generator for NPCs who who are talking our aim is to reduce the time consuming of making animations we make efforts on creating high quality animations the high quality animation should be natural lifelike and human-like they are able to convey appropriate emotion and intention there are variants instead of repetitive patterns in our current work as 15 second facial

expression is generated in less than 6 milliseconds a piece of 20 second body language is made within 600 milliseconds our work has has been applied to several games developed by net is let's think about what happens when we are talking what happens to our compartment yeah indeed amount of academic researchers have reported that our behaviors can emphasize ideas signal syntactic boundaries and stress complement verbal information and also express emotions and intention on the other hand while speaking human facial and body behaviors are closely related to simultaneous speech or spoken content this inspires us to think about

synthesizing animations according to the input speech our solution relies on neural networks to perform the mapping from speech to simultaneous animations the new the neural networks learn and capture the temporal relationships between human behaviors and speech in a stage of computing animations the neural network takes the speech as input and synthesizes the animations while while rendering the learned the Learned temporal relationships into the generated animations as no training neural network requires amount of data to obtain adequate data we use an optical motion capture systems vehicle to capture body movements

and use Dynamic XYZ to capture facial expressions in our work deep and jaw movement are represented with 28 dimensional vectors at each frame eyelids and eyebrows together with 22 dimensional vector and 76 dimensional body movement the motion capture process ensures the high quality data to better display data we manually make reduction to reduce the effect of a skeleton and phase gap between actors and the targeting MVC the training data consists of amount of pairs of simultaneous speech facial expressions and body language each pair refers to an utterance let's make tea represent the audio

length time frame the low facial expression is represented with a 28 dimensional animation sequence lasting T time frames the upper facial expression is represented with 23 dimensional animation sequence lasting T time frames the body is a 76 animation sequence lasting T time frames let's talk about a lower facial expression first involving lips and jaw I think some of you may be familiar with this topic reviewing most of the published work speech features are often used as input to synthesize synthesized mouth animations of course speech features contain flow information on the other hand speech features keep some

information that is not related to animations for example speak specific Timbre is included in speech it cannot reflect any information related to animations if speech features are Disturbed with speaker specific Timbre this speech features damaging the quality of synthesized animation of course we may ask questions that we can use speech features which have left out speaker spec speak specific temper foreign we are also making efforts on refined speech features disentangled from speaker specific Timbre okay now let's move to a useful solution instead of using speech features we propose to use text information for

mouth anime Movement we propose to use the attend content like phonemes volumes are absolutely non-related to speaker specific information it is very easy to extract extract to the volume information with the speech recognition tools from speech from a human speech if using syntactic speech volumes can be obtained directly from the TDS [Music] once we points from a speech we have Atlantis phonemes according to layer dilation specifically we repeat the phoneme label according to each phoneme duration like this we can get time aligned for new sequence to facilitate Computing we set this volume sequence at the frequency as same as motion sequence

this slide demonstrates demonstrates a example oh yeah you don't know to tell me where she is then according to the audio ASR helps us recognizing time timeline for new sequence here as observed the phonemes are repeated according to the lasting time of phonemes as mentioned neural networks are used to capture temporal relationships between time aligned volumes and mouse animations intuitively we take into account aspm that is typical neural network modeling temporal relationships in fact amount of

researchers rely rely on STS STM to process temporal relationships and have successful experience experiments on the other hand STM is recurrent model it computers the output sequence frame by frame the next frame output depends on the output of it is previous frames the latest time consume time consuming therefore we attempt to use another neural network structure CNN convolutional neural network a cam performs the calculation with the parallel computing in our experiments for 15 second sequence why STM takes 500 to 800 milliseconds a CNN takes 20 to 60 milliseconds only

and this slide shows our neural network frame framework your network framework this input is time aligned volume sequence timeline we repeat the phonemes according to the lasting time of each funny and our work will ask all 12 layers each layer is electrical layers foreign specifically T is is 30 frames in our work the training data consists of 865 atlases each lasting about 5 to 15 seconds

foreign I show you a few exam results by comparing syntactic and motion capture lips in total I show you four comparisons in each comparison the left one the left one is animated with the syntactic data while the right one with mock update okay I will show you the first one foreign movement

foreign okay so you can observe the both animations are so closely to each other and we we extend the Lip lip motion framework to generate the photo with realistic mouth movement we can find that a stiff code to distinguish the syntactic and ground choose talking

Mouse okay that's enough let's have a look random tell me the gun was loaded he's my friend of my luck I've never killed someone 's again why didn't Randall tell me the gun was loaded he's my friend of my luck I could have killed someone and now let's move to Upper facial expression involving eyelids and eye blows similar to lip expression generation I repeat two trickies using time aligned spoken words instead of using speech features I emphasize here different from lower facial expression using timeline volumes the upper facial expression generation uses time aligned words

this slide shows the neural network framework for Apple facial expression the input is a Time aligned word set sequence the output is pointing three-dimensional animation sequence which lasts keyframes this Frame is a typical encode decoder the encoder is colored with yellow a decoder with green the encoder extract meaningful information from text into a latent representation and then the decoder translate this latent representation into the output animation sequence the training data the training relies on the 15 minutes data

and here I repeat the above comparison results and this time let's focus on the upper facial expression foreign

are also close to each other and here I show you a clip from our game Justice foreign is automatically generated from the above mentioned framework [Music] okay and here

and this clip show you the example from another game named revelation is okay certainly let's talk about body language generation and for body language we use mmcc the solution relies on a common used neural network unit the output is a 76 dimensional animation sequence which lasts keyframes the use of unit benefits of from achieving large receptive field with little computational cost and better capturing multi-scale features the training relies on the 10 hours data

and here I show you a few examples of generated animations with different emotions foreign [Music] animations by with a different style with different styles of NBC

the first one foreign languages are generated from generated with the mentioned above framework foreign tutor that is used in a popular online application of learning programming it is called code combat

um [Music] [Music] foreign syntactic speech our flame General our framework generates animations let's take a look

foreign foreign demonstrates that AI helps animation generation for instance

a 10 second animation is automatically generated lessening 60 milliseconds instead instead of manual creation with a few days so a while speaking a while speaking it it exists correlations between human behaviors and speech and human behaviors are temporarily synchronized with prosthetic and syntactic structure of speech the difference techniques can build correlations between human behaviors and the simultaneous speech neural networks are powerful for the automatic generation of body animations according to the input speech speech signals or we say the

speech based information with the neural networks the phase of body animation generation requires the time consuming in next name 660 milliseconds for for facial expression and 600 milliseconds for body animations so about the limitations there is a still a long way to go in making the generated animations more explained expressive there will always be a need to retarget the output animations to NPC with different skeletons the Frameworks require high quality motion capture data the styles of generates animation depends on the recorded data set about future work

and how to if we would like to improve the quality of Animation we have two we have to take into both the text and audio information for making making the expression and the behaviors more believable and more human-like but that is it is a challenge how to how to model how to model the interaction between the text and audio information in order to in order to generate high quality animation and in the future work we will take in into account inherent inherent correlation among the three three mortality animations including the upper facial expression low facial expression and body language

and we will make more facial expression Mega facial expression and the body language with more semantic informations and thank you this is my talk [Music]