Using AI to Create Interactive Digital Actors

[Music] welcome to our presentation on using ai to create digital avatars for ar and vr my name is kevin he i started my career in gaming 10 years ago at blizzard entertainment as engine developer world warcraft after that i took technical roles in roblox at disney ever since i enjoyed working on entertainment technology so five years ago we started this company called dmotion we focus on working ai for animation today i'm going to introduce to you the technologies we have been working on and what we can apply ai and simulation to interactive avatars so my talk will be divided into four modules the first one on simulated motion that's

our first phase engine to to use physics simulation to create the characters the second one is our motion intelligence that is an extension of our simulating motion to provide you more diversity in the character movement the third one is about motion perception that will add additional ability to the ai to understand how people how animals move around us and apply that knowledge to enhance our motion brain and in the last i'll give a quick overview of the pipeline we're working on to help you to create interactive content for air and ai and the vr so let's start on the first module which is simulative motion first first i want to talk about what's the difference between keyframe animation and the simulating motion as we all know keyframe is the standard tag for film and for game entertainment but

in general keyframe animation they are static every time you play they behave the same so that's repetitive and keyframe takes a lot of hand crafting to create is also not reactive to user input and user influence which is the quality you are looking for in ir and vr applications on the right hand side the simulate motion we are looking at physics simulated movement that's procedural every time you run a simulation it generates a unique experience it's interactive it's reactive when you engage the ar vr characters created of simulation you can see the immediate feedback how the animals how the character react to your input so that's something we found very interesting for that reason we did a bunch of experiments in simulated motion this this is

the following slides we are going through the path we went through to achieve simulated motion for digital avatars in order to have a fully physic physics similar character we first work on building a very robust physics engine for joint for motors as you can see in these slides most of these demos they are cars and vehicles machines right so we started from here we make a really robust multi-joint physics simulation of machines so that will lead us to the later stage where we can add more intelligence on top of the machine then the step two we went through is to apply apply the physics simulation on human so first we need to create a physics-based character model so this model is basically created with multiple rigid body representation and all the rigid bodies are connected

with joints just just to simulate the biomechanical model of human with such a physical model of human the next step is to apply our physics simulation on this uh skeletal model uh then want to see what what you get so the result is like ragdoll as we know very well in gaming it's a passive animation effect to simulate the character falling to the ground so we do have some physics simulation we have a ragdoll character it's modeled with physics but not so interesting yet then step four we need to apply muscle torque we need to give the rectal some power to activate the rectal so it can stand up so it can maintain its own balance so in this step we basically apply a very simple muscle muscle model at the joints of the skeleton

for those those of you who are interested in the details we apply a pd controller on every joint to drive imagine each joint is a is like a motor in a crane right so the pd controller will drive every joint of the virtual human being to match the target trajectory you want you want the robot to achieve so with this very simple pt based muscular model we we now can can power the rug doll so the ragdoll will tighten up his muscle and you know we do feed the simulation so you get someone who can hold his posture but still not standing there the remaining problem is balancing so in the the fifth step of our simulating motion we want to add a balancing to the digital avatar and it's not actually it's not very hard if we study the human

balance theory when when our center of mass is under gravity when the gravity pulls us forward we will flip forward the main the problem is the projection center of mass is go outside of your support polygon as shown by the red the red square on the ground so all you need to do is just tighten the muscle across your skeleton properly so you can generate the ground reaction force as shown by the green arrows the aggregate effect of the ground reaction force will push your body back to a balanced posture so after implementing this basic robotic balancing theory we can get this we can get a digital robot standing in place performing some simple actions such as rotating his waist do a kick or just do squatting without falling to the ground

so you can you can think of of this like a virtual world analogy of boston dynamics robot in the physical world basically this phase we employed robotics balancing algorithm into our digital agent we can get a standing in place balanced agent and the final step we want to add a local motion to this digital agent so it's not that hard but it's basically we can balance in space we just need to leave one foot at a time and alternate between the two feet and then you want to give it some forward push you know implement a food plan planting logic to place your foot strategically as you move forward by doing that we implemented a very simple uh local motion controller that can power the digital character physically and walk around while not falling to the ground

so so far we implemented a physically simulated character that can do self-balancing and walking now the question is what can we do with this digital agent so i want to show you a few applications in vr which is our main interest area of today the first application of similar motion to vr is a full body motion tracking before going to the demos i want to talk a little bit why do we need full body avatar in vr right we all saw that facebook released this awesome facebook horizon platform for social vr but as you can see in social vr scenario if you only see the half body the upper torso of your friends or fellow that's not as believable as you hope it to be so in multiplayer or social vr setup you do want to see the full body

presence of your friends you know of your your team teammates in the virtual space however a commercial vr device only offers you three points of tracking basically you have one six star tracker in your hmd device you'll typically have two other six star for three dot trackers on the hand controllers you're holding so in short you usually have three point tracking you can utilize that to model the upper body movement another key question is how do you generate or synthesize the lower body movement so together you get a full body representation in virtual space and that's the first application we are trying to apply simulation to and we we we did uh receive questions from users why don't we just use full body mocap equipment we know we can do that right we can apply the you know traditional um

optical based motion capture or inertia-based motion capture there are a lot of options but in general to apply a professional motion capture equipment to achieve full-body tracking that's expensive you have to set up a studio to do that you need to purchase expensive hardware and software to do that the post processing is quite complicated it takes a lot of time and also there's no single standard of motion capture so it's very tailored to project from product to project uh you know in short you cannot really achieve it with consumer level vr device right you it doesn't provide the enough sensors all over your body to do full body motion capture so let's come back to how can we solve the problem in the context of consumer level vr hardware let's say you only have three tracking points or four or two it's not so much different the

traditional technique to solve vr three-point tracking is use ik so ik is a shorthand for inverse kinematic it basically it's basically a mathematical solver to solve partial body of full body motion based on where your your hands are where your head is it's a inverse algorithm but ik is only looking at geometric constraint of human body it doesn't really look at the physics dynamics like mass properties newton laws so with only a geometric relationship considered ik often have ambiguous solution you have to tune ik to make it work right we often have you know flip elbow a lot of complex problem using ik and also ik only solves the upper body it still cannot predict where the lower body are so a standard technique to add lower body is use keyframe you can just keyframe a few fixed locomotion pattern

and slap together with a upper body ik driven aperture and then it it works in some scenario but it's hard to match the lower body and the upper body because keyframe based lower body are just fixed animation right you often have food gliding problem self penetration issue it's sometimes really weird because it's just keyframe it's just fixed uh solution so although the traditional technique of ik plus keyframe is um is a solution for full body after tracking but we think there can be a better way to do it using inverse dynamic use physics simulation so in this example we are um we're experimenting how can we use a simulated character to achieve frequent tracking basically all you need to do is to tell the physics system where the hands are where is the head and let the physics to figure out you know the body's motion state

uh the lower body's posture in order to maintain the full body balance so we want to maximize maximize the the knowledge in physics equation to give us as much prior as possible to drive full body movement movement so with such a simulation driven character we can we can generate uh full body motion more naturally and also because it's simulation right so it can automatically interact with the environment if you uh you know if you push the character it will react to it if the character has run into a wall it will react to it so it's more interactive to environment so that's how we're doing simulate motion for three point so in this slide we are showing two flavor of that three point tracking using simulation the left hand side using oculus quest as the testing hardware the right right hand side is htc vive

there are there they're not essential difference but you know it's just different setting in general um we found the simulated motion we can run a mobile or world tethered environment pretty smoothly as you can see even with a three tracking point we can we can synthesize some reasonable lower body movement as the character is leaning we're trying to predict the torso leaning from the head angle and hence positions so with such technology we can we can create a reasonable full body representation of yourself in virtual space so with such a virtual body you can engage multi-player experience better you can socialize with friends in vr more intuitively and uh yeah another benefit of that is you can

also interact with the environment more easily right like like the we can see the right hand the right hand side the wife demo uh the actor director is moving around and she can sit on the ground right because there's collision with the ground so the seating motion generates automatically in this case she's sitting on a virtual box and you know the collision will just generate the the lag movement you have the leg need to rest on the box so that's three point tracking uh the next demo i'm gonna show a different flavor of multiple tracking this is four point tracking there is additional fourth tracker on the back of the of the uh actor so um

the the tracker on the back with track rotation only you know in other words you could use a cheap inertial sensor for the fourth tracker and just attack attach it on the performer's back it should still work why do we do the full point tracking is because you know even super tracking can track some basic leaning but the fidelity of the torso curvature is still not ideal with additional point as you can see the character can do the leaning you know in a much more realistic fashion for some shooter games for some location-based vr that need like the the like the shooting demo on the right-hand side the shooter really need to hide behind an obstacle and leaning out his body in order to do the um in order to do the do the action now we can do it with four point tracking so still the four points are pretty lightweight it only needs upper body standard vr plus additional

sensor and with this setup we can get a a pretty good uh full body representation like the left hand side uh the performer is doing some dancing kind of movement it can track the uh the hip and the torso reasonably and if you look at the lower body the lower body's leg movement it's not one to one mapped to the real lack of movement but because we employed the robotics balancing theory as i illustrated early when the center for mass of the actor moves outside the supporting area of your feet the ai nose automatically swing the foot right to the right direction so it roughly match right it's you know in most of time reasonable and in a summary here just want to recap what's the difference

between three-point tracking and four-point tracking as we can see a three-point tracking on the left it tracks a basic basic torso leaning but you need to lean very clearly for the vr to pick up on the right hand side with the first tracker on on the back you know we can travel more details in the torso you can do a little bit curvature movement you know uh in the back we can still track it so there's such quality uh improvement as you put more tracker into the system but i also want to stress uh this shows the flexibility of a physics-based tracking system you can just add you know if you only have one tracker you can put it on the head we'll figure out the hands and everything else it has two tracker or two tracker you put it you'll put it on more you put them on more body parts and we'll track more detail and we can incrementally increase the tracking quality by adding more sensors but the system can synthesize everything else

and the last demo just want to show the same technology you can apply in a two-player environment so because both avatars are physics simulated they can collide with each other uh you know you can sort of touch each other in virtual space we can do some dancing or fighting you know or people hugging so that sort of very intimate social experience will be enabled by this physics simulated avatar tracking technology we've seen a lot of demos about tracking the aperture which is your own motion but what about what about npc non-player characters in this case we also use a simulated motion to simulate the dog movement the dog is a very simple one he's just trying to do a balancing okay just trying to stay on his feet and remain balanced but here the player is using his uh his virtual hand being tracked by lip motion to engage with the dog even though we didn't add any keyframe animation to the dog but just because the the ai dog trying

to balance in place you you'll see a lot of richer experience being rendered being generated in real time as a player trying to touch the dog trying to pinch or or doing some you know random movement so this shows the potential of simulated motion in creating non-player character as well i think by combining avatar which is a player character and a simulated non-player character we can build a we can build a full procedural world that's a lot more interactive because the motion itself now is generated by simulation and ai is that just you know keyframe can experience so we talked about simulating motion as the first phase of the technology and the second phase we experimented to upgrade

seamless to the motion intelligence phase and what is motion intelligence i like to i like to quote a definition from deep mind block so the agility and flexibility of a monkey swing through the trees or a football player dodging opponents and scoring goal can be breathtaking so how to master the ai to coordinate your full body muscle structure to achieve this kind of motor maneuver that's the hallmark of motion intelligence that's very exciting area in ai and that's the that's the kind of technology we are moving towards as well to its core we just want to build a digital cerebellum to do balancing right to to do motor skills like a little baby learn from observing for those of you who are familiar with reinforced learning the setup fundamentally is a reinforced learning you have a digital agent

it will it will take observation from the virtual environment in this case that's the vr or ar environment it will send instructions you know to the agent and the generator reward by collecting reward or penalty uh from thousands thousands of iterations experiments you learn from your failures you learn from your success and that's actually that's very familiar to to us because that's how humans learn our experience as well applying that to a digital world build build a deep reinforced learning pipeline you can actually teach the digital agent to learn from you know to learn learn to walk learn to stand as well so this example you know this is the input to the machine learning algorithm it's it's just running right one seconds of running animation so this is uh the reinforced learning result after 100 iterations basically you do 100 times experiments

every time you try to tighten your muscle in in a way you can mimic the input motion of course it will fall you know you will run into the wall a lot of bizarre scenarios but when you collect the feedback use the feedback to do gradient descent to correct your you know your motion you can eventually get something that's kind of working but if you keep doing this after 1000 iteration you will generate the ai that can do running running while keeping balance pretty smoothly and because of stimulation it can also endure your disturbance as the digital agent running you can use a mouse tracker to drag him around you can apply a virtual force he will just react right so that's the beauty of simulating motion and then in the next weekend naturally will will teach multiple behavior to that digital brain in this case there are three behaviors like running sharp turning and the github by combining three behavior to the guy

now you have a agent a little bit more uh smart it can it can turn now uh and if you push if you pull him to the ground he can become him himself this whole thing will be um will be generated by a simulation it's not keyframe so every time you you play it in vr it could be different it's just then you can keep a building on you can keep expanding the motion brain you know to have six behavior eight behavior in this case that's a 6 behavior agent it can do basic indoor mobility maneuver such as climbing ups there such as you know sitting in a sofa yada yada as you can see we can keep building that and this one we want to add more upper body variants to the mpc so we add 20 behavior to a pedestrian motion brain and in this case the guys are you know working while making a call reading a notes so just

add a lot of diversity to the scenario so we could use this to simulate um a crowd you know crowder simulation or pedestrian case we can actually use this kind of motion brain in non-gaming environment such as if you making autopilot car if you make a training program uh involving full-body physical physical intelligence we can use simulation and ai to help this um this example shows a more sophisticated motion brain model of basketball player so each each demonstration in this slide has three behaviors right three three styles of passing as you can see the ai can stitch them together and based on high level commands like where where where to move towards ai will automatically pick the right the right behavior to transition into and blend

them together and who says the technology can only apply to human actually the exact same methodology physics simulated character and driven by ai this you can use on animals as well actually quadruped is easier to do than human because quarter pad has four legs it hardly can fall to the ground so you can apply to animal you know different form factors of creatures and we talk about the motion intelligence so now i want to just show a few examples how we can apply motion intelligence to create ar character so the you know the left-hand side are just some just a boy character spun on a kitchen top in ar and you can mess around with the boy and see you know it it it's done it's dancing it's being pushed off the right hand side is a is a ar dog you can spawn the dog on a

carpet and throw different toys as a dog and do some kind of engagement interaction this is the you're probably familiar with this video already because we've shown similar uh six behavior characters right in earlier slides now if we apply the exact same brain in ar application you can create a little uh indoor mobility demo with a house right this is like a two-story house with staircase uh you can dynamically change the digital environment and that character will respond to it it will try to climb up you know you if you change where the staircase is the character may fall to the ground may succeed just like in real world so um basically everything we did in simulated motion and emotion intelligence can be automatically applied to ar so that you can create a highly interactive character you can create highly

interactive environment we think this level of interaction is necessary to bring the ar world to life right you can experience it you can really um you can really feel the presence of this digital being and then the next module i want to talk about motion perception we talked about the motion intelligence very cool to synthesize stimulating motion but what about the what about the vision what about the input path for for our motion brain right as a human we can understand the external world very well by observing how people move around um so the this this last piece we are extending the motion intelligence to cover the perception path basically um we can do the following five step in order to achieve some basic motion perception the first

um as we know you can apply some post-estimation algorithm such as the open pose from cmu welcome to check it out this open source project very popular you can use an algorithm like open post to extract your drawing position of human from just an image from just camera feed camera input right and the second step you can retarget this skeletal drawing position retarget that the posture to a digital avatar to a digital model and the third step you can apply um physics simulation on that character to give it a physical physicality give it collision give it the weight and mass all these nice things and then your avatar become more alive it's not only tracking the inpromotion it can also collide with the environment and also by applying this simulation technique you can make the tracking quality higher because we can automatically fix some

artifact like a self penetration right like a joint limits violation things like that so by adding this motion perception to your pipeline you can further improve the pipeline's capability to model human motion to capture human motion and convert them to a digital character in real time or offline so yeah this last piece also demonstrate just from single rgb camera what you can what you can track you know from a 2d video or from a camera input and in the last slide i just want to quickly talk about the pipeline so we we talked about a lot how to create simulated character in ar vr but exactly what's a workflow for artists and content creator to achieve this effect so here we want to summarize this

workflow at high level the first you can um you can upload or you know ingest your own fbx file your 3dmax or maya digital assets into this ai system whatever it is and the second you want to add a physics physics layer on top of the artistic model by that i mean you will add the collider the mass the weight the join the muscle strength all those physics and controller parameters to the artistic model after this second step and the third step you need to decide what are the set of behavior i want my motion brain to to master and you you will pick those subsets of behaviors and if you can have reference motion provided to the ai that's good if you do not have reference motion you just need to describe it procedurally or mathematically you know some in some fashion

after you select your behavior and you can start a training process right the training process um you know if for very simple behavior you don't need to run machine learning for example as i showed the earlier styles for the standing in balancing behavior that's a very simple heuristic if center for mass is moved outside your supporting area of your feet then you know try to tighten the muscle to to move the center fast back to move the center back that kind of that kind of simple heuristic you can you can just write it uh from you know a few lines of script so however if the behavior is quite complicated like some of our more advanced demo for the basketball player you know dribbling the basketball without losing it without losing balance that's kind of hard to manually um generate fuel lines script to control the muscles things like that

for those who can run through a neural network just like that machine learning to do thousands of tries and errors figured out so this way or another you will train a control logic and then you download this control logic as an assets file this access file imported to your game engine you know your own environment and then finally you know during the runtime you need the physics engine to be paired up to the controller assets and you will get your controlled simulated character in place so that's that's a overview of how this pipeline could be constructed i mean at demotion we provide solutions for various motion intelligence tech as i described we create ar vr body tracking solution we also help our partners to train interactive characters and provide real-time motion perception and customize motion brain etc so if you

have any questions for the slides or if you need additional how about motion brain training or digital avatar creation feel free to reach out to us so yeah that concludes my presentation and i'm going to take questions thank you uh okay good go ahead hi so it's really impressive it's very impressive um so the thing though that strikes me is do you think there is a way that you can also include uh the next level which is personality is there any way at all that even if you

train 10 people 10 characters to walk is there any way you can think of to inject for the for the artist to inject different kinds of personalities in the different walks procedural as they may be yeah thank you so the question is is there any way to inject personality in in the trained ai such as 10 10 styles of walking how can you represent emotion or personality very good question so the answer is uh yes um although the motor brain i showed in the demo are for primary motor skill mostly like walking you know dancing but uh as when you train if you train like a 10 behavior that share the same pattern like walking for 10 different walking or running what you can do you can easily include a action label in that training so you can label that action you know you can label the mood as happy uh tired angry

with some reference motion to the ai so as you are training you're basically creating a latent space to tell the ai you know what kind of motion you consider happy what kind of sad and the ai will create a interplayed intermediate result representing the level of happiness um so in short answer yeah you can define more parameters to train the ai to hopefully demonstrate some personality thank you i i thought the tech looked amazing it reminded me very much of natural motions euphoria sdk back in the day and but my understanding around that sdk was it was incredibly complex to add new behaviors and and how how easy is it to kind of train within you know how are you finding how are developers finding using your tech i guess because i think that was almost the reason why euphoria didn't become bigger was it became too problematic for developers to use how are you finding that great yeah

thank you the question is um natural motion euphoria produce very impressive physics-based animation toolkit in the past but it's hard to use and how the motion or how the new solution can can help in that space so we were also inspired by natural motion so it's good technology but really the difficulty in deployment is how much time to create a new behavior for example using the last generation tech without deep learning without this new simulation ai it probably takes a month to add a new let's say dancing movement to your physics based character because it's not just dancing right you need to think about the muscle pulling sequence the balancing requirement all the physical constraint the mathematics behind that can be daunting so i think the new tech is exactly trying to address our concern because the beauty of deep learning and

ai is that you can learn from data so we just need to throw more reference motions from motion capture from high quality keyframe to the trainer and the ai will do that thousands of thousands of iteration and figure out some you know some very pretty complex neural nets that represent some implicit math you know to control that so i'll give you an example now in order to add a new running or dancing sequence to our motion brain it probably takes an hour for simple case uh for very complicated yeah it could take days but still it's like machine time days it's not like our mechanical engineers spend months to tune every motor control thank you thank you sure thank you any more questions okay thank you so if you like the talk welcome to give us a good rating in the

review thanks