The Facial Animation Pipeline of 'Marvel's Guardians of the Galaxy'

foreign [Music] welcome to my talk on the facial animation pipeline for Marvel's Guardians of the Galaxy I'm Simon Hadid a lead technical animator at idos Montreal a bit about me I started the first 10 years of my career as a rigging artist which gave me a solid foundation for Tool development and fertility development human anatomy Automation and improving the user experience in more recent years I dedicated my time and attention to my true passion which is facial animation since action adventure narrative games are my favorite genre to play working on this project was a perfect match and allowed me to create genuine and emotional performances based on the incredible feedback that our game has received I'm proud to say that our characters have resonated with players

to kick things off I'd like to present you this 30 second clip from one of our cinematics showing different stages of production throughout this presentation we'll see how this scene was created did you say you have all of this energy Faith energy right but when we first saw you weren't you stranded on Alice hope because your shuttle ran out of juice yes I was but so why don't you just believe that your ship had more gas it doesn't work that way Faith energy is a byproduct of belief focused belief [Music] okay sure I saw this proven to kick things off oh sorry here are the topics we'll be covering in this presentation we'll Begin by talking about the vision our goals and expectations at the start of the project next we'll talk about the photogrammetry scan this is the process in which we

take images of a model in order to get anatomically correct characters in the third section we'll explore the three types of performance capture that we used in our game in the fourth section we'll talk about how we processed all the data from these capture sessions and converted them into animations in the fifth section we'll talk about the Cinematic polish and how we brought these animations to final quality and finally if we have time we'll end with a q a so I'd invite you to write down questions in a notepad or in the chat and I'll do my best to answer them at the end if we run out of time feel free to reach out to me on Twitter or LinkedIn and I'd be glad to chat about any of this I I want to mention that this is a facial animation-centric talk so I won't be covering everything about the Cinematic pipeline or facial rigging it's really about how we capture our actors and how we animate our characters I may get into some technical Concepts but it's still meant to be an overview of the pipeline

so the vision what did we set out to do when we started this project we knew that the Guardians of the Galaxy was going to be a performance driven narrative game sits into the Guardians we knew that there would be an ensemble cast but five characters and that they'd be together for most of the game we knew that these characters were going to be stylized to look more like their comic book counterparts but they had to be very believable we wanted a lot of emotion to come through these characters we also knew that they'd be traveling to different planets so there would be non-human characters as well we wanted to push what is done in our industry we wanted to look at different technical innovations that we could incorporate into our game one of those is photogrammetry typically on games we would have a team of character artists sculpting and using reference images to create their characters whereas with photogrammetry we get the Precision of scanning an actual person as the basis of our characters we'll talk a bit more about that in the next section in terms of audio versus video based animations a lot of games use audio

tracks to generate facial animations and while that may give very accurate lip sync it kind of has to approximate what to do with the rest of the face which is usually procedurally generated using emotion tags when working with video based animations and actors performance is our source and is directly translated onto our characters so we get more believable performances and subtlety we were fortunate to have our own in-house mocap stage and we were planning to use it extensively on this project our cast was comprised primarily of local actors which meant the turnaround from booking our sessions capturing all our data and then generating usable animations was very very short especially if you were to compare that with working with an external mocap Studio let's talk about the advantages of full performance capture many game developers tend to record their voice separate from their body mocap and facial capture and then Stitch all that data together for their cinematics which doesn't always guarantee that they'll all line up and give a cohesive performance in contrast when we're when we're

working with our actors their voices their faces and their bodies are all recorded at the same time which ensures that the performance is as accurate and natural as possible in video games we often hear that we have to either choose quantity or quality meaning we either have lots of animations or fewer at a much higher quality on this project we try to prove that it would be possible to offer quantity and quality by having multiple tiers of quality of facial animation bronze being mostly automated with little to no manual labor silver is the one that comes the closest to the actual performance but requires a manual polish pass and then gold is really that Final Touch at the end that is reserved for the most important shots the first topic we're going to look at is photogrammetry scanning this is the process of taking hundreds of images of a subject from many different angles to build a 3D model given that our game is stylized our

characters didn't need to be one-to-one with the performers so we had to make the decision do we want to start by casting a model and then designing a concept art based on that person or vice versa in our game we decided we were going to design the concept art first and then cast a model to best fit that character this photogrammetry room was designed by pixel light effects and assembled in our own Studio the room itself is composed of 40 DSLR cameras and five softbox lights which would all get triggered at the same time the lights ensured that we have a nice diffuse lighting with hardly any Shadows across the face in total we had 13 scan sessions so 13 different models that came in for different characters when each model would come in we would dot to their faces with roughly 100 facial facial markers using an eyeliner which allowed the character artists to accurately track the deformation and skin sliding in the booth we would ask the models to hold about 25 different facial expressions and these full face

expressions is what we use to separate into 138 blend shapes for our facial rings let's quickly go over the workflow these cameras would give us CR2 raw image files that we would pass into Lightroom classic for some color balance and light adjustments bringing these edited pictures into reality capture gave us a point cloud and generated very high-res meshes we were able to automate this process making it much quicker especially considering the number of expressions for each model that would come in then we would hand off these high-res meshes to the character artist team who would bring them into wrap three apply them onto our base mesh and then bring them into zbrush for cleanup stylization and final polish over here we have a preview in Maya of those blend shapes kind of blending in from one another uh the goal for this first test was to try to get our digital character as close to a perfect match with our real life model before the stylization phase the advantage of applying these blend

shapes to a base mesh is that we could mix and match different parts of multiple characters with one another in order to create to create brand new characters in the upper right corner we can see a sample of different Expressions taken from a single camera and beneath it we have an expression sheet of those expressions converted as blend shapes next we'll talk about performance capture from our in-house motion capture set our actors body motion facial performances and voices are recorded simultaneously there are three types of performance capture that we used on this project and we'll explore each of them in the bottom picture we can see the mocap volume that we used for all of our cinematics even though it's a relatively small volume in comparison to other Studios it was great for conversations some simple Locomotion and even some stunts at most we had seven actors recorded at the same time of which six were wearing head mounted cameras used for facial

performance capture which was the maximum that we actually had available on set uh the body mocap was recorded via optitrack and the helmet cams were provided by faceware in the picture on the right we can see our cast all geared up in their suits and helmet cameras but you may notice that their each actor has their own live microphone attached to the front of their helmets which allowed us to record their audio tracks individually and then combine them into a single file whenever we need it when capturing the body mocap for all of these actors with the facial footage with the microphones and three reference cameras placed around the room all of this data needed to be synced so we would use a common time code across all of them we also needed to to trigger the recording of all these devices at the same time and we also had to name all of our files consistently so we had an in-house solution called Lumiere from the bottom image we can see the monitors with the facial performance videos being streamed in real time

on a typical shoot we would have about five people supervising the session from this angle we had our producer our cinematic director our mocap specialist our audio engineer and myself the facial Tech um we also had a team who would help us Suit Up the actors whenever in the morning whenever they came in a lot of this was under normal conditions but of course our last year of production was not under normal conditions we had restrictive measures to follow and a strict protocol with social distancing masks visors hand sanitizers and all that but what was most interesting is that we had a camera recording the entire room that we could stream via Zoom so that certain people could actually monitor shoots from home and provide feedback through Slack we had to make do with the situation but we pulled it off and managed to complete all the recordings that we had left to do on this slide we have a closer look at the helmet cams that were used that we were using this is the Mark III model of Face rose head mounted camera or HMC for

short even though they recently released the mark IV model we were using the Mark III during production these cameras are recorded in RGB so color footage at 720p 60 frames per second beneath the camera there's a dimmable LED light which helps to reduce harsh background lights and minimize Shadows regarding the calibration we would draw Tiny dots across the actor's face 27 to be exact then match the same tracking landmarks in the software that we used for uh to track the facial footage on the monitors that we saw in the previous slide we could display a grid overlay to adjust the framing of each actor's helmet and to make sure that the face was nicely centered in that vertical video this was useful for ensuring consistency throughout the day and between shoots we would also adjust the lenses Focus to make sure our pixels were as sharp as possible for the tracking process uh it's also really important to mention the comfort of the actors fortunately these helmets are pretty lightweight compared to other models while the

camera and the lights are placed in front of the actor's face they're just low enough that it doesn't occlude their vision we always have to keep in mind that we don't want to hinder their performance over here we'll see the performance capture footage for the same cinematic that we saw earlier at the beginning of the presentation on the right we have the six HMC videos in the center we have the three reference camera videos and in the bottom we have the take name and the time code which allows us to sync all the media files including those individual audio tracks that are all combined to create a single single audio file for this video did you say you have all of this sorry about that it's energy Faith energy right but when we first saw you weren't you stranded on Hallows hope because your shuttle ran out of juice yes I was but so why don't you just believe that your ship had more gas it doesn't work that way

Faith energy is a byproduct of belief focused belief okay sure I saw this proven these tiled videos are assembled by an automation script using ffmpeg commands and are crucial for our editing process our cinematic director would use these to identify the selected takes and Stitch them together to create a first pass edit another type of performance capture is what we call banter sessions we knew that our Guardians of the Galaxy were going to be talking throughout the entire game so not just in cinematics but also throughout gameplay typically these types of recording sessions are done with a single actor in an audio Booth recording their lines one after another without necessarily having any castmates to interact with for our game we have the advantage of recording all of our actors together in one space which allowed them to maintain their chemistry and ensure that dialogues felt a lot more genuine and dynamic in total we recorded 23 000 lines of

dialogue and for every one of them the actors were wearing an HMC the videos of these performances were used to generate facial animations for our Locomotion our combat and in-game conversations by referring to the time code metadata embedded in our audio tracks we can identify the start time and the duration of each clip so that we could clip the corresponding HMC video even though we were tracking the entire face we were using primarily the lower part for the lip sync the full face data was passed on to a machine learning process that would give us an emotion track which we'll talk about a bit later in this video we have a clip from a banter session in this example the actors are standing still while they read their lines from a tablet typically when recording with a stationary or suspended microphone actors are required to direct their voice which limits their movements since our microphones are attached to their helmets the actors were free to move around as much as they wanted during the recording sessions without causing gain

issues or Distortion in the audio this Carnage brings back troubling memories of Worlds I destroyed always wondered what do you use to do that some kind of Mega bombs rocket blades my own hands and feet knees elbows often my forehead pretty good so good I chose this clip because it also shows a bit of the levity between among our performers between takes uh they were often laughing and cracking jokes um and so considering how long some of these sessions can be it was really nice to see that they were still enjoying the writing that was given to them and always maintaining that chemistry the third and last type of performance capture that we used is the emotion capture uh during these sessions we would ask the actors to perform a range of different emotions at varying degrees

of intensity the emotions that we had narrowed down were content angry nervous and sad the theory behind this is that when we speak with different emotions we have a lot of nonverbal gestures both as a speaker and as a listener with that in mind we paired the actors one-on-one and we had them both standing and seated which offered a wide range of different body gestures that we could add as a layer on top of locomotions as I mentioned earlier the facial performance recorded during the Venture session could be passed to a machine learning process to generate an emotion track and this would determine an actor's emotional state and intensity these values would fetch the corresponding animation from our library and drive the upper portion of the face as well as body gestures if you'd like to learn more about this I recommend that you check out tomorrow's session called emotion detection for expressive characters in Guardians of the Galaxy which is part of the machine learning Summit

in this video we have an example of in-game banters you'll notice that the lip sync matches the performance and the body has overlaid animation you'll also notice that there's automated directional Lookouts for the eyes The Head and the Torso so depending on who was talking they would look at each other keep in mind that this is auto-generated so we consider this a bronze performance it would still have three no one's throwing rocket yeah yeah I'm fine just leave it it's your problem my problem is Meatheads trying to hook me over a cliff you're overreacting I did not only because quill stopped you there should be no problem I'm watching you you could tap the in psychopath uh imagine that they could be walking or running through the environment and these animations would be layered on top

of locomotion but for the sake of this clip the characters are standing still I picked this clip also to demonstrate how our facial performance translates quite well onto a non-human character like rocket after we've captured all this data from these three types of performance capture sessions how do we convert them all into animation so batch processing it's the ability to run through thousands of videos automatically track actors facial performances and apply first pass animations to their respective character rigs to be able to batch the tracking of our performance videos and generate facial animations we have to train the software to understand how each actor's face moves slightly different from one another we often use the term profiles to describe The Binding of our real life actor and their respective digital character on the right we see two screenshots of faceware analyzer the official facial tracking software that we use as part of this training process we build a tracking model which is a collection of all the different

expressions and mouth shapes that we have throughout our timeline I wrote less is more in the sense that we want to identify our Peak Expressions also known as training frames avoids too many redundancies and let the software do the interpolation between them consistency is key in the sense that if we introduce too many contradictions between similar Expressions the software doesn't really know how to deal with them and that could result in shaky tracking after we finished our tracking model we repeat a similar process in Facebook retargeter which we were using in Maya to build out a pose library of corresponding facial expressions on our digital character that Library hold on a sec uh that Library can contain a lot of extremes but we also wanted to include subtle Expressions to really have a nice range we decided that we would keep a bit of asymmetry in order to get the timing from what the actors were doing in total we had facial profiles for 20 characters on our project

one way to efficiently build a Facebook profile is to record a facial ROM which is short for a range of motion in this video I've chosen different parts of a typical ROM and edited down to about a minute these videos usually take about five minutes to record we focus on different face groups so we'll start by isolating the eyes the brows and then the mouth and then we end with a mix of different emotions and lines to test the range of our profiles eyes looking forward now wide up down left right one wide rotation excuse left stupid right mm-hmm close to do less like a smartphone

the Human Torch was denied bank loan while the purpose of a facial ROM is mostly for building out our character profiles we found out that it had a lot of other benefits as well it served as a really good vocal warm-up for the actors at the beginning of the day it was good for adjusting audio levels and for testing the the recording trigger across our multiple devices including that embedded time code for those reasons we would do it every session even if we weren't necessarily using the footage to create new profiles some characters were actually able to benefit from the same one facial ROM that we had set up early in the project and we never really had to create another ROMs were still necessary whenever we had a new actor joining the cast so that we could create a profile for them and pair them to their digital character these are the tools that we used for batching but I'm noticing we're running a bit short on time so I listed here the different the the inputs that we normally provide to the batch in order to generate bronze animations if you

have any questions about this please send them to me and um and I'll do my best to answer you but otherwise we'll move on to the next one after batching all this data and generating our bronze animations we can move on to the Cinematic polish this is a technical and artistic polish pass on top of existing animations to obtain the highest Fidelity and most believable animations over here we have the team of motion artists in our team of four we were part of the Cinematic Department one of our team's responsibilities was handling the body mocap which included suiting up the actors when they would come in as well as setting up the actor's body profiles in motionbuilder then we would clean up the data and retarget retarget it onto their respective body rigs regarding faces our team was in charge of setting up the faceware profiles based on the ROMs that would allow us to batch process all those bronze animations for in-game and for cinematics

we also took care of these silver facial pass which I broke down here into two steps the technical pass which our goal was to try to match the performance as closely to what the actors did accuracy and capturing subtlety were really the priority here there's also tongue animation since Facebook doesn't provide tongue tracking this was keyframed for specific close-ups when needed then there's also what we call the artistic pass where we can actually go beyond what the performance performers did so we can emphasize certain Expressions we can hold expressions for comedic timing and it's also an opportunity to accentuate the asymmetry that the batch process gave us in this video we'll see a comparison between bronze and silver so on screen left we have a version that was automatically generated meaning no manual polish and on the right we have a silver quality facial animation which has both a technical and an artistic pass this was captured in Maya so not in our game engine which is why some of the shaders may appear a bit different

oh please proceed nah you go ahead no need I insist no I insist please proceed this pointless pageant of politeness plagues our progress which is why you should go first well I will Voyage Through the vexing vestibule over here is another example of what we did in the motion artist team we generally work in a standalone scene with just a single character and nobody animation and no set this was meant to give us the maximum frame rates and no distractions you may notice the colored sphere in the bottom left corner This was meant to help us identify the frames in which we have close-ups medium shots or if the character was off camera this allowed us to focus the Polish time on

the moments that really mattered uh this is actually this the facial animation scene for the same sequence that we saw at the very beginning so the conversation between Star-Lord and Ricker with all of this energy Faith energy right but when we first saw you weren't you stranded on Hallows hope because your shuttle ran out of juice yes I was but so why don't you just believe that your ship had more gas it doesn't work that way Faith energy is a byproduct of belief focused belief okay sure I saw this proven after the motion artists have completed their work with the body mocap data as well as the Silver facial animation we would hand this off to the Cinematic animators part of their job was to assemble all the scenes and bring the animations to final quality that includes the many different characters props Vehicles cinematic cameras and

environments they would also animate the hints for all of our characters since the that wasn't part of her performance capture aside from humanoid characters they would also animate animals and creatures that didn't have mocap data or facial animation in the images here we have the example of Cosmo the dog while his voice was provided by an actor we didn't capture an actual dog's face we did however capture the body motion of two dogs for several of his scenes another task was the head and look at adjustments since a lot of the characters don't have the same proportions as their characters the best example is rocket where his actor was almost the same height as star-lords but the character of Rocket is much shorter specific to character faces our cinematic animators would bring some of our facial animations to a gold quality level for specific emotional beats and for marketing shots it was up to them to give that final Beauty pass they also had the advantage of animating to the camera as opposed to the motion artists who worked in Standalone scenes even though we had those colored orbs to

know the proximity of the camera The Cinematic animators were able to polish the faces directly in the scene so we're almost at the end of the presentation I want to go over some takeaways from this production I believe we delivered on our promise of shipping with quality and quantity we had 13 scan sessions so 13 models that came in as the basis for our characters providing anatomically correct blend shapes we created 20 unique Facebook profiles that could be used for batch processing for the in-game dialogue including combat conversations and Locomotion we had 23 lines of dialogue almost all of those were bronze animations I didn't put 100 because some required some manual adjustments but we were very close and then for cinematics we had a bit over five hours of cinematic content which includes the multiple branching paths in our story the ratio between gold and silver was about 90 silver 10 gold quickly looking to the Future how can we improve all of this I look forward to improving the overall quality of the animations that we generate from our

performance capture sessions for example polishing them directly in the scenes with these cinematic cameras I also look forward to maybe higher realism but that's usually dictated by the art direction of the project it would be nice to see how far we can push our photogrammetry scans in combination with performance capture and then finally there's previewing our animations in engine with proper lighting and shaders to try to get a faster turnaround than just rendering in Maya uh to wrap up I would like to present you this 40 45 second clip of a gold sequence that incorporates almost everything that we saw in this presentation so enjoy tomorrow wait hold on a sec why some more children can be sacrificed in the name of Breakers up Church of course not just just what Peter if you just let me finish this on the temple ship we wouldn't be here they wouldn't be here I know raker brings back bad memories

my sister if I had just been better at protecting nebula Maybe she wouldn't be dead whoa what nebula's dead how by who tell me so I can find him and shake his critac in hand so it's kind of a heavy scene but I'll leave you with that and um I don't think we have a lot of time for Q a so if you have any questions please reach out to me I'd be more than happy to answer you can find me at simonize23 on most social platforms otherwise find me on LinkedIn and I'll try to organize maybe a post uh GDC talk hangout um I have a zoom link I'm not sure how I'm gonna manage to send it to everyone maybe put it in the chat so that's the end of my presentation thank you very much [Music] thank you