Case Study: Creating Realistic Facial Motion for 'The Quarry'

[Music] thank you just a couple housekeeping items um just make sure to silence your phones like a movie theater we got some imagery to show you that'll be pretty cool my name is Aruna I am a creative director and VFX supervisor for New Media and experiential projects at digital domain our teams work on content and creative that is outside the traditional 2D media deliverables of feature films episodics and commercials which means our focus is on real-time media and non-traditional asset deliverables such as modeling and rigging character animation and look development joining me today are my colleagues Peter our pipeline supervisor and Ricky our official rigging supervisor together we'll be talking about how we created realistic real-time facial emotion from over 30 hours of emotion capture performances and how he kept track of it

all I'll start with a quick production summary summary of the show and then Ricky will dive into the challenges of facial performances and Peter will follow its solutions to this challenge of rendering and managing it all so a core strength of digital domain is is the high standard of our work one of our aims is to push the boundaries of these cinematic visuals in games and part of this involves character animation specifically facial animation uh we can represent an actor's performance by mimicking the emotions captured from the motion capture stage and conveying that emotion onto a compelling character here are some film examples clockwise from the top left the Curious Case of Benjamin Button in 2008 which you want an Oscar for Ready Player one and Avengers Infinity war in 2018 and morbius in 2022 in the lower left corner there all these films started with the facial capture of the actor underneath and based on the character may have been animated to the performances you see in the film our facial technology called masquerade

was originally used to solve a character like Thanos here for reference his performance like this one here was about 29 minutes of Avengers Infinity War so a side note those 29 minutes are more than any of the other characters in that movie with the next up being Gamora so a fully CG character has the most minutes of any of the other characters this in-house facial capture system was rebuilt from the ground up as a masquerade 2.0 to bring high quality facial animation to episodics commercials and now games it produces high quality Fidelity facial animation at unprecedented speed and with less initial setup without limiting actors onset performances which you'll see this Freedom has helped us leverage this technology for real-time projects so since its release the Quarry has received numerous accolades and over the past year we've been on a number of top 10 lists and nominated and won some awards um next week Siobhan Williams who plays Laura in the game is up for bafta so we'll see how that goes next week a lot

of competition all of these character performers all of these character performances come from our actors the expressiveness and unique animation seen in our film projects comes with a handicap the time because of the complexities of the human face the facial shapes that drive our dialogue and emotions take time to render and this required a short test to validate facial performances around the time how many of you have played the Quarry or seen clips of that oh good amount great so there are some spools ahead if you haven't played the game so I'll try to show you these this what you'll see next is a brief performance that was captured in engine during the summer of 2019. this proof of concept with actor Grace zabriski from horror Fame like Twin Peaks um tested the compound compatibility of our pipeline as well as driving human facial performance in Unreal Engine 4. directly after that we'll be a quick behind the scenes featurette from onset capture in our process

hello my friend welcome to the show [Music] over the course of three years our team modeled rigged tracked and solved nearly 2 000 minutes of facial animation these 32 hours of facial performances drove 15 digital characters in supermassive games 2022 game release the Quarry these performances came from 42 days of vocab shooting using new machine learning techniques and leveraging our existing high quality facial pipeline we improved our character animation processes which allowed the team to complete this large amount of facial animation with minimal human intervention only 26 of the nearly 4 500 takes were Touched By An animator you know we can talk uh but then the best we can do is switch it over to the receiver and see if anyone broadcasts anything back which is unlikely but well I mean it is what it is I guess yeah

facial animations are delivered through our new optimized real-time facial solution called Chatterbox which allowed this high level of facial animation to be displayed on all current generation consoles at Xbox One Xbox SS PlayStation 4 PlayStation 5 and PCs you think this is over how the hell am I gonna get down so what now everyone's back to normal this cinematic interactive narrative project showcases 8 700 shots of highly emotional performances which are rendered interactively in real time at runtime so one of the driving goals behind the project was the high bar on cinematic narrative and actor performance these actors these characters are driven by the actor's unique facial features and

these emotions come through from their mocap performances it was why it was vital for will biles of the director of the Quarry to be able to give our actors direction that would be readable by the game audience and convey this emotion so what's involved in creating believable performances and how did we set up our pipeline to be able to process and solve over 30 hours of facial performance so the project was divided into three phases a character test which I just mentioned with Grace who played Eliza in the game the vertical slice the pro which was the prologue and the full mocap production the grace allies a seven minute test uh was to make sure that our pipelines work together and iron out any inefficiencies of data transfer and methodology of importing characters and animations into the engine based on this feedback we captured phase two the prologue across two days in October of 2019 and delivered around two hours of finished facial performance eight weeks later this prologue comprises of the first 30 minutes of gameplay when Max and and Lara meet

Travis Hackett in the woods it ends when they get to the camp and all hell breaks loose phase three began in January of 2020 and then this thing called covet happened after many discussions we pushed forward and attempted a remote shoot in 2020 in June of 2020 and this proved shoots weren't really ideal remote shoots weren't really ideal with all the supermassive team in the UK our state shoot in Los Angeles proved a disadvantage with the beginning of the mocap shoot starting around 4 P.M and ending in the late hours of the following morning around 1 or 2 A.M so you can also imagine trying to direct actors through Zoom which uh will did his best could based on what you see in there however we finally got everyone in person and on location with the Hackett family uh played by Lance Hendrickson Lynn Shea Ethan uh supplee David Arquette and Ted raymey in November of 2020 and then the full mocap shoot happened in January February of 2021

with some pickups in April and June due to actor conflicts so in order to pull this off we needed resources both of the human and machine kind there was a core group of people around 140 artists TDS and production crew that worked on the show and some of the biggest departments are listed here these 140 weren't on during the entire production run but they would spin up um as phase and phase in based on critical moments during the show there are a lot of frames processed as well almost in you know nearly a quarter billion frames processed for these 32 hours and this included rendering of all our QC and validation passes nearly you know a hundred thousand artist hours 53 combined years of people working on the show and while everything ran in real time on Console we still needed to solve and QC all these performances with close to 11 million processor hours so as I mentioned there wasn't a lot of Animation intervention but we did need

some our 11 animators covered a cumulative 30 days to fix items at machine learning couldn't figure out like mentioned in the previous video those 4 500 takes that we captured on stage eventually became the 8 700 shots in in the game so I'll pass this over to Ricky who will cover some of the facial paralysis we used and discovered thanks Aruna hey guys I'm Ricky Cloud still and I will be talking about the facial Technologies used on the Quarry so first up we have our head mounted markless tracking which is through our software bullshite handles this is going to best fit blend shapes to a depth map from the HMC next up there is the machine learning marker removal this is using our internal cyclegan and this is again that

we set up to remove markers from the plates um then next step from that we have our machine learning feature tracking which is leveraging our in-house feature tracker this is going to be for eyelids and Iris and this is called gazeml then we have our HMC marker tracking which is done using Bullseye markers and this is an automatic blob tracking for the HMC plates from there we need to have marker cleanup and this is going to be handled by masquerade cleanup for stabilization and GAP filling of marker data from there we need to go to our marker up press which is going to be used for inferring the high resolution deformation from the markers using masquerade upres from there we go into our direct drive which is a workflow for transferring of performances between meshes which for here this is going to be you know going from actor to character once we're on our character mesh we need to we'll leverage copycat which is our

in-house non-linear blend shape solver for our fax Network and then finally we have Chatterbox and this is our compressed representation of facts for real time and now to discuss how all of these Technologies work together and help to make the query possible this project made complete use of masquerade 2.0 our suite of tools currently used for offline capture to make our solving process as automatic as possible which had to happen due to the scale of the project we could not possibly go in and have to correct every shot in 30 hours of capture a large component of the optimization was in the area of tracking in both markers and features okay and how everything connects and this will start with the HMC taken on stage from there we can begin generating depth in normal renders from the plates these renders are used are to be used in Bullseye handles the cash from Bulldog the cash from

Bullseye handles is used to provide training data for our Bullseye markers software along with the depth renders there is also the generation of markerless plates using cyclegan these are passed to gaze ml for the eyelid and gaze track the feature track data gets converted into a similar format as the marker tracking to allow both of these to be ingested and to masquerade to generate a high resolution actor mesh the performance can then be transferred onto our character mesh with our direct drive process this is then solved to the facts blend shape using copycat allowing animation to pick it up and then from here we can go to the standard VFX pipeline but also now with Chatterbox our compressed representation effects we're able to run in real time in unreal

with the input to masquerade 2.0 starting with the onset HMC capture it makes it one of the most important steps corruption or poor capture at this point will result in poor solving results for the Quarry we use the technoprop stereo head mounted camera shooting at 2K 60fps in IR at the start of each shooting session there's a rigorous calibration process to help make certain we have the most robust camera model to produce the best results leveraging a grid to fill the frame in all angles to capture the most information possible for solving the intrinsics we also use a found object during the calibration process to assure the model is producing accurate results when setting the cameras up for the actor we want to have as much of the actor's face and focus as possible as well as making sure the face is framed in a way and both cameras to allow for the full range of expressions even when placing the markers on the face there's a back and forth with the makeup artist to make sure that the markers are on The Ridges of The Valleys of the fold to capture all that great

detail the first shoot that we have is a tech shoot and this is used generating this is used for generating the HMC actor-specific training data and preparing the asset for shop production the performances for the tech shoot generally consist of ROMs for the face and eyes as well as Visine performance given in a range of emotion the idea here is to try and cover as much as possible for multiple scenarios that could appear in production as soon as the data is brought online we can begin ingesting and processing the plates to step into our first piece of tooling Bullseye handles from the images we first extract rough scams from the head mounted cameras using standard photogrammetry techniques to render depth and normal Maps out as aovs which are arbitrary output variables this allows us to store and later extract the positional data of each vertex from the camera Bullseye handles will take the training data and a set of blend shapes and best fit them to the aov of the depth and

normal map renders from the HMC the solver will try to stabilize the performance and provide a mesh that will be represented of the renders the bullseye handle process consists of four core steps setup register anchor and track each step is ingesting the previous step solve the setup command is used for setting path information about the plates aovs cameras mesh data register as seen above is a static non-rigid registration traditionally a neutral pose that is as close as possible to the neutral of the model internally this is an iterative process where the solve will first rigidly align the mesh and then solve the blend shapes and rinse and repeat to the user-defined number of iterations register will produce a single frame Olympic that best fits our mesh and blend shapes provided to the renders along with the transform to position the mesh back to camera space the anchor step takes a set of frames that are flagged as similar to the registration and fits the registered

cache geometry to the depth map while trying to keep a consistent registration amongst the selected frames this command will use Optical flow between all of the selected frames and minimize the error while trying to find the blend shape weights and head transformations the final step track is using optical flow solving the frames in between the anchors resulting in a single cache that will best fit to the plates each of these steps allow for 2D constraints on a per frame basis to really hone in the solve and make sure you're getting the results you want this whole process is a bit timely is a bit timely processing wise but while not optimal for shot production workflow is helpful for creating initial training data now that there is a consistent mesh provided by Bullseye handles a simple plate projection can happen on the initial registration frame and this can be used to properly bind markers to the mesh and write out an animated microphile to be used for training in Bullseye markers and giving us our neutral marker

position for the day this new method of providing training from Bullseye handles helped remove the need of manually tracking frames and iterating on the tracking training until enough frames have been provided to be robust enough for most performances now with marker training from Bullseye handles we can move on to Bullseye markers Bullseye markers will first produce produce a rough estimation of the markers placement based on the training data this can be seen on the right side plates and then refines the tracking through blob detection and a series of checks and balances with the final result being seen on the left the initial estimation is based on Cascade pose regression given a pair of example of images and 3D position of tracked features this tool learns how to locate the same features in unseen images it can be used on plates where the actor has markers applied or not additionally this tool requires for each image sequence in Associated 3D camera track file so it knows how to sample the images from the 3D positions

the 3D position training data does not need to be perfect as it also relies on blob tracking to snap the markers to the marker blobs on the actor's face during the training phase a greedy-based algorithm grabs n number of samples creating a model with enough of a correlation to automatically produce 90 to 95 percent of the tracks for the day after the initial estimation we leverage Optical flow across the entire range to connect blobs from consecutive frames to assign the same IDs for connected ones for each camera this also allows to connect blobs that separate in time due to things like occlusion then using optical flow between the two cameras at each frame we can check the blobs for the entire sequence and assign the same IDs for both cameras reducing the work of the final job to correctly assign the blobs to the same markers each shoot day usually a model is trained that is specific to that day's marker layout this can be achieved by leveraging the previous model on new

plates and cleaning up the track as needed or by running Bullseye handles on the new set of plates as done initially the main point to drive home here is that an artist will not need to create training data manually from scratch generally on a per day basis a set of parameters are also re-evaluated to get the optimal solves out of Bullseye markers this will include things such as blob size and to help make sure we're only capturing the makeup markers are identified and a bounding box for the area of the face to prune portions of the images that are not important once trained though you can evaluate once trained though you can evaluate the model using any input image sequence the resulting 3D positions are exported in the same format used for the training and a static camera is generated which can be used to QC the results against the plates cycle game models have been curated to HMC footage to remove the marker makeup from the plates allowing us to use the same take in gaze ml originally there were corresponding captures of markless

ROM and Visine takes during the tech shoots to build a robust model that works well on the specific actors but we have a generalized enough model at this point that we do not require corresponding markerless takes now with the markerless plates we can infer the eyelid and gaze track from gazemel to have a cohesive performance with the rest of the face and the markers one of the areas that are lacking directly from the markers are the eyelids due to the lack of coverage in the upper eyelid region as well as there not being any information on the actual gaze gazemel is used to help fill that Gap as our feature tracker for the eyelid and Iris originally the model had been trained on a large generic data set and then further refined with HMC specific training data to help improve the overall quality when using HMC on inference only a set of markerless plates are required to produce a solved feature track the marketless plates are required are required for each assault feature track

the marketless plates are required as the markers can cause confusion for the features to track but due to cyclegan we're able to ensure all of the animation is from the same source to not create any consistencies in the solve these results can be combined with our HMC marker data to create a complete facial performance the gazama comes in the form of 2D coordinates but with everything in the same HMC we're able to take the distorted TD points undistort them and then with a transform we get from our masquerade cleanup tool discussed later we can place the iospheres and the camera letting us to do a projection to find a 3D placement for the lid and the case markers the eyelid marker data can then be fed into masquerade upres allowing for more accurate eyelid Fidelity and the Gaze marker data can be used to drive the animation rig eye gaze now that we have Bullseye marker track it's time for a masquerade cleanup this tool will involve taking the raw track from Bullseye markers which we can see

as the initial set of markers on the left first the markers get stabilized to our actor space and then any missing data gets Gap filled which you can see in the red markers the setup needed for masquerade cleanup is quite straightforward with setting up a Gamo with the needed files for evaluation which is just going to be your neutral markers for the day and then training data this marker training data can come from the same source of 4D training data by simply attaching the markers to the 4D mesh and storing out that marker track file similarly to Bullseye markers this process is done in a per shoot basis to the placement changing for markers on each day stabilization and GAP filling process both rely on standard SK learn covariance modules as a way to best fit the input data to the training data these are automatic processes that happen when running the evaluation with settings for Gap fill to help push things more towards the training or more towards the track marker input these settings are configured on a show level but can be tuned to the sequence or

specific shots one of the larger issues that needed to be addressed on the show were stabilization issues that arose during the later suits with the extreme action scenes when running around and jumping the main improvements from stabilization came from leveraging extra fax information and the training data that was specifically for the marker stabilization as well as a weighted mass for the specific markers to stabilize gimms the main problem areas come from the forehead and how that motion is interacting with the top of the helmet this skin is oftentimes just sliding around which can be a bit hard to represent inside of the training awaiting some of these markers allowed for a much more stable results in action sequences without being overly detrimental using facts also helps ensure that the data is very stabilized and allows for creating new unique poses before going too much further I want to discuss WPSD or weighted post-based deformation in short it lets us store Delta

Corrections for n number of examples based on edge strain of a low res mask called a feature graph this solver is used in various places throughout Mass grade 2.0 the setup consists of a feature graph used to calculate Edge strain a neutral mesh and basic sample meshes with corresponding example meshes The Edge strain at the neutral pose and bass pose is stored out with the corresponding Delta between the base and corrected example meshes a trained WPSD model can be applied to any performance with a corresponding feature graph using an RBF solver the various Target Deltas are triggered based on the current feature graph Edge strain the main benefit of this workflow is the sparse nature of needed frames to result in large overall gains masquerade upres relies on a WPSD solver during the final stage where it's applying it back the actor details back to the high resolution mesh and wpsds are also used during the direct drive process to help push the transfer towards any needed Corrections

the Delta and these Corrections could be reduced or removed altogether over time if those same performances are being used for chaining data in masquerade masquerade upres is our in-house software for inferring high resolution deformations from a sparse Marker set with it we are able to drive the high resolution topology directly with the cleaned up marker data from masquerade cleanup and have all that nice actor specific detail coming through the training for upres is a bit more involved compared to the masquerade cleanup setup it is going to be cashing out the initial barycentric coordinates for binding data to be used in the little plasm deformation as well as the WPSD model information this includes the N number of training frames which are selected using a similar greedy-based algorithm that was used during the for the CPR model with Bullseye markers and all of the corresponding data needed for the WPSD model this will be Ed strain

RBF weights Delta correlations Corrections like the other setups that require marker data as an input this must be trained on a per shot base per shoot basis due to the change in Market placement the upris evaluation process utilizes three main steps internally to produce the high resolution mesh with the last step relying on the training data for adding back in the actor-specific detail the first step is laplacian deformation of the actor mesh being directly driven by the 3D marker data followed up by smoothing the mesh to get rid of any artifacts from the laplacian and to provide a clean base for the wpsds which then the final step is the WPSD step which applies back the actor-specific detail to the smooth mesh having the WPSD model already trained allows for the process to only take minutes per shot instead of the hour or so it takes to initially train

the Masquerade upres result can then be applied to our character mesh using our direct drive process direct drive is our process for deformation transfer on faces a transfer rig is built to handle the deformation transfer from the Masquerade asset to our character asset and to generate the needed data for copycat our in-house blend shape solver in the case of the Quarry there's a one-to-one Vertex relation between the Masquerade and character meshes but this is not a requirement and we're able to rely on triangle correspondences or UVS as well there is a standard UV set used for all face topologies which has a texture associated with it to quickly and easily show the flow and the correspondences this helps to ensure there are no misalignments between the two topologies once the results are on the character mesh it can be applied to a low resolution mask to be solved for copycat copycat is our non-linear blend shape solver and is used to apply animation

solved directly to the fact sliders copycat requires three main files before solving a microphile of the blend shaped Deltas a combo mapping file and a slider mapping file this allows copycat to understand the blunt shaped Network and how the sliders map to the facts all that is needed then is a live marker performance in the scene to be solved the projected Iris data from gazumel earlier can be applied to the animation rig triggering the I look around blend shape values based on rotation these bun shaped look arounds these blend shape values can then be taken into account when solving the performance in copycat and this helps distinguish between the I look around animation and the rest of the animation a Delta is then calculated between the direct depth between the direct drive result and the copycat solved and placed front of chain on the animation rig achieving a one-to-one result of the direct drive once animation has been applied to the

rig any artist edits or augmentations can happen as the results are being derived from the actors training data or after specific facts this can be rolled back into Mass parade as extra training data this results in new solves being able to leverage more than just the original training with the animation on the facts rig this data can now go into the standard VFX pipeline or it can go into unreal as fully functioning facts rate Chatterbox is a node that drives a compressed representation of the facts rig through a linear mapping this allows us to have a full functioning facts rig in real time inside of unreal we will take the full fax Network and compress it down using principal component analysis to achieve the optimal representation of the shapes the main benefit of leveraging PCA shapes was a decreased shape count while still keeping as much quality as possible this was largely beneficial at the time of the start of the Quarry as

we wanted to effectively represent the facts Network live inside of unreal having this plugin in unreal as well allows for live takeover of the facial animation if needed for such things as controlling eye movements and blinks or anything that can be represented from the facts rig here's an early example showing the live control inside of unreal you may have noticed slides with footnotes these will take you to our Publications for a deep dive into the individual technologies that can be accessed from the PDF in the vault I'll hand you off to Peter to discuss the pipeline and the improvements that had to happen to help make the court possible thanks Ricky hi I'm Peter Abel I was a pipeline supervisor for DD on the Quarry and as you may have surmised I'm here to talk about the pipeline over the course of the project we learned a number of lessons about scale adaptability troubleshooting strategies and how to integrate new technology into

our workflows more rapidly um to better understand our pipeline Journey I'll briefly highlight where we were starting from after our work on Infinity War the Masquerade 2.0 process was first production proven the initial facial capture pipeline was pretty Elementary this was due mostly to three factors a limited engineering resources limited scale Infinity War had as mentioned about 29 minutes of capture to process which fell well within the means of a simpler pipeline and a hesitance to invest in locking down the process whilst r d pushed relentlessly forward with new approaches the foundation for publishing Version Control dependency management Farm submission QC rendering was solid and a pretty standard implementation for something like this at our facility the shortcomings were primarily in

Automation and the front end tooling consisted mostly of Scandal and guise for each step in the process uh they amounted to forms for configuring inputs and settings with some smarts to Auto populate Fields based on the state of our database users had the option to daisy chain the rest of the pipeline on submission with primarily default settings mostly to accommodate an initial iteration from which to work for the most part artists would run one step at a time so they had more control over the settings for each step in the process this was sufficient when you had individual operators responsible for one or two steps and artists to fine-tune the outputs here you can see an example of one such interface it's pretty simple but it got the job done earlier Aruna shared some statistics about the difference in scale between the Quarry and our typical magnitude of work on a feature film like Infinity War we had over 70 times the amount of data

to process and we wanted to do it without any animators in the end we did as mentioned have 26 takes where they needed to intervene but we didn't it we didn't miss the mark by far there uh the comparison is an important one for the story of this project we realized that we needed to revamp the whole pipeline in order to handle this class of workload without the typical luxury in the feature film world of having a host of artists to finesse individual shots our mantras quickly became automate everything and work in bulk these are difficult asks when in an active r d environment where the processes and I O requirements are changing rapidly the happy story for pipeline engineers is for reach researchers to prototype a new technology artists to test and develop the process for using it and then finally for pipeline to come in and make a product out of it for use at scale

but the flexibility to continue iterating on both the technology and the workflow and perpetuity was important to us on this project one of the supervisors at the time said he wanted the interface to be just one big red button they could process everything on the show uh well we ended up somewhere slightly more elaborate again I think we didn't miss that Mark by far either as Ricky Illustrated there are several steps to go from HMC plates to deliverable animation and when you get into the details there are dozens more not so significant supporting processes uh that had to be run as well the approach we took was to encapsulate layers of configuration into a nested graph and encapsulate that graph within a simplified bulk editing interface this was especially important not only when you consider the raw magnitude of data to process but the quantity of iteration required throughout the years of the project there were rigs or models that had

changes and when those changes happened it triggered the need to resolve some of the performances before I get into more details about a new pipeline I'll illustrate what the stack looks like at a very high level so this is how our old pipeline worked very straightforward we have a user they interact with the pipeline either via a command line or a graphical user interface and both of those exist as wrappers over an underlying python API in our new system we preserved our existing interfaces but added a few new ones and restructured the our the underlying API as well pgb or performance graph Builder is a GUI for configuring and submitting shots for processing in bulk pgb sits on top of ddg or digital domain graph which is a node-based GUI that executes python code where there's a certain lack of imagination when it comes to the names they're pretty descriptive what pgb

actually produces is a graph that's interpreted and executed by ddg we built a set of templates for ddg to represent components of our pipeline that could be used discreetly or combined together to orchestrate larger sections of the overall pipeline this could be done directly by users experimenting with modifications for the workflow or procedurally via pgp the architecture of our underlying API is now also logically a graph that both clarified its relationship with ddg making plug-in development very straightforward and improve the maintainability and extensibility of our code base now let's get into some more details about these systems at the time we began designing our new facial capture pipeline the internal application ddg had been in development for some time and had seen limited use in other departments in our company ddg is essentially a node-based visual programming interface for python similar

to what you may be accustomed to with Houdini or Blueprints and unreal we decided to develop a pipeline as a python API with plug-in wrappers for ddg representing the pipeline as nodes in a graph and this killed two birds for us out of the gate the user interface as our artists are well accustomed to working in node-based tools and orchestration ddg had a convenient interface for handling serialization Process Management and integration with our render Farm we exposed the pipeline to our more technical artists this way enabling them to continue experimenting with different ways to process the data to achieve better results it also provided our pipeline developers the means to quickly swap out parts of the pipeline without much overhead enabling more rapid integration of new technology here I have a bit of a contrived example for demonstration purposes showing how one can quickly prototype new workflows users can drop down what we call a

script node configure its inputs outputs wired into place and write arbitrary python code that can then be tested immediately in the context of the Raider pipeline later their work can be formalized as new node types and templates for others to be used this is what the graph looks like at a very high level just one node to select performances to process and one node to do the processing by nesting the graph very deeply we control the interface to the end users the notes can be configured at any level but the deeper you go the more Technical and less abstract it becomes if we dive into the processing node you can see what the high level processing graph looks like this exposes the main operations that Ricky described earlier if a user wanted to tweak the settings for any given step they would do so here I'll dive in One More Level just to illustrate some of the complexity that

we encapsulate in those higher level nodes as you continue to dive deeper and deeper into this graph of course we would have been throttled by the artists had we just handed them a bunch of nodes and told them to wire up their own pipeline uh we developed a simpler interface that we called pgb performance graph Builder that could be used to configure and generate graphs for any subsection of the overall pipeline this would become our big red button for processing the show the users could select any number of HMC performances so they wanted to process specify which steps they wanted to run and it would generate a graph accordingly from this tool you could either generate a graph that you could then edit before submission or just submit the processes without to the farm without directly seeing a graph and that ended up being the primary mode of operation for most of our users it's mostly the the leads and the technical artists that we're getting in there and changing the graphs themselves

so most of the time the users did not need to interact with the graph grass could be produced for any subsection of the overall graph to run specific steps as desired the steps presented in this interface ran in one of two modes generator query generate to run or rerun a particular step to generate new data or a query to reuse already generated results for some Downstream processes one of the once the plates calibration metadata had all been ingested the asset development complete a single user could essentially hit the I'm feeling lucky button and submit the entire show to the farm this would however be ill-advised burning the assignment Loop of our farm software to the ground by the sheer quantity of enqueued jobs aside we iterated throughout the production on what settings produced the best results it was better to submit a few performances at a time early on to zero in on the configuration for each step that would bear the ripest

fruit globally also these settings often change from day to day based on shooting conditions after some iteration larger batches of jobs would be run overnight or over a weekend producing various QC renders to review the results our system does a remarkable job of capturing my new details in facial performances that however comes as a double-edged sword it means that it is also hypersensitive to small errors in calibration and stabilization minor issues had cascading effects that accumulated to bigger problems in the performances that made it important to generate effective key surrenders at each step in the process when users would submit a couple hundred performances over a weekend if unsatisfactory results were produced we needed the ability to trace the problems back to their source as quickly as possible this meant quite a few renders for every solved performance

that translates to more core hours and more disk space to account for although we had a limited pipeline engineering team it was worthwhile to invest our time in optimizing these qcs a bonus of using the system that we didn't consider up front was that it made it much easier to debug certain classes of problems on failed processes here I've updated our example from earlier to raise an exception you can see our interface here clearly highlights the erred nodes in red you can simply dive down the red graph nodes to the lowest level to see the cause of the problem you can select the art node so you trace back in a little in the lower right log pane so far nothing out of the ordinary get it traced back that's normal the real win here is that we also get all of the result values for every node that was executed successfully including the inputs of the failed node we get this as a byproduct of ddg's orchestration it serializes all of these values to facilitate inter-process

communication allowing us to introspect every process Upstream of the failure without explicitly logging every value as it's resolved it's similar to having a remote debugger connected to all of your processes at all times no need for a pipeline engineer to take steps to reproduce an artist's problem when the state of the graph is serialized every step of the way in some cases operators were even able to troubleshoot problems themselves without support from our department we frequently encountered some issues that cause jobs to fail while ddg streamlined the process of debugging and rectifying those errors a cost is incurred by virtue of a process failing after submission in the first place a user might submit 100 shots to solve overnight and if many of them fail relatively early on they have wasted an evening or maybe even a weekend of processing time and iteration having to resubmit those new processes

once the issue has been resolved not to mention the core hours burned on our farm nodes for jobs that would just need to be rerun these were sometimes simple problems that could be easily fixed such as data having been taken Offline that needed to be restored that in particular was not an uncommon occurrence for us that quarry had a much longer life span than a typical visual visual effects project so older data would frequently be archived to manage our active disk capacity to mitigate the farm and Human Resources expended on failed iterations we developed a pre-flight check system for users to run before submission this closely resembled a unit test Runner that artists could use as an early Discovery system for problems before submitting lengthy batches of shots the lesson we took to heart here was essentially fail fast becomes a more important design ethos at scale we front loaded all the quarries from

various parts of the graph so that we could quickly verify the submissions were likely to succeed and fix any problems before we hit them the tool would also provide succinct explanations and offer instructions on how to rectify them I'll leave you now with a brief list of some details we encountered when adapting our VFX pipeline to games we ran into a number of small but unanticipated issues we needed to adapt our shot Centric world to an asset Centric system for games no longer were we packaging renders as our final deliverables but 3D assets and animation data directly that had some effect on the semantics of our production tracking internally and required a new delivery tool set to be developed it was easiest for supermassive to receive files with static aversionless names to update Assets in engine while we did generate sidecar manifest Json files with metadata and versioning

information for our deliverers deliveries that proved to be inconvenient to track on the client side so we started embedding uuids into the files themselves so we could facilitate cross facility troubleshooting the duration of each individual performance was also about an order of magnitude greater than uh what we would traditionally see in VFX shots in films are usually quite short on the order of a couple hundred frames performance is on the Quarry were in the thousands and in a few cases over ten thousand frames long as a result we needed to parallelize some of our longer running processes such as the bullseye Microsoft markerless plate generation through cyclegan and gazelle projection to solve some memory issues and also enable iteration in a more reasonable amount of time amusingly this also caused some issues with insufficient frame padding the

devil really is in the details there were some face palm moments there it is worth mentioning that the vast majority of VFX studios are very Linux Centric and DD is no exception our tools are prevailingly developed with Centos in mind we do have a separate virtual production Network that operates on Windows which is what we use on stage and to test products and unreal as a result our tools needed to include more cross-platform testing and had some additional steps for us to deploy and now I'll hand you back to Aruna all right uh thanks guys it's a lot of information learning stuff every day um so let's recap what went well well we succeeded in solving 30 hours of varied mocap performances like this extreme take of Zach who plays Jacob in the game uh tackling a dummy on set and this is a result of the accurate HMC

stabilization that we were talking about which allows our actors to perform on set without distractions and restrictions I mean you might break a helmet camera too but you'll get a good performance no we didn't break any during this show since we scanned our actors prior to the mocap shoot the supermassive games was able to outfit the characters and they could see their performances on their versions within the game this allowed greater feedback on set as the actors and director could talk about the Ambiance and performance in the scene a lit mocap stage is very different from a dark Hackett house if you remember our Chatterbox QC tools uh were enabled us to review each of the performances independently before pushing them into the game it was important to review all shots and takes with an unreal to confirm that Rigs and animations were correct and having a breadcrumb trail of versions that we could double check throughout the pipeline so things we learned or things that went wrong um you can't animate my Brute Force 1

900 minutes of facial animation you really need to have a method to track and automate a large number of processes and QC along the way and fix items as globally as possible in our original training set we didn't capture a large enough variation of human emotion which involved going back into the rigs and adding facial shapes after the initial model creations this led to changing rigs facial rigs changed constantly and with 18 different character rigs across different types of meshes for each of the various outfits that the characters were in ideally only only one facial rake should be required per actor while a large number of our processes are aided by Machine learning artists are still required to make changes and tell the machine when things are wrong and also actually start the machine so some things we are already improving which include getting the full range of actual performances which are very important at the beginning of acquisition is important that the

characters in character it is important that the actor is in character when these takes are done we only had 96 facial shapes for each character to represent all the performances in the game this eventual increase of fidelity is a Venn diagram of gameplay optimization and overhead number of characters on screen the number of facial shapes and of course the platform delivery while our Chatterbox QC tools allowed reviewing facial performances they were very much Hall of President style very head in the jar in the future projects seeing how lighting and texturing play with the facial work is just as important so see everything in the engine and see it in its correct lighting environments as we move forward into the future some of our future work includes the continued development of marvelous High Fidelity facial tracking this shows a great promise as in and is nearly ready for production

we've also improved onset reconstruction and QC with an with a custom facial capture rig for daily photogrammetry these are the daily ROMs we do every day this enables much more refined and accurate facial solving data take corresponds over many shoot days is important and as we said we had 42 of these we're also finalizing the development of Chatterbox 2.0 which is currently showing a nearly 10x speed improvement over our first version of Chatterbox one which we used on the Quarry this will allow for both more characters and more Nuance in active performances so I want to thank you all for coming um we are tiring across multiple disciplines model rig animation pipeline I think we have a little bit of time for Q a so thank you [Applause] yeah there you go

hi hi um I had a question for Ricky you mentioned that you had or I'm sorry you mentioned that you had 96 shapes I'm really impressed by that could you talk a little bit more about how you got so much Fidelity out of the fact shape that you had um like maybe like the PCA shapes 600 um and so yeah it was just the Chatterbox representation for unreal that had the reduced shape count okay that's awesome could you talk a little bit more about the PCA shapes um yeah um uh I mean it's basically uh Peter I don't know if you can describe it better than I can yeah this thing on can you hear me perfect uh yeah so PCA or principal component analysis is a common mathematical technique used for various machine learning processes but

it's essentially a compression where you're taking the most important or the highest magnitude parts of the performance and you're compressing them into shapes that aren't semantically understood so they're not like facts where you you have like a lip puller or something to that effect it's more like all of the shapes are involved in producing all of the outputs so every shape is activated on every frame which has some minor inefficiencies and unreal but because of the limited shape count it performs quite well still and yeah so basically every shape is going to transform the whole head in some way that isn't really understood by a person but when we run it through our processes it produces the right output performance how many components did you have so how many components were there like uh 96 96 96 ships we we used them as well so we use around the same like about a hundred or so yeah it's really cool okay thank you

hello first off awesome I thought you used animators way more than you did so that was really cool to see now that you have a pipeline set up like with everything that goes into it do you think it's going to be viable to see this kind of technology in more and more games as things go forward absolutely yeah I think um a lot of the research and development is kind of in place and we've we've got a good pipeline going here we're also developing newer technologies that require less inputs and less um less involvement on stage and less initial setup so we're reducing cost as well in many different ways it's awesome I look forward to it thank you hello this is astounding amount of work and hype to check quality is amazing congratulations on all the success and all the the awards and everything you've gotten I mean this is again something you want to work under optimal conditions and when working uh working Partnerships and things but can you talk about uh what how much you were impacted

by changes in the game itself in the narrative the design and if you can't talk about it I understand that too oh absolutely yeah so our characters are able to be driven by different Lookouts in the games because you can take over the eyes and you can kind of have that emotional beats in certain sections of the game you do have an opportunity to pick different weapons or choose different choices and there's an opportunity for the supermassive team to actually take those Eye controls over and retarget them to where you might want to see you can look down a path um as far as I know I don't know if they implemented a lot of those Lookouts but will the director is is really interested in continuing to push those types of um user-based uh game mechanics to allow you to have a character like focus on different parts of the scene and allow you to actually really kind of take certain nuances away if you need the character be a little bit more happy or Smiley you can do that at runtime now with this system um so pretty cool thank you

um love the game it's super fun to play um one thing I'm curious about was um how did you deal with um uh motion capturing for the sequences where the player like walks around in an open area um was there any um specific processes you had for that based on like the variable levels of speed that they walk at uh we yeah so we captured a number of different idols and takes at the studio and they were used to kind of help propagate individual different walk cycles for all the characters all the characters had their different walk Cycles um that was done by the supermassive team after they got our data and you can see in some of those characters that their Walks Like Ted Ramey has a specific type of walk um so they we captured all of that and then gave it to the supermassive team to kind of do the idols and retargets and get that going there cool thank you all right uh great work first that's super impressive

um a quick question on the direct drive stuff is that just like a linear transfer of Deltas or do you guys do any patch or we have kind of like two methods where kind of like just like a straight linear transfer of deltas and then we have more of like a kind of like the gradient transfer deformation so looking at gradients um but for a project where it's one to one it was pretty close all right thanks excellent excellent hi um I was curious oh this is a little weird I was curious if you could mention any of the decision making process of using the the compressed shapes on on the face um from your packs to that compressed shape um that custom uh blend shapes on the face as opposed to translating to something like meta-human compatible like like live link stuff um was that just not around at the beginning or were there other decisions

that you were thinking about at the time yeah one of the because we started in like late 2018 early 2019 some of this technology didn't exist that we're seeing on the floor today so we did have to build a lot of that Nuance into that and make something A system that really worked um to hit the director's vision of getting that Nuance you see the eye gazes you see the little Twitches in the mouth and at the time none of that existed in a consumer product yet epic is getting there and is getting really close to that but all of the actors in the game and all the characters in the game are driven by the actors and there are specific human nuances to each of those actors so if you look at any of those any movies with those actors in them you'll see the exact same facial performances and it really bodes well for actors to come in and say yeah that performance that's in that game is me those are my face shapes and that was one of the big requirements of this project cool thank you hi uh could you talk a little bit more about the head Mountain height uh HMC setup uh how you got that standardized across

like I guess like when you show up for a day how you standardize that well uh yeah I can talk a little bit and then Ricky can do as well um so we had about I think four or five head mounted cameras um they were all calibrated per the actor at the beginning of every day and if they um came back we wouldn't touch them so they were calibrated with uh with the system that we had here uh right there so I can come up yeah so every camera system every head mounted camera system was measured locked off for every actor um and depending on if they would come back they would always be calibrated the same to them both the camera lensing information the hmcs the the all the the lens grids that we had and that was stored in the system and then when they came back they would just put that on and we would hopefully be there correct um those day-to-day corresponses correspondence that we do with the photogrammetry rig allows us to

triangulate and confirm from day to day that markers are in the proper place with that as well we do have a vacuform system that allows actors to get vacuformed with a kind of a plastic face mask and using that we drill holes through the mask when they're not in it and put those every day so we have a kind of more approximate position of all the the tracking markers and again now with that as we move into markless tracking that's less of an issue but it helped us keep calibration from day to day because you know actors got bigger and smaller and got pregnant um and it was yeah that's kind of how we keep track of it all thank you and that's a that's a stereo rate right is there a top and a bottom yeah this is a top and bottom techno props helmet camera helmet mounted camera system um and we've experimented with different types of helmet camera systems but this is just the one we have in-house but we can work with a majority of them awesome thank you cool

hello thank you for your talk I did have a question about one of the frames of your presentation where you were putting the high poly model the model I think that you were using to transfer the data and the low poly model um and how you had a system of like markers on it could you explain if that's the right interpretation of how you guys transferred like the blend sheeps from that high poly model to that locally model with differing topologies uh so that but uh so that would be more on kind of like specifically to the facial capture side uh where we don't rely specifically on blend shapes um so we're taking the marker data that we get from the tracking software and then we're feeding that into masquerade upres and so inside of masquerade upres that's going to deform our actor topology directly and then using our WPSD solver we're able to basically well we'll first drive that topology directly with the markers then there's a

smoothing process to kind of like clean up artifacts and give you a clean base and then using the WPSD solver it's able to add back the actor-specific detail from our training data all right uh thank you we'll be outside in the wrap-up interview if you have any more questions but um you know like your favorite Rideshare service yeah please rate US five stars um if there's anything less just let us know in the comments on how we can improve and uh yeah we'll talk to you guys outside thank you [Applause] [Music]