Sucker Punch's Performance Capture for inFAMOUS: Second Son

I assume you're all gathered here today to worship cat Zulu welcome congregation or perhaps you're here for the sector punch facial performance capture pipeline as seen on an infamous second son so the infamous series is this is welcome the second sign is the third installment of series of the infamous series where basically have an open world superhero game that or supervillain depending on how you play and the choices to make so karma plays a big role in the game not only in the choices you make and the gameplay it

affects the story and the outcome and how your powers progress the infamous Ares with its comic book roots has a very strong narrative as all superheroes need a strong backstory and with the jump into new hardware we really wanted to see about pushing the facial performances trying to just you know push the quality on all fronts and as you are probably familiar if you're here with facial animation the the higher jump in quality you get on all fronts the more important it is to have solid facial animation because you'll quickly dive into the uncanny valley so what we had before the previous two infamous games we had the real-time cutscenes anteed animations for both cutscene and in gameplay basically driven by joint only facial rigs so moving forward onto second son we really

wanted to dive deeper into the story with new hardware we feel like we could better push them closer to the characters and have more compelling compelling story bring some actors in actually capture try to capture the subtle nuances that will help you know carry you know connects you with with both the story and the characters and and hence the gameplay ultimately giving the you know the gameplay more meaning giving that more motivation to the player and just give a richer fuller more cinematic experience to the game so we're exploring the facial motion capture it was on the table right at the beginning but if people can go that route we knew that we wanted to have bodies faces eyes an audio and multiple actors captured at the same time there are methods what you know where you can get some facial capture that really require that you're

in an isolated area separate from your body capture your face would be scanned and tracked and applied later you know so you end up having this disconnect between your body and your face as well as just disconnect between your actors so endless and lastly it was still a requirement that it's all runs in real-time so here's some highlights of our results [Music]

[Music] game is released on Friday by the way we're all excited okay we hit the town we load up at the all you can leech Honda with power buffet maybe swing by the space feel we're back home to say [Music]

so there you go so we were looking into you know the various approaches you know for facial performances and you can be on the route we went before with the hand-feed animations there's a 3d scan per frame you know performance capture kind of like I guess I would call it like LA noire style where you're captured you know in a booth I think that you know that that that's a method where you can really get lots of detail directly from the actor but there are like with all these things there are their own limitations and we did want to capture the body and the face and the other actors all at the same time and on stage and you think that's for method you're

really kind of isolating narrowing what kind of performance as you can get there let's kind of keep their head still and they you know I have to use reference objects for where their eyes are tracking and it's kinda all pieced together so you don't wanna go big with animations because that might not this where they come together well and also there's a really consistent topology or you v's you know with with that method that would know considerable work so retargeting to other characters is you know next to impossible and you know having animation on top of it and kind of fixing anything would basically just require a reshoot because that you really can't do that so there's an another methods muscle basing relations and you know things that have been done for future film stuff you know that has relevance and you know it was you know really more of an R&D project that we wouldn't really think was necessary considering that the last option that we

did go with which is the data-driven example based approach that you know is based on a lot of things that have already been proven out and has worked well in feature films in the past so basically what the data-driven example based solution is is you have a bunch of data in our case a bunch of scans of of your actor it's going through a range of motion to get it's kind of infer the physiology of the face you know the other little Salty's that happen with the muscle sliding over bone and fashion and skin you know when you get it from a range of scans you can get all that stuff for free it's the scans are roughly based on what's called a fat system which there was a PhD named Paul Ekman that completely unrelated to CG he had described all the muscle movements in the face as independent movements that

defined everything you could do with the face anything else was just a combination of those so it's been a really good library for what kind of range of motion you can get out of face that can be used for getting quality scans to really fill out that data set of of poses that you're going to use to ultimately you know blend and match to what the actors doing at any given frame of motion capture sort of make that happen you need a consistent facial mark early so the method that we had gone through there's a number of marker dots or placed on the face it's a aligned with muscle regions and the objections that the face moves it per actor you kind of align it in a way that is following their muscle flow and kind of between the creases so it's always exposed to the cameras free gift cameras I'll go into that little bit later but

we did use head rigs that are Wireless Hedwig's that have cameras playing back at the actor that's recording all the markers on the face and they're up close enough to get that high fidelity that you want for the into the subtle movement on the face so that same marker said is what it's used on a day-to-day basis for the motion capture itself so what we do is we actually get a vacuform mask the default pose of the actor and there's where the marker set is we basically extrude little points on that model and have it sent off to an engineering firm that will print out a vacuform mask and drill holes in each one of the spots so we can use that as a template you know on set beginning of any day of motion capture and it'll correspond directly to our fax data all of our scans so so as part of that we need a method of taking that facial marker data and obviously coming up with

a nice blend of shapes that's that's reasonable so that's where the solver comes comes in basicly just applies that data in our case to a facial rig where you can also be used for animating on top of it you can replace animation or just animate on top to tweak or change of performances needed and lastly you know if your game character doesn't look exactly like your actor you want to vary it up you'll have to retarget that somehow onto your game character so I'll go into that also in a little bit here so here's just a sky view of the pipeline in general the whole left column is basically about acquiring the shapes old describe that next we're just kind of floating over the the points here we have our facial motion capture the picture of Troy Baker after wearing the head rig so all the scans are

combined with our solver which in our case is a post based of the former and the motion capture is applied to the rig so that it's directly connected to this solver and just outputs a normalized number of shapes that matches the current configuration of motion capture and you can do additional things on top of that additional deformers simulations kind of sky's the limit with this method we actually have joints per a facial marker nearly a couple hundred facial markers and so we've got a pretty solid skin cluster as a basis where this all starts off and all the deformations that we have our triskin cluster to give you the vertex level changes that are inferred from all those scans that we got so I'll come

back onto those two details later just jump ahead twice okay yeah here we go this is the fact session of this giant chart here and we had a day this is one time set up you know a lot of stuff where I will scan the actor going through these series of facts poses and combination poses and sometimes poses that are just unique signature looks that he's going to have a while in character so we're trying to get as much data this can be repeat and useful and method we're using it doesn't have to be quite perfectly isolated shapes like a lot of a lot of scan process is really required that you kind of isolate those motions so they kind of add up well but there's problems that blend shapes if you're trying to make them add they're not keeping them normalized you can end up with shapes that just don't work well together so I've seen huge networks of fixer shapes that they try to make combinations work together in order to kind of make up for where shapes don't

work well together when actually they should be morphing together you know where they're not adding their own normalized so that you never have the weight of any one given shape over one so it's gone with the process of how we get the consistent topology of all of our FAC shapes I mean each of these scans you know are millions of points if you looked at in wireframe view would just look solid so obviously that you know we're not gonna play that in in the end any one decent topology it has good flow much lower resolution a lot more resolution that we could have on previous some previous games but the actor well we're doing these scans you know you can move around and take breaks and come back through there's you know they're seated so there's a stabilization process that has to happen to make all these shapes relative to the default pose if we take a low res version of that face

based off the default pose we make sure there's a tight correspondence between the markers from that default pose and all the other shapes on that low res version that we make it's basically extracted from our our default pose to start with so that's what's that setup we use a warp algorithm it'll take the marker points and basically warp the geometry and snap it to the surface of each one of those shapes there's a subdivide and then a snap where it's projects itself onto the surface to pull those extra details back into the shape so each of these shapes end up being combined with the back of the head we have eye socks and mouse socks no socks and there's basically our jointing kinda go for the ride as you went through each other sheets almost like it was a as a blend shape initially just to have it hold the neck and mouth long for the ride so they said they're staying combined so just gonna copy them

off along the way you know ultimately end up with your final topology so you have with this processes we're starting with the low res you have a version two versions ability or low res and a high res that can be applied with scans there's always problems with the eyes and around the teeth where you know there's a lot of reflectance you have eyelashes and that causes interference you can get a lot of noise in those areas and the lips in particular you have areas that are not always exposed the inside of the lips and you'd be surprised how much that actually shows with really expressive emotional performances and so we'll go and we'll have a modeler go in and based off of the scans clean up that area around the mouth and make sure that they maintain their nice blonde ability across all the shapes that's one area this it's neglected quite a bit which when you get more realistic characters it's really

important to make sure that that you're maintaining the thickness of the lips that the volume changes as you know as the lips kind of scrunched together and so you the modeler who understands anatomy and ultimately he goes through and you could you know kind of run them off as a bunch they'd see how they all blend together and there's more stats on our skinny process yeah I'm from 60 to 70 scans tractor we as they described we applied that on to consistent topology the hardware we use were to link to structured lights canards and then get a lot of detail out of face if we had them scripted to kick off in succession so you know four cameras two per per unit which takes a couple of seconds to a few seconds we had your faster machines everything but it was cost of getting

that much detail on the cheap I guess but it does get a lot of detail and if you get results right away so if you didn't you know factor moved a little bit at that time you can see it right away you can just you know have it taken again so that was process we use again that'sthat's a scanning process that's done in a static volume you know they're seated in the chair it too has to be you know stabilized you know against the defaults at they all blend well that's good process for the fax buzzer so here's Troy Baker again Delsin Rowe series of his fax poses so quite a number of shapes mouth open and closed giving range of motion and because each those markers has a joint there and actually every joint has their own post base to former each one of those shapes can contribute uniquely to any other part of the face so these are all triggering independently to give you

basically an infinite combination of of what the face can do so here's here's a view of that consistent topology remeasuring that I was describing on the bright we have our low res mesh same daaad configuration it gets warped and snapped to that dense one in the middle which I like I said I couldn't show that in wider wouldn't see anything then that's combined with the back of the head the mouth socket and such so here's this little video just got planned through some of those blend shapes you see I just made a bunch F out of the shape just you can see them in succession you can see how they're working and decide if there's any anything it needs to happen but it's really important like this whole process you really are following those those marker points carefully through the entire pipeline because you want as much consistency as you can because those are go to your drivers and you want them triggering the

right shape so are you describe that marker correspondence you can see what the custom vacuform mask looks like here on the lower left so it's got this series of holes gonna go on there with the marker and place the dots of course you know it would be careful doing this to make sure that the mouth is consistently you know a nice default pose the actors not trying to talk about anything we actually had gotten some back where I think the marker set was a little off and all the animation kind of look like he had a droopy mouth but like we had just animation controls we can you know tweak that right back so I think I've run through all these points can you move ahead it's no be describing them from the motion capture process itself these wireless head rigs are

really lightweight they're Wireless you can just swap in cars and get you know a couple hours of motion capture arm if we actually would have to swap it but and the battery swap out and such as needed so some the other requirements of these head rigs is that well actually I'm jumping ahead of myself I'll just you know follow through here and follow the slides it's very described that we want everything to be captured simultaneously so how do we all that stuff in sync traditional film you know misuse a hard hard piece of hardware that just has cycles that's always ticking at the same moment and was cameras you would find on a motion capture set are gonna allow all the equipment to capture at the same time so they're all in that same cycle on they're stamped with the timecode so when you're going back and looking at the data you'll see that they're all

lining up as needed kind of a follow-up benefit of having those facial rigs is that you can get a video right from any one of those cameras or combination and stitching together to project onto a temporary piece of geometry so you have basically an animated texture map of the actual performance and you can have that delivered right away with the body motion capture so that you can just continue setting up your scene and setting up your camera for cutscenes and well the actual tracking is taking place that I can take a little bit longer as far as these head rigs there's different ones that are out there and depending on your your own pipeline might have different needs we use the the multi-camera setup so we could actually triangulate the positions of each of those markers in 3d other method methods

that have been used for tracking use you know things like optical flow and feature recognition you know if you try to estimate you know the movement of the face but they kind of motion capture there's sometimes there's noise that you have to to deal with and try to filter your data luckily but you know when you have cameras up that close that's not as big of a deal the helmets can slip around you know they're there they're on the priest on but they're not you know bolted to their head unfortunately they did agree to that but the so they can slip around from day to day they're you know they're gonna be aligned slightly different so a stabilization process that can constrain those back to relative to a default is a necessary part of that process so you know you can also have missing or occluded data you know you have wrinkles and folds and brows and stuff that can just hide

marker data so a way of dealing with the missing data and then with facial rigs the lighting conditions are very important the stage that we were on had the entire ceiling was emitting light and the walls and very nice diffuse lighting so pretty much from any angle you're going to get a decent capture of the face other units have lights that are on them projecting back it can be distracting to an actor but again you know depending on the conditions in the stage you're on they'll have different needs which brings me to the retargeting phase so retargeting is be done in a number of ways I think I'll go through describing all the ways we didn't do it but basically for our method we we targeted both the shapes and the facial motion capture you

know so that the facial motion capture is basically moving in the same space as your game character version so it's triggering your game character shapes and it's all applied into a rig so that since we're actually using the joints connected with that marker movement it's all incorporated into a rig that works well using the actual game character seekers one of the actors might have seen in the highlights there she's her name's Karen and we have a Native American counterpart in the game thicker face bagging your eyes wider nose more ethnic look and using these two defaults and all the shapes we've got from the scans we can generate all the other shapes so it basically uses a proportional constraint to take every edge in the face and that all difference between adieu to default it

doesn't solve and iterates basically to come up with where all those would settle as moved through each of those other shapes so if Adrienne stops by here you can have any questions about that specific solver that he he had created he explored a number of different constraint types one nice thing is that with this method it also was regarding the motion data which is very low risk in relative to the really high res shapes the marker points actually line up exactly spot for spots and when over triggering the right you know shapes with the right data so as far as you know how we're applying out shapes I mean you could have blind shapes they're easy to animate because there's you know possibility of having the over adding issues that I was described describing if they're not normalized any shape based system if you

have too few of shapes you can get cosy results as as they're blending between each other can if you have to for your shapes can I kind of take wonky direction you know if you seen that in animation where there's something just a little off it just kind of takes a weird path between poses so that's a problem with any any system if you don't have enough shapes but we do have a method that we can kind of correct for something I just in case there is areas in the post space that you're not actually getting data to avoid that wonky path that they could take while blending so ultimately as our solver is a post base deformer this is basically just you know deriving shape weights from your marker data at its hard it uses a radial basis function which you look at the references do have deep into the math on that if you're interested but ultimately what's really nice about it is that weights that it outputs are

normalized - from the most relevant the closest shape in post space from your it matches your market configuration on any one of your shaped will trigger in that section remember we're doing a different post base per joint so if you have a configuration that exactly matches the scan you'll get that exact vertex data applied in that area so and this is a applied pre skinned cluster so that the heavy lifting is done by the joints and then the extra data is on top of that which helps for some efficient compression because we can stream that stuff into the game where the joints are just animation that's played back in an engine but then vertex data is just that

extra level of deformation that can happen on top of that but you have the joints there for a level of detail if you want as well so as far as like training this post base - former you have all your shapes and you have your marker configurations and basically to fill in that you're the former with all its information it's basically just a matrix that setup of rows being shapes and columns being vector data coefficients are the shape weights and process called a single value decomposition is run on it which gives you a transpose matrix that basically just says if you have a bunch of new marker data from an arbitrary pose from motion capture you just have that lined up you multiply it by the original matrix and you're going to get those weights as a result that I can within the DES former it will blend those shapes together to give you an output so it's led to this performer

node in our case in in Maya so you have a direct connection into the node and you have a direct connection out to your shape and it's just this blended result of all the most appropriate shapes so here's just kind of a way of visualizing the protective time my timer did not work well financial speed-up because again because a lot to cover it was just kind of a visualization of what a post space de forma would be if it's really simplified down at TD let's let's say we picked a point exactly on one of those markers and there was a bunch of poses that we're that vertice is moved around in a different spot and so that that would represent you know the post space here and I'm a little red circle with the fall-off there is some arbitrary pose and from this that radial basis function of it is that it would it kind of reaches out in into a radius in that

space and gives you a kind of a captured volume of relevant shapes and so the closer of shape is to the center it's going to be driven to a value of one all the other ones fall-off to zero now it's kind of like a weighted average but biasing towards ones that are closer to the center so it kind of cancels out any other shape as it gets closer to an exact match of your pose so let's say you had in this scenario that single vertex it has a wide range of motion and why very little in X you can do what's called a regularization of your parameters which you basically just have an input with scale for each of your parameters that brings them down to very similar space so they're similar values that sort of an oblong circle trying to capture that really wide stretch and why you you know you really haven't kind of normalized down and space to where

you're you're you're using a circular radius you know the radial basis function so one cool thing about your input with scale is that you can also if you scale them all together you basically are increasing the volume of your capture space for influence of shapes and so if you have if you have holes in your your post base you you have kind of even distribution but it's a little farther apart even if you don't have maybe enough poses you scale that up it's going to reach out and capture more and get a nice blend of of those shapes and this is something that can actually be exposed as an animator control that if you have you know a nice distribution everywhere because have a few spots where there are holes in your post space where you just didn't have posed you know scan that was relevant for that marker said you need to scale that up until it's getting

influenced by the nearing poses and either anime didn't use it like that or just copy off the shape after you scaled it to so it's blending with those other poses and put that right back into the post based áformer as a new shape so here's this little test we've done all this training to get our post space deformers where we just iterated through our shapes we had surface locators place au some marker and so what that would do is go for they'd go for a right as you're running through all those shapes and as series and blend shapes and we're just copying off the values and it just gets fed into the deform ER and so we took those shapes and just randomly picked different shapes over a thousand you know combinations of just picking random shapes so and blend them over over time to see that your interpolation of your post spaces is working well and it shouldn't obviously look like your pose when it's at the same pose that your blend shape is that so here we have

the actor on the left their shapes the retargeted shapes on the far right and the other ones removing joints this is actually a skin cluster but on top of that the vertex level stuff join us kicking in to make it actually match what is happening on the right so it isn't a blend shape it's actually using that post based a former so the example of some of those poses because if they were only a skin cluster if they didn't have that extra level of vertex deformations from the that post based áformer itself I'm just kind of go back and forth you can definitely see on the lips you know the amount of difference maybe and the lower left there you can see the volume lips is totally lost by just dragging around joints obviously when you get more realistic when you're even for real more realistic animation lips are a particular area that could fall apart really quickly so this is a great way of

just really pulling out the real physiological detail at a vertex level so we have key drivers out of those markers that we can divide up the face into regions and each of those regions use a different combination of these markers Wiest off of distance and offsets and then another layer on top of that one of your got the per joint level uses those key markers and their own position so you can have more fine and control on top of that I'll describe that in a little more detail dyes are a separate thing you might have notice the eyes weren't blending in that one version at all it's because they have different drivers that your eye rotation and your blanks the direction you're looking as well as the markers so for squinting you know as the distance has changed there they have their own sort of drivers that are trained for those separately so kind of like was mentioned

earlier processes fine-tuning your your post base to former so you bail these shapes in there you're going through them and they they're not quite hitting all the poses like you'd like them to you know answer that maybe blend shape test and you see that well it's not really interpreting the way we want so we have a method that kind of goes through and regularizes all those parameters that they're working in the same space as a starting point that you can kind of scale them up until you see that they'd captured all the spaces you needed yeah like I described it like a whole bunch of really nice a distribution of poses they're just certain spots that don't have that decent pose you know you could again just scale them up at the pose until it thick you know finds the neighboring shape to copy them off put them back in the system or even you know with ribbing you can use a lot of things with post based which was really awesome is that you can actually we didn't do this but you know you certainly could go through and just say wherever there's a gap in post space let's train

it to ramp up those scales on the fly you know so that it's normal default scales but then when it gets close to it empty pose regional dis automatically branch out and kind of compass the neighboring shapes good briefly mentioned weighted PS DS but basically this the separating of the the joints into or the the post based performers in into different regions you know basically way of getting a similar effect for those that are familiar with the weighted PST so here's our example of where we had a hole and our post base it's not quite reaching it so you can imagine if we just increased our scalers would just increase that radius and encompass those shapes so we have our one time training phase where it goes through with our surface markers going for ride if that matrix all set up and that once it set up it's just in the breagan and i think about it as you move

your controls around on the rig it's actually interpolating the same shapes so that's one things is particularly about doing this layered approach of using key markers is that you have individual controls on the face at some of those markers you can just move around I'll see the example of that a little bit later how that works but I mentioned efficient streaming here that's that's basically the fact that you're having skin clusters doing the heavy lifting and you have basically small offsets or the difference for your deform er which you could take advantage of GPU decompression to stream that in really fast wait about two twelve thousand vertices perf after you know we think we had it up to three afters at a time so we could do things like on some characters like the Betty character we actually did full Sims that this would drive the face and on top of it we could

do a simulation loosen up the skin and they're slower to be dynamic and get to some of that extra cool stuff out of it on top of it because the fact that we're just streaming it in afterward so so we think about doing this per joint motion that is it's really cool the fact that you can actually get smoother natural arts of motion in this way I don't describe but I'm talking back here so we again will go back to like vertice that's right on when those markers on the lower left there's a balloon dot representing a pose if you're right on that pose you know your your face is going to match up perfectly with that pose along the way to that other pose along at the top you're going to get a another perfect match up a pose but when you're in between there if you don't have poses specifically trained along the whole way it's gonna be basically an interpolating between the the most relevant shapes nearby and

space so you can see it's influenced a little bit more work it's curving over to the right and this shapes to look good in themselves like but if you didn't have quite enough shapes it's going to take a slightly different path while it's blending between those shapes and and so having these joints pulling back to the actual 3d tract the locators that you have just basically warps the whole thing gives it a nudge to it so it's a hundred percent accurate if your track is accurate at each of those vertices and so everything can be between just goes for a ride with a nice fall off so here's the image of what I'm talking about just to kind of go back and forth here you can see that not quite lined with the surface here they are and so that's just actually have a control per joint that allows animator disable the track wasn't really doing something weird here it's solving well but I don't want to fall at track there but you know the Browse and getting a lot more dynamic motion on that so I'm just going to have

them attract to the markers and just get that more fluid dynamic you know if he moves around very fluidly and having that directly tied to the actors motion is is really important for realism so just play this so this isn't in the camera view so there was used in the cutscene so you'll see the hand come in here that kind of snaps in a place awkwardly but we get the idea that you can see that the those those dots stay with the face where is this earlier version once I click the right one yeah sorry here so with this version you can see that that kind of tells us head on there's there's a gap between his face and natural 3d position that represents a an inaccuracy and we didn't have an opposes to completely make it that tight but that little nudge that we gave it

you know makes it as accurate again as your track is so that brings me to the fact that we used a 3d track versus a 2d track the 3d track does require four cameras you know to get really good coverage we have found but the benefit of the 3d tracks is that it allows you to verify your data we have mean of the actual track points there right next to the face and your as your face is solving those should be lined up at least very closely so you can you know tell right off if you're not solving very well and then you have the additional benefit of just being able to warp right to those marking points and give you just that extra level of a detailed fluid motion with just taking the normal natural arcs right from the actor lastly and this is this is pretty key I think is the fact that I mean the lips are the most moving part of the face and it actually moves and Z you

know you can pull your lips out your jaw forward and all this and with the 3d track you you take advantage of that extra dimensionality and your drivers so that there's a you know making some pretty faces but it that kind of shows like roughly like you can make the same pose from a front you and have it look very similar when I'm pushing my lips out one of them really tightening them but kind of keeping that same rounded shape just moving your jaw for it even can do this thinking here I'm actually rolling my lip in and pushing my jaw forward and rolling it out from the front view there's hardly a difference noticeable which means if these two different shapes are getting triggered by your same drivers because they look the same and 2d space you're going to get wonky interpolation in your shapes because it's applying shapes that are irrelevant to the actual pose so that brings us on

to the facial rig so it's mixing that we have key markers they're carefully picked as all the drivers are carefully picked per region to use the most relevant either distances or vector directions of movement of those markers so that basically through motion studies of seeing where the face moves independently and some experimentation to find the best combination of drivers so but once we have our key drivers those points in the face that you know basically move them up the most we can hook controls up to there and you can just move them around and you actually can get all that interpolation between the shapes as you form around also you example around a second and then of course the fine level of control for each individual joint if you want exposed that we had the you I did I described had a picture per scan that was one the pictures that we actually

taken right from the moment of the scan and we we have a UI that exposes those with sliders you can click on any picture and I'll give you sliders and we can pick however many regions of the face we want to break it up into with sliders and it it's basically just moving the joints in that region and it move them in a normalized way such that you move them he moved all the sliders to the right for that face all the joints would exactly match the positions of those markers as they were in the scan especially what that looks like here in a second but again if you're because it's tied directly to this deform on this post based reformer and you go from pose a to pose be it's a big distance of travel between those poses what you get for free along the way is any other Road shape triggering because all in there in the post space and so maybe you didn't some with your jaw open there comes a point where it's ready to compress if actually information is in there it's gonna wait to the right time before those lips touch and start to compress and you get that stuff for free which is

a very cool so here's an example of the UI I was subscribing we're able to slide those regions around and to kind of pick any part of the face from any other picture I like but the eyes are doing here with the left side of the mouth to do what the students over here this gives the both this gives a way of you know fine-tuning what you want if you're starting from scratch or if you're just tweaking motion capture so but this year you're able to slide an attribute that says that you want to have it take over your motion capture so that basically as you dial in any shape it that also way the motion capture so you're not fighting with it or you can just set it to add on top so I'm just pulling around some of those motion controls you can kind of see how like moving the chin makes the your the

lip compress up or let's play that again it goes pretty quickly there but I'm not rotating anything I'm just translating them around and just because if you pull it under the mouth it's going to you know curl the lip in and if you pull it over the front just can try to you know pull the jaw forward and I was just yes fun to play with it when you actions either she's kicking in submission exposing some of those attributes I was jumped in to you some of the ones that I hadn't mentioned sticky lips I drag I've been kind of down on how much I drag the controls have how are we doing on time anybody 37 minutes in okay so this makes me to our or wrinkle or creased maps that it's basically basically we're using the compression and stretch independently

from a subset of markers on the face that drive basically an alpha a dynamic alpha so wherever there's compression it's gonna light up red where they're stretches kind of light up green and that will reveal a wrinkle or crease map or a stretch map so what this does is you can let that subset of vertices that you're using you basically are defining a low-res cage that you can are correct where you want things to to always trigger together if you use everybody's in the Faceman it might be noisy a lot of red and green little dot so this is not only good for that but it makes for faster calculations so I'm going to fall off the way we create our Maps is really interesting is that if you look at stream data from scans if somebody going through a range of facial motions and you look at it in UV space where you're just seeing the color changing and

changing and the geometry is not moving what's really interesting is that 95% of those wrinkles are just just coming on in the same spot and this is why but as you as you get older you start to get wrinkles you know because it's the repeated wrinkling happy in the same spot so that means we could just take the aspects that we wanted from the various maps and just kind of make a single map out of a combination of where those wrinkles and creases are coming out and of course he was going and you know use a normal painting map to can I add any that you want in addition if you say trying to make them older here's an example off that if you've seen in in our highlights video this is a Reggie row character and he's you see corner of his eyes and his forehead you know those wrinkles kind of kicking in on dynamically there and this is what he looks like with just lamb

bird and this the skin cluster and close space to former so there's a little bit stuff there but not that kind of detail coming out and this is what the vertex color being triggered by that compression and stretch looks like so many squints the one I can see that becomes red and the other one goes to green gives just a nice dynamic way of having a trigger right where they need to be so which brings us to actually getting this stuff into the engine in real time and that wrinkle map is also applied as a plugin in Maya that so when you're moving to bring around you see those kick in as well so that you know that they correspond to what's happening in the engine so as far as the real-time performance mentioned that we have the joint animation IG GPU decompression one aspect that something like Maya and these are the programs to do for free is to recalculate the normals if you have

deformations moving the vertices around if they didn't do that you you might not see the details actually come out because it's not reacting to light at the angles that it should so that's one of things had to be added in there some scene fixing we had a shader applying the parts there streaming of the face and a different shader very similar but just wasn't screaming for the edge around it kind of ran into the funny little snag that the round off that my I was using for its skin weights wasn't matching with the round off and the GPU was so it was giving this little seam that once we corrected that one away some additional considerations tracking issues we really learning experience from this round was that is probably beneficial if we have a rigged version that's of the actor before any retargeting that's that your the

tracking company kind of if you're doing it outsource they can apply it immediately to the actor rig and get immediate results to see if there's anything wrong we a camera calibration or misalignment from the stabilization so they can iterate on that before it goes through the rest of the pipeline gets applied into the scene and gets animated on before you realize it it's really not behaving well IRA target process I think you know it has an impersonal improvement as well so something I think you know led to a little bit of hand animation tweaking and spots where they probably didn't need to we infer the jaw motion from couple points on the face with basically aiming I joined at the center point of those so it's moving the jaw with it I mean the skin does move no matter what you do so it's it's doing a decent job of implying where the teeth are but we just give an extra offset control so you can kind of find to align those as needed afterwards if if if it looks like

it's not aligned correctly again it's no attention to the deformation point where it connects between your posts base two farmers may be regular skin clusters and such so which brings me to the end I don't know if Adrienne Bentley is here and he's had his own talk this week oh hey he's gonna be around hopefully I come up and help answer questions related any the the deep engine programming and plug-in writing that he implemented so he's a co contributor and big shout out to him for all the work he did as well as a java old Seabee jostled who did additional plug-in and programming work and why the you and Steven white as well so thank you Cthulhu thanks you and special thanks to

the DDC committee and feel free to contact me through the email or I just got Twitter on there so actually you guys are type of people I would actually have an account for to respond back so feel free to connect with me that way we can have the wrap up room afterwards for additional questions but we have time for questions now and it just couldn't step up to the microphone as you have the question yeah looked looked fantastic I was just wondering if you simulated are considered simulating any of the muscles underneath you said you talked about simulating the skin yeah that's that's something that certainly done in feature films that like these wet as example there this is amazing tissue simulation that starts

from muscle and bone and all these layers are simulated it's just we don't have a giant render farm to be kicking these things off and it's really a matter of time and iteration and animators be able to you know tweak the performances without having to wait to see the results so we didn't you know go that far but you can certainly infer that stuff after the fact and you had a farm to kind of go help you like that stuff it's really good how much yes the question was how much animation was required you know after the the whole process was done we didn't save a lot of time in the end for motion capture so we made for a facial tweaking but you know there was definitely a good amount you know like I said we did we did have you know tracking and retargeting issues and it's easier sometimes just to go through

and just kind of tweak them and then to figure out we'll have to do a post-mortem know kind of yeah check for some of those you know errors came from but surprising we didn't have a ton of that that we had to do considering I kind of we did it on the cheap you know basically compared to like a feature film approach but I can't tell you like there's a matter of hour of man-hours who went into it but it was a very tight schedule at the end where animators were going in a shot-for-shot just kind of tweaking on them and pumping a mallet so there was there was that to it I guess what what facial capture system did you guys use because it looks like it's doing a little bit of 3d and a little bit of 2d with the eyes or something can you you know over that the facial actual capture process tell me the helmet the helmet is again it's a four camera setup

where you actually have to use tracking software that looks through those cameras and tries to triangulate the position and per frame it it follows from the different views to figure out where those points are some companies that specialize in that certainly the film industry we use digital domain was there was a company we used their stage and their helmets and so they did the tracking for us and would deliver those marker points that we would apply into our rigs yes did you have two different runtime rigs for the cinematics versus the gameplay sections and so roughly what was the joint count for the face for those rigs the marker joins themselves not if you don't come to eyes and all hopefully joints on the neck and such we had a hundred and sixty-eight markers the we could certainly have

implemented some of that within within the game but we really decided that you're really not up and close to really need to do all that additional streaming we have an open world game very fast travel so we didn't really feel that was an area that we really had to take up computation power but but since we do have that joint motion that was perfect for just a LOD for the rest of the game but you could drive up with the same system do you remember how many joins you had for the real time character that the same 160 860 for the face okay thank did you use certain markers to stabilize the face for the stabilization process we have the civilization from just our facts session we need to get those aligned and then there's a stabilization process that Digital Domain did using the motion capture to try to get that relative to the default and they have

their own proprietary method of doing that it's a constrained spring basis system they try to keep it aligned but I'm not privy to you know answer any details on that yes yeah I was just wondering as far as using this sort of system for instances where your characters are quite a bit different from the actor like what are some of the challenges you see with that I know you mentioned retargeting being an issue here where Troy Baker looks a lot like your character what if it was a creature or something in that nature you can certainly take one minute left I'd say might have to pull all the questions into the room emphasis but I'm so your question was if you retargeting really far away from what your actual game character was we had a pretty close correspondents been with our main character but it's anyone I mean a giraffe you know you could instead of retargeting all the motion capture and all the shapes make a correspondence between your giraffe for each pose and even if it's a combination of other

blends to give you that pose that correlation is what can give you a path for an algorithm for retargeting those if you do have an exact pose one for one that nicely blends on the other end the out poking weights that are from that before for those two farmers can be just directly fed into the other one yeah and it would drive this the face at the same time that's it well thank you very much for your time [Applause]