(Super)Humans of New York: Managing the Many Faces in Marvel's Spider-Man

[Music] thanks for coming to my talk superhumans of new york managing marvel's spider-man's many faces i'm noah alzaire a character td at insomniac games and as the title of the talk might imply i'll be talking a bit about the work that went into creating the faces in our most recent game marvel spider-man which released last september to be more specific i'll be talking about how we started setting up our process with a third party studio to create facial rigs i'll expand on how we added our own little twists along the way and added edits to the art on our end i'll cover how we track those changes as subsequent re-deliveries came in and finally i'll end with the work we did getting heads in the open world but first for anyone who's unfamiliar with the game or hasn't played it in a while here's a trailer that features some of the facial work that went into the game what is happening to our city yuri i

don't know feels like the end of the world maybe it is peter parker how the hell are you the city is in danger it needs our help all of our help all right well call the play coach gang at costume nut jobs is taking city apart piece by piece time i return the favor this is opportunity knocking you know the closer you get to them the more you become a target right the closer i get the better chance we have to stop them i've ordered silver sable and our team to shoot you on site this city's had enough of your vigilantes you're officially an enemy of the people

[Music] come on peter uh how do i call you i mean who would you have like a cell phone in your in your pock pockets or something thank you so going in one of our biggest goals was to create a living breathing new york since spider-man is a character so is often so inexorably tied to the city of new york new york is often one of the most important characters and in service of that goal every character big or small has a feel reel whether you are in a cinematic or just walking down the street we had a very high bar set we

essentially needed to have a photoreal quality to our characters and the interpersonal relationships had to deliver a punch so we couldn't really skimp on anything they had to have realistic and nuanced performances in order to hit our goals so to fill those to fulfill those expectations we'd also have to have lots of characters as with most superhero stories especially the spider-man ones our story we were aspiring to tell was about big action and big implications but rooted in a very personal story about peter parker and the people around him aspirationally that's all great but practically for character production and all that it usually means a lot of work and it meant that some featured characters in some of our cinematics were only featured once or twice in the entire game and uh so we had to have them up to a certain level though so that they wouldn't stand out like a sore thumb against the other characters for whom we were devoting more time and resources final oh excuse me

uh so finally these higher quality rigs were fine for you know the larger cinematics but it had to feel like new york and little secret new york is pretty heavily populated so you know we would have to have rigs that would also be able to run in the open world for when you're out on the street and you know we didn't want to turn the game into a slideshow every time you were outside so realistic faces are very difficult to pull off and uh you know with this with a game like this it can really make or break it uh and you know it takes a lot of time to kind of get uh a sort of system in place and in the first year of production at our studio we only had three riggers in the burbank uh studio working on spider-man and we hadn't really done a project production on this kind of level yet in our history though we've grown considerably now in our rigging team even doing something now from scratch

you know it takes a super long time and you know there's lots of r d and it would be a steep uphill battle adi even at this point for all the higher quality characters we might have a general idea of where to go but it just it takes a lot of time and experimentation and make mistakes being made to kind of get to that point where we can create uh characters added quality that we were aspiring to uh so you know we we decided to instead approach a third party that knew what they were doing and had a system in place and that party was three lateral as you guys may have already heard of these guys they've been making a pretty big splash in the last few years but a quick summary on them for people who might not know they specialize in data acquisition processing and creating rigs for interactive characters rigs are often based off of photogrammetry scans in order to get those realistic rigs as was as it was done in their work with us now they've worked with many methods of capturing data for our purposes they did

scans of actors in neutral pose as well as doing facts poses for anyone who's unfamiliar with facts or didn't go to axel's great talk yesterday i definitely recommend seeing it if it ever goes up on the internet at some point the god of war talk i'll be talking a bit about it during the presentation uh so just to give anyone unfamiliar crash course in a paragraph it stands for facial action coding system and it's a system that was popularized about 30 years ago by a man named paul ekman and it essentially breaks down anatomically possible muscle movements on a face it was originally kind of developed as a psychology tool so with this breakdown since we're all experts you can tell that i was having a kind of rough day when i took this picture but it tends to lend itself to well to creating realistic facial rigs and you going into performance capture and thus is often a sort of baseline for creating realistic facial rigs terminology can sometimes be different depending on what system

what system you know people are using and things they set up but this is the general thrust of it so the rigs they made for us were to be joint based with corrective facial blend shapes using streaming tech in our engine we were able to stream hundreds of blunt shapes during cinematics and further polish in detail came from blending normal maps as well as blending color maps these were derived from those photogrammetry scans and used to simulate wrinkles and blood pulling under the skin in more extreme poses finally a custom embedded rig node that they shipped with each character uh kind of did all the the calculations for the rig logic under the hood instead of using utility nodes which allowed us to kind of play them back in the viewport in real time which is a pretty cool thing considering how complicated they were but anyways this is all just for context you know for this company and its methods you know a lot of companies and teams are using similar methods but if you'd like to know more specifically about them their work and their process

you can see gdc talks they've presented their website their youtube and uh yeah they're out there a lot so for all these characters we needed we set up multiple tiers for quality when setting up work agreements with three lateral the way things worked out characters uh we set up five five tiers and uh first i'll go through tiers one through three since they were functionally fairly similar and then i'll follow with tier four and five so tiers one through three were any named characters they were actors we had cast specifically for certain roles and sometimes they were meant to be digital doubles meaning it was a direct representation of the actor and their facial features such as with otto octavius and norman osborne then we had other rigs that we coined characterized meaning that they were based off of scans but then edited to feed a design that the actor might not quite fit and in the regular cases it meant slight changes in physique for people

like peter parker or kingpin to kind of widen their necks and like you know give them whiter jaws but then in the more extreme cases of this characterization after the fact it meant things like adding severe scar tissue to electro or fully changing the shape of the cranium and adding more scars or with hammerhead the tiers one through three essentially covered any characters that that were kind of cast to be in a golden path cinematic you know more than once or twice and uh you know and i say at time of casting because there were actually some cert some characters where they had been written as bigger roles but then as rewrites happened for story cohesion certain ones were cut just because we had a pretty crowded story and uh what we ended up doing with those facial rigs was just using them for other background characters that for whom we might have used a tier four or five rig there was actually one character that we had cast uh and we put her into three separate roles uh and

you know completely separate people in the game and we were i think able to get by with it because of some texture and hairstyling wizardry from our character team actually all three were in the slideshow and you may or may not have noticed but it's pretty cool so the rigs were based off of scans of the actors doing the facial poses and the facts shapes i mentioned before with the classification uh for the tiers one two or three we kind of set up how much uh iteration and blend shapes uh would be kind of done by three lateral so the tier one character was essentially the top and had a lot of iteration while a tier three had a fair amount of custom work but it was still one of those things where little inconsistencies between that and the scan were okay as long as things didn't look broken because they were minor characters uh so we didn't go as over over it with as fine tooth a comb as we did with the major characters so the way it broke down major characters like peter or mj where tier one secondary but prominent and

important characters like captain waptanabe or kingpin were tier two or three with characters like tombstone or the roseman's gallery manager being being tier three as mentioned previously these rigs had blending normal and diffuse maps to simulate wrinkles and skin flushing so to that end they delivered with each rig they they sent us three wrinkle maps and three color maps respectively which were then masked out to create a total of 37 tracks that would blend in during animation to replace the normal in color result for those who have played the game might we they might be aware we were fortunate enough enough to have the late legend stan lee agreed to do the cameo in our game and with him we were able to also get the scans of him in the fax poses as well so his wrinkle maps and all of it is actually derived from his data to recreate his likeness it was awesome to be able to have him in our game and have it kind of have that stamp of a true marvel creation for me personally it made for a pretty surreal day making uh goofy

faces at the stan lee while coaching him through the fax poses he's a great sport about it and you know yeah he's great hope he missed so tier four characters in blow covered kind of just yeah background characters as needed the minor characters are background people uh they were based off of scans that we had done from an open call for studio employees and relatives who might be in town to go to a 3d scanning studio nearby where we did photogrammetry scans of them some of the scans ended up heavily edited but that's kind of how like the sort of baseline started so if anyone knows any insomniac employees or relatives of employees and might have seen some familiar faces in the game chances are it was them the scans however were only in the neutral pose and we never did the facts shapes and things like that so what three lateral did was they took the neutral poses and based on some some statistical estimation then derived

their movements and everything obviously it was all still high quality and it had like you know blending uh add wrinkle maps and all that uh and blend shapes and all that but uh yeah it was uh not driven by scan data as none of that was available in this case so to that point instead of the 37 wrinkle and color maps from the higher tiers tier 4 rigs had 24 blending wrinkle map tracks and no color maps they say tracks because since they didn't uh they didn't have that information they just set up the tracks in the logic and what we ended up doing was kind of plugging that in and changing our skin shader a bit for these characters so then wrinkle maps would come in additively rather than replacing and with that we were able to share uh wrinkle maps between all these characters and what we did was we made generic sets one for male female and elderly

finally for tier five we fed in a bunch of heads that had already been created for the game as seeds for a character creator of sorts complete with sliders that three lateral had delivered to us to make new models in maya one of our character artists created a wide array of male and female heads of various races ages for various body types to cover civilians thugs heavy thugs and police officers those character creator values were sent back to them and they created rigged versions using that same statistical approximation model and sent them back to us all told we had about 90 of them so it was definitely a case where having a simple batchable command helped me out a lot as some if some of this is sounding vaguely familiar it's actually very close to the gene splicer technology base that they showed off in gdc 2017 at the i believe it was at the at the unreal engine summit uh but however for our purposes it was

it wasn't about trying to create rigs that would like run in run time in the engine and like have sliders and all that but just about creating a wide array of faces so then we could uh kind of prop out our cinematics with good background people anyways functionally for our purposes on our end they behaved very similar to the tier four heads with the same mesh topology and shader setup so it's pretty much the same protocol except just on way more characters so that's a lot of things to keep on top of as one might imagine three laterals works load aside for creating all 150 of these rigs there's a lot of files to track integrate vet keep track of and edit especially considering that it was kind of only supposed to be me doing this and it was with the expectation also that it wouldn't take up all of my time either because there are always more things to create other tools and pipelines to maintain maya's to crash all of that so i imagine lots more

studios and projects are using these types of separate team situations to create facial rigs in the last few years either through outside companies like we did or in the case of the bigger publishers they'll start to set up internal groups as shared resource groups for all their studios i figure this situation is becoming more and more common uh a common thing that might be happening more and more in aaa and even double a sometimes uh hence the genesis of this talk at least on my end i figure it's a thing where a conversation on best practices might be a beneficial thing to start whether or not ours turned out to be the best or far from it so the first simple problem was processing the raw files they sent us since they weren't working off of our perforce branch they sent clean files based off of their own environment every single time so anytime they we sent feedback a new version would come back to us and we'd have to process and integrate it again and this happened over many months over the uh

with many characters so we wrote a tool to do that boring stuff and uh kind of batch through everything thrown our way this processing tool at first started out is just a simple thing for adding controllers and body joints and repathing textures changing changing their naming conventions to kind of just exactly fit with ours and you know set up attributes to control blend shapes and wrinkle maps and all that with our engine conventions but once we got the basic stuff down uh we wanted to add hooks for you know extra functions for certain characters and stuff and i'll kind of go through all that uh later this tool kind of started out as a as a simple like you know couple button thing so excuse the interface because uh it's kind of one of those things where a small solution had to turn into a bigger and bigger solution and it kind of turned into a gray sea of semi buttons but it got the job done so one of the first things we tackled

next was streamlining our feedback process with them they're based out of serbia so we wanted to stick mostly to uh email communication because in-person communication was obviously out of the question to do like super often and also skype calls would be considered kind of a last resort item since uh you know we have that big time difference as well to deal with so as part of their deliveries to us they would go through they would go through the facts shapes and try and pose the rig to match they would then assess internally and send the controller values used to get to the to match with those uh photogrammetry uh scans and in the margins of that spreadsheet they would list any notes on discrepancies they had spotted on their own so then because of that we wouldn't necessarily try and waste a lot of time communicating issues that they had already spotted so that saves a little bit of time so since we had that spreadsheet i wrote a function to read those controller

values from that spreadsheet create a calisthenics animation and import the scans to match those timings so then with that we could get a play blast in of the delivered rig in maya we could export the animations to a test cinematic in our engine and get a play blast of the scans coming in and all those captures were then brought into a template premiere project to get a comparison between the delivered rig how the animation was uh was exporting to the engine as well as a photogrammetry scan for a ground truth comparison this allowed us to spot extra issues with the rig that where we wanted to you know communicate that's at three lateral as well as suss out any potential issues that we might be having with the way the rig animation was being delivered to our engine for in for fixes that we might need to make on our end with the help of the tool and the workflow we had set up the process of creating these videos was actually turned into a simple enough process that i could kind of delegate it to a junior employee from

our production services department who had very little maya experience and because of that that was kind of off my plate so i could do other things and an hour or so after a rig was delivered and processed then my art director leed and i we would get a video of that and then be able to just do quick draw overs and send back to three lateralis feedback from that i can say at least speaking on our end uh and our perspective it made for a great great working relationship where more often than not uh you know we were on at least the same page about what we were trying to communicate there were rarely that many miscommunications which is i think pretty cool considering you know how often emails can turn into miscommunications when you're with someone who's four feet away versus on another continent so when they delivered the heads to us they always just had the standard eyelashes brows eyeball geometry and all that however in order to hit the art direction we wanted the character art team would

make new assets for things like hair eyelashes beards jewelry and all that and as time went on and re iterative rig deliveries came back in you know it became pretty clear that we would need a way to save out those extra changes we had gotten so then we wouldn't have to just keep saving it out and bringing stuff back in transferring weights all the time because uh you know it would just be a big time sync and then potentially things might be forgotten so to get around this a function was added to save extra meshes to a and associated data to a character post folder and the way it worked was to simply duplicate the selected meshes arranging them into a matching hierarchy relative to the geometry folder for things like led groups and all that and we exported those duplicates with their materials to a separate file called extra meshes and along with those it exported skin weight values and all that to a folder using our save skin weights tool with that reimporting those extra meshes scanned it in their proper respective

groups was a fresh into a fresh file that we got from them was a simple button click with little or no thought or work required by us so even though they were doing very good with the very good work with the face any neck movement was just purely for adam's apple you know swallowing flexing of the muscles for you know that intense performance basically they were tasked with getting the facial movements but neck deformation for head looks and body mechanics wasn't really under their purview under our work agreement so it wasn't really on them to do it what they delivered to us was a face with single joints for the head and neck and the rest was just kind of skinned to a chest joint this is a very simple and generic setup for basic movement but as soon as we started getting those heads in cinematics with some facial animation it was pretty clear that we had to get the rest up in line with a level of polish with the face now one thing that i feel is often missing from from a lot of game

characters is that polish work does tend to end kind of just at the chin a little where no work is done for creating a realistic and plausible uh feeling in the neck when the head turns so you don't get that sternomastoid uh flex in the neck and when you look down there's often you know it's just it's kind of trying to get things good enough so you know you might get some gradual twists or something but that's often the extent of it obviously budgets and and time being the way they are it's it's a simple it's an understandable thing to kind of let go but in our case we we actually did i think since we were freed up in a lot of ways we had the opportunity in time to push things a little bit further and since we already had a few hundred corrective blend shapes coming in for the face adding a few more for left right head up and head down movements was just kind of a drop in the bucket we eventually also started to add fixers for you know combinations where it's looking down into the side because on some of those characters things were

maybe looking a little crunchy at times though the meshes were pretty dense unfortunately sometimes that last little bit of mesh density at the base of the neck to get that sort of sharp profile on characters who had more exposed necks wasn't quite there so moving forward it would probably be cool to integrate some wrinkle map tracks and stuff like that but that would take some uh you know some probably some schmoozing with the uh with our shader tech artists you know in order to change up our shaders so might i mention they're incredibly good looking well dressed and uh very smart anyways for most of our characters we sculpted correctness for them using a tool to use some tools to streamline things using this workflow where it would generate duplicated targets of the sculpted corrective result edit those and quickly update the resulting shapes by extracting deltas essentially just creating a sort of link between all of that then you know a script would

set all of that up and then when you uh extracted deltas again a preset calisthenics animation would uh kind of come up and let you see how the uh corrective craft is blended into each other it might be worth mentioning that we did our production in maya 2016 before they added the shape editor so the creation of these tools was born largely out of some some lack of reliability that we were having when using the old one shape editor and using the sculpt tool where sometimes edits were saving to the base mesh rather than the than the corrective we were editing now that we're on a newer version of maya i'm going to be kind of i guess evaluating the new stuff and seeing if we phase this out if it turns out to be stable and useful just a little note so if we ever needed any other custom shapes there was also a tool to easily set up a cone pose reader on specified joints for post-space deformation

and add that to the arsenal of correctives for some of the more major characters and so it was usually just things like raising the clavicles for people like peter or also there were some cases where we had some edge cases for doing like skin-to-skin contact with the you know the uh with any kissing that the game may or may not have ended on once we were done with the correctives using those same tools we could just export it out to that extra that character post folder and it would store anything to so then it could be brought back in on future deliveries since a pipeline had already been written to export shapes out to a folder and store them we were actually occasionally able to find some secondary uses one notable one was that we were able to create create a pretty uh easily scriptable uh neck jiggle rig for kingpin as he is a uh heavyset gentleman we kind of wanted we we realized that things were looking a little static under his chin uh but the issue was that his uh his rig

already had neck influences from the facial rig so we couldn't necessarily just throw a joint on top of it and uh you know not mess with the facial animation so that wasn't an option but using the jiggle on a middle controller of the standard my muscle rig we were able to set up driven blend shapes so we had three of them for movement in the positive axis positive direction in the x y and z axes which we would then bring in and then drive in the positive and negative and we were basically able to kind of create a joint movement in a sort of faked way so then the jiggle bun shapes and the neck could co-exist for cinematics we would go through and do a pass on baking those controllers for his jowl jiggle there's a subtle thing that no consumers likely noticed in game but it at least made me happy so there you go so that blend shape stuff worked for cinematics but we couldn't have those blend shapes streaming in gameplay because that is something that is very

much just delegated to very much you know specific to cinematics so we need a gradual neck twist rig for the neck and also while we were at it needed a way to load in clavicle skinning since that area was just skin to the chest so we had to add them all after the fact but at the same time we wanted to avoid messing with three lateral skin weights for the face in order to edit the clavicle weighting and add gradual weighting in a clean scalable way as with all other edits which was the goal we added a a feature to our import skin weights to tool called relative mode where weights could be brought in to only affect the weights of influences from the imported skin data file so that way if the tool loaded a skin data file with weights only for the head neck twist joints and clavicles and chests it would only affect the weight share values for those influences while leaving the facial weights untouched making it easy to load

in fresh deliveries to other rigs completely as long as they shared the same topology so you can see this extreme example here where i'm going to skin his cranium to the clavicle i can save it out and then load it back in in a relative mode onto a onto a version of the rig that has all the face joints and you can see that it now only affects that area while leaving the facial joints untouched so to break down exactly what we're doing to achieve this i'll go over a quick example warning there will be a bit of math but don't worry it's simple and quick because i went to art school and i was only required to take one math class and it was actually just called math so say we have a head if we were to look at vert 7729 we see its weight values here if so if we load in a skin data file the skin data file has neck head and neck twist joint influence data only and not the hundreds of facial joints that are included in this

sort of thing so we look at the weight data for the same vert from the skin data file skin data file and we see that it's 0.5 neck for twist and .5 head all that really matters here is the 50 50 split since we're working in relative mode we're only concerned with the influences that are in the skin data file and so jaw bulge and jaw line aren't influences in the skin data file we'll leave them alone and only affect the 0.5 0.35 remaining influence then we basically multiply out the incoming influences by 0.35 zero out any other influences and match them with the skin data file and we have a normal a normalized result so once the weights were set just like everything else we needed a sustainable way to export them out uh sans facial joints so essentially we needed a way to do the inverse of that process that i just described so at insomniac we do have a some tools

for saving out skin weights what it does is it basically takes the weight values for each vert and then just exports them out as a json formatted file it's a thing that we just tend to do a lot quickly to just quickly rebind a thing or just share some weights between things as long as they share mesh topology so since our save skin weights tool output was a collection of of data in a simple dictionary i was able to interact with that data to get what we needed so while the regular one was saving out joints for the entire skin the entire skin cluster i was able to write something that would output as a separate file with edited values so along with exporting skin for this full skeleton it exported a skin for the skeleton that had the body joints and none of the facial joints as you can see there are a lot of facial joints but fortunately they were all under two sort of root joints one for the head and one for the neck so i was able to make something that could generically detect all face joints whether they were

under the hierarchy of the face or under the hierarchy of the neck the tool would then transfer weight values for all face joints to the head and all neck joints from the face rig under the end twist joint for the head the result would be an extra non-face skin data file for the mesh that could be brought back on the rig in relative mode for uh for new deliveries since we shared topology between all the heads this would actually work on other files as well if we wanted to do some recycling to get that clavicle weight in so if there are any math quizzes in the audience going through these numbers in their head they might have noticed a little bit of a problem with the approach from the last slide if the facial animation from the neck joints were being saved out to the end twice twist joint it creates a situation where the end controller gets more influence as it's saved out because it takes more and more relative influence from the resulting file let me break it down if we look back on our old friend vert 7729 if we go by this formula of saving out

the non-face skin data file we take the facial joints and migrate the weighting to the head joint and if the head and neck for twist ratio which used to be one to one is actually now very unbalanced so if we were to turn it back around and reload the weights back in in relative mode we actually have a lot more waiting on the head than before and uh the more and more these weights are saved out the worse the problem gets uh you know it still isn't messing with the facial joint that's the uh the facial joints but that's still uh you know that was the main purpose of this but things are getting more skewed so really all i can say is uh i concur with dr octavius on this one luckily this process started to kind of come online when the rigs were closer to being completed so even though it was possible that it slowly started to offset things it never really became an issue because we were always re-extracting deltas for those uh those blend shapes because we were all

saving out those bases and then just redoing it it's just a little like clerical error on my part that uh luckily didn't really balloon into anything and if i'm being honest i didn't even notice it until i was preparing this presentation uh but you know what i'll probably change in the future now for this scheme is basically have it export out to a joint that can just kind of serve as a dummy that exists in both so like we actually don't actually use the base like neck joint at all like the one that drives the choice joints so i'll likely just put the stuff into there so then it just serves as a container and then it all just goes back out to the face anyways when you load it back in but yeah if you guys want to do this sort of thing just keep that in mind and uh be better than me so for the rest of you who have been glazing over for the last couple of minutes uh time to unglaze because we're going away from the land of numbers and into triangulating quad meshes so maybe keep facebook handy

so with all that workflow solved one confusing snag we ran into for getting our models in engine was things would look fine in maya but when we exported things out there was this odd disconnect between the way it looked in maya and the way it was in the engine we were getting this uh sort of sawtooth thing and jagginess along the silhouette of the neck at certain times i asked around to other riggers our engine people and did lots of frantic googling to find out what was going on and i was pretty perplexed we ran side-by-side overlays had no idea what was going on i was thinking maybe there was something wrong with the engine but i was wrong because our car department is amazing please give us more features uh and uh you know what i accidentally stumbled backwards into was figuring out a lot about the way maya triangulates quads for any four-sided face maya accepts that shape but quietly triangulates those faces by connecting the two closest corners since it does this all quietly you don't really notice it except for the telltale

pops that you kind of get when when a mesh is moving and two corners are suddenly the new ones that are closest to each other there are ways you can kind of display it in maya which you might be able to see here uh but either way yeah you can turn on something where it'll softly show it and you can see it switch when when you move verts around so there was kind of the rub it was uh it was dynamically re-triangulating that stuff in maya and but your average game engine actually just takes the triangulation at bind pose and keeps that so that's where the mismatch was coming from the resulting model was different so unfortunately there's no way i actually know of to disable that in maya but fortunately i found a little hack where you can just throw a poly triangulate modifier to get a more accurate visual of what it might look like in the engine but one trick to getting it to come in before the deformers is you know that that's something that's actually necessary because if you throw it in after the deformers it'll still dynamically retrangulate you'll just get

the hard line instead of the dotted line in that sort of mode so it's uh instead of triangulating the head mesh what you need to do is triangulate the ridge mesh what's the ridge mesh for anyone who might know not know what i'm talking about it's essentially a copy of a shape node that maya creates when you throw a deformer onto it and that shape node kind of serves as an input to the deformer to kind of give it a baseline to offset for the things so if you run an off uh if you run an operation like polytriangulate just on the regular head mesh it you know you can rearrange deformers you can rearrange inputs and stuff on anything in maya but maya will actually block you from rearranging anything that like edits the mesh until uh it still always has to be after any deformers it's probably something that they kind of put into place to prevent chaos and uh and tears but we tech artists uh you know we deal in chaos and tears all the time

so if you know what you can do though is just throw it on the ridge mesh instead and then you can kind of just trick it into coming in before the deformation and luckily the polytriangulate modifier doesn't actually you know rearrange vertices or anything like that so the resulting deformations are fine and here it is in the node editor it's all a little bit more simple than it probably looks but that didn't solve the deeper problem the fact was triangulation in biden pose was fighting our ability to get a good silhouette the triangulations weren't going along the sternal mastoid as we wanted but they were actually going perpendicular which was creating that sawtoothing so along the lines of the hack to to solve the dynamic retriangulation issue i found a surprisingly simple way to kind of feed in a new base mesh what you can do is you can connect a new mesh to the in-mesh attribute of the hidden a ridge shape and then immediately disconnect it and you've essentially kind of done a

convoluted way of doing a set adder to just replace that mesh and as long as the order is still the same you're good so we would create a duplicate of the mesh run a poly triangulate on the offending area then flip the triangles in that area and delete history pipe it into the new mesh and we've replaced the resulting triangle flow with something that's a little bit more pleasing to us i included all this because uh i figured it might be useful for other people who kind of might run into this problem uh but i didn't really want to turn it into a big tutorial or anything but if you do want a big tutorial or something i actually did write one about two years ago when i discovered it in a blog post at the time i was very heavily buried in the land of ndas so i had to rely on placeholder assets and snarky flavor texts but you should be able to you know get the point of all of it if anyone wants to do this so rather than listening to me go into it you can just snap a picture of this slide ask me later go to my twitter or my

website noelzaire.net and it should be easy to find because i'm kinda lazy and don't blog nearly as much as i should so uh this actually came in handy as a secondary way to do other things to edit other base meshes if we wanted you could pipe in a new mesh for heavy thugs because we were deleting the bun shapes anyways and it was just saving those uh as deltas in the deformer so we wanted to give the heavy thugs thicker necks so we were able to just do that and no need to propagate that all out to the hundreds of blend shapes so we did this to a few characters to change small things if we wanted if we didn't want to deal with like you know sending it back and all that this also came in handy as a nifty way to recycle neckbone shapes between characters so you could bring in the base mesh of a character that had already gotten neck corrective blend shapes then you could feed you could import the correctives as you you see here with mj then after that's in

then you feed back in the original mesh and you found kind of a hacky way to to recycle the bun shapes between the two of them and you know kind of the heavy lifting is done at this point and it's just uh you know polish so that covered cinematics but for gameplay those heads were way too heavy to be in the open world for the amount of civilians we wanted on the street the plan was to create a bunch of uh separate but very simple head rigs to do blinking eye looks maybe jaw flap all that simple stuff we played around with a few things at first one notable test that i actually thought was pretty cool was doing facial animation through basically through the shaders where we sort out blend shape deltas as textures and then they just kind of came in i couldn't actually find a video because it was super long time ago but anyways that was like kind of just feeding in blend shapes in a sort of cheap way what we ended up electing to do though

was a simple joint rig like the one shown here and we were going to load those heads and attach them to the bodies at runtime uh like per cycle and uh using our built-in engine retargeting animations could then just be shared across uh across all heads and here you go you can see uh you know the sort of first test at that just you know simple stuff for a kind of face idle however it became clear pretty quickly with the massive amount of civilians we wanted in our open world at once where you could just see a super long distance even that was actually starting to become too heavy for uh for our needs as doing that searching and matching per per frame was uh actually starting to slow things down so luckily our engine actually has a vanity mesh system for keeping multiple mesh groups within one model that can be toggled off and on and then you can set up groups for drawing things in randomly and all that from a predefined set

so we already had all possible shirts pants gloves and shoes in our civilian master files so what we ended up electing to do was have the heads also be in that body model as sort of other toggleable submeshes however it wasn't all sunshine and roses in that case either so ballooning file size aside the problem with this approach was whereas before head models were sort of separate so they could have bespoke positions for things like eyeballs and eyelids and and all that uh they had to all kind of live in the same sort of environment so kind of to uh like to keep the situation we had before we would have either had to move all the facial landmarks to be in the exact same spot or uh just have like joints for every single head thankfully there wasn't much consideration for either of those as keeping all the joints in the head would be just a massive pain to track and just

kind of grow out of control quickly because we had like 20 heads that could be kind of brought in for each of these uh types of characters but then also we didn't you know we we wanted to have a variety in our open world so we didn't want to you know have basically just everyone have the exact same face but just slightly different like skin color and stuff we actually wanted more visual variety so that wasn't an acceptable option either so what we liked to do was create a 11 joint facial rig that only used translate to move the brows lips and even eye blink the way we handled different sizes of eyes and stuff was just to adjust the waiting where the animation was always the same but smaller eyes just basically had less less eyelid waiting and you know only the people with the largest features would then approach anything near 100 waiting on things like the eyes only the jaw used rotation and the lower lip joint was parented under that and you know it just

honestly didn't really cause that much in the way of problems because the jaws did tend to be in a similar spot and again these were just kind of open world people and they weren't necessarily going all like you know snake swallowing a cow or anything like that on us so uh still left eyeball rotation though if we wanted to do it for that we actually experimented with using blend shapes to do rotation i even wrote a script to mass produce them and we were getting surprisingly good results considering that blend shapes only do that linear interpolation what we did was we just kind of got it to a rotation of about 15 degrees and then we capped things at a weight of 1.5 and with that you know we didn't necessarily have them that much in the way of issues and we could rotate to about 20 degrees in each direction and uh yeah it was pretty cool i think but before rolling it out to all characters uh in order to just suss out some things we ran a sort of lighter version of the

civilian rig in a sort of open world diorama so hold on to your seats because i'm about to blow your minds turns out eizel's eyeballs are really small i know what you're thinking gee noah that sure is the industry redefining hot take we came here for sure glad we flew people out spent thousands on hotels and registration and all that it's crazy but before you crown me king of the nerds basically what was happening was we had five professional game devs you know standing around a tv and squinting and we couldn't really tell the difference because they're on screen about that big on a smaller screen typically than we have right here so it was determined from the minuscule bit of benefit we got from it it just really wasn't worth it and uh you know maybe when we have 8k tvs or whatever it might we'll revisit but it's still a pretty cool result i think so sorry before i go i'd like to

quickly talk about batch tools and naming conventions because during most of the production it was just kind of me interacting with the files and it was coming from a company that also took its naming convention seriously and uh you know it was fairly drama-free and stuff then uh towards the end with the tier five rigs uh we actually rather than having me just be that intermediary we had a character artist go through and add uh art to those files to add the hair and all that but as he was doing that i gave him this big list of rules and so i was like you know you can't use pasted textures you have to keep naming you can't name it this this and this i was probably annoying the hell out of him with all those rules but he stuck to them i occasionally even jumped in files and checked just to you know give him that sort of random inspection he was following them to his credit and as we were closing out the game as breathing room became more and more scarce you know we were starting to hit performance problems and uh you know it became pretty clear we had to like

do a lot of uh changing of leds to be more aggressive and things like that problem is we had to do this on 130 rigs so you know thanks to the conventions that we were keeping we shared we shared uv sets and all that kept strict naming and we had a pretty good batch tool that we just kept upgrading through the projects to kind of prepare for a moment like this going through and editing all 130 of them was just a rig it was a script i wrote in you know a couple hours and then just ran over one i ran over lunch came back checked it we threw it in game oh leds need to be more aggressive okay whatever just rerun it again it was a pretty simple thing or you know we wanted to change certain weights on all the characters just all all pretty simple so it basically turned what would have been a garbage week or weeks into yeah just that little simple thing probably would have had to drag in the whole

department you know because you know i think even doing that manually even if i was sleeping under my desk and all that just wouldn't been possible so it's not really much of a story but a story about a story that didn't happen it's like one of those boring stories where every where something goes right but you know while those stories are fun to watch about other people there's those the crazy stories aren't the ones that you want to be in so follow your conventions kids they do work so as with all creative endeavors we learned some lessons along the way and uh so what might we change if we move forward first a small one uh i probably should have from the beginning included a better way of doing these conditional additions for certain characters it was kind of those things where i didn't realize how much we'd be replacing eye eyebrows and eyelashes and stuff and this kind of just kept getting added to the sort of master file and all that uh so probably what i'm going to be doing is just kind of have

stuff where it's like separate scripts for each character that then just gets sourced in according to whatever because well it didn't really cause any issues it kind of just hurt my soul a little bit to keep doing that and there were just so many edits that were just like okay i'm actually making it may remove her eye eyebrows and stuff when they come in because we're bringing in a new set of eyebrows so again another one of those small solutions that just kind of grew and you know probably should have anticipated it so we had a lot of heads that were shared and randomly loaded on different types of characters especially at the thugs and the thing is all of those thugs were then just skinned kind of separately never really kind of followed any sort of convention for no particular there was no you know reason that we thought to do that but then towards the end we were getting a ton of clipping bugs because you know we had these shared head models that were coming in on unlike sort of body models

so in the future i'll probably push to at least try and do some sort of baseline for clavicle weighting so then you know we'd still want to try and be able to support you know changing the weights a little bit more if it just if it's really needs to fit a certain look but either way there has to be a reason to do that rather than just kind of what we did where it's like just do whatever next part way through production i wanted to add those uh combination shapes that i was talking to talking about but you know the way you have to deal with like blend shapes and all that and all the attribute connections and stuff i probably should have found a better way to track it because it was just kind of annoying manual work and like i sometimes had to disconnect things and reprocess them from scratch so i started to add like a lot more just general tracking through network nodes and all that to be able to track everything but either way it needs to be kind of done from the beginning i might investigate in the future saving out a way figuring out a way to properly save out

blend shape deltas rather than the absolute shapes because then things would be able to be shared between characters or if the character model changed for whatever reason there's no need for all those extra steps that sort of method of sharing blend shapes that i talked about before it was discovered pretty late in the project either way like yeah having something from the beginning that would have allowed us to share bun shapes to then just polish on top of it would have probably saved a lot of reduplication of time and effort and uh having a proper plan in place from the beginning for the pedestrian rigs probably would have saved us a lot of uh time and heartache because uh i think we just we didn't necessarily stress tests as often as we should have and yeah again still worked out but you know could be better in the future and lastly i probably need to make a way to empower the character artists to do the updates for things like eyebrows and all that because especially towards the end with like the important characters

it was often you know move one vert you know adjust this slightly change the uvs here okay and then i need to rig it and get it into the game because part of vetting that was actually about seeing it on the animated characters in the engine with the lighting and all that and either way i was kind of starting to become like i'd get these things like two or three times a day it was often just like trello comments or slack messages or just you know walking down the hall to the kitchen they're like hey can you update that thing it's kind of turned into a bit of a a bit of a tracking thing which was like pretty bad and more more sort of damaging was that i was becoming a bottleneck and you know kind of getting in the way of artists so i want to remove myself as that sort of red tape and give them some sort of uh method for importing the new stuff with an emphasis on keeping things clean so yeah something i'm investigating anyways that's all i have for today before i go i'd like to give thanks to

all my fellow insomniacs for helping make a game that's been a dream project of mine i'd like to thank my fellow riggers past and present we share everything and you know we all i think you know if i walked out of here and got hit by a bus we'd we'd be pretty good for a while you know because we all just know each other also the the the shader tech artists the animation team character artists our awesome core department my advisor jeff hannah through lateral cubic motion and our amazing voice and p cast pcap cast and our producers at sony and marvel lastly thanks to you for coming to my first udc talk hopefully i didn't bore or disappoint uh as is obligatory i'll mention we're hiring a lot so if you'd like to come and help us build some really cool things in the future talk to me or someone else wearing an insomniac badge we don't bite and if you do maybe you'll get some superpowers or something [Music] anyways if there are any questions i'd be happy to answer to

the best of my knowledge and legal ability i i have a question okay uh so you talked about how you had to simplify drastically the the rigs for the tier four and five how did you handle simplifying the the one for tier one to three uh so that you wouldn't have all those many blend shapes or was it the same rig in cinematics and in the rest of gameplay um well the tier one through three heads actually all of the heads were usually delivered with the lod zero the tiers one through three had a shared topology and then the tiers four and five had a shared slightly lower topology but then the tier one had all the blend shapes then the two sorry led led zero had all the blend shapes then led one was usually like the exact same mesh just minus the blend shapes then led two was a kind of half mesh

half density one and then uh yeah they shared uvs so in the case where we had to go through and make the leds more aggressive we were able to just use transfer attributes though the eyes were a bit of a tricky thing because you know they overlapped uvs uh so what i ended up doing was just writing something to kind of separate the side to side and figure out the size of the eyes to bring in like new versions of the eyes okay so the the guys from three lateral already gave you the like the second that lod one yeah they they did and yeah it was just basically it was literally like half like you know every other edge loop was gone and then yeah in the case of uh later than we used simply gone to create you know more uh downgraded ones or more more optimized meshes and then kind of bring those in thanks yep okay anyone else going going yeah all right thank you