Animation Bootcamp: Script to Screen: The Development Diary of Marvel's Spider-Man

[Music] all right so welcome to my talk uh this is the development diary of marvel spider-man um specifically i'm going to be speaking more about the cinematic creation in marvel spider-man but i worked a little bit on combat in the beginning of the project and uh we'll just get rolling here so my name is brian weiser i'm a lead animator insomniac games i've been in this animation racquet since the last century been doing this for quite a while i've worked in vfx i've worked in games mostly spent some time in tech and even had a period where i had my own company about five or six years ago i came to insomniac to work on sunset overdrive and then shortly after we completed sunset overdrive sony came to us and said you know we um we'd like to work with you guys on a marvel property what do you think so we're obviously excited um and we let me hit one thing here sorry about this there we go i'm gonna time myself better here and we got the opportunity to choose which character we're going to use

so me being the animator or the artist i really want to do some obscure character but boy was i wrong we chose this guy spider-man so i hope to provide a little bit of insight into our cinematic process and insomniac games i mean maybe for some of you this will be a review for some of you this may be what you do every day but hopefully i'll either give you some tips on how you can improve what your storytelling or your projects turn out to be or i can validate some of the things that you're faced with every day and you can realize that many of us face these same problems the other thing i want to say is there are definitely spoilers in this talk i apologize there's a few moments that i wanted to highlight and really though our game has been out since september and if you haven't played it maybe you can't afford it go borrow it from a friend what i will say is swinging through the city is pretty damn fun and i had nothing to do with that that was an entirely different team but get it try it it's awesome um all right so here we go spoilers walk out if you need to all right so early on in spider-man like i said i was working on combat uh our

animation director bobby coddington he was serving as the main animator and then a few of us rolled onto the project in pre-production we spent a lot of time focusing on comic book imagery as the source material for our posing and i'll step through a few of them here and at one point in time my talk had the exact comic slides that would match up with these images but i couldn't clear those through legal so we can't get those you can't see them you can just guess some of these are pretty clear some of these are similar to covers we also jumped into tests this is the thing i did on the first day of the project i was just playing around with our sunset overdrive based spider-man we really didn't have a character yet and i was trying to figure out ways for us to present an alternate form of combat something that isn't just punching and kicking and along the way in comma there was another test we threw in which i think made into the game the slide under dodge this was actually something that was introduced by myself and an engineer during a little bit of downtime right after a milestone the counter and kick after that i think got cut but at any rate we got some of it to stay in addition as

you guys can see there's a theme here i was really into getting spider-man to climb all over his villains i didn't want to just punch and kick i wanted him to just disable them in other ways that only this villain can to differentiate our project and lean into those differentiators again none of these tests are actually final this is just something i was throwing together so to give you a little bit of background and insomniac we have a system that is based on uh maya we export our scenes from maya and they get funneled into our cinematic editor so this is a view of our timeline in the cinematic and to anyone in here that has edited before it's really incredible for us that we have a tool set that looks like a non-linear editor we're able to slide around shots add tracks add effects add lighting and all the other aspects of our scenes can be easily entered into this editor this is also accompanied by a viewer but i didn't want to show you that it just doesn't run as well and anyway it's not there the other i think advantage that we've added to our pipeline is our camera we've spent a lot of time working on ways to improve the way we shoot

most of this work has come from trial and error and insomniac but a lot of us have come from other studios where we've experienced creating films or shooting for other cinematics and there's benefits to what we do one of the things we're always trying to think about is where is the camera who's operating the camera who's holding it based on the answer to that question it'll define how the camera functions with a little bit of frequency and a little bit of noise to aid in the animation work and make our job faster we don't hand animate all that all the time we have a number of different sliders here that you can kind of see in this initial column and really all that these are are a way to dial in the weights of a number of different frequency or a number of different wavelengths that apply to the translation channels on the camera so for instance shake is something that you'd often use if you were in a helicopter shot where the operator might be sitting there and something that's vibrating and wave is something we use often when we're trying to do a little bit of handheld the other advantage that we or the other

addition that we've added to our camera to help us be more cinematic in our thinking and the way we're trying to tell stories is we've actually mirrored real world lenses from an alexa prime camera package that you would find on a film set this came from one of our incredibly talented animators josh gridley who had joined our team after leading the number of productions at third floor and working on film previews uh so now back to the story uh after about a year or so working on combat our narrative team started to shape the scenes and and the overall trajectory of our story in in a way that we could start working on it so i moved from combat over to cinematics and we started to realize we were looking at about 90 minutes of story cinematics and that's just our story cinematic team that i was the lead of there was a number of other animators involved in two other animation pods that worked on in-game set pieces and massive moments and other mission-related cinematics so on my pod we had 12 animators that does include myself and the animation

director who were very involved in the scene creation we had roughly two years at that time to finish our story and yes it was freaking daunting i mean so many of us were really concerned with the fact that we were jumping into a character that was i mean arguably overly saturated in the market maybe people are bored with him were they gonna like what we made so i was pretty freaked out so just to give you a high level view of our process when it comes to creating cinematics we start with previs most of our animators will jump into uh doing thumbnails or animatics on their own but that's not something that we've actually incorporated into our workflow because in my experience in the experience of our team it's much more efficient for us to jump into 3d as fast as possible once we identify our previews we get to a level of time lock then we'll identify what we can animate hand with hand keying which is a lot of the spider-man work or a lot of the villains that are in the scene or we'll identify the scenes that we need to take to pcap after we get data back from performance capture we'll lay the scene out again

part of that is because we don't want to infect our animators with or excuse me our actors with our previous we in most cases we don't even show them our previous we give them good blocking instruction we tell them where the cameras will be but we want them to help bring about the performance we want them to help us make a better character or a better scene which requires us to lay out again after we get through a number of weeks of animating and working on layout and coming to a final edit animation is finally ready to release that scene to the other departments that will help bring these things to final actually make them so much better than we can create on our own so we lock the edit we release it to visual effects lighting and audio and score and anyone else who has to contribute to the scene and finally concurrently now at that point we're polishing the scenes we're fixing bugs and we're banging away on many other scenes so previous generally speaking in insomniac we assign one scene per animator and we try and cast these scenes out to animators that have strengths and weaknesses but we're not limited to that we don't

want to pigeonhole people so we allow them to try different scenes we iterate exhaustively on previz especially on i-20 we would meet twice a week where we would gather together the entire cinematic team and sit for an hour hour and a half in a room and all go over the scenes provide feedback a safe place to weigh in on what we're working on this also helped us because we didn't have a singular person shooting them or editing them so it helped us to remain connected and the other benefit of jumping right into previews is we're readily able to add these into the game like this little joke that our animator threw in there at the end of the scene where we didn't have all the pages um once previous is built we can put this in the game we can identify hookups or as many of the people in this room might know who work on cinematics that's the point where the character leaves gameplay and goes into cinematic and then leaves cinematic and goes back to gameplay we spend an incredible amount of time just focusing on those little bits it's actually a disproportionate amount of time all right so as i said the animators in our team are cinematographers as well so we spend a lot of time working through previz and when we get back into

layout thinking about how we can use the environment to help tell the story uh we're thinking about depth planes we're thinking about ways to really work in old 2d animation styles and as you can see here in the bottom two images so we had a shot with peter or spider-man suiting up in the very beginning of the game and in the one on the left late on the project we realized was a pretty flat shot there wasn't it was a lot of information but it was equidistant from the camera so we just threw in this bed to give us a little depth of field element in the foreground and that i think plus the shot one of the other things we do in my opinion is we shoot scenes overall twice the first pass is to get comprehension it's to allow us to build something that makes sense we're not crossing the line of action unintentionally scene geography makes sense you know where actors are and then we take another pass we jump back in and try and infuse subtext as best we can and that could be altering compositions moving the character to the center of the screen or offsetting and having them look into the short side of the screen to let the audience feel awkward or looking for frames within frames or

other opportunities and again this is a massive art form in and of itself i won't go too deep into it but it's something that so many people on our team are really working hard to do in addition to being animators so another thing we have to shoot a lot of dialogue scenes when it comes to the story cinematic story cinematic pod and it's some people's tendency to want to shoot the words that are coming out of a person's mouth and i want to raise the importance here of reaction shots reactions of characters within a scene help the audience understand how they should feel about the scene they help us understand the emotional journey that a character is going through i say no dragnet because that's an example that i found in a walter merch book where he describes that old show i don't know if anyone in this room knows that old show um but the way that they would shoot it is they would point the camera at the speaker and it ultimately felt like a tennis match you'd see one person speak and they'd cut and put another person they cut and they put another person and they cut and you didn't really understand the emotion of the scene and i suppose it

will work for a factual based uh story but not necessarily something we were working on so this scene again big spoiler you know what i'll pause this if anyone does not want to watch it and go back but anyway go back to that again sorry about that this scene was actually the topic of conversation we got feedback you knew that i was told to put the camera on on may when she's talking to peter i want you to worry i did and i and that's just one small example where i mean it's small but i think it's important in that scene where we understand peter's journey we feel with him and if the camera was on aunt may it wouldn't have had the same impact on the audience so reaction shots are important so the other thing that our animators work on is editing where animators cinematographers and we're also editing scenes and just to quickly gloss over some of the things that we think about we

certainly want the edit to mirror the emotion of a shot of a sequence so in these initial shots this is a conversation between peter and mj and it's a very low tempo calm scenario and we give you long shots we're not having cut it we're not cutting around this very frequently we're not affecting the audience with how fast we're going from shot to shot we're trying to allow the rhythm of the shots themselves to support the action in contrast what i'm about to show is a fight scene where as you can see these shot lanes are distinctly shorter we're bouncing around very quickly we're trying to get you to feel the chaos of the action and in addition we keep the camera moving and really just kind of keep the bass rolling and lastly to talk about editing there's the power of discontinuity so sometimes within a an action moment you want to show an action twice so it heightens the importance this is seen often in john woo films and jackie chan films where in one shot they will show an impact of a fist hitting the face and

then on a cut the next shot will show three or four frames prior to that impact so you're seeing it twice now the audience doesn't catch that but it adds importance to that shot it adds importance to that action and it helps to sell what's happening in the scene so in addition to editing when it comes to time close but i'll let this play i'll keep trying i will find a cure i will i love you son so our one of the first things we do when we assign previews is we record temp dialogue we get our animators in a room and we record the script as best we can some of our animators are better actors than others i am not one of the good actors uh this animator alex zimke is actually fantastic he's a great animator and he's a great read he's in so many of our previous moments but the point that we want to get at is previous for us is so much about selling the writers ideas and selling our ideas to the broader team and we can get buy in to build these moments to quote walter merch an editor

who worked on so many incredible films from apocalypse now to a number of other things sound is a huge influence on people's attention and with that in mind we really try to make sure that our previous contains scores sound effects vfx and really anything we can do to help sell our ideas better um so in a mocap based game as anyone in this room can attest to mocap is definitely not everything even when you have a scene where it's just two people sitting at a table talking well there might be a moment where they got a kiss and it's kind of impossible with the hmcs so laura bailey and yuri lowenthal fantastic actors in their own right but this is a perfect example of where you know mocap is not just going to slide in the game it just doesn't work animators are needed don't worry anyone who animates here your job is safe so something else that we do a lot on spiderman that i think is a huge advantage to the way we're telling our stories see

here we go is we tend to shoot some of our action sequences with two actors on the stage at the same time but delivering a singular performance while this takes a little bit more time in our prep and our walk-throughs all right we'll pause that um what it gives us is a unified performance with voice actor and stunt actor and we allow the experts to bring to the table what they're good at um you know i yuri's fantastic but i'm not going to ask him to do a backflip on the stage we need him we know he's too important he's going to get hurt but what this allowed us to do and many of the people in this room probably understand when it comes to animation and cinematics you'll often have 80 yard lines or lines that are added after the fact or the script was changed and as animators we need to go in there and add head accents and connect those to the body this is a very quick way for us to capture and get the complete performance all in one it also helps us to avoid stitching which is something that you have to do a lot with mocap where we would have if i was to take this sequence from our black cat dlc you might have a moment where we're

working with the performance actor and the stunt doctor comes in and the performance actor comes in and then we as animators would have to blend those together that's just a big pain in the ass it's so much easier to have all this together so another thing that our previous allows us to do is innovate and solve problems early so this was a shot that early in our very first cinematic we were putting into our vertical slices actually the first previews i put together on the page it said a lot of crazy action happens truck gets takes off and peter's going to chase the truck and we leave peter on the rooftop sitting in a perch position so as an animator i just kind of didn't think that was right come on spider-man's never going to wait he's going to chase after this action so with this i pushed for our ability to blend two blend from cinematics to gameplay on the move um early on you know a lot of people blocked at this and thought this was a big concern and maybe we shouldn't try that for our initial vertical slice long story short we now have a system that allows

us to blend into many states throughout the game most specifically falling and in the air which is actually quite easy for us there's no contacts and we do this all the time now all right so the project's going well everything's fine and then you open up your shocker scene and this is what you see or you go back to your mj scene where she's in grand central station and this is what you see and we all know gameplay is messy so when otto and norman talk this might also be something you see now i'm just pointing this out because as we all know there's a lot of problems when it comes to creating games there's so many of us involved things change every day no scene stays the same trust me the scene looked better when we shipped it but to solve this one of the things and i'm going to talk about a meeting i'm sorry everybody we actually introduced a meeting midway through production where we realized that most of these problems come from communication gaps so we have an assets meeting now that's every week

and what what happens in that meeting is i sit in the front of the room and i go through our previews i go through anything that's in production and we talk about it with all the leads of other departments to identify holes or identify areas where either they don't have the bandwidth for this coming down the pipe so we need to change course or they can introduce their ideas early on and we can get buy-in and finish our scenes together so yes meetings are important one of the other things we have in spider-man is many costumes for peter when he's as spider-man um in general the narrative team in writing helped us to avoid many of the problems with him where we had to remove his mask and it wouldn't have been very easy for us to do that with all of the possibilities that he has for his costume but it wasn't limited to our hero so we had this moment in the game dammit where spider was to stow away on mj and this spanned a couple of different missions and it was through a number of different cinematics high alert and once that spider got to the feast center it was supposed to

leave mj as you can see in this incredibly animated scene by one of our animators liam murphy and then it's going to go on and bite somebody else i won't spoil that even though i spoiled other things here however this is what we saw again when we went back to the scene later on because the designer rightfully so had put in the proper suit and the spider's gone so the solution to this was simply brute force unfortunately we had to assess the problems we had to go to each of those scenes and any any other mission or anything that's related to those moments and identify where the least work would be needed and then we'd fix that uh moving forward our lead animator gavin goulden essentially mapped out all of the clothing that any of the actors can wear in our game and we had a kind of a timeline of their life so this is something that while it becomes forgotten pretty quickly you need to remember that continuity is important all right so another problem when it comes to creating cinematics is it's rare that you're aligned with the rest of the game so in this particular moment we built a

scene that was solely based on the concept below we shot it we had this great idea that spider-man was going to be tethered to the ground it was going to be really exciting and then midway through production we realized that gameplay might not even be building that we may not even have a tether it's not a thing that'll work unfortunately we had to pivot and we took what was a part of our stable agents bola that worked but i guess the point i'm trying to make here is cinematics themselves are often challenging to keep in alignment with mission development and you just need to remain aware and communicate with your teams scene rot is a term that we've coined for when you leave a scene alone and something's going to happen and this is no one's fault it just there are so many of us making these games so many things that are going into them that something is bound to change inadvertently and it will affect the scenes as we go back and edit them so you might have a moment where spider-man was supposed to be on a building you see a little spec there in the middle well there's no building anymore or you might have a moment where one of the stable agents is also supposed to be on a building there's no building there or you know

our villain after a huge climactic moment was supposed to look unconscious in a helicopter but there's a chair going through his chest and finally i think one of the most egregious of these ones that i'm highlighting you saw this a little earlier where spider-man swung through an open space in a billboard well one day we opened the scene and this is what it looked like to our team's credit though the moment these are identified everyone jumps to and we we fix these very quickly in all departments is this great support and insomniac so we've had a bunch of problems we've developed a a workflow we're working pretty well we're starting to rise above and and fix those problems and we get to the point where we need to start releasing these scenes to other departments and and finish them and finish the game and it's at this time in our production that our animators almost become or they're also polishing animation finishing their work they become shepherds for their own scenes and they're communicating with others to make sure that the remaining details are are completed this scene in particular epitomizes the

the collaboration with other departments and other people to create a performance so we have an entire fight scene that's on a rooftop at the end of our raft mission and he said i would say there's good idea not a single animation in this moment that doesn't include the work of a stunt performer or an actor or an animator completely changing the performance in fact there's a moment in the beginning of this where uh spider-man leaps off the scorpion's tail lands on scorpion then lands on the ground and i happen to be working with one of our stunt actors just the other day who was involved in this and he joked to me he said he was playing through and he was watching the scene and said well there i am oh no that's not me no no that's not oh there i am again and just kind of identifying how much we kind of work together to create one performance arguably i think it would be great if we could go back to crediting the way disney used to where they'd have actors and animators and anyone else involved in a character credited on the same slide all right so one of the other problems that we have in a massively well in an open world game is

we will often have scenes where we want a continuous continuous narrative to pursue to to move through a number of different locations that are nowhere near one another on the map this would have been completely unacceptable to have loading screens between each of these vignettes so we kind of started building these frankensteins we work closely with the art department to identify all of the parts we need kind of pull them together and make a unique set that we just throw underground for these cinematics so while ideally it would be in the open world it was just simply not possible and through their support we were able to complete these in a way that didn't really ruin the game can you um here's another example of a frankenstein where we go from the fee center to mj speaking at a desk all the way to norman's apartment but in the interest of keeping my talk moving along here maybe i'll get to some questions i'm gonna kind of skip through some of this over all transit points throughout the year so not only is this frankenstein notion one way we hide loading but sometimes we can't do that the thing

we want to hide that's getting loaded is far too massive and the ins and specifically at the in the beginning of our game spider-man shows up at fist tower and it's going to help the police go in and take down fisk so we wanted to go from open world directly into this entire instance of our lobby and the entire building with this tower do you see the cut see where it happened someone there anyway so our goal is to again work directly with all of the other departments involved in creating anything for our game we have to collaborate to make this work this was really a long process in fact to the extent that people told me we shouldn't be doing any of this this is not a good idea but we work closely with design and lighting and effects to identify the singular frame on which we are going to teleport peter from being within the open world to going to the instanced interior that he's now going to continue on the the bigger challenge we have here is constraining our cameras and working with environment to provide

visual blockers but again this is something we hope to do more of in the future it is our goal that games at insomniac ultimately eliminate the loading screen and we'll see if that works i don't know it's a big challenge facial animation is another challenge that we have sorry that just burst the mic um so when it comes to facial animation is critical that lighting is involved as early as possible change in six months so when we look at these scenes why did you ask me here pete you can see a dramatic difference between final lighting and even a passive lighting that was pretty close to final to the extent that as we're reviewing facial animation that comes from cubic motion incredible team did some incredible work for our project can't thank those guys enough there's a lot of baggage you can see the distinct difference i i might not have approved something with that near final lighting versus the final one head of stable international so really what this boiled down to at the end again working with another department we get into this cycle where we would get first pass animation air has a pet army now next time you get in my way past this i

will not be we'll get first pass animation and then we'll get first pass lighting in and then we'll go back and get more fix the animation we'll adjust head and neck because in our facts based facial animation system there are blend shapes that'll emerge if the head is pushed back too far things we can't see before the facial animation goes in there and then lighting will light again and so on and so forth until we kind of arrive at this final that we can improve and feel is successful and ready to ship and i can change in six minutes uh one of the other challenges that we have in our game is the living world um we often had scenes that were in the exterior through which we had to contend with the fact that we have this incredible system of vehicles and bots that is a technical and creative uh achievement if anyone notices that's my head actually getting beaten down there but at any rate so within these scenes it would be impossible for us to shoot this as an entire cinematic to contain every character that we have here within our scenes and actually work

with it there's no way maya would actually even function so we would break these down and try and leverage the in-game systems as much as we can to tell the whole story and there's a shot that comes up here that can better articulate what i'm trying to say let me pause this one so this is a perfect example so within this shot our cinematic is constructed primarily of these three actors with miles morales rio morales and j.d morales and then we have some mid-ground characters that are coming from a secondary cinematic that is funneled into that same one and then we also have other actors that are coming from the living world so it's in combination with all these other departments that we can create this one scene otherwise there's no way that we could handle it and the scenes would no longer be performing so one of the other things that we had to attend contest with is see screens within uh our scenes so often on the page it'll just simply say and peter walks by a screen and sees x those were often their own cinematics we had to actually build those out as

their own thing and it was a challenge and often forgotten because it was one of the things that would either fall on effects or lighting or no one really claimed them we have another one here where norman is supposed to be speaking on a tv the entire time if ever you see that in a script go back to your writer and make sure they realize they need to write that whole speech whether or not you're looking at norman he should be talking the entire time and in addition we have other reveals that are on screens again these are their own cinematics that were unaccounted for kind of came up in production we had to throw together so in conclusion i hope that uh you enjoyed this first of all um secondly i think in insomniac we've leveled up our storytelling techniques and our abilities to uh give the audience emotional journeys i think cinematics has had their challenges i think i've highlighted a few of them i'm sure they're i mean i don't have enough time to go through all of the problems and and issues that we face and many of you i'm sure can deal with that here too or under can speak to that as well

and lastly i hope there's at least a little bit that you can take from what i've said here and maybe it validates what you've already done in your studio or you can use a little nugget of information to help improve the way you guys tell stories with more confidence in the future and that's pretty much all i got i'm open to questions at this point but thank you all so much for coming to my talk thank you it's great my kids are gonna think i did a great job all right so i'll take questions as long as i can and uh i know there's a another room that we can walk to if there's more questions after this so feel free i can take one question all right you sir hi so uh where i have two questions and they're somewhat related to each other okay where in the process do you loop in like the downstream teams like lighting and effects and sound into what you're making for a cinematic

and then you mentioned that you like to include vfx and sfx in the previous process those your animator is like doing temp versions of those assets or is that after you've looped in the other teams and they're providing temp versions so the question was where do we include the other departments and are the animators doing the temp versions so i'll answer those in the inverse order so in previews the animators are putting those in but we're not trying to drive the other department's work we are not the experts we're just trying to again like i said sell ideas and get people to buy into these scenes um when it comes to when we engage other departments we try to do so as early as possible they see scripts they see what we're working on but um with the introduction of that asset meeting that i mentioned we actually engage them in early previews now we show them the things that we're working on they're not actually actively working on the scenes themselves but they at least understand the work that's coming to them and have the opportunity to you know tell us whether or not they can support that or introduce new ideas that will help us and improve the scenes in general we are not the experts in those departments at all most i mean myself i have a history in

animation and so does my team and we're branching out into other things as we develop these scenes cool thank you thank you thank you everybody