Polishing Uncharted 3

[Music] okay all right I guess it's time for me to wow that's echo alright big room alright Sartain welcome to the talk this is the last 10 going from good to awesome or as we like to say at work if you're a South Park fan my name is Benson Russell and I'm a senior game designer at Naughty Dog and just a little bit of history on me briefly I used to work at EA Los Angeles on the middle amount of franchise I worked way back in the day at ritual entertainment on a Star Trek game and for those of you who've been in industry way long time I was a founder of 2015 where we did the original Medal of Honor a light assault for PC I worked on both the Uncharted - all Uncharted games and I'm from China - and three I was the primary and combat designer so just quick notes don't forget the little email thing it's gonna send out you know rate me however I swear to God I will not buy your vote and nothing is to talk I have tons of information I'm gonna go right to last minute so I won't have time for questions but if you want hit me up outside wherever and we'll go from there

so let us go on with what this talk is actually going to be about no don't do that you stupid thing alright I love it my notes crap up on me so quick overview the last ten you've heard the phrase last 10% takes 90% of the work also refer to this as polish and at Naughty Dog we believe this is what can take a great game and make it absolutely awesome and to us it's arguably one of the most important aspects of the game development cycle but in our experience it seems to also be one of the most marginalized aspects of the game development cycle but I'm here to be able talked about how the fix can be easier than it might seem it's basically it's gonna involve the change in mindset what you're going to what you need to look for the level of detail you want to aspire to and then also a better distribution of your development time and pretty much a good tell the industry for us is every person we've hired at Naughty Dog goes through the same process every game when we go into the polish phase the new people are like you want to fix what you know it's like literally that level of detail they're like you're nuts what we don't fix this before it's like well we do so

hence it's kind of at ellis house that now maybe more the industry isn't doing this as much as we kind of would hope they would all right so this also does talk to me about how we approached polish on uncharted 2 and uncharted now this quick disclaimer the first thing to note is that this is by no means a perfect system this is what works for us this is also the system that we've been refined we're continuing to refine over time in fact it it's controversial internally like when I was talking people getting information for this talk or asking them what they you know what they think of it yeah I got a lot of these that's not right we hated that why did you want to talk about that that's crazy that's not what we did for policy with your stupid you know so yeah even internally we have a little bit strife over it but the keep Roach and takeaway or the key takeaway for this entire talk is that you need to allow time for polish and don't just let it sit in the rest of the mix and I'll go over that a little bit later so what am i defining as polish well the first thing is it is not about iterating and finding your gameplay mechanics you should have that known by this point in

the game this is about wrapping it up and getting it ready for for public consumption you want to finalize that product and just like you do with a car when you polish it it's the act of removing the blemishes and the imperfections anything that can take the user out of the experience and make them oh no longer immersed some high-level examples of what I'm talking about my slides work here there we go gameplay related issues and this can be anything from like say glitchy controls you know are they frustrating can the player easily what translate what he wants to do into his actions into his avatar can also be wonky world interactions like bad collision so you know artists love to make like hey I'm gonna make collision around all this little tiny wonderful stuff and the player comes to walks along with his feet I k and his feet are danced in the air and then he turns into the physics ball of goo that bounces off into space and you want to go smack your artist in the face um it can also be like difficulty controllers or sorry bad difficulty spikes and pretty much all those translate to controller into television screen which is what you really want to avoid it

could also be visual presentation issues like a confusing interface or your menu items clear and concise can they find the data they need is the HUD overcrowded you know does it do they can the player get the information needs at any time it could also be anything from like bad animation pops or blends or awkward timing which is exactly what I'm doing right now in this speech so and the third thing I kind of group into a category this we as game developers seem to think is called that is quo and the litmus test for this is if you sit there and you go it's a game don't deal with it no if you say that about anything you probably need to actually put your head down and be like well maybe I should actually consider fixing this all right so our approach that we took on uncharted 2 and uncharted 3 well the first thing you have to do is you need to set the quality bar really high and this is what I mean specifically about a change in mindset and I'm gonna I'll go over here in just a little bit some examples you want to shoot for perfection don't settle don't sit there and accept something as the status quo don't you

know if you see a little bad pop or a blend or something take the time to fix it you know as long as you give yourself the time which is well another thing I'm gonna go over here all the small things that you think stop doing that all the small things that you just let slip through yeah you might be able to have one here and one there and you're shipping product the more of these small things you have it just chips away at the overall feel of Polish in the game like even if on a subconscious level it's just the player is gonna realize kind of feel awkward like I'm not really playing a completed game here so why do I care see and now it's for some video examples and I've got some better ones even later but these are just wanted at least spread them out so you maintain your interest I'm gonna quiz you on it so this first example is from uncharted 3 it's where we're training the player how to do pull around stealth kills in the under London Underground the guy who want to watch in this video is cutter he's the one on the left take a look at what he does we got ninja mode attack yes kill yes and oh and go

ahead and desecrate his corpse why don't you yes just stand on him you stupid AI that phrase wasn't under enough so what it should look like when it's actually polished is this ninja mode again hit the button and just like they say in tennis and golf a great follow-through it's exactly what you want and honestly that was a real simple fix it was just a minor lineman script that basically was to when they were playing the animations on the two guys as he was going through his swing the animation on the the guy that was supposed to die ragdoll him too soon so he didn't get any moment to follow through with the rest of it and it was just a small line and change and fix made it that much better all right game blitt our camera glitches now this one I might have to play for you twice because every time I've tested this one people they don't notice it see if you find it that did you see that pop I will play it again watch as the camera tilts wink there

that little glitch fix that it's just it it makes your game just feel awkward and unpolished in fact well I got the video so might as well show you what it looks like when it's fixed if it see no glitch much nicer much smoother and again this was an easy fix in script it was basically the camera was playing an animated path and we have a we have a camp on command in the game that pauses the game puts the menu up and says wait for this button to be pressed for training and so then what happens is is even though that command was issued to pause the game the script would actually cute one more line of code because of the way the engine works and thus the camera started moving so it's just a matter of putting an extra like weight one frame in there to to give the script enough time to get that buffer in there and then it was not a problem all right now this one is a sequence of events and I'll just let this one play and I'll go over it afterwards what are you doing Elena why you standing there what are

you doing just get the door you stupid AI geez so in that whole sequence you can I don't you could see there was a there was an awkward dialogue moment where as she's walking over telling me we need to go to the door and Nate goes yeah good call but her dialogue stomps it so they're both saying things at the exact same time which is difficult for the listener for the player because now you have to kind of filter out what was going on what did she say that I just say response and then she decides to walk over the door and is like oh yeah this sorry you want you want that oh yeah why there was no reason for that no polish version of it should look like this [Music] I could call help me with this and she goes right to it and there was no stopping of dialogue everything's wonderful orders restoring the world peace in the Middle East yay so that was that's what I'm talking about with the mindset change I mean some of these these might seem really small I've got some even smaller ones for you that'll come later on the

presentation Evan is Evan wells our co-president is the master at breaking things and finding little things to polish and oh my god we'll go on all right so that's what we're talking about in terms of polish now let's talk about how we do it so the first thing is schedule now it's easy to say this let's say this is kind of a representation of what a standard development cycle is at most companies you've got pre-production a very long production very short alpha a very short beta very show gold and everyone's happy we hit gold the key thing to note though is with a short alpha on a short beta most people tend in their studios tend to be like alpha is like almost a gold candidate it's like final art near zero bugs you know everything is polished and ready and wallets it's fantastic the problem with this approach is people just leave polish into the production cycle because they don't actually schedule for it so you're just like oh you've got this much time to work on your level make it as awesome as you can before alpha but of course in reality what happens is is the hits and everyone's running around with chickens you know headless chickens just get the

game done you know get it out the door that's what everyone starts falling onto and hence polish falls by the wayside so what we do at Naughty Dog is we change it up a little bit we actually use our alpha and our beta time for polish instead so we reduce our production time and then we specifically say we're hitting alpha in fact an uncharted three it was four months we gave ourselves and in future projects we want to give ourselves even more time because we felt that wasn't enough but where did I go I lost myself in my notes pardon me for a second so if you for those who want to put a number on it if you want to polish in the alpha beta phase like we do allocate at least 20 to 25% of your schedule one gotcha in this approach though is you kind of want to hit a solid alpha as much as you can and internally this was one of the controversial subjects of like we hit maybe sort of kind of an alpha well maybe it was an a very least you know not quite the Olfa so now there's a quick side note the one of the one nice thing about shortening your production schedule is it gives you

more time to make an evaluation on difficult things earlier so if you have systems or parts of the games that are not quite look like they're not going to hit their schedule because you've reduced your production time and you move that out you can say oh well maybe we need to cut it maybe we need to trim it down or eliminate these features or whatever but it allows you to self edit you know and go for quality over quantity and that's something that we feel is very important all right so back to this whole scheduling fiying in order for this to work you have to not fear change most of the studio's I've worked at in people I talked to you the moment you talk about like get it alpha beta the standard practices do not touch the hell's wrong with you I'm trying to shift this game you're gonna break it it's like well no you're not if you give yourself enough time and you allocate accordingly you can actually apply your policy in alpha so what I'd like to propose is let's a little bit of a following change in definition when we go through this stuff so pre-production is still you know you're prototyping R&D fleshing out your core production is still the grunt work of the game when you get to alpha though

our definition of alpha is the game is playable from start to finish all mechanics are implemented but they're in a rough form puzzles and systems and gameplay like combat exploring sections even the art they're all in there or they're in a rough form into some way and then you're not really concerned about your bugs you cuz now is the time when the beta testers are going to start playing the game any bugs up to this point where pretty much you know are you next to useless unless you're you know having an e3 demo or something because of course it was still in production so and also at this point stability is paramount so Class A Bug's should be top priority any time the game is broken it doesn't matter who's on what whose job it is to fix it they stop what they're doing that you go fix it so that way everyone could keep working and then we're gonna get to beta our definition of that is the majority of polish has been applied levels should be looking and playing really well of course the game should be really stable because you're about to dim put the thing out the door and as many critical anti-party bugs are pretty much fixed and at this point you're also like bugs

are being waved it like lower bugs at this point are totally being waved it's like no we're not fixing that polish can still happen but there's an approval process that we had to go through and I'll tell you about that a little bit later as one of our little tips and tricks section now one thing from the controversy that talking people inside the studio is we wanted to add as this hands-off section so the idea is that all right so we did this work and during alpha beta yes there's a chance that a bug might have been introduced this is where the entire game is beta and it only is in the hands of the programmers and the testers you're just playing the game looking for anything that can cause a major breakage and that not way you're not polishing anything more you're not fixing bugs you're not doing anything you're not touching anything that can cause more bugs it's just and actually we were lucky enough to find one and uncharted three that was a really nasty level loading bug that would have caused a crash I'm probably 20 to 25% of the system's out there it was a really hard one to find so it was kind of like yeah hey thank God and then of course you had gold and everyone's happy so then we get on here rolling demo sorry that if that's the scheduling side of it now I'm going to talk about some of the

tips and tricks that we use on how you know we manage all that stuff so the rolling deadlines one thing that can really fight against you when you're working on this wonderful magnificent opus that you're creating is the charge and tourism that you have made it is huge it is relentless it wants to suck your soul so I mean you get to this problem of like hey personaiiy needs to talk to person B to get his stuff done but person D is doing person B is doing something for person C because he's gotta work on a different part of the game and then all some person D is pining for person a and you know nobody's getting anything done at all so what we like to do instead is we take the game and we split it up into sections specifically for uncharted 3 it was three sections and each section will go through its own alpha beta gold period and then what we do is we stagger them on the schedule so the idea is that if we have that four months now I talked about the ship to get the rest of the project done we split that four months into the across these four sections the sections that are most ready to be completed during section one the

sections that are next most ready section two and the least ready go on section three so you'll notice section 1 hits alpha section 1 is dedicated all to itself until there will be a time when section twos alpha will have to kick in and there will be some overlap and that's fine but at least you know if you're working on a level in section 1 you have the artists and the animators are focused on you and same with programmer time that is the priority there getting that done right now and the same thing as you move through the rest of the sections big gotcha on this one though is watch the overlap this killed us a little bit on charter 3 hunts and other reason we want to say we want more time in the schedule to push these things out even further yeah in actuality our section 1 didn't finish until about the end of section 2 did I say that allow crap so tell me leave it up so one other nice thing that we use for this is we use bug count totals to go with the sections in terms of alpha beta gold and what I mean by that is so level hits alpha bugs are coming in you're just you're balancing polishing and fixing balancing polishing and fixing at some point Kristof sends an email saying okay this

is your new big this is your bug count told you want to have no more than this and it's about halfway through alpha and then so now you want to have that for less so that helps you focus your attention of like oh I need to make sure my bug count is at or below this and then as you get further than alpha that number tricks again you go to beta shrinks again halfway through beta shrinks even more so the concept is is that we can sit there and see like okay who's consistently below their bug count and his above well maybe somebody's getting overloaded with bugs too much and we need to have somebody who's got less bugs help that person and you also can track your game as a whole to make sure that are we going to finish this thing on time in fact here's a little bits of data that might be interesting the chart on the left is this is an actual snippet of what the bug graph would look like that from our bug database that Christoph would send out every day the blue represents that at that phase in time that's your bug limit and if you're in the red that means you're way over so obviously Joshua is having a bad day at the top of this chart but it also is a sign that he was our cinematics lead so he was getting tons of bugs for the entire cinematics team and he had to

dole them out so maybe it's better if we get one of the guys on the Cinemax teams to start fishing through the bugs for him and find his own and you know or come along and help him take them off of his plate cetera now the one on the right is just an interesting side note is we also when we were in alpha we did we started in focus testing every week we had focus testers coming in playing the game once a week and we have automated data capture of like you know per breakdown over the task how many times someone died and this was a little spreadsheet we put together it's called the attempt Canaries so anything that started creeping up above more three or more deaths like either the medium and the median or the average then that's something we as part of the Polish process we would go back in and try to fix it hence the task my favorite RPGs as' is looking a little tight right there so yeah keep talking with the graphics on level RP Jesus please so anyways tiered approvals this is the other this is another phase another tricking tip that we do and one this is kind of what I was going into earlier is like we have a group of key people that

are high up that you have to get approvals from it pools from at certain points in order to get anything fixed or even polished the key people in this case where Evan wells co-president he's our main highest designer in the company Christophe Palestra he was the copra other co-president highest programmer in the company and then the game director adjustment Richman and the nice thing is you want to pick people that have an understanding of the technical the end also the quality bar that can end they have to be high enough to be able to be decision makers so that way they can sometimes have to duke it out amongst themselves to whether or not does quality take precedence or does the risk take precedence and therefore we don't do it and this goes as the Alpha and the alpha beta gold stage has happened the approve this is where this kind of goes into play so at the beginning of Alpha you're fixing your polishing whatever whatever whatever but halfway through alpha that is there's going to come a point where it says okay from this point on out if you're gonna do a polish pass you have to get one of these people's approval doesn't matter who just have to get one and then as it goes on later like especially as it approaches beta now you have to get to and then once you're getting near that hands-off face that's

gold then you have to go all three that's what the real fun begins gotta bring in like chocolate cake for one guy and a cinnamon bun for this guy a little bribery all this goes a long way but anyways this system helps them mitigate risk because that we you're all running around like chickens with your heads cut off you know you it's people that are that are up there high enough in level that they understand the balance of shipping the game but making it awesome and they can make sure that you know the right balance is restorative like yeah we can waive that bug or like no we really should fix that that looks ugly and again the IRS didn't make sure you put the right people into approval okay and the last little note to go over is what I call the all-seeing eye so again while you're running around like headless chickens I love that phrase headless chickens so make a game with about headless chickens please I'll download it right now alright the all thing I is everyone's crazy at the end of the game we have a high-level person in our case it was Evan wells looking for Polish daily so he understands the quality bar we're trying to hit he understands the the trade-offs that need to be made but he was the one who's going through make

sure nothing slips through the cracks this person must have a deep understanding of what to look for very important and it's also important to note I'm gonna put these two up at the same time it's not a dictatorship role even though he could use the president of the company he'd come at any time and say you will fix this right now like the hand of God and what are you gonna do say no well he it was never a dictatorship role he would once a level hits alpha he's going through the game every day constantly looking for problems in that areas we have video capture inside of the game where you can capture second in fact the videos you've seen so far use that video capture he would grab the segments ship them out an email send them to the appropriate people and if they were important enough we'd go get them put on the bug list to make sure that they would get done but the point is you would have to polit you at the balance in fixing those with fixing your bugs and the other polish tests you have and it was always a discussion you could go to him any time and say I don't have enough time to fix this how important is it or you could say you find out this is a really difficult fix it can break a lot of crap what do you want to do and then it's always the discussion with him he's like

well I really would like to see this fix can we find a compromise and you know you'll work it out he's very he's very approachable but it is like I said a critical role you must have to catch things in the craziness now these three videos are actually from Evan I have no polished version of these because the fry had to dig back for an email friend's email to grab these and I'll play these more than once because these are a little crazy to see so [Music] did you catch that I'll play it again look watch the Drake as he wigs out right before the water comes the they're like that crap it's like even I've been at this company for since uncharted 1 and even when I got into uncharted 3 I started like oh yeah my levels polished and I started getting videos I'm like oh that's right we do that I forgot so I mean it's freaking yeah it's crazy this is another good one we call this the one frame off category a bug did you see it big white flash right

there in the door and now this one turns out because you're in a fixed can't a pivot Adickes camera up here and then you're into another animated camera in the sequence between the two what happens is is it was another one frame off bug where the animation I think finished a frame early so what happen was is the camera said oh I'm done here and it went all the way back to wherever the hell the player camera was off in world space for one frame and then it came right back for the next animated camera hunt and causing that white flash because it could've been a black flash or whatever but yeah that got a little craziness and nice little just give example of a render pop see the world right there it happens right near the beginning just watch the background how the light changes weak that like that's another one of those things he's like hey this looks bad go fix it and I have I mean I hadn't if this was an hour-long talk I could fill the entire talk up with this I mean we have thousands of these videos of like really you fix that I'm never walking about at Travis's desk and he's walking around in the sand and he's trying to show them this little

shuffle stuff that he's doing and I'm looking at it and Travis is looking at it going can you point it can you do it again we're not saying what are you seeing here it's like literally that's the craziest little miniscule detail but all of that helps to add up and make your butt and make your game that much more solid in tight so quick takeaway on all this well I'm actually slightly ahead of schedule that's what nervous nervousness will get you last 10% is the hardest but it gives the highest level of payoff you know and schedule your time for polish make sure you put it in the schedule I'll just put it as part of your production time saying no no we're actually going to whether it's throughout from beta or you want a dedicated time for and you cut your production cycle sort and say no now we're doing nothing but focusing on getting this thing as rock-solid as awesome as you can if you want to do the alpha-beta thing like we do at least twenty twenty-five percent of your schedules what you want I just said that a rolling schedule you can use those to help focus your team tiered approval systems to help make sure things stay in check in the right balance of what to fix versus what not it's taken care of the dedicated

overseer as we just went over and then most importantly you know aim for perfection set that quality bar high don't don't accept the status quo just accept something because it's like that's the game's the industry everyone deals with it their players silly players and that's pretty much it that's me down there and then a quick pimp for our HR person we are hiring she wanted me to put the slide in if you contact jobs at Naughty Dog comm or you can follow Candice Walker under Twitter it's at Candice underscore Walker and one more quick note we have two more Naughty Dog talks in this very room later today the art of Uncharted that is from our art director Rob rupal at noon and Caitlin Bernal a new hire she's doing another Dame design talk at 2:30 in this room and that's it if you want questions outside