Running the Halo Multiplayer Experience at 60fps: A Technical Art Perspective

[Music] hi it's great to be here good morning I appreciate everybody coming in here I know you have a lot of options here at GBC and we really appreciate you coming here for the boot camp so I'm Brian Moss I'm the lead technical artist at certain affinity today I'm gonna be talking about running the Halo multiplayer experience at 60 frames a second a technical art perspective I'm gonna give a brief overview of how we kind of run performance at our studio and a couple of the tips and tricks we use to hit 60 ok so who we are certain affinity was created in 2006 with the goal of creating innovative to have quality action games we were founded by max Holderman who's the lead multiplayer designer of halo 2 once he finished his work on halo 2 we decided in one's own studio so we traveled on down to Austin and started out with four guys now we're up to a hundred plus employees since

2006 we've actually had the opportunity to work on quite a few games everything for left4dead multiple versions of Call of Duty multiple versions of Halo we've done a few IPs a djibouti series crimson Alliance now we are primarily a focused multiplayer studio what that means is we actually primarily do the multiplayer we get contracted out by lots of studios to actually work on multiple multiplayer we're a co development studio and that means we're not a we're not an outsourcer so we don't get contracted to let's say create a thousand widgets then hand them off and somebody implements them we actually work inside our partners toolsets excuse me work inside our partners toolsets who actually work inside their servers if they're artists use Maya we smile if they are just use max we actually switch

all our artists back over to max it's a very fun experience the responsibilities of what we do depend on our engagement sometimes we'll actually have a project that's only a couple months and we'll work on some multi-player design all the way up to working with several years on an entire multiplayer engagement everything from level design technical art shaders tools you name it we'll do it all it depends on what our partners really need we like I said we embed ourselves into their tool chain and pipeline their process is really kind of a foundation of what we do sometimes we're actually able to add our own layer of tool sets on top of it sometimes we can sometimes we'll actually be able to write rewrite all the shaders in the multiplayer sometimes we can add new exporters for Maya sometimes we can't there are projects where we literally are stuck into exactly how they like to work and that's just kind of one of the strengths of certain affinity one of the things we do a little bit about our staffing we're

really arc focused we've had 56 artist on staff past that goes from design done for engineering we're mostly senior artists we have a lot of senior talent I think our average time in industry for our artists is a plus years some of what we consider our junior artists are actually like three to four years in industry so wow that's a lot of fun most of the times you don't actually have to teach artists like hey this is what a texture map is you know this is what how you normally map something they're very talented people and what they like to have their pipelines set up in a very specific way so supporting them actually takes a little bit more effort okay so take our responsibilities to ca we have five full-time technical artists we have our own specialties we have some guys that like to write shaders we have some guys that just love to write tools some guys of performance will do a lot of like general tool creation scripting depending on the project a lot of what we do is actually tip to the spear work so when we get a new project will

actually try to fly off the technical artist and embed with our partners anywhere from a day two days really until they actually take us out and make us leave because we're eating all their donuts and coffee all the time so we need to get as much information as we can as how they like to run their pipelines all the way to what is broke them in their pipelines will actually spend a lot of time on something just to realize oh wait a minute that doesn't work that you I didn't get updated that with that feature was deprecated three versions ago so talking with their artisan producers was actually helped us save say the time of finding those mistakes who else to do a lot of graphics word we do a lot of visual development for our art directors and our artists we do a lot of shader writing of course performance news and that's what we're here to talk about today okay hey look back into that 2013 we got the call and say hey we want you guys to do more halo work and you know we love halo down a certain infinity so this halo project was actually going to be pretty exciting because we knew this was going to be fed Xbox one after spending so many years on

the 360 and a ps3 I thought of moving the next-gen was was really exciting for artists and people really uh really jazzed to work on this project so we found out that we were going to be doing a sort of halo 2 anniversary for these levels we're gonna be doing six level remakes also for the first time we're actually going to be taking over a lot of sandbox work now sandbox that's what we do anything has to do with that the characters weapons vehicles modes we got to create a few new modes and really kind of sink our teeth into it for the first time we got to help getting dual wielding working for our part of the gate now as this project kind of snowballed into something bigger into the Master Chief collection lots of studios around the world actually ended up working on this so I'm really going to kind of contract arianna what certain affinity developed and put up for this game and of course it's the Xbox one and who doesn't want to work on that as well any project you actually

gonna have a few surprises in the beginning well the first one we were hit with was you know we were going to have to run at 60 frames a second that was non-negotiable and I being a performance guy I loved 60 frames a second I think it just feels better I mean as a shooter of course you run it at 60 it's just such a smooth experience and Hill has never run at 60 before there's traditionally a 30 frames a second game and I've always wanted to run it at 60 and I was really excited about it mostly artists were terrified we also found out that the next-gen engine and toolset was going to be developed off-site we were half excited and half kind of dismayed when we found this out some of us really kind of wanted to upgrade the engine and really be able to add our own spin to it other other people were kind of ready to kind of let it go we wanted to focus on the work that we had to do because we have a lot of work to do and we only had about seven to eight months to do the project the engine was actually developed by

ruffian games they were just amazing guys I just loved working with them there they were really really great so what do we do now knowing these two things so we started doing our initial planning for this part of the project so we knew okay we need to hit 60 frames a second it's got a hitch got a shift on the Xbox one and it's next-gen halo a I mean it's gotta look fantastic if it doesn't look fantastic and we don't want to let you guys down so shipping on xbox one so with the engine being developed off-site and we didn't have any Xbox ones yet like what do we do well we had to start working on the Xbox 360 we developed a plan of how and when we were going to get the transition according to our partners we were going to get the engines somewhere halfway through development so we were gonna have to start developing on the 360 transition the next box one shipping on the Xbox one and hopefully run at 60 at the end mm-hmm constant communication with engine team that was really pretty crucial we had weekly phone calls with

them we were in communication if I email anything that they needed or we needed we were right on the phone right on top of it it was it was really pretty great today helps herself knowing exactly what was working and when it was going to be working and also what was broken some things they didn't think they were gonna be able to fit in the time line that was really pretty crucial because if we were to depending on a certain kind of process to actually get these levels working and looking as good as they did and it just wasn't going to ship that would have been disastrous we would have been really struggling at the end he's look amazing so we didn't have a lot of control over base engine was developed off site well we did have control over we actually there was the shaders so we decided to go to a full PPR system and Louise cruel is actually speaking right after this and he actually rewrote all of these shaders to use the PBR system so please stick around check it out I didn't want to put any screenshots in here and then was stealing his thunder but it's it's he did some really pretty amazing things so check it out

okay so we need to hit 60 frames a second we immediately started developing on the xbox 360 the upside to that it was it was a known pipeline we've been done a lot of work on the Xbox our team was very familiar with the tools very familiar with processes we had the levels up and running within a week I think and just really started putting a lot of lot of the content the game immediately we started also planning to see what our 60 frames a second budgets would actually look like so I had a couple slides in here talking about in milliseconds and I even kind of ripped them all out basically a millisecond is a thousandth of a second and when you're talking about 60 frames a second you have sixteen point six milliseconds per frame to work with and that's it that is not a lot of time although when you start working with performance a couple of milliseconds will actually start seeming like an eternity and artists will check in an object and then wait that object takes three milliseconds to render were you crazy that's forever so and you have 60 frames 16.6 milliseconds

per frame it is important to know really that for the remainder of this talk maybe talking about work on the GPU so if I'm saying 16 milliseconds so this can be all actually on the GPU so how does tech card help with all this well we work with design the very beginning now these levels were all remake levels so if there was a rock with the canyon in front of it with the turret on the left that was what was going to be on these levels the fans were very very particular to make sure that we've made these levels exactly how they remember them but a lot of the times we'll work with design to actually make sure that they don't go over the budgets or hey if they want to have an underwater city you know like okay great you can have another water city but that's going to be 12 milliseconds that means you get one millisecond to put everything else in the game we work with art for look development and shaders so our director wants to have a certain type of lava or a certain type of ice technical artists will actually help develop that and actually optimize those shaders to make sure that they're going

to run a framerate daily communication with math leads for anything that they need when we're doing our multiplayer levels we had the concept of a mathlete for every map that we do there's environment artist that is the the key point person for that we call them a mathlete they are the guys that we go to with any kind of communication that we need for their team best practices it really helps us have a having a point person for every single every single map technarch managing the frame rate so we work with QA to find it for the problem areas our QA guys hit certain affinity are just fantastic I've never worked anywhere where Qi guys worked so hard they'll run through all the maps find all the areas that aren't running a frame rate record the positions and we have a tool that will actually be able to snap the role but camera positions for each level that aren't running properly then of course we communicate our best practice to map leads a team will troubleshoot fix problem areas a

lot of times we find framerate issues two things can happen either technical art can fix it ourselves we can get in there and we can actually fix the collision of the terrain or whatever's causing issue in there or we can go to the Napoli and say hey if you want to do this here's what you need to do sometimes our map leads are like oh tech art go fix it awesome I'm busy polishing this thing over here sometimes we have map leads that are like don't touch my stuff ever we don't want you to ever touch it okay whatever you need communication I really cannot emphasize enough how important communication is when it comes to performance especially when you're trying to run at 60 60 frames a second is a it's difficult it takes a dedication from the entire team the artists are going to have to give up a lot to run at 60 the programmers have to make sure that they have everything running like perfectly in order to hit 60 or you'll you'll you're just not gonna it's not gonna work

so certain affinity strives to always run the framerate every day I'm gonna show you guys a little bit of our data that we send up to the artist and now it's not always possible to run at framerate every day you know development happens sometimes you're gonna be running at 50 sometimes you're gonna be running at 70 but you it's important to always try to run a framerate it's really just a winning solution those of you that have been in game development before when you're at the end of a project trying to hit frame rate is the most painful thing ever a lot of times what ends up happening is you start stripping out all the artwork and the art director is very upset that his vision doesn't look as good anymore and the artists are upset because they spent six months putting that that that beautiful room together and now you have to have to strip all the objects out just so they can run at 30 frames a second or 60 frames a second but if you always try to run at frame rate everyday and you build your process around your frame rate then it's always going to win that's always going to run and your artist will spend more time polishing

art instead of ripping all out to try and hit fragrance so it's important to have a budget obviously I'm not going to break down our budgets completely because I would just take too long but our environments had a budget of 13 and 1/2 milliseconds out of there out of our 16.6 now let's say environments that includes all the effects and levels and includes the post effects and includes things like godrays high-quality shadows or whatever kind of features that the artist wanted to put in there if we weren't hitting framerate we had to know why and we had to watch the numbers every day so so you might be asking yourself how do you actually get these numbers you know how do you figure out if you're running at 14 milliseconds or 22 milliseconds well when you're working on the consoles each console has their own kind of performance capture software we use picks a lot especially on the Xbox one razor for the ps4 or G pad for the ps3 really what it does allows you

to capture a frame of your game loaded up in the tool and you can step through individual draw calls of your scene you can see what every post of that costs it will actually show you what is wrong with your scenes if you've ever walked into a room in a video game and all of a sudden the framerate dumps down and you don't know why loading in and depicts will actually show you this it'll show you what's behind that wall it'll show you what every single pixel are not on that wall cost to render it could even show you the pile of a hundred thousand polygons that have an invisible material on it the senator a room that's what's causing the framerate to dump out like oh wait a minute there's pile of invisible leaves in there that the artist forgot to remove and that's worth killing the framerate it's really the only way to truly understand what's going on okay so running a sixty we're splitting up her budget thirteen and a half our environments and with the character's weapon the vehicles in all our sandbox work we had about 2 and 1/2 milliseconds and that's about an average for the characters sometimes we had a player

maps and sometimes our big team battles were 16 players so we had to roughly be about two and a half if 16 players on screen with the vehicles and maybe some grenades going off and maybe a banshee flying by okay two and a half milliseconds is about at average now when you get into your multiplayer scenarios you're gonna have things like server lags if you don't have dedicated servers and things so you want to try to even run a little bit lighter than that that's why if you notice we're a little bit under sixteen point six milliseconds here so trying to account for any kind of server lag that we can you might your host might be running at you know 17 milliseconds so any kind of extra buffer you can give it it's just it's just gonna help you out okay so we start talking about our daily performance numbers here now it can be difficult and time-consuming to gather all the data I will admit on this project of what we ended up having to do

are qwq a department every morning would get the build and they would load up every level and they would go to every single camera and they would write down the numbers of where the frame rate was that we were being bad TAS at that point I'm happy to say now we actually have multiple tools that will automate this entire process for them so in the mornings they had open up the build they hit a button and it runs through every camera spits it all out to an excel file loads it up and we send it out to the team and it saved them hours of work but unfortunately from what we've had on on this last pillow project it was less than optin ideal but these guys did a great job now this data can be good confusing at first so this is an example what the team saw from the mornings it's not quite as confusing as it looks I'm gonna break down one of these cameras here in a second we work with our math leads on any problems the math leads will load up this and be able to look at it and anywhere they see green they know that their shots were passed anywhere they see red they know that they failed so they zero in on the red parts and we

the Technic part we're gonna see this first and we'll go directly over to them like hey yes shots three four four three through four failed and here's why we've already run it through pics to know exactly like what's going on so here's our camera shot so the picture on the top is a shot of the camera location this is the location when you're looking through this is where the framerate is bad and the column on the left the GPU work that that is a custom counter we had in pix actually showed us hey just accounting for the GPU this shot and the top number is twelve point six twelve point eight milliseconds and that's how what they would how much it would cost to render on the GPU now the numbers from the top to bottom the numbers on the top was B those are the numbers from the daily build today that would be today's build so today the camera is passing it's great now yesterday's build was running at fourteen point one that means it was failing so between yesterday and today we actually got on there and fixed a few things they checked it in they ran the numbers this

morning and all sudden hey the cameras green and all the artist care about is if the cameras red or green if everything's green it's all good the column on the ref left is the GPU usage that's was another custom counter we used it's basically a percentage of how busy the GPU was when it was produced in the frame but once the artist kind of break this down they would only really look at their individual levels it was a really quick process here's a quick screenshot of our multi player performance numbers I've never worked anywhere where we test multiplayer as much as we do with certain finiti being a co development house that focuses on multiplayer I should have expected it but we have three to four multiplayer today and each test sometimes it'll focus on design and sometimes it'll focus on modes sometimes it'll be an art review but behind the scenes we're recording these numbers for everything for every mode for all the amount of players get quite a lot of data this is where a lot of times we'll see a lot of CPU issues of running and running in

here and here's my handy-dandy attempt at explaining GPUs is stalling and being CPU bound so up here on the top if you look at the top graphic the on purple the GPU let's say is busy bit rendering the first frame and it's going along and it's rendering at 13 milliseconds now while that's happening the CPU is actually busy getting ready putting together the second frame and once it's done it releases off to the GPU GPU renders a second frame 13 milliseconds no problem everything's fine so now the bottom now here is where what happens what it looks like when your GPU actually stalls out so while the GPU is running the first frame 13 milliseconds that's fine the CPU is busy putting together second frame and then it runs into issues and there's too much data it's got too much whatever whether it's collision or there's audio problems and it's taking a long time for it to put together the second frame now what happens is it's causing the the GPU to actually stall out to render the second

frame it can't render the second frame until it gets all the data it needs then it ends up by the time it actually does get released to render the frame it's taken 24 milliseconds to render that frame that's because the CPU is actually hogging all the time to put it all together now what are some of the issues that would cause the CPU to kind of stall out well for halo we have collision issues we have lots of different collision types our artists don't like making custom collision even though we have great tools for it what happens is the artists will actually take the render model dupe it off and go that's collision and I'll go ahead and optimize that later well later comes around and end up with hundreds of thousands of polygons the collision in your scene and that ends up just really choking out the CPU audio issues can kind of would hit us we had draw calls poor occlusion calling we would have way too many draw calls and our scenes we ended up combining objects together we're with portals and make sure the objects are excluding properly and to a lesser extent maybe some animation or AI but

multiplayer we don't really have a lot of a I don't okay let's talk about our levels so we'd six level remakes I want to talk about bloodline bloodline was our hello to coagulation Matt it's really kind of a classic map everybody loves it's one of my favorite Maps it was a nightmare for performance it was our worst case scenario on on all accounts so this was our big team battle map there's 16 players so we had 16 players out there hunting killing driving around throwing grenades and causing a lot of havoc multiple vehicle types our designers put basically every vehicle in the game we had flying vehicles we had driving vehicles we have tanks we had both of our new vehicles in there we nee extremely long drawing distances and I'll show you right in here so here's the screenshot bloodline and this is a little over halfway through the map this is a sniper perch you can get up there and you see the entire map there's

there's nothing to hide from you get snipe from this position so I mean it has to render perfectly and if it doesn't people are gonna let you know you know when you were working with multiplayer audiences if your framerate isn't up there and it's not perfect they will get on those forums and it will let you know so here's another sniper perch this shows off a little bit of our grass system here one of our issues we had with this map is these mountains we'd actually combined a lot of them together because we ran into draw calls we were way over our trial called limit for this so we'd combine all the mountains together well then we got hit with a GPU we're the mountains weren't calling properly so we had to come up with the clever ways to cut all of these mountain ridges the mountains would come in and kind of go back around and we had to find clever ways to cut them up tour ok well this angle from this sniper perch and get the left hand side of the mountain out and that's a quarter of a millisecond we can get that okay you know we have a half a millisecond to go where else can we cut these mountains up to get some of that out our grass system was it was pretty robust and we'll talk

about that in a sec here's kind of another sniper perch looking down inside one of these bases these bases were pretty complex and there was entrances and holes all inside the bases so calling out the inside of the bases was quite a challenge we end up attaching a lot of objects together these bases work if - together in the beginning and there was lots of geometry in there so it took a lot of optimization passes for artists to get that right that little glowy ball on the top there that's one of our new dynamic dynamic events we add it into the level basically the player can go up there flip the switch in this giant EMP would kind of burst forth and fill the screen with this massive you know electrical storm that disabled all the vehicles and it was really a lot of fun it was very cool and it was awful for performance but we ended up making a lot of particles additive which took a lot of a lot of load off our GPU and it was only on screen for a short amount of time so it generally kind of worked out here's a

picture of the inside of the base beautiful I loved it it was it was really really pretty the art it's just an amazing job with this shows off our new PB our system really really well but you can see that there's a hole in the center of the base and behind that other column over there there's another hole that's a hole behind you so there was no good way to really cut out a lot of the environment we had to do again a lot of creative cuts when you inside the base to cut out some of the environment on the outside it was it was it was very difficult we also had a cave system to make things worse these caves when they first got in they were kit bashed together with with a when the arts created a rock he's like I'm gonna take this rock and he could bash the entire cave system together with these rocks and I forget exactly how many polygons it was but it was a lot as soon as that got checked in our frame rate went down about 15 frames a second in that area when we go over there take a look at it and the artists were like oh yeah yeah we know we know we got to optimize if we're gonna attach it all together like well okay well that's a lot of polygons

and it kind of looks something a half-joking he's like oh it's next-gen I thought polygons for free now so if you've been in the industry long enough you're gonna hear that you're gonna hear Oh polygons would be free of next-gen polygons will never be free never it's some kind of urban myth that's been created from 15 years ago and I heard that when we were switching to the ps2 Oh polygons are gonna be free on the ps2 I'm not even too sure exactly what that means you know but we ended up with attaching all these rocks together optimizing it out and actually it ended up running really great but I love that polygon his free thing okay so our grass system with a very very dense system of grass in here blood line is basically two giant open Canyon okay we couldn't have a lot of objects in there because we were very bound by a draw calls so we needed grass to really kind of break it off and our artists were very very passionate about this grass system to make it really kind of blend and really look really good and

we were like okay let's go for it at one point we had 70,000 clumps of grass in the system if you look into 20 to 50 verts in each and that's over a million vertices now the system actually handles grass very well it's it's it's pretty slick and how it does it the tool will actually go through and combine a bunch of these clumps together and and two big groups of grass and then we'll have these groups of glass which we can LOD them differently we can turn them off on the GPU a distance and it works really well well we've ended up with thousands of these clumps now this system actually hits the CPU and the GPU at the same time so GPU because it's got to render you know all of these little vertices and you'll have tons of overdraw at one point this grass was almost waist high and now you have all this overdraw going through so we had to make the grass a lot lower but it also hits a CFU because that's a track every single one of these groups and tells it okay well what LRD is it on is it on or off every single one of the group's hits it so we still

didn't render enough grass and at one point we were gonna have to take a lot of grass down to me the artist very sad and nobody likes sad artists they start crying and it's very depressing so so what we ended up doing one of our amazing technical artists actually came up with a few tools to get the grass out of our grass tools and brought him into Maya we actually converted a lot of this grass actually into regular geometry and we were able to call it you know on a normal basis just like anything else so wouldn't hit our CPU as much it turned out looking really great other things to help the GPU the lod system in halo is actually it's it's pretty neat it's got some cool things to it at a distance it will actually create a very low res version of the geometry it puts a cheap shader on it actually bakes some vertex colors on there well when it samples aligning it if lens pretty well flip for bloodline we had to bring that way way in because the shaders are very expensive now so you actually most people couldn't even see them so it did a pretty good job very few detail objects in the middle of

math when in the middle of map you're actually playing if they did a good job clearing everything out without it feeling too naked and open the grass doesn't help with that we had some strategically placed rocks but we couldn't have a lot of objects in the middle we just couldn't afford the renderer and the poly counts we had to keep as low as we possibly could this map had the lowest poly count of any of our Maps it was just so big we couldn't hide anything things that help the CPU the vehicle physics needed optimization again we had almost every vehicle this map and each vehicle has multiple damaged states and destruction stage and each one of those states had their own little bits of physics that kind of pop out so we had to get in there and optimize that to really help the CPU again with our draw calls we cut them down very dramatically and it via effects distances we cut those back as well we turned off our VFX really aggressively unfortunately there was just nothing we can do about that so after a certain amount of distance there just isn't any of the effects because we

just couldn't afford it so blood line was very CPU bound get rid of that slide wrapping it up all right so trying to hit 60 frames a second really we had to do a lot of careful planning with this basically you know having to start out in the 360 and then convert over the Xbox one we literally we've had plans of exactly where we needed to be on every week okay where we at this is that going to help us get to our plans in March when we get the engine constantly communication between us all the other studio is working on the project all of the different teams within certain affinity that working on this project it was really tremendous and you always have to keep an eye on your data if you're trying to run at 60 frames a second having a system where you have that ten of daily updates was is really really helpful there's no way we didn't been able to do that without it and this was definitely

one of the most challenging products as certain affinities ever done.we was the first time we ever had six maps running at the same time it was the first time we've actually switched consoles in the middle of development it took seven months to develop six levels full set of weapons and vehicles with the effects that went around with it on top of everything else we had a very pretty large team for us we had over 60 people working on this product thank you I want to say thank you to Thomas Maillard Louise cruel Matt ah slay Neil Davis the entire Hillman team it was an absolute pleasure to work with every single one of these people it was just really a lot of fun lastly you know certain affinity is hiring we're looking for you know talented developers we're looking for technical artists so if you're interested I'm talking about the words or go on our website check it out thank you if anybody has any questions microphones

yell scream em at me I just wondered because you you do daily taster yeah how many positions and the view angles do you taste each day so the question was how many positions and few angles do we have a few angles do we have daily as many as we can find our QA Department is is devilishly clever finding all of the angles that are actually within our play space that are actually don't run at frame rate when I sit down and we work with them and usually they would be between 8 and 12 depending on the map sometimes they'll run up to 14 when what ends up happening as each individual camera hits framerate if they're consistently at frame rate for 5 builds will actually drop that camera out and we no longer report that to the team okay QA team positions are

determined by the QA team yes okay very well do you think now to the tens of tens tens math do they test them enough to feel according to my experience with multiplayer levels it depends on the level it really does like coagulation had quite a few there's always levels areas that don't fall within those cameras but we'll catch a lot of those on our daily play tests as well we didn't want to overwhelm to our team so let's say if we had 50 or 60 cameras and you're sending those out to the our team every single day they're gonna open that up see an endless amount of data and they're gonna shut it down and they will never look at it ever so behind the scenes that technical artists will actually take a look at the entire level as well we ride around in the vehicles we attach to came a fly around and we'll find other little areas that will try to

fix those and sometimes we work with the math lead seems like hey we found a new area but we're not going to report it so let's let's work on this layer as well do you ever try automation automate all the process yes yes we have new tools now that actually automate a lot of the process we still place the cameras by hand I would love to have a system of heat maps and I prayed it even more thank you thank you I had a question with regard to your shader replacements in general were there objects that were more aggressively replaced and Howard what were they replaced with like did you have different levels of PBR did you fall back on a spec or a pure diffuse we ended up replacing most of the shaders Louise is going to talk a lot more about our PBR system and the neck talk so stick around and talking about that thank you hey I was very curious about your how you deploy your partners pipeline so that that's really

interesting i was at XDS last year and basically like working with external developers was a giant nightmare for almost everybody and one thing pretty much no one talked about except for me was the concept of deploying a pipeline externally so I'm curious how your partner's pipeline gets installed and configured differently every time most of times we have to be on a secured network so I'll actually have like two or three different boxes at my desk and every single one of those boxes on a different network or sometimes I'll have to have a box on a desk in a different room but they'll come out and they'll help us install it sometimes they'll deliver boxes with the the tools already installed on them sometimes they'll actually send guys out and install it for us you know but once we get that networks up and running generally we'll run a little slower than their guys you know because we have to send data all the way up to there let's say up to three for three and then back so it kind of it's different every time so it's a

it's largely a manual process it's done automated sometimes the candidates parts of it thank you I noticed that you were sending out the reports via Excel files have you ever have you guys thought about like maybe having a back-end for aggregating a lot of the data or was that like how to scope yes and no again sometimes we have to rewrite things for different projects sometimes our partners actually have tools for us that we actually have to end up using we're currently investigating a lot of different solutions to try to come up with like a certain affinity set of tools so we're really looking to everything right now thank you thank you the team as a whole embraced performance especially from the th perspective did you find yourself fighting with art over visual bar look development versus shader performance the whole team loves performance when I'm in the room you see the beginning of project when we got that Mandy to go 60

there were a lot of artists that were really bummed out because they wanted to have all of the milliseconds and hoard the milliseconds and we're gonna do all of these amazing things and we're gonna have all these vehicles and and do all everything we've always dreamed of but even they kind of came around once we started playing the play tested 60 frames a second they really came around and they had to give up a lot of things but really once our artists go through an entire cycle and they get to an end of a cycle and they don't have to take anything out of the games we actually have one of the gate processes that it's a tech gate to where they actually have to add in any of the expense of graphics processes they want for this level so if they want godrays plus the expensive fog and a higher resolution shadow map we have an early gate they have to actually put that in turn it on and actually account for that frame rate early on in the process let's say all three of those things took two-and-a-half milliseconds well now but I actually build a level around those processes and so for the

entire time they have that on it may not be tuned and looked super pretty but it's there so that way at the end instead of trying to turn those uh processes on at the end and now I'm not hitting a frame rate they're always rotating frame rate so once they go through that process they're actually pretty cool with it they really like a lot so they can polish their stuff at the end awesome and where those features would have already developed you said that there was already some existing pipelines I'm sometimes yeah sometimes we'll get features halfway through development and we'll have to do a lot of testing on them so we have to be really flexible we try our best thank you I am really curious uh curiosity about the parking accounts so basically the the halo2 exactly the Romaic so when you import it original asset so the polycon account already fixed so when you add more polygons so where where where do you added those polygons to very facts already make the original

SS okay so the question was was how did we get our polygon budgets from the old assets and the new new assets actually we used a lot of the old assets just as visual reference we we rebuilt at least in the environments almost everything we were built all that from scratch now when coming to the characters like the Spartans and and the weapons and and the vehicles we really just kind of like polished those up there wasn't a lot of time to rebuild a warthog from scratch and there really was no need to but we did redo all the shaders and the materials on things what's what when we're coming to the polygon budgets we stayed pretty close to what we what our probably our budgets were for the 360 just because we weren't getting the next-gen engine for four or five months and we had to be very very conservative with our numbers I think we only went up by maybe 20% because if we completely

blew our polygon budget got to the Xbox one and the next-gen engine didn't handle polygons very well we would spend a long time reducing our poly by the poly counts so we were very very conservative with our numbers okay thank you I how you mentioned being CPU bound and in the past I've always had a lot of trouble figuring out that we were CPU bound I always it was kind of like prove that it's not the GPU yeah so I was wondering if you had any more details about what okay so have you ever used picks no okay so picks is a profiling software for the Xbox it's really really amazing you can kind of let picks run in the background while you're running around in the game and you can have these counters that actually live live update and show you you know what's going on and there's a counter in there that'll show you what the frame rate is right now so if you're let's say or you're walking around your game and that counter will tell you okay you're at

let's say 18 milliseconds 19 milliseconds you're like oh I'm trying to run 60 so it's 16.6 that's what I got a hit what you can do is that you can write that number down and then you can do a full graphics capture of that exact same frame and when you're doing a full graphics capture and your kind of loads it up and you can see all your draw calls and all your textures you can actually calculate and see if the GPU wasn't busy that is the that this is the number what is that it would take the rent on the GPU and now that number could be 13 milliseconds so you can actually get the Delta between those two numbers and okay well if the GPU says it only takes 13 milliseconds another counter says it was 18 you know okay we've got five milliseconds that's unaccounted for and that's straight up because your CPU is way too busy I you mentioned that you had the engine developed off side right yes sir so do you have two questions actually do you have programmers on side like certain if it is yes develop the engine as well or

we had programmers on site that would help us with things like we implemented dual wielding on our end we had programmers that actually would merge the builds from from our partners to us and do all the testing and all the helping the technical artists with the testing sometimes our programmers would help with features if ruffian couldn't get to a feature our programmers be like no no we really need that so they would dive in there and fix that feature as well our programmers depending on our engagement do all kinds of work so whenever you've run into like problems that you couldn't solve with the the standard means like reducing draw calls merging meshes whatever do you also have like programmers that take care of some feature works that are performance yeah yes yeah we have great programmers that help us either if they'll do some heavy lifting on the graphics engine if we can because a lot of times we can't actually change the graphics engine if we're working on a call to duty the you know when Call of Duty's not gonna let us change their graphics engine to fix a

feature and a couple multiplayers levels but we do have programmers that'll actually dive into the do a deep dive into the code and tell us exactly how its rendering that feature and what we can do to get the most amount of traffic now some projects will actually our programmers will go in there and actually change the engine if it's a little bit more open there are partners really trust us a little bit more thank you thank you hi all right so I was just kind of curious how you got around the problem with split screen play because I know I love this game and I'll be playing with friends and whatnot and you know single-player is great and then you're like yeah come on let's go jump in a big team battle and split screen and then you're losing frames oh you're one of the guys it's your fault [Laughter] okay so for those who don't know halo you can do two player three player four player split-screen it's always been a

popular feature we got really lucky with this product and we were told that it was okay for split screen to run a 30 now we actually do track our split screen numbers as well if you go back to the multiplayer slide where with our numbers out there you actually see on the right-hand side that says two player four players split so we play test those numbers as well so when we're actually in a play test lab we'll have a bunch of guys actually on regular computers I'm sorry regular monitors you know playing on a single player and then we'll have four guys on a couch squeezed in playing on a quad screen and that's where you go if you didn't you know those were the four guys I got their last because you got squeezing on the cows and you're playing quad screen you know you never win on quad screen but yeah we tested that every day we had special throttles that we could actually go through I can turn things on and off when it goes to like to screen three screen and four screens so on quad screen and I would do things like turn off the grass I would turn off the fogging or I would turn off certain features and then I can actually profile that individually

No thank you very much everybody I appreciate it [Applause] you