I Shot You First: Networking the Gameplay of Halo: Reach

[Music] good afternoon everyone I could have your attention really quick before that I took it started with a talk proper I want to share two quick notes with you first please silence cell phones laptops anything else that might make a disruptive noise second this is mentioned in a few places in the program one second all right sure so it is mentioned in a couple spots in the program and at the door but there are also too few spots where it's not mentioned this is a two-hour talk we are gonna run until five including QA obviously there's a whole bunch of sessions at 4:30 some of you may be very attached to those sessions I have good news for you this slide should run about an hour and 20 and then we'll do Q&A so if you're willing to skip QA catch on GDC vault you should be able to catch a 4:30 talk thirdly and lastly I've been

asked that if possible if you could all scoot towards the center of your sections as late comers come in we want to allow them to enter the audience as with a minimal disruption with that welcome to I shot you first I am both honored and incredibly excited to be here today to talk to you about the gameplay networking of Halo Reach when I started working on reach three long years ago I knew very little of what I'm going to describe to you in this talk and I would have killed to get my hands on it I hope that you will find its lessons useful in your own games the very first thing that you're probably wondering is who the heck am i to be talking to you about this I'm David Aldridge I'm the lead networking engineer at Bungie I spent approximately

three years working on Halo Reach networking I've been in the games industry for quite a long time before that starting with this gem over ten years ago the second thing you're probably all wondering is what is Halo Reach fortunately I have prepared a small video here to show you what you're missing cool sound please pardon the technical difficulties [Music] [Music]

[Music] [Music] to me [Music] triple kill so that's Halo Reach we're pretty proud of it so there's three things that I want you to walk out of this talk with three things that if I do my job you will leave here with first we're gonna

talk about a proven architecture for scalable gameplay networking proven across three Halo titles this word scalable is very very key here and we're gonna return to it again and again there are many ways to do networking that are not scalable we have a way that we have been we have had very good luck with scaling across games with very very large numbers of objects and very diverse gameplay mechanics second we're gonna talk about how to design solid networking for your game mechanics which includes things like ensuring that your game mechanic design and your network design work hand-in-hand to reach the highest possible quality third we're gonna talk about how to measure and optimize your networking systems something that is incredibly important there are a few things that I want to get out of the way upfront that this talk is not about since there's the word networking in the title there are

certain expectations first we're not talking about halos campaign or firefight networking we're specifically talking about the competitive multiplayer experience 16 players pvp campaign and fire fight use a different networking model that will touch on very very briefly but we won't go into any detail on second we're not talking about sockets we're not talking about traversing a gnat and punching a hole in a firewall and connecting to a remote machine everyone has to solve this problem once but it's not really that interesting in the long run third we're not going to talk about what we call high level networking at Bungie things like matchmaking getting players together into games creating your online ecosystem that further ado let's go into the game play networking architecture all right so what is game play networking what is this problem we're trying to solve here i sat down I tried to write a definition this is what I came up with it's a little wordy it's very explicit I don't

actually like this definition very much at all communicating sufficient information perceptually shared reality this sounds almost like you could you could like test and algorithm against it that's that's crazy game plan already does not really work this way in practice this is what gameplay networking is in my opinion technology to help multiple players sustain the belief that they're playing a fun game together what underlies that does not matter all right how can we solve this problem how can we create some technology so the players believe they're playing a game together there are some simple approaches simpler approaches nothing's really are simple but simplifying approaches that will make your life easier first you can use locks tab also called deterministic or input passing methods this is what halos campaign and firefight networking use in this model you rely on the complete determinism of your game loop you pass the non-deterministic player controller inputs over the wire and voila everyone's running in the same game

simulation the downsides to this are that your input latency from the controller to the game includes the network round-trip time to the post of the game this means that for first-person shooters you end up with significant input latency when you try to shoot your gun or jump or operate on the world in any way unacceptable in a multiplayer context where players are fighting each other the second way you can make your life easier is to use reliable transport protocols you whether TCP or a homegrown reliable UDP to try to reduce the latency of it this can make your life much much easier because you don't have to worry about things being reordered or lost in the wild wooly internet this works very well as long as you have relatively simple Network State or you're willing to use a very large amount of bandwidth and restrict the number of machines and Internet connections that can play your game third you can always send all of the network state about your game

specifically you can always send it atomic what this means is you will not have problems where the network state of one object is out of sync with the network state of another you can guarantee that remote clients will receive the entire network state of the world in a consistent way this fixes an enormous class of bugs that quake three model is famous for this this works very very well as long as you can express the total network state of your game and no more than a few hundred bytes let's say does not work with very large numbers of objects unfortunately for us and the reason I'm here to talk to you today halo has to solve the hard problem we have hundreds of networked objects every one of which can stop a bullet every one of which can affect the gameplay in a critical way we have a highly competitive skill based action game that players devote thousands of hours to mastering we do not use dedicated servers to minimize latency between peers players expect the game to always

work for many of you from the PC world players on PCs can tolerate a certain amount of cutscene with their connection you can show them some interesting debug information like lag indicators things like that we don't the console audience doesn't really tolerate much of this at all they pretty much just has to work and lastly there's sort of the death knell for any of these methods for Halo multiplayer is this N squared problem that comes up in so many contexts across engineering for n players we have to send the network game state two and minus one remote machines and the amount of things occurring in the game world is proportional to the number of players because they're all shooting they're all creating havoc this leads us to a classic almost N squared graph something like this where the very lower left there is the amount of bandwidth that you would need to transmit everything that was happening in the world if you were in a two-player game and the far right hundred times is the amount you would need for a 16 player game

this is totally infeasible for Halo if we've graphed this out and it comes out to something ludicrous like 20 megabits a second of data fortunately we are not the first game to have faced this problem even more fortunately one of the games that did wrote a beautiful paper over ten years ago and presented at GDC the tribes engine networking model by mark Frohnmayer and Tim gift they outlined a host client model resilient to cheating at most PvP games the key attribute of their model that made it really interesting for us and made it the only model or the top model when we were considering model for halo is that their protocols for how they how they communicate about the game over the network are all about semi reliable data delivery they're all about providing network state that is not fully reliable so you don't you're not forced to resend

specifically they support this idea of persistent state and transient events we'll go into detail on what those are and what they're used for and the result of all this the result of being of having these semi reliable guarantees is that the whole thing is highly scalable to match available bandwidth we never have to build up a huge latency debt because we flooded the connection with too much data alright before I can tell you about how our system works I have to define three key terms gameplay networking terms unfortunately or not that standardized so let's get our terms straight right up front first replication replication is a term that we use at Bungie I haven't heard it used elsewhere but it might it might very well be it specifically is the idea of taking some state that exists on one box and then transmitting sufficient information over the network so that it can be replicated on the other box the analogy to think of is like a Star Trek replicator you take a description of a cup of hot Earl Grey tea and you give it

to the included in the replicator and the replicator can create a copy of it that appears to match it's basically the what we mean when we use the word replication we were going to give you a description of the object or what's happening with the object and then it's going to appear and look to you to be real second term Authority this is fairly intuitive this is the idea of who's the host who has the authority over objects for almost all objects for almost all things because we're a host client game because we're very sensitive to cheating and to attack surfaces for cheating the host has almost all the authority in our game yes it's the throat ativ over object health damage events projectile positions things like that clients can exert some limited authority over their own bipeds and we'll talk about that as well thirdly prediction prediction is what you do in between receiving updates about an object in this little animation I've got here if you assume that this

moving circle is an object and this is a simulation occurring on a client machine and the green stars are updates coming in from the host about where this object really is and what direction it's really moving all the movement of the object in between the green stars and afterwards and predictably bouncing off walls and things like that that's all prediction that's all us applying rules and guesses about what we think is going to happen with the object I'm gonna throw this word around a little bit okay that was pretty painless now we're that much closer to looking at how we network beautiful things like this networking stack there's any old-school Network programmers in the room you're probably expecting this to come up a little sooner but it doesn't look much like the OSI networking stack let's run through it at the very highest level we have the game code everything everyone else works on who's not on the networking team it simply runs the game

the next level below the game we called a game interface the game interface is responsible for scraping the game for extracting all the data from the game that we might potentially want to replicate the prioritization layer then takes all that data that game interface has gathered and decides what stuff is most important to send once the prioritization layer has selected what to send we hand it off to the replication layer which implements essentially the tribe's protocols again we're gonna look at all this in more detail below that we're gonna we have things we're not gonna look at in detail the channel manager which has flow in congestion control to keep us from flooding flooding players connections and the transport layer which looks like everybody else's transport layers just across platform socket layer okay in this talk we're gonna cover these three layers specifically in pretty good depth we're not gonna talk about anything below or above okay some meet some

actual real algorithms and instead of just overviews state data this is the first tribes protocol and the one that accounts for the vast majority of our bandwidth state data provides this very specific guarantee eventually the most current state will arrive sounds simple it's incredibly powerful what does this mean let's say I was the host and my position on this stage was being replicated out to all of you as clients say further this position is being replicated as state data if I start in this position and you all know that I'm here we start from time 0 everything's in sync and then I walk across the stage then I stop state data guarantees that if I stand here for long enough and the connection is not dropped you will all eventually see me appear at this location it does not guarantee any of the intermediate locations that will be

sent all the intermediate information can be dropped based on available bandwidth so this is used for all the vast majority of properties in our game objects physicians Health's timers tons and tons of properties the second replication protocol is called events events are all about justifying state data transitions with richer data events have much much better screenshots than state data as you can see because events are all about telling you the thing about why something happened why someone's health died or dropped why someone died why a warthog has just lost a wheel all these words of things the event reliability guarantee is non-existent unlike state data which guarantees the eventual delivery of the most up-to-date state events have no guarantees whatsoever every event could potentially be dropped this is important because

events are describing transitions if you are not in a location where you can currently see the transition occurring like if you are for example outside this room you don't care about transitions that are occurring within this room you only care about the final state so if you were to enter the room you would see the updated state last replication protocol control data control data is all about improving the accuracy of our prediction systems for control data we take the sub sample of the players controller inputs and a few other useful things and we transmit that sort of as frequently as we can from the client to the host and reflect it back out to all clients what this lets us do is it lets us in Ferrer's in the game or predict changes in the game far in advance of when they would actually change the game for example if I'm running forward and then I push the left stick to start strafing left you won't actually see a

change in my object for several frames plus acceleration time plus however long until it actually moves more than one pixel you know whatever significant amount of time control data allows us to transmit the fact that I'm pushing my stick left so that everyone else can start to predict that I'm going to begin to strafe to improve the accuracy of their prediction this is a very very tiny amount of data control data for one player is something like 20 bits very very carefully hand packed data because it's sent so often okay let's look at the big picture putting together all these protocols in one we have a host box in the client box as all good networking diagrams should firstly the client sends control data to the host this includes both his stick inputs and the current information about where he thinks he is in the world like Maya biped is currently in a certain location this is how the client exerts most of his agency in the world second clients will transmit events to the host these are transient things the client can do that he needs the hosts permission to do

or that he needs the host to to execute he needs the host to actually change the world in response to these things he's trying to do from host to client the picture is a little bit bigger we have control data which is stripped down to just the minimum at this point just the controller or inputs that are useful for prediction we have state data this is the vast majority of all our bandwidth again this is transmitting all of the final states all the current states of everything that is that is important over time and we have events which are just like with what the client sends to the host they are justifications for the state data transitions that occurred so take an example right here we have in state data the Warthog now has a broken windshield that's a piece of state data that's transmitted windshield state is now broken if that was the only thing we transmitted and we didn't transmit this Warthog just took damage at this point you would just see the windshield permute maybe there'd be some fake effects to cover it up but we wouldn't really know why the windshield got broken if you have the event you see

beautiful sparks you see all the appropriate transits transitions you have special sound effects you know all the beautiful flavor material that describes why the state data transition took place ok again I beat this drum several times now you're gonna hear me beat it several more times replication is never fully reliable unreliability enables aggressive prioritization which lets us deal with the fact that we don't really have an object cap which lets us deal with the fact that we have 200 synchronized objects or more in a 16 player game I mean it Peaks well over 2,000 so what can we do with prioritization we can give the power to our flow control layer to the guy who is looking at the current connection status and figuring out whether it's safe to currently send data without causing packet loss or a latency spike we can

give this flow control layer that full authority he decides when to send a packet he decides when it's safe and how big it should be when he decides to send a packet he talked to the replication layer and says hey we're sending a packet now what would you like to sequester right replication fills up the packet how are big it is obviously we don't have room to fit all our data we never do we don't really have 20 megabit connections and even if we did we wouldn't have the perf to actually encode it all into a packet so we write high-priority data first of course right pray or write data in priority order all right we've sold our souls in terms of complexity to get this wonderful prioritization system so let's take a little quick look at what its capabilities are it's sort of what you would expect it's it's a similar to the problem of culling of like figuring out what stuff is important to render it's a little bit of a software problem but basically we're looking at hey what stuff is close to this client what stuff is he looking at what stuff could hurt him what stuff has he recently damaged because that's a pretty strong indicator

of interest there's some really unintuitive stuff you get in here which we have evolved over time things like Oh grenades as they fly through the air our prioritized based on how likely they are to damage a particular player that seemed reasonable to us it's not going to damage you why do you care but we discovered significant lag artifacts on the players own grenades because players watch their own grenades like hawks even though they're completely irrelevant to them after they've thrown them they can't affect them in any way but in general they can't affect them in any way you can shoot grenades out of the air crazy sandbox designers there's tons and tons of these special cases that we have so sort of built up over the years in response to player feedback to give the the best behavior that we can let's look at a quick example of this this is a screenshot just taken from a Halo Reach game this player here and the

lower-left has just killed the player who is now ragdoll and falling to the ground below him let's take a look at the network prioritization of data about these objects - the guy who's still alive so everything we're going to see is about sending data to the guy who's still alive firstly we have a prioritization overlay which I'm sure most of you cannot read don't worry this is our end game debugger that sort of shows all of our prioritization decisions as they occur some examples the dead-body so first off the legend there's three numbers here for each object first is the final priority this is the final like literally if this number is higher you get to go in the packet first second relevance relevance is sort of a pure measure of how noticeable something is without regard to its importance to the game third update priority this is in milliseconds and this tells us hey how often should we try to send updates about this object so zero means we're sending updates as fast as we can about this dead body every single packet gets an opportunity to do so to send

something the 1.0 relevance is basically maxed out and that's because we know that players watch ragdolls fall for a few seconds after they kill people they're proud point five is a fairly high total priority there's all kinds there's tons of tons of range that has to be encapsulated into the priority range do things like stacked bonuses so 0.5 is actually a fairly high priority in this stage in this scenario conversely we can look at this gun here this strange pink thing is a covenant needle rifle the update period for this is 127 milliseconds so we're trying to send them updates about eight times a second about this gun its relevance is almost as high point nine seven because it's quite obvious it's in it's in the players space it's very very close to his screen it's on screen but the priority is much much lower than it was for the biped 0.2 - that's because priority mixes in all the information that we have gleaned about what is actually important to the player over and above what is noticeable guns when

they're falling generally not that important to players meant they might go pick them up but in general they're watching the body they're watching protocols in flight they're not watching the gun even lower priority example this is a grenade same shot same exact moment in time dead body in the background there this is a grenade the inactive grenade that was dropped by the guy when he died sitting in in the water behind the shooter this grenade is off-screen we're trying to send updates about this grenade about three times a second here is 339 about 3 X 3 times a second relevance 0.73 still seems fairly high from a 0 to 1 range 0.73 is not actually that high because the relevance range has to encapsulate everything from objects that are near to you to unthreatening non-critical on your team objects that are across an enormous map so that's why you see like a point seven three here instead of a zero point zero just cuz it's an off-screen grenade and the end

result priority is quite low it's point one nine here so this grenade is gonna get updated maybe three times a second if we have bandwidth certainly not more than that all right that's the core of our architecture this idea of using unreliable protocols to enable prioritization to give us unlimited scalability is the core of bungees approach to Halo unfortunately our job is not over we have to actually design the individual pieces of gameplay networking and this is actually quite difficult let's look at some examples firstly throwing a grenade [Music] [Applause] not that exciting pretty similar to a lot of other our first-person shooters what is it what happens when we throw a grenade well this is a sequence diagram

which is a tool we use quite a bit when we're analyzing how we're going to network a gameplay mechanic let's take a look at it in terms of a single box if we were throwing a grenade what does it look like from the controller the player presses the left trigger very simple that goes gets consumed by the simulation loop at some point fairly quickly and then a grenade throw animation begins there is some lead-up Telegraph animation while the player winds up and throws and then at a key release frame somewhere around here with his hand the grenade is detached aimed at a target using the players current facing directions and other code and then launched exactly how you would think a grenade thrower would work it's beautiful this is how it's worked since halo 1 how can we network this grenade throw there's actually a surprisingly wide variety of options first let's try something simple now we know we've talked to design we've played halo so we know we played the prototype at that

time for Halo so we know grenades are very important to the game we know we want them to be host adjudicated we want them to be authoritative we don't want to allow players to cheat with grenades so let's try this this sort of permission slip approach we send a grenade throw request to the host once he confirms that we are allowed to throw a grenade we do our whole animation and throw a grenade will this be a good networking approach and it won't but how do we know that for sure one of the most useful analysis tools is to use these sequence diagrams but with two boxes instead of looking at the controller controllers a single machine picture okay first thing that happens the client pushes a button and sends a message to the host I would like to throw a grenade now very first thing you'll notice this arrow is tilted unlike all sequence diagram standards why on earth is it tilted it's because we are dealing with a distributed state machine this period is the one-way network latency from the

client to the host these these tilts of these arrows are the key that makes this sequence diagram tool so useful to us because you can see all the potential overlaps of all the pieces of your mechanic all the different ways it could be network all the vulnerabilities okay so what happens pretty simple the host gets the request he says ok you could throw a grenade that sounds fine starts to throw animation when he reaches your release frame he throws a grenade so that you're happy he sends back a yeah that's cool you're starting your grenade throw go ahead start your throw animation you start your throw animation you play your delay for a certain period of time you reach your release frame about the same time you reach your release frame you'll get a message from the host that hey this grenade totally exists everything's cool we have a problem here we are waiting our round trip time before we're giving the client any feedback on his button press players hate this players hit it with a passion

this is totally and completely unacceptable so we have to find a better way ok let's try something new something a little more direct a little bit less asking for permission we'll just throw a grenade locally we'll just let the player like he hits the button he runs his animation he throws a grenade and we'll ask the host to also throw a grenade so that everybody else in the game can tell that we have a grenade in flight what if this look like sequence diagram button press we start our automation immediately it's nice and responsive everybody's happy and playtests huh so good tell the host I've begun a grenade throw host does the same thing as he did before triggered his throw animation on the release frame he throws a grenade where's the lag where are we hiding the fact that the host is adjudicating the grenade throw or is it judicata the fact that in the end the grenades damage will be owned by the

host the answer is there isn't any lag in this scenario we have not hidden the lag anywhere in this diagram we have never asked the host for permission which means does not mean that we've succeeded you do not want this you very much do not want this this means you have incurred latency debt you have created an artifact a dissimilarity and a synchronicity in your simulation which you will have to pay for in some way either you will have unhappy players due to inconsistent results in the game or you will have some sort of warp later on or you'll have additional code complexity to try to reconcile things in flight you have to pay for this sort of thing somehow ok so let's look at the actual way we do grenade throws just like in the second option we predict the throw animation so as soon as you push the trigger button on a client machine you immediately begin your throw

there's no controller input latency but we do not predict the grenade release we wait for the hosts this means that all grenades in flight will be real the host is authoritative we haven't incurred any lag debt but where is the lag in this view somewhere we're paying for a round-trip time let's take a look similar to before client pushes a button tells the host host does it's so far so good oh but at the moment when the client reaches his release frame he does not throw the grenade as we said the host must be authoritative over all grenades in flight he deletes the grenade so he gets to here and the grenade vanishes from his hand and he continues his animation at the same time as he reached that release frame and deleted the grenade he sends off a message to the host please create a grenade with this aiming information round-trip time later the host comes back and says yep this grenade object got created here his position and velocity and the grenade appears where we put the leg fairly

straightforward right here you could tell you see this pattern over and over it's a call in response to say please do this thing okay here's the feedback from it clearly round-trip time how is this better than the other way well that's where networking becomes a bit more of an art than a science this is much much better than the original solution where we asked for permission before we triggered the animation two reasons one players are extremely sensitive to input lag they're extremely sensitive to anything any button they can push on the controller that doesn't cause some reaction immediately in the game so having that arm start moving immediately is infinitely preferable second during the time of this lag during the time between the normal release frame and the frame when the grenade actually appears round-trip time later there's a giant arm on the screen it's covering like a third of the screen

it's moving really fast got really high res textures on it it's got all my custom crap on it that I've bought with all my credits it's really distracting not only that I'm probably not looking at what's in my hand I'm probably still looking down my crosshair it is to make sure that I'm aiming my grenade correctly players don't notice they literally don't see this this can get up to a hundred even 150 milliseconds casual players and getting to 200 before they notice anything this is completely shippable this is how we've shipped grenade throws of earth for three games now fortunately having such a beautiful grenade throw networking allows us to get results like this [Music]

okay that may have been a little the grenade example may have been a little artificial we never actually shipped any of those earlier sort of our made-up methods of networking grenades the suboptimal methods let's look at some trickier game play examples let's look at some features that actually were in reach that had multiple iterations of networking some of which we shipped in our public beta that were deeply flawed okay first up our merlok let's take a look at what our milan armor lock is an invincibility shield player pushes a button there's an intro animation during which he drops into this locked pose at the end of that intro animation which is three frames long incidentally very important three frames he gets

invulnerability and also infinite mass he has that for some period of time and then he'll release the button or run out of energy and his shield will go away seems like a fun mechanic how should we network this first version we did all the animations the entry into a into an armor lock pose and the appearance of the blue shield and the infinite mass on in the physics simulation all were predicted by clients so you locally when you decided to use armor lock would activate all the attributes of armor lock in your simulation this felt great this played great in our system length play tests in our land play tests at in the office we had this in for a few months and then we started doing take homes where we load the game up put two boxes and take it home and and play together and we discovered where the lag was let's look at armor lock v1 the client pushes a button he begins his animation immediately everything's responsive there's a three frame delay

here tells the host hey I've activated my armor lock host runs the intro when the intro animation completes on the client he gets invulnerability as depicted by this green box when it completes on the host he also gets invulnerability on system lengths this worked great what's the vulnerability where is the lag that's right here grenade explodes on the host it's very very special time what does this do well host evaluates it this is okay grant exploded you're not invulnerable yet you're still playing your intro animation so I'm going to damage you hey by the way this grenade blew up you just took damage the client says oh sure that's what makes sense I took damage no he does not he says what the hell I was armor locked I did the right

thing I saw the grenade coming I pushed the button I said I fly through the air push the button I was armored a lot I saw the beautiful blue shield this is blue [ __ ] so this was no good we had very furious players I just explained the lag so version two let's try it again that version one never saw the light of day no no that's too terrible we've never shipped that version two let's have the animation be controlled by the client sort of inspired by the grenade or the grenade approach well let the client immediately start his intro animation but once he reaches the frame where he would normally enable the shield he does not enable the shield he stays unshielded there's no beautiful blue shield there's no infinite mass he waits for the host to tell him that he's actually invincible this is the method we shipped in our public beta where did we move the lag to here now let's take a look at the sequence diagram client begins his

animation on button press yadda yadda tell the host host runs the intro client finishes his intro animation does not turn on a shield as we said host completes turns on the real shield and then he tells the client hey you're vulnerable now you can turn on your blue shield and your infinite mass everything's great there's a really ominous hole there in that diagram and indeed same exact vulnerability just a little less obvious grenade explodes here hey by the way you got damaged again just like last time and client did not have a blue shield so he doesn't have any real strong reason to believe that he's invulnerable so in some sense this is consistent networking is this correct networking players unfortunately are not very understanding a bit of this and they say why does my armor lock not work properly like this mechanic is broken this shift in our public beta

million people played this and they told us loud and clear this was not acceptable well actually they well they told us loud and clear and also we day to mind their use of the equipment and discovered that our lock was not as popular as the others so we knew we had something we had a problem we needed to fix this lag shows up there it's just wonderful period of sort of broken mechanics from the player's perspective so we implemented a version 3 for ship with one last tweak one very evil tweak let's take a look same thing as before client begins his animation he does not turn on his shield when he finishes it tells the host standard stuff oh this is different why did we just do we just said instead of waiting for the three frame delay to trigger invulnerability on the host we shrink that three frame delay by the measured round-trip time

between the host and the using client what is this pain well means this we activate the shield on the host early before the intro animation actually ends so somewhere up here the shield will activate instead of down here where it should and then we tell the client and lo and behold it lines up with where he expects to see his armor lock activate it's almost like we planned that and we did but it's obvious right by shrinking the frame delay by round-trip time we have exactly brought that we've paid for the lag we have brought the host time stream up forward in time so the client can see a consistent view of the world and we haven't incurred latency debt the world's consistent there's not something out of sync that's waiting to bite us later so what did we just do like grenades it can explode everything's fine everyone's always in sync there's everything works great players are happy what do we do this

feels too good to be true what we did was we broke the game mechanic in a very targeted way we went in there and we grabbed the frame delay number that the designers had so carefully tweaked with so much love and we slammed over it for networking we actually changed the way the game played we changed the real simulation this is the pretty much the only other way that we have yet found that you can hide latency other than paying for a round-trip time or reconciling afterwards you can change game mechanics fortunately we got away with this because players it was discovered once again somewhat by luck players do not care that someone else's armor lock activates a couple frames early sometimes they care but rarely they care tremendously that their armor lock activates on exactly the correct frame because they are very attuned to that timing so we got away with it we

shifted the lag by changing the game mechanic we shifted the inconsistency from the player who cares a lot who's using armor lock to the host and all the other players in the game who don't care as much just let us ship another wonderful feature tie the leader double kill I'm really infinite mass I mentioned that's why they wanted that turns out it when a rigidbody hit something with infinite mass you know the mass doesn't give the rich body has to give entirely so we deals with more or less infinite damage all right third example another

reach mechanic that shipped with somewhat broken networking in the beta assassinations let's look at what these are first strike the lead these were a signature multiplayer feature for us in reach we wanted to provide an exciting way to sort of take a take a small risk and humiliate another player so what does this mean in terms of what are the high level of problem we're trying to solve you have to buy pads that are happily running along in their own simulations they're both predicting their current positions as I mean they're both clients they are somewhat ahead of what the host believes their head by one way one way network time to the host of the host current simulation and then suddenly out of the blue we need to force these two bipeds to perform a joint perfectly synchronized

animation with very very tiny error margins the animators have carefully scripted this so that the knife slides exactly between the 13th and 14th rib of the elite whatever their Anatomy is they do not we can't fudge this this has to be precisely accurate this turned out to be something we had never really done before in multiplayer so we implemented a system which we shipped in the public beta local prediction of participant positions and orientations what do I mean by this what this means is when the assassination order comes down from the host which says these two players have just begun assassinating each other each individual box is permitted to make that assassination occur in whatever location is most convenient most pretty for them based on their current beliefs about the positions of those two objects so everyone runs the assassin in a slightly different spot potentially with a slightly different animation so that it can fit in that spot and we

thought this would be great because it would minimize lag on the way into the animation it would solve this problem we were very afraid of which was all these bipeds are in their own little predictive futures and we need to bring them back into the into the hosts present that seems like it's gonna cause artifacts so we did this and it did work that way like it definitely prevented lag on the way into assassinations and it worked really well in our play tests and take-homes unfortunately it totally failed in the public beta let's take a look at some of the bug videos from our wonderful bug database they came back from the beta

this one's my favorite this black screen is what you see in a film when there was a host migration it will surprise very few of the programmers in the audience to learn that that location that we got warped you after the host migration was zero zero zero all right so this wasn't very

good there were clearly a lot of problems they fell broadly into two categories first the animations didn't always fit in the predicted positions on client machines clients would decide okay to minimize network lag I'm gonna run the assasination over here in this corner but turned out there sometimes was no assassination that animation that would fit there so they would try and then the system would panic and havoc would freak out because you're pushing people through walls and then we would break up break apart the assassination you get people pushing through walls you see all the artifacts who saw in those bug videos second class of bug was at the end of the animation any survivors need to get back in sync on all boxes remember we allowed every peer to play the assassination in a different spot which means all the survivors are the survivor or both survivors possibly are in different spots on every box but when you come out and the guys still alive everyone has to get back in sync somehow

so our system took care of that of course if you received position updates we said oh sassá nations aren't happening anymore get in sync so there was those artifacts like what you saw in those videos that people like warping and blending right after the assassination but this was totally unacceptable for one very very specific reason we just got through playing a very expensive very beautiful piece of content to draw the attention of every player in the game they're all staring at this survivor they're all wishing they were him and then he warps and then we get thousands of forum posts saying that our game is laggy and the player perception is everything you can't draw their attention and then have an artifact so there were a couple of fixes to produce the shipping version first all peers including the participants obey the host strictly this means that we get rid of the problem of discrepancies on exit survivors are always in the exact correct position at

the end of the animation because we paid to get them in sync at the beginning now we've still had the original problem that we were worried about to start with of hey on the entry into the assassination we have to get rid of all these discrepancies how are we going to do that and so we very carefully implemented a you know visual only object state interpolator so that at the beginning of the assassination all of the havoc state all of the is our true logical object state would be warped to so that the assassination could begin and be in be in perfect sync with the hosts but we delayed bringing the physical and our pardon me the collidable and the renderable properties of the bipeds into those locations we blended them over time using a quick exponential blend for like three-quarters of a second even that didn't look great when we looked at it in our debug cameras when our animators looked at this under network test conditions they said that's nice not gonna work that's totally unacceptable there's still these big blends on the way into my beautiful

animation then we played it in game and we discovered that once again heaven had smiled upon us as you may remember from the first the video showing how assassinations work when you start an assassination on the frame you start the camera pulls out into third person to better show the ballet turns out the camera pull to third person takes about three-quarters of a second and during that time the camera is traveling at hmm 25 feet per second the players perspective is changing so fast they can't tell what's happening we can blend 20 feet they don't even notice we tried it their brains fill in somehow consistency because they're confused and then they see the animation happen and then there they just backfill like oh that must have been what happened so no visible

latency no reports of lag out to 300 milliseconds or so totally fine and once again we could ship beautiful things reinforcements all right so all of our experiences on reach and on prior games have led us to these four rules four rules for

designing your game play networking systems first you have to decide for each gameplay mechanic and sort of some some high-level principles as well which parts of your game are adjudicated by a single authority versus the parts that are allowed to be predicted and allowed to be out of sync for brief periods the more stuff is adjudicated by the authority the fairer and more consistent the game is the more stuff that's predicted the more responsive the game is most often you will have to mix these paradigms even within every individual game mechanic as we saw you have to split the mechanic up and make predicted the pretty pieces and make adjudicated the important pieces second always always ask where you're hiding the lag you can save yourself from shipping ugly public beta and networking like we did if you have hosts adjudication occurring you will have lag somewhere if you don't have lag

somewhere you don't have those adjudication you have incurred like debt third don't be afraid to change game mechanics to improve networking this is incredibly important we we have in the past several times had gameplay mechanics that we liked internally that the design was very happy with and that played very well on land games halo 3 is shipping melee is a classic example it was this fairly complex well somewhat complex thing where when two players melee to each other at the same time the player with more health would win the other one would die they felt this added tactical depth to the game in practice on the internet with a few hundred milliseconds of latency you couldn't tell with great precision what the health of your opponent was versus your own health which meant that when you went and in melee it felt arbitrary it felt like sometimes you died sometimes he died you had no control we patched three months after ship two simplify mail it to tear it down and

make it this much simpler thing where if both players shields are relatively low they simply both die designers weren't as happy with this it wasn't as tactically deep but it worked on the internet it was fun and fair to players forth this almost goes without saying because it's so common across all of game to have pardon me and really all of engineering reserve time to iterate networking is incredibly hard to get full test coverage for as all of you who've worked on it know you will test it at your desk you will test it with your coat your co-workers you will have an army of testers bang on it you will take it home and play it then you'll hand it off to a publisher alpha with 10,000 participants and an executive at your publisher will load up their game onto a kit onto a retail Xbox and he'll put it in his car and he will take it to

his cabin at Whistler and he will plug it in and he'll boot up your game in his cabin at Whistler up in the trees and then he will play and he will proceed to tell you at great length on your feedback forums how bad your networking is seriously the Whistler forum thread as it came to be known inside Bungie I'm not making this up it actually was the source of several significant fixes between our alpha and Delta because his connection had attributes we had never seen before he had latency variants we model the latency variance on a connection as like a sort of normal distribution his latency variance the square of a standard deviation was over 700 milliseconds his packets were bouncing around all over the place I don't know what the hell there's a buffering he had with his ISP up there but he had unbelievably interesting bugs and he got the game into all kinds of crazy states

because of his bad network and we were able to go in and fix them took a lot of time so reserve that time all right after you've got your architecture and after you've networked your gameplay you have to go in and measure and optimize your networking to make sure that it's actually as efficient as it as it needs to be this is really really hard for networking unfortunately both because of the test coverage problem which we just talked about and because networking is a magnet for entropy it's a very very complex system it gets more complex all the time because there's always tons of impetus to optimize it it's invisible it's only really noticed when it goes wrong and perhaps most insidiously networking has to work on an unreliable underlying layer on the internet which means it has tons and tons of internal safeguards and fail overs which means

that if you have some system that is horribly broken somewhere in your networking tree you probably won't notice it for a very long time because one of the failed one of the safeguards will save you while consuming a ton of extra bandwidth or being inefficient in some other way at the end of Halo 3 and when he'll tree shipped we had built up a couple games worth this sort of entropy and as a result 16 player games in Halo 3 did not work very consistently there were a great percentage of lag artifacts and so this richest most complex most chaotic version of Halo which we all loved so much didn't really work very well on the internet and we were sad so we said hey let's go in let's attack it let's optimize it optimization is super dangerous it's really easy using your expert knowledge of the system and your belief that you yourself are a smart person which we all share I'm sure to think about the architecture of your networking decide

something that seems like it could be better implement an optimization gain some small amount of efficiency and introduce a ton of bugs again this all comes back to test coverage you can't really fully test your networking or it's incredibly incredibly hard and requires just vast vast resources so changing architecture has to be something that has to be done very very deliberate what this means is the kinds of optimizations you will think of from thinking about how your architecture currently works will usually not be the best bang for the buck really we don't want to be optimizing without good data we don't want to just be you know going in and thinking about functions that are slow on a CPU optimizing or events we think are big on on Network optimizing and going and go in and change them what we want is data this is one of my favorite quotes and the other Michael Jackson first rule of program optimization don't

do it second rule for experts only don't do it yet well we knew we knew we needed to do it unfortunately we couldn't follow mr. Jackson's advice so inspection tools are what you need to get past this problem to get to where it's okay to optimize deep inspection tools will help you identify the things that are the best bang for the buck so we thought about this we sort of figured this out we said okay we need some tools what do we need well let's think about how we do CPU optimization like we boot up the game and we run some representative test case and then we gather data which gives us a nice hierarchical view of where all of our time's going then we figure out the best bang for the buck pieces and optimize them so we built some profilers that seemed natural they would track bad with use for us and priority calculation results and show them on screen let's take a look at some of the capabilities of our profilers that we built so this is from a lab game of reach this video

recording and we bring up this graph view a bunch of configuration stuff at the top data series in the in the center and then graphing on the data series at the bottom you can't read these numbers or you can see a lot of these lines don't worry about it not important what's important is the kind of content being presented here the first line here is showing even the entirety of replication used bandwidth the total amount of data being sent now it's split into five lines this is showing all of our replication bandwidth used by protocol type so state data as one events as another control data and then voice in particular you can see here well if you could if you take a look at the GC vault and look at a good resident version of the video you can see that state data takes up about 80% of our bandwidth hey there's a lot more lines now we've just gone another level deeper we can go inside state data and events and look at individual types of events and individual classes of objects so we can look at bipeds versus weapons versus

items and look at all their bandwidth use we can go even one level deeper which is shown here we can look at every individual property of every subtype of every type of replication so we can look at biped positions and compare them to weapon inventories all of this data is shown of course with a nice graphing view so you can see which types are spiky very very important in any kind of optimization now this is all very useful this gives us a nice like sort of a high-level classification based view of bandwidth use but one other really interesting thing especially in combination with priority information is per object bandwidth use which I've just turned on in the background as a sea of numbers there our profiler tracks for every individual object all of the bandwidth use associated with that object as well as all of the bandwidth that the object would like to use in an ideal world and it tracks all those and

shows them here and they're color-coded by who's using the most bandwidth versus the least and this can let us identify things like individual objects that are sitting off in a corner and chewing up bandwidth for no apparent reason which happens all the time so that's our profiling tool we built this and I don't know about a month maybe three weeks and we decided to go and attack well meeting ahead of myself we had one other tool besides profilers to work with this tool is called films films are something we've had since halo 3 they provide us with deterministic playback of gameplay sessions they're called films instead of deterministic playback of gameplay sessions because they're a user facing feature players can actually record their gameplay sessions and and play them back these are very very useful for debugging gameplay but they've never been really the very for Network debugging because network systems are of course idle during film playback networking is all about live decisions it's all about looking at the current state of the internet and making decisions so that was sort of neat so we

looked at these and we said hey we've got this profiler on the one hand seems pretty sweet we got these films on the other hand everyone loves although the gameplay guys sort of thumb their noses at us with their incredibly easy to repro bugs and we said hey these things sure would taste good together can we make that happen and of course we're engineers we can so we spliced the network profiler data into the film at the times when gameplay was occurring so after every frame of gameplay we would inject a little debug blob into our films that's being they're being streamed out to the console hard drive containing all of the new network state that was sampled during that frame so all the information about every packet we sent and received and all the prioritization decisions for the first time in Halo history we could analyze network performance after the fact which just enabled all kinds of wonderful deep investigations we'd never been able to

do before all right profilers and films are all very well but we still need some way to actually gather data to actually figure out whether the game is working well so when you play test we need to play the game that's the the ultimate test till we ran network / focused play tests about once a month we ran network filters on all the boxes in the lab so that we could simulate all kinds of different network conditions packet loss limited bandwidth latency spikes brief connection drops all sorts things like that and then we had players play and then we ran up into a problem very quickly players can play and they can sort of tell us whether they had a good time or not whether they thought the networking was laggy or not which incidentally correlates almost perfectly with whether they won or not but really this wasn't very scientific and it wasn't very granular it wasn't helping us figure out whether we were

actually making our networking better we needed a more scientific way to measure success so what did we do we allow players to report every time they see lag by pushing a controller button especially a sign for that purpose in these play tests this injects an additional debug blob into the wonderful film allowing us to jump to it later and look at what they saw and investigate why they saw lag this is how we turned this from an art a guessing based game and an incredibly labor-intensive game into a science when we could run the game at our bandwidth goals and get no perceived lag events coming out of the players we knew we had succeeded there were a few bumps along the way most notably as we went through our network optimizations we started discovering that perceived lag events we're still getting put in by

players they were still pushing the button we didn't understand it we would go in and we would look at the at their films and look at what they saw and they didn't see any lag there was no network loggers plenty of bandwidth available instead they would push this button whenever some game mechanic confused them if they got shot from behind by two guys at once and the Deaf camera didn't show them how they got killed clearly enough they would hit the button it like that looked laggy it got incredibly subtle things like the at one point we had it were tuning grenade damage in the Alpha and grenades were tuned up somewhat from from their halo 3 level so a grenade could kill you in one shot much more much more often than at a larger distance and in pre-alpha nerf network earth tests we got a significant number of perceived lag events inserted when players would get killed by these grenades because getting killed in one shot from nearly full health is very confusing and feels very disorienting so we would bring these things to our

designers and send them these little sub snippets of films and say hey these game mechanics are confusing players they think they're laggy and we had science to back it up all right the culmination of all of this I'm gonna show you the same section of gameplay that we looked at before I'm gonna talk about everything working together so this is not a live game it's not a video of a live game this is a video of one of our films playing back so we can go into third-person and view the game from different perspectives we can fly the camera around and view what's happening in the game to our hearts content even more powerfully we can rewind we can go back and we can view a questionable section of the game over and over with these powerful profiling tools available to us there is no excuse to us for not tracking down lag problems given this tool set of

course it gets even more interesting all the so I've shown you up until now is all of the network data of the host box in one big bucket like the graph is showing all the bandwidth use of the host to all clients that's not really the most interesting thing the most interesting thing is this which looks sort of the same to you but you have to take my word for it this is showing bandwidth as sent to the particular client there with the sword only this is the contents of the packets that we were sent we sent to him in this game at this time we can also go in and look at a particular object like this shotgun and press a button and filter all of our data being sent to that one Lille with that one guy with the sword to show only data sent about that shotgun similarly this biped we can say oh this is all the stuff we transmitted about that biped this is the total output of our prioritisation systems the movement of that guy recently every attribute would care about but these are still one second averages oh we can get way more detail than that we have all the data we can show every single packet that we

sent to this client and look inside them in this very visual way to see exactly which packets included updates about this guy what properties they included we can scroll through time we can scrub back and forth exactly like we could before this whole picture is what allowed us to reduce the overall bandwidth use of Halo 3 heard me halo reach from halo 3 by over 80% the total time for development of these tools the film integration and the various visualizers somewhere on the order of 6 weeks plus a significant amount of time ongoing reviewing perceived like events after play tests all right so we took this wonderful

process taking you back in time again we're at the end of halo 3 we're starting up a reach we built this beautiful toolset we integrated it into our films we Unleashed it on Halo 3 on internal builds we said let's see what we find I wonder what the what the problem is and we got this wonderful high-level breakdown of data and this is the first thing we harvested this is even before we had network per fly tests and we got ok 50% of our data as positions velocities and orientations 20% player control so forth we said hey let's go optimize the heavy hitters just like we wouldn't CPU perf this was a false start unfortunately turns out it's very very difficult to further optimize positions velocities and orientations this is very much like seeing your math functions and your CPU profiles they are things that have to be done a lot they're in a lot of algorithms they tend to be already fairly optimal because everyone working on them knows they're gonna be used a lot and so they we didn't get much out of it they were already pretty optimal from even their original implementations so we need to

go up to a higher level here are some good optimizations first reducing always-on bandwidth use this was sort of a traditional networking optimization we would call it we remove data that was duplicated sorry so the control replication from hosts to clients accounted for a significant amount of upstream and we looked into it and we found that it didn't contain positions and orientations and velocities it contained things we could compress pretty well so we went in and we removed a bunch of data that was duplicated we removed data we could infer in other ways and we optimized some encoding by hand very carefully and we got this down about 60% by doing this just because we knew we had we knew it was a significant amount of bandwidth we knew it was worth spending this time so we spent like two weeks on this tiny section of code trying to just hand tweak the hell out of it and this worked out this is the only optimization we implemented that I would consider a success that was actually a networking optimization but yet I have many more slides let's take a look fixing a prioritization bug so when guys

die in Halo they drop all this crap to a couple guns couple grenades that sort of thing these fall to the ground that are stimulated by havoc we discovered a problem with the profiler idle grenades rolling around on the ground had incredibly high Network priority it just showed up to us like they're using all this bandwidth so they're red on the on the object bandwidth display bring up the priority profiler is a ton of data they're kind of very very high priority numbers showing up and we said what on earth is going on so we trace it back and we walked through the prioritization algorithms and we discovered that a bug fix was implemented at the end of Halo 3 which was causing this to happen so we traced back why was this done like this is crazy why our island Grenada Idol grenades have such high priority turned out that at the end of Halo three about two weeks before our sea equipment was given a huge priority boost equipment in Halo 3 was like these throwable things that could spawn bubble shields or drain

people's shields and they were an equipment was laggy and so they gave it a big priority boost and they said alright we're you know that's all that I tested it worked turned out through an accident of design on the gameplay coders side grenades and equipment had the same parent class which was called equipment so all the idle grenades on the ground without their pins pulled totally on threatening to anyone receive this same priority boost consuming tremendous amounts of bandwidth as they rolled around gently on the ground and waited for someone to pick them up simple fix one-line apply priority boost only two active equipment this by itself freed up probably 20% of our bandwidth the whole battle was ident buying the issue alright another example a little different changing game mechanics halo 3 used a constant artificial friction on items these these

idle grenades and weapons on the ground as they rolled around on the ground we we sort we applied this artificial friction to them to keep them from all rolling indefinitely until they reach the lowest point in the level the problem with this was if you had an object on a hill a relatively steep hill the artificial friction would combat gravity and keep the object rolling slowly for a long period of time consuming network bandwidth so not that relevant to the game so we didn't really want this behavior so we went into the item code and the physics code and we implemented this fake friction system totally arbitrary gameplay code no one really understood why I was there except the network encoders but it's this crazy thing where designers can say oh yeah please apply this friction curve over time so that we allow we allow objects to travel down steep hills at full speed without applying any friction to them whatsoever so they're just subject to Newtonian physics once they reach a relatively gentle slope we apply this friction curve that designers specify which starts with low friction and ramps

up over the course of three to four seconds to infinite friction and bring them to a screeching halt so grenades dropped on hills would fly down them in a couple seconds looking better in the process if I do say so myself and then come to rest nobody noticed this nobody the design team cared nobody on the sandbox team cared but this probably say is another 10% just because there's so many items players dropped so much stuff and it gets to cerb so often next example rag doll network King rag dolls are kind of a pain they're deeply deep physical simulations they have a lot of nodes that move around there's a lot of it's very hard to capture all of their state in a small package like if you have a guy who's alive and he's running around playing an animation you can pretty much capture his state as current position direction he's traveling maybe the animation he's playing and you're done like 100 bits of a deal ragdolls on the other hand is like oh god we got to transmit all these bones they're all physically separate they can bounce off things independently what a

mess so we always sort of come Krauss we always come up with different compromises trying to network less of like some subsections the ragdoll never looked that great it burned a lot of bandwidth so we said hey do we have to network ragdolls like really so we're trying we're trying to turn the problem on its head and say hey can we just get rid of this whole problem and we brought this over to our sandbox design lead sage and Merrill said hey can we can we not network ragdolls and we went through the street three stages of gameplay design response to this sort of thing you'll probably see yourselves first shocked what are you doing Network ragdolls we might want to do something with them someday then skepticism I'm sure this whole this will work like that same sort of crazy like does known everybody else network ragdolls and then consideration where they dig in and tell us what what really are the problems with not networking ragdolls for real and they came up with two

things first ragdolls blocked bullets like every other object in our game ragdolls are physically real and they can block bullet trajectories so we needed to deal with this somehow if we were gonna let ragdolls desync because otherwise you'd have invisible objects blocking bullets on different peers second problem was humpin quite serious since halo 1 there has been this tradition in the Halo community this tradition that we didn't want to mess with when you kill someone and you have some time to spare because you're not under attack right now you walk over to their corpse and then you press your Crouch button drop them down like this and you release your Crouch button and you push it again and so forth until you get bored very important for the ragdoll to be in the right spot so that that's humiliates him properly so that he can see that you're doing this to him totally serious this is a real thing so

we made two fixes to deal with these two problems first we went into our collision system in our projectile system and we allowed bullets and grenades to penetrate ragdolls freely we allow them to over penetrate without any side effects we use the debug harness to verify that there were no side-effects was that it added fairly late in the game second we have an assistant to synchronize the initial state of ragdolls so we wouldn't Network them over time as they were flying through the air bouncing around on the ground but we would Network them very exactly with the debug harness to verify we would Network their initial state at the moment of death on all boxes and all boxes would be guaranteed to start with that which meant as long as you didn't interfere with the ragdoll too much it would be in perfect sync in most cases and this worked well enough once in a while if there's enough grenades going off you can have ragdoll just completely desync and it becomes obvious but by-and-large worked quite well and we saved up another 10 12 percent of bandwidth maybe

lastly I have sort of a bucket of interesting cases of smoothing out bursts of bandwidth smoothing out worst cases very much like smoothing out worst case frames of CPU performance first we had problems with high rate of fire weapons machine guns bullets were networked optimally but not the damage that they were causing people who designed the original machine gun networking knew that they would fire very rapidly and so they needed an efficient bit of networking for them so they did that but when the bullets hit targets and dealt damage it would trigger individual events for every bit of damage and that could take all your bandwidth very very quickly if people with multiple you with machine guns were shooting targeted a lot of hit points so we went in and we said okay first four bullets that do a small amount of damage we're gonna just let clients predict some of those damage effects so that the Coast doesn't have to tell them oh yeah this kind of spark occurred and this sound played and yeah yadda yadda yadda like it's a machine gun it's firing 15 times a second like just just play your own sparks another example we had

periodic update of game statistics data so we have this blob of data we send that shows up on the and endgame stats that took priority over gameplay traffic because it was on a protocol below our application layers this would come in every like 10 to 30 seconds and just take all the bandwidth in every packet until it was updated two three four packets and we discovered this with the bandwidth profiler because we get lag events during that time because at the end of that three or four frame period the networking would turn back on for the game and everything would warp and we dug into it with the per packet information in the profiler and discovered lo and behold all the gameplay did is just missing from four or five packets and so we found this guy and we beat him into submission you can only send 10% of each packet maximum statistics data can be can be late it's fine lastly low priority objects were getting updates in perfect sync this is an interesting problem so our priority system ramps up the priority of objects over time to make sure that they eventually the state data guarantee will

be met eventually the most updated state will arrive problem was if you had a lot of objects that were out far far away and they all tied for lowest priority and they all got disturbed at the same time like a pile of crates across the level that gets bumped into they would all get lowest priority they wouldn't get sent and they'd ramp up and priority slowly slowly slowly slowly and then they'd all get really old and then they'd all panic on the same frame and say oh crap we haven't been sent in five seconds we need to get in sync and they would eat two packets to say oh my god this whole stack of crates over here has just got updated and all the guys a guy you're standing in front of the guy you're fighting doesn't get didn't get any bandwidth during those two because he wasn't panicked he got an update recently and so we got lag artifacts from this simple fix again limit objects that can take this panic priority to a fixed number per packet so we always reserved space in the packet for the current live gameplay that matters all right three rules of network optimization gleaned from all this first measure twice cut once

just like CPU optimization used tools it will be worth the implementation time for the tools I promise second in general as long as you have some very basic encoding and compression like you're not mem copying vector threes into your packets you're at least doing you know a little bit of encoding and compression you won't get much benefit out of going looking at that further that that mind goes you know empties out pretty quickly you need to look at the big picture much like CPU perf lastly make friends with your game mechanics designers and coders because as you probably saw four of those five network optimizations were pretty much game mechanics changes you're identifying and changing the game so you make friends with these guys take them out to lunch and so on and then abuse those friendships very important all right I'm almost done in less than five minutes to go it's 4:20 for some of you may have to rush tidbits in the

future I want to leave you with a couple things first some numbers from reach some actual quantitative stuff most important not going to read through all these most important is the top number our goal for reach was to have 16 player games function on a 384 kilobits stream consumer connection so we could be considered to be sort of the baseline DSL at the end our nerf tests showed us that we could we'd have a solid game with no lag artifacts with 250 kilobits per second of total bandwidth in a 16 player game we were very happy with this related best practices these are not strictly related to gameplay networking but they're really important first flow in congestion control just dumping stuff into sockets until the connection gives up the ghost or floods or starts dropping packets not optimal for the quality of your game you implement your own flow in congestion control you can guarantee the ability to send responsive

data at all times or send a low latency data at all times second once you have that flow control layer you can record the quality of each individual players connection record how good their upstream is and how stable their connection is you can save those records and then you can use that to drive your host selection the benefit we got from smarter host selection in reach was on the same order as the benefit we got from bandwidth optimizations because finally we were actually able to select the good host select the guys with the education at the university connections the eyes of the megabit up and completely rock-solid connections third post migration this is hard to add players love it really nice for for gameplay networking quality because if you have a bad host if a host is having a bad day because his wife is downloading Netflix or whatever you can host migrate in the middle of the game to a better host and get rid of all the lag artifacts in five or ten seconds instead of forcing players to live with them consider having a multiplayer beta or

demo almost all the top of multiplayer game timing gears is doing this cod does this it's it's become almost common practice but definitely do it you can get a tremendous amount of test coverage from this we would not have shipped anywhere near our quality if we did not use regular internal play tests with traffic shaping to measure your game mechanics under some levels of latency if you play everything on a system link game it's not gonna be or partly a LAN game it's not you're not gonna get an accurate picture of how your game plays on the Internet consider having full time Network testers this is something that Bungie has which I had never seen before and working on networking on past games and it was something no revelation we have guys a guy in pre-production and then more guys later on in the project whose only job is to stress and attack the networking of the game these guys will keep your your game mechanics designers and your network programmers honest and it will give you an accurate picture of the quality of your game of

your networking over time few more resources a couple of papers you can read this deck will be will be online these give you some sort of more high-level information and more detail on their application protocols many many people work to make Halo Reach play as well as it does online especially these guys Nick Girone Paul Llewellyn John cable and our fearless leader Luke Timmons lead of our networking and UI group lastly just sort of a quick look at the future our system is not completely perfect we have usability improvements to our replication systems we want to implement currently it takes a lot of boilerplate code to implement to implement new mechanics we want to fix that we also want to extend our protocols to support more low bandwidth complex use cases like I have a state machine off here in the corner I just want it to be networked I don't want to get a PhD in replication I don't want to think about it I just want this state machine to

stay in sync on multiple boxes we want to make that sort of thing easy but of course there the real question here is what's really next for bungie what are we doing if you've been paying attention to the the news recently you may have heard the rumor as profound prepended by Kotaku and others that we're making world of warcraft in space as cool as that sounds that's not true it's not what we're doing the only thing I can tell you today is that if you love working on or playing massively awesome multiplayer online action games you should come talk to us we are hiring across all disciplines for many positions I will not take questions

it is 429 if your talk is also if you're going to 84 30 is in South Hall you can totally do it but I'm sure the Q&A will be interesting too yes on the grenade throw yes there is firstly because we didn't need to because we had because we had happened upon this other design which player which were players were ok with and so we never really revisited it secondly because it actually would not have worked for the grenade throw because there's a very specific expectation that the clients have or the players have in Halo which is that their grenade aiming is sampled at the release frame it is not sampled when they push the button or at any time earlier and players are very attuned to this they will took they will hit the button when they know they're

thinking about throwing a grenade and then they'll aim it during the little telegraph so doing anything where we lock in the aiming before that release frame wouldn't work please also be many more questions but please fill out your sheets the correct way to do that is to circle the excellent but please pull them out yes there are two phases to Hana the question was how do we handle host migration in particular when the server just vanishes off the place base of the planet there are two phases to this if the server just drops ungracefully the first thing we have to do is what we call a host election which is a distributed algorithm to determine for all the individuals sort of leaderless peers that remain to determine to elect a new host among themselves once they have elected a new host the actual process of doing a doing game play host migration is very very complicated it could be a talk in itself the core of it

is that in most cases the new host takes his current game state reinitialize --is his networking state his replication state from that does a consistency pass to look for any things that might be might have been out of sync because at the time that the host migration occurred he was in the middle of receiving Network data things like that and then he becomes the new host and everybody joins in progress to him joining progress is fairly simple because we have this split between state data and events to join a game all you need is all the current state data and you're guaranteed to be in sync with all of the current network state of the game I'm sorry can you speak louder yes and that is sort of what the state data panicking that I mentioned gives us

the are to be the priority panicking where we ratchet up priorities over time nothing in the game is more than a couple seconds out of date on any peer so if something is truly irrelevant to you you'll get update and it's moving you'll get updates about it every couple seconds instead of every frame if you do not have there's one special case which is interesting if you do not have the current game state because for example the hosts dropped off the face of the earth while the game was starting up before the the original host could tell everyone what was happening in the game we we call this a double host migration and it's we have to initialize the world from from a clean slate and just we inherit only the statistics data like the current score and things like that and we'll reset all the game state that happens fairly rarely because it in normal gameplay requires two hosts to fail in rapid succession yes priority and relevance is a very

interesting thing it's possible that it's over engineered it's not something that's that's unknown at Bungie well the idea what we think about it is relevance is a pure metric of how noticeable something is so relevance is calculated purely by is it on screen how far away is it what is its radius what is its perceptual like slice of the screen sort of very pure things like that that don't take into account game mechanics or what type of object it is and that that's sort of an algorithm we can write in isolation then we take out all those relevances for all the different objects and we mix in special rules like what type of object are you have you damage the player recently things like this and that is the output of that set of rules is what gives you priority and that just gives us a nice sort of split for be able to evaluate each algorithm in isolation we could we could mix it together detail next

sure this question is whose authoritative over hits when you're dealing with bipeds that are all inside the different positions on all the boxes how we do networked aim-assist what networked bullets and network to damage could be a talk all on its own and in fact one version of this talk was that let's see if I can give a quick summary the core idea of that is at the moment bullets are fired we do not transmit the worldspace vector of what you are shooting we transmit a target relative vector which says you're shooting relative to this target who is likely you're the guy you're trying to hurt with a particularly vector on remote boxes including the host your bullet is reconstructed to have the same lead the bullet is then simulated on all peers and on the host peer when it hits someone it will deal damage that's the broad stroke of it it's a lot more interesting stuff like how we deal with Rockets versus machine guns and how we

deal with the really interesting case of you pull the trigger and fire your gun and then before your bullet your fire request reaches the host you die on the host so when he gets it he's like oh you you think you fired your gun like you saw a muzzle flash on your screen but you're dead how do you deal with that and and the answer is that in most cases we have we have sanity checks and whatnot but we we will construct a fake new gun out of whole cloth fire the bullet and then delete it specifically so that whenever you see a muzzle flash like you will you will generally get your bullet shot with correct aim assist and all that sort of thing next yes yes

the so your question is about the the splicing of the profiler data into the films and and how we get the data about every client into one film the answer is that the host film includes all of the upstream and downstream that he experienced so it includes everything he sent to every client everything every client sent to him and every individual client film only includes their traffic only includes their upstream and downstream so the way the way that we investigate these things is we always work from the host film if you are watching a film in Halo Reach a user user facing retail film that was recorded by a client box and the game had any sort of bandwidth limitation if you then detach the camera from that guy's view and go fly and look at the other side of the map you will see certain artifacts you will see objects updating once a second and things like that yes

yes correct yes so I'm not gonna repeat that question because it was very long I hope everyone heard it the way that works is the appearance of the shield on a guy who is armored locking is always hosts authoritative it is always transmitted from the host to everyone including both the user and viewers so the fact that the host shortens the the timer

automatically causes the shield to appear early on every other box so we don't have to worry about that yes we don't yeah we don't shorts look at the animation in any way just just the blue shield will appear at frame 1 or whatever you're sure yes good question good question fantastic question I love that question it does not work at the field level the priority is per object for state data however we do look at which fields are dirty which fields currently need to be sent and we use that to bias priorities in some ways so for example if the state of whether you are alive or dead has changed your priority is quite high because almost everyone needs to know that so we do a little bit of work with that there's a lot of fancier stuff we could do like oh how dirty is your position like your

position could be dirty by a centimeter and we'll send it and it doesn't we don't really feed that into our prioritization versus something that's dull you know rolling down a hill rapidly it has a 10-foot dirty position we are considering adding stuff like that currently we are constrained primarily by cpu perf in the complexity of our prioritization algorithm the total prioritization cost for a host at maximum bandwidth 600 kilobits ish is around 1.8 milliseconds on the 360 no 360 Harbor thread so sure very good question the the state data updates are not ordered with respect to each other you can absolutely get you wouldn't out of order is an interesting question so you can't absolutely have

this scenario you can have an object change in two linked ways like like yes his position his position changes and his health changes those can be so those updates can be sent in separate packets which means that the client can absolutely get one of them without the other so if there's any any sort of interplay between those two properties there are vulnerabilities there where clients can have some inconsistent state briefly the only guarantee we provide is that we will never apply state we will never apply old state data updates about an object so if you have a you have a state data update about the position of an object and then another one following it on the wire and they get cross packet reordering on the internet happen all the time we will apply the first one that arrives and then we will notice that the other one is older for that entity and we will discard it and that's important because otherwise you lose the guarantee that you will eventually converge to the final state

good question good question we are not considering occluders you're quite right that that evaluating that sort of thing for every potential player camera would be performing would be prohibitively costly on the cpu so it's purely based on metrics that we can derive without doing massive work like that we do have fairly accurate information about every player's camera position and orientation we we do a lot of work to make that happen fortunately for us client to host bandwidth is very cheap so we can shove extra data into the control data for the client to host about like here's my current camera position because I know you can't predict it because I'm using a third-person camera on my vehicle right now things like that or I'm dead and I'm

currently like flying around the world in this flying camera or orbiting camera on various teammates will transmit that data explicitly to the house so we can use it but ya know occluder is nothing like that okay fair enough I guess I guess accurate may be the wrong word like yeah we can't we can't guarantee you that like at the time this data gets to that client the camera position and orientation we use to fill that packet will be the same as his camera that's complete nonsense of course yeah to some extent like for by at least round-trip time he can turn his camera do whatever he wants to to that extent he can see artifact that the prioritization system would have blocked likes we could have some off-screen thing that we deprioritize he looks towards it and he sees under the the underbelly right the way we handled that is we limit the amount of D prioritisation we do for things that are off-camera because that can change very

quickly we are primarily based on distance because that's harder to change quickly and we use a frustum with a wider fov than the basic one for for on-screen tests sure all right big wet sorry yes absolutely as far as vehicle populations and using that to sort of limit network problems on reach we are entirely bound by cpu costs for pretty much everything we do so we do not limit object counts for networking reasons at all we can support something on the order of 30 vehicles

maybe like 30 full suspension vehicles in addition to the 16 players before networking will start to run into problems or do you know gradually degrading priority to answer your question specifically yes vehicle updates are a huge amount of bandwidth because they are huge they're highly noticeable players care about them a lot they catch the eye in Halo 3 we spent a lot of cerf on them in reach we we actually had a major development initiative to fix that and I can talk about that because we have a few more minutes that's pretty interesting so in Halo 3 we had a blending system which we use on bipeds so when you got a network update about a biped and it was out of sync and some just by some amount some number of feet and you had to sort of fix up get in sync with the host we did this this visual only interpolation that I mentioned with assassinations where your your logical state warps and your physic state warps to that location but your your body gets this little blends the exponential blend that consumes the error as you continue to predict forward so it looks pretty good for vehicles we couldn't do this in Halo 3 because of their interaction with

havoc slamming the blending vehicles turned out to be very complicated because of the level of complexity of their rigidbody front hierarchy as it was represented in havoc and because of annoying issues like the dirt kicked up by the warthogs tires is based on an impact system that is driven from havoc contact points of the wheels on the ground so when you warp the physics all the dirt starts coming up from the new locations and the fact that you're blending this object to it looks like garbage like you're clearly cheating so we for reach we rewrote that system that visual interpolation system to work with all object types we fixed all the issues with vehicles so we would we would calculate you know the transform from the the visual object to the real physics object in the inverse of that and we would use that to move things around as needed so any visual represent to any visual elements like a dirt from my wheel would get blended back to the visual position to correspond with the the that visual match being blended the result of that is that we could cut down our vehicle update rates a lot on Halo 3

we tried very very hard to keep vehicles at 20 update seconds rarely below 15 because they were very high priorities how we did that because they look really bad when they warped because we didn't have any blending with this blending we found we could drop vehicles to anywhere downs like 5 updates a second and they would look good it is grass it's very specifically very very precisely the logical position of the object is warped immediately the haddock representation all the rigid bodies and things are worked immediately which allows you to do you guarantee that the the guy will be able to run he won't run into invisible walls for example a bad example let me finish and I'll go back to that the visual representation and the what we call the collision representation which is distinct from physics are blended and the collision representation is all the things that can be shot so if you shoot a guy who's

blending and that you see it hit his his body it will actually do damage it'll be--it'll work correctly so the thing I'm saying about physics the reason we have to work physics is because you can have this case with like a thin wall and you're predicting that you're running on the right side of the wall and then you get an update that says oh no you're on the left side of the wall this is a solid wall but you know fairly thin if you try to blend physics and not warp it it's very hard to get that to sort of consistently well won't look right patek will predict all sorts of terrible things as you push the the rigidbody through the wall right it'll start exploding so we warp havoc and a logical position to the other side of the wall so prediction can be happy and you know think I'm on the right spot everything's cool and I need to be this visual only blend visual and collision blend that's great nobody's behind you

[Music] yeah that's a very good question um v8 when you are driving a vehicle in halo you have the same level of control over it that you would normally have over your own biped so you the way that works for the biped is you as you're running around you transmit your worldspace position and orientation and other things to the host and he sort of copes with it he applies the same blending logic the clients use unless it's insane and then of course he'll correct you vehicle work the same way as soon as you get into the driver's seat of a vehicle your control stream switches instead of sending your biped data it sends your vehicle data the host copes with it everyone blends all the way around and everything you know it looks beautiful on the others there's no latency whatsoever the hope the client isn't actually authoritative technically because he can be corrected but he's authoritative within his bubble sure all right thank you very much everybody [Applause]