How predict rollback determinism will revolutionise real-time multiplayer game development

[Music] so hi thanks for so I'm Eric from exit games I'm gonna share with you here few insights on what are the challenges whether you want to use determinism for building more player games so a little bit of context series we're talking about online player experiences and from the player perspective which is what matters they don't care about how you do it so they just want to perceive that they are in a veritable world that's populated by many players and that's it they want to have that with as little slack as possible so they don't want interfering with a good experience so on the other end of the

spectrum we as developers we have a challenge which is essentially it is there is a game state which presents what the game is and we have to keep it synchronized across all game clients ideally again without lag that's it but that's the hard challenge that we have to solve but because this has been done for the past 30 or 40 years or so there are some commonly strategies they're really well established in the best ways to do this in different genres or different games and these you can divide them in two main ways so so one idea that you can come up with is essentially okay let's have somebody have the game state and transfer data about the game state itself like positions rotations of all the objects in the game from this server etc across all game clients so

this is the essence of known strategies for for example first-person shooters and there are two main ways of doing this when it's called no it's known as an action interpolation another one's called representation is state transfer I'm going to give you guys a little bit details on each one of those there is a completely different way of doing state synchronization which is essentially not sending across the network data about positions rotations of the objects themselves but actually sharing whatever the players are pressing as input and this can be done in two ways that the the lockstep mode which is very well known for RTS games and there is also the predict rollback mode which is the subject of our talk in general so one by one here what is the whole thing about snapshot interpolation it's actually in essence you have a server and as a player you send comments to the server you might have local prediction or nots this is now adays is pretty much mandatory but in essence the server has

to send the full game states to all clients every tick you might think this is kind of like naive it is but it works in certain scenarios it for example games like csgo is to use this approach and it's it's nice because it's simple to implement but then you have to work a lot of stuff like for have like compensated Ray casts and it still uses a lot of bandwidth because essentially you're transferring the whole game state every frame across the network and this is also both uses a lot of bandwidth but also limits the number of objects you can have in the game so so in essence you're limited to what demand the thing you can transfer there is a more complex way to do this which is state transfer so in essence you still have the same thing about simulation running on the server and still transferring the data but now you only transfer partial stuff you only transfer to my client what I should see and okay this might sound ideal but this is actually a lot more

complex to implement there's a pretty famous talk about the overwatch team show it I mean they have a one-hour talk explaining just how I mean half an hour of this talk is just explaining how one weapon was implemented always using this because of the sheer complexity of this approach but it still pretty good it also has another issue which is it uses a lot of bandwidth the reason for doing only partial transfers is not much because of it's better it's just because it's the only way to do if you want to have a complex game world and still use state transfer which is transferring the data about the game itself so if you go the other route which is transferring input the old way of doing this is known as lockstep and this is common on on RTS games although it was also used on even first-person shooters like the first doom which is let's say that the whole start of eSports great stuff out say in 90 was it was it 94 93 that was

using lockstep but um it's good because the only thing you're transferring is input so you have a really really low bandwidth you can scale pretty much the number of units you have moving it's pretty much limitless is limited by the CPU and not by the bandwidth anymore and it's it's you have a lot of interesting features out of the box like replays are very easy to implement and so on and so on the tricky part is that you have to program things in a deterministic way and the really ugly things is in pure lockstep you end up with the problem that you have to deal with input the way because that's the way to do you have to program in a way that the game accepts that you do something here but it will be applying on the game state later because you have to wait for the input to come from the other clients and this might be 200 milliseconds might be half a second on a mobile network so it's very limited in the applications it's more like for for RTS games but this is

you use today Starcraft 2 is to be used stays you have a lot of mobile games in lockstep using this because it works it scales really well as well but fortunately there is another way of using the benefits of inputs based determinism that does not have the limitations of locked lockstep and this is known as the deterministic predict roll back mode this is what firing games use this is what brawler's use and this is what can be used in a lot of mobile games and in all those things it's essentially you can come up with a determinism idea for action games the way this works is you still only share inputs from the clients but what you do is on your machine here think about it as soon as you have put from you you simulate that tick right away without waiting anything and what do we do for the other players the framework predicts what's gonna be their input

based on the past and then eventually in three or four seconds for example the input confirmations come from the other clients may be based through a server or not but you receive the input input confirmations what the framework does is compares with the prediction and if the prediction was right there's nothing to do the gamestate is already in sync because it's deterministic if the prediction was incorrect what the framework has to do is roll back the entire game state to the past and recently all the frames again now with the correct inputs and in this surprisingly is the best possible player experience for twitchy games where we have a lot of interference between players for example fighting games brawlers anything that you need accurate physics it's much it's done much better if you take this approach and if you still have the ideal things of low bandwidth and all the very very complex

part here is to achieve the right level of performance knowing that you have to roll back the entire game state and we simulate the entire game several times when when input prediction misses every now and then so this is what we've been working on for the past three years in a inner product that's well known but I'm here actually to share some of the insights we came up with while building this this is just stuff that you can use your own or just talk to us at any moment we're happy to just talk about in any way so first do you really need this why do you need robots for so first thing is this is the best player experience possible in terms of speed if your game is action basis twitchy and so it's not I think that it's not so well known because of the prevalence of the first person shooter model when people think that maybe I'll just have a server running the simulation just like I do in

pub G but that's not the best way to implement all games the same way that this is not the best way to implement all games but there are a lot of jurors that this is the best way and the but then the other advantages is you have low bandwidth but more importantly especially for teams so the guys we're talking about okay we're four guys doing the whole back end here in the the previous talk so is that essentially online multiplayer is no different than the local game actually you share the same code that you run for the local split-screen version of the game is exactly the same code that runs when you're running the online game so for the development team this is really the idea so what are the challenges then the reality is that this is likely hard to accomplish first because engines are not deterministic so no none of the engines out there out of the box are deterministic at all not only that they're even if they were made to be deterministic they would not

support grow backs out of the box you need not only to be deterministic but all also to support very very fast roll backs and no physics engine is built with that in mind some of them are built to maybe be deterministic but they are not built for roll backs so it also kind of like requires you a different mindset in how to think about the game although this is the easy part because it actually simplifies a lot of stuff but in our experience it seems I mentioned this before it seems like because every a lot of people is thinking about FPS games a lot they always think that that's the only solution for solving multiplayer is one of the possible ones so I'm going to talk about two challenges that we were facing that we're happy to give you a few tips about it so first one is how do you achieve cross-platform determinism in summary you need to write a game engine so we had to write from the math code to

animation tools navigation physics both 2d and 3d as I mentioned before there is no physics engine out there that can do roll backs out of the box and about fixed point math that so this was what we chose to use and talking to other people seems to be the best solution anyway it's like using 64-bit integer to store a number and use 48 bits for integer part in 16 bits for decimal part you might think oh there's another way using like 32 32 bits that's more accurate etc but it has a very very important drawback which is the multiplication operation between two fixed point numbers when using q 32 32 is four times as lower than you can achieve with this because you can do without over four checks so don't even think about it go for 48 16 if you think if you're doing this I I just I found one of the games here that's using the term in ISM etcetera the guys did the

right choice so they came to the same conclusion so it's it's it's interesting so if you think about it really a lot and you do it for for two years you see you come up with this conclusion it's the best possible for fix point and this becomes cross-platform deterministic out of the box as math engine can do so of course we do everything we inline everything in c-sharp we don't rely on dotnet compiler to inline it because on unity would not work maybe it will work in the future but we inline it manually everything we use look-up tables for the things that would not be so fast to do directly doing in code with variable density when needed so we don't explode the memory use the other challenge is how then to do fast roll backs so what you need to do roll backs you I don't need like two instances of the game state one verified with the data that's already like confirm the input and one that's running let's say the tip of the

simulation based on the clock or a ring buffer of game states that you keep keep moving forward we use internal in the ring buffer solution but it's it's fine if you use like the two instances only the slightly simple but it's but for the predict roll back itself it works in essence what you need you need to have it did have a way to copy the complete game state from one instance of this structure that represents the game state to another and you have to do that very fast in essence you need a deep copy so if you have an object-oriented game state with a lot of classes and instances of them all across memory you're screwed because you have to implement the deep copy because something like this would not work I mean if you have object a and you have a copy of our basic baby but the it reference this object B this is fast to do but you keep a shared copy of B and this is a problem because if you modify B you have the modification on both copies and this is not what you need so

you need to create a deep copy mechanism and in the object orientation this is gonna be heavy I have a few numbers to share with you guys on the experiments we did with different kinds of data structures and different kinds of implementations of this on c-sharp so the the number itself is a specific machine but for comparison sake it's all on the same machine same size and same thing so if you have a very very basic gamestate structure kind of like that and you have a hundred thousand instances of that thing is just an example the most naive way of copying this would be serialized and then deserializing another copy and that's of course not a useable thing but for comparison that would take three seconds to do then if you implement a manual deep copy or we are much better you get you can do that in 100 milliseconds and if we're talking about 1k instances we're talking about one millisecond but it might be good really because 100,000 units might be too much but notice that this is a very

very simple and shallow gamestate there's not much data in here and the thing is that if you think about structs in c-sharp for example and not not objects which are blitt able they live in a stack or they can be put in a buffer or in an array just using array copy puts you down from 100 milliseconds to copy the whole thing to 3 to 5 milliseconds right away so just moving from classes to structs in c-sharp then you can you can understand which was the option we went for if you go one step further instead of using a normal shop robbery you just have a buffer in memory and you do a mem copy on top of the same amount of data it's like even five three to five times faster so that 1,000 instances is going to be one hundredths of one millisecond to do the rollback so that's what we choose to go to so if you can do that if you're going with array of breathable struts could be okay

as well but don't try anything else so a few tips to end here so we came up with this thing that's at the game state that our framework helps you create ends up being like this memory align little struts we're using a lot of old-fashioned pointers and so on and so on we never allocate anything on the heap so no garbage collector being called because of simulation so depending on the game this might fit into the l2 cache of the processor this is so nice because you also get a lot of performance benefits from getting cache hits while you're doing the game loop it's it's surprised it's just something that comes out of the box because your game state is on a buffer and it fits into the l2 cache and this is really really common and even in mobile games some of our customers are able to do this because of these ideas always in line everything even your code so you start without it then you later in line

it you can get 10x performance improvement just by in lining stuff we do a lot of multi-threading in internally for now we're not exposing this to the customer so it's more like that the engineers do that and it comes out of the box for example on our physics engine and so on so if you're thinking about doing something yourself it's it's nice to first have it working well for you as a team later you exposed to whatever else whoever else is using your stuff even on inside the team about API one idea that we came up with that is working well is don't expose anything networking wise so the game developers write the game as if it was a local game that's that's something that it's by design we don't even expose rollbacks on the API they from the code you don't know what robach is happening although game design wise you are aware of it and you plan some of the visual Corrections

and interpolations you apply are gonna be aware of the amount of rollbacks you're gonna have but in the code itself you don't have to about it like do a different code when a rollback is happening or not because that would just actually break determinism we use a lot of cogeneration this has been the subject of several talks have been to in different contexts it's always nice if you can use cogeneration it helps a lot it helped us a lot as well in several stuff and also write good simples even as a internal team if you write a to write a sample of how to use that tool it's going to help the other developers to get shop and use it correctly with the right patterns and that's it I think we still have one and a half minutes thank you for listening so we are close in time but I don't believe the next speakers around quite yet so we have time for a couple questions up here

there is we are we try to be pragmatic so we started with a buffer a pre-allocated buffer it's working really really really well for all the customers we have but we already working on the customer locator idea anyway because it also works it's also a nice idea and of course it it makes sense to move forward to that that's a very good question I think it depends a lot on the speed of the game what uh this this game in the

middle here it's it's a very very fast-paced shooter and it's available on Steam as a tech demo it's called blueness you can download and try yourself and then you see how much rubber-banding you see and you can compare two screens you see the pink and all but what I know is that most of the time we're not gonna see any rubber-banding and what they did is that there are some we have a lot of control and how much interpolation you want to do for the visuals what they do is they do an adaptive change on the interpolation factors so when when they notice that that character jumped they use a little bit more interpolation so it catches up fast but most of the time they don't use a lot of interpolation so in essence you don't see a lot of rubber bending when they move direction it's it's just a matter of it's more a matter of you find tweaking the dynamic

interpolation you use on the visuals to reduce rubber bending that's only it only exists since very very very fast-paced games so fighting games for example they normally don't add any interpolation that just corrects it's the players are fine with it yeah it's what we try to do as a framework is we expose it so you can choose what to do [Music]