Insights on determinism

[Music] good morning I'm Eric Christa's here at the audience so maybe I will have some questions that he might be interested in yes in answering so first very quick thing about like we're gonna talk a little bit more about like traditional tech of how to build multiplayer games so why are we here in a cloud-based session so this actually makes a lot of a lot of sense it's related to the question I asked about like how do you communicate between servers because you don't want to add more latency knowing that your engine does a lot of latency and error controller in your TV do a lot of latency so cloud gaming is not the problem in here but we used to have the latency problem so we don't want to add any more latency on top of that so this is one thing that might help a lot in that and I hope that will be clear and last thing is before we start is

although we do have products about this stack etcetera this talk is not about the product is about the ideas and attacks so please ask any question you have we'll be happy to answer so just like cloud gaming determinism seems to be like a game developers dream so you you focus on doing only the gameplay code you don't think about network you just think about okay I have a computer there are many controllers because there are many players but you don't have to think of them as being from different computers this is the premise that you can this is the thing that you can achieve with determinism so it sounds like a dream as I describe it why aren't like everybody using this why isn't everybody using this right now so then reality kicks in and here is the thing it's it's very hard to accomplish determinism it's difficult to to implement these things and especially the most known implementation of

determinism which is the traditional lockstep model is very limited it has a lot of issues including a lot of latency so this is clearly not very useful anymore although we has been very sick in the past and last thing is used to use engines to be the games and engines are not built to be deterministic neither of them are built to be deterministic like this so let's dig a little bit deeper first by understanding a little bit about the history of this so by the terminus tech architecture like a pure deterministic architecture what we mean is that the clients either them being like different servers in cloud gaming or the real clients in traditional sense the idea is that they only exchanged input with very very low bandwidth requirements and they do simulate tick by tick based on the complete input set and if you use the terminus like algorithms and data structures at the end you're gonna have

the same gain state at the end of 10,000 frames so it's very simple premise but in the practical approach number one which is traditional lockstep has been here for 30 years with a lot of success especially on the RTS chair for example Starcraft 2 still runs pure lockstep model and but it has a lot of issues first one is that you have to apply input delay because you cannot fake the speed of light and you have to wait for the inputs from the other clients you let's say be able to simulate exactly like the other clients so clients have to exchange input so what did you normally is you apply input delay before you have you can actually simulate a frame that's clearly not good and it also all breaks down if one of the clients starts to run slower if it has high latency especially pink spikes which is pretty difficult to handle in those old times then let's move forward and let's bring us to 2018 or bay post

2010 so the idea here first one is let's get rid of lockstep completely so maybe some of you guys have heard about gGPO and what it did for fighting games and all that so the idea here is that you don't have to wait for the other guys as input anymore as long as you have the input for the local player you just simulate that tick so you go back to let's say like a single player experience with zero lag well not zero in all the controller the TV input lag etcetera but er you get you get to the same experience as single player and what do you do with the remote pointers you you just do a very very simple heuristic prediction of what he's gonna do if he's going left he tends to keep going left for at least 20 ticks running at 60 Hertz for example and what you do is when eventually the input arrives you can roll back the entire game state and really late to fix the miss predictions but but normally these miss predictions

are only for the remote players not for the local one so the local one doesn't change anything and you just have the single player like zero latent zero latency experience but then this is what you could do when gGPO was still let's say available but it only worked for one versus one fighting games for example it was used on Street Fighter 5 and latest mortal kombat but then there are few tricky things if you want to do this for 3 for 1030 players games or more and then you have to have a server in the middle to coordinate input exchange and in take care of keeping the actual game instances completely in sync in terms of clock because hardware clocks drift you also need to manage what to do when their latency spikes even thinking about running this between two servers you

want to have players in the US with the last mile server playing against players in Europe in Amsterdam the server is running you still have to communicate between these two servers and there might be some issues between those so here's the thing here is that you do stuff in the stuff that you need to do to achieve non lockstep predict roll back determinism requires a server in the middle that coordinates the input exchange and this is something that you normally don't find on the internet forums there is not much information about but then let's just dig a little bit more on the actual challenges here so essentially you need a new game engine to run this you you need you need to run different math because floating point math is not deterministic so you need a complete library for that you need new physics because no physics engine that you get out of the box is deterministic you need same for path finding same for

if you want to try control of animation so just a little bit of teaser this is exactly what we do we have everything in-house because we needed to have this but if the good good news is you can do that it's not it's not impossible and then the real challenge is how to achieve this near zero latency experience with rollbacks and here's the real challenge is how to do this very very fast how can you actually save and restore the game state to a previous version and whatever is the the architecture you use for the frame buffers etc you end up having to do this you end up having to do a deep copy of your game States so this is just a quick example of what that means if you have an object a that references an object B if it was a simple copy of a be a clone of object a you still keep the reference for object B this is not what we need we need to do a deep copy which is we need to copy the whole contents to another part of memory for example

that's the only way to do rollbacks and how can you do this really really fast so just to cut the story short I'll just show you some numbers of an experiment I ran so if you are very naive you can try to just serialize and deserialize the game state but then you like this is like 100 K instances of an object that has to what for other inner objects of two different types so serializer will be like three seconds to do the whole thing so it's pretty bad clearly if you do a manual implementation of a deep copy which is what some let's say Stack Overflow forums are recommended to do you get down to 100 milliseconds to do the the rollbacks you're not good enough but any let's let's not use objects and classes anymore let's just use whatever is your link of choice let's use plain value-based structs that you can just clearly copy without using the heap so if it in

c-sharp this this was running c-sharp so if you use just a struct array and you do in a rate copy of this you get it down to five milliseconds and finally if you decide to do just a native man copy of course it's gonna be better so it's one millisecond oh and you can you can understand what was the choice we did in terms of what to do for robots so basically please do mem copy if you're gonna implement that anything else is gonna be too slow so in summary that's these are the design choices the technical choices we did which is basically keep game state as a memory round block of starts whatever the language you choose and use native mem copy for roll backs and old fashioned pointers in line everything so math library you were talking about fixed point not floating point so in line everything don't think about anything else it doesn't help you know of course never allocate anything around time that's that's regardless of what

you use so in terms of API designs we chose this is about product we have so we never expose networking code just like cloud you don't think about a network at the game as networked you just write forward simulation as if you had more local players we try to protect common pitfalls with pointers even if you're in doing this in-house you you have to try to achieve this so new developers don't fall into the normal pitfalls and just quick thing about multi-threading our internal idea now is that we we're exploiting this a lot internally but we're not exposing this with the developer too much we think it's it may actually add more overhead because many of the tasks you actually do in gameplay not in rendering in gameplay are inherently sequential you are modifying data that depends on another modification of data etcetera so if you try to do that multi-threading adjust at the overhead in most cases so

we do exploit it internally but we try not to expose too much so that was my summary in the in the time we have here I have time for questions I think so so yes we do have time for questions people are so serious come on guys I know the party yesterday was hard but you can do it and you know more technical things that I do I could try to be smart and try to very interesting so so do you provide a developer some sort of libraries that takes really care of all that and how easy is to use and where can you get it yeah that's I ever said I was not talking about the product but the ideas behind it yeah we do have a product called photon quantum that is

exactly all that we do have the physics engines 2d 3d nav mesh agent it's it's completely decoupled from the rendering engine so it's agnostic to that as well and it's from exit games photon you can we can just talk and yeah so so people with custom game engines are using different engines they could be utilizing your secrecy exactly exactly because it's not tied to to any particular engine although we do have a pretty good integration with unity is not at all I mean the simulation is completely decoupled you write it in a separate videos to the solution that does not know that does not even know the engine exists so it's just pure simulation right are there any requirements for the engine like you mentioned like you need to basically have a flat array of memory like what you can mem copy so like a stereo requirement or you we have flexibility for you to add custom stuff on the game state if you want to do in advanced way actually we have several layers you can even not use already fault approach which uses the main

memory aligned based struts for this one we have a custom is yes where you we have a DSL for you to define the data structures and this get converted in superior memory aligned blocks but but you can actually do everything custom if you what all right all right [Music]