3 Layers Every Online Multiplayer Game Runs On

I always think about multiplayer game development as the holy grail. Multiplayer is the way to develop a game and single player is a subset of that. And this is a conversation with Eric Pasos, a developer at Photon, a company that creates online multiplayer solutions for video games. Most basic is you have a transport layer. Second level would be object synchronization libraries. At the highest level for me, it is SDKs that really tell you how to develop games. We have products on all three levels.
The tech powers a variety of titles both big and small such as peak repo content warning, wind blown, stumble guys, fantasmophobia and more.
There is no silver bullet. People normally come in as oh I have a I have this game I want to add multiplayer to it. Although possible, this is rarely the right way to do. You design the game as multiplayer and single player is a byproduct. Online multiplayer has always been one of the most requested features in my own games. So, this is part of my

personal research journey. It's time to finally wrap my head around this online multiplayer thing. So, you need to have a single source of of timing otherwise uh clocks will drift.
I found the conversation extremely insightful and I hope you do as well.
What I'm talking about here is not the common thing people talk about when they develop multiplayer. They think about, oh, to synchronize my objects, I need to spawn and synchronize. For me, this is yeah, that's the basic. I I like to think about multiplayer in three different levels of of a I mean multiplayer SDK or tooling development. It doesn't matter if it is a commercial tool like the stuff we do. If it is you're doing in-house for your studio, a custom one that works really well for the type of game you're building or it's agnostic to that. I think about it all in three different levels. most basic is you have a transport layer that can send bite arrays can send bytes between machines either client server relay it doesn't really matter is transport layer you send data have a way to send

messages that can either be reliable but without guarantee of time and then unreliable that can be faster but it's not guaranteed to arrive so that's the most basic level second level would be what people talk about more commonly which is I call them object synchronization libraries. So they they help you serialize properties on spawn objects based on I spawn it here, it will spawn there. I spawn on the subway, it will spawn on the clients and if I modify the data in the properties that's going to be transmitted one way or another. There are more efficient, less efficient ways of doing it. But this is just object synchronization. You have many many of these libraries and some of this stuff out of the box on on these engines stops there. So if you think about Unreal's net code, the net code itself just does that. You have in and RPCs, right? So we RPCs are message passing with a little bit more metadata. They convert they're they serialize some

types in as parameters and they can match where to call the method, but that's it. It's it's transport layer level combined with object synchronization. Then yeah, you have the character controller in Unreal that does client side prediction and then you have game ability system which also does client side prediction but they don't talk to each other. They're individual independent implementations of a client side prediction which is something I believe should be done at the net code level but they have a legacy. They have something that works that that's done that way. Uh I'm not criticizing it. I'm just saying that it is an object synchronization layer if you think about the net code at the net code level itself. Unity's net code for game object is that as well. So it synchronizes objects but it doesn't really go too deep into the concept of time and so on. Then at the highest level for me it is SDKs that really tell you how to develop games because I always think about multiplayer game development as the holy grail. Multiplayer is the way to develop a game and single player is a subset of

that. So, so people normally come and ask, "Oh, I have a I have this game. I want to add multiplayer to it." Although possible, this is rarely the right way to do the right way. You design the game as multiplayer and single player is a byproduct
because it's if you solve multiplayer, single player is solved on the technical level as well. Um, and some tooling that you do for multiplayer is also useful in single player like replays and and things like that. This last level is SDKs that actually solve the concept of time and accurate presentation of the objects and the game state in general. Solve this at the net code level and and inevitably they influence on the way you have to develop the game. You essentially have to develop the game already thinking on the APIs of these SDKs and
we have products on all three levels. So if you think about photon real time is on the most basic level. If you think

about our legacy library like pun which is our most famous product it is on the second level. It's an object synchronization library that's client authority.
But then if you think about quantum and fusion they are um on the third level and that there are other um people doing that kind of thing especially in inhouse stuff. Riot uses um Unreal for Valerant but they built their own net code for that. Or if you think about Valve on Counter Strike, so this is top-notch stuff and and they use this they they do it this way. So they they they solve the concept of time, they do all that and and or if you think about games like Brawhalla or even Rocket League, they all had to go down to this type of problem to solve. Some of them solved it uh in a specific way for one game or one genre category. Some went a little bit beyond. But that's that's where we try to position ourselves is solving this type of

problem at the net code level because I think this helps a lot the developer. they can focus on on game play and on on uh the challenges that are there instead of bumping their heads into these problems that are reusable. You can solve them for multiple different games cuz the you you ask the original question how complex it is multiplayer cuz it is complex on the technical level like we've been discussing here but it is also complex on the game design. If you're going to do a multiplayer game now you're going to see all of the different things. Okay. What happens if the the the other guy's a troll? What happens if you have three, four players? How do you balance that? How do you think about different roles for the players to that they like to assume? Our goal is always to try to solve the technical challenges and help you focus on the game design challenges and and the and the stuff that really makes the game uh fun.
Would you describe with the three layers? So would you for example say that the steam relay servers are sort of

the spot layers like uh how do you call it again? How was the first layer?
Transport layer. It's a transport transport layer. Exactly.
Yeah, it is exactly a transport layer and and steam is specifically it's even more restrictive because it is specific to Steam. If you're releasing the game only on Steam, yeah, fine. But what if your your game is successful now? You want to have it on Switch, you want to have it on Xbox, it doesn't work. Steam only can only be used on Steam. And yeah, you cannot you would have to rewrite everything and make sure and and and even if you say okay, I'll leave it like that. My port for Switch is going to use something else. Then you don't have crossplay. We have two main products that implement multiplayer in two different philosophically different ways. And there is some intersection on the type of stuff that they are good at that you could do with either. But in general there is an envelope for one of the SDKs of the type of games that work better and then other types of games would work better with the other approach which is more similar to what

other tools do as well. We don't believe that there is a single solution. So if the like and real code only has one option and one way of doing things. So whenever people are doing something that doesn't fit that they end up building custom stuff in house because that would not work. So we try to cover a bit more. We do not solve everything. So there there are some genres that if you come and askism no maybe maybe you get a transport layer it could be ours and and you would need to implement this this and that's on top but we try to cover the most popular and and challenging things that's what we try.
Got it. So the the game needs to match the tech or the tech needs to match the game that you want to make
or you design a game around around the right envelope of a certain tag. That's another way of thinking about it.
Um because it's better. You were always designing based on the envelope of the media you were designing for. You were designing for a screen. You were designing for a certain hardware that is

capable of some type of presentation. You were you were designing for a certain type of controller of some type of input. So you always have envelope restrictions on what you're doing. So the net code is another one of them that you can think about. point is if you think about quantum which is something you you mentioned you were interested in in another video of yours um it it does have an envelope where it shines where it it's really unmatched in terms of what you can get out of it but it doesn't solve everything. It it if if you try to do some things with it it would just not work. It would not be the right solution for you. And we're the first ones to say when you when you when
I mentioned this to you when we're discussing before um that um when people come asking us questions some many many times the first thing we do is ask back what is that you're trying to build what is the type of game can you describe it a bit the mechanics are there similar games in terms of what you're trying to build even if it I mean you're not

cloning this but can you uh give us a couple of references of the type of stuff you're trying you're building and based on that is that we then start giving um suggestions and and recommendations. Oh yeah, probably you should use this and that that's an option. So here there are these couple of samples or maybe when it's a very very interesting thing sometimes we even say let's prototype something together for you and uh we may help with that. It's rare that we do that but we do sometimes. I personally love to work on prototypes. So yeah prototyping is fun. We we love the protoing phase.
It's the fun part. Let's maybe um work our way up this this pyramid, the three layers. And like I guess in can you come up with any examples for games where maybe it would make sense to start with that very bottom layer with just the transport layer or is it or should you almost for any game start with with something higher or in which case should you build your own stuff from the

transport? For me, you should always uh start from the top.
Okay. Okay,
the only reason for you to start from the bottom is okay. There are some cases where you need to have multiple layers at the same time. So I can I can think for example of an MMO.
Mhm. People many times ask, "Oh, I want to build an MMO or should I build an MMO?" Was this your last video?
Yeah, that was the last video.
And the thing is, first let's define what an MMO is because there are different types of MMO. So you mentioned Tibia which is a very interesting game and so on but some people mention as MMO uh clients of a game that runs up to 200 or 100 players on one instance and MMO is normally a bit more in my definition is a bit more than that. You have to have a more seamless big world where you don't notice that you're going from shard to shard. they were moving from instance to instance. But you try to make that a little bit more seamless.

But point is that many people have been trying to build MMO tech that solves all the problems of MMOs in one shot. But if you go look at really how MMOs are made, they need different solutions. So sometimes you have you use one paradigm let's say for the overworld. Maybe what you do for the overworld is you use client side authority for the movement. you have a very very lightweight thing running on a server that validates just that you don't teleport or something like that, but it lets you pretty much move around freely on the clients and you're just updating your positions here and there and you're chatting with people and you're inviting them for some raids or some instances. Then what do you run on on a raid or or or on an instance that is a PVP instance? Then you can use a different paradigm. Maybe use determinism. Maybe you use a more uh isolated client server that runs full server authority for some stuff. So um sometimes you just need a

transport layer to do chats or some other things around. So an MMO becomes such a beoth that you're going to use all different layers in one shot. But
okay,
when should you just use a transport layer? when you're doing something so special, so different that there is no offtheshelf full solution on on the different layers. One thing I like about our attack is that even when you're using a transport layer, it is a little bit different than the others in that because it's always relayed. There is always um our cloud on the back. It's not like a dumb relay. It allows you actually to write custom code to run on our server if your game grows big enough. And this is true for every single one of the text stacks. So even if you're doing quantum, maybe you've seen that it's deterministic. It's cheat proof in that people cannot modify the

game state. They're just drifting their simulation. They cannot influence the other. So you cannot move faster
on the other players machine or something like that. But and together with the web hooks we have you can validate loadouts and a bunch of stuff. You can put the replays you but you can also run the full simulation on our server because the simulation is agnostic to Unity. You can do that but even Photon realtime which is the transport layer allows you to write code to run on the server. So and this is kind of unique because if you go to steam relay okay there is no option for you to upload code to run on their servers. No. Unity relay? No. Epic. No. So, we're pretty much unique in that even if you're using the transport layer, it always allows you to put lightweight code on our server that intercepts every single message that's going between the clients. You can block, you can protect, you can do state authority stuff there. You can validate, you can do whatever you want. So when we're doing something that is so custom

that does not work with any offering from us or from other people many times it is interesting to take a look at our transport layer level and build something on top. We rarely recommend that because it's very rare that you have something very very custom but if it happens it happens. Some people built MMO tech on top of our base stuff, but they they went a little bit beyond that that they didn't they don't use the concept of of uh sessions rooms that run on that we offer in our cloud. They actually very early they got um to develop with our Photon server but a totally different application on top of it and they built an MMO on top of it which is Albon online. So, uh, it's a successful MMO that's been running for, I don't know, a decade or so, maybe more.
Yeah, I've heard of that.
That uses our transport layer and our server tech in a different way.
Awesome. I see. And then the the first product basically that you built on top of that was a second layer product,

which is just object synchronization. And I I guess the idea of object synchronization is sort of, right, every player has their own little chessboard and you go on a phone call. and I'm like, I'm moving my piece to E4. And then the other one just does that as well as soon as they get the memory.
Practical. It is practical. It's easy to use and it works and it's incredibly successful. You
I'll I'll list a few games for you and you you recognize. So, but a little bit of history here. So um vote on exit games as a company started in 2003 and and this uh transport layer attack with the vot server cloud I think it's it was from around 2010 2011 and that's when unity was becoming extremely popular because they released their Windows version of the editor.
Mhm. And um they had a racknetbased um client authoritative uh API called Unity networking which was an object synchronization layer built on top of recnet but it rarely worked because they

were expecting you to be able to just without doing a proper n net punch through it was actually net net punch but they didn't have a relay fall back and it would fail 20% of the time and then people would just not be able to place. So it was not reliable. So back then we built a similar API always on top of our relay and allowing you to write server code and that became pun photon unity networking and that's our most famous product the product we're mostly known for which is an object synchronization layer that's extreme extremely convenient and has been extremely successful. It is simple. It is client authoritative. So it it is it is it's not uh super cheat proof in that sense because it's client authoritative by design. Right.
Mhm.
Um we've moved beyond that many years ago when we started offering photon boat back then and quantum and then fusion to

replace boat and pun in one single uh SDK. But pun is still super popular and people still build super successful games with it. Let me list a few for you. Peak.
Mhm.
Peak by landfall that uses pun.
Okay. Crazy.
Then then you have Oh, Ripple.
Fasmopobia. It works for these guys and and
people like it because it's so simple to implement
and there's there are so many tutorials about it. But I I I mean I would strongly recommend you to go to the higher level because it is so much easier and it protects you from cheating. It it does all this stuff that people complain about older generation stuff. It's solved on the new stuff.
Timing is solved.
Cheating is solved with the exception of stuff you cannot really solve at the net code level like aimbots that you cannot solve on net code. You solve differently. Let me mention some successful games built with newer tech. So gorilla tag VR incredibly successful. That's fusion

that uses uh photon fusion. Then you have uh quantum stumble guys
which is massive. Massive,
which is a game I built a prototype for
5 years ago or so.
Oh, that's that's cool. Awesome.
33 Immortals by Thunder Lotus, which is in Game Pass, which is a pretty
interesting 33 player game. A little bit like Hadis, but I'm sorry about my comparisons and descriptions. I'm I'm I'm really a tools developer. I'm not really
a game design,
but I've I've seen that game as well, actually. It looked really cool.
Wind Blown by uh Motion Twin. That's also quantum. It's super fast. That's a super fast quick twist.
Quantum.
That's quantum.
Yeah. Yeah. Yeah. It It played really well. We like we played a while ago and we were like, "Oh my god, this feels so responsive and juicy."
That's qu There is a story we wrote with them. It's on our blog. I'll send you later the the blog post about that that they describe how they used quantum. They they they had some pretty interesting uses for replays in

development that we didn't thought about before. I I forgot here, but I'm going to send you later.
Okay.
So, you you have a look at it. So, it's there is another one called um so maybe you know Moon Studios by AI in the wheel of the wisps or blind forest. So they they have this new game called No Rest for the Wicked, which is a top down. They're pitching it as an action RPG, but it's a mix between ARPG and Souls like game that's also quantum. And they're about to release the co-op uh in in the next few months. And uh I guess that's going to be very interesting to see player reactions because you played Wind Blow, right? So and oh, this is juicy. This feels good and and all that. So imagine that kind of accuracy and twitchiness on a Soulslike game where you have to plan your your dodges and your attacks and you have friendly fire and all that. So that's no rest for the wicked.
Um and it's a gorgeous game. So that's

also quantum. So there there are many things being done in quantum a lot in the mobile space. There's a there is an interesting tower defense from Korea. I think it's called uh I forgot the name of it anyway. It's not relevant. But there there are many people who found out about quantum especially after we released it as a freeto use SDK and there is the
Yeah, let's get let's get get into it. So your two flagship products at the moment are quantum and fusion. Can you quickly describe sort of the philosophy of each and what their strengths and weaknesses are? So there there are basically two ways of developing multiplayer which is if we go back to the beginning of our chat here the challenge is you have multiple devices multiple computers and the games running on them. So the the actual goal is to make sure the game state is synchronized is that you have the same game state at all times without too much latency in when things change. That's the goal.

There are two main ways of doing that. So you either keep transferring the state changes themselves. The other option is to if you have a deterministic simulation, if you are capable of implementing a deterministic simulation, you can transfer just the input. That's incredibly hard because you have you cannot use floats anymore. You have to you cannot use Unity as the game engine because Unity is not deterministic. You have to have your gameplay layer and that's but we ended up with a solution for that. The point is these are the fundamental ways within the state replication variation in which you transfer state. There are two main subvariations. One is that when you have a single source of authority a server and you have client server model in which clients only send player intentions input and the server is responsible for actually changing the state and transferring back to the clients. Clients can still do prediction of changes but the server is ultimately

responsible for the final variation of it or you can have distributed authority or client authority in which each client is responsible for changing the state of some of the objects in the world not all of them. So um you can have some mixed variations but these are the fundamental two ways of representing state and then there is determinism which is another way. So we have fusion covering the state replication. So it has client server model pure client server model with server authority both for you to run as a dedicated server a unity headless instances dedicated server or as one of the players being the the server with a net net punch and um relay as a fallback uh with a host migration and all that. But programming wise it is the same as dedicated server model just that the server also has a player and a rendering layer. And there is this what we call shared mode which is the client

authoritative var variation of fusion that uses the same APIs with a couple of exceptions that are not available. So you don't use the input loop because there is no need for it but otherwise the API is the same. And then you have quantum implementing deterministic predict roll back on this other side. So with this we cover most of the different types of genres. What what do you use them for? What kind of stuff? I mean if you're doing a shooter we're normally going for client server with client side prediction and lag compensation. That's the stuff we have built in interfusion. There's a reason for that. Everything that we do in fusion in terms of client side prediction actually quantum does even easier and better client side prediction than fusion because fusion for physics prediction for example is limited by physics that cannot roll back on unity quantum is not because we use our own physics engine that's designed for that but but we have like compensation we also have sample for light compensation in quantum we can do the same
but there is a big reason for you to use

client server model in a in a shooter Because when you use determinism, it's cheat proof in terms of people cannot modify the game state at will, but they can see it. So every single client has the full game state. So what we call map hacks, which is if you scope into memory, you may be able to see uh data about the entities that maybe you don't want to show to every single player. So in a shooter, that's very very sensitive. I mean it's still a problem even in client server when you have somebody behind a wall because you already have the information about that guy on the client anyway otherwise you would not have it when he runs out. So people can still do aim botting with that even on client server but
uh at least you don't know where everybody around the map is unless you because state synchronization can have what we call interest management and of course fusion implements that based on area of interest or custom options. this player should have access to this object or not. So you can set that kind of

stuff. So if you're doing a shooter normally go for that and
do you if you want to do a server authorative do you have to write a separate program for the server probably right?
No in in client server mode what I'm talking about is that you is the same code that you write that runs on the client runs on the server. You can if stuff so you can have stuff that runs only on the server if you want to but you are writing your server in unity as well. this okay what I what I mentioned before that even in shared mode which is the client authoritative version the advantage we have is that even though it is client authoritative it's not a dumb relay it's not that our cloud is just passing data around it actually keeps a copy of the game state so you can write server code to intercept and validate and modify the changes but that is on the client authoritative variation it's a lot to grasp right so
yeah yeah so I Hey, I guess to break it down a bit, there's fusion and there's quantum. And both of these are each

available in server authoritative or or basically not just relay versions, right? No, let me try again here. Yeah, client server is one way of doing net code. Is one way of doing net code. Fusion implements that mode. By client server here I mean there is a server running uni running the same game engine as the client but it is the only one responsible for changing the data on the objects. Clients cannot change the objects. That is one way of doing multiplayer determinism which is what quantum does. It's a totally different way. You don't need a server to have the game state at all because the only thing you're exchanging is input. There is still one server but it's to coordinate timing input stream validate some other stuff but it's not running the simulation
right but I I I guess for quantum the the question I have more is like if if you want to make something highly competitive then you probably still need

a server to verify game results at some point right excellent question we have an entry in the docs exactly for that but server is say is it safe yes absolutely but it is also expensive so quantum is interesting Because when you use determinism without the server, without the validation of the game results, you're already protected against a lot of annoying stuff. So people cannot ruin the game experience of others like by teleporting, moving faster because they cannot really control the simulation. They can only send their player intention. Let's say, oh, I want to move in this direction. I'm moving in this direction. They cannot really say, I'm moving faster. Unless you make a mistake in your programming model. But if you as long as you follow the examples, you just understand the basic concept, you're sending just intention. I want to do this. Who says you can do that or not? You're your simulation implementation that's running for everybody. So if I'm not a cheater, nobody can cheat the game state that's running on my machine. They can only send their intentions. I call it

simulation authority in that sense. That's the first layer of protection you have.
Mhm. Second one is that we also have hooked in the base model web hooks hooked in. So we have web hooks for you to validate the game load out. We call it runtime config which says let's say has your game mode or something like that or the player loadouts which is I I joined the game. I want to add a player that's going to spawn my entity or something like that. You can validate the load out with your choices. I want to play with this character. You can validate that with your back end. So we have web hooks for that. This is a second level of protection. And this web hook can these web hooks can also let you upload the replays to your back end, the full game replays. And that means um you can also store the replays from the clients for convenience for the players to watch or whatever reason you want. But you having the replay is interesting because if you have a discrepancy in the game results, you can use your backup replays to retrofit to to post fact

validate matches where you you had non-matching results. So 90% of your matches, 95 in in our experience, 99.5% of the matches are not going to have anybody trying to do anything and they will have matching results anyway. Yeah, unless you have a bug in your implementation that is nondeterministic. But we have tools to help you with that with check summing and frame dumps and all that. But but those that don't match first level is like you have a replay. I mean second level you you already validating loadouts. Second extra level is like okay replay validation. Third thing you can move towards is okay I can run a concurrent spectator that goes into live games let's say important games like like uh if I if you're doing a tournament a ranked mode you don't have to go into every game but you can statistically validate some matches with a live client that does not even run

Unity. It's just spectating the game and running the same simulation and you trust this guy, this machine because it's running on an Azure instance, whatever.
You trust this guy to push the game result. You can also do statistic validation with replays and that also helps. But then the ultimate level is yes, you can if you grow big enough, you can run dedicated machines with this with on our clouds and you can load your simulation layer which is not Unity. You do not need a Unity license for that running on the server. It's our stuff only. And we can run hundreds of these in one bare metal machine. So costwise, it's still super effective. Anywhere. In that case, yes, you are running the full simulation on the server. The game result can come from our server directly to your back end.
So the idea here is that you don't need you you do not need to go from zero to 100 in one shot.
You develop your game the right way. Play test. Does it resonate with players? Yes. Okay. You go to the next

step. You release the game. Does the game sell? Are people like it? Are the reviews good? Yes. You continue. If not, you didn't have to spend stuff up front with servers or anything. You're just testing it out. You use the free 100 CCU layer that you can even run commercially just to test things out. You don't have to compromise anything. You become successful then. Okay. Am I having a problem? No, fine. Oh. Oh, yeah. There's some problem. Let's start with replays. Let's start validating stuff. Let's find out who is cheating. Because here's the thing. You validate based on replays when things don't match or when you find strange results on your player back end and you caught the guys who were doing nasty things. You ban, you apply a penalty. They're going to stop doing because they know you. You find out unless they're just annoying and then you go for the next level. you get remember the annoying parts would not

affect the other players. They're just
so I like this fact that you have this um more gradual approach to adding adding validation layers. You don't have to upfront uh compromise. This saves you time. This saves you development effort which is time and
yeah you're you're very much right on that that is extremely important for indie game developers. We don't know up front if our games are going to be successful or not. Yeah.
Yeah.
So that that's why I I I like uh determinism as an approach so much because it allows this kind of if you have a tool to do that. We're still kind of alone in this. There there is people doing this in house. I believe Brawhalla is a bit similar to that, but I don't I don't I've never had a chance to meet the team. It's an awesome game. Um I believe they have something like that internally, I would say. But um yeah, when you have this kind of thing and you know how to use it is really really powerful. I I I'm not sure if you saw

the podcast I did with Johan Grval Grunval. He's the developer of Bppel Battle which is like a an online multiplayer party game and he actually wrote his own deterministic simulation and I I was quite impressed by that. But I think um one thing that differentiates uh your product even further is um all of the client side prediction and roll back stuff right which is I think that that part is very tricky to do if you want to do it yourself um or what's your estimate on that I mean it's in one way it is simple but again it is it's built layer by layer right? Mhm.
I I really admire these guys that built a specialized solution because if you see how much there is in the stuff we build you how can you do that alone? The

point is that you cannot but we're building for multiple different genres for multiple of games and and for people who are coming to it without knowing how to do this. Mhm.
If you know how and you do specialize, I still admire because this is still a lot of work. But you start with the principle that Yeah. So I'll give you an example. So the timing stuff I talked about and the fact that we always use a server to be the distributor of input and timing. Why do we need that? Because we our product must do more than two players. If we're doing only two players, you can actually synchronize the clocks with just the two of them. But you still have something called rifting that GGO style stuff has. That's what we don't suffer from because this problem only exists because you don't have a synchronized clock.
But if you're if you try to do three or more players, you need a central source of the inputs streaming and and everything else. Otherwise, it's impossible to solve more or less. Yeah, you will fail in some edge cases. With a survey, you'll never fail in edge cases.

Um, we do up to 128. But if you do if you're doing something more simple, you can go away uh you can go around some some of the complexities. Um then it all depends on how many things you want to get out of the box from an SDK or from an internal tool. So we have two physics engine, a 2D and a 3D physics engine fully fledged with a continuous collision detection shape casts uh stable solver. Uh it's it's a lot of code just to get that running. And you have the 2.5D and the 3D ones. They are independent. We have fully deterministic nav mesh. We have flow fields. We have a fullyfledged AI SDK for you to build behavior trees, hierarchical state machines visually with a visual editor, with visual debugging with a so it's layer on top of layer on top of layer because you're building for multiple different developers doing different things. If you're doing something specialized, it's a lot less code. So,

but the predictable back which was your question, it is built in the basic core. So, the basic core contains the concept of a game state that we are constantly rolling copy. We what we do is we get confirmed inputs. We simulate forward with confirmed inputs. Then we copy that into a clone version of the game state and and that one is predicted forward. Then no input came. We just continue predicting a bit more. Oh, new input. Reim simulate these guys. Copy from again. Reimulate the prediction. There are some variations we can do uh with uh intercalating between moving the verified, moving only the predicted and to to organize CPU consumption. And there are some some some options that you can advanced options you can use but that's the basic principle to it that is console be on. So the fundamental ability to be able to copy the full game

state very very fast is one of the key aspects of it. If you cannot do that fast enough you cannot do predict roll back. Mhm.
We have to be able to copy from to do me copy of the whole game state physics whatever it is. So we designed the whole API the whole ECS the whole game state on top of this approach. But in in quantum it it's basically I think it's called ring memory in the API a couple of times right where you have a bunch of buffer
ring buffer and you have a set amount of frames that you can store. We don't do that anymore. We only have that quantum one. Yeah, we don't keep a ring buff of all the frames anymore because that's more expensive both in memory and in in me copy. And I'll explain you why where that comes from where people think that's a good idea, but it's not.
The thing with the ring buffer is that yeah, you can always not roll back. You don't need to roll back if the input is

confirmed. So if you have the full ring buffer of the predicted inputs and you then get two new frames of inputs confirmed from the server. If the predicted inputs you used match what the server confirmed and our the basic principle is that theistic tries to copy the input from the previous frame for all players and for the local player we're actually using the P input.
So if it matches I don't need to roll back. I just move a cursor that the verified frame is now here and I continue. Mhm.
That's a bad idea. It sounds like a good idea, but it is not. Three reasons. Number one, in practice, this rarely happens because in practice, mispred small mispredictions are actually called by the input varies. Player, especially as you move towards 10, 20, 30 players, there's it's guaranteed that somebody's changing a bit of the input during one frame. So you will always be rolling back anyway. And why is that important?

Because you have to prepare for the worst case. If you think about your delta snapshot, sorry, your delta time thing. What we're fighting as game developers is not frame rate. I hate frame rate as a as a measure of performance. I mean, if you're doing free frame rate, fine. But you say, "Oh, I'm instead of running at 60, I was running at 48." No, you were not running at 48. you were running at 60, but you had 12 frame skips, which is your scripts didn't finish in time for you to be able to push one new render frame to the GPU, and then you got a frame skip and then you had that 12 times over the course of a second. That's what happened. So that's yeah, 45 is worse than 60. But for me, 59 is equally as bad as 48. You want locked 60. So because you don't want to have a single frame skip, what does this have to do with what we were talking about? You do not want to have let's say one update where you don't have to roll back you are just that is cheap in terms of

CPU and then one time you come and you have to roll back and you have a very expensive roll back and prediction that creates fluctuations in how long your simulation takes. What you want to have is you want to have a very predictable pun intended here,
double pun intended because of our product name, but you want to have a very reliable or stable time for the simulation to take. So what you want is to actually constantly rolling back roll back and make sure that your simulation never exceeds the ceiling you want it to be at. So don't get fooled by not rolling back. That's not really relevant. You you you want to what you want to do is you want to make sure your prediction is lighter weight. That's why we have prediction coloring features in built in because you want to make sure that your prediction is as light as possible and the verified frames are the expensive ones where you run more complex stuff in these running at a steady pace constantly. Um that's much better than having a fluctuation in simulation time that just causes frame

skips when you have roll backs because it's very easy to develop thinking about oh yeah in my tests here roll backs are not happening. So the and then you start getting spoiled. You add more complexity to your game and it never hits the ceiling. And then when you test with players, bang, all of a sudden your frame rate is at 45.
You don't want that. You want to know exactly how it would behave.
That's one reason. That's the one I like to talk about the most. At the same time, a ring buffer takes a lot more memory because you need to take into account that you may be um predicting half a second in in ping spikes and and in in highly loss conditions. So at 60 Hz that's 30 copies of the frame. In reality we only keep four copies now constant constant four copies. So the the number the amount of memory we use is fixed. Uh second thing is when you do a ring buffer if you have to do authentic prediction. So you roll back simulate forward the verified frame then you have

to do copy from and which is a me copy and then for every new frame you have to copy from copy from copy from copy from. So the amount of times you call me copy or call copy from becomes proportional to your ping and you don't want that because that's instability again. So because the grow the bigger your game state becomes the more expensive the the the frame copy becomes. As efficient as it is at some point you get it to 2 megabytes or something like that or maybe more. So what you want to have is you want to have a constant time for that. So that's why we only have four frame copies. We have a verified copy. We have two predicted copies for for interpolation and we have one extra copy for error correction for for view misprediction error correction to smooth out error errors prediction errors and you can actually disable these copies and do them yourself. So we only do one copy from verify to prediction. So we have a constant amount of copies every time there's a roll back. It never goes it doesn't matter if you're predicting 30

ticks. Three copies. Three copies. recall. So it it helps you control the time. We're we're this is the kind of stuff we're also uh trying to improve for the next versions of the SDK. We're constantly getting to the edge. We're introducing a new ECS model now. We're trying to work on a copy on the right heap. So even these me copies are going to be cheaper. Even these three uh we're trying not to copy the full game state on on the two extra copies. We'll copy only the full game state for the predicted frame. But for the the the ones that are used only for for um view interpolation, we're going to copy only the transform buffers, not the full frame because we don't need to. So we're we're constantly optimizing stuff, but no ring buffer.
Okay, so the ring buffer has already been optimized away. Essentially,
years ago, we abandoned the ring buffer idea. It was it it was the way Quantum

One used a ring buffer. A ring I think it's the way GPO works that works on tiny game states. If you're doing a fighting game or it may work,
but you don't gain anything from it.
Okay. This other model is better in every single aspect.
Cool. So you can now develop the game um and simulate a couple of rollbacks maybe so that you can get realistic performance in editor and don't have any
you can always you can always test online you can induce anything you want you can introduce laws you can introduce and test and
we encourage people to do it right so to test all the time we try to build tooling around that there are people who really get it and and in our public discord like indie guys just doing some free games and some stuff. So that that really get it and and they would advocate and they would help a lot. So there is an interesting community being built around it. I I really like to

go there and and chat a bit sometimes. Is there any reason to use uh fusion over quantum except for uh hidden information that you might not want the client to have access to? They are I mean quantum has you said this yourself has some of somehow a steeper learning curve.
So it is it's a little bit hard that you get at the beginning although you we tend to see that once you get it you just think on game play and it you flow you go much faster in game development in general but fusion is much easier to start with it it you use a lot more unity stuff. So this let's see a more practical stuff. You you have a you have an asset from the asset store that does AI or something like that. Forget it. You're not going to be you're not going to use that in quantum. It's not deterministic. It's not bu quantum is the game engine. Unity becomes a passive presentation layer. So yeah, I love

quantum. It is it's an amazing tool. I helped build it. But if you're coming from a Unity perspective and you have a lot of assets you like, we try. That's why we have to build these things like the our AI SDK, our our flow field implementation. Some of them end up being better than anything you find somewhere. Some are not not necessarily. So for example, I'm responsible for building vehicle physics stuff. It's something I love to do. I have a ton of experience doing this, but I love some of the Unity asset store stuff. So I specifically like Edd's vehicle physics. I know him personally. Uh we worked with him to help us build our own. So he helped us understand the the most important aspects of of that and I learned a lot from him.
But but his stuff I mean Ed is Vehics Pro is a great tool. So if you're using that on Unity, that's ultra easy to integrate into Fusion, especially Fusion uh shared mode with the new forecast

prediction model we call it for physics that doesn't do rollbacks for fixing predictions. Um it's super easy to integrate that into Fusion whereas in quantum you have to not use it. You would have to use my vehicle physics. I'm proud of it. But it is a different one. You have to learn it. You You're not going to be able to use your favorite tool. You can use anything visual, any shader, any models, animation tooling. I mean, you can use any maner, you can use uh rewired for input mapping. You can use any of that. But code, gameplay code assets, forget it. You're not going to be able to use them in quantum. Um whereas in Fusion, you can. So especially in shared mode, this is not this is an interesting thing that for me it is it is still more important to use the tool that is more suited for the game play you're doing because this at the same time this can be an advantage to some teams to reuse

some stuff they already have but it can also be a trap. So that's why I like to think each case is a case and you should always carefully consider do not be afraid to learn something else. Everybody who really learned quantum they don't look back they really like and most people at the end gets that yeah there are use cases for this and for that you you should always use the right tool for the right job it's also easier this is another controversial one especially fusion shared mode the client authoritative version is the closest thing you would get from I have my game I want to add multiplayer to it it's not a click this button to add multiplayer here, but it allows you to try to do it one step at a time. It's more friendly to that. Again, it's controversial because I'm not recommending that this is the right way to do your multiplayer game, but sometimes it is the only possible way. It's it's an existing code base that you already released.

Although we have cases of people who rewrote the whole thing in quantum to get the advantages of it with a released game. I know at least two or three cases that did that successfully.
Mhm.
But yeah,
in terms of performance, are there differences? Cuz I think in in quantum you have to be a bit careful with performance so that you can essentially simulate multiple frames in one frame so that you can do roll back, right? You work always rolling back. Although that's the less I mean you you can when you get to that you can control. So think about Unity's fixed update, right? So fixed update can happen multiple times. It it is it depends on how your rendering time is. So if you're if you're
So think about Unity's fixed update that runs at 50 Hz, 50 frames per second by default. You can change that, but default is 50. If you're rendering at 30 in a mobile phone, whatever, WebGL or something, your fixed updates on Unity, including physics, sometimes you're

going to have two calls, sometimes you're going to have one, sometimes you're going to have zero. Most probably never zero, but can happen. You can have aliasing and not have a single call of that. Oh, no, not at 30 Hz, but at 60. Yes, quantum's verified frame is basically the same. So, because they only move forward. For example, you can put 50 Hz in Quantum. It's going to work as smooth as as anything because the the view is interpolated. So, say you use 50, same as Unity.
Quantum's verified frames in an online game are only going to move forward and they're going to behave very similar to Unity's fixed update. And you can even clamp. We even recommend you to clamp that to three. There is a it's not a setting, but it's something you can pro procedurally set into the session object. You say, I want to clamp to three. So you never exceed three verified frames in one Unity update.
And two is a little bit dangerous to death spiral if you go behind a bit, but three is safe. So you you can make sure it never goes above three. But then you

have the prediction. But the the key for the prediction is to make sure your prediction frames are ultra cheap. How do you do that? You have multiple techniques you can use. The basic stuff is just set a prediction area to what you see. If your game has is a little bit bigger and there is stuff you're not seeing, you set the prediction area or you use a custom frustm culling based prediction which is on our dogs. how to do you you can make sure that you only simulate the entities that are in view for prediction verified you always simulate everything but that behaves like unity fix update so it is the the the problem becomes can I run the complexity of this game in a single machine
well if you go server you have to run it on the server but the point is that my target devices like Steam Deck or something like that are they capable of running this game forward at this step with all the complex ity I put into the game. If the answer is yes, then there is a way to make sure that the online variation runs as performant. It's just

so you design carefully. There is an envelope for it. It's not
you do not if it's something that does not fit then you should use something else. But the point is that if it fits, if you can run it forward, it can run online. One interesting example for stuff not to run in prediction. So when you're running AI and bots, which is complexity of multiplayer game development, single player, you design um you design boss encounters um very specifically and all that and and you can make AI dumb because it's just AI to for you to beat them up and easily and have fun doing it. Today games are as easier and easier with some exceptions like Metroid venas and this kind of stuff but also like but when you do multiplayer one added complexity for you as a programmer is that you start you have to write bots because you say ah no no bots are not fun to play it's fun to play with other players. Yeah. Yeah. when you get to a certain level, when

you get to the competitive level where it's only fun to play with other players, but you always want the first time players play to be with bots so to make sure they beat the the bots. Maybe you want to pair a couple of players in the same if it's a team based you put real players in the team against bots. Then on the second match, you put these second round players to play together. You match levels of them, but you always have to have bots. It's very rare that you're going to have everybody feeling games to to full to fill a 4v4 game in matchmaking is incredibly complex. You have to have so many players playing at the same time to always feel 10minute matches with eight players each. People don't make the math. So if your game lasts for 10 minutes and you need eight players in each game, the other variable is how long do you want people to wait in matchmaking. So if you put 30 seconds, which is half a minute, which is a very long time, but let's say 30

seconds, it means that within 10 minutes, you have to fill in eight player games 20 times. Every 10 minutes. So you need to have 20 * 8 every minute coming into the game. How many players is that in total is
160
in one region just to have an eight player game. So if you want to have a 100 player game that becomes so you all and if you
and that's only if all of the players are at the same skill level which is also
that's for each bucket. So if you think about oh yeah I want to have 10 second matchmaking time at the beginning so people don't get frustrated. Multiply that by three then.
Yep.
So you have you need to have 500 players at the same time. Do you have any idea how many players you need to have to have 500 players at the same time? You do because you have a uh you you have a release the Steam game so you know your CCU numbers versus the number of copies you sold. So you have an idea that it's on launch this is different because it you have a peak but when it settles and for multiplayer this is super important.

you get a 2020 relation which is what we like to call it's like for every online player you need to have 20 times that in daily active and 20 times that in monthly active. So to get a,000 concurrent players, you need 20,000 daily, which is half a million copies sold. That gives you
roughly thousand concurrent.
But yeah. Okay. So bots are important. Multiplayer more complex complexity on your development because you have to think now about bots playing as a player, not bots as enemies only. So okay, it's it's not only game play anymore. I have to write a bot code. And bots are some of the more expensive code you also write. AI in general. So that's one of the stuff that we always help. Sorry, I always take very long arcs to come back to the original thing. So bots are expensive to run. So it's I I always like to recommend people to if you're writing when you're writing bots, make sure your AI has at least two different

tiers. So you have a decision making tier that has the expensive stuff and you are only going to run that on verified frames. So you'd never run that on prediction because prediction can fluctuate and you never even for visible sub you don't want to run the planning stuff every time. But the operational tier of the agent, which is the thing that steers the agent, that does the movement, that uses timers to shoot or to follow a character, that one you always run on prediction if the entities is within the visible range, if it has a target and so. So you always plan to implement your AI into these different tiers because you want the visible aspect of it to to run in prediction but you do not want the expensive planning part to run every time. You want to run it verified only because again if you can run the game locally forward only you can run it online with the same performance

with the same amount of entities.
Is there built-in functionality in quantum to run specific code only on verified frames? Yeah, it's an if away if f is verified. If frame is predicted, you have that in the in the frame class. Frame is the game state that you receive from the update methods in systems. You have frame is verified is predicted.
Okay. And I can I'm allowed to do checks like if if this is a predicted frame, then I just don't run the section of my code, for example.
Yes. Yes. Just early exit on the update. Okay. And this will not cause any desyncs or whatever we
never because only verified frames are matter for the syncs. Why? Because we always roll back and reimulate. There is no roll back skipping.
Okay. Buffer is a bad idea.
Okay.
You wouldn't have see that was one of those really valuable insights because I

don't think I would have figured this out on my own or or in the
I'm I'm telling some secrets. Yeah. No, this is this is really good to know. Just don't run the expensive stuff in the prediction frames.
Run on verified. Yeah.
Cool. It is a different I mean you have also the prediction culling layer that if you have a game that spreads around an area, you you you can afford to run stuff in prediction as well within a certain area. So
it really depends. I mean I I rarely think about this in advance with the exception of the AI. I like to think about the AI that way, but anything else I don't think about it in advance. I I start adding cuz I know prediction calling is going to kick in as soon as I enable and and so on. So, and again, don't underestimate the performance characteristics. It it is it is damn quick. It is where my my rule of thumb is like I always like to run my simulation within less than two milliseconds ideally less than one millisecond for the whole simulation

including predict and roll back
for at least for prototypes I write but in uh I like to get my verified frames even in production to max 02 05 milliseconds worst case because then my prediction frames are going to be 01 or or less. Um I'm I'm a performance freak. So
I like that
there is a there is um video I have I think I linked it to you before. There is a a few chapters on optimization. I explain every all these things I talked about here are explained in steps. I mean uh how to enable prediction calling, how to run stuff on verified frames and examples of when that happens. You know another cool I I forgot to mention this one. This is also cool. You can do what I call time. It's time slicing. So um so say you have 100 entities that are enemies that you want to run complex AI decision. You already

moved this to run on verified frames only. But still your verified frames are running above 1 milliseconds or something. You want to optimize it a bit better. Then you can think okay the steering of the AI is already fast enough. But do I really need to run the planning agents every single frame for every single entity? Maybe not. Maybe you want to do what if I run every fourth frame for each. How do you split the entity? So 25 25. There is an easy way to do with uh the entity index mod operation with four compared to frame number mod operation four or update rate you want to. And then you compare you only uh update the entities that match. So you're going to have 1/4 of them in one frame, one/4 on X and and so on and so on. So you you time slice the stuff not only to run on verified only but every fourth verify every fourth verified frame
equally splitting the entities um according to in sequence. So you get

again a constant time that is four times smaller and instead of spikes because you can think oh I I'm going to run this system every fourth frame. That's a bad idea because you're going to have every fourth frame I spike. What you do is you you time slice the entities having 1/4 of them every frame and inverified only. So you combine these techniques together. That's the bread and butter for having a very smooth gameplay. Some of these techniques are valid for single player Unity in general and time slicing for complex AI on a local RTS game. Do it. Mhm.
So, what if I wanted to make uh like a strategy game like for example something like Thronefall uh or I don't know Starcraft or Age of Empires or whatever.
Is it crazy about strategy games? It's because well pretty much every single one of them uses determinism.
So you you probably know that. But the

thing is that what I I like about that is the challenge because RTS's for me are the combination of everything is cutting edge. So if you think about okay entity count is at the limit of what you can do. You have tactic formation which is so path find is incredibly difficult because you have group formations and all that and then you have oh the economy or the the the tech trees and then you have to have AI on every single one of these entities and you have to also have a layer of an AI for you to play against as a tactical commander of the other army if you want to you don't want always skirmishes with other players. So everything is cutting edge and you cannot use a state transfer because it is so yeah you can if you optimize like crazy but honestly you would be basically spending time on that and you still have to do the other type of optimizations for everything else anyway. It's so to me running state

replication in RTS's is is an exercise but it is not really a good solution. Yeah, you don't get map hacks if you do interest management, but it's so much work to me that
nobody does it in at least the games I know about with one exception which was a special game planetary anhilation that uses this chrono chem. It's not really state in that they send this curves that is bit like input. It's deterministic. So, but they send us scurves. Um, but
so yeah, you use quantum for that. We we used to have an RTS template um or sample that in quantum 2. We didn't port it to the latest quantum 3, but we did port and improved the flow fields implementation which is the basis for group navigation. So, we you want to have 2,00 units, it runs um with group

formations reusing patches. It's it's a it's a verb not verbatim. It's it's a by the book implementation from the AI. It's not AI gems. It's uh it's it's a it's the definitive uh article on flow fields that there is in game AI. Uh we we implemented all the techniques in there. It was um a friend uh a colleague Thomas uh Thomas uh who implemented this and we keep kept it maintained and it is updated to latest version that you probably going to use that for because if you're doing multiple units but there are many many tricks in RTS so um you can use so some so if you think about Starcraft my favorite game zurglings they're not really individual units visually they are but entity wise you can combine four together you can you do some
you can do some tricks to get more eye candy without actually going crazy on on that.
But there are lots of interesting features we have. So you mentioned that

we have this thing called um prediction culling that allows you to you don't need to simulate stuff that's not in your camera for prediction. But in an RTS maybe you don't even need to predict everything. But it's a good Have you seen this latest RTS by these ex Blizzard guys who wanted to build some like a spiritual spiritual successor to Starcraft? It's these guys that build
Are you talking about uh Frost Giant uh
Yeah, of course. Of course.
Yeah, Stormgate. So, what they have is a mini uh specialized quantum that they built themselves for 32 players. Um
Yep.
Yeah. It's they're doing predict roll back at 60 Hz essentially with uh I think last time I talked to them they didn't have yet prediction culling but they they they were talking about it they have they had to implement it at some point I think
but it's that idea right so besides prediction coloring because in an RTS maybe you don't need to run prediction you can run basically kind of lock step verified only but it's nice to run

prediction because you click you see it moving right so and it doesn't hurt it's so cheap to run prediction on something very close to you. But there is something more interesting which is how do you prevent all of the game views the the game objects that are around the map. I mean the game objects I mean here you have entity moving on the other side of the map that you don't have to know about.
Mhm.
And maybe quantum is spawning the entity view for that. And because it's in your fog of war you are going to have to write code to disable that or something like that. We we recently introduced and it was I I implemented that which is a way for you to call views by basically by you can overhaul our basic behavior for which entity views need to be instantiated locally. So you don't even have to spawn at all a view for an entity that's out of your scope. This sounds obvious, right? But we you could

do in previous versions of quantum but you would have to reimplement
what we call the entity view updator or something like that. Now it is built in
that you can um you just implement an interface that and then it's very easy for you to decide what to see and what not to see
and then the view is not spawned at all and then it's something less for you to worry about in terms of scalability of rendering and all that. So you use the fog of war data in your simulation to decide if the local player should have access to it or not to the view. So you're going to use all these little tricks together to make sure you can run an RTS game. It's it's still one of the interesting genres that is challenging to do for sure. I also think if we were to make an RTS, it wouldn't be a fullyfledged one. It would be a somewhat simplified one. Maybe a base defense game that plays a bit more like a tower defense or or whatever. Yeah.
Yeah. The game um We have a lot of tower defense games in mobile being done in quantum, right? So

yeah,
but I works I've seen your games and and um yeah, that's definitely something I I would personally do with the quantum or something similar.
Yeah, makes sense. I see.
So it is still possible to run quantum in lock step if you don't want roll back
in multiple ways. Yes, we still have a setting for you to do that. I would personally never use it. I don't see really a reason not to run a prediction especially
because prediction culling allows you to run it so quickly.
So if you think about the systems there is a way for you to not even schedule the system to be on the task graph on predicted frames. We have some uh base classes for systems that are convenient to use. But if you use the very base system, it has a schedule method that you you you decide if you want to schedule it or not as a to go into the the task graph or not. And you can decide not to schedule it on predicted frame. So the there is no

overhead at all. It doesn't even get scheduled. So
you only run the stuff that's very local that's like okay the click to start the movement of the things. You only run those things. You don't need to run for example progress for you for units or stuff like that on on prediction but you run just the movement stuff with prediction culling enabled. You run only that. Don't run the path finding with um flow fields. You only run the initial because it's going to start moving on the local flow field initially because you probably already have you run the there is the main a star for the for the patches that decides the act the exits between the patches and then there is the flow field update itself. So if you run just you don't need to update the patches because the patches probably already exist but you just run the main a star you already know the direction to go to. So that already shows when you click and it's incredibly cheap to run.

So I would not go in lock step tab. Is it possible? Yes. But there are better tricks for that. So Moon Studios for example, one trick they use is that the worst offender case is when you have a rob input confirmed and you get more than one input tick confirmed. So let's say you received five different input ticks from the server in one shot and you still have to predict. So simulating five verified frames plus that it's too expensive. So but let's say you were clamping to three. So okay simulate three plus eight predicted that may be expensive but there's a trick you can again use a times licing strategy which is quantum allows you to when you when you have multiple frames verified frames to simulate you can decide to I'm going to simulate my verified ones but I'm not going to roll back the prediction and reimulate from the beginning I'm going to just do one extra prediction and

accept that it is kind of like out of sync with the verification Oh yeah. Okay. Mhm.
And then on the next unit update that I may not have any verified or maybe one or two verified, I do the full roll back. So you keep intercolating between that. The def if you just download quantum and start using it, it would not behave like this. But it allows you with a call back to decide exactly how many verified frames to run. how if you want to copy for the prediction to do the full roll back or just update the prediction and accept a little bit of out of syncness in in the visuals. So you and that's what a complex game does. So I would use all this before deciding I'm going to run lock step because
okay
it's so incredible to have initial feedback.
That's true. I guess there's one case I can up can come up with where I'm like I I think I really not I really wouldn't need any roll back for this and that is an auto battler where basically both

players place their units and then they just go up against each other that I would not run lock step still but what I would do is I would run everything you were talking about in verified frames only so they cost nothing on prediction. Prediction is just a empty thing.
Okay. And I would run the views in snapshot interpolation mode because here's one thing. If you run stuff in verified frames only, they're going to be jittery
just like we discussed at the very beginning of our channel.
But quantum also allows you to have snapshot interpolated views which are
interpolating between previous verified frames. Do I keep a ring buffer for that? No, not frame ring buffer. But the transform I keep a small ring buffer of the transform data. So we can smoothly move the view here even though the verified frame is kind of oscillating the speed at which it updates forward.

Um so just turn on snapshot interpolation on the views for these auto battle units and yeah that would be my suggestion. So yes you don't need prediction on that at all. So run everything on verified frames but but snapshot interpolation because maybe you want prediction on some other stuff going on like a card selection or unit placement. You may want to have that with a with prediction so it reacts immediately. I I would cuz I would almost never use lock step mode nowadays. I will always find something where prediction feels better.
Even if 90% of the game is going to run verified only and some snapshot interpolation, I would use that. So there's not really a good reason to turn it off because you can always just disable basically a lot of the systems and then
and it would cost the same as lock step and
and you get some initial feedback that always feels better. So,

and um for an auto battler where there is hidden information for some time maybe, right? Maybe there's like both players need to pick an upgrade at the same time or both players need to place units at the same time. What I was thinking is maybe you can do something like um make both players send their input at the same time. So, the input would be uh where they placed all of their stuff like more more of an event or something. That's an interesting that's an interesting thinking you can do that. So one thing is that you have to be you just need to be careful that so there there is one input type in quantum called command which is reliable. So they can also be predicted but they are reliable. So what you can do is if your game design allows for it and and the player can plan his move and and you can basically even take the the frame data as you see as it is available in your machine to for the player to

think about his movement and you can let's say run the path finding from the view without modifying the game state but just okay where if I put here if I use this command this guy is going to attack that unit and you can predict how much damage you're going to Ju just by running the code that would do the damage in trial mode, you can get that code accessible so the view can run it just for the player to see. Yeah. I mean, let's for let's for example say with a with a more simple example. For example, both both players get to pick an upgrade at the same time, but it's it would be bad if you knew what the other person was picking because then you could counter pick it. So would be important for both players to pick an upgrade at the same time or not know what the other person picked.
So So what will happen is you're going to have kind of like a timer that runs that players are seeing. They have to pick their selections before that timer runs out. And when the timer runs out, the clients are going to automatically send their selections as a quantum

command. You just have to make sure that internally in the simulation there is a little bit of slack after that timer to allow that to because you you have no guarantees that the command is going to arrive within that tick uh that you predicted. It can arrive one two three ticks after depends on the networking conditions and commands are reliable. They are guaranteed to be delivered uh but they may shift in tick if the network does not allow it to be delivered in time. That's different than the regular input that can be totally ignored by the server if it doesn't reach the server in time.
So just give a bit of slack, but you have to send them here. And
if somebody is very very clever, they could actually write a tool that waits for that moment to see if the command from the other player arrived. But because you are receiving the command from the other player half well full RT because the guy has to send

it to the server. The server has to send it to you. Maybe there is not even enough time for your client to wait for that and then send your
and you would have to be an automated tool that decides what to run yours
what you send as yours based on the other guys. So I don't think that is yeah a crazy guy could do it but um you have other ways to think about that based on you you would find the pattern that he always sends delayed and yeah I think that's that's an interesting idea. It's a turnbased game that you basically time the commands just with a bit of slack for protection
and yeah you don't see what the other guy did until you guys do at the same time. I think that probably works for a for a first implementation for sure. And then if if you notice that maybe there's still a cheating problem, then maybe you can uh write some server side code that makes sure these get delivered at the same time. You can Yeah, you can you can do that. You can that's actually a you you're a clever

guy, right? So I I should have thought I'm I'm ashamed because I should have said that myself. Yes, you don't even need to run the simulation on the server to do that. So, you can intercept a quantum command. You can decode the data in a quantum command and know what the data is and you can hold those commands. Do not allow them to actually be confirmed into the simulation for a few frames until you get all of them. But you so so if your turns happen in if you have the simulation running on the server, yeah, you can do everything. But uh but say you don't have but but your turns are in specific frame times. You can say yeah I'm expecting commands be here for this before this frame. And what you do is you make sure you don't dispatch them until that frame arrives. So the guy sent 10 ticks before, 15 ticks before, but you hold them and when that very frame is the time, then you inject them all together into that frame and you don't accept anymore the frames for that the commands for that round

after that moment. So yeah, you can write server code to that if the game becomes successful. Then you it's absolutely guaranteed that the commands are going to arrive on the very very same tick on every machine and there is no way for the other guy to wait for your command to decide for his. No way.
So yeah, you can you move to that safety layer again.
Yeah. Well, that that's really what I what I like about everything I hear about quantum is that the first implementation of everything is like so easy and then it's like if I want to make this foolproof later, I can can still do that.
You can add in little steps, you can add this protection layers. Yeah, exactly. That that's what I also like about it. Well, one of the things
there there's um one sort of silly question I have as well, which is like I think there might even be some cases where I'd be interested in having a deterministic game simulation but without even using your multiplayer. for

example, um, for example, for procedurally generation stuff when I'm like, I always want the same seed to output the the same terrain or something. or maybe even for multiplayer games that use a different networking layer where I'm like, "Cool, there's this new kit on the blog which like Spacetime DB and maybe I want to write an uh online multiplayer game with that, but I need a deterministic simulation to make it work to make my auto battler work with the system or something." And like, do you really believe in that?
I don't know. This is just see what happens. I mean the there are always these new players coming and going.
So u
we we don't try to solve everything right? So we try as I told you before we try to solve
particular envelopes. I don't I don't I don't I don't claim we can solve this or that problem but I tend to recognize

solutions that are interesting and that I have a certain skepticism on the costs. So there are some that that are technically interesting as an exercise but are questionable in terms of okay but how much does that cost and we've had a few of these in the past and they came and they went
um got it and there are some that have questions about the the technical solution feasibility or the scalability in general and there are some that are really interesting and and and welcome as competitors and that We hope sometimes they they can convince people to use there. There are examples in all of these, right? So, I'm curious in this case to see. Yeah, we will see. I mean, the the question went went off track a little bit, but but I guess the question I had was more like is there a chance for just the the game engine part of Quantum to be available at some point

without the networking attached to it? Cuz I I even see like
the answer is no. Mhm.
Long answer is that you have to talk to us for specific use cases, right? So, we've we've been asked this before. So, a couple of notes on this. Um, you can run quantum when you run offline quantum. I mean, you're building your multiplayer game with quantum with a valid plan. You have an agreement with us and all that, but you also have a single player mode on the game. You have a split screen multiplayer. uh that is that is offline split screen because we also allow split screen two players on this machine against one player on that and three players on that that is built in the solution but let's say offline when you're running offline you're not connected to us you you as long as the game is part of a multiplayer game that uses quantum that's already expected that's already allowed that's within the terms that is just fine and you're not paying us when you're running

anything when you're running the local version you are paying for the license, you're paying for everything with your online plan for the online part of the game. So, that's already expected and that's part of the legal agreement. We're giving this SDK that I have a 20 developer team of top-notch guys building and our business model is the online plans.
Mhm.
To run it completely offline without an online plan, that's not within the terms. Even if you're building the game in quantum itself and and so if you want to do something like that, you have to talk to us to see what is the exact case to see if we open an exception for that. We we are always open to talk. I mean we we we
it might be possible, it might not be possible. using the tooling really outside of quantum is something that we've never done that is not in the plans and but if you have an interesting case then you can convince people about

it we are always open to talk right but each case go is I guess the point I was trying to make was also a little bit like I'm kind of sad that there's no game engine that does what you're doing with quantum out of the box because I think uh determinism is highly undervalued I really really like determinism um for for all kinds of reasons, not just multiplayer. I I think it's awesome that you can do replays easily. I think it's awesome that you can do basically deterministic procedural generation. I also really like the turns out I also really like the ECS architecture that you use in quantum.
Yeah, you do have sort of a bit tricky to get into, but once you're in it,
we're going to like it even more with the next version that we introduce table components that are even faster. So we don't we have not only sparse set now we are also table. So we you can decide if you want sparse set or table per component and when you filter table only it's much faster to go through the entities and all of our internal

components are now table like collables.
It's similar to archetypes. So an archetype is a is a composition of an entity and it's stored on the same buffer.
So instead of well it's a it's CCS architecture right? So there is a sparse set which is
so sparse set is ultra flexible and the biggest point of it is that
if you add remove components it's a very cheap 01 operation for add remove and get uh get has they are super cheap as well on table components has and get they have the same performance as a sparse set but the differences are in spark are set. Add remove is super cheap. In table components, add and remove are more expensive because you have to move the whole entity with all of its components to a different buffer if you add or

remove one component.
Mhm.
Um in sparse set that is a one ends in. The filter which is traversing a collection of entities that match a certain collection of components is much cheaper in tables than it is in sparse set. So it is always a trade-off. Sparse set is normally more forgiving to developers and that's what we chose quantum to be designed with. Whereas tables can get more performance
if you design things for it carefully and you know exactly what you're doing. you can get much better cache coherence and and and cache hits um if you're very careful with your system design as well when you benchmark everything but you can achieve interesting much more interesting performance level. So introducing that with quantum 3.1 we decided okay it's time to do a hybrid ECS. Yeah, we we do cool stuff. I mean, it's cool to work here. It's we have super cool guys. It's super fun to do.

Yeah, I I in a certain way I I also I agree with you that it would be nice to have a lot more stuff that is open source uh that does good things, but you can say the same about end game engine in general. Like so I I love Unity and I I I've used it since 2006 there, but it's important to have competition. Now you have good and you may agree with some stuff. You don't agree with everything but it is not yet there
in terms of what you can get out of these engines. And even if you think about stuff like Unreal, I mean Unreal is not a unanimous thing. I mean it's visually impressive but you have tradeoffs and yeah you have source available but it is it is what we live in the tech world and there is some stuff some deterministic stuff around that you can rely on that is open source but yeah but nothing as complete as this but I think it's yeah we we don't have an open source game engine that is as complete with all

the platforms with all the stuff that's supported like Unity yet with all consoles with everything with the facility Unity has. So
yeah. Yeah. I mean I wasn't even specifically talking about open source. I was just like it's sad that there's no actual game engine with this specific profile of features
in the engine itself like the engine is built on top of it. Well
well if someday somebody comes along and is like I I will build an engine on top of uh quantum make a visual interface for it then It's kind of like a pipe dream. So if if a bigger player would come and say, "Hey, let's build a rendering engine on top of this. Let's pump this gameplay engine to even a next another level of
uh clean up and and organization and everything. Let's push this to the next level, including the runtime and and a rendering engine. That would be an interesting bet. But would you want that

as a game developer? Would you want to rely on your you would be building because this would restrict the types of games you build with this engine to these types of games, right? It would be a it would be a specialized engine. But also one thing you have to remember is that a lot of games also aren't uh multiplayer. A lot of games are just single player
and you could get replays, you could get all the benefits of of determinism with that. And also also the dream of basically having a check mark in the engine that's just like make a game.
Yeah. But that's if you build with quantum you you get that uh that that check. Yeah, that's what I'm saying.
Yeah, it's it's Yeah, it's something we think about. I mean, we kind of like dream about sometimes, but it's not a realistic dream, right? So, it is
I I think I think that's that's an answer I I'm happy with. like if if it's on your radar at least the idea
it's an idea we're not crazy against I

mean we're and we're trying to integrate with other engines we did we did do an experimental test with good
Mhm. back on the Unity, how can I call that? When they had the previous CEO bonkers business model switch thing that scared everybody. Um, when that happened, we said basically to answer the community, people were asking a lot about it because uh quantum is C# it was agnostic to Unity as gameplay engine. So although good car integration is not nearly as good as their JDcript integration but okay let's see what we do here and let's share uh the results and it was an interesting we took one existing sample we had that was already implemented in quantum totally isolated let's run that
in the what we call the console runner which is just a net version and we have the photon realtime transport layer that runs on C without Unity let's load it

with Let's run that from Godo. And let's port the view interpolation code to running some good nodes and the entity view updator script to run as a good node. And let's see if we can export the gameplay scene. I mean all the assets, the the the static colliders for the map that was exported from the Unity sample because it was already ready as JSON. So we would load it on the C# simulation from there. But the visual we could actually export the Unity scene in a format that was friendly to good. We just had to fix a few shaders and something. And then we okay took two days and we one guy and we got that running and we did a very very interesting test which was a cross engine deterministic play test. So we were running the the Unity build against the GDO build totally in sync without checkum errors. It was I we still have

the video for that, but it was not practical to release it as um as um player. We waited a lot to see how things would settle down, but that's still an open question. I mean, uh I I was asked yesterday
uh what about a monoame layer for quantum? I said, I mean, guys, if you if you guys build a team that would be interested in doing that, we could help support this. we we could get you guys in in our premium program for free because you guys are doing something for a different engine. We're interested in that. So, we would help you guys with support on this and we would get the legal aspects checked to make sure this can happen and yeah, we we see this with very good eyes. Of course, it's it's it's customers in the end. So, we would like to have it. So, I I that was my answer to the guys, right? We're actually now cuz we have a new generation of Fusion being built as a native core for Unreal that's been in

tests by some customers already for fusion for now. We also have a quantum unreal implementation which is a totally new C++ implementation of quantum ncs and so on. But that's a different story. But this fusion core is also going to be integrated with different engines. I'm talking here default which already has an integration with uh photon real time where there are couple of other engines I may not be able to talk about because it's internal discussions still but popular stuff not yet good but that is on our road map we want to try to integrate fusion this new core of fusion that is native to integrate it in good and I would say I would actually be willing to integrate it with GD script not with uh with the if the car integration ation of GDAU is production ready and is really good. Yeah, absolutely. We all love C. But so this is all in our road map in that in stuff that we're interested in doing. We're not locked into one engine only.

And um
that's the closest to your question about let's build an engine on top of quantum.
Okay. Yeah. Let's shift gears a bit again. I guess there are a couple more game ideas that were floating around in our head which I wanted to ask you about which like for more openw world type games. Let's say like it's like a co-op game. It's a small party. It's like four players or something and you are dropped into a very large maybe procedurally generated world and you kind of explore together something like that. Would you would you tend to build something like that in quantum or would you tend to build something like that in fusion?
It's a good question. It really depends a lot on on what kind of systems you would need in this type of procedure generated world and how much of it you need to be loaded in memory. How much of it you need because if you need to be a lot of it to be loaded in memory for the gameplay aspects of it, it means quantum

would have to have it loaded in every single client. in if you go fusion then maybe only on the server clients don't need to load everything but then again for players your cost effectiveness you're probably talking about one of the players being the server a bit like Valheim does but our normal answer will be fusion probably playero player hosted client server would be the usual answer to a survival or open world type of game it's the but it the the devil is in the details it depends a lot on what type of procedure generation you're talking about and I'm now for example working on a vauil voxil based collider check for so we don't have you don't have to naively create triangles for the physics engine because that's prohibitive it becomes too big but if you run directly against the vauil and and your collision checks are proportionally the same size as the

vauels it is 01 for the checks And so I'm running it's essentially Minecrafting quantum if you think about it. I'm doing a prototype of that. If it's that kind of a open world procedure generation, maybe quantum is interesting because you can load in memory only the chunks that are um interesting now. But then you can think you should think okay but can I unload one that is not being nobody is in that chunk anymore. Can I unload it from memory but can I reconstruct it as it was? You always have to keep somewhere the data too. But it again it depends on what exactly is the type of procedure generation, what type of terrain, what type of modifications you do. But the normal answer for would be fusion probably player hosted.
I see. So can quantum deal with big open worlds if there are no so if there are no is the question like how many movable

objects are there in the world? For example, like if I can for example reconstruct the position of all the trees from a seed, then I don't need to hold them in memory, right? I can just reconstruct that chunk when I get there again.
Yeah, the that's um so for example, the numerical representation accepts actually quite big range because it doesn't change. This is the one interesting thing about fixed points is that you don't degrade the accuracy as the number grow big. The accuracy is always the same. So the the usable range is plus minus 32,000 which is pretty big.
But the problem is how complex is updating these entities and how many of them you want to have. If you think about entities being loaded into quadrants, chunks if you think about it and there is not a single player in a certain quadrant. You could for example tag all the entities in that quadrant as non enabled. So they would not be

updated at all. They can they can still exist in memory. So, it's a trade-off between how much memory is that and but you can actually make the update cost scale well if they do not even get into the futures especially with the new version with with table components because that skips much quicker.
So, um and so you don't need to think about rebuilding them. You can keep them in memory and not and again if we were planning to get the copy on write stuff. So because you're not modifying these entities, they're not also copied for the predictions at all because the their heap pages are not being modified.
So it may be feasible to just leave them there. It depends how many do you want. Do you want to have 2,000 3,000 of this? Maybe this is not a big problem on a PC game. You can probably run a game like this at 30 Hz simulation. This always helps because the rollbacks are going to always be less predictions and the verified frames are also steady at 30 Hz. So you can always
the thing we're talking about like

running max three or maybe at 30 Hz on a PC, you can actually run max two verified frames per update and that would be enough. If you've never been to a certain quadrant, you do not ever have to load the entities, the first version of the entities. You do not spawn them at all. And then you can think and and and if it's really big, for example, even the ones you touched that were dormant, you were probably talking about at the end of the game, you want to save the game state as you was and somehow reload it for the next play through as you was like you want a persistent kind of thing. We do not solve persistence, right? So net code cannot solve persistence. So especially long-term persistence because you add a new property. So our frame serialization does not survive with a new version the moment you add different data. But you would be custom persisting only the data you need and then when loading again the

game you can actually pre-spawn the stuff you need on one of the clients and then let the other players join the game and they are going to receive that as a snapshot and then they continue from there. Quantum supports that out of the box as well uh for late joining. So you could do that. So you can solve that. So you don't only spawn entities that a player at least one player ever went into that quiz and you don't load it again. And the ones that and when we're reloading the game, one interesting way of doing it is you do not spawn the dormant entities on the next game instance unless somebody goes there again. But you you load that into a database that everybody downloads because they have the save of the game. It's kind of like assets for that. So there are solutions to scale that and still use quantum and have all the advantages of the predict roll back and physics but but it's always challenging that and and the the array modifications the procedure generation incredibly this is the most uh challenging parts because

we built a lot of stuff based on unity tooling converting from the unity editor into quantum and people struggle to build implement the procedure generation themselves within creating triangles in our physics uh APIs and so one. So, but I guess that's kind of stuff you would not have a problem with in your physics API.
Can you create triangles in your physics API?
Yes, you can use what we call dynamic map and you can add remove triangles. You can create one from scratch or you can base it from an existing one and I normally don't recommend it is it is well maybe you would not have a problem with it but some developers have a little bit of a struggle to understand but yeah it is possible.
Okay, this is good to know.
Nav mesh is more of a problem though.
Nav mesh is more complex. We're we're working on that to get uh we have a port of recast that can do deterministic baking and
that helps but we're
we're still working to get a new

generation of the tooling for that to to simplify procedure generation of nav meshes that's it because it has to be deterministic that's the big point right so
we do have deterministic nav mesh but deterministic generation we have partially we can convert the data deterministically to our format but the full generation Uh it's we haven't covered yet everything but we're working on that.
Okay.
And for I guess the the fusion approach to do procedural generation would be to do all of the generation maybe on on a server and just share it with all of the clients.
If you're if you're if you're so so if you're thinking about like a static collider for the terrain big terrain or something like that. If you if you have a deterministic generation for that on Unity in general, um you can actually just sync the seed from the server and do it on every client. That would not be

a problem. But it's all about like do you want it to be modifiable? That is where the challenges start in both cases, right? But generating is generally possible. And then for example, if I do procedural generation in quantum, that's maybe also one of those cases that's a bit more expensive. So I would maybe also try to only do that on confirmed frames and maybe also
to generate the stuff. There are multiple options here. So if you're talking about like the initial version of the of the game for a new world, you're creating a new realm, Minecraft or Dwar Fortress style. So you can run that on one machine. It then it does not even have to be deterministic as long as the late joining data becomes deterministic. The stored data is deterministic. So
um or you can actually run with the same seed deterministically on every machine before they start the quantum simulation. even if they're going to

late join to receive the mutable part of the game state from the guy that's already running but they the the bigger chunk of a static asset that is the world that they can generate before starting quantum. So that doesn't affect simulation performance. You just run it before in these two cases and then
but you can also run the procedure generation from within the simulation itself but then it would going to impact simulation performance and you have to be careful with it with how much we're doing from there. So it it depends on on how how much it is.
Yeah, I guess it depends also on the kind of game, right? Yeah. Maybe you have a fixed size island that you just generate at the beginning or if you have like a an endless train track that you want to keep generating as you go or something.
Yeah, different problems to solve. Yeah, I see. But yeah, but yeah, when you can generate incrementally, you normally can get along with generating within the

simulation because you can do in smaller chunks and so you make sure your performance envelope fits. Where do you think are all of the game engines at with their multiplayer implementations? Cuz I'm always like the first game engine that's going to crack multiplayer will have such a huge advantage. But so far it seems like everybody's like uh source source engine has good multiplayer. Um Unreal has a functional object uh synchronization layer that's very reliable. Um very it's a bit legacy but it's very reliable. uh it's on that mid layer and they have a proper client side prediction for the character and for game ability system and and that's where they are. They don't have an offering in determinism. They don't have an offering in client authority variation. Unreal Unity, they have the object synchronization layer net code for game objects that has both client server and

client authoritative, but the their client authoritative version doesn't have the middleman stuff capabilities we have. Um, and their client server doesn't have like compensation uh timing information. and they've been improving on that, but they they're they're uh they're trying to get from an object synchronization layer to a bare net code. I don't know what's the current state of it. I honestly don't take too much don't pay too much attention to that.
Um and they have their ECS net code which is built on top of good principles. It's some tradeoffs. their ECS is still not really mainstream and I don't know when if that's going to ever be really popular or if they're going to crack what they've been announcing of making the game object world be backed by ECS that that would be interesting to see but in any case they're doing B that is pure client

server only for now and if you see their demos they do 10 player maybe they've never shown a demo with we do 200 fusion but it is built in good principles I know they're some they're great guys so they good developers they
I'm not I'm not saying that they cannot crack it it's they are capable they're excellent developers it's it's that they have their goals they have different targets they have different milestones and they're getting their things but
we're still competitive. We're still doing and um we have a very good relationship with them as well. So, but that's how I see them. Maybe some other guys from our team would have more details than I have. I on on the technical implementation of these ones, but more or less that's how I would tell they are now. Other engines don't have a net code uh solution at

all. So, they're purely single player engines. I mean, it's a it's obviously a a way to make some reoccurring revenue for them. So, I think game engines generally are are keen on getting this to work somehow, right? You can get keep developers in your ecosystem and then
Yeah. But I think they overestimate how much of a market that is.
Uhhuh.
We are successful. We're a profitable company, but but I would say we're small to but we're a very lean 60 people company, right?
Yeah. And we have a we have an interesting portion of the indie market and the mobile market. It's not huge, not crazy, but we do power important games and we have a sustainable profitable model. But compared to what the size of Unity as a company and what they need in terms I think we're not the revenue even if they would get the same size as us in terms of business I don't think that would move much of a

needle for them.
So maybe they overestimate this.
I don't know. Well for for what it's worth I think the reason why not more developers are making multiplayer games is because it's so difficult. So, if at some point somebody figures out how to make it easy, I think we will have a lot more multiplayer games. I think everything will be almost everything will be multiplayer if it's easy enough. We're we're trying to I I believe it is possible to make it easier. I'm not sure I agree with uh what I see out there. I'm not talking about Unity in this case, but I I see other people trying some stuff. and
not sure I agree that especially when you try to make it easier but make it extremely more costly that for me doesn't work in today's world.
Sure.
Uh we're we need to be lean we need to keep wasting less energy less resources in general even for the world itself but also as a business as business models to to stay competitive. So but certainly

that's a goal to make it as easy as possible. Sometimes the easiest is you need to do two steps up but then it becomes flat. That's certainly what we try with quantum.
Mhm.
But it's couple of big steps. But I I like that idea of making it easier. It's uh it's definitely number one goal making it easier without compromising performance and and costs. That's the
right.
Yeah. Cost is cost is obviously also a big one. So maybe we can get into that really quick. Um I think you have a free tier, right? And you you you charge based on concurrent users for all of your solutions.
Yes. For all online running online, it's always based on concurrent. Running local single player version of split screen, it's not connected. So it's doesn't count.

Right. that is free but only if you intend to use the multiplayer part as well otherwise
and and we we have um SDKs are free to download and and use for development you have you can create app ids for development to release a game with you need at least the 100 CU free which is so 100 CU may not sound a lot but we were talking about this before this is kind of like yeah you have 2,000 players playing the day game daily. That That's what you get with 100 CCU. This this is like you have you sold 50,000 copies of the game. 100 CCU is going to get you with that and that's totally free.
Um you just keep renewing it. You would never we don't plan to make it we we've never in u made things we we only made it easier. So we we added this CU3. We didn't have it before and so on. So then

the next layer is still subsidized. So if you get the 100 CU because this is for only one app ID for your account. The next one is that if you have want to have that for multiple ones, you you can pay the 100 CU version to have it in multiple app IDs that is per year 95. That is totally subsidized. That doesn't cover the costs. It's it's really
okay.
The other tiers are especially the higher ones pay for it. But we've never had anybody who gets to the paid tiers asking or saying that this is expensive because when you get to get to need 500 or a,000 CCU, you are making so much revenue to when you got there that you realize that the costs of our stuff is like 1% less and you are paying 30% to Steam and you think their relays for free. Well, you're already paying for it at 30%. Right.
I'm already paying for it, so I might as

well use it.
But, um, if you want to get switch and other, you you don't get it, right? But well, we already talked about this, but the point is that it is not it is cost effective and and and these are this is the model we have. It's proven to be fair. You have all these ask the guys from Landfall. Uh, peak is not their first game with us. It's the third
all successful.
Mhm.
So they're they keep running with us because they trust us and they know that the cost is totally fine with what they do.
Uh Ripple, it was the first one for them, but they're already thinking, "Oh, we have to do more stuff. They're incredibly happy as well."
Yosmophobia and all these guys. So, I guess one of the the fears you have as a developer is always because for single player games, it's it's very nice. It's kind of set it and forget it, right? You upload it and once you fix all of the major bugs, it's kind of like you can can leave it alone

for a long for a long long time without having to touch anything. So, I I guess one of the worries you might almost have is that your game becomes too successful, but it's like a one-time purchase kind of game and people just keep playing it and keep playing it and uh causing very long tail of server costs. But this is not something you've seen in practice
normally. Okay. When you when when this long tail style of thing Yeah. Even for games that people do not monetize, you are you are not going to get thousand CU. I mean, if you don't touch the game anymore, normally keeping the 100 CU thing is enough. And and think I mean, even if you have to pay for the 500 CU,
that's going to cost you how much in 10 years. It's really
you can plan in advance to Yeah, we're going to fall down to this. If we get down to this, it's going to be this layer for 10 years or for 15 years, you

definitely will have stuff. But yeah, to be fair, if you if you have 500 CCU over a longer period of time, then your game's probably still doing okay. Oh, you are maybe yeah, you are probably still selling some copies that would definitely pay for that, but you may also be building other games that are selling. And keeping your old game alive with a decent 500 CU is a lot.
Check check Steam DB. Try to find which games are at 500. Right.
So,
that is a lot. Yeah.
So, say you're doing another game. So, keep funding your old game with the revenue for the new one is a very very good message to your community that you still care, that you still uh support them. You're you're not updating the game anymore except for a critical bug fix or something like that, but you are still paying for the for the infrastructure for the game to run. And so this if you do not have

um some some people actually keep doing DLC's that you do not have to pay for it again because if you already cop paid for the game, you get the DLC for free, right? But this um normally stimulates people to new people to buy a copy of the game. So normally it's a good way to do it's not really a live service kind of thing because you're not asking people to play to pay recurrently multiple times right so it's just that yeah you buy your copy but there are many ways to keep this afloat either with um and again yeah it's it's never been a problem that but do you want to discuss about game conservation aspects of this because you mentioned
that is also that is also interesting cuz it's it's also like you want your game to be around 20 years from now and it feel is it requires a lot of trust to think that like every company that provides online gaming solutions for example is going to be around in 20 years. I think a lot of them will

probably not.
We've been around for 22 now but so we've seen many come and go but
yeah the bot is a positive one. So it it's a a topic I care a lot about personally and occasionally we get companies asking about that what happens if you guys go out of business. we have some signed stuff about um if we if we sell or if you do something I I'm not familiar with these things but we do have some very very big companies. So, so we've talked a lot about indies, right? But we have big publishers like Ubisoft, Facebook, Meta, and some I cannot talk about. These guys all have agreements signed with us. They use our stuff for many things. Um, and some of them have but in contract some some stuff about what happens if this happen and that's this and that. But let me just talk about practical stuff that matter for

the solo developer and everything. So we've we've discussed internally and we we're one thing we want to do we're we want to make sure that if you're running fully online and you are running with um patch punch through you are using our cloud services and all that for that you have to have an active plan. But you know that if you're running offline single player, you're not paying anything. It's it's if you turn off the servers and you're because your studio is out of business, but the single player is still going to keep working. We don't have any type of pinging to validate the license of the local game. That will never happen. But more than that, we've we've discussed internally a lot and we're trying to see if we can make especially for fusion player hosted to make learn only games work without the cloud. So, so if we can reliably guarantee that

you're not using this as a loophole for something um we have to be careful, right? So
uh
internally we would like we we everybody from CTO to to our us engineers and everybody we want to do this and we're we're trying to solve any legal aspect that exist and everything to try to make it possible and so we want to offer this which is okay people ask a lot about this although nobody plays land anymore directly but it would be a way to keep the multiplayer alive even if we go out of business and the studio go out of business you would still have the land variation working. So I cannot promise because this is not written in paper but I'm telling you uh it's a topic I care a lot about and we we do care about all these developers who think about this and and and it doesn't hurt on on the business model perspective to allow LAN only to to be done without without

binging backends or anything. Mhm.
I cannot guarantee that the other stuff you use will do the same, but we're trying to. So, you try to be I I assure you that this the local single player variation doesn't touch anything. You can check the code. You can monitor the activity. Just unplug the stuff. It works normally. You will never stop working. And we're we're aiming we're trying to get this also for Fusion on LAN. For quantum it's a little bit harder because the the the server side the coordination piece we only have it existing on the server. It doesn't exist on the actual client. So it's a little bit hard that you get it there. But we do have deals for example for people who run a local instance of a Photon server for say you're developing your game and you're going to do it with quantum you want to go to PEX East to show your game or you you want to go to games. Let's go to to code.
Um, and then you wanna you are not gonna trust to run on our infrastructure with

the com with the Wi-Fi from the conference, right? So, you want to have a local setup.
Can you do that? Yes, you talk to us. We're going to give you a a a local license for a Photon server with a quantum plugin that you can run locally for the conference. It's it's going to expire in that case. Um, but we we solve it for you. So all your game is a special thing that runs on an attraction. It's like like on theme parks or something like that. We we can discuss solution for it to run offline in that case as well. It doesn't solve the game's conservation aspects yet. Although for the land the land variation would be interesting for that. But um
but yeah, we do care about these cases. We were always open to talk and I personally am enthusiast of that. I would I think games should be always possible to cons conserve and uh I hope I wish every tool you use would think it the same way and would be open to or at least work towards solving this in the

near future. But yeah, it's it's an interesting thing.
Yeah, I guess we game developers are also sometimes a bit uh hypocritical about it. We want the tools to do the work for us and then we ourselves often don't put in too too much effort to to make our games conservable ourselves. So
there's also that's an interesting aspect because you
you asked if the tool is but you have to be careful because you also have to do some some work to make sure your game
will keep working. Yeah, it is
clear.
That's right. I guess interestingly, one of the best guarantees you can give is that your your business model model works, right? If you're if you're like this is making us money, then I trust that you're not going to shut down the service. So,
yeah, we're not a startup.
We're privately owned. Yeah, we've been

initially based on investment, but that was solved long time ago when the company was reacquired by the founders and we've been running a profitable model. We're safe financially. We're we invest profits back into new products into the company. We hire very good people all the time. So, I was interviewing more guys this morning and um
so yeah, we're we're good and we we try to keep
that way for much longer. I I guess that makes sense, right? Cuz if like my reasoning is just like if
if the servers are profitable, why would you shut them down? The the first thing you do would be to to stop developing new products. And I I don't see that either. You're sounds like you're still cooking. I'm we're
we just released Fusion 2.1 nightly builds full circle uh which is the premium support program. So subscribers of that they are seeing now nightly

builds of the new fusion the new quantum 3.1 and we are about to release the nightly Fusion uh Fusion Unreal and Quantum Unreal are in beta for SEO subscribers. We're planning to release Quantum Unreal publicly as an MVP beta still this year and I'm already thinking about Quantum 4 and what should we put into it and other engine. So yeah, we're in constant active developments. We have stuff been reported uh rewritten in Rust some old C++ code base we have. So, we're we can even afford to do that, which is to get uh some older stuff we have to be refactored, to be modernized, and uh we're always doing that. So, and we're always happy to talk about and show what we've been doing. You just saw a little tip of the iceberg by talking here. What advice would you have for people who are

building their first multiplayer game? Uh do you see any common pitfalls that a lot of developers tap into?
I think the most common pitfall is the same in general as game is that the is that you you get a too big of a scope. So you you go for that and you you try to build your dream game the first time. And I would advise you to take more a more relaxed approach in that. Build much simpler stuff first. Release much simpler stuff first. Try out different things. If you're thinking about our stuff, learn quantum. Even if you're do something that fits really well with quantum, learn it, apply it on a game jam or something like that. See what you do with it. Prototype a lot. Then learn also fusion. When you learn fusion, you also learn a lot of other SDKs which are similar in in in principles. And and then you're going to learn it's like when programming if you think about

one paradigm only you be you you are bit of restricted if you if you learn how to program functional then you have procedural with separation of code and data and then you have object orientation which is kind of like villain at the moment but you but when you see multiple paradigms it opens up your your mind to groing programming in general better.
M
the same is multiplayer. The more you learn, the more you understand and you know what to use for each case. So start small. That's the main advice. Try something very very simple and don't don't try to say, "Oh yeah, I'm going to build a fighting game because that's simple. fighting game is ultra complex because of the timings involved and it has some very specific patterns in it that it's a crazy community that is fanatic and so but it start with something simple and and then try to multiplayer it try to multiplayer it in

different ways see what the challenges are do not go for your first as your dream game as the first one that's the first advice
cool makes sense
yeah and prototype What is what is the correct way to prototype a multiplayer game? Cuz you have a lot of usually the idea of prototyping is you you try out your ideas very very quickly so that you see if they make sense or not. And multiplayer sort of stands in the way of that cuz it's it's something that slows you down and that requires you to do some upfront tech investment basically. So, it's almost like you you got to know what you're going to build before you build it with multiplayer. Do you have any experience with prototyping multiplayer games? And what do you think?
That's what I do every day all the time.
Okay.
And I mean, I'm spoiled because I like to prototype with quantum. And because I am so proficient with quantum and so efficient in quantum I don't have I

don't I don't prototype slower more slowly because of I'm doing multiplayer. I just do it with quantum. And you know that when you program in quantum you don't think about networking. You just think about game play. And because I prototype as fast in quantum as I would do in pure Unity,
I don't get any slowdown because if it's multiplayer at all.
But this is specific to the kind of stuff that quantums work better. Um, if you're doing fusion, I would say the best way to prototype and and not get things too quick. Oh, sorry. And do not slow down too much is prototyping shared mode. Don't be afraid to use shared mode. client server is a lot more investment in terms of prediction and and all that. Don't worry about cheating if you're just prototyping. I mean, if you're doing in quantum, the prototype is can even become production ready because it's not going to change the way you do it. But if you're doing quantum

client authoritative shared mode, maybe we're going to do in the end as as player hosted. As you get more proficient with it, you may get as efficient in client host in in client server as you are in shared mode. But I would say from a coming as a unity developer, shared mode is the easiest one for you to think about. You invest a few days learning the tool and then you start prototyping with it. It becomes then it doesn't really add much of a time.
Here's the devil on the details. So prototyping small things with uh state replication does not add a lot of complexity to it if you do it carefully and you understand for example a tool like fusion or another SDK works. What is but that is the deception because state replication because the replication is is there showing in your face as a developer when you're building your game play as your

game complexity grows as your prototyping is becoming the production version of the game it al the complexity also grows with it and it becomes if you're not very careful the complexity becomes unmanageable. So prototyping is not the the problem on multiplayer going into production and maintaining it a saying is and that is what quantum does not suffer much from if you're careful. Of course, there are exceptions. You can mess it up, but normally because it's the same as a single player from the perspective of the simulation code, the complexity grows with the game complexity, but let's say it's a more linear relation. Whereas in in state replication in general, it is quadratic or exponential with the complexity of the game play. A prototyping is never big deal. But if again in both cases, you would have an initial learning curve. But once you get

there,
I think I think the best answer to prototype multiplayer is that you need to start once you start thinking multiplayer. You will always think multiplayer and you start thinking about single player as the byproduct
and then it comes naturally to you. So you would say prototyping multiplayer does not slow you down that much once you're familiar with the tools whether you use quantum or fusion like is
no it slows down less in quantum. It slows down a bit in Fusion, but it's not as bad as as as people think.
Um,
it it gets slower as you get more complexity. And this is not specific to our tooling, right? So, in state replication in general, it grows in complexity. Um,
but what is more complex in multiplayer always is the game design and and the and the game play implementation, right? So, forget for a moment online multiplayer and our tooling. So think about a split screen game or even a

single screen game but that you have multiple players multiple player inputs and you're playing that game implementing that that's different. You have to think differently and but if you think about the game as a local multiplayer game well for quantum that's already what you do. You think about it as a local multiplayer game and that's it. That's all you need. In Fusion for prototyping, you have a little bit added stuff here and there, but it's not crazy. It automates a lot of stuff for you and you can still treat the local input in Unity directly to move the character and all that for client authority shared mode that I suggest you should prototype with because it's easier to start from and it because it's also cost effective is the choice if you need to go state replication is the recommendation we have for simple for a mobile game. PC game is more forgiving. you have the full choice. All four options are valid. So, you really go for the one that makes more sense. But I think I think the the the answer is more on that that you have to think

multiplayer from day one when prototyping regardless of online multiplayer, but multiplayer thinking about that every day and then it becomes second nature. As someone who mostly builds the tech, do you do you sometimes spend a thought on what what makes for a successful multiplayer game?
I do. Steam DB is my friend. I watch Steam DB every day. I mean, I scroll through and analyze the stuff every day. I I prototype a lot. We're all game developers, right? So, we all come to this because we like developing games. I've I've been a game developer uh in indie. I founded my own studio many years ago. I ended up not successful as many of us do and I landed an opportunity here and to never look back. But I like to think that I actually am part of some of the teams that that ship this stuff with us. It's a different type of thrill. It's a different type of work, but we work together with a lot of our customers. So we we do workshops

with them and I I get to play test games before everybody. We we we see things. We know what's coming. We we've we get as excited as the team because we see it's our tech, it's our code going there as well the same way but and and I still do um prototype a lot and but there is a practical reason for that. So I like to to analyze trends to see what's happening. That's why I watch your channel. That's why I watch many other channels. I I
okay
I I I like to understand what's going on and what's what's the stuff that's prominent, what's working with players, what's resonating at the moment because we we have to make sure that we build features that sometimes we create a sample or we or we create an add-on which is a helper set of libraries that fits a certain genre that people are developers want to do. Now we don't do games. We never compete with our customers ourselves. But we we like the

tooling is not only the net code. So quantum is a game engine. I have to build so much stuff on top of it to help people build stuff with animation curves uh splines stuff and a lot of these uh friends lop style games kind of and and also the frustration with trying to do full predict rollback physics with uh physics that doesn't scale for that um led us to develop a different a different way of prediction predicting physics um we we name it forecast that comes with the new fusion uh SDK. We already have some very positive feedback on our public discord that you can see. Um and so that was because of this genre and uh and and
it's not going to work for everything. It's not going to solve magically. It's not going to replace full prediction and roll back like quantum can do for physics. That's still that's much more accurate and better. But we cannot do

that with physics at all. It doesn't scale. So this is interesting because it solves these more simple generas carrying stuff around pushing stuff around and and and incredibly it works really well for car physics which is may sound as a surprise but if you do it right it's actually very good with that um but uh yeah we we I look at trends all the time so and um I'm constantly prototyping stuff mashing up between two different thing
do do you think and what succeeds or not is Do you think it in trends or do you think it in like more like design choices or like like what do you think what makes for a successful multiplayer game? Is it hopping on the correct trend at the correct time or is it something else?
No, it's when you make the trend.
When you make the trend or if you ride I was thinking riding riding the wave at the same time. So yeah, um if you're behind if you're trying to mimic a trend, you may have lost already the the the wave.

You you need a bit of luck. It's a mix between capabilities. I mean, your qualities as a game designer and developer to execute and a bit of luck. But, um, maybe this is a question you should ask to the landfall guys. They are so good.
They they landed three hits in sequence. How many times you've seen that happening?
Yeah. Yeah. I mean,
which is the sticky game uh AC totally accurate battle simulator. Then
content warning and now peak with collaborating with uh crab. So I forgot the name of the other studio.
Yeah, they have a crazy success rate.
It's but they're writing the trend and which is
they're writing the trend and creating the trend.
Yeah. So that's what I but it's fun to talk about it after it happens. But what is next?
I don't know. I wish I I did,
but I don't. So you do think you need a little bit of foresight to maybe

anticipate these these waves.
You need you need or to make so something so good that you create the way.
I would like to believe these guys are
I know some of them I talked to they're in our support channels and everything but I I do not work directly with some of them but I see from other studios I've worked with that you need to be creative. you are need to be. These guys are smart and um they're smart, creative, and they're competent in executing stuff and they're incredibly passionate about what they do. But is that enough? No. There is some other magic recipe plus a bit of luck or a lot. I don't know. But uh yeah, I wish I knew the answer. I I I try to I we we sometimes we try to anticipate some stuff and and we occasionally we prototype something that's unique that it's not same as before or worst case sometimes we do I mean we see something that is PC only

and we know that these guys are not going to do that in mobile because the way they did would not port to mobile and then we may create a sample for a customer that is similar in some ways and then the customer is responsible for doing it but it's not the rule. But but I personally occasionally create some prototypes based on on okay what if we mash this and that and see what there is this new trend going now on on PC games in this direction what if we get this trend but twist it a bit and adapt it to this what would become the game then I like to prototype this and then if it's fun when we play test um we may share it with customers to see if they do we may share with the studios the games that inspired us to do to see if they would like to have a spin-off that is different.
But this is just prototyping. I mean, I as as a I'm both a tools programmer, engine programmer, and a gameplay systems

programmer. I
Mhm. I I am I like to think that I am very good at if a game designer describes me something or if I see something as a mockup or something, I am very good at implementing that as a proper interactable game play. So that's something I can do well and quick. So that's why I prototype so much.
The reason why you prototype is to figure out how to make your tools better.
You eat your own dog food. I I occasionally I find bugs that because I'm using slightly different than the usual developer and then I find a case that was not checked and then it's giving me an inconsistent result and then I report it to the guys. Sometimes I fix it myself if it's part of the code I originally built. Um but yeah, it is constantly to be testing the limits to see how our tool can be used and and to both exercise how to do that to show to

customers and and know how to answer and know how to give them suggestions to know how to give you a suggestion on how to implement this or that but also to find bugs. I mean we it's not only me. I mean uh everybody in the team they're all game developers so they all take some time of their work to prototype and we have a we have a channel in our internal Slack called work in progress and play test that's essentially for us to share our internal prototypes that we're working on our we we have we can spend some time normal work time on these and then we always share and we have play tests together on these fun things and yeah we We do this. All of us do this. And I think it's a very healthy thing to do because we're constantly using our own tool to build the kind of stuff we like. I think it it it does help. It's I don't know how much it shows, but we do this a lot.
Oh, now that you say it, I think it shows a lot actually
cuz you you don't want the actual

developers using your tools to to find all of the bugs and report.
And when they do, I mean, we If it's a bug, we fix it. It's But yeah, but definitely we want to improve and and there is no better way and we have fun doing it, so why not?
I think that sounds extremely healthy. That's really cool.
Are you based in Germany as well or
I'm based in Germany as well? Yeah.
Yeah. Do do you go to Gamescom and other conferences normally?
Normally not, but I might go next year.
Okay. Interesting.
Also also I think I met one of your colleagues at uh
was it Robert at the port side.
Uh games assembly could be. Yeah.
Oh, games assembly. I think it was order mark maybe.
Mhm.
One of these two. Yeah. Cool. I mean, one thing we we occasionally do, I mean, if you if you if you are not too far from Hamburg. You are also welcome to come to visit our our headquarters. It's uh in the Huffen City. So, just feel

free to touch based on that. It's uh we we like to to to to get people here and to visit us. We're cool.
Yeah. I mean, uh, Paul and I will probably reach out once we start making our next game once as like once we know what we want to build, we'll probably reach out again and try to get some advice from you guys. Yeah, we'll be happy to.
Cool.
Yeah, I mean that's that's it from my side for for now. Do you have anything else?
No, I mean it's just it was super chill, super fun for me and I just want to thank you for the invitation to chat with you about multiplayer. I hope it was these were interesting topics for you and I I hope it was not everything obvious and it was really some useful stuff for you as a developer and also for your audience. So um as I told you before I already I already follow your channel and I knew you from the videos before. So it was a
nice surprise when you first reached out to ask questions about our stuff. So

we're
we're happy you did. So
yeah, I mean, thanks a lot for for taking the time. Yeah, I I was kind of asking some boiler boiler plate questions for the first hour there, I think, or so, but like the the last two hours were all like just stuff I've I'm I've been very very interested in, and I definitely got some some cool answers. Um, yeah, you see, I I I like to talk a lot. So, um uh yeah, even if you ask some boiler plate stuff, I I I like to talk about it. So maybe maybe that works.