Netcode for Multiplayer Games | Glenn Fiedler, Avinash Pandey, and Chris Wegman

welcome everybody today we are going to be talking about netcode for multiplayer games and with us to talk about this topic we have some incredible speakers including first of all glenn fiedler who has been a programmer and multiplayer lead in a lot of really incredible game studios including irrational pandemic playstation respawn and currently glenn you are the founder and ceo of network next which is hopefully i've got this right essentially creating a new internet for multiplayer games welcome glenn secondly thanks secondly we have avinash pandey the cto of lila games leela is best known in my opinion for having the handsomest ceo in the games industry right avinash anyway yeah uh most recently uh avinash was working at a different multiplayer free-to-play game studio and released

mass gun a multiplayer shooter game that amassed over 40 million installs and finally we have chris vegmon founder and cto of photon engine photon is a world's number one independent networking engine and multiplayer platform and kristoff has been working on photon for checking from your linkedin chris almost 18 years now so assume you know a thing or two about about this area and it's uh great to have you all on with us so i thought we could go ahead and dive right in starting with the biggest challenges in terms from a technical perspective was wondering if we could talk about you know what are the big technical issues that game studios will encounter as they try to develop multiplayer mobile networking games and chris since you've got 18 years here maybe we could start with you uh sure um that's uh

that's a good question but um pretty broad as well so i could dive into like tons of different things we see so our claim is we make multiplayer simple so that was always what we try to provide and provide different techniques so usually what we say so the first thing if companies approach us and then ask okay how should i approach multiplayer it basically boils down to what engine do you use so the first choice is unreal where there's unity where there's probably a native uh engine then what genre are you are you building right are you building a moba first person shooter turn based game action rpg cooperative competitive and then basically on top of these briefings we we help customers to decide okay what would be the best netcode model and but still we see even

experienced teams struggling with with netcode for sure so you have all the aspects of networked physics right so a game like four guys where you have 32 people playing and you have network physics in the game is super challenging to build or a competitive fps i mean the session is is focused on mobile networking so um i think this is it's good to have a focus because if you talk about let's say pc or console type of challenges it's it's usually completely um different so um so network physics matchmaking is is definitely a challenge so um if you launch your game you have low player numbers you you need bots to to integrate into your game you need to figure out how you roll out the game globally so um where do you launch in the us in europe and asia and you do you need to ramp up service everywhere

you want to have a good latency everywhere you want to handle disconnects and reconnect so if your game becomes popular you have the challenges of cheating so suddenly a lot of cheaters get into your game so there's a lot or a high motivation let's say for for uh for people uh to to hack your game uh so then of course you have the challenge the challenge of latency so your game may become popular in let's say uh we we just uh launched a game uh together with a partner in south america in india in africa so it's it's quite challenging to provide a good latency in these countries and you want to serve your your customer base really well um and then what we what we see that um the the games which are launched recently and become successful are

mostly driven by streamers so streamers are major force to make your game a success so how do you design a game that it's actually well streamable let's say and it's really fun to play so we see the the landscape really changing and it's it's getting so professional so it's not easy to launch a game these days and then just have a hit right and then avinash for you when you were working on mass gun and now with this kind of new game design that we're working on which has different kinds of requirements what are some of the challenges that you're running into as you start to prototype some stuff right uh so when we talk about mobile so there are some certain constraints which apply to not only netcode but overall game and when you are just starting to develop any game you have to think about all of them so on top of my mind like performance and bandwidth these are like two major limitations which you will have on your uh

on your game so whatever solution you choose you have to keep these constraints in mind so let's talk about performance so whatever networking solution you use depending on again i think this answer will come up in all questions like uh what type of game are you building so if you are building a moba where you know large number of players then the solution is going to be completely different like you cannot send data of all this for example 500 plus from mobile device to the server so you have to have plan and architecture to kind of you know mitigate those kind of things then one of the hidden things which uh from my personal experience i have seen is banned with cost so game developers don't realize this initially like you know send as much data as possible have really high tick rate but then when once the game is live you have whatever sees you thousand two thousand twenty thousand or million csu then this factor starts kicking in and sometimes it might also happen that you might end up paying more for bandwidth

than actually the server cost so you have to make sure that you optimize whatever data you send on the wire and how much data is required so if for example you are building a shooter game of 100 battleground then you are only interested into players who are in your interest area so you only send data of those players from server and from the device as well so i think these are the top things and the other thing i think for mobile games which which is one of the attributes of deciding whatever solution you are going to choose is cost so multiplayer is not only about net code but there are other services which you are going to use so you will need api you need database and then it is not that hard to kind of extrapolate some of these numbers so if your game has 20 player account then you can just to build some prototype see how much data you are transmitting for one player what kind of server you need and then you know calculate net code cost server infrastructure cost database cost api cost and calculate everything and see whether you know this makes sense for your game or not so i think these these are some of the

challenges which i will add on top of what crystal talked about thank you and glenn for you and by the way since you're actually taking more of an approach in terms of trying to actually fix the network does that suggest that some of the solutions from a client perspective are intractable to some degree or maybe you could talk a little bit about that as well sure so i don't i don't want to dive right into the network next thing because i do think it's something you would experience right at the end right at the end of the journey of all the things that chris chris and avinash mentioned um let let me start by basically saying like look when you're making a multiplayer game as an engineer what is your job um as a network programmer you've got to find a way for your team to be able to write networked game code right so kristoff that's particularly a great fit for for a photon quantum because it basically networks itself with a deterministic lockstep model so that's that's really really powerful but then if you have teams coming out

saying i want to make an mmo or i want to make a an fps a top tier fps with a high player account um i'm not sure that there's actually an off-the-shelf solution right now for those sort of games that that is at the at the top tier of what i've experienced working at aaa studio so in in many circumstances like respawn um these these folks have been shipping these games all the way back for for 15 20 years so it's sort of like the the sort of code bases that these games have in the networking approaches i don't see a turnkey product out there that does this well not even not even unreal for example the way that call of duty is networked is quite different to the unreal network model so but let's let's sort of go past the real hard part which you should think of when you i want to i want to network again which is what's my network model and how do i make my my gameplay engineers productive right when you launch it you've got this really interesting challenge

which is where do i host my servers and what provider do i host them with and what's it going to cost and and there's a really interesting thing here um you can try you can try to host servers everywhere right and what happens is that you actually end up fragmenting your player base and it's really challenging because when you launch a small game you actually don't want that fragmentation you want the whole point of a multiplayer game is for people to play together right so there's an interesting trend that i've seen over the last 10 years where larger games are setting up regional play and trying to get connectivity to that central area for that region really good and that's probably best demonstrated by riot with right direct because riot created league of legends and realized that the internet performance wasn't good enough for their game they basically built their own private internet and

league of legends is hosted for north america entirely out of chicago right there's no east coast west coast servers they did originally try that and then they they consolidated centrally so so the tension that i see when you launch a game is like where do i host how many places do i host um do i try to get my game close to everybody um do i try to go central and and how do i make sure that people can play quickly together and then if you get really popular it turns into how do we do skill based matching and make sure that people can play together at a similar skill level um but uh there's this tension between hosting in many different places and hosting in few against fragmentation this is why some of these moba games they're very popular because you know 10 years ago player bases would be fragmented by i'm playing one of five different game modes right so like you couldn't play you know

an esoteric game mode on pc at tuesday at you know 2 pm because no one else was playing it near you so you might notice that many games now have only one map right and only one game mode so there's this interesting trend towards trying to make everyone play together and to reduce fragmentation and then the natural extension of that is hosting fewer locations but have better connectivity to those locations so that's what we're doing in network next so glenn if i'm understanding you correctly i think what i'm hearing you say is that for a game studio to provide a great multiplayer experience there's one way to think about it is on the netcode side and there's different approaches whether it's you know client lockstep or kind of mmo type of model but then there's also like a network approach and so from the perspective of the network how you manage that it will also factor in but in terms of like if we could focus on that netcode part uh yeah i was wondering if

we could talk about what are the primary approaches in terms of all the different types of ways that you could attack netcode so you know client lockstep server mmo and chris maybe you could talk about quantum's approach and maybe your your thinking in terms of what the major types of approaches are um yeah sure so um when we started back in 2003 and in the in the past 10 years in particular where we focused on real-time multiplayer we started let's say with the classic approach of of synchronizing game state which is to say the the most common sense approach where you say okay you have whatever we are playing and you are exchanging positions and velocity and rotation and you you synchronize that continuously and then you have to make sure that all the clients are kind of seeing the same thing and do a good job there or you let the

server do the work which is synchronizing the clients at some extent so which is still the major model what what companies are doing and then about four years ago we we started to to investigate into the into predict rollback networking which is only synchronizing input so it looks like all the controllers are attached to to to your local machine so you're not synchronizing anything inside your simulation but only the input and this is revolutionary in the sense of that as glenn mentioned so there's no net code anymore so from a developer perspective it simply doesn't exist it looks like everything is local and it sounded too good to be true it still does [Music] but it's quite fascinating how well it works um

of course there's a catch so this approach has limitations and you couldn't build a fortnight with it or an mmo so it's limited in terms of players which is the major limitation of that approach but we believe that for a lot of genres and game types it's the best possible approach because it takes away a lot of the pain you usually have so this is again you have to to check okay what are you building is this what is the best netcode strategy so yeah basically you can divide the whole networking world into synchronizing game state and synchronizing input from from top and then synchronizing game state is still very popular and it's not going away especially if you if you have games with more than 50 players and that's another trend we see is the number of people playing together are getting bigger right so we have customers asking oh we all want to do a

battle royale with more than whatever 50 people so it should be 100 200. probably fortnite would would like to i mean there's a reason why their games are limited to whatever 100 players i would like to get it as large as possible um because at certain scale but certain game types are just fine right you wouldn't play pool with 100 people that doesn't make any sense or soccer um so yeah so these are basically the the the different approaches right and avinash and glenn would you agree with that characterization in terms of the different approaches uh i think when you are thinking about net code uh the mental model behind choosing netcode should be what your game what kind of game you are building so you should definitely think from what is your core game play mechanism and what kind of feel you are going to give to your users so for example if you are building 8 ball pool then the

choice is very obvious you cannot build edward pool with traditional approach you have to have deterministic physics on the client like you cannot sync uh every boss position every frame you know you need some kind of simulation on the client where you just taking player input some force and then you know the entire thing is simulated in the same way on every other client but if you take an example of shooters i have seen shooter games which work on client authoritative where the gameplay doesn't worry too much about cheating or sometimes if players get shot behind the ball they are like okay i'll respawn and i'll play again so you know what kind of game you are building think about that and then choose the solution so if cheating is a critical problem for your game then you have to go with authoritative physics on the server or if player count is you know limited then you can go with deterministic approach what uh quantum is built upon so i would say like the the choice of net code or engine should be from game play and mechanic the the field

point of view rather than you know finding a good solution then trying to fit it in the game uh gameplay i i have um an interesting point of view on this one so i've actually worked on uh peer-to-peer open world games like mercenaries to do the networking for that and then i went and networked god of war and then i went and joined respawn and got schooled basically on how they've been networking call of duty and so on for 10 or 15 years and learned a lot of best practices there and one of the things i learned from that is i'm never doing peer to peer ever again right but that's like old news 10 years old news now at this point so let me let me kind of outline how i think of the problem space and what's really interesting about it i think is that um it's hard to to see it at first but it's actually a series of different it's a continuum across a few different axes so there's actually a lot of similarity uh kristoff between

the rollback with gtpo and photon quantum and how fps games like call of duty do roll back although they think of it very differently right and they don't they don't require complete determinism but they're very similar elements so here's how i think about networking look the main thing you're really going to want to decide is does the game run everywhere and i'm trying to have it in sync so like if there are 16 players the whole game is running everywhere and all 16 players are somehow in sync option a right and option b does the game really kind of run on the server but each player kind of sees something that makes them think that that they're there and this is the major divider right so um an example so i'm playing a fighting game of four players right and maybe it's ggpo style networking um which which kristoff i

think is it's fair to say is basically a similar strategy to photon right so deterministic and then there's rollback to high latency and all the good stuff okay but what's really happening is that you're really trying to have the same exact simulation run out on four different machines and and the limitations of this is that if you want to have a world that's larger than what any of these four players see or is computationally very expensive um you can't really do that because on the mobile phone you can't simulate that whole open world that's like 10 kilometers wide even if you only see a tiny part of it the whole game has to run on every device right so that's option a right and option b is kind of like look clients are just kind of viewing a ghost of the world or a sub part of it or a section and the real game lives on the server

and this is really the divide and it's the way to think about it uh am i having the game run everywhere or is there really just do i think in my mind the server is the game and i'm just a client in it like kind of like that thing with like you know i'm touching the elf i know all about the elephant's tail someone else knows about the elephant's nose right but you don't the two people don't know the whole elephant right um a deterministic lockstep model is kind of like no no you get the whole elephant all the time or it doesn't work but uh a more classical fps and i'm going to go and limit say it you know mmos must also do this although i don't have any direct experience with mmos it's about having a really large world on the server and then the client has some sort of relevancy and they see a subset of it as they move through the world for someone like myself the simple view like more from a business perspective that i think about when i think about

these different approaches is that one just trying to understand the trade-offs and it sounds like from what we're talking about the trade-offs are really about player count potentially also glenn to your point about having to run different instances on the client you know is there it's so you're running cpu does that have an impact on lower end devices for example if we're going with a client kind of deterministic approach and then also from a server cost perspective something that you mentioned avinash if we're running with more of the photon quantum approach then certainly server cost would be lower because we're running a lot on the client relative to server and so when i think of the other way to think about this is more from a market perspective right in terms of the two biggest genres of games mobas and shooters right and mobas are the number one monetizing category at least because of honor of kings in china but it would seem like for a moba more of the photon quantum approach would be

superior because there's a lot of engagement a lot of gameplay extremely high server costs probably and then on the shooter side if you're going battle royale high player count wanting to have broader penetration across lots of devices then i would assume more of this server authoritative model would make the most sense that is that a fair representation of uh or is that a fair way of thinking about the the different approaches i can give a concrete example do you remember titan 4 1 when it came out like the original respawn game with the giant max and yeah you know you know what we did on the consoles render okay because the whole game ran on the server okay right so what the server side thing did in that case even though the server costs were very expensive is it allowed us to spend a lot of time on the xbox console and

the playstation console purely on rendering right all the ai logic and the level scripting and everything while you're playing multiplayer in titanfall 1 um was running on a dedicated server somewhere so actually it's interesting that you can kind of choose where you want to do the work right but when you view games like call of duty and you know those sort of first person shooters um if you do those right almost the entirety of the simulation and everything is running on the server you might only be running simulation for your your own character and your your shooting and everything else is server-side um and kristoff that's quite different to to the the lockstep model where everything's running on all of the peers yeah i mean you combine it right so you don't want it's not the extreme case where everything is running on the server so this is more the game streaming mode like stadia or the other whatever hatch on the

mobile side of things where everything is running you send the input to the server the server is simulating you're sending the state back so you as you said already so you have this smart combination right so everything you do locally you try to predict in your local bubble and be smart about it so it feels snappy because fbs competitive fbs players do not accept lag so they want everything instant right and then you have the other challenge of favor the shooter right so i see len sneaking off a corner i'm i'm shooting him right with the sniper rifle and i want to hit him because i saw him on my screen and you need to hit him so but in reality probably len already is is not there anymore right so you have to make this decisions and and say okay i saw him so you you hit him in the head right for example or um but this is mostly let's say on the pc side of things where it's like super sophisticated and you have where glenn worked i mean these teams

are the best of the world right so on mobile it's it's different type of teams i mean these studios are coming to mobile for sure and you have the supercells but looking at the at the mobile network titles like the successful ones are slowly getting into this competitive hardcore genre so whatever supercell invented inventing um the clash the clash genre right which is was the first big real-time multiplayer hit in the number one or top ten charts um fps's are since we exist on mobile super success so i would say most of the um successful games running on on our cloud are probably fps built by really small teams and they are far from having this professional competitive let's say experience compared to

to a pc game but yeah totally it's um you have to make a lot of uh choices and people are we we see people are very confused uh let's say uh making a decision between all the different choices and and the thing is they ask okay what should i choose and say well you know i you can do everything it just has everything has pros and cons so it depends on your team mostly on the skills yeah so maybe we should talk about that i know avinash you've been looking at a lot of third-party solutions out there that provide various types of netcode solutions and obviously you're i know you're a big fan of photon quantum as well but could you talk about kind of what's out there and then after that it'd be great to hear from chris and glenn maybe you guys could also talk about okay yeah there are kind of third-party solutions out there then the decision between external versus building all the netcode internally it'd be great to hear from you guys about

what that effort would actually involve but maybe starting with you avinash on on what's out there all right i mean it's uh there's no one perfect answer for this question because it depends on so many factors so it depends on what stage of your studio is in like do you what's your runway how big is your team what is your talent pool and then again uh as i'm just reiterating on the same fact like you should think from your game's point of view like what kind of game you are building what's the goal field you want to give to players so all these factors are really important when you are deciding on this solution so uh if you are going with unity on unreal uh these choices differ a bit so unity kind of duplicated their old networking solutions so there are some uh apart from photon and when we say photon photon itself is like multiple products in itself so there is put on real time put on quantum photon bolt so all these libraries they are built for different type of games so you have to be really uh sure about

what type of uh solution you need based on your game uh apart from that there are some open source solutions like uh you know dark reef mirror mirror which people talk about but when you're going with any solution you have to always know what's the ecosystem around it so if tomorrow you get stuck so on high level all these solutions would work like you know when you are evaluating any multiplayer solution you would want to send some data some player data i want to do authoritative movement what does it mean like you send some data to server server send data back to your device and if you are testing in your local scenario it would work perfectly so you have to make sure that you simulate these things in a real world scenario so if it's a mobile game and you think that you are going to soft launch in asian countries so you know host some server in asia if you are using photon then you know take asia a reason then if it is a 20 player game then come up with a way where you can actually simulate with 20 players whether those are ais or you have 20 devices or some sort of simulation and then you know evaluate all of these

things because when your game is going to go out in in the life then you are going to face lots of challenges which you would not imagine right now like people play from weird locations with weird network lots of server hops so you might optimize your game for 100 millisecond thing and you are imagining worst case we could go like 150 but you would be surprised by seeing like you know people play at 500 milliseconds or even to a second pick so you have to be ready for that whether you support those kind of things what's the user experience like do you just let players you know uh jitter do those spring roll bags so i think all these things you have to evaluate when you are evaluating a networking solution okay and chris maybe you could speak about if we were trying you know if we were trying to think about using photon quantum versus trying to build that internally does that take us 18 years or what kind of how do you estimate the effort to build something like you guys are providing so of course you can always let's say you have to make this build or

by decision with quantum it took us four years and we have some really really smart guys building it in the past four years so i would say it takes that time usually you have it takes two years to build a version of something right and then it takes another two years to build the second version and then you you are really embarrassed about the first version you launched basically so this is usually the case and um with quantum we we took a different approach and we are charging um like an excess fee of one thousand dollars a month which is very unusual so it makes quantum not available for a lot of companies because it's too expensive for them especially independent teams but we were able to to convince some really big customers about 70 teams

using quantum so why because they made that decision that let's say they can't build it in-house and they need it now so usually you have a mission you have a project and then you want to have something now to build it no we don't have a chance to convince ea or supercell or [Music] riot games to to use uh quantum most likely maybe if they have a little prototype team maybe we get into prototypes right so this but for ev everyone else it's even ubisoft square nx and these type of companies are definitely looking for a solution right now and this is so we are i mean with the product you always have to be at the the right time at the right place so like glenn is i think we talk about latency a bit

later and how you enhance this that's i think this is a time now because multiplayer is becoming big and um and there are like four guys is is a great example within one day one million downloads right or phasmophobia another hit by just one guy in the uk who built this running on photon so becoming the biggest customer in in just a week or so and then or among us right it's a single guy from brazil building this game which is number one everywhere um so and then if you have if you build this yourself and have to scale everything yourself that's that's a tough job so it's it's good to have good partners for sure and glenn and maybe you could also speak to is there a difference if you were trying to go with the more client based approach versus server mmo based approach

um yeah so like look even if you were to take excellent open source software which there's a lot out there i've written some of it and and people come to me like you should use should we use netcode for this or whatever and i'm like yeah do you have five years right because because you see the hard part isn't a client server protocol the hard part isn't sending messages between a client and a server the hard part is working out how do i network my game code right so so christoph your product solves that very neatly in the areas where it applies right where it's challenging is when people want to make an mmo or they want to make an fps there isn't that turnkey product that they're at least not that i'm aware of right at the same level of quality um and you know my my advice to these people generally is you know i used to consult in this space a lot five years ago it's like most of what i did was telling people just how much work they were signing up for

if they chose to build it themselves and and generally the only team where i would kind of suggest this might make sense would be like this is a core competency of us like maybe a team that has its own they've been doing single player for a while they want to do multiplayer and they have their own custom engine they've been working on for 20 years right in that circumstance perhaps building and designing it yourself might make sense right um almost no other circumstance like let's take the call of duty team right they worked on medal of honor then they went and worked on call of duty and then modern warfare right and then they all got pissed off at activision right and then they started respawn and then five years later they made titanfall when it launched and i got to help out right at the end right and you know what they did the whole time they used the source engine for titanfall they didn't even write their own engine i mean this is a top a world-class top tier you just can't do it in the time

you need to do it to ship your game if you try to do everything so um it's really really really difficult making multiplayer games so if there's a if there's a turnkey solution out there that is going to let you use something and and and get to the important part which is making the game and shipping the product right rather than five years noodling around with reliable udp protocols right that's the important part and even even if you're even if i was on a team even if even if i created a studio you know um i would probably even hesitate to like go you should use my netcode right because i would need two years to like get all the stuff in place and it would have to be a run rate of the studio where it's like okay we have two to three years just to prototype and get this done and and we might have a game out in four or five years

right and and and i'm i'm i'm world class at this right it's like it's going to take time the hardest feeling as a network programmer is being the bottleneck and the whole team you have to have all this stuff finished before they can network the game code and you want to get to that networking the game code day one right so um if your product doesn't fit within the turnkey stuff that's out there maybe your best choice is to make it fitted and maybe speaking to the challenges like one of the things i hear a lot about is from let's say some of the bigger framework providers like unity and one of the things i hear and i'm more of a the business guy not the technical glide but i just keep hearing that the net code in unity is so bad and there's the i think they restarted stuff over there but in terms of like what's available

from unity and from uh epics unreal engine could you guys talk about what is there as far as like that the netcode support or the the capabilities there and i just give a like a bit of an overview on unity like yeah there's something i really like about unity actually which is that unity is less of a framework you can use the parts of it that you really want to use and not the parts that you don't and that's actually something i really respect so the problem is is that networking is a structure it is a framework you have to do things certain ways right but there's no one framework that works for all type of games so i think unity's actually realized this and and you know going on from their component system that they're doing and all the good work to make it modular um i i think unity is personally a superior solution but but you can take the unreal engine networking code and i mean it's all there it's shipped games there's nothing wrong with that but it's rather inflexible when you're gonna kind of take all of it

right so so that's the major difference but i will also bring up that the way that quake derived games network like call of duty and these best practice games um is quite different to how unreal engine is networked so maybe maybe that's better described as a flavor of fps networking but but i don't actually think the unreal engine networking approach is actually the best one for an fps type game even but this is sort of getting into like wonk territory chris you have any thoughts on what's there in unity or unreal yes so unity is an amazing ecosystem so what they what they did is you know they they do a lot of things not perfectly whatever it was initially the the ui stuff right it was instantly fixed by someone in the community on the ecosystem building it and it's not considered competition and the same is what we did

10 years ago we were one of the first networking solutions in the asset store back then right where unity was pretty pretty small as a company and they always support us supported us and and once they started to build something they reached out to us and said hey we are going to build a competing product just that you know we respect your work we respect everything you did and you're part of unity but we feel we need networking in-house so and they did this a couple of times so luckily for us there's no um nothing ready yet right so it's good for us so we can we can grow but in the end it's not a competition or we are in a fight so we are in very good terms so that's what i like about unity so i cannot talk too much about unreal because um we have a couple of customers using

unreal but most top notch games use the in-house engine it's like windows internet explorer basically the default solution you use it no matter if it's a term based game or anything a lot of companies just use it blueprints drag and drop but um yeah i think there's a lot of space in the unreal world but but as glenn said it's um it's it's quite complicated right to understand unreal so to to get into unity is much simpler you can just grab the stuff you need you have this ecosystem you can you can mix and match of course there's a lot of things to complain about about bugs that never get fixed right you can endlessly complain but all in all it's it's an amazing solution and i think both solutions have their space and because of the competition of unreal and unity we are seeing all the good things right because

both companies are totally motivated to to build the best engine on the planet and this is benefiting our are basically all the developers including us so but unity has the healthier ecosystem because it's a much larger base of developers and you can monetize right alone in the asset store you probably earn a couple of million a year if you are in the top sellers it's not enough to to run a big company right you need a subscription business what we've built up or services because the asset store doesn't allow this yet um but yeah they hire very smart people so and it's for me it's just a matter of time if you hire like very smart people that at some point they will have a proper transport layer and relay server but we try and as a small very fast chip we try to be ahead right so we try to be already in the next stage so we believe

the time of building a relay and transport layer that's kind of boring everybody's doing it amazon azure including unity right it's it's a commodity basically very soon um we have that since 10 years so we are trying to build templates we believe that you want to give developers a moba template rather than telling them how to build a mobile right it's like amazon was developing from giving you machines to oh yeah there's a database and i think now we are entering the area where the era where you where you help developers with every with actually giving them what they are looking for oh you're building a competitive fps here's a template for you which is exactly what you're looking for right or a mobile or even a turn-based game i think turn-based games are not going away and they are not bad right you can you can earn a fortune with eight ball pool or having the first four

guys mobile or um yeah and glenn since you're kind of you know you i would say that given where you are in the market you're kind of neutral if you were to give unity advice on what they should do to in terms of netcode to just like forget about it and rely on you know folks like chris or to try and build something internally what advice would you give unity to for with respect to netcode i've actually given this advice in person oh yeah okay um look i'm kind of torn here because you see what i love about unity is that it's flexible and it doesn't sign you up for a structure that you don't necessarily want to use so so i actually think that um that uh i'm not sure if you can actually network

unity while having unity remain unity because you what i like about unity is i take the bits that i like and i've got an editor and a renderer and you know and off i go and i maybe i i'll i'll run my own dedicated server somewhere and you know and and i love that i i i chafe with my experience in networking i don't want to do things that i don't want to network things the way that unreal engine does and if i was to use unreal engine i would and i have um i would be contin i would rewrite the whole network in day one it would take three years and then you know that's that's why i'm not working on that right but like i love the fact that it's actually wide open on unity and um products like like photon quantum can come in and meet a need i like the modular nature of it and and my fear is that you might have to create a little bit too much structure to create a useful networking template it might be kind of philosophically a

culturally opposed to how unity works right um so i my advice would actually be lean into your strengths and continue making everything modular and foster a community that will solve problems in different ways and surface the best i i i think that i think the things that unity you're doing with the component system and the core and making everything open source and more modularity and more flexibility is is the way to go and and less frameworks right so so i would almost say just don't okay just don't yeah services are too tempting right so you earn a lot of money with it and so and and public traded companies are driven by monetizing everything and growing yeah so so unity did acquire multiplay and multiplayer do hosting and that's

that's an interesting play and somewhat adjacent to the idea of you know building a framework for for a network netcode right um and i think that was a good acquisition and that's a good team and uh you know clearly providing hosting services around the world is of value right so that's that's good um but like you know i would kind of be sad if i had to if i had to wake up and kind of be like i'm using unity after network at this particular way right there's the unity framework because because i know that there's just so many so many different flavors of how you can network things right that that um and it would be almost impossible to express all of them in a single solution right and unity is very much a company that tries to kind of make a product that can do everything for everybody right do you see so it's almost it's

almost the impossible problem so i don't fault them at all for not having a solution i think is actually true to their values and uh they they should probably provide good low level solutions and modular components and lean into what makes them in my opinion better than the unreal engine um and uh avoid the temptation to try to create additional structure that may or may not in the end be the correct structure for the top tier games right or the indie games because it's probably different right glenn speaking about that in terms of having different solutions for different kinds of problems maybe you guys could check me on my very simple basic understanding of for a small studio what would be like the best in class approach again for a small studio and it sounds like if you're going to be making a like a supercell type of game like a clash of clans clash royale or a moba then something like a unity plus

photon quantum would potentially be a best-in-class solution if you're gonna do a battle royale based fps then maybe actually just jumping over to uh unreal and using the unity networking would be best in class and then if you're doing an mmo then maybe something like a smart fox i don't know and avinash plea please check me is that would that be a kind of you know high level very basic you know understanding of different solutions for different types of problems i think my take on this is not to scare off the audience uh you know with if they want to make a multiplayer game they have to spend like two years before writing uh any line of code i think be very specific about what you are trying to build and just choose one solution whatever fits right to your uh type of gameplay which you are building and then try to make a prototype which can stimulate the type of gameplay you want to have as close as possible so like you know sometimes you know

unity is very flexible very open you want if you want to go with dedicated server solutions you can create a headless build uh but your networking is going to run inside the engine so you have to be aware of the limitations of england like you know if unity cannot support more than 100 players as an engineer then you have that hard limit but does your game need more than 100 players no like your game is like it just needs 20 players so that's a good solution in terms of low level networking there are some open source libraries even glenn has written some libraries so just choose one whatever works best whether it's quantum board photon real-time then just implement what you need and see whether it works create a playable if you're building a mobile game i think it's very important that you do all these testings on a mobile device so you create a flavor test in different environments all these tools i think come up with their own testing environments so like you know you can simulate lag you can simply drop and also go out like you know i i always say that developers always keep one build in your pocket if you are in a mall just play there if you are in a

taxi just play there if you are in uber just play your game just play a prototype and see whether it works because you know you have to add then understand if this game goes live on a hand off a player how would they feel if they're feeling right that's the solution for you so i think that's that would be the way to go uh in this sense got it so so i haven't actually you touched on a really interesting point you got to play game you got to play your game and you don't play on the land you're not building yourself right so so the best practice um so it's one of the best practices that i i learned from from the original folks is you know people people not might not be aware of this but when you're playing titanfall 2 single player you're playing a client and server in the same process and they're networking titanfall 2 single player is a networked game huh right because there's this really there's this really bad pattern that i've seen um which is single player works but then

multiplayer doesn't this often comes out for for teams that have a single-player background and then the idea is kind of like yeah the network the network programmer will like they'll network it right right no that doesn't work because because the network programmer is like walking behind gameplay with a little poop scooper networking little bits right and that never works right so the real the real goal here is to empower gameplay to network their own code and then they test it right so there's a what better way to test the networking works is that the single player again doesn't work unless it does right so so that's that's kind of sounds a little crazy but that's how the call of duty folks make sure that they're single player and they're at least on titanfall one and two there was never a situation of like you know this code works in single player but it doesn't work in multiplayer because they're the same thing and if you can do that that's really good and then the next thing is always

play test with simulated latency and packet always right um you never want to fool yourself right so so i i would generally suggest that adding at least a 50 to 75 milliseconds of of of latency would be like kind of a ball back you really don't have to go crazy and add like you know 255 it's it's like an anti-pattern to kind of think like what if legs 500 milliseconds or more it's like yeah you got other problems right like you got other problems um the answer to that is you show a bad performance network icon at the top of the screen that's the answer it's not really your fault um but like the average you know sort of like 10 to 100 milliseconds of latency that you'll you'll kind of see for the good case um there's there's a lot of effort to be done on gameplay systems on design making sure that things feel good under those situations so

one of the most important things that you can do while you're developing a game and creating that code is playing it under real world conditions and for a mobile that might mean playing it on a train while you go under a tunnel right and thinking about that what that means on your game design see some of these hard problems they're actually not solvable from engineering their design problems right like what sort of game should we make on a device that can go into a tunnel and disappear for 30 seconds right right yeah and speaking to that the other thing that we mentioned earlier is server cost right and so i think that's one of the key challenges for like a moba and i think what we've seen is that in high rpu countries like china where you can absorb all of the engagement thus paying for the server operating cost one of the things i really worry about and one of the advantages of quantum chris would be that if you are you know uh more client deterministic then maybe you can reduce your server operating costs as a percentage of that revenue from like

30 to 10 or something like that and so and i think that's been part of the problem with mobas in the west right because the rpu curves for a western audience or other audience is just not high enough to offset the the server operating cost but um maybe that can also lead to this next question that i have for you glenn so talking about the other side so we talked about the netcode approaches you you also mentioned in terms of also kind of trying to solve from the network side could we talk about network next and you know or any like new infrastructure on the network side whether it's 5g or what you're building how will that impact you know sort of the multiplayer experience or netcode so here's here's the sort of critical thing everyone needs to internalize when you're making multiplayer game the internet doesn't care about your game it doesn't care right makes no guarantees kind of says like whatever right um and uh you know

it's not really okay right it's not really okay um you know we all work we all work so hard on making these experiences and and livelihoods and and everything is behind it and obviously we also want to we want our players to have the best experience they can so um i created network next because after like 20 years of doing this i was like kind of pissed off that like no matter what i did from the netcode point of view or the server hosting point of view a significant percentage of players will still have a bad experience from one time and it wouldn't even be the same people each day it was just like this big whack-a-mole thing going on you can't you can't write your neck you can't make your neck code better it doesn't get you know there's nothing that nothing that i a network engineering expert can do to solve this problem because the problem isn't actually my net code the problems the internet right so so i started looking into it and i realized like we started measuring and we started working with partner companies who integrated our sdk and we started getting real data and

we started to find that fairly consistently 10 to 25 of a player base are getting network performance that's like easily fixed like i'm talking like 20 millisecond reductions in latency or ridiculous packet loss that there's no reason for it to occur but it's just that the internet is routing them a particular way and it's not malicious it's just kind of like whatever right like it doesn't make a guarantee but for the sort of sort of applications that we're all creating that's not really okay so so what we do at network next is we have an sdk that integrates in the client and the server and it just watches all of your players and when they're getting a bad route we find another way and um behind all of it is a marketplace where all these private networks are kind of effectively competing to carry your traffic if they have a lower latency route or a route with low packet loss from where your player is to where your server is wherever you host we'll find that every 10 seconds for your player

and and get a good price without having you locked into a single vendor so that's pretty much what we do and the results are like kind of ludicrous um can i just can i just share my screen yeah please like this is what we see and and and these are all players and if i can sort of these all players are getting improvements versus the default public internet routing and it's we it's everywhere but like you know are there bigger improvements in africa and south america yes other bigger improvements in the middle east yes but like and then but then you find weird stuff like at a certain time of day someone in google fiber gets a big latency reduction taking network next to a google data center what right like like it the internet is just says this like weird weather right so like it's just it's just weird but like five or ten percent of players probably getting pretty bad experience at any

point in time no matter where you host your service no matter where they are and it's not the same people from one day to the next right and here are the sort of improvements that we see like we see like uh like this this this is ludicrous right now of course this is the best one on right now okay and they're in a remote island in the pacific sure right but like you know these other uh he's he's happy definitely he's definitely doing better so it's it's it's really gratifying to like to spend so much time working on that code and then kind of realizing you know like okay like i've done everything that i can but but something else needs to happen to make sure that my players get a good experience so that's pretty much and networking i'm really i'm really happy because these people have done all this work to to make their game as good as they can and then they launch

it and then we get to hop on and actually catch the five or ten percent of the players that need real help and make the experience better for them people just don't realize how bad it is out there um see this and they're kind of surprised wait there are people with 100 milliseconds additional latency that you can just fix yeah it happens all the time yeah as a player being the laggy person on a game i i've seen that myself so yeah and it's it's transparent so you just add it to your game right so it's of course it's work but it's just something you you just patch into your game and you have a better experience for whatever ten percent of your players or even five percent yeah well some some of our customers even tune it that half of their players are being accelerated um some some customers with more professional player bases because one of the things

that we do is depending on the type of game right well the the question really is what's what's meaningful meaningful for you what makes sense with your our if it makes sense for your players to save two to three milliseconds we can do it but for many games that's actually not meaningful it might be actually you know like if someone's having really high packet loss or 20 milliseconds or more yeah that's worth it right um so we work with every game customer we work with um and we the first thing we do is we pretty much ask them like what's what improvement is meaningful for you right like what what what is the the range of network performance that is okay and and uh when we can help take something that's not okay and turn it to okay um what percentage of your player base is that is that gonna apply to and that's that's

um it's challenging because people don't really understand how how bad it is for a certain percentage of players in their game it's those people in the forum that are kind of complaining like you know hey friday night at 7 00 p.m like i had like lag for five minutes right or or sometimes we see like you know like for a given data center where they're hosting servers um there's just something going on in europe for three days straight and everyone we've seen it we've seen situations where people just have everyone in europe suddenly gets an extra 200 milliseconds of latency for three days right this doesn't make sense but like this happens like often right so like a lot of people you might be wondering like these people complaining about my game for real probably yes right um because we we have the data that shows that these these events are real they happen um and and you can't really do much about it unless

you use network next uh did you know before building this that it was so successful um from my past experience i kind of had a gut feeling because of i kind of knew there wasn't much more that i could do to make the netcode on titanfall 1 and 2 any better than it was right that we kind of felt that we were getting diminishing returns and on god of war ascension as well like it's like okay the the bandwidth usage is minimal the protocol was very latency tolerant but a significant percentage of players still having a bad experience and the lesson from golovkin was like peer-to-peer is a dumpster fire don't do that right that was that was the learning there so it was really interesting to go to a very client-server dedicated server team and become transition from being like peer-to-peer is amazing you know it's it's not right to like dedicated service client servers

um and see that transition as it occurred and then to realize even with dedicated servers in the best hosting companies in the world um that's not a guarantee of good network performance for every player everywhere in the world all the time right we've seen multiple companies uh multiple customers we work with have like um you know four or five different hosting companies that they use at the same time and they end up just playing a whack-a-mole where like hosting company a has really good performance now hosting company b doesn't what do you do do you migrate your servers around on a daily basis it's crazy so um we've we've sort of seen that like game developers when they hit a certain level of success can get frustrated with server hosting companies but that the server hosting companies aren't really at fault it's sometimes it's not even the player's isp that's at fault

it's the things that are in between if you do a trace route from your client to your server the route that you're taking is sometimes the problem uh and the hosting company the game developer and the isp are not fully in control of that route on the internet and there's not really any guarantee that that route will be the lowest latency lowest packet loss route for you so we hop in there and make it that glenn isn't league of legends is that peer-to-peer and from a netflix i don't have a lot of in insider knowledge okay for that i believe their client server okay i would i would be really surprised if a competitive esport was peer-to-peer for just for security reasons but but um it's a game that's been around for a long time and what what many people don't know is just like there's a reason why a lot of aaa folks don't talk about their net code it's just because they're kind of slightly embarrassed about it you know because you got stuff to do you do hacks and then five years later

you're like that wasn't the best idea right um so maybe there's some peer-to-peer elements in there but i know i know their servers um but uh like there's no there's no perfect net code out there it's just a series of compromises and and hacks all right guys so i have one last question so in terms of like anything emerging as far as trends or technologies that can impact netcode or mobile specifically what's coming up is there anything that we should be aware of or are we pretty much where we are and maybe starting with you chris okay i i try um i think game streaming is becoming a thing again maybe the fourth or the fifth time uh now it's it's coming again every whatever every year so it's interesting to see right so um is this finally happening and then you see the devices are getting

super strong so with um apple building these crazy chips into the phones and graphics cards and everything so you have to balance between these high-end phones which can do pretty much console like rendering and on the other side you have the the countries which have more the low end android phones and you have to balance your game so between those i see more and more visual programming stuff right unity bought this company bold which is similar or the same name as our product but not related of course unreal engine with blueprints i see a trend in gl um so more and more games using the the browser and webgl as a as a host so it's it's coming from from the casual market right so what and facebook and then these casual games are getting very strong so these are the let's say the emerging trends i see or

of course augmented reality but it's not really of of big use yet right and chris i also saw the announcement that i guess via geforce now you'll be able to play fortnite in safari now is is there any from your perspective will you be able to play just as well as you know in a mobile app or regularly or do you see any problems there what you mean was game streaming not with games streaming but you you mentioned stuff going to the browser and kind of webgl and stuff like that is that what you were talking about oh yeah i see that's that's pretty yeah i think it's games particular build for that so snap purchase a couple of companies and then just focusing on on making games just inside snap all based on webgl and 3d and and you have with webrtc you basically have udp in the web browser now so you have all the good stuff and you

have so the browser is becoming an operating system and there's almost no limitations right so it's becoming more and more powerful so it's interesting and the only thing you need is a url and then you're able to play so this is the the viral effect is extreme there it's it's not yet there right it's it's for simple games and facebook and so on are pushing it okay but apple of course wants to control the ecosystem with the app stores and so on right cool avinash what do you think so i think we were talking about trends so one thing i always think about any new trend for example 5g like it will definitely improve the internet speed but for game developers we have to always make sure that if our audience is not hundred percent there in that new trend or new platform then we have to build for whatever is most common right now the same analogy is like you know now we have really good smartphones we have iphone 12

really good smartphones on the right side as well but we always optimize for the medium level or low end devices even today so whether 5g comes or whatever tech comes unless most of our user user base is there we have to like still do the things the way we are doing things now like the same optimization same challenges are going to be there in terms of new trends i recently heard apple's newest new announcement about that chip where you can run your ios games or ios apps on macbook so i think that will broaden up the market more so like if you are building games for iphone now actually macbook users will also be able to play your game so mac itself was not considered a gaming device gaming laptop so far but this kind of opens up a new market so i feel like that is kind of positive for developers got it glenn final thoughts oh i think we're all staying at home and uh you know like a year ago i would have said there's two types of games that i see that there's the location based games

where you play with people around you and uh games where you play at distance and i almost wonder if the first category is going to go bye-bye for for another year or so but i do still think it's it's the case so so from this point of view i think sort of edge compute and the ability to have compute power that's secure close to where your game is is occurring could be very interesting and then for other sorts of games connectivity across distance so regional based esports are a trend that's playing out right now and more and more games are creating things that are more like sports and less like games so i sort of see it splitting into a a casual location based gaming and more casual and more competitive sports-like regional connectivity for player game and of course the best example of that would be regional play for league of legends but i don't know if the anyone's planning people out on the street for for a few

more months unfortunately right all right well guys thank you very much for your time i definitely enjoyed this discussion and for for me it was very very helpful to get all the different perspectives and and thoughts so thank you very much for your time guys and uh unless there's anything else i think that's it thank you thanks for having me [Applause] [Music] cheers [Music]