Tech Infrastructure for Multiplayer PVP - How To Reduce Latency for Competitive Games?

hello everyone and welcome today we are going to be talking about the kind of technical infrastructure required to manage latency for multiplayer pvp games now to speak to these issues we have two superstars first we have zach blitz head of infrastructure and live ops at one of my favorite game companies of all time riot games and glenn fiedler founder and ceo of network next and considered to be one of the world's best netcode guys right right glenn it's morbid my mustache just to set the stage kind of one of the things that we've been seeing increasingly has been the rise of more social and multiplayer and pvp games and i thought we could start with thoughts about the current state of technical infrastructure for truly competitive pvp multiplayer games and in particular we could start with you know how difficult is a challenge to ramp up an infrastructure for competitive pvp

and what are some of the key variables that increase or decrease the difficulty of this problem and maybe we could start with you zach yeah yeah it's it's you know it's it's interesting especially with cloud plural proliferation if you have a knowledgeable distributed systems engineer devops person architect the design and scale up and ongoing maintenance at a smaller studio isn't super heavy as long as you go with the cloud uh i i would not suggest going into the data center business unless you have extremely deep pockets or you need to go where the cloud is not but most of the problems that you're going to face are business or product or game design problems um and and there's generally two main considerations when ramping infra the first is the experience you want players to have and that usually means competitive integrity uh which which is that everyone has a level

playing field and what makes things complex there is how you define a level playing field for valorent we we defined it as uh sub 50 milliseconds globally to eliminate peaker's advantage and that made life extremely difficult for us because of the second consideration which is where your players are and at riot we serve a global ecosystem of players um let's look at europe for a second europe is generally easier to ship latency sensitive games to because it's physically smaller population centers are reasonably close together and it has a pretty robust cloud and metal presence the united states by comparison is enormous physically and population centers are spread way out and to simplify the valorent problem if your data center is in chicago you have a player in akron and a player in miami is the playing field truly level um do you introduce latency for the person in

akron like we would never do that uh just just uh from a values perspective um so do you account for that with matchmaking or do you spend more money so on infra in atlanta so that uh players can play fairly and we we certainly chose the latter um but now you start to see how especially in the united states launching a pvp game is expensive and really difficult uh but the problems are more business product game related uh than they are technological um and i'll give one more example and and you know glenn feel free to riff like we've we've been pretty public about an mmo and that introduces a whole new set of challenges for us like imagine for a second two people about to fight an open world game with no signal to our infrastructure that it's about to happen we've never done a sessionless competitive experience how does pvp combat work when we don't know the location of the participants that are about to fight how do we make

that fair do we just let one person get wrecked because they're in boise idaho versus like terre haute indiana um and and i think the answer to that would certainly be no but these are some of the considerations and challenges that i think make it make it pretty complex right and zach before going on to glenn could you also for the audience that's not familiar with peeker's advantage could you uh go into what what that is and what impact that has on gameplay yeah sure so the classic sps challenge is something called peaker's advantage um and it it you know games like counter strike and its predecessors i think had a lot of it and um basically it's when when a player peaks or looks around a corner first because of the way the internet works they will see their opponent before the person who is standing there um sees them or the so what valoran's thesis really was is if we eliminate peaker's advantage to the best of our ability globally

uh that players will really really want to play a game where that is true and for the most part we've done it you know um but it's it's it's very difficult okay glenn hey cool so let me give a like a maybe a completely different point of view so um i'm i'm a little bit unique in that i struggle both the business and the design and the engineering side and i can maybe give a holistic view and also maybe skewed more towards people who um don't don't necessarily have the resources of riot direct um which is a massive advantage to write and i fully acknowledge so um the first thing i'd really like to say is look the number one thing is is your game fun that's it is the design compelling will people stay around will they have high engagement and will they not turn out that's absolutely number one and if you don't do that nothing else matters you must have a fun game the and then after that here's maybe the next most important thing and i'll think

within the realm of what we're talking about it's it's intelligent selection of where you host servers and trust me you will distribute servers in the united states across uh seattle los angeles chicago you will go to new york you will go to probably virginia miami you'll definitely go to sao paulo brazil in the united states you'll get in in europe you'll definitely be thinking about uk amsterdam frankfurt at minimum middle east is a little tricky south america outside of sao paulo is definitely a little bit tricky uh and then you go to aipac and holy cow everybody get your servers in singapore right because singapore is a big deal for apac region especially for western australia right and eastern australia sydney it's a good location itself but there's something really cool coming up that i'm very excited about is in addition to singapore being a hub there's this company called one code one

qode and they're actually setting up regional server hosting in guam which used to be an internet exchange and that is central latency wise to to a lot of countries including australia japan china etc right so the general trend here is is that you want to try to find servers that kind of satisfy like a like a gravity sink people around them regionally and then you want to try to subdivide that when you find that latency isn't good enough and hey mainland china is even trickier because you can't host servers in there unless you have a game license right so there's a lot of regional complexities in the middle east infrastructure oh boy right um you know it through politics or geopolitical situation um there's not necessarily a lot of great connectivity between countries in that region and sometimes the routes by default that they get to servers even in their own region i see people in dubai going to frankfurt on the way to dubai

so so that's that's what's called a hairpin so so what do we got fun game absolutely number one geographically the distributed servers and i literally just told the locations that you'll need right and then the next thing is matchmaking how are you making sure that people are playing together and and alluding to your point i personally think everyone under 50 milliseconds is having a great time and i think at a pinch you could say less than 100 milliseconds is is essential um but uh you know it's it's like uh it's it's essentially look what's the latency above which where you're feeling that your players are not having a good experience and you can look at the data you can look at the data and you can back it up and you look at retention and engagement goes and you can actually look at the point where people churn out because they're having a bad experience so and matchmaking is the first thing to rule out so it's

fun game excellent server locations really excellent matchmaking that's trading off skill versus the time to find a match versus distance or latency and then only after all of that is solved you can get really strong benefits from riot direct or a competing product well not really competing per se um very different but if you're not riot there's an advantage that you're missing because riot has something that you don't have riot has the ability to say the internet route is terrible for no fault of the player and no fault of the server and say take our private internet that we've built five five or six years of work on this tens of millions of dollars every year i would say closer closer to yeah closer to 10 but yeah this is this is not something that uh i'm going to open even a small aaa studio i'm not going to start by building this i mean for you guys i agree and and honestly

financially it was was kind of crazy to build that initially because nobody did that um and we were just so frustrated playing you know games that we all loved where the internet wasn't optimized for gaming so we we built it in house which i don't think a lot of studios even i don't i don't know if riot today would have made the same decision uh to go build it um that said though i do think cloud proliferation is different today than it was then like you can stand up some sort of aws presence or gcp or azure presence in almost every spot that you just listed where uh 10 years ago when we built that we couldn't right and that's that's also part of the reason why we built that so i mean i think glenn and i i think are largely saying the same thing it's like what experience you want players to have and like where are your players and um you know those population centers put your servers there and you can generally get to a to a good spot in terms of latency and

fairness right yeah i i've actually got an additional point i could make to to back this up actually some data i can share and this is from a game launched in 2021 yeah um on bare metal in these very locations we found that 83 of players on the public internet even to the best bare metal and cloud data centers in 2021 yeah have high latency defined as greater than 50 milliseconds or a packet lost greater than one percent at some point every week that so so i would i would sort of step up and say it's really great that the cloud offerings are better it's really great that you can open up servers you don't have to build your own data centers necessarily but there's something afoot still which is the vgp the default internet routing protocol man there's a perverse incentive out there they take the cheapest route not the best one for your players and that's why honestly i think every game needs something like riot direct because that's what fixes that right it's it's interesting because

there's a bunch of businesses that have tried to open up in that space and they just haven't generally found traction i don't know if that's because they're aimed at aaa studios as opposed to the little guy but uh it's it's been yeah it's it's interesting to have watched that space i think over the last few years it's i i would say too that you know the question on i'm like what's wrong with uh what's wrong with that working i think that's that's a really good question i'll settle i'll settle with this statement um the public internet treats your gaming the same as someone reading their email and that's why 83 of people every week get high latency or packet loss and we have hard data tying that same threshold of latency and packet loss to three times reduced engagement and three times increased churn okay so i think there's an opportunity out there for people making games

when they have a fun game when they've selected good regions for server hosting and when they're comfortable their matchmaking is doing the best job and they've tuned it specific to their game design there is as a number four and it's not number one right but as in number four there's definitely benefit to okay but there's like one percent of my players at any time or get maybe even two to five percent depending on your game are getting an experience correlated with three times lower engagement and three times higher churn fix that and if someone comes along and offers something that's cost effective that fixes that i think that's a winner game buyers are not going to go and build right there's not going to be 50 right directs for everything it's not going to happen right but there's an opportunity for something that's really cost effective that targets honestly like price sensitive game developers making money off a live service product this is something that has to be turnkey and reasonably priced and that's what i'm trying to create with network next

got it so maybe we could talk about that topic around like the smaller developer is it realistic for a small developer so i'm with a small developer and when it comes to trying to develop a game that is let's say sub 50 milliseconds or sub 100 milliseconds how should we be thinking about trying to create a competitive game is the kind of infrastructure because we don't have things like riot direct and all the resources of a big company like a riot what how should we be thinking about developing our infrastructure uh what do you guys think can i take this one initially and i'll pick it to you zach um there's an amazing amount of really good stuff out there in google cloud aws and azure and as a small indie studio focus on making the game fun right focused on making the game fun play tested and hosted in google cloud the premium networking for group is actually really really good we find it to be the best out of the three clouds and uh you don't need to go to bare metal to

be a serious game you can get really good performance out of cloud and the network performance of the cloud is actually in general superior to the bare metal performance that you'll get by default with with the medium to your hosting company so i would suggest if you're a sub 10k ccu focus just just get it into a cloud pay the transit fee pay the cents for a gig which is higher than you'd like and know that you're getting better than the full internet networking um in in a hosting company like google cloud that's my recommendation yeah i i the the interesting thing about the question too is like for us we're trying to be more industry standard we we built a lot of custom things to roll out league of legends um but but things that we use that are pretty ubiquitous i mean val launched uh i would say 60 or 70 on aws um and that's compute databases uh you know manage kubernetes route 53

uh on you know we use terraform we use docker we use cloudflare we use new relic we use github i mean all of these things right are are cheap or free uh depending on the tier that you're using them at um uh you know things things you can't do that we do like physical data centers like i would probably not do that and all that entails from a management perspective inclusive of like the software to run it and people that's a lot um and then riot direct obviously we've spoken about that uh a bit here um but you know i i would say that like what glenn advised i mean aws has has latency sensitive products as well um that that they can help with most of the clouds have offerings that that can get you there and you know we we manage our our stack using pretty standard tools and and trying to get more standard every day right and then just speaking about the the network infrastructure i know it's been steadily improving over the years and we've got

things coming like whether it's starlink or what glenn's working on with network next but is there anything in particular that we should be looking out for or anything any other thing that we can do is us whether it's as a small developer a bigger developer to address some of the challenges with on the network side i mean i think the current state of of the iosp ecosystem regarding speed and stability it's like you know speed they generally send traffic the cheapest route for them glenn's kind of covered this it hurts competitive games uh and it's not a technical problem it's a problem of incentive for a network that has no incentive other than to treat your your traffic as if it but a route they could they could just be like 50 of traffic goes left and 50 goes right right depending on the hash of your ip and your port and the distance you could have from one game to the next plus or minus 50 or 100 milliseconds

just because they load balance you down route 80 instead of route b and to them an additional 50 or 100 milliseconds is okay yeah right this is the internet today everybody and and i think the the problem that glenn's company is is solving is a real one like i you know i i have this theory that in the next five to ten years there's going to be a big split in this country between like the boomers and probably gen x and younger um because the boomers aren't don't care about this routing thing right they get their 50 to 100 milliseconds they can watch you know whatever videos they want on youtube or go on facebook it really doesn't affect them i've already i've already got half a second response time when i press a button anyway yeah yeah but there's going to be games that come out or ar or vr experience yes you're going to throw up you're going to throw up because yeah because of a network glitch yeah and people will be cleaning up puke and yeah we as a country are not going to be able to participate

in those things the same way as asian or european countries are because our internets are slower and our isps are not getting less monopolistic um and that's going to create a generational divide in this country um and a potential political nightmare if not solved i hope things like starlink can bridge the gap but but i i think it's going to become an absolute problem uh for for for our country um and i think one that people outside of working at the internet in in video games don't i think really understand so let me let me sort of step up here there's there's actually really the starlink thing leads on to a really interesting sort of like what what is really going on right and here here's what's going on there's the last mile and the last mile is your isp from they have a data center called the head end so it's a data center somewhere probably one wilshire or whatever in la and there's there's points where they and they all connect there and uh vgp is a thing that kind of works out how it gets from the head end to wherever it's going on the it's like

you're on comcast you're not on comcast's internet you're really only comcast to the nearest head end and then it goes ip and then bgp is going to route you stupid right it's going to route you like hey he's checking his email now right she's she's browsing the web no no no you're playing valerian well in this case you guys are taking it from the head end and you're saying no internet don't do that go on our private network from a to b okay and network next is kind of the same thing now we can't currently fix the last mile but the middle mile from la to chicago man a lot of stuff can go wrong between and that's that's a that's a that's a city to another city and i can tell you just the perverse incentives um the things that i see in the united states and europe at scale with games we're working with i mean i feel like rutgers are here like i've seen things you people wouldn't believe like attack ships on fire like this it's absolutely crazy out there some people just get randomly 300

milliseconds for no good reason and they're in la right it's nuts sometimes people get an additional 300 milliseconds for four days because a route went down for an isp and they're rating them around the country the wrong way and you can't as a game developer call somebody up and say hey comcast can you my game's on fire can you so i totally understand why riot direct was created it solves a real problem and uh i don't see the solution for this coming from the network itself um i see um i see a way with my company but you know there's no guarantee that we succeed but the thing that i'm really hoping to solve first is the middle mile so that we can kind of say look guys hey hey this traffic is actually different it has different characteristics to reading email please treat it differently and let me create an incentive for you so that you're paid when you do and that's what i'm trying to do now the last small stuff wow starlink breaking the last mile monopoly it's brilliant it's really smart it's it's

breaking that fiber optic or uh dsl last mile and starlink is not you're not going up and then to the game server you're going uh i'm in i'm in upstate new york right now i have starlink and i use it as a backup i go up to starlink and then i go down to nyc or secaucus right and that's that's the hulk that the satellite does and then the rest is regular wired stuff now eventually starlink is going to have um interest satellite links so i can go up and then go around the world potentially full speed of light in a vacuum people don't know this but your two-thirds speed of light inside a fiber optic cable due to refraction so even if you've got a direct line of sight from one point to another theoretically if the difference between two thirds and full speed of light is less than the up down to the satellite with the low earth orbit satellites starlink might get you faster japan to new york times when you're playing i'm really excited about that um what do you think zack yeah i mean

especially the the implications it has for things like an mmo um you know sessionless games with like you know thousands of people in combat scenarios i mean i mean i just remember playing wow you know 10 years ago and you know the instances basically crashing as soon as as soon as the fight would start because you know of of internet issues or servers being overloaded or whatever um i think i think it it starts to um i think open up another arms race for what pvp means uh which which could be which could be really really exciting so you see higher player accounts and uh more engagement across distance being opened up by increases in both bandwidth but also uh better routing across distance to perhaps a central location yeah i think i think especially i i think a lot obviously about our player base when when you know it's just the frame of reference i have i i would say most of them are interested in being the best in the world or to know or to know that they're among

the best in the world at whatever they're doing yeah and so people and people people talk to me all the time about this they're like i don't want to play in the little leagues i don't want to play in the minor leagues i want to play in the major league i want to play and that's why when you guys had league and you did everybody in the united states going to chicago that was so smart that's really smart it was it's a super controversial decision i think what that that our pros would tell you you know they still you know are pretty open about not liking um just because the pros are based out of la but the reality was that was the fairest that was the fairest point for everybody um and so you know that's that's what we're tr we're trying to pro you know provide competitive global esports um and i think i think things like starlink you know et cetera provide a more level playing field globally um you know potentially potentially you know getting rid of regions uh which which is something that's been a bit of an albatross for us because

it's totally like north america south america aipac middle east europe i mean it's very much regional right now isn't it yes yes and obviously the language barriers to a certain extent like do do help to create that but i mean you know in games like valorent that have pretty specific sophisticated ping systems in terms of like what what you can uh signal to players like you know i mean there there are ways for us to play without uh having to speak the same language so it's it's interesting it's if you need to do the same central hosting thing for valerian or do you have more locations for servers and valorem because it is a lot with that peaker's advantage it is a lot more latency sensitive i think i think if we had league to do over again we would have more server locations than we do i think i think the reality is is you know we and you know but but for val yes we certainly have highly distributed uh server locations just based on the nature

you know of the game and the way that uh population centers are distributed speed of light yeah all that you know all that stuff but yeah i think i think yeah it's it's been and actually i did a thing for for uh re invent this year on aws outpost that was a critical uh a bit you know thing that helped us a product that helped us roll out in time especially because this the speed of metal in specific locations was vital to uh launching that game at the millisecond target that we wanted that's awesome so you actually you actually found that your your game was so latency sensitive that you had to work with a specific bare metal option with aws to get the perf you needed yeah it it so yeah well it was also a supply chain problem i mean there's there were a couple things that weren't like was covered last year and all that exactly it was like who am i betting on am i betting on riot supply chain to get servers and networks yeah and and i you know i opened the

checkbook and we paid the money but you know it's it's that is that is exactly you know that was the calculus right yeah so very smart cool yeah um but it it i mean i think also like the ease of ramping up on that versus having to you know install an os on every box and you know there's there's a lot that goes into spinning up your own infra just from a just from a time perspective you know uh rack and stack you know the whole nine so um you know i'm sure there's vendors out there who are gonna listen to this and be like i could have done it in 45 seconds for you but you know i think we we have a very specific sku that we installed guys maybe we could talk a little bit more about like you know you guys have been recommending using cloud-based uh solutions but for some you know some of these bigger companies still do have on-premise based infrastructure and so when you think about you know what are the if we can dive a little bit more deeply into

what the advantages are or are are there advantages of still having on-premise-based stuff and then you know what does that what does that mean or what what specifically does cloud give you versus on-prem i actually don't know any companies that i work with at least who actually manage their own data centers except perhaps valve and riot and maybe ea yeah i think some of the bigger console guys do i'll give i'll give i'll give a little bit like when i think about cloud providers right yeah like for still stability they're great if if you've written some business logic to fail over gracefully um because they're going to take a note down you don't expect at some point um like you know their business is really stability so long as you're you're kind of smart about that and that's their whole pitch it's like we can we can run infra more reliably than you and uh cheaper than you can but and i'll tell a story here they don't care about latency at all and

and and i have a story about that and i'm not going to mention names um but in the lead up to val we wanted to share some learnings about some of the latency studies we've done and some latency reduction we wanted to do with a cloud provider to make their offering more attractive and to hopefully offload some work from us and they kind of laughed at us and were like this work is completely ungeneralizable to the rest of our customer base uh and none of our customers care about latency they care about stability and i know i know i know a couple i know at least a couple gaming companies that that are specific to this cloud provider who absolutely care uh about latency um but i think latency is largely still a game maker's problem and so when we talk about the decision on-prem versus cloud in specific areas of the world right especially ones that don't have a ton of cloud presence

metal or some metal vari metal hybrid like an outpost really is is your option if you want to launch a highly latency sensitive game can i give a shout out to some companies that i've worked with with nate with network next we've we've worked with stackpad zen layer limelight we work with g-core around russia uh one code and apac and uh data datapacket.com really really good uh servers.com 100tb total server solutions performance there's a whole bunch of really good bare metal but here's the challenge not everybody has not all of them have all the locations you need you'll have to have a a a uh heterogeneous infrastructure where some are in this group like g-core in russia really good right zen layer amazing in apac and asia and obviously all of these are really good everywhere else um and you know like there are really good

companies that will host servers for you and and have good connect and then there are some other ones that are really cheap right so let me let me sort of take this and like i've i've worked out like a really weird sort of thing it's like man the cost of service can be pretty high even with bare metal and with cloud what happens is the the monthly recurring cost is maybe a little lower if you do tend to at the scale of most people get a little bit whacked by the egress bandwidth from that cloud they they hit you on a cent per gig on the on the cloud on the bandwidth going out which is tricky for games because we're paying a lot we think for the cpu and the egress is usually down to this down to the player is usually more than up right so so what happens is is that when you're primarily considering the bare metal and the cloud decision the simple fact is look the bare metal is going to be cheaper because you're not you you can buy that bare metal server and has a certain amount of bandwidth

of which you'll certainly stay below built into the price with the typical medium and small scale right like hundred thousand big ccu or roundabouts very common size when you're in cloud um you you will find that like four five six eight ten twelve cents per gig adds up very quickly so one way to look at network next is you can go and host in really cheap bare metal and only for the one percent of players that get a massive improvement that's relevant for your game design we can get you the performance of the absolute best servers in the locations where you are yeah maybe could i ask you guys a potentially controversial question about prem versus cloud do you guys think that for some of these bigger console companies that have been running on-prem infrastructure for a long time that to some degree do you think it might be a political decision not to move to cloud to retain jobs and things of that nature or is it generally just a technology

choice um that's a good question obviously like i don't think the general the general like sales pitch from the cloud provider is always we're gonna save you money we're gonna save you money we're gonna save you time and we're gonna save you a headache and we're gonna save you maintenance the reality is and we're we're in the you know full disclosure we're in the middle of a cloud conversion ourselves right the reality is you don't save money it's about it's about at cost you don't you don't always save labor because the work changes right and so i think i think the answer to that is probably no i think i think like executives that are you know who sit at my seat at other companies are generally really good at telling the truth about these things which is that look the cloud provider is going to come

in and they're going to give us you know potentially millions of dollars of training credits and we're going to try and take all the staff who really only know on-prem and try to convert them into devops engineers be and that's in the best interests of both the engineer and and their future career as well as the companies um because most of these people want to retain them especially if they've been at you know at the company for a long time and have you know good performance and all that stuff so i don't i don't think it's actually that controversial of a change internally i do think yes there are people who are like look i've done metal my whole career as with any change right in any industry i've done this thing my whole career i don't want to change like i i think there will always be those people but i don't think it's like a political decision i think i think most of the time when they look at it i think it's more you know they don't know where the skeletons are buried and like changing out the physical infrastructure that they have is going to take so long to do or and

and like they don't know what's going to break when they do it and you know it's been sitting there i i i work with some people from disney who are like some of the some of the some of the rides still run on like floppy disks you know like and that's true like some of the disney park like rides that you love run on floppy disks and like the idea that like they'll ever change that out like that once the ride is gone they'll change right but they'll never like they're never gonna like try to figure out what you know some dude was thinking 20 years ago who's been you know gone from the company like this was was figuring in the initial setup i think that's usually the main thing that discourages companies from doing it it's not like a cultural revolution more of a stability issue is what you're saying 100 yeah like i can add a i can add an interesting take and also i can talk a little bit about the unders our own decision at network next right so let me let me start by saying look there's really two things you have to think about where are you hosting your backend and

where are a host where are you hosting your game servers and there actually could be two different places so so for network next our backend is written in golang and runs natively in google cloud and out of my cold dead hands right cold dead hands like i i love google cloud um we use bigquery pub sub the golang stuff that we do runs native we don't use kubernetes or docker or any maybe that's controversial right um but um we just run native goaling instances as migs on google cloud and auto scale and i tell you what we are done we are done right and we have a somewhat micro services based architecture for horizontal scaling and look it works great and i would never i would never advise to anybody you should build your own bare metal on-prem for a for for a non-game server component my recommendation is that when you're hosting your game servers um and someone else is running the data

centers for you and that's their business you can get really good price performance from bare metal and uh you know network next can help bridge the the absolute best connectivity some transform cloud sometimes from other bare metal providers but there's there's a cost decision to be made between the the cents per gig egress from cloud primarily for game servers versus the bare metal bandwidth costs which is typically much lower so um but for the love of god run your back end in a cloud aws or google or azure i guess um but but man i love google um zac you're you're an aws guy we we run both aws at google okay cool so yeah but i i think i think um did you go to pick one though now we're getting controversial now you're going to give me trouble you're going to give me trouble with something so yeah team google over here

so in terms of the next topic i thought we could talk about cost and basically you know there are some game types that are very expensive with respect to net code server operating costs that can potentially eat up a lot of margin as you're operating these games and so in some situations they can turn some games that you know may have been fairly popular from a user demand perspective but economically unviable you know it could just be like games that have very low monetization on a daily basis but you know your cost of operating could be higher now i've heard that things are getting better where we don't have to be as careful like in on some some games in the past we've even had to limit the number of events that we instrument to like you know let's say 10 15 events per dow or something like that and and you know some console games will instrument a thousand events per day so i was wondering if you guys could talk about

when you think about cost relative to the the type of game that you're operating how do you guys think about cost for for the games yeah i can i can go really macro and i'm sure glenn glenn can go a little bit ma micro so for us we always start with our core principles which is like we want players to have the best experience possible the game needs to be fair and globally available uh and then we optimize cost around that right like very very similar to what glenn's been saying originally like build the fun game like make sure players really like it and then and then you start to optimize um this is now this story is from four years ago okay so this is like long before we even started the initial rollout of valor like when we looked at the the cost modeling we looked at three global millisecond latency targets um and ultimately we landed on the middle uh of costs and the curve was pretty dramatic the the cheapest option was extremely riot friendly the middle option was extremely player

friendly and the last option was not really achievable from a cost perspective or achievable based on on internet in some areas of the world and the amount of effort and we had a few people in on this meme in the office but it was like we just can't build a data center in the center of the amazon rainforest uh and that players some player would be upset at some point that we that we hadn't done it but but play it out for a second right like let's say ballard's margins are great right are they great enough to build a data center in the amazon rainforest regardless of your ethics you'd have to get permits to destroy the rainforest you'd you'd have to pay for construction you'd have to run fiber across a good chunk of brazil you'd have to cool it in the middle of the jungle you'd have to buy servers and import them hire people to run it maintain it for years maybe long after the game is relevant what's the exit strategy do you sell it like how do you consider that do we even have staff like

currently who who could design this and then you have covert come in too giving you a nice wrench in the gears right your your margin would have been gone in the first 10 seconds of what i discussed um and so we we spend a ton of riots money every year whether it's in the initial like game design or launch or in our day-to-day lives with a game like league we're constantly thinking about how we help the business save money and give that money back to players in terms of new games new game modes new cinematics new entertainments experience etc if we save a million dollars that's a whole new game dev team um and that's a huge win for players that's generally kind of the way that we think about it so we spend a lot of time you know um kind of underneath the covers trying to optimize um optimize for four players but that's that's probably a little different of an answer uh right that i think you were going for exactly to that point that you just mentioned though like to the point that if you're able to save money that value can accrue

back to the players and so in terms of that concept like whether you're taking some functionality from the server and shifting it to client or things like that what are some of the things that you guys think about to try and save save cost so one of the one of the big ones right now is is we're cleaning up from the valorent launch where we we'd split some routes uh some network routes with it with aws in in certain parts of the world where they're preferring aws versus uh riot direct obviously we we have a huge cost advantage if those if that traffic is moved to aws uh sorry right direct because we're already paying for it right in bulk and we're paying for it in volume and so one of the major things right now is we're trying to move that traffic back over to riot direct since we're not in launch mode but that gets tricky because every time we have an event um in in in valorent and you know there's been a couple this summer that have that have done really well in terms of player counts uh those those sorts of

things like drive demand so it's like okay so when's the right time to actually shift the traffic over so it's a lot of considerations like that um because we don't we also don't want players have a negative experience like hey we we tried to cut this route over to save money in the middle of like an event and now like you know we we have we have a two-hour outage like that's that's an absolute mess for players and something we would never want to incur so though that that's that's something we're doing even right now okay i gotta say like look like zac you hit it you hit it right there it's there's there's this really really weird kind of and i'm not talking about any specific game especially not riot but like when you look at a lot of sort of the the scarier fans that it's like you know devs with their terrible net code and then their broken servers and you know like everything is and it's it's it's like really it's really not true like these are these are game development teams who are packed players themselves they're super passionate about um the games that they work on

they i mean a good example sammy duke uh lead online engineer on apex legends like that guy engages with fans on twitter and i think i think maybe four out of five times like there's like nasty replies and he's just like hey guys i'm helping how you doing and what's going on is the devs are not pinching panics those devs are not buying cheap servers they're not like they're they're spending a lot of money on servers and they're putting the players experience first because if the players aren't having fun they've got nothing they've got absolutely nothing but but where it gets really challenging is when people hack games or when people ddos games um what happens is that it drives costs up for game developers and they have to spend more money on infrastructure and less less money on making new items new game modes um new events and it's just a really and cheating as well any cheap i mean all the effort put into any cheap could could make a billion games as far

as i'm concerned any cheat is even billion billion dollar industry like both both on the positive and the negative side it is it is yeah it it's a thing people have opinions and that's cool but but it's like the one thing i want to do is as as a dev myself i want to stand up for the devs it's like these these are not people buying shitty cheap servers these are people investing millions tens hundreds of millions to make your game as good as they can and and sometimes the internet just isn't giving you the best route and honestly it's probably not even the fault of your isp it's not it's definitely not the fault of the dev yeah and i don't blame the devs the hidden war here is like the dev versus the internet right like it's it's and and i think a lot of people don't see that like we we had i think uh in latam a couple weeks ago like this fiber cut right where it's you know and the game's down game's out for 12 hours luckily that doesn't happen in the united states as frequently probably as it used to

and like they're you know we're we're getting killed on twitter you know our tests are getting killed on twitter what's amazing is that you guys do it in your own dark fiber for god's sake oh yeah you know how much money that is i know but they don't you know but all the player understands is hey my my you know i can't play the game that i really want to play and i got my friends together and we were supposed to do this and i i can empathize with the position but but to go once we always strive to do better but like sometimes the reaction can be a little kind of like you just gotta okay right yeah yeah yeah yeah oh i was just about to ask so for like i i know at the bigger game studios there they have full anti-cheat teams and things of that nature for a smaller game studio how how would you recommend that people address staffing or focus on anti-cheat security things like that you should do a client server with a

server authoritative networking with udp yeah preferably with someone else's engine and don't worry about it that's a problem you want to have yeah i mean you need a scale to hit that that you will not get until you've succeeded and once you've succeeded go go for an off-the-shelf solution and then on massive success do it right yeah yeah pay pay somebody to do it got it okay don't don't worry about ddos don't worry about but but do do the best practices don't don't do a client trusted network model in 2021 most cloud most cloud providers have some kind of you know ddos solution that for a small dev is totally like totally okay okay and final question for me but when you guys think about let's say opening up new game server regions how do you guys think about that is do you have like minimum ccu requirements or you know when when is it time to open up the amazon exactly yeah how do you guys think about the

players how many players are there and what's the improvement that can be provided yeah i think i think that's it i i think generally what we see the way we do it is we will see people playing our games comp with completely horrible experiences like i i think the reason yeah the reason we opened up uh middle east which is i think our our latest one and i think we're gonna we will go to south africa at some point uh as well which we're super excited about i didn't mention it because africa is this this just there's just so little there in terms of infrastructure and the players and i mean it's almost like there's like a little bit on the top and then a little bit on the bottom there's a massive open like there's a blue ocean there in africa for someone to help improve connectivity and everyone playing in that that continent can be massively improved that's a that's going to be a huge mobile region once it's uh once it's yeah once there's infrastructure there i i would say that i would say you know for us we were watching players

in in the middle east uh basically struggle playing valorent uh in europe and we were kind of like this is unacceptable you know we we need to we need to put servers uh in a place now we now picking locations within you know the middle east is super tough because all the countries uh do not play nicely together in terms of the internet exchanges which is really difficult uh and additionally there's no real center like there's not like a chicago right uh and it's big like we talk about we think of the middle east we we probably think a lot about like israel egypt you know jordan and not necessarily the entire arabian peninsula uh and and and things that go a little bit east it's it's it's incredibly large uh geographically almost the size of europe so like trying to figure out where we put it within um you know the service within the middle east is

is is really challenging and it's almost as geopolitical as it is uh like necessarily like latency based um and and and so a lot of these you know to answer your question joseph are like kind of you know region by region uh thoughts you kind of you lead with like hey people are gonna play the game if we put it there great we'll put it there but then like how do we kind of deconstruct it and make it make it really make it a really good experience and most of these things are a work in progress i mean there are i want to extend on zach's point here it's like people think that there's a relationship between distance and latency in all cases that is not true like i'll give you a very concrete example um uh i have i have many korean friends and i have many japanese friends and because of this i know that if you're if you're if you're a united states tourist in in seoul you're not gonna you're not gonna make comments about how similar

korean culture is to japanese culture that is those are two completely distinct and different cultures although they have some similarities historically and i can tell you that the two countries are actually quite different and also don't always get along and there's a really horrible shared history that that the koreans are you know like there's there's some reasons why the koreans are not happy um going back to the second world war right and earlier so is it somewhat surprising that the route between seoul and tokyo is not always the fastest one it's political now it's no one's out there going i'll just no no it's it's just kind of like no one has the incentive and we see other things like if you're hosting google cloud and you're trying to go cross-cloud to aws like you're not always going to get the fastest route between the two because who really cares about that in fact maybe they don't want it i'm not saying that there's an intention here but there's sort of like the internet is one hundred thousand different ascents as of 2021 and they don't all like each

other or or put to put it more politically have really strong incentives to ensure that they have the best connectivity between their networks right so sometimes the path is a little weird and then with a hundred thousand cooks in the kitchen sometimes somebody in dubai going to a server in dubai goes through frankfurt because that's the closest exchange we actually have deployments in middle east and we're often able to fix all those middle east to other middle east regions like we have little mini pops in every city and in every city we have 10 or 20 different providers with different connectivity and we can steer through those points as we choose so so that's sort of like the magic of what we do um and it really helps in these geopolitical um in these sort of geopolitical areas where you know israel and egypt might not have the best connectivity between each other and certainly not to iran right and it's no fault of the people of iran or the people of egypt or the people of israel but and and i'm pretty sure that everyone just wants great connectivity

as people um but um the internet does not have a straight line path physically between two points and nor does it always have the best drought picked between two points on the planet it has its own topology and that's influenced by geopolitics the geopolitics and often the route is not what you think it should be so sometimes just putting a data center in the middle east won't actually get you what you need because you know you have all of the political stuff in the middle east in addition like you know you can deploy a lot of infrastructure to different points in the middle east but still those different points won't necessarily talk to each other because of geopolitical differences okay well guys i think we're out of time but i really want to thank both of you for this really in-depth discussion a lot of incredible information but in closing do you guys have any final thoughts or recommendations for the audience and how can people get in touch with you

maybe starting with you zach yeah sure um i don't know i think i think this is one of the most fun areas of game development today because i do i do think there's a lot of innovation happening in the space we've talked about ride direct we've talked about ddos we've talked about infra i think um i think the cloud versus metal debate is going to continue i think i think all of these things especially as metal provider um kind of metalcolo providers uh step up their their managed service offerings i think i think things like that are going to become super cool uh over the next 10 years so just have fun with it have fun making games i'm i'm at riot z blitz on on twitter uh i just started that because people were like uh players may want to get in touch with you i have like 50 followers so i think like 20 tweets so uh but but i'm there and i'm always happy to answer whatever questions you have um and if i can't answer it uh i'll i'll find somebody who who's a pretty good expert who who can help so but it's been a pleasure joseph

thanks for having me on man yeah thanks cool good so anyone who's interested in uh latency ddos server hosting questions what's it like i'm launching my game or i've already launched a game i'm concerned about some of this stuff what can i do even if it's just you know like just general advice what are you seeing with other games just reach out to glenn2enz networknext.com and i'll help you out awesome all right guys well there you have it zach blitz glenn fiedler two world-class experts thank you so much for your time until next time everybody bye

uh [Music] you