When the Shit Hits the (RGB) Fan: What Can Go Wrong When Making Multiplayer Games

[Music] [Music] we have um matia from edge gap now talking about why he trailed in the previous panel when she hits the fan it's a good a good sweary title always like that so i'll let you let you talk about uh about that thank you good morning everyone so um what can go wrong when launching in multiplayer games or aka what happened when the hits the fan what you're about to hear was here before in the meetings maybe with some of you um any resemblance is purely coincidental

my name is matthew dupree i'm the ceo and founder of edgegap we're gaming gaming backend providers we provide services and platforms to for one-man army studios up to aaa's and with tens of thousands of employees i come from the software industry i've worked in the infrastructure ops and telecom for 20 plus years and i love driving racing cars as you'll see so if you look at the journey of a game developers or a team of game developers you initially start with energy at 100 your team is built of great players who are experiencing all sorts of the game development cycles your concept is awesome right you're funded by a very prestigious publishers so you have money in the bank your prototype is impressing everybody

including players which are telling you that they want more of your games uh you're shooting for the moon really so um but what are the challenges that are gonna lies ahead let's take a look so as a as a racing driver i like to to compare what what my fun time is to what a game development is let's look at the four different turning points where you should be on the lookout for problems so the first one is around development cycle so when you're building your game what are the hiccups that you should look for the second one will be around before launching your game in the world just right before the third one will be when you launch the game and the fourth one is right after it what can you do so as as driving a car every sharp terms require prep so we're going to be checking for that so these are the things that we may or may not have here during development cycles so turn one

or we have a nightly build but the qe team only gets a release every two weeks or ci cd pipeline is more like a spaghetti code or a bunch of script that was written by the interns last month replacing that third-party tool would take months of development and now they're trying to ask us for more money right um let's let's make everything ourselves so we're going to code our net code or game engine and the back end that's going to take a few weeks for philip to build let's save some money on this great open source project right i've heard that many many times in the past why do i have to wait um five hours for uh to test this new new features that you've implemented these new bug fixes oh we're not going to look at cross play at this moment it's not required i think we're going to look at that in a few months when the game is launched so avoiding those crashes

you need to ask yourself if you guys really follow the devops methodology properly right are you really following that or do you stop at releasing a new payload or do you really push toward and deploy it in the field get the qa get the players to to play with the thing and and mess around with it are you trying to reinvent the wheel or many source main tools out there many projects many companies like us are you really trying to reinvent the wheel could you use automation to make your team and your development cycle more efficient are some of the tools that you've written internally job security for people and talking to some of the tech director uh in the room or that i may or may not have met in the past have you selected technology agnostic vendors so that you can replace them if if needs be and your team can remain nimble is this open source project supported is

there a company i can pay is there a solid community will developer behind it drop the project in six months and decide to do something else and we've seen that a few times in the past how could we support cross-play right now that's a good important one so these were here before launch date we picked three regions in the public cloud but uh we're again we're launching our game worldwide that's a given for uh a big crash in latency uh we're good with gdpr it's not important and i think we're fine um we think we're going to have this amount of ccu let's pick this and 10 person buffer where we should be good to go and this happens a lot i mean just google game launch crashes and and that's that's the most common one so um hey hey mr um i'm making this this

deal with this vendor and to get a good rates and credits uh we need to sign this 24 months contract um we have this vendor who will divert the traffic for ddos but they're unable to let us know how much it's going to cost but they're saying it's free at the beginning so we built our own data center we don't need anybody else i've heard that many times from triple a's or game can scale we tested the game with a thousand players sure we've used spot instances we're gonna save so much money so let's try to avoid the crashes at turn number two so you don't know the location of the players you don't know where they're going to be coming from you don't know how many of them there's going to be so be flexible in how you approached it i was hearing the the lady talking about localization before the panel and

she was right you don't really know whether your game will be popular in south america in asia be flexible and be open about that have a plan if your k your game is a success seems trivial seems silly you're probably wondering what the heck is he saying that the thing is we've seen that many in many many cases i mean how many times have we seen launches where the game got a hit overnight and the day before it crashed have a plan for that you need to be ready in case it happens you need to be mindful about your budget right pick everything as pay as you go as much as possible long-term contract are our hard sell um and if if you're not make if you're not managing the money in the studio um ask the person who does because it's the most important one your game will hopefully be a success but it may not right and it what if it happens what are you going to do about it

virtual machines at those scaling is 2010 tech um this is wasted money don't waste money on underused servers and we'll talk about that in a few minutes if you value your players stay away from spot instances we've heard our story about studios being forced to kick players off because they had to leave alone the the resources um use the capital wisely so solutions like ours um we support multi-infrastructure whether it's on-premise or the cloud or edge computing larger studios sometimes have done prior investment on existing other partners vendors infrastructures you can leverage that so scene on launch day um i need a coffee so we've been awake for 36 hours straight the game is half

working but uh we need to keep pushing i've heard that a few times in fact we have an employee who left the studio because it was the case and last launch everything is fine but twitter and steam says differently i have streamers on twitch complaining about my game but yeah no i don't think we have any problem queuing time is in the hours so i i think this one is is i guess it's debatable because riot and blizzard seems to be thinking this one is fine um we're getting ddosed um yeah it's gonna happen we don't have any visibility on what the players are our experience is we know what they're doing in the game but we think it's working well for them from an experience perspective so we're going to need 100 servers but we've reserved the thousand i have the cfo on line number two

oh we needed a thousand servers but we've only reserved a hundred i have the tech director on line number three so let's try to avoid those crashes you need to get real-time visibility tools get as many logs as possible not just from your game itself but from the flows what are the ins and outs of your games what are those flows and besides knowing that something is is either working or not working do you understand what's working is it working well and how well is it working don't assume that your cloud provider has unlimited scalability it's not it's not real it's not true they have a limited amount of servers um we've seen cases where a whole dc in sweden for example went down and resiliency is also another very hot topic because i'm i'm sure that uh you've heard about um aws outages in the last few months right

have a 24 support plan 24 7 support plan for your game i come from the telecom space up until five years ago i was purely working in the telecom industry and i'm used to what we call the nine the five nine availability i don't know if you remember when was the last time your wireless phone stopped working you probably don't and that's because those wireless carriers are bound to giving you the service their their emergency service what if you were treating that live game that you want to make as a service what if you were planning at to support it 24 7. we've seen a lot of cases where developers told themselves that oh if my game goes down at 2am in the morning i'll just fix it up when i wake up we've seen that many times i've seen a game down for more than a day because the guy was out on pto he didn't give a about the rest of the team was oh it's it's it's john's job

poor experience the amount of churn that they had that day was probably really really high matchmaking needs to be flexible you need to be able to implement different modes in your game very quickly not just one because you don't know what's going to work pick a ddos mitigation solutions which will work but also will be fairly cheap in price because they can be they can get pretty expensive so no range of ip different vendors pick a low server density so you don't want to have 50 000 players stuck in two physical servers that's a it's a recipe for a big mess use an orchestrator you want to make sure your game will scale if it becomes successful or when it becomes successful and it's going to be invisible to the players so some of the problems we've heard after launch and that's not me why you need three engineers to keep those servers running and i'm saying three but we've seen cases where it was

30. not going to mention the name of the studio let's implement this new mod in our game matchmaker everything is going to be fine it's not going to break yeah we have this bug in production and fixing it will take two minutes but putting it in production will probably take two weeks updating our game servers require an hour maintenance and during that maintenance the game is not going to work i've stopped playing pubg because of that not the mobile version the pc version so if anybody from 10 cent is here your game is fine so let's avoid the crashes at turn number four you need to choose a back end that supports multi-versioning a b testing i've seen it a few times this morning rolling updates rolling updates we're on 2022. if you don't support rolling updates like please come and see me there's no reason you don't support

rolling updates your game should have should support having multiple versions running at the same time and you should be able to upgrade a certain area a country do a b testing in production automate the whole pipeline and that's including the deployment you want to put your game in the player's hands as quickly as possible that's going to help you down the road i know it's a pain in the butt sometimes it's quicker to just do it manually but automate the whole thing pick a platform in a vendors which won't requires you to to hire tens of devops engineers and a few phd to understand try to avoid underused server instances your cfo will thank you for that very pure waste of money so we've seen cases where actually i'll mention a little bit later oh see i don't think that the mac likes my my powerpoint here can you help uh how can we help you so who are we we're a fully managed and

automated platform we provide hosting and control for game servers and relays it's an all-inclusive psu-go pricing model it's very easy to budget we support every game genres and studio sizes we have mmos we have fighting sports fps moba and we have one man army actually a few one of those launched their games this week and we have triple a customers it's 100 turnkey solution no complex no learning curve um so it's one top stop shop for gaming backend services we work with an ecosystem of provider of partners and we can provide everything from a to z this whole platform is supported 24 7 so you will be able to sleep you can spend your time and effort on making the games and rely on us to make sure that the game is available for the players we are integrated with every existing engines out there and other backend services we provide free engineering for a special project we have a few studios

uh who reached out and they wanted to do different design in their games which called for a different kind of management in the back end and we work on that we provide that for free in some cases and we do have a feature and credits for trial and development these are some of the studios we work with i mainly want to mention the the partners and we work with net tension a parallabis branch who provides netcode and braincloud who provides player services like leaderboards and inventories and what's not if we compare ourselves to the public cloud we're container based for those of you who know about containers we can deploy any applications in 1.7 seconds compared to what the cloud will typically gives you which is 45 seconds to two minutes we support 250 regions this allows us to reduce latency by half that's why most of the aaa studios work with us

by being so spread around the world where we're able to lower latency by getting closer to your players which is not what the public clouds will typically do they support 2025 regions very centralized we have unlimited scalability if if a region is filled if a provider goes down like it happened a few times in the last few months with azure and aws we're it's still going to be fine your game will keep up and running so it's kind of an insurance for your game to be available there's no idle scaling so we deploy an instance of your game server and your relays only when you have players if there's no player we don't deploy anything so you don't need to have this overflow buffer of unused capacity like a service like gamelift will will give you will require actually they need to start to preemptively start servers so that they can be allocated by a matchmaker and there's no engineers required

uh the whole system is fully managed uh whereas with a cloud you typically need two or three developers engineers at the minimum so in a few closing notes so we provide the lowest latency on the market for multiplayer games our platform will scale instantly worldwide we caught we we helped you to reach uh all part of the world so you never really know where your game will be popular we provide we protect against outages we're going to help you tap different markets and increase revenues our technology technology is already integrated with most engine today if you go in their asset stores or marketplace you will have plugins for edge gap that's going to allow you to focus on your game make sure it's work and make sure you know when the players will spend money on on the hard earned their hard earned money on their on your games we manage the whole infrastructure and we support custom features for larger and smaller studios really so

it's just a matter of uh reaching out i think i have a minute or two to answer a few questions but we also have a booth here and we're going to host and happy hours where i will uh give you a few free drinks thank you very much sir the booth that's five o'clock today i think so six i believe is it five okay well okay maybe six i thought it's five fair enough well it's it's on the uh it's on your uh your planner so has anybody got a question uh anybody questions a lot of things thrown out there sorry i can't really see you couldn't see him here i you i'll come in you've got quite a lot of different sort of services there you seem to be doing everything and you've highlighted quite a lot of problems um you know in terms of your target partners is there a limit of how of scale before they can come and work with you or is it from everybody in the industry it's for everybody in the industry we um we're providing a human touch to integrating with studios right whereas with the

larger providers it it's it's very hard to get someone on the phone we have we use discord slack and we have one man army we obviously prioritized i'm going to say full-time developers those weekend warriors we help them when we can but we mainly work with uh established uh studios even if you're a single uh employee working okay so for even in indie dev studios it's it's still something to talk to you okay yep but we also have a dedicated did they get a team for triple a's because we do work with largers and they have a different um different expectations i'm going to say yeah of course of course fantastic okay well um thank you very much uh for your presentation and we will come for joining you for bears later on um awesome thank you very much [Music]

[Music] you