How to Launch an Online Game and Still Attend the Launch Party

[Music] hello uh welcome I'm I'm Chris inkland um CTO at Odyssey interactive Odyssey is kind of a very small studio uh focused on competitive online games um I should remind you to please silence your phones if you could and uh you're going to get like an email I presume or maybe it's in the app I've now already forgotten um about a survey please uh you know be nice to me that'd be great um and so this is how to launch an online game uh and still attend the launch party and uh thanks for coming um so the structure of the talk here is kind of largely oriented around two case studies um before we get to the first one I'm going to introduce the game uh that the case studies are from the game is called Omega Strikers and I'm going to briefly discuss sort of the architecture of the game services that we used uh from development all the way

uh through release um and you know since I put in the title I will point out that launch day wasn't exactly boring or peaceful but um I was first call for Pedro Duty I made it into the pictures that we took you know um and and the dinner afterwards um and so I think in the end you know we sort of had confidence in our tech and and it seemed to work actually um so part of the discussion about architecture um I will then Define sort of observability um and explain why it's necessary uh the first case study is a really s simple Trace uh that's uh for a player's matchmaking status um which will be a bit of an orientation into what observability can look like um then I'll dig into some of the tech underpinning kind of observability and and how we implemented in Omega Strikers and then the second case study is a little bit more complicated uh hopefully to show kind of where things can go uh with the observability

um it revolves around kind of a matchmaking problem that we had in in the actual uh public beta and then finally you know just some summary stuff and uh hopefully there will be time for questions uh so starting with the game itself uh it's called M Strikers like I said uh here's a short video you know because it's cool [Music] [Music] be get

down en [Music] [Applause] thank you video team um so Omega Strikers like I said it's uh it's odyssey's first uh released game uh it's a three Onre competitive online game um kind of many of the features you expect from such a game there's matchmaking there's ingame friends presence chat uh we have custom games you know the typical progression systems that you might have daily weekly missions that kind of thing there's battle pass entitlement shop um and all the things that I just listed are what we at Odyssey call our services or the sort of

around game uh which is somewhat distinguished from the core game and I will only really be talking about uh the services uh in the infrastructure kind of to support that uh for this presentation uh Omega Striker launched on eight platforms uh PC Mobile console kind of all of them all together um we did it with a pretty small team we're about 10 10 engineers and for services in particular we were two to three at at any given time uh just to give you a sort of scale of of uh people to to do what we're what I'm explaining um and so being an online competitive game you know as an engineering team I think we have sort of the responsibility to provide not only sort of stable gameplay uh for our players but really just ensure that they can log in and play uh whenever they want and I think failing to meet that uh especially right as the game is launching is like a really really negative impact on any momentum that we might be able to build and I think just sort of personally it feels bad if somebody can't play your game um

and so in order to achieve sort of our successful launch and the continue live service operation uh we focused on scalability and operability as our kind of principles um so essentially we wanted to be able to respond in minutes to some problem uh while also simultaneously handling significant fluctuations in concurrent players um and so the goals there of scalability and operability kind of led us to focus on on managed infrastructure uh and that's very pervasive through the entire uh architecture of the game uh and we kind of identified the general overhead and and inflexibility of we call it owning a computer um as something to be avoided as much as possible uh so here is a picture um of the high level architecture of the game services in Omega Strikers um sort of pretty pretty briefly you know game clients go and send htttp requests to

Services uh via some kind of Gateway um they receive then notifications about stay changes back down over a websocket um the actual logic of those Services run on a big bundle of uh lambdas that's like a serverless compute offering um persistence we're using mongodb um they have a managed Service as well and then sort of sprinkled around in there there's a bunch of Rus nodes that do caching for us uh also on a kind of managed offering um there are Game servers as it turns out um and and we as may not surprise you use a manage service to to handle scaling them up and down and as well as uh Place matches um there is a computer that we might describe as owned uh in the middle that's the match finder it's a single box um and it just sits there and kind of spins over an inmemory list of players uh and then spits out

matches when they're found uh and then all of these pieces uh of infrastructure talk to our observability service uh which I'll talk about a bit later um and so the the key thing I think about this is that as I've said repeatedly they're all managed Services uh which gives us you know the ability to to scale uh quite quite nicely um and as a result the computer doing the actual work may actually be ephemeral uh uh you know it may go away or may not have control over it and so if we want to do a typical thing uh like you know log into the box and see what happened you may not be able to do that because either like it's gone or you just aren't allowed to or whatever um so that's kind of a big blind spot and it makes it really difficult to operate uh these kinds of services and that's what observability really that's the hole that it's plugging um and just before we move on

I'll sort of highlight two notable decisions well I think they're notable anyways uh about how we actually made use of this um and I would describe them as probably opposite sides of the conventional best practices Spectrum uh the first one which is the probably good side um everything was automated uh in terms of deployment we we aimed for like one button uh we ended up with three buttons but you know it's not that many um and that was sort of a core safety blanket uh for us in terms of really responding to really really high demand or if we had a really significant outage um we sort of had this in the back pocket where we could spin up the entire stack in another zone region wherever um or potentially concurrently in multiple zones uh if we really had big big scaling problems um and there's a there's a requirement um to really maintain uh that ability um you know us usually it'll be every

couple of months or weeks you spend a few days yelling at computers because it's not working um but it was totally worth it uh an example of the value we had a play test that we heard about only a few hours before it was supposed to happen in Asia um and we were able to just like click the buttons and it worked and we was like great and so it didn't disrupt any of our daily play tests or or any other really development process so move on to the unusual Choice um for Omega Strikers is like there's a single environment just just one uh it handles you know live game uh it handles internal play tests it handles literally everything um so when you make an update it's going live now and that means that like development features upcoming content absolutely everything is always in production uh so this forces us to implement sort of logical barriers uh to enable testing or protected actions but the upside is that there's only one process proc and we do it every day or

potentially multiple days and so we're really used to it and as I will talk about later that's really really helpful um so changes get put out quite frequently um and there's there is a risk to that um but you can kind of mitigate that or we mitigate that through pretty extensive automated testing um in the sense of like end to-end testing uh and then there's you know feature flags and actually rolling out A change is quite quite quick so if it goes out and it's bad you just undo it it's not the end of the world um and I think again the big side big upside of this single environment is that the in the scripted deployments um that for each individual change is actually relatively low risk and like I said you can kind of just undo it and it's very incremental in that way and the other big thing is you're never migrating or you're trying to do like any integration between these environments it's always there

um so like I alluded to a major downside uh to the managed infrastructure and especially these kind of ephemeral compute uh offerings is the significant increase in difficulty of debugging the thing when it's actually running um there's several factors I think that contribute to that but generally it's just like hard to isolate a particular change or figure out what the heck it was doing at that time um or or or reproduce it in any sort of really controlled fashion um so solving that is obviously something that we are going to need to do so that we can actually use these things uh at scale and so I'm going to Define observability kind of informally and it's it's really the ability to inspect or sort of otherwise know what the state of an application is um while it's doing processing and logging is an example of a common but I will classify it as somewhat rudimentary form of observability in Omega Strikers what we use is essentially a highly structured logging uh systematic instrument instrumentation and then a

queriable UI Al together uh to to solve these problems for ourselves um and we'll dig into the really sort of more specifically how that how that looks uh again a little bit later um but for now I just want to highlight a couple of things that this allows us to do which are really significant Time Savers in debugging and I think as a result really increase our our reliability um the system is always on it's always recording there's there just kind of it's like a thir day buffer but um it potentially could have been longer or shorter um but it allows us to kind of go back and investigate problems um after they've happened so we do daily play tests and this was like our primary testing of the game more or less uh and so someone would say hey this was weird or this was busted or whatever it was and like we were playing the game so we were not like monitoring this stuff but we were able to go back in time and look at the state that they had at that time was all captured and just Trace

through and be like yes that's right it was busted it was busted here or no you're wrong that's actually how it's supposed to work um and and it's a proactive as well it's sort of real time uh so when you release new code you can just like look at if it's doing what you expect uh so you could you fully inspect the the system okay so we'll start with the first example um it's hopefully kind of illustrates now what I've been describing for the last few minutes um this is like a sample of the UI uh that you would see uh and it's it's a really simple matchmaking status query the focus here is not on this particular request but sort of on how this data is structured and and presented that hopefully will illustrate some of the benefits um each row so some nomenclature here each row is called a span uh spans are typically organized in sort of a tree like hierarchy the one at

the top is called the root and the collection of related spans is called a trace and typically that's going to represent some logical operation it's kind of context dependent um every span usually has some common data like it'll have a name uh a duration and then it'll point at who its parent is uh and this allows us to create you know this kind of waterfall type visualization um and you know just with that we're able to see how long individual functions take basically Within These requests you know we can aggregate them across multiple requests or or by players and this is essentially what we use to do uh load testing and profiling U so by itself I think that's pretty valuable um but then there's a bit more uh so each span you can give it kind of this custom bag of data uh so here's example of a few things that are useful um and then you know those get indexed basically so you can query on

them uh and then look for related spans or or traces so here's like you know a I don't actually remember what I highlighted there but you know the game version for example um would be useful things to to query against um so here's a different Trace just to sort of highlight the the value of the built-in timing um it should not take 30 seconds to log into the game uh and I we pull this up and we know exactly what step actually took all of that time you know immediately kind of like a profiler but it's always on um and finding this takes like 10 seconds you're like hey find me things that took a long time oh this is bad um okay so we'll move on now um to sort of what an implementation might look like and and what tools are uh out there to to use uh for your own code um at its core right this data is

kind of just semi-structured Json like that's what's being passed around um fortunately there's sort of a standardization body uh it's called open Telemetry they identify a few common property names uh that are expected to exist so these are like that's the name and and the parent and and that timing and those things they have like almost like a schema um and outside of those the rest of the data is pretty context sensitive or context dependent um you kind of give it whatever you want uh so we ended up building a pretty small set of utility functions that made it really easy to integrate like spans and traces into the services code um and we through fairly continuous effort um we ended up with like about 1,200 instrumentation points last time I checked um the number goes up like every time you write a function we just always wrap it in a trace or a span um there are lots of Frameworks that you could use that have like you know sort of built in uh instrumentation for this kind of

thing so it's not like there's this massive labor cost um we didn't actually use anything like that so we did it ourselves um and then we're able to use that integration then to facilitate sort of debugging across all phases of production so development Stacks your unit tests are running this code so you can actually look at them um and then having them all visualized in this single tool is really nice it's like again back to the sort of your used to this process um from all uh points of development um and then to to actually aggregate things and visualize them we use a vendor uh to store and Index this data um there are lots of backends uh you can kind of just Google them and pick your favorite one uh here's like a kind of cool picture that open Telemetry made which is great uh to sort of show what the internals of the observability system itself are doing you know there's got

your various producers and they're sending data to you these service or or services that are called collectors The Collector then is you know doing something probably indexing maybe a database is involved uh and then it's feeding some kind of front end uh to assemble you know however the visualization is could be the waterfall could be something else um and then typically it'll provide some kind of querying interface as well uh so you can look at interesting things uh I'll just point you kind of to the open Telemetry documentation uh for any really specifics or or to to go deeper on that um yeah so now we'll move on uh to the second example it's a little bit more complicated um but the idea here is to show that you can look at data from multiple systems uh together uh to solve problems that I think would be pretty challenging without some kind of centralized system um this is sort of a recreation because the data was is a out but a recreation of a real problem that

we had in our first fairly large public test um you know we were sitting there playing the game it was great and then all of a sudden people were not matching for relatively short periods of time it was like you know 15 seconds is it's kind of when you get like a DM from Discord it's kind of hard to know exactly what they're doing uh and then some groups like weren't match we're sorry weren't matching at all yeah and so the first thing to do this is a matchmaking problem right so we go look at the Matchmaker and it's like okay this this is bad um but what is going on here um so we can basically zoom into the last Trace before one of the dips and again the useful part of that is this may or may not be actually happening live but I don't care because I can go back and find a case of it that it was happening uh in the past um and so you kind of drill down into this particular m finder Trace um

get that last one that's great uh and we can look at some of the context data we've got here which in in our case we were specifying you know these are the actual groups that are in the queue um this actually looks fine so this is probably not the problem but maybe this one at the front is is suspicious so let's go ahead and dig further into that um like query hey give me things that are related to this group um I sort of filtered out all of them but you would see like the construction of the group so you see the invites for them and who accepted and how that was actually done um but the relevant one uh for us is like okay well they're getting added to the matchmaking queue so who was in their group what happened when that uh addition happened maybe that data is what's suspicious and it turns out um unsurprisingly because it's kind of a guided tour that this is the one that's that's wrong um they're all forwards which probably doesn't mean anything to

you but in our game you have to have a goalie or at least that's what the matchfinder decided and so it was sitting there trying to construct groups where one of the teams was full of forwards and it was like I don't know how to do that uh and so then they would sit there and plug the queue for essentially the timeout uh for them so they would fail and then people would go through would be a big burst and that was the the chart okay great so now we know the problem um we basically then like hacked in a really quick fix so that okay we just ignore what you wanted to be for now so that people can play the game um throw a test case in actually fix it properly in the Matchmaker uh update the Matchmaker and then we're we're good again and like the total time kind of to do this was in the 25 minute range um which I think is pretty good uh we're like getting data across multiple things there's you know

hundreds or thousands of matches being made uh per second at this time not not the picture that I showed you but I promise it was like that um and you know so being able to drill down into that data really quickly is is solving a big big problem um but it's not as it's not quite as valuable when you aren't able to also operate on it so that's kind of our principles uh in action um um yeah so uh kind of in summary we kind of kept our infrastructure as simple as possible while still addressing the the problems that we were trying to solve um we use the same process every day from like early early development uh all the way through launch and and into life service as well so we were really really familiar with it and so when there was a problem it we weren't like trying to remember like oh how do you do whatever in production or how do you do it in stage or what's at no there's just the one thing um I think the investment

commitment to observability really meant that we could understand what was happening uh so we could drill in directly quickly there was again a single tool you just go here and find it um and then lastly on launch day itself we really significantly over-provision the infrastructure just give ourselves a buffer um this flexibility is is exactly what the manage services were supposed to provide and they did it and that was great and yeah it costs more than it maybe should have but the thing ran and then you can scale it down very quickly over the next you know a couple of days or or weeks once you identify like what your players are doing and how many of them there are and what you need to do there um so I kind of think of that as a marketing expense realistically if you're trying to sell that to someone get some marketing money to run your servers um okay uh that's it uh thanks for coming

uh can take any questions will you ask that you go up to the the microphone if you have a [Applause] question hello hi talk um I was curious about your match finders you said is it literally one machine yes it's a single C5 large it's like a little baby machine running and it can produce like 3 million matches per second wow okay how do you patch it if it's one machine without taking prod down or do you just take prod down for a brief downtime oh uh we just spin up a new one so there's like cues and redus around it right right right so it you just kind of pop one up and that one takes over and you Dr down the other one interesting yeah um yeah so for my matchmaker I recently decided you know against against external uh advice to you in memory as well I was like well it's probably it's probably fine so it's pretty fast yeah like not I think we as a distributed

system sort of tech space shed away from any kind of in memory anything when we figured all this stuff out but I think if you're calculated about it you still can do in memory sometimes so it's validating to see somebody else do the same thanks welcome thanks alandro producer but uh ex Apple software engineer for seven years did Sr live Ops devops kind of stuff uh very interested to see your basically like monoo style like single Branch production because that was not my experience but curious to see if you think for a game like Omega Strikers maybe other games where um maybe you aren't developing too many features that would be maybe taking months to develop and then needing special integration do you recommend that maybe other games maybe take this similar approach using feature flags and end testing um and using that just for Simplicity sake especially for the Simplicity of automated one to maybe three button I would yeah I think it was great and

and we have features that took months to develop uh you just build them in the branch like turn off the code don't call the code um you know obviously there's like two or three of us so it's a lot easier for us to communicate with like hey don't call these functions please they're not ready um I think there are probably processed ways to to address that at a larger scale [Music] um I had another thing that is just gone uh I'll just steal in that time that you just lost on that thought um testing you do unit functional and then end to end or just do you focus on like the last function we pretty much just ran full end to end uh spin up the whole stack locally um you know we use some Docker for that um and just run the whole thing because I don't know that's what's actually running and so especially you know we have to be able to observe those things anyways so that's just part of the stack and then we know it's going to work cuz we tested literally the the whole thing cool thank you thank

you uh hey hi uh I'm also a big fan of open Telemetry uh one of the issues I've seen teams try to solve is volume of events so you showed a trace that had every function call uh generating uh span and being sent to the vendor I wonder yeah have you had to deal with volume problems and costs and yeah so uh I I have a slide that I that I removed but yeah it was it was massive uh when we launched we were sending like almost a trillion events a day um you do some sampling basically is is how it boils down um it's a really challenging problem I I think to solve where where we are now is there's Baseline flat sampling like at the actual event creation uh because otherwise you have to centralize the the whole volume of data and trying to do any analysis on it becomes really challenging because the volume is so high um but then we have special paths through so like any requests for example they're never sampled we we want to know what's

happening in CT or we want to know what's happening in Dev um we're able to turn traces on for individual players you know these are sort of just like overrides that we've learned are very useful for us um I think it probably depends on what your particular context is and what is useful information but yes that is a hard problem okay cool that's good to know did um is it basically like you could turn it on the root span like okay the root span and all child spans are going to get yeah so we keep most of our root spans because we like to see um kind of the how many requests are happening and but we throw away the rest of it essentially and so it basically just rolls rolls a die at the beginning of the ban and says H we don't need you um if there is then a fatal error or something it'll keep the stack basically because that's kept kind of in memory um ideally we would buffer all the requests probably the the whole Trace while request was happening so that if we do hit these error cases we

can like dump the whole thing um but it's been pretty useful just to be able to see that particular stack basically cool hey thanks yeah um hello uh uh I don't know if if this is okay to ask but with the with the sad news of Omega Strikers uh I want to ask is is there still a team that looks into the problems and all these uh for example errors or bugs or anything El is there like a small team or is it like when it pops up it's like oh can someone like check that out or like deal with that or how does that work in yeah we we still have people uh full-time on Omega Strikers okay got thank you thanks cool um probably last one but yeah okay great yeah so uh yeah running a live game as well um and I think one of the interesting things you brought up is that your your server backend is entirely all prod All Forever um but that also means that you have a high potential of leaks and stuff like like when that also means you also have like

code duplication right so if you have something that's not backwards compatible for example um you're going to have to have old code and the new code how do you like maintain that and make it so that people aren't duplicating all their work over and over again um when they're back comping yeah so I think there are a couple of questions there the first one is like worrying about leaks um we worry about them but not like too much okay like it's probably stuff we're going to release anyways if it leaks it's not maybe the end of the world obviously we try to to avoid the leaks um but it's not going to end of the game um and then the backwards compatibility thing we basically duplicate apis if if they're going to change significantly you just go for v2 and great and then you in theory to lead it later I don't know if we've ever done that but we probably should right right okay so it's normal normal code duplication is fine just deal with it later coffee and paste is there for a reason okay cool thanks oh jeez all right thank you

[Music]