'Rocket League': Scaling for Free to Play

[Music] hello and welcome to rocket league scaling for free to play my name is matthew sanders and i'm a lead online services engineer with psionics i'm giving this talk because i was responsible for coordinating the back end preparation for transitioning rocket league to free to play [Music] let's start by talking about my team and quickly covering the scope of the problem what does the online services team do at psionics we are responsible for all game related databases web services and some websites this includes the software development and devops deployments for all of them and what is free to play just like it sounds the plan was for rocket league to drop its price tag this lower barrier to entry was expected to bring in lots of players that didn't want to or couldn't pay the sticker price all right so just scale up the back end to accommodate more players right well this change could ask our back end to handle an order of magnitude more traffic we had gotten to know our back end very

well since its original launch in 2015 and therefore we knew immediately that significant improvements would be needed as we'll see in this talk after nearly a year of preparation we did see a huge increase in player traffic but we were rather well prepared for it and had a successful free-to-play relaunch to describe rocket league's journey to free to play we'll discuss it in three parts part one is about planning and preparation we'll briefly cover setting up new partnerships along with some methods of analysis and then load testing to figure out what improvements were needed the most part two is where we'll discuss our scaling improvements in more detail finally in part three we'll discuss the launch experience and the new normal that has developed since then let's dive right into part one planning and preparation now what shall we talk about we went through a lot of different planning and preparation and it spent a lot of our calendar time but here we'll cover just a few areas

first up our analysis tasks mostly revolved around trying to figure out what to expect we spent some time with load projections to figure out what player population levels we might see and we also spent time analyzing our architecture to set up a backlog of potential issues next our partnerships are definitely worth discussing we lined up several support contracts ranging from simple ticket-based support to much tighter working relationships that directly helped us prepare for the launch finally we'll talk about load testing and how it fit into this project let's start with load projections how many new players will be brought in by going free to play it might seem hard to divine but there are techniques one of the best ways is to find a comparable game release that's similar to yours unfortunately rocket league is a multi-console sort of sport sort of action game with rockets so there were no incredibly close comparisons available ultimately we settled on an increase of

three to five times our current scale we use the upper end of this projection as our load testing target we'll cover much more on this load testing later next let's talk about our architecture evaluation what did we do to evaluate our architecture we wrote up a thorough description of how each component works and called out any scalability risks in its design those risks went into the backlog destined for validation via load testing as we worked on the first few we worked out a template for consistency and reuse this documentation actually made good reusable training material we also added a peer review step for each write-up we have peer reviews for most steps of our sdlc including documentation so this was no exception to that we also wrote up an audit of our source control repositories to help make sure we didn't miss evaluating any of our products these evaluations were pretty straightforward but the whole exercise

took a while to get through since we had a few dozen components that we wanted to evaluate now let's talk about our partnerships from very early on we knew we wanted to line up the best possible external support we don't have a huge engineering staff so we wanted to augment with dedicated experts wherever possible the primary intent of signing support contracts was to be able to summon expert specialists at a moment's notice we quickly noticed that having specialists on call like this is actually pretty expensive but our thought process was if the expert help reduces our downtime at all it would quickly pay for itself by restoring our revenue stream sooner the first company we have to talk about is google our back end has always run on gcp so we already had a working relationship and a support contract that said after discussing our upcoming release with them we decided to take advantage of some of their additional programs such as cre and gtla

next we wanted to augment our in-house dba expertise percona are the mysql experts so it made sense to sign up for their support service especially because we use pracona server which is percona's own flavor of mysql finally redis labs was a new addition we were running some open source redis clusters but that has no formal support we got in touch with redis labs and started migration to redis enterprise this got us support but it also offloaded a lot of devops work by using their new managed offering in gcp bringing on these companies was like buying insurance it gave us a lot of confidence that we'd be able to deal with scaling issues during the release but it also presented a golden opportunity to leverage their expertise during our planning and development phases too we took advantage of our support partnerships immediately in the form of several different types of exercises one handy exercise was called the pre-mortem this is similar to a postmortem but predictive instead of retrospective

basically you just get your whole team together and brainstorm possible causes of outages based on what you know about your systems score each idea with severity and likelihood then multiply those scores together for stack rank this stack rank helps guide which problems you should focus on items near the bottom of the list are most likely not worth your time since they'll be either low severity or unlikely to occur another exercise from google was for resource planning since we were not a new game launch we already had a firm baseline of resource usage including cpu cores memory and disk across all different gcp services that we were using our load projection that we mentioned earlier gave us a times five multiplier which then yielded our projected resource usage this resource plan was directly useful in two ways the first was to get our quotas increased preemptively and the second was to give the affected gcp zones plenty of lead time to make sure they have enough hardware for our increased scale hardware usage can often be ignored or

forgotten under all the abstraction in this age of cloud services but if our services are suddenly scaling up by a factor of five it's only courteous to give this heads up besides google we also talked to redis labs quite a lot we had several technical q a sessions before even deciding to commit to redis enterprise where we learned about the specific advantages it offers they would also support us during our database migrations later this brings us to load testing this was the most important feature of our entire release cycle and it's not even a player-facing feature our original intent was just to use it as a research tool to figure out what needs improvement but it ended up with far more uses and was massively underestimated writing load tests was unique compared to the rest of our code base they're totally different from writing services so we got almost no reuse from our existing code base load testing was also expensive both in engineering hours to write

and just running our load test environment this environment contained our full set of services and was necessarily scaled up beyond our production environment in order to handle our times 5 load projection lastly for the load tests they were hard to get right because it took both research and iteration to get the client requests simulated realistically we wanted to get the request patterns to match which involved calling them in the right sequence and with the right timing and we also wanted the request volumes to match which involved tuning the weighting of how often each service was called once the scope of this work became clear we realized we're going to have to level up our load testing so we made load testing one of our primary focuses as we continued into major feature development so we continued upgrading our load test code base now as full-fledged feature development we already had some engineering work invested in locust and after evaluating some other frameworks we stayed committed to it in particular we wanted to make sure the

framework we used supported websockets which eliminated many other frameworks also it ran in kubernetes which is another plus for us the one relative downside was that it's written in python and we have almost no other python code amongst our services a quick note on terminology locus scales up by hatching locusts where one locus simulates one client as we started writing and running tests we felt out several different practical aspects of load testing the first was that our tests were growing in size worthy of the term new code base orchestrating this many tests required its own debugging sometimes even to make sure the framework is working properly one notable gap in our original design was that our locusts did not coordinate they simply generated randomized data for their test runs running certain test suites was generating far too many validation warnings this was a big problem because while the call volume might be correct the work done by each service call would be completely inaccurate

for example most validation warnings cause our services to return an error without even hitting our database our solution was to set up an independent redis database just for coordination of test data locusts would then register themselves and some of their actions similar to how players coordinate in the real world this led us query for active player ids and other sensible inputs to reduce those validation warnings and thus exercise our services more accurately having to stand up this new redis database and write the related queries and coordination code is just a classic example of how the scope of our load tests grew quite unexpectedly another big issue we found was that running load tests had multiple heavy time costs the first was that scaling up a test took much longer than we were expecting proportional to the scale of the test being run for example it was common for us to run a million locusts and our stable hatch rate was a thousand locusts per second

some quick math shows that it's going to take a thousand seconds or nearly 17 minutes that's a hefty commitment for one test run so what should we do about it we could figure out how to stabilize a higher hatch rate but that's another whole engineering problem and less more development time to spend we were content to pay this time cost because we could just multitask while waiting since our locus deployment runs entirely in the cloud this hatch rate was actually also realistic since it roughly matched our target player authentication rates anyway scaling up more quickly than this could actually make each test less realistic the next big time cost we found was the run time once the test was running at full scale was it just an immediate pass fail or should we let it run for a while given the ramp up time the latter was definitely our preference longer runs helped reveal non-obvious issues such as resource leaks and running semi-randomized tests for longer help test more input combinations

now the scale up time and the run time combine to form one test iteration as multiple iterations are run of course this time stacks we ran many experiments using this iterative testing with our load tests and they were incredibly useful for experimental performance tuning but they paid this heavy cost in time speaking of experiments part of each iteration was gathering the results as i mentioned our load tests were not just pass fail so what did we look at locust reports http response codes in aggregate but we also reviewed our services logs and looked at our stackdriver dashboards with many metrics plotted over time the goal was to determine what what errors the locusts were receiving if the services had unusual warnings or errors in their logs and of course what the resources used to look like for our services and databases now that we were gathering test results how sure were we that the testing was accurate how accurately and completely

did we simulate our clients we started with plenty of knowledge about the global volumes of our service calls and we definitely studied the call patterns of real clients but in the end our load tests were still just a simulation of real traffic the biggest intentional difference is that our real clients call hundreds of different services and it was prohibited for us to simulate them all instead the research from our architecture evaluation helped inform us which features have the highest scaling risk so we can make sure to focus load testing on those first the bottom line our load tests were as accurate as we could make them we designed and adjusted the best we could we couldn't know anymore until the actual launch still these tests were definitely guiding us to many improvements we ran into probably dozens of scaling issues and had to fix each one to scale up higher to uncover the next issue to do this under pressure of live traffic would have been disastrous so what did the future hold for our

locust load tests unlike the cicada we couldn't just take a break for 17 years given the amount of effort it was taking to develop and run our tests we resolved to continue load testing up until the release it's always better to know the limits of your services even if it's too late to make improvements you can have mitigation ready instead looking even beyond the release we realize that load testing can and should be more fully integrated into our sdlc once running at a higher scale we could expect all of our services to be more sensitive to performance issues changes will be more likely to have unexpected impacts and continued load testing was going to be the easiest way to maintain confidence in the performance of our services and as we begin thinking about what it will take to get to the rapidly approaching launch this brings part one playing in preparation to a close [Music] this next part is all about scaling improvements this is where we'll discuss the feature work that comprise the bulk of our development time

the first part of our journey was all about planning and preparation to help decide what improvements we needed to make and this part is all about the improvements we had time enough to complete first we moved our core set of services from google app engine to kubernetes second we overhauled our matchmaking service third we migrated all of our redis servers to redis enterprise fourth we made a set of improvements to our mysql databases fifth we added a web service rate limiting system sixth and finally we'll spare a few moments to discuss this development cycle's postmortem the first feature we'll talk about is the live migration of our core services from google app engine to google kubernetes engine gae is google's platform as a service similar to amazon's elastic beanstalk or azure's app services just upload your php code base and the service takes care of everything else gke is just google's managed kubernetes service

we had been using gae since our original launch in 2015 but in the last few years it was showing its age and we had actually started the migration to kubernetes before free to play preparation had begun still this effort was folded into our free to play release as a prerequisite there were several reasons for us to move to gke we wanted increased control over our php runtime since we were missing out on some significant performance improvements among other things we also needed more control over our resource scaling another goal was more deployment consistency by running more components of our architecture in kubernetes this also happened to make us more cloud agnostic another big part of our core services migration was introducing separate gcp projects to better separate our deployment environments a project in gcp is simply a high-level organizational structure to divide up your gcp resources with this change the paradigm is to have

nearly identical project for each environment with no sharing of resources between those environments having separate projects made it much easier to determine which environment owned each resource and enforced independence of our lower environments gone were the days of production affecting our test environment or vice versa our logging became simpler to browse because it was simply a matter of selecting the right project rather than setting up the right log filters finally our security and access control was massively improved although we had to rethink it completely due to the way everything was reorganized when this migration was complete after many long months incorporating load testing was indispensable gke was still fairly new to us as our small devops team only had experience with it from deploying and migrating our services this is where our support from the gcp teams came in handy as they were able to assist us by monitoring our load tests from the platform side and suggesting adjustments based on their extensive kubernetes experience

completing this migration to kubernetes was a fundamental part of getting our core services modernized and tuned for higher free to play scale the second major feature we worked on was overhauling matchmaking the architecture of our matchmaking system has some insurmountable scaling performance problems these issues did not affect its correctness but rather meant that it had a strict performance ceiling to fix these problems we knew we were going to need something new and it was going to be difficult because rocket league's matchmaking has a lot of complex rules our matchmaking service was a single threaded.net application pretty close to being maxed out and definitely not going to hold up anywhere near a low testing target and as this cat could tell you it contained a lot of spaghetti code so with a blank slate where do we go we needed to fundamentally change the way it scales yet preserve all of the original functional requirements the most obvious direction was a

horizontally scalable map reduced solution we considered building from scratch but we also found open match after much analysis and deliberation we decided to go with openmatch it seemed like it would let us skip some framework development but still allow us the freedom to implement all of our functional requirements it's also open source in contract with our matchmaking service openmatch was designed for scaling which is where the mapreduce algorithm comes in it runs as an orchestrated set of docker containers this orchestration also enforces a division of responsibilities for each container which compartmentalizes the different areas of our matchmaking logic this was a huge improvement over the original spaghetti code sorry cat lastly it was cutting edge so cutting edge that it was not up to its 1.0 release yet integrating with a pre 1.0 product actually went as you might guess we were held up by or found blockers

that required us to wait for the next revision this also forced us to fix all breaking changes when that next version was released so it was not the smoothest development cycle but we launched vulpa match 1.0 and it certainly follows better versioning practices now for our last bullet of course we mentioned load testing again having load tests that incorporated the matchmaking path were essential we were able to tune open match in gke for running at higher load levels just like with our core services more importantly we learned the dynamics of how even subtle configuration changes affect the matchmaking system as a whole this allowed us to react more quickly and correctly during our free to play launch as well as know where to go with further development overhauling our entire matchmaking system was another huge undertaking lasting more than a year but it was desperately needed and directly contributed to the success of our free to play release next let's talk about migrating to redis enterprise

in our view the best feature of redis enterprise is its fully automated resharding this means we can increase our cluster size that is add shards to scale horizontally with a few clicks and the data in the database is automatically redistributed across the updated cluster with no downtime and yes we use this during the free to play release of course there is a performance cost while your keys are being redistributed so hopefully you don't need to do this because your database is already pegged if it is you won't see any performance improvement from the additional shards until the restarting is complete as part of deciding to use redis enterprise in the first place we also learned in detail about how it can be deployed specifically with respect to its proxy pictured here are a few common redis enterprise proxy configurations as pictured on the left most deployments only need a single proxy because it has very high performance however for our largest cache database we did end up needing a multi-proxy

configuration to guarantee consistent performance we initially tried to enable support for open source's cluster mode pictured center which deploys one proxy per shard and has all clients connect to all proxies this was the simplest change but unfortunately we couldn't use it due to lackluster support in our web tiers redis client library instead we are using a multi-proxy setup that uses dns load balancing pictured on the right this has the typical disadvantages versus a layer 4 load balancer specifically for us our web tier had recently moved to kubernetes and we set up node local dns caching to improve some of our network performance unfortunately caching dns results is incompatible with dns load balancing so all dns queries on the same kubernetes node would always resolve to the same redis enterprise proxy rather than rotating that said we run enough kubernetes nodes that the connections still end up

reasonably well distributed across the cluster so it works just fine [Music] once using redis enterprise we noticed that a few of our databases were performing as if underprovisioned despite our estimates why did we have to scale up these databases to get the performance we expected it came down to improving observability measuring performance in redis is somewhat difficult measuring cpu directly is too opaque and measuring operations per second is also misleading if your commands have a mix of o1 and on operations performance so we did tune some of our commands to simplify from the application side for example we switch from using mget and mset to the single get and set sent with pipelining thanks entirely to the pipelining there was only a minor difference in performance from the client's perspective and the reddish shards were able to burn through the smaller commands more efficiently

furthermore this made the reported operations per second far more accurate since mget and msat both counted as one operation no matter how many keys they updated this observability change gave us the right information to explain the performance issues and then scale the cluster appropriately tied closely with observability is how we monitor each database redis labs provides a guide for setting up prometheus and gerfana integration including some fantastic pre-configured grafana dashboards once we got this working it was far better than anything else we hooked up for our open source redis deployments the screenshot on this slide shows operations per second broken down into reads and writes and it's just one of the many charts on these dashboards redis enterprise also has a few other nice advantages that we leverage redis enterprise handles redis pub sub much better than open source in a few different ways but primarily because it doesn't distribute all publications to all shards

next standing up a temporary replica for key space analysis or other inspection is just a point-and-click operation this has already come in handy several times for looking at production data without interfering with the performance of the production clusters finally none of the features of redis enterprise that we are using result in vendor lock-in with redis labs the only risk is not wanting to leave besides redis our other primary database system is mysql it has been our most serious known scaling bottleneck for a long time specifically it only scales vertically give the database server more cpus and memory until no higher spec machine is available it was easy enough for us to add replicas but this could only offload the read traffic that could tolerate replication lag we implemented countless optimizations over the years which gave us more headroom but eventually there's an upper limit on a single server especially when anticipating a quintupling of the

server's workload our best option over the years has been feature shards this is where we move sets of tables to new servers and it's relatively easy well as long as you can identify a set of tables that can safely be separated out unfortunately having all of our tables on the same server for many years has led to many tightly coupled features due to convenient table joins we were left with a large core set of tables including our largest overall tables that could not be separated out without significant refactoring effort notably this does not solve the vertical scaling problem but does buy us a lot of headroom for each feature despite the scaling risk we estimated that there was not sufficient time to safely incorporate a horizontal scaling solution to our databases that needed it the most we thoroughly evaluated the tests it would have let us spread single tables across multiple database servers this would scale better but to use it

properly we'd need to solve new types of problems one big example that's both mundane and labor intensive is that the test would have required us to eliminate stored procedures and move all of the sql into our web tiers code base we have over a thousand sore procedures so ouch in the end we split out one last set of tables in this release cycle and our load testing demonstrated that we were able to exceed our load target but this was not our only mysql improvement [Music] we also incorporated proxy sql to proxy the database connections between our core services and our mysql databases proxy sql is deployed as a horizontally scalable set of identical proxies behind a load balancer we can scale out as many of these as we need each proxy maintains a set of persistent connections to our mysql databases and listens for connections coming from our web tier the point of this setup is to offload the connection churn caused by our php web tier unfortunately we can't use either

connection pooling or persistent connections from php the process model makes connection pooling impossible and persistent connections are not feasible because it would establish one connection from each php worker thread to each database and we run too many worker threads for that to work processing a login to mysql is a surprising amount of work for the database server offloading this work cut the cpu usage on each server by about half this gave us a lot more performance headroom which is critical for our vertical scaling we also gained instant failovers our previous technique for promoting a replica was to swap ip addresses this would take about 50 seconds which was a rather large unacceptable hiccup to deal with so it was only reserved for emergencies or maintenance windows proxy sql lets us promote a replica almost instantly this allows us to upgrade our servers much more regularly and without a maintenance window

proxy sql has some other benefits too it speaks to my sql protocol so it looks like a mysql database to your application just point to a different ip it also has dynamic configurability for routing queries to replicas we had already done this manually in our web tier but it's a handy tool for a scaling emergency without having to modify our services there are also some cons from using proxy sql we had some downtime due to proxy sql bugs there were socket leaks in some error handling situations which took us a while to detect and mitigate in fact we're still dealing with one which we can't reproduce for the maintainers to fix also proxy sql is a bit complicated and the learning curve plus our lack of experience with it resulted in a misconfiguration in production within a few days of our free to play launch this is another good reason to release major features early all in all proxy sequel was a definite improvement we only discovered these cons after using it for a while and the pros still clearly outweigh them

the last feast or the last feature we'll discuss is one of our handiest small features rate limiting where did this feature come from originally we designed a few options for introducing a login queue as a major feature and the engineering effort for most of those designs was significant eventually we thought up a much simpler solution by understanding the relationships between our player population levels session creation rates and player churn [Music] we reasoned that a configurable limit on session creation would allow us to control population well enough in the event we had to use it of course the hope was to never use it so it was more of a load shedding technique added to our tool belt this is also why we didn't want to spend tons of time on a more complex login queue that would be rarely used we went with a custom implementation instead of integrating an off-the-shelf product because the logic is very simple to add to our web tier and having our own implementation let us use our existing infrastructure

such as our feature flags logs metrics and dashboards we also leveraged key exploration in redis to reset each tally automatically which keeps this logic flow chart really simple rate limiting turned out to be one of our most versatile and useful features we could configure rate limiting for any set of services this would let us limit or disable problematic features to this day we have rate limiting setup to protect our immensely popular tournaments feature we could also limit service calls globally or per player for example we could limit overall session creation to 500 per second to control how quickly our player population grows or we could limit friend invites to one per second per player if we see a client bug or something like player reviews this simple rate limiting feature definitely takes the cake for cost benefit in our development cycle it was cheap to write and has demonstrated its benefits many times

as our future development was drawing to a close it started to seem like we hadn't accomplished as much as we wanted to weren't there many months available on the timeline don't be afraid of asking these kind of questions in a postmortem let's see if we can come up with some answers the first thing that sprung to mind was that our team like many others transitioned to full-time work from home in march 2020. as teams go much of our work is in the cloud or at the very least remote desktop friendly it only took most of us a matter of days to get set up in our home offices but of course there are many other less tangible impacts that the industry as a whole is still sorting out this makes it hard to put a finger on any specific impact or lack thereof next major features will take a lot of effort and are very involved if you follow your department's sdlc changes to live services will always need live migration plans and rollback plans bigger changes need bigger plans often with multiple phases

our matchmaking and core services migrations were the epitome of this since they both fundamentally changed how they scale that's quite different from your typical feature change something else to be aware of don't overanalyze a hefty backlog containing great ideas does no good if you have no time left to work on it it's probably best to have some engineers start working on features while research is ongoing next define intermediate deadlines respect them and coordinate them treat them as legitimate deadlines escalate and call out the risks if the work is falling behind make sure everyone on the team buys in this can be hard to sell to the team if the real deadline such as the launch day is still much further away most of our major features actually slipped far past our intermediate deadlines we were almost in a situation where our major features were all trying to roll out at the same time right before the free to play release that was incredibly high risk from our purse modem we realized that

our feature deadline slippage was also also partially the result of not properly coordinating the teams as i mentioned earlier some of our features were already underway and simply folded in as prerequisites to our free play release this weak association allowed those features deadlines to slip without considering the wider free-to-play release schedule we should have brought our teams closer together which would have clarified the overall schedule and the impact of any individual delays [Music] so after reflecting on our development postmortem we can now start thinking about the big release this part is where we'll discuss the launch itself followed by our new normal first we look at the schedule our final lodge schedule is as you see here releasing the game update a week early was an important part of the launch logistics this had a few nice impacts historically every update has had a significant upswing of users who sign in

to check out the new features for this launch releasing some of these new features early allowed some of this user bump to subside which helped flatten the curve before going free to play one of the features we released earlier was account linking with epic games accounts this was a very complicated feature and releasing it early let us start monitoring it and get in some bug fixes before the additional influx of players similarly we were able to watch the newly patched clients for any abnormal call patterns and and had time to react to the hotfix before the free to play release the free to play launch itself was on a wednesday and midweek is significantly less busy than the weekends as we shall see and here's the launch this chart shows the game update the hotfix the free-to-play launch on wednesday and how the population climbed into the

weekend look at that population increase compared with the previous week we exceeded a million concurrent players for the first time ever this was an incredible milestone but achieving it actually felt like a foregone conclusion because our load testing had demonstrated that our performance ceiling was much higher than this sunday's highest population peak actually exceeded our five times high end estimate but it did not exceed our how high can you go load testing this reinforces that you should continue with load testing and improvements up until the release this also shows why releasing on a wednesday was an excellent choice the first boost on wednesday was big but then we had a few days to react to issues as the population climbed ever higher into that first weekend but if you look closely at the charts the release wasn't entirely sunshine and roses as good as the launch looks we did have some severe issues including

two near total outage indicated by the red arrows the first outage was on wednesday the 23rd which was our free to play launch day [Music] we had an unexpected scaling issue related to signing in with epic games accounts and our new account linking feature this was an integration point between psionics and epic game services that was not covered by our load testing also note that account linking had released a week earlier so this problem only showed up at the higher free to play scale the second outage was on thursday the 24th we saw climbing redis traffic getting too close for comfort and made the call to double the number of redis shards for the affected redis enterprise database this was a preventative measure but this database was already stressed and adding the overhead of the resharding operation ended up starting a death spiral resulting in this outage the third issue was on friday the 25th it's not visible on the chart because it did not affect player population

this was the proxy sql configuration issue that i mentioned earlier while preparing to promote a mysql replica we removed the read-only flag from its configuration and proxy sql picked up on this config change automatically which was premature that replica was added to proxy sql's right group which led to a split brain situation this wasn't an outage but we had to spend quite a lot of time reconciling the data over the rest of the weekend we had no load-related outages this is despite far more load compared with the first few days these were the only issues major enough to make it into our release postmortem also a reminder to the viewer don't forget to keep a record of your issues to write a complete and accurate postmortem we've referred back to ours many times this was a pretty incredible record for the first weekend but how did the team handle it our team was tense but confident the rest of the studio just bombed us

with memes the peak load set a new record every day for the first five days which meant every day was more stressful than the last when was our all-time peak going to occur amazingly our times 5 projection turned out to be extremely accurate the entire team joined a war room zoom call every day up through this first weekend we were more prepared than any other release prior but it was still tends to monitor it was so difficult to try to load test everything we've ever written did we somehow miss something important lastly it's worth saying that just because there are no other big outages doesn't mean that we weren't actively identifying and fixing issues before they became problems we spun up a simple spreadsheet for lightweight tracking of potential issues it ended up with over 50 entries many of these were fixed quickly and the rest were put into the backlog to fix once things calmed down after the population began to subside on monday the team was able to take a

breath and start to think about the future and our new normal halfway through 2021 we're still seeing a sustained increase in users it's holding at about three times our previous normal with this new normal the higher population makes our services more sensitive to performance issues this applies to both code and configuration changes the specific danger is that if the client call patterns deviate from our load test simulations we end up in uncharted territory one example of such a configuration change is when we enabled a limited time mode like heatseeker or spike rush we're up to a million players all switched over to that game mode nearly simultaneously that's very out of the ordinary behavior different from our load tests and therefore not something we tested thankfully we were able to react quickly to the situation by extrapolating from our previous load testing of matchmaking a second example is when we set up an

easy challenge that rewarded multiple items this caused players to be rewarded items at a much higher rate than normal and it ballooned our inventory system to its breaking point this has left us watching our inventory system very closely until we can migrate it to a new database engine that supports horizontal scaling on the upside now that we've put all this work into our load testing system it's readily available for testing any of these situations finally we've renewed our diligence in following best practices for designing live migrations promoting code and always having a rollback plan this concludes our look at the launch so let's wrap up takeaways what do i think you should take away from this talk these are my picks for takeaways from the planning phase seek expert advice and support stay organized and get working on the long tent polls first but be aware of what we're planning set a deadline for it the next theme that permeated this

entire talk was load testing it's our most significant feature that came out of this release and we would have failed without it it was far more work than we expected but it was unquestionably worth the investment for the free to play release and beyond next when thinking about major improvements take careful take care to carefully coordinate their deadlines and roll outs and respect those deadlines finally you should always have some versatile load checking controls like rate limiting they're low effort and will get a lot of reuse even if your load testing demonstrates you can handle your load target all day long what if you actually get 10 times or 100 times that target rate limiting might just keep your services stable rather than have them go down entirely so in closing thanks to efforts described in this talk psyonix had a really successful free play launch for rocket league and i hope you will too in your future endeavors thanks very much for attending please ask questions if you have the time