Start-And-Discard: A Unified Workflow for Development and Live

[Music] in this presentation I will not dwell too long on the past but instead I will share with you the current shift in mindset that we are undergoing with our online development I hear at massive and by the end of this presentation you'll understand how by opening a cloud first mentality you'll be better prepared than ever before we're smooth and stable onshore we gained but first some background and and Who am I and what perspective do I bring to this presentation well I am online focused have been at massive for 15 years and the company itself is over 20 years old and since we can trace our online roots back 15 years we had this long legacy and I I just want to state for the record that I think legacy is is a good thing because it enables us to rest

assured on the quality of some subsystems so that we can focus on improving others rather than having to reinvent the entire way for every game we do so let's take a look at a typical massive game so there's no spoilers here there's just considered a normal classic mess again whichever it is running on your TV or console or PC this game is connected to a online infrastructure that we run in in data centers around the world contains a lot of micro services over 20 and that deal with things like your say gaming your character and progression and leaderboards and now what have you and furthermore this your game is also connected to a game server and this game server is running the AI training missions and controlling NPCs and and of

course also multiplayer activities and then of course this game server is also connected to the online services in infrastructure and one thing that that this enables us to do is to or that it forces us to do rather is that we take care of the say game since the game state sits on the game server and not on the game client the say game is synced between the game server and Lauren on back-end so with that out of the way let's talk a bit about what's cool about our online the way we do this and as you can understand it enables us to run larger simulations because since the AI is uploaded to the server we can we're not limited by the compute power on the console or the client but rather limited by the compute

power that we have access to in the cloud this enables us to have a bit of a different performance budget than all the titles mostly we focus on better graphics and and and better you ice or higher budgets whether for this since there is less a AI that needs to be computed on the client and also we have multiple clients connected to the same game server so since they're already there it's easy for us at seamless multiplayer and and it's up to us whether or not we enabled flip the switch if we oversimplify it so to replicate the players so you see them this happens for example in the division when you walk into you can work into a group of players and that's because they're all dirty all right they're there so now then let's talk some consequences on on our

online strategy because of course there are choices and there are pros and cons with with the way we build it and and these consequences are well to start with it's a complex mesh of online services you saw very simplified picture of it earlier and but it is quite complex and there's a lot of interdependencies in there further or since the AI is running on this server we cannot have an offline mode for the client the client has to be always online otherwise the game simply doesn't work this means that the online stability is absolutely critical even for single-player so we try to think of our online infrastructure as a critical infrastructure and furthermore since all the teams regardless of what you work with on pretty much in the on the game you're affected by online so

this means that that you need to be careful so you don't break things or order other people you know if you break the game server everyone all the players on that games are kicked out for example even from their single-player session which we don't want to happen and we need to ensure that everyone is up to speed on the latest on auntie-ji technology etc so here it master then we of course have all eyes on the online and to deal with this we have a very efficient LiveOps organization here at massive and who take online seriously on the quality of the service to our gamers and and adding adding more stuff to to the game as we all and and we do lots of open betas etcetera for testing for example the in debate on the division was the biggest ever for a new game and

that reached over six more four million players which at that time was a record but let's go back and talk a bit about development and it's a story of unintended consequences because since we've been doing this for so long and as time goes by and different teams may start to use different testing methodologies for example in different technologies and that I'm best suited for their specific task at hand and then also different tests may use different stubs and shims and different teams may test on different hardware so there's a lot of special cases and and what what can happen then is that special cases become the norm rather than the exception which increases an already complex system

results in lots of isolated testing which is not always relevant for the shipped game and as we grow as an organization fewer people understand the full online system so let's talk about some some examples of unintended consequences well I mentioned the say game system earlier it's very important that it works so we have built it to be rock-solid it's essentially a son however this son came online very late in the project so a local stub had to be used in most cases during the years that the game was in development and the team size is not large enough and because at the end of the the latest project we did a server could handle 1,000 clients

because of all the performance works that had gone into it but we didn't have 1,000 members on the team so on on a playtest we couldn't actually feel a server we had to rely on boats and and boats don't necessarily load the servers like players do so it complicates capacity planning for launch and some stubs and shims were were testing more than the system and that we shipped so we were not always focusing on testing what we actually so that was actually very much something we wanted to improve but furthermore that this is more than a technological issue because for for many developers live is it's a special case because it's not the case you see while you are working on the game and we have different processes therefore it has evolved different processes for when we develop the game we're building data

managers handling ops with QC monitoring and but then when the game goes live we had our live ops organization then I mentioned earlier with a war room for monitoring so it's it's quite different and and of course few developers actually see both sides of this fence so it's hard to anticipate the needs of the other disciplines so we needed to be better testing what we ship and and we have this mantra now that we test what you shipped on the ship were to be tested and and this sounds very easy but but really it's it's not because how often do you test the client well you do pretty much every day I'm sure the services your account system somewhere near any other thing you depend on I hope you test it a lot the game server I also hope you all test it a lot

auto-scaling and how often you actually test that and do you also test scaling down which you most likely need to do at night do you test disaster recovery or how often do you test that so you know it works and furthermore how often do you actually test the war room so what what happens here is that as as as the impact on live increases our ability to test decreases so the less something something is tested the more it relies on a few experts with specialized knowledge and these are people that can get sick so it's a potential liability and and a risk but that we don't want and then if

you couple this with the way the industry is changing the way it has been working for a long time as you spend a bunch of years and development making your game down and the ship it was live for for a while and of course as you all know this is changing shorter time in production longer time you live in light and ideally no end of life so this new era where we're Triple A games are live longer than they were in development is pretty interesting it has a couple of consequences for example few developers stay on a project from the beginning to the end simply because people have different interests some people want to build the foundations of games some people want to maintain it etc and what happens then is that processes or what can happen as our processes tend to be even worse strict and formal to

compensate the natural knowledge loss that you have and we really want to break this chain of complexity we want to focus our attention on one single unified workflow where we define processes in the most strict and formal way we know how and that is with code instead of with people so if we take one workflow what we really do is that we start looking at the ideal light workflow and then we work back first from there and and we try to apply that everywhere and that means that we have a single code path for all features a single way of the bug in your game single way of monitoring your game deploying your game and testing your game and this is all defined in code with automation by default so let's talk a bit about this infrastructure as code

in practice what we do is that we're using terraform to script the complete infrastructure and we're using Google cloud built to build virtual machines and when we do this we needed to embrace infrastructure on demand which was a big shift for us since we relied completely on on on-prem it just couple of years back so start and discard is what we call the set of tools and principles that steer this attention to one workflow where you start complete live like environments you use them whatever person need and you can start how many you want every single team member can start their own complete live like environment if they want to and if they need that for testing and then of course you discard it once you're done with it so what I mean done with a live

like environment well it's actually it is a copy of the real infrastructure it is defined in code so it's easy to copy it's easy to instantiate and in the cloud so you boot it up with the exact same auto scaling rules exciting firewall rules etc and the exact same way to deploy debug your monitor your system as an example if a gameplay programmer wants to add a dashboard to graph an ax to monitor a a gameplay feature they would check in that feature and the dashboard at the same time in the codebase and then at every start installed environment thereafter they're able to monitor in real time that specific metrics neither one that they wanted to track regardless of if it's a play test for handful of people or going

live so since this infrastructure done is sets in code it's it can of course be branched and track together with with the code that uses it and it means that we can add 100% we pitiable deployments no special sauce on any of these servers no special cases and everyone can go in and see how it works if a developer wants to add in use that uses a new firewall your new TCP port for example or UDP port for that matter and you'd go into the infrastructure open up the firewall for that port as needed and shake it in together with their code and have it submitted for review and and it's all it's all done by by a single person can be so in order to to do this as you

understand by now we need to have a cloud native mentality and we use 100% Google Cloud currently here at massive and and and I guess some of you will react with oh using 100% cloud of any type of cloud is there expensive and and yes it can be and but typically it is if you move all of your on-prem and just pull up up into the cloud that's not exactly how we how we do it we actually cover later because once we are inside Google Cloud it makes sense to use some other on-demand services and in one case for example we replace the subsystem that we have built ourselves and with on-demand services on Google cloud and that saved 90% of the operating costs

for that particular system and furthermore since this particular system then and any others like it or on-demand there is no overhead cost of actually letting the entire team boot up their own complete live like environment because you don't pay until you actually use it so that's a pretty nice nice benefit so we we test what we hip and we ship or to test one thing that we and that we did was that we moved say games to Google Sarge I mentioned early in the past we had we had this Sun just made more sense to put it in groups now storage and earned it and have it there furthermore we have

smaller game servers so that we can fill them with human players at every single play test and then of course with with design and technology invented or inventions we hide the fact from players that it's a smaller game service it's not necessarily something that you would notice but but it means that we can have more relevant testing oh everything every single time we do it furthermore has a lot of nicer benefits it's easier and faster to scale down when you have fewer players to wait for it that leave this orbital everything up latest utilizes the same tech from deployment through monitoring and debugging as midlane live game does and the process done as I mentioned already it devolves along with the code and and we can

branch it which I find pretty cool and knowledge transfer is improved with new hires they can get the entire complete live experience in the safe environment we can break stuff and have them and have them troubleshoot it or they can break stuff just exploring how how that works and then discard it if they want to start up a new one we start learn under proper live like situations so every play test is then a dress rehearsal for being light and and assess the full path from build system deployment live monitoring debugging but but we can also do some pretty creative testing that we were not able to do in the past for example we can simulate Australia or rather we can simulate how

it feels for Australian players normally what do we do is that you add latency to the client and you play Justin you see how try to get an idea how it feels to play the game in Australia it's not really feasible to some depth across the globe because the test game so it's normally how will you do it but but a complex mashup services all have different agencies so it can actually be quite complex to simulate Australia in a proper way and and we can actually test better than that so let's take a look a lot here we have a classic setup and core service service in Europe and there's some third-party services in the US and game servers in Australia and and

then the players are normally here and in Australia we'll start on this card which we have defined in code we can easily flip the table put all the course servers in Australia and the game servers nearby our office because then we get a true feeling for the Australian experience and since this is sitting completely inside Google Cloud it's literally five minutes to change so it's very easy to test this out so yeah so maybe I actually want to get bad never mind the animations Telesat and then of course they are the same dashboards available as we had before and now we can see the effect in these dashboards on top of playing the game and feeling the effect so so hopefully the

experience for Australians and and others are improved but there will still always be some special cases and what in the third party services that I mentioned in this example we always have third party dependencies for for multiple to do reasons it may be for example telemetry or tracking so the way we deal with this is that where they had a dedicated staging and a dedicated program moment we have a proxies Isana proxies that sit in between our start and guard environments and the internet if you will and where we proxy the traffic to and this means that all all of our start and discard environments at

the same view outside the emitter we cannot control the inside so of the third-party infrastructure but it's still hidden it's a it's a nice compromising and it goes to show that we really try to minimize the special cases in our infrastructure as much as possible yeah so what would you do if you were to if you want to embrace start and discard what you need to stay clear of special cases and you should use a single code battle zone but but at a fundamental level it's a shift in mentality that's supported by technology it's not purely technical solutions and cloud native and on demand is critical for the success of it and in order to to

be at a cost parity or sheep earn an on-prem hosting and but remember that if a programmer needs a special case for feature X then you're probably doing it wrong there hopefully you can find another way obviously new code paths we do it the same way as when you're alive and then stick with that it's worth doing not help to maintain multiple parallel code paths and if you also plan to hand over to a live team when you're done yeah when you're shipping the game then you're also probably doing it wrong because as you see here live is truly part of development and we are try to be live from day one of any new game but

but then what's what's may be next in store or what's were you my challenge to you guys why stop here and because what you can do as a logical next step is that you can define all of the infrastructure in encode do you need a port opened and you can do it yourself and submit exchange for review so in a way you can open source office IT you could have lower and upfront costs by using compute or at lower cost when it by using compute we're in half where it happens to be cheapest at the moment anywhere around the globe there's spot pricing etc that is way cheaper than normal and you may use that for your entire build pipeline for example furthermore yeah we all know that the stadia works great for games so of course and you would be able to

stream your level editor so you could have a workstation in the cloud and be able to upgrade it and with new and more memory etc as you need and working from home with an oar could then be the norm rather than the exception where you pretty much just need to fork the code in order to integrate the new dev team but that would be like all games 2d 2.0 and that is the subject for another tour than this one that's what's my presentation about start and discard here at use of massive thank you and stay safe you