Stress-Free Game Development: Powering Up Your Studio With DevOps

[Music] game-development sucks it's hard and it's stressful our industry is notorious for queuing people up and spitting them out as burnt-out husks so much so that those of us who've been around for a while are called veterans as if we've been at war do you know what they call experienced people in other industries experts are we experts at what we do what is it that we do and if we call ourselves experts what are we actually experts in are we experts in working 80 hours a week are we experts in dropping a huge day one patch to fix the thousands of bugs that we missed during testing what about losing sleep for a week because you're panicking about server outages and game client crashes are we experts at that I don't know about you guys but I'm an expert at

those things because I've done all those things a lot but I don't want to be an expert at losing sleep or burning myself out or putting out a thousand hot fixes for a month after launching a game I want to be good at making games losing sleep and pulling heroics and crunching that's not making games in fact that's what I would call that not making games but what if game development didn't suck what if it was easy imagine if your team could deploy a major patch and take the day off to celebrate because you knew everything was going to be fine what if you could deliver a major content update to your game every week without working overtime or if you could take your weekends to actually play games well I have news for you you can do that I know you can do that because I did it I'm doing it right now and if you

want to know how good because that's why I'm here my name is Seth Koster and I'm the CEO game programmer KO game designer and co-founder of butterscotch shenanigans and today we're going to go on a journey that's many years ago when our studio was flailing around crunching putting out fires and burning ourselves out I'm gonna walk you through some of those struggles and how we learned to use the three ways of DevOps to change everything about how we work we'll talk about QA testing production bottlenecks reducing waste small batch deployments scaling up your team and more and by the end of this talk you'll be able to use the power of DevOps to make your studio or even solo operation run like a clockwork machine our story starts in December 2015 when our studio was three years old and was comprised of my two brothers and me we had Sam as the artist Adam as the web developer and I was the game

programmer we had launched four games up to this point and they had varied in success ranging from extremely bad to decent but this time we thought we had a real winner on our hands we were building a huge open world crafting based action RPG called crash lands we had been working on it for two years and it had truly been a grind but now at the end of 2015 we could finally see the light at the end of the tunnel at this point we were feeling pretty good about the state of the game and as far as we were concerned it was ready to ship but just to really make sure it stuck the landing we thought let's do a little beta test you know just to shave off the rough edges so we reached out to our community and we enrolled 150 players in a closed beta test immediately we were hit by an avalanche of bug reports I mean broken quests crashes missing recipes people saves getting deleted anything that you could think of that

could go wrong went wrong and suddenly we found ourselves staring down the barrel of over 2,200 unique bug reports and we had eight weeks to launch so we did what everyone does we crunched we put in 60 hours during the first three days of the beta and fixed as many of the hugely problematic issues that we could and we hardly slept for the next two months Lee to launch as we tried to wrangle the remaining 2,000 issues so we weren't able to fix everything but we did the best we could and then we launched and the game was well-received so in the first week we had about a hundred thousand players across our three platforms steam the App Store and Google Play so this was great news but it also had a dark side because those hundred thousand players crushed us with another enormous wave of bug reports some of these bugs were repeats of the issues

that we weren't able to fix earlier during the beta while others were completely new bugs some of which we had introduced during our sleep-deprived crunch on the death march toward launch so we did what everybody does we crunched again and instead of being able to enjoy the launch or even get started on some new interesting content patch we spent two months putting out fires scrambling to get a patch out just to repair our broken game so eventually we did get that patch out and the game became relatively stable so it's time to move on to the next phase content updates we worked on some new features for a couple months and we were excited to get our first content patch out the door but by this point some things had changed it had been just long enough since we had deployed a patch that none of our builds were working so we had to get everything back up to date download SDKs and test builds and finally weeks

later we released the patch and this time everything went perfectly just kidding just like before we were hit by another wave of bug reports which we crunched again to fix we carried on like this throughout 2016 and we managed to create only two content patches the entire year with each patch we released it was like some kind of nightmarish Groundhog Day scenario where we had to relive all the same stresses all over again crunch heroics putting out fires just generally being in a panic in the midst of all that chaos we thought this is too hard so we did what everybody does we assumed that our troubles were simply due to us being short-handed and we hired more people scaling up from three to seven we immediately had a difficult time integrating these new people into our production processes because it turned out we didn't really have any processes almost everything that

happened in the studio was ad-hoc without any concrete systems and the three of us founders were doing so many things that we weren't able to take the time to create any kind of coherent onboarding or training structure so essentially we brought new people into our studio and then just kind of set them adrift this was extremely stressful for them because even though they wanted to contribute and help they couldn't insert themselves into this weird shape-shifting amoeba of game studio that they found themselves in and worse when they did get involved it backfired one of our hires was an artist who we brought on to assist Sam in making content for crash-lands and whatever our next game project was going to be and she was able to produce a pretty good amount of content which became a problem everything the art team created of course had to be brought into the game and as the Soul game programmer I was suddenly responsible for implementing

twice as much art this left a lot less time for developing game systems and we ended up struggling to figure out how to make reasonable progress on the next game so the art team was getting demoralized because they were creating all these assets and those assets were just lying around not getting used because I couldn't keep up so we thought maybe we should take a look at the processes by which we add our to the game and perhaps we can find some way to make that process faster so that's what we did at that time our art process looked like this the artists would create an asset using Inkscape and they would manually export it at some point later I would find that asset in Dropbox and manually import it which I would then review for errors and if something was wrong with the art asset you know like a technical problem then I would send a note back to the art team about what needed to be fixed so that artists would then fix it and re-export

it and the loop begins again and eventually once the asset was finally usable than I would implement it so to speed up this loop we decided to focus on this part here the import step if I didn't have to individually pull files into the game project by hand then I would save a bunch of time which I could then use that time to implement art so Adam our web developer took over this project and soon he had built a program called the ink Bob the ink bot allowed me to mass import all of the new and updated assets that the art team had created so now I can import hundreds of assets in seconds instead of one asset every few minutes which is pretty sweet right except for one important problem for the ink bot to work the assets had to be standardized so that the ink about would know where to look and how to import them into the game and to

standardize those assets Adam also automated the export side of the process so whenever our artists would finish a creation the ink bot would just pump those art assets right out the end result was that while it was faster for me to bring assets into the game it was even faster for the art team to create new assets so we ended up with an even bigger pile of unused art the art team became even more demoralized and I became even more stressed all right so we carried on like this for a year tweaking our processes and making things both worse and better at the same time and constantly shrinking our cash run away without much to show for it in terms of new projects and by the time 2018 rolled around everything buckled we knew this wasn't working and we had to change something so that we could move forward so we

scaled our team down from seven down to four and we scrapped the game that we had been working on at that time because the game had also become unwieldy due to our out-of-control processes so with our scaled-down team we started to work on a new game called level head the vision of level head was to capture the magic of something like the original Mario maker which is a game where people build levels and then share them with each other but we were gonna do it as a fresh new IP with new and interesting mechanics a better social system and make it available on way more platforms the first year of level heads development was pretty much just like our crash-lands development lots of hard work and brute force and within a year level head was far enough along that we started to think about launch what was the launch gonna be like and we thought back to the crash-lands days and we realized that level head was going down the same path launching and maintaining

this game was gonna be painful and expensive and difficult the game wasn't being actively tested so it was probably full of bugs and we were gonna have to crunch and panic and go through all the same motions all over again so if we wanted to make this game right something was going to have to change as part of our research to find a better way to do our work we started learning about something called DevOps and we were inspired over the next year we started applying DevOps principles to our work and we turned the whole ship around we were able to put level head into early access and maintain a 97% positive review score on Steam while deploying two to three content patches per month to our players on average it's so easy for us now to make deployments that we deploy up to four matches to our new internal QA team per day and level had also now supports 11 languages and adding a new language

takes almost no development time in the background of all this we've developed stable builds of level head on Xbox Android iOS switch steam and uwp and we're gonna launch on all six of those platforms simultaneously and we even scaled our team back up to six people without any problems and with all of that going on we're not crunching we're working regular work weeks and we take weekends off like normal people we get to have hobbies outside of work during our free time and we even take vacations but I hope that piques your interest because the rest of this talk is about how we managed to pull all this off over the past year as I mentioned earlier we were able to do all this through applying DevOps principles to our work so let's jump right into DevOps first some definitions DevOps is a set of operating principles that focuses on faster deployment cycles high stability

and reliability of your product and better communication between teams it recognizes that there's a mutually beneficial relationship between development the creation of your software and operations the deployment and management of the software DevOps is broken up into three parts called the three ways and each way is essentially a different lens for you to use to get a better perspective on your work the three ways are managing the flow of work amplifying feedback loops and continuous learning then so we're gonna go through all these first let's jump right in to the first way managing the flow of work the first way is all about understanding and then managing the flow of work through the company ensuring that the work flows in one direction from development to operations to the customer the core tools of the first way are to eliminate waste make the work

visible deliver the work in small batches prevent defects from moving downstream and align everything toward the global goals we're gonna start by talking about waste because understanding waste is the pillar that holds up pretty much everything else in the box so first what do I mean when I say waste waste is anything that you're doing that isn't delivering value to players in other words waste is not making games waste comes in all shapes and sizes and it's the silent killer of your production process because it's always a lot worse than you think and it tends to be self-perpetuating because waste creates more waste we're gonna go through all the types of waste that you will come across and as I talk about these I want you to think about ways in which your own processes might be creating these types of waste so here's a little chart that I call the circle of

waste as I explain each type of waste this chart is going to become increasingly tangled to show how all these different kinds of ways to have spillover effects in ways that you might not anticipate so let's start with motion there the top left motion is waste produced by the movement of materials or information between people or departments so think about this situation that might sound familiar you need something from someone so you send them an email but soon you end up emailing back and forth a dozen times trying to communicate what you need and eventually a week later you just have a meeting instead that's motion and motion is waste because the whole time you were doing all that you weren't delivering value to your players and even worse that process produces a lot of waiting and task switching to cut down on motion you need to create self-service systems like good documentation so that people can

treif information or materials themselves when they need it instead of requiring somebody else to bring that information to them task switching the next type of waste is when someone has to move between various contexts like perhaps going from a meeting to production to back to a meeting than to launch then back to work then another meeting not only does task switching make things take longer which causes waiting all that mental exhaustion from context switching read leads to excessive creation of defects task switching can be minimized by batching work as much as possible so for example save your meetings until the end of the day and check your email in 1 or 2 big bursts during the day instead of constantly throughout the day so there's tons of ways to batch your work and even doing it a little bit even just yourself will go a long way next up is defense a

defect is when something that you have produced has something wrong with it making it unusable by your downstream customer and by downstream customer I mean just the next person to receive that thing so maybe it's an art asset that's the wrong size or a sound file in the wrong format or a blog post full of typos when someone runs into a defect they have to stop what they're doing and send it back to get it fixed which creates motion waste this forces them to tasks which because they planned on doing one thing but now they're responding to an error instead and this also leaves that person with an unfinished project while they wait for the defect to be resolved it's even worse when a defect makes it all the way to a player because if it's bad enough like a gamebreaking crash then you have to engage in heroics like pulling all-nighters to deploy an emergency hotfix to cut down on defects you need to fix the processes that create them in the first

place which we'll cover in a little while next up is waiting waiting is when something is supposed to happen but it doesn't waiting is obviously waste because it's basically just standing around but it also leads to other problems when you're stuck waiting for something you will have a tendency to not work to start to not start work on a project because the thing you're waiting for could arrive at any moment so either you're gonna keep waiting or you're gonna start working on a project only to have two tasks switch away from it to pick up the thing that you're waiting for which then leaves that project unfinished so to cut down on waiting you should try to deliver work continuously in small batches which we will cover in a bit alright so unfinished projects is next speaking let's talk about a scenario that's probably all too familiar to my fellow game developers so you start by working on a feature in a game and it's a really

big feature it's taking long enough that other demands on your time starting to pile up and eventually you have to put that big feature on hold and it's it's off to the side maybe on a git branch waiting to be picked up again months go by and pretty soon you realize that if you were gonna pick that feature up it's gonna be full of bugs and it's gonna cause a huge amount of problems so it's better to either abandon that idea or just start over that's an unfinished project what a project gets left unfinished even for a short time the context of the rest of the world changes and when you pick it back up there's a high likelihood of creating defects as you try to bring the project up to speed with everything else and you might even need to pull heroics to get that project finished to avoid this break projects down into the smallest deliverable phases so that the project is always in some kind of finished and usable State even if that

state isn't fully ideal all right next up is manual processes a manual process is something that is being done by hand even though it's done the same way every time the worst part about manual processes is that they tend to create defects because people get tired and distracted and they make mistakes by definition a manual process also demands task switching because someone has to stop doing something else in order to do the manual process and this also may create motion because people may have to communicate about the manual process this also makes it difficult to scale up your operation because to do more manual processes you need more people and this takes longer than necessary which also creates waiting if you do have manual processes in your organization you need to think about ways you can automate those so that your people can be freed up to deliver value to your players in better

ways extra processes on the other hand are just things that you're doing but you don't need to be doing they lead to the exact same kinds of waste as manual processes for the same reasons but the solution is different with extra processes the solution is to just stop doing them in our studio we use the focus in question can we not to cut through the waste of extra processes an extra feature is something in your game that doesn't need to be there every feature adds complexity to your game and complexity is where defects emerge when deciding whether to add a feature to your game whether it's a super sophisticated AI a new game system or just an expansion of an existing game system you should always be vigilant about whether that feature really pushes the vision of the game in a way that players will care about because if not you're just gonna make things way harder

for yourself without delivering value to your players the best way to deal with extra features is of course to not have them but if you do have them already see if you can cut them alright so this is a mess but don't worry it gets worse because last I want to talk about heroics heroics are when you have to perform far above and beyond what's reasonable in order to deliver a required result like crunching to get a game ready for launch once you reach the point of pulling heroics you've fallen into a dangerous feedback loop because when you're in heroic panic mode cutting corners and moving as fast as you can on very little sleep you will create more and more of almost every other kind of waste heroics are the worst kind of waste because they are cancerous they're self-perpetuating and they consume everything else you're

gonna pull heroics because your processes have broken down but heroics also make all of your processes break down leading to more heroics so I want to hone in on this because heroics are the signature thing that the games industry is known for every single week you can find an article about some studio big or small crunching their employees in 70 to 100 hour work weeks and we Indies are no better because we just crunch ourselves and then we pat ourselves on the back because we're crunching by choice sticking it to the man I think we should all take a moment though to appreciate what this means heroics are a signal that all of your systems have failed this means that our industry is best known for being really bad at our jobs and we're stuck we're trapped in this heroics loop so how do you get out the

only way to break that loop is to completely stop production and take a step back to rebuild your processes and you need to get every person on your team on board for this because this is everyone's problem so that's what we did we totally stopped production for several weeks and set out to reimagine how our studio would function so our first order of business according to the first way was making our work visible so that we could find all the places where we were creating waste to do this we decided to have twice weekly production meetings where we would all agree on what was going to happen in the next roughly 16 hours of work because the Sprint's were short the meetings were also short which allowed us to be very responsive and iterative and we were able to have such short Sprint's because our team was so small larger teams may

end up needing to do a bit more planning and have longer sprints but this would this is what worked for us so we all transitioned to using Trello and we came up with a structure for managing our Trello boards that would allow the boards to guide the flow of work in a way that we could easily see so we gave our boards five lists in box to do doing which is the current sprint testing and done and we gave these lists a few important rules for production planning the first rule is that new tasks should always go into the inbox first they can't be put into any other list and each production meeting we evaluate the new tasks in the inbox and decide their priority and this meant that you knew exactly what your next two days of work were gonna look like and you wouldn't get derailed by some random unplanned thing being dropped into your lap because all the new stuff just went

into the inbox instead of at the top of your current pile the second rule is that time is a constraint only 16 hours of work could be handed to any one person during that sprint and if we gave someone 20 or 30 hours it meant that we were demanding that person to crunch which going forward we were no longer going to do and refusing to crunch is the core way for us to bake quality into the production pipeline and the planning process the third rule is that all work items have to go through the whole flow including the testing step the testing step creates an explicit requirement that you verify the thing that you just finished is free of defects before you send it to the next person this keeps work from flowing backwards because errors get fixed at the source instead of being found out later so as a practical example of how this flow works we can see how this testing requirement

allowed us to identify and fix a major problem in our art asset pipeline so under the old process Sam would create a bunch of assets the Inc bot would export them and I'd bring them into the game and then I would review them for errors and when I found an error I would have to send it back and that's a defect so we would have defects then coming to me and then flowing backwards and going back upstream but let's take a look at what happens if we take the workflow from our Trello board and apply it to the art asset pipeline but as normal Sam creates an asset and then the Inc bot exports it but now he has this testing step in here so instead of just immediately sending the asset to me now he is the one responsible for reviewing it for errors and if he finds an error at this point it's easy to fix because he just finished making it so

he's already got all the files to open and everything and since he is now getting hit by his own errors in real time he's able to update his processes to cut down on his error rate so when he first started tracking his errors using this process he discovered that his error rate was actually 40 percent but with all the fixes that he implemented into his process his error rate is now closer to 2% which means by the time I get the assets I can usually just go ahead and implement them without any extra work so if this small update Sam spends a lot less time fixing errors because now he makes them a lot less frequently and if he does make an error he's not making it anyone else's problem most importantly he's not making these errors my problem and something we learned very quickly once we started making our work visible is that if something is my problem it's everyone's problem because I am the studios

production bottleneck what's a bottleneck I'm glad you asked a bottleneck is part of a process that's receiving more work than it can handle and every production process has at least one bottleneck it turns out that understanding where your production bottlenecks are is super important so let's take a look at our art pipeline example in our case we had Sam over here able to produce three are assets per hour as the game programmer I was only able to implement one asset per hour on average because I'm programming I have a bunch of other stuff that I'm working on - I'm working on game systems fixing bugs developing new features all that stuff so in this case I'm the bottleneck for art and it doesn't really matter that Sam is producing three assets per hour because our process makes it so that the game

will still only get one asset implemented per hour so the total output of our team is not determined by Sam it's determined by me the bottleneck so when Sam started doing his own testing it cut down on the rate that he was able to produce art because now he had to spend extra time testing his art but he took that testing task from me which actually saved me a bunch of time which I could use to implement the art so with that one change we were able to get twice as much art into the game in the same amount of time so just to reiterate this by using the same people with no extra work hours and simply moving a single task from one person to another we doubled the amount and pace of art that we could get into the game which

brings me to the golden rule of workflow improvements the only way to increase your team's output is to increase the capacity of the bottleneck to move work through the system if you're making changes anywhere else other than the bottleneck you're either making things worse or you're just not making them better so once we got our work flow under control we were ready to tackle the next bottleneck in our production process deployments by deployments I just mean making builds of the game and getting those builds delivered to players deployments had been a problem since day one but because of how painful our deployment process was we had been avoiding them but one of the core lessons of DevOps is if a process is painful you should do it more not less if you don't confront the process you can't learn about it so you can't fix the thing that's making it so difficult

so for us then the ultimate goal was to have continuous deployments of our games in other words it would be ideal if every time we made one single tiny change to our game we could deliver that change to QA in a fully compiled build for testing having continuous deployments brings a huge number of benefits it allows you to detect problems immediately after they're created instead of weeks later and if you do release a patch with very few changes the likelihood of that patch containing some kind of catastrophic problem is quite low and if there was some huge problem it's very easy to track it down because hardly anything has changed so you can easily pinpoint where to look and last the more deployments you do the faster you find problems in your deployment process so you can fix your deployment process as well so this is the ideal but for us our

deployments were anything but continuous just before we learned about DevOps we had hired a few part-time QA testers and we were making builds for them once per week the process of the time looked like this at the end of the day every Thursday I would write up and compile all the week's changes into patch notes in a Google Doc which would take about a half hour then I would make a build and verify that the build was working so that process could take me 30 minutes to an hour and if the build was good I could then send it to the testers by putting it into Dropbox and if the build wasn't good then I would have to make another build and start the whole process over the testers would then go ahead and make a checklist out of the patch notes and they would begin testing and going through that checklist if they

found problems they would report those problems to me which I would then review and if the issue was something that needed fixing I would move that issue into Trello and then I would fix it for the but this whole process was a one-week and this was only for Windows builds if we wanted to make builds for other platforms then each additional platform would take me another 30 to 60 minutes to deploy to so at this point the rest of our intended target platforms were completely untested so we've set out to update this process with a new automatic deployment pipeline for our games which we would call the game pipe we had a lot of really big visions for what the game pipe could do but if we did them all it would take months before we had the game pipe up and running so instead we asked ourselves what would be the simplest version of the game pipe that we could

start using immediately so this is back to that small batch delivery of work so out of our tools developer took on this problem and he created the first working version of the game pipe in about a week and this first draft of the game pipe whenever I push to get commit to the game's master branch the game pipe using a bit bucket web hook would create a Windows build of the game and then deploy that build into Dropbox where our testers could then access it so this is very simple but even though this version of the game pipe was about as minimalist as it could get we found that it solved a bunch of our problems right away so first my time was no longer being used to create builds so I could just keep developing while the build machine was just churning away compiling builds I also no longer had to manually track down the builds and put them into

Dropbox and deliver them to the testers the game pipe was now doing this automatically so this saved me a bunch of time it saved me task switching and it cut down on the chance that I would mess the process up and then create defects at last environment configuration was no longer a problem so all this means is that no matter what we can ensure that all the builds always come out the same because they're always using the same versions of the same SDKs because it's always being built on the same machine no matter what machine we triggered the build from and just like that we were able to go from making builds once per week to making them once per day so we rescheduled our QA team to spread them out throughout the week instead of coming in just on a Fridays and so whenever we made a change to the game it would be tested within about 24 hours but our goal was continuous deployments and once per day

isn't continuous we weren't yet able to make the leap to continuous deployment because there was still one manual step left in the deployment process writing patch notes fortunately there was a solution right under our noses because every time I made a change to the game I would write a get commit message explaining the change but if I'm writing get commitments for every change and a patch note for every change that I'm just doing the same thing twice so to cut down on this redundant manual work Adam built a versioning system into the game pipe that would automatically convert my git commits into patch notes and then it would post those patch notes to our website so for example if I wrote this git commit campaign three new levels added to the end of the campaign blah blah blah then it would get automatically converted into this on the

patch notes so with this patch note delivery system up and running we found a few huge benefits first and most importantly this fully eliminated the manual creation of patch notes which finally did allow us to make continuous deployments we could make deployments at any time second since the patch notes were generated out of my git commits it was no longer possible for me to accidentally leave something out of the patch notes so that cut down on defects and made our testing more reliable and the third benefit is that because my git commits were now publicly readable by other people outside of the programming team it forced me to be much more clear and deliberate about my git commits so in the long run it became far easier to navigate our change logs so now if we look at our deployment pipeline you can

see that we've fully automated all of the manual parts of getting builds made and delivered to QA so all the automated parts of the the gold boxes there and over time our platform specialists sure took over the game pipe project and and he worked with Adam to continually improve the game pipe piece by piece so instead of deploying to Dropbox we got it to upload directly to an internal staging branch on state so that meant that our testers would always have the latest version of the game automatically downloaded later we upgraded it to deploy builds directly to all of the rest of our platforms like Google Play and the App Store sure and Adam also created an automatic localization pipeline and connected it to the game pipe making it trivially easy for us to add new text and translations with every new version so by this point things were looking pretty good the game pipe had

turned a fully manual process that would take up to an hour each time into an automated process that would only take about five seconds to trigger we went from weekly testing to continuous testing which gave us the fast feedback that we needed to rapidly iterate on the game and it allowed us to more easily track down bugs so this caused the builds of the game to become more stable and most importantly patching the game became incredibly easy instead of something that we had to agonize for weeks the most most important thing I hope that you all take away from this game pipe story is that for starters you should have an automatic deployment pipeline but also it doesn't have to be perfect make something simple that you can use and start using it right away because once you're using it you can then iterate on it to fit with your processes and eventually you will get it

doing the things that you need it to do so with our new continuous deployments we wanted to make sure that we had quick testing on every patch this meant opening up jobs for new part-time QA testers to the week which meant we had to hire new people so we had to screen them interview them and if we brought them in training so this started taking up quite a bit of time which was putting a strain on our production we needed an Operations Manager someone to take charge of the business side of things fortunately we knew someone who had been doing operations management work for quite some time and was looking for a new job which happened to be my wife tough enough she was our first hire since the studio had collapsed the year before so we were determined to ensure that her onboarding and training in the studio when as smoothly as possible and that she was able to contribute immediately now we already had a process

in place that was making work run really smooth and easily which was our production meetings so we took that same exact framework and we created a training Trello board we populated that board with a list of training tasks with time estimates and some of these were solo like reading certain books while others were collaborative like one-on-one interviews with studio members and the goal of each of these tasks was to generate knowledge and insight about how the studio works and what her role was inside the studio twice per week we had production meetings to review what she had done in her training and talk through the next training sprint and every sprint we had her tasks broken down by hours so that we never overloaded and she always knew exactly what she was gonna be working on and why because of this approach she was immediately able to add value by taking over several business aspects of the studio and she quickly built up the context of how the

studio fit into the games industry at large despite not coming from a gaming background we refined this process further based on some bonus feedback and we later hired a new full-time QA tester Jordan and just like with Sampada his onboarding was simple and stress-free for all parties and all of this is thanks to the process of making the work visible so we can easily spot problems give guidance set clear expectations and receive feedback from our new hires with samba de and Jordan on the team we increased our capacity in places where it mattered we hired for roles that were specifically meant to relieve pressure on our production bottlenecks and it worked so before we move on let's recap on the first way of DevOps and how we applied it to our situation remember the first way is all about understanding and then managing the flow of work through the company ensuring that the work flows in one direction from development to operations to the customer the core

tools of the first way are to eliminate waste and make the work visible and we did this by building new processes to guide test and review what work was being done through our production meetings and Trello rules to deliver work in small batches we created the game pipe and we were able to continuously deploy patches and changes to our QA team to prevent defects from moving downstream we updated our workflows to move testing closer to the source so we could fix our problems before handing them off like what we did with our art pipeline and last to align everything toward the global goals our goal is to deliver high quality games to our players and we do that by adding staff and developing tools to ensure that the game development team is to work unimpeded and that our bottle XR clear second way is about amplifying feedback loops so the second way says that while the work is flowing in one direction we want feedback to flow in

the other direction feedback is what allows us to catch problems improve our processes and build quality into the production pipeline and to get that feedback we need to develop tools and systems to allow the people downstream from us to report problems so by this point we did have our QA process giving us continuous feedback on our deployments so the question then was how do we amplify that loop to get better and faster feedback on the things that we're doing to answer this question we took a look at our QA testing pipeline and we discovered that if we ramped up the speed and volume of reports coming from our QA team we would be unable to handle it because we still had this process every report had to be manually turned into a Trello card before it can be dealt with although this only took a few minutes per card it was a manual process which meant that

it created tons of task switching motion and defects and because of all that waste a few minutes per card can easily turn into five or even 10 minutes per card as the volume of cards increases because fatigue sets in and I would become increasingly distractible trying to process 60 cards in one go so we used the Trello API to make it so that anyone reviewing an issue can just click a button to auto convert the issue into a Trello card and all of the file attachments tags comments and relevant information would just be baked into the card and we can then review that issue at the next production meeting so that was it just like with the game pipe a cumbersome process that would take many minutes or more per iteration became an automatic process that would only take seconds so now that we were able to review and process feedback

faster and easier we could start looking for a way to amplify how our QA testers delivered their feedback under the current system our testers were manually making checklists of what to test by looking at the patch notes and it took a lot of time out of testing for them and it also caused them to sometimes miss things because this was a manual process and there was nothing keeping track of all these things automatically so the checklist creation step was clearly the bottleneck for everything else in the QA workflow if we could alleviate that bottleneck then our testers would be able to spend more time testing and they would be able to give us more and better feedback since we already had automated patch notes we upgraded that system to turn the patch notes themselves directly into a QA testing checklist under the new system each tester would see little check marks next to every change in the patch notes and when they had tested something they

could simply check that change off so now the testers didn't have to spend any time compiling a checklist at the start of their testing session because they could just jump right in and on the back end of this system when I log in as a developer instead of seeing checkmarks I would see a number next to each change which would tell me how many testers have tested each change so this lets us know whether we're missing test coverage on a particular issue which we could then deal with that more directly so that brings us to the current version of our patch deployment and QA testing pipeline which now looks like this as you can see there are more automatic processes than manual ones and we only have people doing the things that only people can do with all those changes we were suddenly getting better and more feedback without being overwhelmed we knew what was being tested and our

testers were able to each add a lot more value to the process so that rest of the second way amplifying feedback loops the goal of collecting feedback is to allow you to catch problems improve processes build quality into the pipeline developing tools and improving the means by which you receive feedback will allow you to get better feedback faster which lets you make better products but before you amplify the feedback loops make sure that you have systems in place for how to handle that feedback so with all those changes we made to our deployment pipeline and feedback systems we could tell that everything was moving along much faster things were easier and things were cleaner but DevOps is all about continuous improvement not just one big improvement and that brings us to the third way continuous learning your studio is always changing and the world is always changing around it if you want to truly reach your potential

and get the most out of your work you need to be able to adapt improve and grow and you need to build that philosophy into everything that you do to do that you need to ensure that everyone on your team has the right to experiment and try to figure out better and faster ways to improve the company processes people need to be able to offer up sometimes risky ideas and a management needs to be able to hear those out without just striking them down out of hand which means you need to create a work culture that reinforces psychological safety where people know that they're gonna get a fair shake and where they will always speak up about problems but culture doesn't emerge in a vacuum called collection of practices and beliefs that people develop in response to their circumstances and what DevOps has taught us is that your work culture is a direct result of the structures and processes that you use in the workplace so we had

to ask ourselves what structures do we have in place and how do those structures affect our learning our culture of learning and growth although we were doing twice-weekly production meetings we realized that those would only get us so far they're too narrowly focused on specific projects and they're all about just doing the work so we needed a higher level way to evaluate and make changes to the studios processes as a whole for us this meant creating a formalized quarterly review process every quarter we have our team members fill out a questionnaire to identify how they're feeling about their work and ways in which they believe that their time or their talents might be getting wasted we use those responses plus a studio waste analysis to make changes to the work processes of the studio as a whole so every time we've done this we have found huge gains and solved big problems with some aspect of how we handle our work whether this

review process is quarterly monthly or somewhere in between the most important part is that there are structured ways to make improvements to all the processes in the company without that structure the evaluation doesn't happen and then the improvements don't happen so that brings us to the end of the three ways so before I tell you what happened after we did all this let's quickly recap the first way is all about mastering the flow of work make sure the work flows forwards watch out for bottlenecks cut down on defects and eliminate waste the second way is all about amplifying feedback loops make sure you're getting the information you need from the people downstream from you so you can make their and your lives better the third way is about creating a culture of continuous learning and experimentation so that you can constantly improve and deliver value to your players it's a good thing that we started down this path when we did because our world

was about to get flipped upside down or at least it would have if we hadn't been prepared by February of 2019 we had been making our DevOps transformation for six weeks we were planning on launching level head around June so we had four months to go then on February 13 we got hit by curveball Nintendo announced that Mario maker 2 was not only in development but it was essentially done and Mario maker 2 would be launching in June this was really bad for us we had started making level head because we sought to fill the void left by the fact that there was no Mario maker on the switch pretty soon that void wouldn't exist and there was no way that we could compete with Nintendo in terms of marketing power so we didn't have any options other than maybe beating Mario maker to launch but how could we do that we looked at across the media landscape

and we discovered that if we wanted to beat Mario maker to the market then pretty much the only available launch date for us would be April 18 2019 two months earlier than planned so we thought let's go for it and better yet let's try to dodge Mario maker entirely by launching on steam instead so on February 13 we completely changed course and settled on a new timeline on March 1st we would do a small closed pre alpha test and we had two weeks to get this put together on March 8th a week later that would be the start of the main alpha test where we would bring in a larger group of players so we could refine the game further and six weeks after that on April 18th we would launch into early access so suddenly instead of having the game in the hands of players in three months we suddenly had to have it in their hands in two weeks

this is the exact kind of situation that would have been nearly impossible before and would have resulted in an incredible amount of crunch Inuk crashes bugfixes made of everything else but because of our updated DevOps workflow we already had stable builds on Steam that had been thoroughly tested so we didn't crunch we just shifted some priorities around a bit to wrap up some final features that we wanted to get in before early access and we just carried on as normal no problem and within two weeks we had players playing the game in our pre-launch testing period and it was fine although our players did find some edge case bugs it was easy for us to quickly patch them up and throughout the five-week pre-launch testing session across thousands of player hours of testing we only had one single crash which was easily fixed tested and deployed that same day on April 18th we

launched into early access we got the Steam pop up that weekend and got a decent influx of players and everything was fine we had no game breaking issues we didn't have to put out an emergency patch we just hit the launch button and then we just got to hang out with our players in discord and chat with them about the game it was actually pretty relaxing so that was 11 months ago to this day we are still doing our continuous deployments and delivering routine content to our players level head is sitting at 97% positive on Steam early access and the game is getting better all the time since we're deploying 2 to 3 content patches to our players every month and every time we send out a patch it's calm it's no big deal and we're not crunching at all even though we're gearing up for a six platform simultaneous launch supporting 11 languages with a team of 6 people and all of this is thanks to our whole

team's willingness to break the loop of heroics take a step back and ask does it really have to be like this and that's the story of how we reboot in our studio I hope what I've shown you all today is that the stresses that we all take for granted as a normal part of game development are not they are the result of bad development practices our industry is caught in a loop but we don't have to be now you might look at all this stuff that I brought up today and think this is a lot you might think that your team can't afford to do this no you can't afford not to do this because you're spending far more time and energy and money than you need to and you're getting worse results that's not to say that making this kind of change is easy but the goal here isn't perfection its improvement the most

important thing is that you start down this road because as soon as you take even one single step toward applying DevOps principles to your work you'll see immediate benefits and when you do we'll be there on the other side ready to meet you where we can all just kick back relax and make great games for our players without setting ourselves on fire thank you all for your time I hope this was useful for you to give you some resources to pursue as you go down the path of DevOps I've put together a little cheat sheet which has all the core points plus some books for you to check out I definitely recommend starting by reading the Phoenix project and if you have any questions you can feel free to just shoot me an email enjoy and good luck out there [Music]