Adopting Continuous Delivery (In Sea of Thieves)

hello my name is Jeff assault ani I'm lead software engineer working for air game studio we are part of Microsoft and today I'm gonna talk about where is Jamie in adopting continuous delivery for our latest title called sea of thieves is there anybody here who doesn't work in games industry okay just a couple so this slide should be familiar to most of you I'll quickly go through that it's games are developed as monolithic applications in C++ and we use three main phases and we use waterfall process to develop games we have a pre-production or prototyping phase when the size of the team is quite small and the purpose is to come up with the idea about the game come up about the story define what the game is about prove out the technology and come up with the roadmap for the game and once we get the funding and greenlight we go

into production phase that's where the size of the team increases significantly and the main purpose here is to develop as many features as possible that we defined earlier on that can last sometimes in one to two years and once that's finished we going to bug fixing phase that's where we're getting close to the release date and we can't really move that and we had the same time we face mountain of bugs maybe about 10,000 bucks that we have to now go and fix for the release date and everybody's point everybody crunches everybody work long hours it's quite stressful people make a lot of sacrifices to make sure game is out on time at high quality and at this stage we heavily rely on manual testing to prove that game works and catch bugs for us and usually for Triple A games we have one big release followed by handful

of updates and the main team moves on to the next project and they don't have to worry about maintainability of the codebase could you raise your hand if you familiar with your thieves or you place your thieves good so sea of thieves is a multiplayer cooperative adventure game that we are working on a rare and the idea is that there is no predefined story there are emergent stories that develop as a result of simple mechanics in the game when you go and form a crew with your friends or with strangers and you go on an adventure for this game we decided to change our business model and develop this game as a service the difference between developing this game as a service compared to traditional process that I just explained is as opposed to working for three years and then release the game at the end we supposed to release the game much earlier and are involving our community

and our players we evolve the game based on their feedback that hopefully leads to a better game there's more fun for our players so far we've done over 150 releases of CO thieves there are technical alpha players and that's by far the most number of releases that any title have done on Xbox platform and a game is live this week it came out this week earlier this week and this is the amazing team at rare will work on the project now I'm going to talk about why we decided to adopt continuous delivery because we changed our business model it meant that we wanted to sustainably deliver new features to our players of a long period of time and we know that every time we wanted to release we used to crunch so if you didn't change anything you would have had continuous crunch as opposed to continuous delivery so it was quite important that we reduced the crunch as much as possible

and therefore have happier developers we wanted to get fast feedback from our players so then we can deliver a better quality game that is more fun and on top of that we wanted to reduce the cost of having large manual testing I did not even rely on them to prove the game works the problem with relying on testing to tell us if the game works or not is what we found is the moment we don't have that manual testing our developers were not feeling confident to go and make any changes to the code base because they were not sure if they're going to introduce any new issues or not I wanted to clarify a distinction between doing continuous delivery and doing frequent releases there are probably some games out there that release regularly but that doesn't necessarily mean that they are doing continuous delivery the difference is for those companies they still use the traditional process to develop their features they've reduced the time so as

opposed to spending three years they spent let's say three months and they plan these features that overlap with each other so then they can release them what let's say one month from each other and on the other hand when we are doing continuous delivery we might still choose to release once every month but we have other candidates that if we choose to we kind of see it release when you're doing continuous delivery you can easily go from let's say releasing once a month to releasing once every two weeks because you have many candidates to choose from whereas in the traditional process you don't really have any other candidates so you can't easily go from releasing once a month to releasing once every two weeks now I'm going to talk about how you're adopting continuous delivery our developers are responsible for quality of the work and prove that a feature works we found that it's very inefficient to manually verify the game it can take days until we get feedback where the game works or not so

therefore our developers write automated tests we'd have about 40,000 automated tests and we run about 4 million tests a day 90% of our tests are unit tests written in C++ and the remaining 10% are larger tests the tests bigger systems like these are end-to-end tests some of them are performances and memory tests take longer to run and when our developers fix parts they usually try to add some tests to prove that they fix the bug you work hard to make sure our game is always shippable we prioritize fixing bugs and broken tests over working on new features if our build is broken or we have an incident which means we can't ship the game we lock our Depot we don't let other people to add changes on top of that I'm not sure if you have experienced this or not but we were seeing that a lot of times if somebody's trying to investigate when the build is

broken and somebody has add more changes on top of that makes that investigations more difficult so we wanted to avoid that so therefore we locked the depot and what we do is we try to identify the change that caused the problem and back that out again what we found is it's a lot less stressful for our developers if they come up with a fix in their own time as opposed to try to fix the build was everybody's blocked and we have build lights and TV screens across the studio to notify everybody when the build is broken so you can't really miss that and as a result of this the quality of the game looks something like this green line compared it to traditional process where the quality is low generally for most of the project and it only gets to the high level towards the end of the project when they fixing bugs and that's one of the main reason that we can't really ship the game when we are at the stage like for most of the project because the quality is very low

compared to the Green Line there we try to make sure our bills as much as possible have small number of changes in them because what we found is when something goes wrong and there's only let's say a handful of changes in there it's a lot easier to know which change broke the bill as opposed to when we have 500 changes in there or let's say even more on top of that when we have so many changes they might interact badly with each other and so there so what we found is actually the risk grows exponentially the more changes we have but on top of that our developers try to break down the work into smaller chunks so they can ideally submit once a day so now I'm going to talk about what traditionally we used to set up our branches we used to form break the game into different features and form teams

around each feature and each feature team used to go to their own branch and work on the feature and once that was completed they would manage it back into main or trunk and at some point before the release needs to create a queue a branch fix the bugs and then create a release branch and release from there so there were some problems with this approach we could get really painful merge conflicts especially if you're working let's say on a feature branch for a month or two months and now we want to measure all of that back and on top of that because we are using centralized source control systems like perforce major are not easily measure conflicts are not easy to resolve some people argue that if you use git it helps with merge conflicts but the problem is that about 95% of the content of the game is binary files and as I'm sure you're aware it's very difficult to

mention this binary file so if two people end up changing the same binary files in different branches when we try to measure them back we have to override one or the other one on top of that we might give semantic conflicts semantic conflicts means if somebody goes let's say try to refactor a function and split it into two because it was too long everything might merge fine but we're not going to know that that functionality is not complete until later on when it gets into queue a branch and again because we don't really get any feedback until changes are in queue a branch the feedback loop is extremely long we're not going to know if our feature works whether or not until maybe weeks and on top of that if we identify critical issues that requires fixes now we have to go and integrate that change into my many many branches and this is kind of the idea behind continuous integration as well well it says that if something is

let's say like mage conflicts our natural reaction is to do them as few times as possible let's say you do a marriage and it's really painful you want to you don't want to do this again for another 3 months but actually what happens is the next thing we're going to do this is even more difficult so what we should do instead is we should resist that temptation and try to do that integration as often as possible ideally daily so therefore we use the trunk base development where all the features are developed simultaneously on main and it relieves from main we don't create release branches so everything is work on main and release from there and what allows us to work this way is using feature toggles we use two types of toggles the first one is compiler on toggle which we define the toggles that we want to switch on at compile time and when the game is compared it would compile out the feature that we don't want to ship to players this allows us to have

in progress features and hide them when we release we also use dynamic taco where all the features are still compared in and we ship them to our players but then we choose to switch that feature on or off based on some criteria let's say we want to test a feature that we just developing we want to give it to small number of players to learn something quickly or prove that he works and build confidence so we use dynamic toggles for that purpose and here is an example of implementation of compile toggle we use Jason farms to define our features and our C++ code is simple if statement so he checks to see if it's enabled do this otherwise do that the dynamic toggle is slightly more complicated we still use the same JSON file to define these features and the way it works is when our server starts it talks to our services to get the list

of features that are toggled on when the client starts it talks to the same set of services to get the list as well and when clients try to join the server they send the list to the several compares the two lists if they match it allows the client to join the server otherwise you will reject the client that was quite intentional because we didn't want different clients with different set of toggles to be playing with each other we also try to continuously improve our cycle time cycle time is the time it takes from somebody deciding to make a change analyze their work implement the world get it revealed and submit it to our source control system a CI server - verifies the change and pass the manual verification and finally its release to retail or production and here are some of the benefits of having the short cycle time but having short cycle time

we can get fast feedback which we can find out if we can release let's say make a change and give it to our players we can soon find out if it works or not which hopefully leads to better quality game it allows us to work in small batches which leads to lower risk because we don't need to worry about piling a lot of change because we are going to release regularly because it's so easy to do that I also reduces handovers if we want to have fast cycle time we cannot afford to pass the work to different teams and hand it over to them which means we have to restructure our teams therefore they are cross-functional and the team is responsible for the end-to-end time and the whole process from working on the feature on to give it to players and get feedback from that and in addition if you find a problem and they say we find a critical bug we can easily respond and release a patch deployment pipeline is the

implementation of our process for working and delivering these features and for this project for Co thieves we formed a new team of many generalist engineers to implement this deployment pipeline and that's the team that I lead at rare and here are some of the examples of the things we do we've developed our test framework we are responsible for our workflow for our developers for our artists to make changes and get them into source control system we look after our infrastructure and implement our pipeline and here are some specific things we've done to improve the cycle time we've developed a system to identify flaky tests flaky tests are tests that you run them once they fail we run them again on the same set of change and this time they pass so you're not really sure they're not deterministic I talked about them in latest slides we implemented the feature toggle in our code base we regularly try

to improve our build time and our cook time and we have tried to paralyze running our tests across many machines next practice we do is tackling technical debt if you on the traditional process when we don't address technical debt the velocity looks like this red line here and velocity is the rate that we can deliver features to our players at the beginning the velocity might be high we can work on features easily but what we find is after wide you get as the code base grows it takes longer and longer to work on features and develop them on the other hand when we're dealing with technical debt the velocity looks more like this green line where maybe at the beginning the velocity is lower but over time as the code base grows and more people work on the project the velocity doesn't drop it actually might even increase and the

important thing here is that there's a point in time when these two lines crosses and the people who argue against doing a tackling technical debt they think that a project would be done before these two line crosses but what we found is unless the projects going to last maybe three to six months you're definitely gonna cross these this point so it's well worth the effort to tackle technical debt but at the same we don't have zero technical debt all the time sometimes we intentionally accumulate some technical debt because let's say we want to prioritize delivering a feature to our players quickly to learn something from that but what we do is once that's done we go back and pay that technical debt I would argue that continuous improvement is probably our most important practice because we try to build a learning organization to learn from mistakes problems failures and get better every

day our teams do regular retrospectives usually every two weeks which they sit down and talk about what went well what went wrong and what can they do to improve if something happens which means that we can't ship the game like we have an incident usually do playing let's post mortems where the purpose isn't there to blame somebody who caused the issue but to understand what we ended up in this situation and how we can avoid this so we don't not in the same situation again in future and more importantly we try to allocate time to complete the actions that we identify in these meetings because what we found is if we just sit down and talk about these problems or we don't allocate time then people stop coming to these meetings and they will lose faith in it and these sort of practices that I just explained here we didn't have these when we started the project we didn't sit down and say these are the things we want to do what we did instead was we

read what other people have done talk to them especially from other industries and apply them to our own context and kept the practices that make sense and help us solve our problems and I fully expect in probably in a year's time it would maybe adopting some new practices because we want to keep adapting to our to our situation now I'm going to talk about challenges we've had in a docking continuous delivery one of our first challenges was changing our developers mindset to adopt testing mindset we're coming from a place where some of our developers would argue that they know their area of code base so well that they can easily make changes without introducing new bugs but what we wanted to do is promote shared code ownership we wanted everybody to feel confident that they can go and make changes to different area of the codebase so therefore we had to have some tests to catch issues if they're

not familiar with that codebase so what we did was we started with a small core team who all believed in this idea and we gradually added more people to the team and everybody who joined the team went through an onboarding process including some test training we also had a separate prototype team using different codebase and even different game engine because what we found is in the past that prototype code could end up being the production code and by actually having separate codebase our developers had to go and we implement that feature using a different level of G we also do mandatory code reviews for every checking's and as part of that code review we always check to make sure we have adequate testing when people trying to make changes commit stage having a fast compensation is another challenge for us commit stage is the first stage of our

pipeline and its job is to reject changes that would invalidate our release candidate as quickly as possible and here is example of the things we do in commit stage we compile code we sync from source control system we compile code and cook assets and create our build artifacts and we run small set of tests to prove that game largely still works and then we use these artifacts are generated for later stages of our pipeline you have to bear in mind that just because coming stage past doesn't mean that we can ship the game because we have other tests like as I mentioned like performance tester we run later on that give us better indication and our developers wait for when they commit their change for the commit stage to pass before they start the new world go home go to meeting or go to lunch because we expect them to deal with it if anything goes wrong [Music] and here are some of the lessons we've

learned in trying to deal with having a fast commit stage we need to continuously improve we made a mistake in this project begin with where we spent some time try to improve the commit stage and he reached an acceptable level then we stopped we moved on to do some other work but what we found was after a couple of months we were back to where we were before so we try to continuously improve commit stage because as people add more code add more tests and more content everything takes longer sometimes some of the our optimization can lead to having more complicated system so every time we try to look at an optimization and a solution we assess it to see does it improve the complexity or reduces the complexity of our system and generally we go in favor of the solutions that reduce the complexity of our system and we want to monitor the stability and confidence in the build as we optimize it because we don't want to have fast commit stage that people don't really

trust because we stopped running tests because they were taking too long to run and we try to identify and stop doing unnecessary work as opposed to trying to do those unnecessary work quickly and here are some specific things that make commit stage long as I'm sure many of you have experienced come party large C++ code base takes very long time so we use distributed build systems where it distributes all the compiled tasks on different machines and then bring the results back and link on the on one on the machine we work really hard to make sure we can reliably use incremental build and this goes back to what I was saying earlier about stopping unnecessary work let's say in a build if you only change 5% of our code base we don't need to go and compile that 95% of the code if they haven't changed we should be able to rely on incremental build and only compile that 5% of our code has changed we have a relatively speaking huge build farm about 150

physical pcs the help with this and they have two CPUs with 64 to 128 gigabytes of memory with envy and as much as we like we cannot afford to run all of those 40,000 tests that we have so we need to make sure we identify and prioritize the tests that are most valuable for us and to begin with what we did was we prioritize running tests they were failing more often in our coming stage and stopped running tests that were not failing and that was a good start well then there were some problems with this let's say an area of code place has been stapled and nobody's been changing it so we stopped running those tests but as soon as somebody went and make some changes to that area now they didn't have any coverage in our commit stage and they would find a later on when we run that there's our later stages so the feedback loop was too long so instead what we did is we develop another system that runs each test and

creates a map to see which code farms it covers and then when we trying to run the commit stage on a build it looks at the changes in that build and see which files are there look at this map and find the test that covers those code files and only runs those tests we found that that gives us much better confidence in the test every run game packages are quite big these days between 100 250 gigabyte and we may not need the whole game package to run some of our unit tests but still they are tests that we run in our pipeline that requires the whole game package so having just fast network connection to transfer this helps our build agents have local cache to store this build artifacts let's say it built an agent DiNozzo build artifact to run some tests at one stage of our pipeline later on he tries to run another set of tests on the same build as opposed to have to go and download the build again

they can use their local cache and our build agents can serve artifacts between each other which helps from all the agents going to the server to download the build flaky test has been another big problem for us especially when we started you are not sure how to write to test so a lot of tests that we wrote at the beginning and are being flaky and here is the problem with having flaky test because when we have flaky test they infect our automated test and when they fail you're not sure if it's because of a change we break you introduce or because of flakiness and what's even worse you can also fail other people's build which means that then we lose people developers lose confidence in our system and therefore they are not feeling comfortable go on making changes which means then eventually they are slower to develop new features and here

are some of the lessons we've learned in dealing with flaky tests actually having fear deterministic test is much better than having lots of tests that are flaky and we try to work hard to make sure these flaky tests don't infect our pipeline and we try to quarantine them and we either try to fix them and move them out of quarantine or we just go and delete them if they're not valuable and didn't provide coverage for us here are some of the common causes of having a flaky test for us using random weights let's say sleep for five seconds as an example let's say the service starts as a new test is waiting for a client to launch so it waits for five seconds everything might be fine at the beginning but as we add more content the client takes longer to boot which means that when it gets close to that five seconds sometimes you might fail or pass because sometimes you might just go over

five seconds and sometimes it's not and tested are not isolated they can fail depending on what tests ran before them let's say if a test relies on some state to be set but doesn't set them depending on what tests ran before those other tests might be overriding those states so these tests might randomly fail or pax depending on what I'll test ran before them and for some of our tests that we rely on external dependencies like some remote services they can just fail if the remote service goes down without actually us introducing an issue so another challenge here is how much time we should allocate for improvement we have three types of work the first category is planned work or the work that we do that we plan to do like let's say develop features and that's the kind

of work that let's say business and production wants us to spend most of our time but the reality is that we also have to do some unplanned work let's say fix bugs does a game crashes or serious problems that we have to go and deal with that prevents us from actually developing feature the third category is the improvement work and these are the work that would address the root cause of those bugs or the incidents that we've had and there is no rule about how much time you should spend for each category but there is a very good feedback mechanism there if we notice that we spend too much time doing unplanned work it means that we are probably not spending enough time doing the improvement work which indicates that we need to spend more time doing improvement work and here are some of the challenges we've had in releasing weekly with confidence as I mentioned

game packages are quite big so we want to make sure every time you release our patch sizes are as small as possible to make our life simpler we only allow one version of the game that everybody use at any given time we don't allow a different version of the game to be live which means that every time somebody wants to play and we have released an update they have to go and update and if that update is let's say 20 gigabyte people would soon stop playing our game the other challenge is delivering new features regularly because traditionally like let's say our designers they want to only release the feature to players when the whole experience is complete but that might take let's say three to six months unless for a big feature so it requires a change in mindset to say how can I break this experience that might take three months into smaller chunks that we can still release ideally weekly to our players but still is the full experience when it's completed as much as we like to we

cannot prevent issues going out to our players because games are complex systems therefore we try to minimize the impact of something when something goes wrong as an example let's say one of our services is down which means players cannot buy or sell items in the game we don't only go into crashes every time somebody goes into a shop to let say buy or sell items it's much better if game detects at and let's say put the sign on the shop to say shopkeepers away come back later and then removes that sign when the service is back online and the other thing is we try to identify these issues as quickly as possible so then we can thanks for having fast cycle time we can respond quickly to them and finally the certification process platform holders like Microsoft they have a certification process and certification team traditionally for us that process could take one being sometimes up to two

weeks so our release frequency is bound by that so for this project because we wanted to release regularly we started a lot of conversation with the certification team and we did a great collaboration with them where they evolved the process and as opposed to again trying to prevent these issues from going out they trusted us that we can do it ourselves and instead they agreed to monitor some key metrics for us in retail and as long as those key metrics let's say crash rate as an example was below a certain level they they trusted us to keep releasing and the agreement was if we find a problem suddenly let's say crash rate increases significantly then they can jump in and have a conversation to understand what has gone wrong and we got to a point right now where from this point that we have a build on available for a player to play it's about six hours that's much improvement from what we had before that was a week just under two weeks and here are some of the takeaways

from my talk we try to work in small batches to reduce the risk of reducing regularly because we want to make sure we can release regularly in safety so we can learn fast and therefore have better quality game that is more fun for our players as much as we'd like to we cannot prevent issues so we are prepared for something to go wrong but instead we work really hard to make sure we can respond really quickly to these incidents and we try to continuously improve so every day we are better than the daily fare day before and we are not done with continuous delivery by no means we've been on this journey for four or five years and there are still room for much improvements so you're gonna carry any proving before I go I wanted to talk about two books that have been very influential on me and my

friends at rare continuous delivery book by Jace humble and Dave Farley it's a great book that goes through much more detail about the principles that I talked about here and the goal by Conrad's which talks about theory of constraint it's an amazing book if you want if you're working on improving your cycle time and optimizing your pipeline by reducing bottlenecks the authors are continuously literally book sorry have a test which they say if you can ship your product to your audience by push of a button when you are on the beach having a cocktail it's a good indication that you're doing in a good job but we released the game this week I'm not at the beach having a cocktail but I'm here at the conference giving you a talk so we could I don't think we could have done this if you're not practicing these continuous delivery practices so thank you for your time and listening to me [Applause]

does anybody have any questions all right you mention about bringing your small core team on board and starting bringing more people how long did that process take in terms of timeline and how was what was the size of the team you started with and growing I think it was about four or five developers when we started and then we gradually added more I cannot remember exactly how long it took but thank you maybe after the first month then we brought new developers but he was gradual he wasn't southern like southern hi I've got a question about the automated testing that you're doing were you doing test-driven development where you're writing the tests first or were you retrospectively adding the tests to two new features and a subsequent question for legacy systems how did you implement tests for them did you ensure coverage for them or did you just leave them as is so I think very

few people did test-driven development not everybody did that but the important thing was when they were trying to check in their changes we made sure there was some test the important thing for us was that when somebody added some change there was some accompanying tests for that there are some other benefits with doing test-driven development like which leads to more modular code smaller functions are but testable so it has other benefits there but we didn't mandate that we left them up to developers and I think very few did that and your second question could you repeat that again and about legacy systems were you testing legacy systems in the same way as new features and no so for the area especially that we were not touching we didn't bother adding any tests because we couldn't and we were using unreal so which is a huge legacy code so we didn't try to go on add a lot of tests for unreal code okay thank you hey-ya fabulous to look thank you I

could ask a hundred questions but I'm specifically you mentioned you've done a lot of work to make sure that you could trust incremental builds that's right if you could talk a little bit about what work did you do because one of the problems we had because we were had about hundred 50 agents and the state of the bill wasn't the same because if an agent wasn't used for a while then the cash could have been quite out of date or because our developers new personal bills which they upload their changes by our CI system to be tested before they commit so some of those are trials might be newer and doesn't match the code that exists so we did a lot of work to make sure we did not in this situation where like somebody can't do a person or bit effects uh much worse and we end up using that object for that does a match the source code in our game that was the main thing did a lot of cleanup work to make sure they work reliably - thank you okay so then you cheap version I suppose

any time that happens you still have features some other features still in progress that's you which you turn off video toggles do you do anything to prevent shipping content belonging to those soft on features so for content if they are not referenced by the game because like if it's at a very early stage that is just being prototype they're not reference pile in the game so therefore they not being shipped but if they start to be referenced in the game we would ship them so that's one of the things that we can improve because essentially yes we are shipping some content that are not being referenced by the game yet or used by the game so do you have some mechanism if you turn a feature off then it's no longer reference do you detect no no we don't have that yet though right but that's something that we can definitely do thank you thank you Hey how do you deal with version and content

mismatches between client and server when you have like a bunch of content creators and a bunch of client developers a bunch of server developers all trying to work on their own boxes with their own features how do you like have a shared environment that everyone is hitting or is everyone setting up the services on their own boxes locally so everybody work like we don't keep multiple versions we because everybody work on the trunk or main we just have the same version so if we let's say change an asset format everybody who update would get that version answer your question no like Stan I'm making that change right sure and I want to like manually test it just like quick smoke test it sure without having to like figure out exactly how I'm going to like automated test it is there a way I can set up the game and

services and and so I can do this yes so we have we develop some tools that allow to improve the local workflow that like you can just press a button there like compares this client the services of the services and cook the assets so then yes you can then I didn't even deploy to your xbox and tested on your xbox or your PC yes we develop a tool to allow us to do that because manually doing it was quite difficult one quick question how big how many people are currently working in this one branch Depot I think probably 150 people well how do you deal with the problem of the builds been broken and now it's unlocked and everybody just like dumps all their stuff in so what we found is that longer is locked there more this chance of people dumping a lot of things so that's why we try to back out the change because if somebody trying to come up with the fix it would take a lot longer so we try to usually deal with the build break within 10 minutes or so so we

don't keep the default lock for a long period of time great thanks thank you hey I wonder when do make code review before integrating community to Toronto after so yes it is before before we do that so we do the code review if it passes then you're allowed to check in your changes to your track thank you how do you deal with the multiplicative like complexity of different feature toggles so say there's five features that could be on and off are you testing all the different configurations of those or do you just you don't know the test pass for it until a month later when you do finally turn it on that's a good question so what we tend to do we we first of all we try to keep that set as small as possible so if a feature is done we try to remove the toggle so it's always on but also we treat the whole set as as one set because it makes the testing a lot easier so if let's say we

make a bill and there's a problem in one of the features that we toggled on you just read off all of the features because otherwise testing them would be very difficult the combination could be I mean the test matrix would be very complicated very soon so yes we just take and to make our lives easier we just treat them as a set then we switch all of them off I see cool thank you I thank you for the informative talk but I had a question on the unit testing can you elaborate further as far as are your developers bringing to the table initially with the start of the project their own set of unit tests or are they sharing or propagating unit tests across the team especially if there's one where it's found to be a key metric or a key set of tasks that they're evaluating that could be shared I guess I'm looking for more information as far as the continuous development improvements and process improvements if that didn't make

sense I I'll try to rephrase myself I apologize okay basically you're developers especially if they're newer ones on the team they're focusing on unit testing are they bringing to the table their own set of units can those be shared or are they developing them on the fly yes they develop them as they work on yes okay but as I said like we had this problem that like with flaky tests and that was the main reason was because when we started we didn't know how to write good tests so a lot of these unit tests severe at were flaky okay but people learn from that and improve therefore that the tests that they added later on were much more deterministic okay thank you you're welcome hi a good talk because you have 150 developers on head when you're refactoring for technical debt ubiquitous feature that's ubiquitous across the code base touching lots of files integrate interacting with lots of

other systems did you run into any issues I have any tips on dealing with changes that need to be made to reduce technical debt that actually affect a lot of developers because it affects a lot of systems yet so that would that was one of the problems we had as well fortune not many people had to do that so the number of people had to deal with this was small and the the trick was then again using feature toggles how can they as opposed to let's say editing tens or hundreds of far anything submitted together how they can ideally try to use feature toggles to hide some office know so they don't suddenly have to commit all of them together it goes back to what I was saying about breaking down the work into smaller increments so they can like make a few changes submit that then make another change sub me that and touch a smaller number of forces the post is going quite a bit but sometimes you can't avoid that sometimes you have to do that and it's very painful what we try to minimize that as

much as possible while the sorry I don't have any specific I don't think other than try to minimize that as much as you can hey where does that a worker fit into all that sorry like how do you manage your data because this is very very oriented yes sorry that's in the same sort trunk in main so all our data everything is there as well but your for example do Europe I don't know your audio designer they can't really code themselves automated testing so who does it for them oh they write some tests really yes our sons and guys they write some tests as well but there's a difference because you have the source sound and then this could being converted that would use in the game they don't need to write test for the source sounds it's only for the when you they can easily work on there's the source files as much as they like without having to add any tests but when they try to hook it into the game that's when they need to put some tests okay so

for example for your level design and your I don't know VFX your customization so they all write their own tests to make sure that nothing's broken yeah well obviously the test that they write is at a different level to what program is right now sometimes the programmers have to go and help these guys to write tests or ask programmers to write the tests for them but we encourage our like artists to write some tests if they can and also they try to do code reviews as well fun for artists but yeah thank you hi I'm when you said that you were toggling these switches on and off to enable code did you mention if that is its own like you have a pull request and a code review on that change or is there like an out-of-band way that you update those switches no it's part of the same change let's say you want to toggle the feature on but it needs some like a small pieces left you complete it and then it allows you to toggle the feature on so you include that change along like

the toggle along with your change your last change and you submit both of them together and gets revealed because sometimes these tacos when you want to turn them on you should have to check with like other people from other disciplines as well and make sure they kind of sign off because like potentially you releasing this feature now to your players so you want to make sure that place is production signed off on that as well but usually we encourage it to like the last change that enables us to now release this toggle feature to our players to turn the toggle on along with that change together to make that change atomic awesome because if anything goes wrong then we can back out and the toggle is offers oh yeah great thanks you're welcome hey for your targeted testing so you said you had 40,000 tests you don't run the entire test suite when you're doing your targeted testing was that based on code coverage and was that a hand curated map or is that you used a code coverage tool to then map that in so we

use we used to use a code coverage tool and when we stopped using that so we came up we wrote our own tool which said it runs each test every basically a once a day he runs all the tests each test and then tracks which code fancy touches they create so it's all automated we don't do that manually because it's gonna take a long time great time map I maintain that and that was all internally developed but you guys are way too bad thank you so what if you wanted to upgrade the Unreal Engine could you do that or did you already do that yes we did a couple of times but then we at some point we stopped because he was taking far too long to integrate a new version of engine but because we had these tests again they were helping us but otherwise he would have been very difficult to integrate the new version and try to release we got to a point when in the same week that we integrated a new version of engine we also released the game couple of days later to our internal audience but it was thanks to these tests otherwise it would have been

very difficult to test them right just kind of covered most of that or it it covered the area that we we were modifying and we were changing what I mean it was quite common that like we take a new version and then something will be broken but again we were trying to detect them as early as possible and try to respond to those issues or as you fail it's possible to do that that's doable yes okay cool hey great talk two questions firstly does this involve both the clients and the whole back-end part of the of the game yes okay cool and secondly do the same teams is it one team that implements them both the clients and the backend yes that's right go so they write tests for everything and integrates all right when a test gets broken who fixes it the tests are broken the person who caused the break okay if they said because like we try to keep the number

of changes in a built as small as possible so it should be really easy to identify if you look at a couple of changes in there you should be able to easily identify who broke it and then you ask the person okay and do you have any dedicated people to basically take care of the automated tests the framework is there a dedicated team for that or is it line work well the framework and the tests themselves we have been tested to test the test framework tools and everything they are in the same Depot in the same main branch and the deployment pipeline team are responsible for writing the test framework but at the same time we don't write the test our developers write the test because they want to prove that their feature works otherwise I have to do it manually and they don't want to do that manually okay so there is no one that's actually dedicated to working exclusively only on writing tests on fixing tests no okay thanks hey at one point you had said that prototyping is critical to effective

testing could you elaborate a little bit more on what you mean by prototyping do you recall what I mean sorry I don't recall the specific instance but you there was something to come that came up where you said there was prototyping was important might have been when you were bringing team members on let me see is it you mean no the third point there which having a separate prototype yeah yeah right there so so what as I mentioned so the what we found in the past was when we are working on prototype the quality level is quite different for production code like you don't have to write tests if you prefer hard things because you want to quickly test stuff and I see if that feels right and that kind of stuff so we didn't want them to kind of put that level of quality for prototype so having separate

code base and using even different game engine it allowed them to work on that it can be isolated okay well the same thing what we found in the past was that could easily end up being the variation we used in production because they might be difficult to justify because there might somebody might say ah but this works kind of works and why don't we use it so having this separation they had no choice they have to go and re implement that feature that they just prototype in the production code with tests on everything great that make sense thanks this is ask you a question yeah cool like great tool thank you I was thinking in terms of the commits themselves presumably each commit is from a single developer that's right so how how much come how weighty is each commit because they frequently use feature branches for working on something so it kind of you know maybe a couple of days do you see individual commits having that much work in or they smaller no there are quite small because we encourage our developers to do check-ins every day not everybody does that some

people do but some people don't on average I think we do about three to four checking's each like our developers do three to four check-ins a week which means if they're not doing it every day they don't every other day and therefore then the number of changes like far as they have is relatively small so and they don't use another flat feature branch or anything everybody work on trunk right and when your code review in one of those commits presuming that source in line with development so I would do a commit someone is there ready to review pretty much straightaway so they do the review before they commit I can again one thing we found is the smaller the change easier for people to review because if you try to review an is small it's a lot easier for people to reason about a revisit because we had often like sometimes we had this problem where the if the chain was too big the person reviewing it will send it back and ask the person to break it into smaller chunks so their review different pieces supposedly I mean this one big change that they want to review right so does the review process have much latency to to that connec process yeah

right so that's one problem we had and but one way to tackle that was to encourage developers to have smaller changes because more people are willing to review smaller changes than big right so he helped definitely oh great thank you you're welcome okay first thanks for for the talk great talk and excellent discussion from the back beer I'm gonna be selfish I have two questions sure we have for five minutes okay first for the delivery pipeline team is it a single team supporting all your programs and your entire studio yes that's right okay excellent and then second question when you and the team at rare first decided to stand up this team and kind of restructure your approach to development how did you the engineering team and leadership kind of build a business case to get buy-in from the entire studio so our software

director also was quite supportive in this and the leadership team but at the same time because we were changing the business model and we knew that we want we have to sustainably deliver new features for a long period of time we couldn't really rely on manual Testament like the type of game that we have we can't really manually test everything so these were the main reasons though we had for for adopting testing specifically because we couldn't really do it otherwise we had to I don't know how how many like testers four for four years originally what we didn't want to do okay excellent thank you hi great talk oh I have a quick question that sort of tangential to the talk here you mentioned briefly that you would release toggles to the world with a small set of players having them turned on can you discuss a little bit of detail about how friends playing together deal with these different toggles so if so people with different

toggles we don't let them to play together because the server will reject the clients that have different versions and the way we implemented it is we actually use there's this concept of title IDs for Xbox games and actually we had to title like these one title ID for these features that we want to give to small number of players and then use the other title ID that goes to wider audience which meant that actually they couldn't play with each other so in that case end-users know already that they're playing two different versions of the game it's not actually like an a/b test to evaluate necess over so it's not like kind of transparent or anything it is actually you need to know that you are playing this kind of almost bleeding aged version or this kind of more stable version but again I mentioned that there are room to improvement that's one of the things we try to do that how we can avoid that so we don't need to keep to title like this how we can do the same thing we just using one title I think so these are the things that we aren't going to work on so in that sense it's a relatively limited test for home in that sort of situation ok cool thank you you're

welcome ok thank you for your time [Applause]