A QA Perspective on Live Game Production

[Music] hello and welcome to my presentation a qa perspective on live game production my name is michael volklarsen and i am the lead tester at ritumoto in copenhagen and we are making and maintaining the game heroes and generals and before we begin i've been asked to just remind you to turn off your any silence or silence your noise making devices and also i would appreciate it if you rated this talk in the survey that you will get after this i will try and leave some time in the end for questions so and if we run out of time i've been told that there's an area we can move to and i will happy to take questions so let's begin first my background

[Music] my background is i've had in seven years i've been working in development so i've developed systems myself content management systems and and i've also created some some logistics handling systems so i've been through the whole loop of from concept to design and development and on to test and maintaining it and that has come in handy now as in my function as tester and as lead tester and before i started at razamoto i was the editor of tom's hardware in denmark where we started up so i was helping with creating the brand and the format and also i had the the task of creating or writing news articles from the hardware and games perspective

and also i translated a lot of the technical articles that came from the mother side and then i've played a lot of games over the years expanding down to the classic arcade games and home computers every console has been and of course on pc so that gives a good base also to uh to have an idea as tester what to expect of bugs i work for rachel moto it's a studio in copenhagen it was created by the initial founders of io interactive and the founders are creators of the hitman series so when they sold their company to idus they moved on and created rachel moto and they created the game heroes and generals which we operate to this day and to give you an idea of the scope of our production i can tell you a little bit about the game

heroes and generals world war ii is a first person shooter and they're free to play and as the name suggests it's said in world war ii belts all over europe are going on between the factions americans and germans and soviets and they all fight for dominance no predetermined alliances which one might expect but it's open to the players and the clients to make alliances in our game so we have a combined arms combat where players will be able to play as infantry or tankers or pilots recons or paratroopers and all this going around in the same battle and yes all three factions can't be in the same battle and and the action client as you see it's uh it's not based on a licensed engine so

it's a proprietary engine that we created ourselves which makes it uh we have a lot of different tasks that we otherwise might not have when we change anything in the render engine or if we change anything to the network code we have to test that from scratch and on top of that we have a strategic layer and as you see here this is the uh this is just a replay of one of the wars we've had it went on for over a month and each blip you see on the map is one battle that starts up so we have our generals move their forces around on the map to create these battles and that will fire up a battle for up to 18 or up to 36 players 18 on each faction or 12 on each faction for a three faction battle and if not enough players are online then it will simulate the results from the resources available

and it came out of closed beta in 2013 so we went into open beta at that stage and in 2016 we we released it fully and since then we've had regular updates and that has added new weapons new vehicles new maps and new exciting features and it's available for free on pc at heroes and journals.com or on partner sites like steam and now on for the interesting part so we've gone through different stages and in the six years i've been there it started out in closed beta and that was a comfortable stage to be in because our players kind of expected there to be some bugs and we had some leeway

and it was still not cool to let a bug slip out so we still had to be good and we still had to to use our resources efficiently and we figured out that we could we had some dedicated players where we took the most mature of those and invited them into a closed beta group and we used them for feedback and gave them the opportunity to see upcoming features and we found that as a very valuable asset at that time of course you have to have those dedicated players and if they're not mature then you might have to have some battles with them but we've been lucky in in having some good players to assist us then open beta we kind of removed or we didn't uh we no longer had that control over how many players would come into our game and put load on our

servers so we had to adapt and find out how to best test for load issues and for scaling and for steam early access we were one of the first games that were greenlit for steam early access which was a program that valve made to give the players or the users of steam the opportunity to decide themselves what to which games they would see on the platform and that put pressure on us in the sense that we could see that there was a lot of interest and we had to of course always look at our processes to see how could we prevent the issues that could lay down the game to come out on our live stream but until release we had a three year period until there or from we went from open beta to release and that time we we spent to

hone in on the best practices and to make a really tight process and make sure that we we caught the most of the the broken box that would otherwise go out and beyond that since 2016 we've had a lot of updates where of course we have frequent updates that has to be uh tested overlappingly so we have always new upcoming stuff that has to be tested in a way where it doesn't have to break or break the or prevent the development process and that's something i will go into more detail with and of course the main goals might be obvious but it's good to remind ourselves we are in qa and quality and stability is what it's all about and with a live game that is not always

as easy to uh to maintain so over the years we've added many weapons and many vehicles and skin variations new features new maps and with each new data that we add that has a lot of new failure processes and we have to scale up and see how can we best prevent all of these issues that could i mean the regression testing of it just grew and grew so we looked into how could we better perform in our qa and make it better or hone in on a better process and one really obvious answer to us was to be more agile and of course there's a lot of literature on that but we tried some ways to implement from what we read and what we heard from others to implement

but we also tried out what we thought would fit best into our way of working and a good way to to illustrate that is by this comic this is one type one way we try to implement it and one that didn't work very well and for example in this case where we have a new house master in the game we would know that there were some issues with it because it was not it was not finished it was testing it while he was making it our graphics artist so we tried to figure out what could we test what could we benefit from testing early on it and we would look at it we would know exactly what this should do the food should be placed on a map somewhere and it should just be working as somewhere on the map that that added to the level so we could see from the get-go that there was a hole in the roof and we

would expect there to be a bug or two somewhere and what would happen if when we approached it we turned the knob and everything just broke down and went to flames and we had a developer that was really not happy about it we had a tester that was really not happy about that he spent some time to to to test this and it wasn't ready so we looked at that and figured out how can we better implement that process and communication is a good thing communication is something you can always improve on and in this case we should never have turned the knob because the knob was a part that wasn't done yet so we should have done it more synthetic synthetically so we should have tested the parts of it that was actually done like the coley like the stitching of the textures and of course we should have made a more specific test case from it and

that's what we learned from it and now on to how we work with streams if you're familiar with performs perforce or some something similar you're probably uh you probably know the term stream or branch and the way we use it is that we whenever we start working on a new feature we take the code base that closely resembles live copy it down to that stream and make sure that is as up-to-date as possible before we start to work on that feature and that means that if we start working on a new game mode we can make sure that no other feature in progress will be have any effect on whether it works or not so if we find a bug on that stream we would know that the one responsible is probably one guy or those guys that are working on that so the blame game is kind of eliminated

from whenever we find a bug who's actually responsible in fixing it this also means that we can have parallel tracks of testing where we can prioritize into what we want into next update and focus on what are the areas that we want to make sure are working in the process where we are in the early process in the sense where some of the features might be very prone to errors and we would know that we would have to have focus on those in early on so we could check all the test breaking stuff and make sure that those get fixed before we move on so in this case we have six different features that are ready for tests we might only have time for three or four to start with and by doing that we will have one tester be assigned to one of these features and apply a first qa pass and the first qa

pass for us is that he takes an approach that fits best and uh and works just by experiencing it either from the black box white box or grey box approach and from there you will go through finding all the bugs bug fixes come back he checks them if they're all fixed he greenlits the stream and when it's screen lit on the stream it's ready to be merged up to the main development screen and from there if it's not ready uh we will of course not merge it up and we will still have features that are quite not ready and some that we have not have time to test fully and that will be a risk management process where we would see do we really need this in the next update do we have to move our test focus on that before we move up and this is also a process where we would use some test automation to make sure

that we don't spend too much time on the stuff that are very time consuming and it will be able to assist us in finding the bugs that that can occur so once we're ready to move it up we have all the green lit features from our first pass and these don't have to be perfect these don't have to be bug-free these just have to be non-test breaking in a sense where we know what to predict from it so in the feature phase where we are now that would be two to three weeks before release we would know that we're pretty sure that we can make this release so predictability is what we are gaining with with this process but of course it's not always that easy it's not always in the perfect situation sometimes we

have a main content driver that we need to get out like a new map for example in this case where based on player feedback from play tests we can see that there are some areas that still need work and in that case our level designer will still be working on it when we're going to the phase where we need to move it up to our test history so that just means that we would have to focus on other areas get our test resources focused on the stuff that is ready and assist the level designer if we can in the process and and move in to test the stuff that he does get ready early and of course something like the sound update that might be something that is not a critic critical asset or critical addition so that means that there's low risk in merging that up because we would be able to test this fully before before release

but then also some other things can come in the side and this can be based from metrics and based on decisions where we want to move a little faster to see the results of it could be an iterative process where we want to iterate on on a process where we just need to find the the issues with it the bugs with it get it out there and then measure the metrics from it on the live environment so it's often not optimal it's often not perfect when working on a live game but that's where we have to be good at scaling it and scoping what we test and and say i mean report to our project manager what the risks are with if we move on with this and then optimally we have all the features greenlit or we will get to that stage before release and

in that case we have the screen green lit and we will move it up to a staging stream which will be a stream we reserve for the critical fixes the hot fixes that where we need to have a clean stream that is closely resembling live and to make sure that nothing else impacts anything if we are fixing something isolated right before release we move it up there and we do a short or longer smoke test depending on where what the risks are with the build just getting around all the corners and once that's greenlit we release it to live and in this process i've seen during this gdc there's been a lot of focus on automated testing and that's also something that we have had a focus

on that we could see we could benefit from i think everyone should in qa because it's something that both can spare you a lot of time and they can actually add new cases that you will never be able to test so we have two different tools as we've explored different ways that we could implement this the first thing being a ward data visualizer because we have a game with a lot of items and a lot of attributes it can be daunting to look into all of these data and figure out if one value somewhere is way off so what we have we have a tool that across streams can compare all the different data all the changes and then we have a manual process where we have to look at these results and see the red flags where the problematic issues could arise and this means that we can focus our testing directly

towards the areas that actually have reason to be tested where it's not just we think this might be affected but this is actually affected might be a bug it might not be but but it helps and the other track we have is an automated compliance and performance tester which runs a nightly build every night and it's tests for it's scalable so we have two worker pcs or more that we can set up with different hardware to make sure that it can actually run the nightly build we can set it to a higher degree so we can set it to each commit we have but but it takes a while for it to run through all our maps from a different set of camera files so so we usually have it set to a nightly build and that will catch so we when we meet in the next day we will catch if a build has failed it will catch if something has been committed that is

very taxing on performance and we will be able to see where is the performance historically moving on our streams that's also something that is very time consuming if we had to do this manually and looking at the methods of course there are a lot of different methods and one of the points i want to make today is that what we found to be very important is diversifying with the resources we have so in the perfect environment we would have all the resources in the world we will have all the testers to make all the different test cases that we want to complete but of course that's not rarely the case so we have to diversify and look into what methods are good to use sometimes it's obvious depending on the test case but other times there might be some that are outliers in what will we benefit from using this

method and for the first test pass it might make sense to do the obvious and for the second test path it might be it might be obvious to try and diversify into another method and another approach in that case we would get around other corners and get around to other areas of whatever we are testing and approaches are also something that we are diversifying with so with box testing i look at black box testing as a luxury something where we can if we have the time i would love to have a first pass just being a black box where we would have a different outcome of it instead of having the focus of having all the details from the get-go so our tester if he knows nothing of a new vehicle we've made he would experience it much more like the player would he would look more much more into how does this fit into what else is in the

game and in the other end if we don't have that much time white box testing is the way to go whitebox will be able we'll be able to dig deep into the data dig into the code and see what has actually changed and from that make a decision on what should we test from this but oftentimes it's a gray box situation where we are looking into the data behind or the code behind and then we are doing a task where we give that information or some of that information to our tester to test from but it's much dependent on what time you actually have for the task and another approach to other approaches are what i call authentic and synthetic the authentic being the way the player experiences something so this will be if we have that new armored vehicle that would be log into the game find it in the store find out how you

unlock it then unlock it purchase it and equip it to your character and from that flow go around all the corners figuring out what can i do with this new item and finally matchmaking into a battle driving around in it and experience it like the player would this thin synthetic way of testing that that would be just firing up a local build firing off a command just to spawn that vehicle fly around fly through it add a lot of debug layers and outputs and testing it that way both are important and necessary but it's something where if we diversify by focusing the first pass on one or the other we can we see that we can actually benefit from that then we have the methodic all the chaotic way as i call it the methodic being might be more of a temper-based approach where when we are looking at hiring people we are trying to fill out different

profiles so a profile could be a very analytical person very very what's the word someone who takes it slow and quiet and kind of looks at it steps back when he occur experiences something and then test it from that approach another profile we have is the chaotic guy the guy that just clicks on everything and if you just respond within a nanosecond he will click it 10 more times and in that case he will probably find a lot of crashes that way but that's also something where we can benefit from having one type of tester testing something in the first pass and have another tester with another approach in the second pass and then the last one is objective versus subjective this is more dependent on i guess this also carries over to the black box testing where if we take objective testing well that is the

definite box that is if you are testing a new vehicle and the collision is all wrong on it you can walk right through it that's just a definite bug that's a defect that goes into the bug reporting system and that's getting fixed the subjective part that's more of does this look right so this vehicle when testing this from a black box perspective for example does this fit in to that category of vehicles so if it's an armored vehicle like a medium armor tank and we look at this and see well it's it has an incredible front armor and it's pretty much impossible to penetrate you cannot deal damage to this tank maybe that's based on historical data maybe that's intended by the developer but maybe that's not intended and that's where the subjective part comes in from the tesla's perspective he will look at where it fits in see well this doesn't seem right and make a report from that

and sometimes not just situations that you can test sometimes that well you can test it but it will not be in an authentic manner as our players would experience it and uh that's a good way for us or we opened our eyes to this when we had more and more players join our game where we started to have much more open tests where we are inviting them in on our prototype server testing a new feature and then seeing what actually comes from that getting the feedback from them and opening it up for anyone who had the time or the interest and this is from a player on live that experience something that we if we put this out we would never in a million years be able to reproduce this i think and that's something where we we can see that the physics system and the network code that are actually working

properly and uh and we could reaffirm that that that it's working like we would like and another example here from a very good pilot what he sees is uh well he doesn't see his friend but his friend is dropping a bump and we can see the hitbox actually and the netcode being up to par so the bomb passes right through the the model as it should and now for the conclusion i know i'm a bit quick i can see so i will expand more on these points and the first point is gets qa involved early on and that of course carries over to the agile part of it and it can be a battle so it can be a constant battle to get qa involved early on and the way we've done it is we can see we benefit from having a

tester in the design meeting itself where we can see or we can raise a flag and say well this decision is something that would result in this and it would give a problem in this area where the designer might not have thought about that and of course that's a very diplomatic process and we have to make sure that we're not killing off the creative process because then we wouldn't be invited back but we've seen results of this working where we are we have prevented some issues to arise where we haven't thought of the bigger picture and so before we started production but still before we started programming it we've actually prevented something to come to to the test phase and another point i want to make is

test features on dedicated streams and if you are able to do that it's something that we of course have benefited from in the sense where we can see that we had had issues where we tried to compile different features on the same stream and that just made a mess of things so what we did with with separating it on each stream we could see that it was not only much more simpler to test but the bugs that came from those tests were much more in a merit that they would take these as granted as okay this is my bug and it's not something that someone else has caused and it also allows us to much more diversifying the the approaches we make and it will add that extra step into before we go to release

we actually make sure that we have proper time for some of the features that might not be as important so for the lesser features that we still want out those shouldn't hold back and release because we haven't had enough time to test and of course also by automating some of our processes on the lower streams we'll be able to catch something much more easily than if everything was jumbled into the same stream so especially without data visualizer that that's possible to isolate a few different changes instead of a huge set of changes that clutters around if it's a lot of different features and of course diversifying is one of the things that we've benefited most from and that's much dependent on the resources you have and for a live game sometimes you have to move fast and by

moving fast you have to take a different approach make sure you cover all bases make sure you cover all different areas where the risk is highest and of course you can never prevent bugs from getting out especially if you have to scope it down into where you have to allow for certain processes to finish early so if you only have a day to test something that takes two or three days our approach would be to first look at it in a very deep manner just look into the data and look into the processes and and the code and then figure out what do we need to test from there and also with the hardware we have available that's also something that we diversify with that's something where we try to with the hardware we're using on a daily basis we use some different graphics cards and cpu setups also operating systems

and for our automated performance and compliance testing that's something where we have different setups of course that will test it on a daily basis and then involve your dedicated and mature players this is also something that might take some effort from our own part something that sometimes takes some maintenance but it's something that we benefit a lot from and something that we can see the value of so that is also something that i would want to point out as if you have that opportunity to involve them being on slack or any other sort of of communication discord or just from a forum then that's something where you can benefit a lot and automated testing that's been a big big word on this gdc and that's something i would also say that maybe you can maybe you don't know exactly how to approach it if you don't have anything

yet but it's something where i think every everyone should explore what is possible and what would make sense and of course there are different approaches that will never make sense we've made the decision that we would never automate our ui testing because that's just something that changes a lot in our game so we would have to use a lot of time just to maintain that test and the benefit from that would be minimal so we are identifying the areas that actually make sense to test and then focus on maintaining that because there will be some maintaining but it's something where if you don't have any automated testing you probably have that the task for your testers where you see that this is just a tedious task you don't see you see them being maybe annoyed or it's tiresome for them but if you can also make that process

that's something that everyone can benefit from and that leads us to questions if anyone has any questions then please please all right oh thank you uh i had a question uh very interesting concept with the separate streams um where do you do the integration testing like you had an example with a new map and a new vehicle so when those two come together where do you do the integration testing for that we do that when we move it to our main testing stream so that means that when we tested it isolated we know that that part works so when we move it up to a higher stream

and they are collected there we will do the integrated test so usually if it's something that is if it's their related we will probably develop it on the same stream and test it on the same stream before we move it up but we've seen larger updates larger features where we have collected too much on the same stream and that has just ended up in a mess where you can't really we couldn't find hit or tails in it and it took us a long time before we actually got it to a state where it's usable and it's ready for release thank you okay thank you [Applause]