How we do Quality Assurance at King and How Agile Jenga Helps us With our Processes

[Music] hi everyone I'm here to talk to you about how do we do a quality assurance at King and how agile Jenga helped us with our processes but actually we don't do Quality Assurance we do quality assistance and but who am I to talk to you about this my name is Anna Karolina I'm from Brazil but I lived in Mexico Chile Germany Sweden and now here in the UK my phone fact is that I graduated in psychology so a completely different background from this my first job in the industry was as a QA in behaviour interactive in Chile six years ago working in fallout shelters so my name is still there which is pretty cool and

my current one is esse QA in candy crush friends so quality assistance to talk to you more about this I'm actually gonna go through a drown Jenga first and then you're gonna understand why so how many of you here played Jenga ever cool nice number so as you know it's a tower game you build a tower take off the pieces and as many as you can and the person that lets the tower Falls loses right but Jenga is kind of the opposite the objectives is to actually build a tower using all of the 36 pieces that you have in the full stack the tower must be at least four Jenga blocks high with the the pieces of standing in the smaller pot part and the fastest team wins so you have a lot of teams to build a competition so it's it's more fun the game rules are all the pieces of the Jenga must be used all of them have to

be numbered all of the sides because you're gonna use those as bugs and each team has at least has to have at least three people so two builders and one person that's gonna be the designated QA and the game has four rounds each round has a specific set of instructions and the bugs that you're gonna give to the people which are four numbers that you're gonna choose between one and 36 which are the pieces of the Jenga set and the only person that can see the rules are the queueing I wish we could have time to play because it would have been you would understand this so much better but anyways round one house it goes you give the bugs and the QA sub the instructions to the QA s which are they cannot show the builders the bugs and they can only remove the bugs while they're being on the tower once the tower is done so the builders are gonna build the whole tower and once it's done the Q A's are gonna take piece by piece of the bugs of course what's gonna happen oops this right the person is gonna take

off the top the bugs it's gonna fall because basically the bugs are gonna be the likelihood of them being the the most power part of lower end of the tower is low so of course the tower is gonna fall down and this is basically a waterfall system right one person is gonna tell everybody else what to do what is wrong what is right and you build the whole tower QA tests in the end it's not the greatest model in the world of course because you're gonna find if you find the bugs only in the end hopefully this is not good you're gonna have to redo the whole thing round two you give them the bugs I just didn't put them here because it's an example and there again the Kin shouldered the builders but they're gonna test the tower after each 9 blocks so stacks of nine blocks they count nine blocks in they can take the bugs out and

then they can build a tower up to the end while you're doing the whole rounds you keep score of the time this way you can see how faster is gonna start going and what happens usually in this round is still not the best right scenario you're still checking by stage which is usually what we do in the industry right Quality Assurance you build stage by stage QA checks this stage by stage and if you work with JIRA and stories and and tasks in your game in your industry or in your companies this is gonna be basically checking after the end of each story or if they're like a feature something big not the whole gang but something big and still not the great thing to do because you know you're still gonna find a lot of big mistakes mid-game end of the game and not in the beginning or meeting the issues round three is a bit better

they still don't show the bugs to the builders and after each block is placed they can test it so the block is placed they can check if it's a bug or not they can take it out and so it goes until the end what happens is the tower is a bit more secure so you can poke a few holes and I'll have the whole tower fall down but still a bit iffy right but it's a better scenario than the two and before that I showed where the testing is a constant and not just a face so you have development test development test and this would be in the JIRA scenario that I said it would be at the end of each task so if you have something like playable you would go and test if it's just a smallest thing you have to make sure that this is compatible with the rest of your game instead of just merging directly into master and messing

it all up and this is what we call quality assistance so for didn't last round you're gonna just give them the instructions give them the bugs and now they're allowed to show the builders from start what are the books and then they can do whatever they want with this information so they can usually what happens is that the teams gather together figure out other or what the bugs are take all of the pieces before they start building the tower which is the best scenario right you're gonna have a much more stable Tower or you can pull calls you can play around and nothing is going to fall ideally right so this is what happens you investigate potential risks mitigate them and then you have less bugs and during the production which is the the place that we're moving forward at King for that we do have some other things that we do not only including QA in all of the steps of the production since the kickoff of the features in

every planning session what else retros if you have to reduce something and because you're not gonna make the deadline and the feature is too big you have to regroup and reanalyze everything QA is there to make sure that you are meeting in the risks and investigating everything that is possible to be find found you have plant we have planning sessions with those to make test cases so once we get the design of the feature we make sure that we know what it's supposed to do what it's supposed to look like investigate wall are the potential risks as they are and edge cases and everything like that sit down with the devs and make sure that they also know this we sit down go through together with like just basically the titles of the test cases to make sure that they know what we're looking for when you're testing the the features or the game in general and that we know what they're making for us to test and we also do team testings with the whole

team maybe 50 60 people at a time we sit down with them at least an hour a week and test the whole game it's not such a specific test cases based it's more of an exploratory for each feature that we're making but just to make sure everybody has the ownership of quality we think the game team so everybody plays the game everybody knows what we're talking about once we get there and have like oh look this is not working or this is not supposed to be like this everybody knows what's happening so this is what I live with you guys and it's just suggestions 1/2 QA in all of kickoff meetings maybe planning maybe for feature a whole game small things big things just have QA there just to make sure that you know what are the risks because we are in the end specialists of quality so you know we must be put to use in all of the places that we can to have planning sessions

for test cases with the devs between Q and EPS is the best scenario because then we know of being produced they know we're gonna be looking for and then we don't have to go the back and forth back and forth like Oh have you thought about when you're playing offline have you thought about when they click here instead of there and this things can be mitigated early and then also testing what the whole team is always a plus it's never a wrong thing to play the game with your whole team if you have designated QAS for each with each game more often than not what's gonna happen is that the queue weighs mostly just plays the game nobody else maybe a dev or another sometimes the producer but it's in the end is mostly the QA and if you have the whole team playing the game you're just gonna endorse that ownership of quality from the whole team everybody's gonna see what they're making everybody's gonna check it out and be like okay this is pretty cool I didn't know these other team was doing that this is nice so have claims in playing sessions with your your whole

team and that's it thank you if you have any questions yes right now we don't have a lot of optimism ultimate ultimate automation thank you but if you do food do is mainly for checks on the dev site before they they merge everything very some games we have very playing play through a level lose win very easy found stuff we're just starting with that thank you kind of more in improved methodologies like image recognition so the AI knows where

to press what to do and stuff like that yeah or we're starting with that they're doing mostly around candy but we're starting with on any any kind of like software you have in mind you may like suggest or do you prefer making it in-house I don't have one really in mind but what we doing right now is in-house so yeah thanks a lot that's a problem so if a team wanted to kind of move to this method what is like any major reorganization of that team or how do you organize the team testing like how does somebody go from what they're doing to maybe using this cool so what we do right now is that we have an each gang team has a team of Q ace

depending on the size of course and we are the ones that basically organize everything so what we do is we have Sprint's right we try to do two times each sprint to the testing and it's like one day a week we call everybody around everybody knows what time is gonna be because it's the same day like for instance I don't know Wednesday in the afternoon everybody knows that Wednesday at 2:00 we're gonna have team testing and you just have to fit in into a planning everybody has to be aware of that like all of the producers they have to be you know in everyone everything and supportive as well and then everybody just goes along we play if anyone wants to showcase something that they have done in their particular mini team they can come and say look I have this to be tested they then we sit them with them and figure out some test cases for the whole team to play so like basically a happy

path what is the future gonna look like what is like what are we looking for or the potential risks and we play together like they present the test cases and we sit down play together and yeah usually we have I don't know three or four things being tested at the same time so we have little groups and if you're done testing and we have still like I don't know twenty minutes you can play around the game make sure that you you know the music is on everything is working for like as it should but that's what we do and then by the end we just collect all of the bugs that were found if there were any and go with the teams or the people that worked in that particular feature or things that have been tested make sure that this is something that has to be fixed now or it can be gone going to the backlog and yeah organize that way thank you for the presentation I would like to know if you have specific strategy when you want to test the quality of implementation or later to the analytics to collect data okay

[Music] sometimes it's not it's obviously it's a bit it's not as easy to do that because it's too many people but yeah it depends on the data that we're trying to collect if it's a particular a be test that it's gonna you know change a lot of stuff inside of the game then yeah we make sure that we have all of the the the specs for the a/b test before that it is implemented already that we had a pass through before just just a smoke check although of the builds just to make sure it works and then we test with the team once it's all done once it's all there all the specs are are there it is difficult because you do sometimes have to be changing people from one being be tested the other and it's not the most simplest thing to do with a big team but if we think it's necessary because it's a big risk then yeah we totally do it yeah thank you thank you so much for coming

[Applause] you [Music]