Why QA is even more important for F2P games

Okay. Uh so next up uh quality assurance uh panel discussion. Uh I have here on the stage uh Dean Bent from from QA. Uh then Yukah Hilfon from from Nitro Games. And then Nikon who is actually one of my students at Guyang University of Applied Sciences. Former former student former student. Yeah. You actually graduated which I think deserves applause. Yeah. Fresh from the oven and straight into the frying pan. Yeah. Yeah. Yeah. Good for you. Um but yeah, um guys, can you uh elaborate a bit on on who you are, what you do, where you're from, what's your uh kind of relationship with the with the games industry or games? I guess I'll start. Yeah, go ahead. Hi, I'm Dean Bent. Um been working with games now since 1992. And uh I started doing QA uh around 97. And since then I've worked for various companies around the UK

um in Germany and with quite a few companies in Finland as well. So uh yeah QA uh is one of the areas that I I seem to specialize in. So user experience testing, functionality testing, anything QA related. So Okay. Right. My name is Yukon. I'm chief development officer at Nitro Games. So we are using outsourcing QA to the external parties at our company and I'm in charge of that among other things. Okay, good. Then so hi I'm Niko noan uh currently working as a lead tester for dean uh and a self-proclaimed incoming kua specialist. Uh uh I guess my history with games starts when Dean started working on them. I was like three years old then when I pic first picked an NES

controller. Uh and of course I grew up with games. I breeded games. Then I first started, of course, I always volunteered for all kinds of beta tests when I had the chance and uh during my school years at Kumalaxa University of Applied Sciences, I volunteered for local companies like for example Nitro when you had a chance for me to test. I came and tested stuff and one thing led to another and then I started working for Dean uh as a student. I got internship points at first and then he hired me and now I'm working as a lead tester. Okay. So, you you're actually getting paid now. Yeah. Yeah. Good. Lucky me. Yeah. Good. Um well, uh as we all know, mobile games and uh especially free-to-play games are the most competitive branch uh of the industry nowadays. Um which affects user

acquisition uh marketing and the products themselves of course. uh how important uh do you see QA and game testing in in this equation in this market situation? Free-to-play games, it's uh it's pres presented us with quite a challenge actually. Um the traditional way of testing the game is is typically you have a tester sitting there testing the game. And uh with the free-to-play uh mechanics in many games, it's like uh you get maybe 10 minutes of gameplay time before you run out of lives or whatever and you have to wait for it to recharge for maybe 50 minutes. And so you you know the clients aren't really getting great value for money if we're testing it that way. Um we've had to be quite creative and come up with some some new ways to test basically in order to satisfy the needs of the products. Um, one of the things we've been doing for some clients

is some field testing. So, uh, people have been playing the game at home just as they would any other game. And, uh, you know, uh, that way we don't charge the client, um, for like a lot of dead time as it were. We just agree a price and and we'll go with that. Um, okay. So, do do the, uh, testers wait for the uh, for the kind of kind of lives to regenerate or Yeah, that's it basically. I mean, obviously you can cheat your way through um, which you would do in a sort of typical uh if I was working out the developer, I would do it that way. Um but we, you know, the developer also wants to see um what people think of the game. Are they do they like it enough to wait? Do they hate it? Yeah. Yeah. And obviously it would take a lot of time to do all that waiting. That's it. So yeah. Okay. Yuka. Yeah. Uh free to play mobile games adds a quite a bit of of aspect to the game testing. At least what comes to my mind is the basic

functionality testing. So basically testing out how how the are the functionalities inside the game working as intended. Then of course are the like the basic server backend side testing. Can how easily can players for example cheat in the game? Can they do any other cheat gaming? Then of course the gameplay testing testing out the the fun factor of the game and is it is it actually driving the users down to the funnel or not then uh basically I don't know is there any any specific term for this kind type of testing but basically breaking the game like finding loopholes in the game game design and game game rules like can you can you cheat yourself on top of the top list or not using the same mechanics. Then of course the platform testing does the does the game functions as intended in in all the platforms where the they

are intended to and so basically a lot of lot of things to do in the QA side of things. Yeah. How about uni code? Can you any opinion on free testing? Yeah. Yeah. Well, let's begin. Uh in this small head of mine uh all games can be divided to three categories. Uh there are bad games. Sorry if you have made any. There are okay games and then there are great games. Now there is a ladder from bad games to okay games. And that ladder is called an idea. If you have a good idea, you can make an okay game. Okay games are the games that might make your money pack money pack. Maybe not, but can do that. Make maybe even make some profit. But great games now ladder from okay games to great games is the QA process because if you don't test all the features, test that

the color loop works and stuff like that. It it's going to be only an okay game. And as for the free-to-play question, uh for free-to-play games, GUA is even more important than for uh pay-to-play games because with pay-to-play games, uh when you buy the game, the p purchase decision is already made. So, it doesn't matter if you if the game has bucks or so to so on. Well, of course it matters with further sales, but not as much as with free-to-play games because with free-to-play game, everyone has access to the game. And he start he or she starts playing the game and he notices he or she notices a bug. They are going to think I'm not going to pay anything for this. Yeah. And they're going to stop playing at that point. Yeah. So, it's more important than with payto-play games. Yeah. Okay. Uh so Dean uh can you give us a

kind of professional opinion on on the benefits of uh you mentioned kind of very early stage testing or play testing? Uh what's important? What kind of things do you look for? Uh and what do you take into account when planning and and kind of implementing the the actual testing? All right. Okay. Um well, whenever I do um play testing, user experience testing, however you want to call it, um I think that the the most important thing is what the user actually thinks. You know, that's the information I'm trying to get back. Um, I think we're all aware that when a game is being developed, um, it's very, very, very easy to become too close to the game and to not see the problems and to just gloss over them because we're used to seeing them day in day out. And within minutes, if you do a user test, you'll, you know, all these problems become apparent. And uh in the certainly in the early stage play testing or user testing um the main things that jump out are anything to do

with the user interface. You know if it's not right it'll just be mentioned again and again and again. Um so I think early stage user testing is vital just to get rid of these um these problems which the developer may not even know are there um but the users will you know it could make or break a game. Basically the user can be totally put off if the user interface is wrong. Obviously the the other thing that comes across as well with the early stage user testing is the very early levels in the game have to be superbly polished you know so that it maintains the user's interest. Players these days have a very short attention span. I mean you could you know I download a game and if I don't like it in five minutes I delete it. You know seriously that that's that's everyone right these days. Um, so if those early levels aren't spot on, then um then it's going to cost you sales. So that that's where I think the earlier that you start the play testing

the better. So So what what kind of kind of testers do you use? What kind of people do you use? I tend to use it depends on the game obviously, but um if it's a casual game, I I I'll uh just use anybody really. Um I have a fairly large network that I can call on in Helsinki. um very international crowd of people and so it's never really a problem to um to organize a testing session. Um I mean when I say large I mean thousands of people I can call on um and uh whenever I do a play test I I don't really structure it too much in the sense that um I I moderate the sessions. I want the users to basically explore and discover for themselves. I always have a hidden agenda. Um, my clients usually have something that they want feedback on. For example, it could be the control system. I might do two versions of the game with two different control systems or there might

be an option to switch at some point. So, I'll let the user start to explore and discover for themselves without, you know, I'll help them if I see they're getting stuck or they're struggling. Uh and then um at some point during the test I'll try to um ask them about the control system if it if that was the what the client wants to know about or something. So um I what tends to happen is the client is always amazed by the amount of the the breadth of the feedback. They get way more than they expected. Yeah. And and I'm sure you get some very honest feedback. Absolutely. From testers. Very honest. Yeah. Okay. Uh so from the uh developer standpoint you um what do you see as the kind of main focus on on play testing? So what do you guys at Nitro look at when when doing this stage of testing of development? I think quite the similar things which Dean already mentioned. So basically basically what the what the players

think about the game at that stage when we do first impression. Yeah. first impressions of the game because that that's like the first things of course to test out. And then uh depending on the stage of the game also like the balance and optimization of of the of the game like how's the how's the like currency are going what's the resource allocations of of different resources how are the levels going up in difficulty or not and then then of course the UI is very important nowadays it's you have to be like really top-notch in order to have it and make break it to the top then the game. So basically these three things. Yeah. And uh how do you take these results from the testing into account in the development process? How do you integrate that kind of into into your uh uh depends for example we use a site called useresting.com for the our UI UI

uh testing and there's there's quite a lot of people there. So basically we can have as uh qualitative and quantitative feedback. So basically if some something some error or some issue happens many times for many people then of course we can take that's a more important issue than some other where there might be only one or two two issues mentioned in the same same thing. Okay. Uh so you guys obviously do a lot of co cooperation with the with the local universities. Uh can you elaborate a bit more on that? Uh what is what exactly is it that you do with the university and and uh and how starting with Dean? Oh okay. Um I work uh quite closely with Kuwam uh your university your ex university. Yeah. And um yeah the students there do a lot of um the testing that I I require the functionality testing. Um it's basically a situation where they

gain by two ways. One one they get course credits for it and two uh they're actually working with with um professional products that they're actually contributing and working in a professional way. So they're getting good experience for this. Um, and uh, it's been an interesting ride because uh, there's those guys. Um, I've had to learn how to work with students. Basically, these aren't like 100% full-time testers. Um, I've leared to tell my clients that they're not going to get much testing done on Tuesdays because they have a lot of classes on Tuesdays. The classes come first, so the testing has to be fitted in around it. So, it's kind of a weird situation where I can't guarantee testing um like eight hours a day, five days a week. Um but I can say there's going to be sufficient testing and uh you know usually about 30 hours a week I guess 20 to 30 and that's usually

sufficient for the clients. So they're happy. They're happy. Yes. Yes. Okay. Um yeah I also do a bit of work with different universities. Uh I do some workshops with uh 10 or two Yensu and uh and Ala and um that's gone down really well and I've also done some user user testing with those universities as well. So yeah. Okay. And you you have some experiences as well. Yeah. Yeah. Basically we have mainly worked with with the studs through Deans Dean's company but also some some stuff by our own. So basically what we have done is we have uh ordered play testing from from the students and they have gave pretty pretty good feedback depends of course on the like the time of the year do they have lot of tests and that usually uh relates to the schedule of the things but then then also what we have done is that just invite some some students to

our office have them look at the first playable of the game have their first feedback back like directly into the site like what do you think, what's good there, what's not good, what you would change. So it's really good to have the university and the students who are interested in in games nearby. So you could you don't have to like go much away but basically just call to the university and ask if there are students who have free time. Yeah. And of course it's probably quite easy to to have the kind of local uh students test the game. Yeah. Okay. Okay. Uh, Nico, you've been involved with the actual testing. Uh, what types of testing have you guys uh done at the university? And and what kind of a team did you did you have? Because obviously you weren't alone playing all those free-to-play games. Yeah. Day and night. No. No. Uh well my team consists of uh software engineering students uh ranging from I

think youngest is now a first year and uh the oldest are last years. So from the beginning of school to the end of school well university anyways. Uh so the most important thing I think about them is that they are software engineering students because I hear that way too often uh game testers or gua testers are thought of as some persons that you can just pick off the street and give them bread or something and they test your game. That's not a good way to do functionality testing because what I've come to learn that in functionality testing the more you know about the software the more you know about the code the better results you get you know how to break the games again once again sorry for breaking your games that's what we

do so that's our team uh we do functionality testing basically we give feedback if such is wanted. But uh I I think that UX testing and so user experience testing and functional testing should be two separate things and uh user experience testing should be recorded from like one person at a time so you can point out the flaws and such. So I haven't done touched that. So functionally testing mostly. Okay. So what's it been like uh working with uh Dean and and QM? uh working with Dean has been uh he's a slave driver. Yeah, it's it's perfect. Nah, it's been great. Great. Uh it's the other students that are harder to work with. Okay. Because but so far it's gone quite smooth. Okay. Of course, we have those classes and I would love

to work full-time and but yeah, there are issues but so far it's gone. Well, so what have been the kind of main learnings from the from the whole process? Oh. Oh, I've learned a lot. I mean, when I first started, I was one of the guys who thinks that you can just pick people from the street and give them bread and make them test your games. Like mentioned, but now I know that GUA is important and really you should have every company should have a GUA specialist. If you don't have uh hire it and I've learned a lot about the process as a whole. Yeah, good. Um what about from the uh kind of business side? Uh what has it been like to work together with the university and the students? Has everything kind of gone smoothly or has there been friction or

uh I can imagine there's been there's been kind of a clashes here here and there. So So give us the details on on kind of what to look for when when starting uh cooperation with with university students. I I've not actually had much friction, I have to admit. No, it's gone pretty smoothly. I mean what I'd like to do is to take this model and maybe work with different universities and and expand it um different parts of the country maybe. Um but so far it's it's it's a model that that actually appears to work. It's you know it it's it's not uh I know that some of these large companies for example in Asia have hundreds and hundreds of testers that you could hire and if you have a huge project you probably need to go somewhere like that. But for the kind of mobile projects that most of the companies in Finland work with, um, you know, having a team of students working on them is is is probably sufficient as long as it is the right students. Like you say, it can't be just anyone. Um, but a handful of of

student testers um with with a supervisor who's able to to um ensure that the work gets done correctly, you know, that that's that's that's a pretty good model. And if if we can get that working across to with different universities, I think that would be a um a good way of ensuring that products get the right QA um at the right cost. Yeah. Yeah. Now that you you mentioned it, uh we actually have our uh game lab uh engineer somewhere probably in the crowd. Marco Oras, is he here or has he already left? Oh, I see. Okay. So, he has kind of been the the uh internal slave driver for the for the for the QA University. So we need people like that as well in in the university to kind of we need a champion inside the university to make it work. Yes. Yeah. And and we need a champion student as well because and Mark is alwaysounding me. He wants more products for his for his students because at the end of the day there's no better experience than to actually you know work with real products that real

companies are developing. um you know the soon as we can we can get that the sooner they can start to sort of get a taste of what the industry is really like. It's not just a classroom experience then. Yeah. All right. Um thinking a bit ahead uh and and considering obviously freeto play. Um what types of things do do you guys think that we should focus on when when testing freeto-play games? I mean I've had some uh discussions on on kind of monetization testing and and stuff like that. Do you think that that can be achieved somehow? Yeah. Yeah. I I um last year I had some some meetings with a company in Spain uh who make free-to-play games and uh we were discussing the QA on on of their games and they they they really weren't bothered with crash bugs. They they you know I said, "Oh, seriously?" Because to me, crash bugs were always like the the the bugs that you couldn't ship with and

they didn't care about that. They cared about monetization bugs. Anything that cost them money was what was important to them. And I was like, so they kind of turned the whole testing thing upside down. They they thought, well, the people aren't playing paying for the games, so um they're not bothered if if you know, they're usually a bit more tolerant with crashes. I disagreed with them a lot. I mean, I totally understand where the monetization uh testing is needed, you know, that's effectively what earns them money, but um I also think that there should not be crash bugs, you know. Um so yeah, that that we had a difference of opinion there. Um but most companies thankfully tend to agree with me. It's like uh you want the game to be as clean as possible. You do not want it crashing all the time. Um, but then at the same time, you you definitely do not want to see monetization bugs. Um, which is strange because with this Spanish company, I pointed out that they did have a monetization bug and they weren't happy.

Yeah. Yeah. Okay. How about you? Sure. Uh I think regarding freeto play is like if you can have some number of people like tens of people playing your game one one crucial aspect to test out is also the analytics side of things like how the analytics really working or not and that's that's quite neatly you can see through testing and also I think one one important thing especially if you have a multip multiplayer game is like the breaking the multiplayer rules. Those people are quite hard to find or at least I couldn't haven't found many of those. But those people who can see immediately how you can use the system in your favor. Those are the things I think I would focus on. Okay. Well, uh obviously there are some benefits um when working with a university. Um but can Can you kind of

give us uh your opinion on should the industry work more with the with the academia and and what are the kind of pitfalls and and and benefits? I I've uh worked now with like I say with about three of the main universities in in in Finland. Um I've done workshops where I I've started by giving a effectively a lecture and then in the afternoon we we've switched to some actual live uh play testing uh and during those sessions we I I've used products which my clients have allowed me to use and what we find is that it's kind of a situation which is win-win for everyone because the students under NDA will get to see a new game which uh they can then comment on and their comments will probably get recognized and used in the game. Um the the client loves it

because they get some free free feedback from the students and they know that they're expert comments uh because the students are usually working in game or hoping to work in gaming. Um and uh obviously the university is happy that you know the students get the the this this this great experience and uh it makes the workshop more interesting. It's like it's not just a a mock exercise. It's it's it's live. Um so I I really like that approach and if I could keep doing that with different universities and you know keep keep uh keep the whole the whole I guess everyone will be happy then if if we can manage that. Yeah. How about you Yuka? Yeah, I think I think main main points than Dean had but I think also one point is that you actually get get to know those people and some there's small percentage inside all all students who are really exceptional and you can like have the contact from early on and then build a real relationship and possibly

hire hire the students get like an inside view of those. Yeah. Good. Okay. Well, I think our time is up. Um, like to thank you guys for for joining me for this panel discussion. Uh, it's been a pleasure talking to you all. So, thank you. Thank you very much. And let's give a big hand. Yeah. Oh, anyone with any questions before we let these guys leave? One over there. Can I get the uh the mic? Here you go. So, hello. My name is Tuli. Um, I'm interested in when you're not released your game yet, how do you measure if testers are having fun?

I guess I'll start. Um I um for several clients this year um so before the game has launched or before it's even gone to soft launch, we've done some field tests. Uh during those field tests, we typically get maybe 30 players to take the game home on a device for two weeks in order to um evaluate the game and play it as though it's just any other free-to-play game that they would download on the app store. And um during this, there's two ways we evaluate them. Uh firstly, there's um usually analytic software built in. So the the game developer is able to um see when they're playing, see how long they play for and get many many many other bits of information from that. And secondly, we also give three questionnaires uh one right at the start, one midway through and one right at the end. And uh we try

to capture some uh some some different feedback that way as well. And between the two, you can basically get a lot of information. Um you know, you can see if someone really hates the game. Um, you can you can see if someone's waking up at 7:30 in the morning on Sunday in order to play it, which usually means they like it. There's a lot you can do to uh to try and try and gauge that during a fairly simple test. It doesn't take too much to organize that kind of test. Yeah, the same same rules apply. So basically basically we take a look at what kind of feedback we get from the uh testers and depending of course how many they are uh we might use quantity quantitative analysis like which how many% of the players actually like our game and in which scale if there's like from one to five the scale. So if we have four or five that would mean that we have a good game and if that's below

then we have a okay or or poor game. And of course during the soft launch of freeto play games that then we actually measure in in mass like how how the mass of gamers actually feels the game. Okay. Thank you. Any other questions? I think that's it. All right. Thank you.