How to conduct playtests when you have no budget, time or resources

[Music] hi everyone my name is becky nunn I'm going to talk about plate tests and how to focus your efforts and really cut budget cut time cut resources which I find really important both for Indies and and for big companies a few words about company and from gizmos we're doing a mobile app developers will focus on music app majorly and also do publishing we have two offices one located in London one located in Minsk we have at the moment around 12 apps on our brand have around 300 million downloads have around 15 million monthly active users have a number of apps on top 10 on App Store and Google Play so majorly do a music gaming which includes rhythm games simulations we also do some products for professional musicians our

top brands would make a girl which helps it to create a tonic music right you have a number of similarities and a number of utility apps like metronome or tuner which help professional and amateur musicians as well so getting paid to pract getting back to play test so raise your hand if you've ever done a play test yeah and raise your hand if you thought that it could took less time then it actually took Sona y'all like super efficient well that's great but I mean from my experience sometimes I'm doing a play test and I feel like yeah I mean I think I could have thought I could have prepared better I could I think I could give people a task something to test really nice and I think play test so been a great and really important thing to do for products it can also be a drain of effort and so we decided to

share some tips and tricks and framework how to make all your efforts focused and how to get utmost from this process a few words why plastic latest are important in my point of view so the the first thing is that you receive information earlier and this is important because the earlier get information the more conscious decisions it can make right so this kind of gives you focus and knows in what direction to move it allows you to study various user patterns because well basically what we create an app we're thinking more tellement out ourselves or a team but when it gets down to life it will figure out that there are many different people who treat output really differently and studying this view user patterns early helps us to get this product more applicable to larger audiences and finally it helps you to reduce things last because well we all know how painful it is to reduce things especially when you asked your team to do something and then you say well

figured out now you have to do redo it again so instead of you know the motivating team and even the motivate in your budget so I think like introducing playtests in right places in the right way help you to eliminate all those issues and really get down to what actually works right so in a deal world it's best to introduce the test as early as possible because the more information you have the better decisions you make so if in ideal world if you have an idea you test it then you might might change the idea while you're happy go to prototype it has the prototype it gets once happy with a prototype you go to implementation you test implementation and once you're happy you release and it as well might lead you to new ideas actually not only at the final stage but at all stages so Dale what you want to have play tests and ever ever stage in reality I mean it it sounds costly and

it might be costly because this is a lot of iterations and this is a lot of people who vote in the process yeah so I'd say that it's best to start doing player tests as well as possible but I found it like the many people are not really comfortable with starting you know from the very beginning and the earlier you start the more vague the artifacts are which is an issue so you have to be more bachelor better prepared so I'd say like starts doing play tests where you feel comfortable and move like la la with every other product so yeah and when preparing yourself for proper test we'll split it in three parts which is a product you in TV and yourself so we start with a product so yeah I think you'll agree that plates are really important for a product but the question is how to make it really work for a product and bring the best and really quickly because I mean as a business anyway one really quick results and like real nice and

quick results so and probably cheap rate so I'm be using for example one of our apps is which identity choose a smaller app just so we don't have to explain it yourself the story is that we have a metronome app which will launch it 2012 it was really popular in morning's audience but at some point you've got you know a bit outdated so we decided well it's time to fresh it up so we're not only fresh stop the design a little bit information architecture we since we made you know in you know music gaming we decided yeah well good at games why don't we do some music games for for the app so we introduced a few rhythm games here right so we kind of started the audience figured out that majority of people use our app as students amateurs so we thought well helping them to practice a little more with rhythm games would be nice and engaging experience just so this way kind of moved from the TLT apps to game here as well and we conducted a number of play tasks so we

will be discussing rate so and we came with a framework who found really useful I mean it I mean it's not only useful it's really simple to introduce to use so what we figured out most important I think this is like the main idea is that when you're doing a play test you're actually not a test in the game but you're tasting the hypothesis about your game and this helped us a lot because it's not like basically asking a person to use your app and trying to find something just drink up this proper preparation with making a list of hypotheses really makes difference because it helps you focused and it makes this whole process so much more scalable because well I'd love to have a number of clones of myself just to send them to many people talk to so many people many other you know interviewees but at the end of the day it's like we have limited capacity and so what we did is we made a brain strong with the team we made a

list of hypotheses of things we like and we don't like about the you know the app the process the part of the game and so allocated its to each screen and this is like was the starting point so and so I'll show you the way works for one of one of the hypotheses so one of the games were called blood beats is it a video yeah so the there was a that beachner anterior tree trunk and you had to you know repeat the rhythm and it goes and all the you know melodies get harder so keep playing with this thing and at some point you notice that well for some of it is that at some level of complexity they get a little bit too hard so we want my want and listen again button so and as you see the top boy semi-transparent so we like okay so that would be semi-transparent but already a design stage we had an issue maybe it's like not hard to notice so had some you know discussion within

the team and instead of arguing and just basically you know making decision out of nothing would it play task figured out that while actually people cannot find the button and I mean turn it took more colorful one which worked perfectly and it's like a great example like how you can you spread house not only for only for large you know testing of ideas but already at implementation stage 15 up things and avoiding you know making a more conscious decision avoiding you know any ruffle any you know argument within your team and now like everyone is happy with the button because we know that this is what actually works so so this way you kind of create many many hypotheses it can be minor things major things things you have concerns about or maybe things you don't have concerns about but you can't try and think of think around the things so idea is I mean it's a good if you make analytical processes either put them all in a positive way or in a negative way because when once you've been filling in TVs it would be easier

for you to go for pluses and minuses and now you need to create scenarios so here this part what's really well so you see to which screens your hypotheses are allocated and then you create a scenario well to use it sounds like really simple it's battery if it is one sentence because uses tend to have short memory and you see there's a lot of information for them like app is new you and use and so many things so but I'd say like to Sounders max so for users it would sound like this complete a round of blood we game but for you it's like a map you like expect you know like they will go here they go game cover they need to do this this they go to instruction and you like basically have this map yourself like knowing that well I want to check this hypothesis here this is people--this is here there etc so like that sounds simple on the interface at the background you have all these things and after you down the interview you

basically fill it in I think it's a nice way to have a comment section down below the great you have rate and maybe put some more information about the interview himself or herself great and this this worked really well for us because this is easy it's easy to introduce to other people and to teach them how to use it and on the one hand on the other hand it gives a kind of statistic type of things because I mean they'll play test this you know you don't probably want to go for like really you know mathematician type of statistics you don't want to go to for confidence level etcetera but you really want to have understanding because no matter how much time efforts you spent on your game there will be people who will get certain things and won't get the same things and that's fine because that of the day you know game design it's not like creating a perfection but it's about creating an entertaining nice experience and it's fine that some they

use a pattern when people probably won't get something but I think like it's not that like 80% of the game this is still okay rate so going back for this framework there are more things that you actually have to prepare so a few words about how to prepare you in TV because there it is a challenging experience where people can conduct an interview at Fletcher's it is also challenging experience for people after Darrell like to major issues like well one thing is that people don't really want you to give a negative feedback and you actually have because negative feedbacks it's really useful and like you actually go for a play test hoping that they will really really like it and love it and be happy but actually what you actually want is a negative feedback so I'd recommend like before you do later and especially you know people see how you're passionate about the product you created it it just brought it to them and they're like oh I don't want to hurt anybody at least I don't know these people etcetera so I pay special attention to like that I'm here for

negative feedback and now the happy to hear that and even like kind of navigate the interview trying to figure out this minor thing is that candy you know and the problem another thing is like there's a different type of in TVs is that this who really want to help you in a way that they give you solutions they don't really want to discuss the problems that I have added a feature for you and another feature for you and this is really great but I think that's what's important here is to take the solutions but also to understand why they want it and what the issue is and there are a number of methods to understand this I think the easiest one is to use 5y so you ask questions trying to figure out the problem area or maybe what what they why they like it because you know like you're more experienced was designing game well then you users so but your users know better what they want so this is how you kind of meet here - I test and I think it is no less to you know

prepare yourself because the test is usually it's like really encouraging experience it also is like I don't know but I've been like I have done so many play chess but I still feel a bit nervous that I'm going in there because well I'm showing something and it's like a little bit nerve-wracking to me so I think problem wrong and I think this is the reason why many people avoid play tests at least avoid them at an early stage is that this feeling and that product is not trying to be displayed because without you know doing a play tasks we know that this is just a matter of research anyway we feel like wow I want to be praised for things for show so this is I think similar that to people who don't want to get that negative feedback is hard for some interviews to get this negative feedback so I think same mantra what's here as well and I'll show you like a brief you know sample from the app this is like another game which you

know happy to study inert values and I mean we started doing some play test I read it you know they the prototype stage so we imagine like this is the way we imagine pretty soon we figured out that people cannot associate notes and time signature so we moved to this variant with you know a bigger time signature nice you know layout so people now got the idea but they couldn't get you know they didn't like the numbers like all the numbers had to address if it were like oh come on but okay so we changed the numbers and now like yeah the numbers a nice but it looks a little bit down so we ended up with this type of game design and of course I mean we could have you know moved and just spend some time with our designer and end up at the number four but there's high probability that in our brain strong within our team we'll probably end up at number three or maybe we didn't didn't know what's what we don't like and what users need so I mean there is a low probability that we'll end up at number

four looking this way and knowing that this is what our user want so basically I think play tasks at early stage is useful because it gives you the right focus and right direction and kind of if you have this feeling which is I think really important that you know what you are doing and you know like that this is like nice things which would be accepted by users and actually needed by users and just another like quick problem I personally have issues all the time it's like we still want to be praised and we want you know to think about many things beforehand which is the nice thing is that if any product design a product manager but still you know like protests really cover not only like major issues which is nice but you can figure out that really small things you could talk talked about but I think it's important to understand that this is just a matter of research and it doesn't matter who tells you yourself tell you well this is an issue we should fix that like a user tells you because anyway is like a giant

effort like a small example was I don't know you can see here but I mean there's like a yellow sun on the yellow background which is I mean yeah I cannot even see from here so at some point there was like a design and they she was that there are like different like some users considered it as a button and so we change it and yes of course we could have thought about it ourselves like at the earlier stage but I mean having all this you know minor things covered by you know playtests really brings you to help you to focus on things that are important creating new things are polishing music creating new logic so it's also like this minor issues are covered just by plateaus you don't have to worry about it and as a final thought I'd say think of play chests not as tasks but more as a journey to learn your customers in product better because I think this is what matters the most and don't think it's as you know

platinum we have a negative experience as its testing your ability or your design it just matter research and much of joint research and joint efforts together with your audience thank you for your time one of the biggest things I have when I've done these kind of play researchers we did a lot of this a Playstation home as well the difficulty is making sure you're open to that feedback yeah not try to solve it yeah exactly I feel like it's really important to be mentally prepared and that's why I was saying that well you as an interviewer yourself are also important and the open to negative feedback they'll have a positive feedback and the open like to not get in hurt by it because I find this is why there isn't give a reason many people give up on it I'll just avoid it Aleksei yeah I mean understand this is important let's do it at the very very

very end of our launch and I think the reason for that is not like money it's not resources but it's like this emotional experience so I'm that's what I'm saying like be prepared for this mentally and be fine with other people like finding out things that happen you I see one of those we used to do is we used to not be in the room have independent people running it because the trouble is I'm going to show you how to use the UI and I know how to use the UI and you're stupid if you don't addy do it my way not sure obviously and but the thing is you have to if I had to tell someone else how they've got to then do it to all the people it really does show you where the floors are and we actually used to sit in a room with a one-way mirror so we could look out at them and they couldn't see us and it was amazing how just things like that really do change the way people react if they're in a

nice cozy looking room that's got a nice mirror they may know that there are people watching them but they forget and you get a lot more honest responses I thought my family yeah I mean I'd say it's my dad say we do as product managers would do some of playtests at initial stages where it's you know more influences business decisions because I find I figure like prototype stage prep prototype stage like pitching etc I think it's so hot it's so vague so it's easier that you say and going forwards to implementation stage for many reasons because in the the further girl the more people you want to be involved in your play trusts so we do part of them are done by me majority are done by do some okay a team loves to do play test some independent people have to do play tasks so it's it's like we sent ambassadors all over yeah and I find it really useful when different people conduct play a task because different people notice different things and have different you know patterns well we had a so have you

make working on a rocky Horeb show game but really much not yet and we had a lot of very you know passionate fans and we got some great feedback from them but it was all very positive and of course the moment we went to soft launch it didn't work because they weren't looking at it as a game they were giving us feedback as a game they already had a frame of reference which is are we representing the brand of Rocky Horror right which we were so it's incredibly important to just actually like you say you know the hypothesis and setting that hypothesis at the beginning and I would argue the sooner you do that the better but the sooner you can give up having anything to do any knowledge any involvement with the actual players as soon as you can get rid of people who don't care about you emotionally the better so you know I think this is a mazing process and soon you can go from this to shifting to our live open experience the better yeah I can agree with that yeah thank you thank you

silence yeah I hijack the questions there because it's something I'm catching over [Applause] [Music]