Playtesting: Avoiding Evil Data

okay so play testing is in my opinion one of the most important topics in game design I find it exhilarating it's the moment that everything comes together uh or falls apart uh the moment where your banies turn out to be monsters or your quick fixes turn out to be lovers however most Indies I talk to see play test as a necessary evil a painfully confronting moment and honestly I don't think the biggest problem here is that it's difficult to receive feedback on something you deeply care about no that's something I think you can overcome relatively quickly I found that the biggest problem is that a lot of Indies use conventional play testing methods in finding testers in setting up play tests in gathering feedback and using surveys or using analytics and they end up with data that is hard to interpret that is confusing or even contradicting itself evil data it's dangerous data I'll explain how I find

it in a bit but it's the stuff that makes G that makes making games only harder and makes play testing way less appealing so the goal of my talk here today is um to to basically go over some of the things I found over uh making seven games in the seven games in many years so I've been trying harder and harder to avoid evil data and I've experimented with play play testing itself a lot I've rebelled against a lot of conventional wisdom uh um sometimes that fill tremendously but I I also found a lot of these play testing 101 things uh just not to be entirely true and so the goal of my talk really is to um make you think about how to make your play tests more valuable and how to make the play test easier to organize so on the schedule is um I'll be talking about what evil data is where it comes from and how to avoid it and how to organize more valuable play tests uh the talk will get more practical towards the end so bear with me

before I start I think I need to remind you that QA is not the same thing as play testing simply put I think QA focuses on getting things to work at all while play testing is much more about the quality of the things that work um thought I should mention it okay so my name is Adrian dong uh Adrian in simple English um I'm a game designer I'm from Amsterdam the Netherlands uh I am best known for the igf nominated fingle which is a finger rubbing game uh which I made with the now closed game muffen I talked about that this morning um and I'm also known for bounden which I also made with game oven a sort of mobile ballet dancing game made in collaboration with the Dutch national ballet and I just released hidden folks yay awesome um thanks a lot so uh this was in collaboration with the illustrator svan t uh in the game there's a bunch of targets they are

hiding behind things or inside of things and the only way to find them is by like unfurling tent flaps or like cutting through bushes or slamming doors or poking crocodiles um I've been thoroughly testing H folks and all the games before that for more than seven years now and evil data was lurking all along so to explain to you what evil data is here's an anecdote a long time ago I play tested hidden Folks at some event this maybe two years ago and at the third area of this there was a hint for a a Target that was in the garage most people relatively quickly told me they didn't know how to open the garage door they like tapped on the door but nothing happened and they couldn't figure out why it wouldn't open and people most people quickly quit after that so the situation was that you need to drag it open right it's like a simple drag but previously you would only have to tap on things so the next day I Implement a solution to this problem that the play testers suggested to me

which was a tiny hand that shows you how to open the garage door and this worked really well people know they can open the garage door now they sort of like act like their hand is the small hand but now the game has become really boring any previous feeling of exploration is completely gone and people feel like the game is treating them like a baby and this bothers me for like a month or so before I realize I made various mistakes in play testing and implementing this feature I listened to suggestions from people who didn't understand the problem or did not understand what I wanted to achieve with the game the uh people at the event weren't really patient enough and so they didn't really take the time to explore hence they didn't really find out how to open the door so turns out the problem wasn't necessarily that the game did not tell people what to do the problem was that the game was not telling people what not to do I realized that players would almost always step on the garage door right but in reaction to

that the game did nothing and so what I did after like a month of thinking about this I got rid of the tiny hand and added that if you tap on an object you're supposed to drag it Wiggles a bit in the direction of where you're supposed to slide it and this worked like a charm people more easily discovered that you had to slide it open but most important of all it people had a giant smile on their face and they discovered it by themselves you know um and this is just one example in which the play testing result I got from you know the all the play testing distracted me from making my game better I was looking for the wrong things and I listened to the wrong ideas and I think this clearly demonstrates that some data is timec consuming and it can make your game worse and that's evil data so evil data is play testing results that are distracting that are unclear or that are misleading um and in turn that may lead to you guessing what to do it may lead to a worse game it may lead to a less unique game longer develop time and you

know many other bad scenarios okay so that's evil data now let's talk about um where it comes from and how to avoid it because I've I've been seeing it in many places and I've been putting more and more time into modeling the play tests so I can avoid it entirely uh and I I basically distilled this down to eight points um eight things I'm now actively paying attention to physical location I remember John and Brenda Romero uh at fingles igf Booth uh my second GDC a couple years ago maybe six years ago gez um after my like it's like finger sex they reacted all like oh I'm not doing this this is disgusting who would ever want to do this um and they were saying it in a playful way sure um but they didn't play it then right um

I could have seen this as like a a sort of I could have concluded now oh you know maybe I work need to work on my pitch um I I should you know maybe make a different game um but I bet that if I if if they would actually be at home sitting together on their couch that exact same pitch would have probably made their fingers make love um so the data you gather from play test is hugely influenced by the setting by the the context of the play test a likewise I found that um people playing hint Folks at events tend to skip the bonus targets more easily and give up discovering the interactions more quickly than when they would play at home bounden the mobile dancing game I talked about suffered terribly from this evil data in hindsight the game was was built on the results of hundreds of play tests that happened at events playthroughs that lasted 5 minutes as opposed to an hour and as a result we designed the game around those first 5 minutes and the rest of the game feels

more like a repetition and this graph of bound in sessions lengths is really painful stuff for me as a designer like nobody 1% not probably like around 1% has been playing more my game for more than 30 minutes that's painful stuff so I want to avoid this and to avoid this to avoid this evil data resulting from the setting of the play test you just have to make sure that you play you know you play test in a setting that fits your game this could be you know some at someone's home at some you know at a party if you're making a local multiplayer game U wherever it fits your game okay so that's one two people it is tempting to play test mainly with all the other developers at your office or you know you have this group of testers the board game night that you're always testing with um but different people have different understandings of games and different experiences with the vocabulary with controls with conventional game ideas

with structures different experiences with prototyping with testing giv feedback so in an early version of hidden folks we had many of these like dialogue boxes and they appeared above these these folks these little characters and tapping them would bring up a little joke a very meaningless joke uh to give them uh to give the characters in the game a little bit more character um so a lot of people who don't really identify as Gamers thought they were actually really funny and then just continue playing the game but a lot of people who do identify as Gamers really fixated on these these meaningless jokes as if they were like very important information uh which it was definitely not and it was actually a giant problem and uh uh I gave um these these actual things gave very young kids the idea um because they couldn't read that they were missing out on a lot of information and I actually had parents come to me and complain to me about having to explain how the jokes uh would make sense and that was actually very confusing to their kids so it was very

different that like different groups of people perceived this speech bubble thing completely different so generally speaking I think this may happen to elements in your game as well I can imagine that if you're making a game for young kids it's for instance often played by parents first so if the parents don't find it like sort of amusing or they don't understand it the kids probably never get to play it and I found that uh people who play a lot of games all are almost always more impatient than non Gamers and it makes me wonder what the other differences uh what other difference there might be that influence the way that people perceive and play your game and so yeah because of this I think it's easy to miss bottlenecks entirely depending on who's testing so how to avoid evil data well never restrict your testers to a specific group of people even if those people are not part of your so-called target audience uh everyone I think has something to say about the game and you know my my mom has given me amazing advice on the game and she's not a game designer at all so yeah I mean think

about like co-workers other game developers you're into game friends you're not into game friends family young people old people people from different cultural backgrounds fans online community just like try it with all these different kinds of people and see how they perceive the game uh and you'll get different kinds of results and it might actually help you understand how different people play your game okay third thing your introduction this might be a bit of a beginner mistake but a lot of Indies tend to give a full introduction of the game before the testers get to play the idea the story the controls I think that is a huge huge missed opportunity when people you know buy or download your game they won't get that introduction and they might hit like numerous bumps that could potentially ruin the experience of the game um so the trick in avoiding evil data here is to sco the data as little as possible I have learned by far uh most about how to introduce the game the controls the UI and many other elements

and even I I even learn a lot about marketing um by just simply throwing people in and seeing where they really get stuck or what they eventually say to me uh you have to sort of allow people to discover how everything works uh and how the make game makes sense and do not mistake the time that they need to discover everything as a problem so don't tell people what to do do not explain the story don't explain what's missing don't explain the controls don't explain the interface just don't explain anything you'll see where uh the tester will actually hit hit hit any bumps there's maybe one thing I would mention and that is that the game is not done yet because this gives testers a mental slack for when they you know when they're staring when you know when they know you are staring at their actions like um while they can't figure out something and uh it sort of incentivizes them to speak their mind and be you know be honest about and don't feel so awkward of not getting it um okay number

four surface problems for maybe a whole year I pretty much ignored the difficult design problem that the difficulty of the targets varied very much um actually per player and that some players would get stuck on finding a required Target and quit the game prematurely I noticed this pretty quickly after introducing the um required and bonus targets Paradigm but as I didn't know how to solve it immediately I slow slowly started to become blind to it over the months following and so I call these sorts of problems surface problems they are like really obvious right there on the surface but you neglect them over time and another example would be that you can focus so much on the details of a character jump in your four player local multiplayer game that you may forget that not even 1% of the people who would probably like to play your game even have four controllers now that's a problem you should seriously be thinking

about um and it's very easy to sort of like forget about these problems um and there's actually design things you can do about this so how to avoid evil data coming from these surface problems well keep looking at the player's experience as a whole don't always test with a specific goal or a hypothesis or subdivide your play test into groups like concept testing or UI testing or usability testing or level design testing play test um off so often that you get a lot of data on the whole thing and it becomes harder to neglect uh all these problems okay this might be a little bit of a controversial one but um we'll see whether you agree with me on that or not because what happened to me super often in hin folks uh were layering issues making it difficult to see something sometimes because a character animation was randomly assigned and a Target would like blend with another Bush or something making it very difficult to see or I would like accidentally make a

build of the game with the setting screen on by default so every time you loaded a new area the setting screen would be right there it's like super annoying and the player would have to close every time and these sorts of tiny issues can have a big effect on the player's experience um and so it also has a big effect on the data you extrapolate from the play test people might uh quit more more more more quickly um this issue though would probably take less than 5 Seconds to fix and so what I do and this is really against every advice I've ever received from other game developers is that I often make changes to the builds in between play tests um and is yeah it's made a lot of my play tests like way more useful because I wasn't afraid to know then actually move the target a little bit or disable the setting screen um right in between builds um so yeah that's a that's a thing I do and I can recommend it um okay okay next so what testers say

testers start playing and they start talking the things they like the things they don't like the things they don't understand Etc but it's your job to filter out all the bull crap that they're saying and distill it to the root of each problem because people are great at saying what's wrong while never actually saying what's actually wrong um I I encountered this so often it's it's really frustrating the first hidden folks UI is is a good example um a lot of people told told me I don't want the targets to be on the screen all the time so I what I made is a thing that made this giant interface in the right disappear so you could press a button and it would like go away um but then I made that and nobody used it I was like huh what's what and no and also nobody complained anymore so it was very strange um and later on I figured that the problem wasn't necessarily that the targets were in the screen but that the pedal was simply too big and it was obstructing like a giant area of the screen duh but but the Dozen people that

played it and you know complained about this um didn't actually mention the penel being in front of everything so nobody said it but everyone was like sort of like pointing at it um and so secondhand information like what people are saying to me AKA evil data uh can be a huge huge distraction so what I encounter a lot is that people explain the solution to a problem they don't fully understand or a problem that might not actually be the problem or people give suggestions based on like this thing they saw in another game that must be way better than what you have um so yeah on top of this like social structures between you and the tester can make what people tell you very biased even when it's Anonymous through a survey or something so I think feedback filtering is probably the skill you'll have to train um to do your place as well so you take in what people say you always consider why why why the hell is he saying

this okay next surveys so surveys are by definition secondhand data um think about ratings ratings have not ever helped meet pinpoint problems so this one star review is like useless to me I cannot make level three better without simply guessing what's wrong with it I have to guess oh it's a shitty level okay now what um open questions people often these open questions with the solutions to problems that are difficult to read in between the lines uh here someone said I didn't like level three because it was too difficult I would make JN easier to find so you'd say oh maybe JN is the problem but actually it turns out the problem wasn't with JN but the fact that people didn't understand they could drag a car away that would then make JN appear so I I need to focus on making it clear that you can drag the car away nothing in this question ever hinted to that and uh yeah that's dangerous I could yeah that's dangerous stuff so I also discovered that the way way players perceive the game or its

features it changes over the course of the game I know this because when I ask people to record videos of their play sessions and this is something I'll talk about later and compared their videos to what they emailed me there was a very clear correlation between the emotions and the frustrations that they had at the end of their video and their emails um I I could see like the tone of their mail was way way more in line with the end of their play session so how do you avoid evil data with surveys and open questions and like tech walls of text and feedback well my opinion is no surveys and no questionnaires ever just don't there are other methods to know what people think that might be more consuming but at least they're way more reliable um next so making video games you've probably sent over builds of your game to other developers to friends asking for feedback maybe you've done a bet a

beta of your game the results coming from these play tests can be full of evil data uh easily turning your designs and challenges and player obstacles into guess workor I explored a few ways of getting valuable data uh valuable feedback from from online testers but each method has their own challenges when it comes to when it comes to avoiding evil data so just asking people is of course by far the simplest thing but you know it's also the most evil data PR way of gathering feedback for reasons that we've discussed before you'll get secondhand experiences it's written after the player is done playing and it gives the player time to think about the experience and give give you their Solutions instead of the problems so I actually don't aim to get feedback from testers this way anymore uh it's just usually too General to act upon using analytics to figure out the things you don't know paradoxically requires you to define a hypothesis and set up Trackers in a way that the accumulated data will tell you anything useful so

how can you prepare for something that you don't know so in this example using Game analytics which I can recommend if you want to go down this this this route I can see how uh how long it takes on average to find Targets in the camping area in hin folks so the mushroom in hin folks takes people much longer to find on average and maybe that's not even bad but this chart doesn't tell me why right I know it's longer but ites doesn't tell me why and that's so I have to guess why and I think that's evil data so another common analytics method is using funnels um seeing how many players convert from level to level or menu to menu um or anything that has a sequence to it what you're seeing here is the level progression uh of hin folks for the players who download the game in the first week so only 7% of the people uh who bought the game actually reach the end which I've heard is actually a really big number um so in this funnel also from Google

analytics loads of people quit at level three you can see a giant drop um but I have to guess what it is and again like guessing work very prone to evil data um I might change the game for Worse here okay heat Maps though are way more spatial than regular analytics data so it's a bit more specific and you can remove some of that guesswork uh what you're seeing here is Unity analytics running inside the editor and I was I was able to conclude from this image that that people are eager to tap on the back of the truck and the motorcycles somewhere like in the middle right um and those things were actually missing feedback and so I added those and now people would have a much more satisfying experience playing the game because more things would react to it so yeah I can do a a little bit of level design based on this it's it's a little less uh guess work um I can identify some problems but it still doesn't come close to actually seeing people play which brings me to using gameplay recordings I am super into this right now um I'm not talking

about YouTubers or twitchers who try to entertain their audiences um but I'm talking about people making videos of their play sessions dedicated to me a lot of the evil data you get from physical play test is still present in these kinds of play tests just to a lesser extent this is currently the closest you can get to a firsthand experience of the tester through the internet you see what they do when they do it uh you hear what they say when they say it and they are doing it in their own time at their own pace in a relevant context so evil data still comes from testers they still give suggestions while never describing what the actual problem is fortunately you can actually see what they're doing so you can you know it helps you filtering and also see what the actual problem could be um on top of that people are still very aware that you're watching uh their every move so they they sometime actually most of them just talk to me personally as if hey like ohy Adrian

look at this thing that I see here it's it's it's almost as if we're having a a conversation sometimes um even though it's just a video for me um but my biggest problem with this method is that it's a hassle to set up so there are people that wouldn't bother doing this uh like my mom which is a problem if you know the target audience you have in mind isn't that techsavvy so this kind of super this kind of footage is actually super useful to me in many ways think about timing level flow big things and small things and I think that if you want to avoid evil data this is probably the best starting point right now but do do keep in mind that this will not save you time over physical play tests you still have to sit down and watch hours and hours of footage um yeah okay so those are the eight things that I found that you know evil can uh help you um think about evil data and how to maybe avoid it so how to organize play test with minimal effort let's talk about how you could do that

the thing is if you try hard to play test throughout the development of the game instead of when it's almost done you'll have a much better grip on the player's experience um you'll save a lot of time because you won't be working on the wrong things and you'll be able to work towards version one confidently if you play this throughout the project you will see the nuances of your design for instance we couldn't quite put our finger on what it was that made some of the hin folks areas more fun than others and it took us I think maybe a year to understand what the parameters were that made the game fun the density of the visuals versus the density of the interactions the 17 different kinds of hints that we found the different kinds of uh interactions that we had the uh visual order or the chaos Etc all nuances that we found because we saw so many people play the game and uh thought about it so long but you can just play test every day right

there's a lot of overhead uh you need to make a working build you need to get people to play for physical play tests you need to be there take notes uh you have to reflect on your findings and all that before you can get back to make to making the game um so the way I've optimized it is to trigger play tests at certain moment in my development I try to do a play test when I Implement a major Feature A minor feature or when there's an event if if any of these things occur I would set an almost automated first step into the direction of play testing and I would just roll with the the play testing structure that I had in mind um which I'll explain in a bit um with hint folks we set out to play test with every new theme for instance uh I organized a few offline play tests with non-developer friends and family uh and online play tests with people who signed up to test on our website and the way I organized them um was really um uh

I have all these template emails I can

pretty much copy paste uh iterate on if needed and uh and just send out uh it's it's really simple and it's surprisingly effective everyone I ask for physical play test literally everyone just joins me for a session I maybe this is also a social thing but yeah everyone just likes to do this and about onethird of people I ask uh for online play tests they respond within a week and then another then another 1 six to a third uh within 2 weeks after I send them another email so with for hin folks this meant that for 4 months especially during the last uh um development of the game uh this meant about 20 to 30 hours of super useful gameplay footage this was like my breakfast every morning I would sit down with someone else playing my game um and these are actually really great numbers for something that I you know I initiate within a couple minutes uh so yeah uh it all starts with a simple email minimal effort and I had a pretty pretty amazing responses so think about think about

doing this it's it's really simple to inspire you to think about sort of alternative ways um here's three other things I've also tried I play tested Bound in every other day so we like made a level on day one we play tested it on day two made change it on day three play test on day four very intense to uh to do uh super fun super fun super motivational um yeah it didn't really give us any insight into level flow though uh fingle is like a party game you know and all the finger rubbing is just weird and awkward and social so the parties were just really really the perfect setting for it and I was still a student when I was prototyping it so I just like take the game to parties every weekend uh that was perfect um playtive Club it's an event I organize for game designers in the Netherlands to come together and play each play test each other's games as all play testers uh as well attendees are game designers you can really talk about your game design and get feedback in the form of brainstorming together really trying to

understand the design of the game uh and people will play for longer than uh than for instance consumer events they stick around give feedback but this definitely doesn't replace uh play test with consumers designers tend to see problems that aren't problems yet um so yeah I hope maybe these three things will uh will inspire you to figure out something for your game so to conclude how to avoid evil data play test in the right setting uh make sure you know the location fits your game play this with everyone so that you don't you don't stare yourself blind to all these surface problems don't explain anything just throw people in the game see what they uh see where they get stuck always play test the whole experience make those one minute changes if necessary it might have a big influence on your play testing results filter feedback always uh never just listen to people and do what they say

[Applause] [Applause]