How Design Can Ensure Impartial Scientific Fairness in User Research

hello all right hi so my name is deborah henderson and i'm going to be talking today about how design can ensure the impartial scientific fairness in user research but first i want to introduce myself and ask you to please turn off all of your phones and various i don't know noise devices that's what they're referring to as though you've brought a set of symbols or something but um i'm going to talk today about a bunch of different things um but first i am a user researcher i have a phd in cognitive psychology i've worked on a whole bunch of games you'll see a lot of examples from the games that i've worked on in this talk but first and foremost what i am is i am a designer i am a designer of research and i have come to ask a favor of those of you in the audience who design games which is i'd really like to stop talking about methods and this this may seem like a very odd request to make as a researcher i in fact love methods boy do i love

methods but the reason i want to stop talking about them is because what methods are for me is there are tools they are tools that i use to design things and when people want to know whether or not my designs are good the question you want to ask is not did you use an axe but are you an axe murderer right the way that you use a tool is actually vastly more important than the tool itself and so instead of talking about methods these are the three things that i want to discuss one i want to talk about what ur needs from game design because let me be clear without the assistance of game designers i will never be successful two i want to talk about how i design bespoke research because every single piece of research i design is in fact bespoke and three i want to talk about how to ensure a just interpretation of data because i think this is probably the largest friction point between game design and research is making sure that the interpretation

is in fact fair so to begin with i'm going to talk about what ur needs from game designers and to talk through this i'm going to use a case study it's a little bit of an unusual case study because it's from the 1740s allow me to introduce you to a designer this is samuel richardson and he is a very good designer he designed he's known as sort of the father of the modern novel so the field that he is working in is the novel right that's the hot new uh technological wonder for entertainment of the day and he builds basically the first successful franchise and that i mean he writes two books one's named pamela one name one is named clarissa in the tradition of all fine franchises they are essentially identical books with different titles and he is hugely successful with both of them okay and the real problem that he wants to solve is how do you get people to care about fiction and this might seem like a problem that some of you in this room are also

working about how you get people to care about your imagination imaginary worlds and things like that but he means this really literally because this may not be obvious but the problem with fiction is it's all lies why would anyone care about a character they know is imaginary this is the kind of problem he is trying to solve and his main theory for how to solve this is something that's going to be known as the willing suspension of disbelief he thinks that if you just get people to treat fiction the same way they would treat non-fiction you can get the right kind of engagement you can build presents right so this is an idea that coleridge is later gonna sort of codify when he says a semblance of truth sufficient to procure for those these shadows of imagination that willing suspension of disbelief for the moment which constitutes poetic faith and so the goal that richardson has is sort of minimize the gap between fiction and non-fiction so

people can treat his fiction as though it is non-fiction and his solution for this is actually a really interesting one because it is a technical solution it is a solution that can be applied independent of content he thinks the future of the novel is the epistolary novel all right so the epistolary style is where you write letters back and forth and he thinks this for really beautiful reasons he's it sounds and just tries to calculate um sort of how you minimize the gaps between fiction and non-fiction he points out with this letter writing style you just minimize the number of leaps of faith you ask your consumer to make so you have to assume the following things one you have to assume that people are slightly more verbose than usual and remember this is a time when letters are actually delivered three to five times a day so it's more like email and less like snail mail um second you have to assume that your letter writers have good memories so they can include all the good bits and their letters and third you have to assume that people have an unusual propensity to keep their letters because in theory

what the novel is is a collection of those letters and what i love about this is this is both a beautiful demonstration of design thinking this is good good good clean crisp design thinking and it's also just fundamentally wrong and it's in that sort of tension between really good design thinking and like anybody who's ever actually read the book and thought no this the epistolary saw i don't need that like that tension that's not only where user research lives that's actually where science lives because what we have here is falsifiability okay and this is the one thing that every scientist will have in common no matter what their field is is a deep and profound belief in falsifiability and falsifiability is the idea that a scientific theory must be capable of conflicting with possible or conceivable observations okay this is how science builds its identity in

fact when we talk about science we will draw a line between science and not science that is entirely based on falsifiability so for instance albert einstein is not science okay in 1905 albert einstein has what is known as the anus mirabilis where he publishes four papers one on the photoelectric fact one on brownian motion one on special relativity and one on that old chestnut equals mz squared of all of these papers he wins the noel prize for the first one the photoelectric effect and the reason he gets the prize for that is not because his thinking was better in this paper than it was in anything else but because a man named millikan could come along and test it and he could run an experiment and he could set einstein up to fail and you know what einstein didn't fail and because of that we had evidence we had scientific evidence that he was on the right track right and it's it's important to recognize that falsified is actually a

relatively new idea so popper for instance died in 1994 right this is all 20th century thinking for science but for games i actually think this idea is core because as lou gehrig once observed i love to win but i love to lose almost as much i love the thrill of victory and i also love the challenge of defeat game designers understand that a victory isn't a victory because it's a win it's a victory because it could have been a loss right the value of the win is not in making a whole bunch of cheap wins it's often and really sort of riding that ragged edge of destruction right getting really close to failure and pulling up because that's a real win you really earned that one so what i need from design is i need you to be einstein okay i need you to build me that theory i'm going to talk through a sort of practical way to actually do this

so the way that we do this at microsoft um is we have designers the first thing you'll do when you when you start onboarding with ur is we won't run any studies instead what we'll do is we'll sit down and talk about your design intent we talk about it a very sort of systematic way we want to build out a theory that we can then test in a falsifiable manner and we do this in a couple of ways first we try to build out game specific experience goals what are the promises you as a game designer are trying to make to your player a second we break this down into sub-components because generally speaking uh the people that we work with are very ambitious we need to make them into sort of more consumable units the next thing we do is we make sure that all of these units are measurable and by that i mean falsifiable um and then finally the the sort of last and most important thing that we do is we write these in a way that's feature independent because here's the thing games start off they're scoped in a certain way and then some features work and some features don't right and it's okay for a feature to disappear as long as the experience ends up being there in

the end right so i'm going to give you uh just a real example of this i've neutered it a little bit to sort of hide privacy and things like that um but here is a sort of basic way that we break down sort of open world right frictionless exploration that's the goal right and it and what we mean by that is i'm motivated to explore and do so without limits or friction i then break this down into sort of more useful components all right need pushes me out my base is not self-sufficient but needs me to go out and explore curiosity pulls me further i may head out with an initial goal but the environment i see and the people i meet tempt me making me want to explore exploring as frictionless exploring is rewarding and as i read through these these may seem very generic but they're not and they're not in a very specific way so think about the word curiosity this probably aptly describes a game like fallout 4 where you start off and you're walking out and there's an awesome silhouette and you wonder what the hell that is and you move in that direction and you just what's over there i should go look over

there but i'm not sure that it accurately describes a game like don't starve where the reason to explore might just be fear right so this is specific to the games it's just not because of the features that they're building in it's because of the experience they're trying to give a player so if you can build out a theory like this i can then design research for you that's specific to test this particular theory you as the designer have defined what success should look like and i'm going to measure against that and i'm going to walk through here a number of the sort of design pillars that establish good ur practices and these are things that independent of methods should be true so the first thing i want to talk through is power so power is basically the relative strength of a signal is the it's the loudness of the thing that i'm trying to measure so i'm going to give you an example this is a video they're actually um five people here the fifth one's in the missing corner until they'll show up shortly and what i want you to do is i want you to watch

these videos i want you to look at the eye tracking that's little red dot and i want you to tell me is there a pattern or isn't there can i do this [Music] so when i look at this i think those people are behaving in a very very similar way and this tell me tells me there's a lot of power to the signal in this data and this is an important concept because oftentimes the first question that somebody asks after i run a study is how big is your end and what they mean by this is how many people did you run and this is oftentimes viewed as a way to judge the sort of goodness of a study and the answer people are looking for is oh it was terribly big i ran lots

of people right and non-researchers hear this and they're just like this is awesome this is a very expensive study i'm super excited about this they like lots of people here's the problem i will ask this question but here's how i'm asking this when i ask this question what i'm really saying is are you cheating do you have a real thing there or you did you just run enough people that you could overcome a very small power thing and discover it because one of the things about things that are not very loud things that don't have a lot of power is that you can test them if you just test a lot of people and they show up and they are fancy things like statistically reliable right and that's not actually a good reason to care about something you probably care about something if it actually is really truly meaningful and what's interesting about this is there the way that you calculate an appropriate end does vary based on the

method that you use but this is actually just a bunch of math um and any you are is going to know how to do this math right sometimes you can use a small end sometimes if you're asking about emotional questions like is the game fun or something like that you are going to want to increase it but overall when i say increase the end i'm talking about maybe running 30 35 people at once in a given condition and if you have somebody who's presenting you data and they're like this is statistically reliable don't worry i've run 2 000 people the question you need to ask is not how big is your n but what's your effect size okay this is again is a very basic sort of math thing there's a standardized way to interpret this you can ask that instead of what's your n you can google what the correct answer is but it should be big if you want to care about it right and what's great about this is that power is one thing but we really sort of balances against is an assessment of risk and there are a number of ways to judge risk i think the most common way that people judge risk is in terms of just popularity right how many people does

this affect um as an example of this if you're ever captured by hannibal lecter and he gives you the option do you want to break your leg but it'll heal or i'm going to mess up your ankle but you're going to limp forever the option you want to pick is actually to have him break your leg and the reason is because this is a sharp painful thing but our stress systems are very capable of dealing with this um conversely if you have a very small thing that nags at you for the rest of your life this is awful it makes people terribly depressed it's really bad an example of this in game play is retraversal if you mess up your checkpoints you make people rerun that like half a second of like information that they didn't want to see that cut scene that they didn't want to watch four or five times this becomes stunningly grating right and it's because of just the the pure repetition that that generates risk the problem is there's another way to generate risks so one of the things that we'll do oftentimes is we'll present people with graphs that look like this and a common reaction from designers is this is awesome most people love my game those people at the bottom i plan to ignore them they are outliers

from a math perspective you might have a point though there's actually again a mathy way to actually calculate that correctly and they may not actually be outliers but from an experience perspective i will never throw those people out and the reason i won't throw those people out is because those are the people i am most worried about those are the people i need to pay the most attention to because something has gone terribly terribly wrong for them and i need to know what that is so i can make sure that never happens again let me give you an example so this is an example from a game called haunt it was a connect game so the way that worked is you would fight various ghosties in a house and they would give you instructions on how to battle them so in this case you're supposed to cover your eyes so you put your hands over your eyes and the connector would see this and you would be essentially be shielding against an attack but here's another prompt that they used so there's cover your ears and then there's this one and what is interesting about this is this is in some ways not a failure

in any way shape or form this player in fact beats the ghost he is a winner because connect can't tell the difference between this gesture and this gesture when you do them repetitively but i would argue that you need to see very few people make this type of error before thinking i should i should really fix that i should not use the word thrust here i should maybe use push or shove or something that implies that i'm using my hands right this is another kind of risk and one of the things that we will do just in all of our reports is we will go through and we will give a severity assessment based on this kind of risk so every time we will write up a report some things are very important some things are less important we'll just communicate them to people so another pillar that we want to talk about here is control and control is the idea that if you subtract information out of a situation you can in fact reveal so i'm going to give you an

example i'm going to show you two videos from a usability session we are playing by standard usability rules here which means that none of the participants know anything about the game um and i want you to just spot the differences [Music] oh so that's the first one and here is the second whoa whoa what just happened what are you doing you're supposed to get in the hot lap what is that so this second reaction turns out to be i would say

the closest i've ever seen to universal in a sort of like you have an infinite number of things you could do and what do you do you get under the lava um if you are under five and have not played this game before um and have never had the experience with the idea of hot lava everybody just climbs underneath it there's in fact an achievement if you do this which presumably only children receive because it is so common right and what we've done here is we've controlled out for the most common thing that i have to control out for which is knowledge in this case it wasn't actually the designer's knowledge it was the parent's knowledge he looked at the game and he said i know what this is i know how to play and they played that way and they had a wonderful time it was delightful and fun but this also was pretty amazing for people right and controlling for knowledge in the lab is important but it's also actually important as you sort of expand outwards and the other big place that you need to consider control is actually in your org chart so this is my org chart in its uh game designers here represent

go up through phil spencer there is a layer cake of useless people in between and on the other side there are user researchers and i report all the way up through somebody else this means that for a game designer to come into conflict with me and in some way limit what i'm able to say in my analysis um they have to go over phil's head and i'll be honest nobody ever does that conversely this means that design is always also free to bet against me right because the thing is what makes a good designer makes a good user researcher are very different things and what you want is somebody who knows something about research judging me and somebody knows something about design judging a designer so the main thing that we control out of things is noise and i'm gonna give you an example from toy soldiers cold war so in this game this is a screenshot from the released version but i want you to notice the little bar at the bottom that says barrage um this is something where players would go through they're getting a gun

they'd start killing things and if they killed enough things quickly enough they would unlock basically this awesome reward which is a little rambo and you could run around and shoot things it was amazing um the problem is that when they first built this that bar and in the corner it wasn't blue it was red and this meant everybody thought oh my gun's overheating so instead of going and actively pursuing this barrage they would get close to it and they'd be like i gotta stop and everybody started actively avoiding it and what was going on here is that players just didn't understand the mechanic now imagine for a moment if i put um this game where people actually understand what's going on and the previous game into a play test and i asked people to say which one was fun odds are this one is going to be more fun the released version simply because people actually understand how to use it right comprehension was a blocker for me it added a bunch of noise i couldn't ask about fun until i could be certain

that people i i knew that people understood the mechanics we talk about this a lot at microsoft um and we basically think there's sort of a consistent flow in terms of how you need to test things first off you need your core mechanics these are like what you're doing in the game walking around shooting things picking stuff up that kind of thing but the next thing that we actually need are learning systems and the reason we need learning systems is because we need the game to successfully communicate to the player how they need to interact with the game and these don't have to be overt to tutorials i've seen a number of lovely talks here um this week already about sort of subtle ways to do this kind of teaching um but people do need to understand your game and they need to understand your game before i'm willing to actually ask them what the experience is like is it fun because if i don't ensure that they know how to understand it they're they're going to judge you very very harshly in an unflattering way and for really unfair reasons so with all of these principles i can go through and i can build a great sort of design for research and that i mean a process to run in a lab but the next

stage is actually what i do once i get the data back and there's another question of just how do we ensure a sort of just interpretation of data and i think that this is a really important one for us to talk about because i'll be honest i think most of the time um game designers look at you are and think you're a big flamey sword wielder and don't really understand how we maintain the sort of fairness of our interpretation and again there are some basic principles the first one is a priori so this is latin for what comes before and it's a basic rule that before i run any study i should know how i'm going to interpret the data okay so an example of this is uh at microsoft we will go through and we will do things like fund scores we have a whole bunch of scores that we do and oftentimes the first question i get back from a designer is well i mean i got a 4.1 what the hell does that mean and that's a fair question right is it good is it bad i don't know and the reason i don't know is because um it actually sort of whether or not

4.1 is good depends on two things one it depends on the quality of the thing that you're actually testing so it might be it's going to be a reflection of that but the other thing it's going to be a reflection of is the willingness of participants to actually pay you compliments um and this turns out varies based on the type of thing that you're asking them about so in this example i've got three things graphed up here we have collected for years and years and years in the exact same way a whole series of questions and we test hundreds of games with them and this is our distribution for them um and so this tells us sort of the average score that every game gets and what's interesting about these is that the distributions differ so if you look at them hud doesn't score very well ever okay and this is because typically a good hud is an invisible hud and if you notice the hud so you could pay it a compliment that's a that's a little unusual right most of the time when hud's working well nobody has anything negative to say about it um conversely graphics people will lavish praise on graphics

like people know how to talk about it they know how to judge it they feel really good about it it is beautiful you get five out of five right they're willing to do that and fun falls somewhere in the middle so whether or not a 4.1 is good it depends for hud it's probably friggin amazing for graphics no you that's embarrassing right and what's interesting about this is that i know all of this before i run my first study if you sign with us and i go and i say hey i'm going to be using a researcher i can tell you three years in advance what i want to see before you actually ship i'll be like these are the scores i want to see this is how you should be doing before i'm going to feel comfortable right so the next thing the data needs to be is it needs to be actionable and what i mean by this is you need to actually be able to use it in some way if i go through and i tell you you're a terrible designer but i don't actually tell you how to help that or fix that in any way that's really really problematic um and this actually means that i do testing not necessarily on games but on my

own methods and i gave you an example of this so i'm going to show you two trailers with some eye tracking one of them was a prototype and one of them was the release version and the question that i had when watching these was could i actually test the prototype and would i actually get data that was representative of what was going to happen in the release in other words could i use the prototype to predict the trajectory of this trailer and so i could test it early enough so somebody could actually fix things and get ahead of it what's convenient about this clip is that because they edited things out of the prototype um the prototype and the release are slightly out of sync so you'll be able to look at one and then look at the other so that didn't work where is my mouse oh hey welcome to spectacular sunset city okay here's the situation a contaminated energy drink

so if you're watching this clip one of the things you should notice is there's a really close relationship between the prototype and the release and what was fascinating about this is that there would be a close relationship even when the monsters weren't yet in the early prototype we because of the way that they were moving the camera and the lighting and the positioning and the sort of scene they could get an entire cluster of people to all focus in the right spot before they put the enemy in right and what this tells me when i see this kind of pattern is that if i see problems in the prototype they're probably going to show up in the release version right and this is just a test that i do for me and i have done this on every single method i've ever run okay so if you have questions about whether a method is actually predictive any ur should be able to give you a reason for why they think it is the other piece of this is that when you talk about things being actionable it's not just a mix of the right question to ask it's actually also when to ask so one of the things we did in halo 2 is we actually had a little

survey that popped up during the game and we knew what we wanted to ask these this was the question um that we asked people but we didn't know how frequently to ask it because this actually is a problem for design right this this depends on the level design right if we ask it every hour maybe that's useful for some person but in our case it turns out that the the cadence that we need to ask this was um every three minutes and this was because that was the cadence that the game moved on um and again this is an example of if you for some reason are not in contact with your you are um you probably need to be because this is the kind of information we really need from designers in order to design the right kind of study to make sure you can actually use the data the next thing that we need from data is that we need it to be consumable so i'm going to give you an example out of science this was the first visualization that was made using scientific data it was a gentleman named von humboldt he goes to south america he does a lot of measurements on mountains and what's neat about this is he drew them out and he saw that there were all these layers in the mountains and they went back to germany and he handed this out and people looked at it and they looked

at the mountains outside their window and thought hey i think there's a pattern here and this is actually what visualization should provide to you they should provide that kind of high level pattern so here's an example from game design this is how we visualize all of our difficulty questions so you have four different missions we asked each each on after each of these missions how difficult was the mission and you get four different responses it should be i hope relatively clear if you study these slides a couple of for a couple of seconds um that the one that you want is the one with the tallest green bar right because this is the one that has the right balance for the most people and conversely if you compare the bottom two visualizations if you get that feedback on two different missions you should respond in very very different ways because one of them is broken because it's too easy and one of them is broken because it's too hard right and the way that you handle it is just fundamentally different finally the last thing that data should be when you you're presented with a report is

diagnostic so i talked a lot about falsifiability to begin with there's actually another way that science sort of generates hypothesis and this is known as inductive reasoning and before popper this is sort of what what a lot of scientists did essentially they went out in the world and they observed things and they looked for patterns and they generated hypotheses um and so when newton went and gave his sort of theory of light one of his methods that he had to actually demonstrate so that other people could replicate it was how to build a prism so you could actually go and break apart light correctly so other people could see it right here's an example from game this is a terrible graph i made it the reason it's terrible is that it was actually based on telemetry so it was a bunch of currency that we were earning in a game um and what i found from it was that there were these huge spikes like people were starting off with um a certain amount of currency and they were getting these four times increases from certain encounters and when i presented this to the game designers their response was that's impossible that your telemetry is bugged i don't believe you that is not okay um and it turns out the

telemetry wasn't bogged this actually was an error but we just couldn't tell why this error was happening from this graph instead we needed to actually go through and visualize the behavior in a different way so this is rise and the way that you earned um currency was typically through building up your combo and as your combo got higher and higher you got an exponentially larger reward and when we went and we looked at sort of the low brakes in terms of combos and the high brakes there was something really distinct about the high brakes which is they all tended to occur to occur during these instances where you weren't strictly speaking in combat you were instead in these kind of moments where you would be put in a ballista and you could kill a bajillion things all at once which meant it was very easy for people to get combos of 300 rather than 12 right and this actually was what was throwing things off and why the designers needed to sort of fix that the other fallback that we have here is actually to just look at behavior so

this is one of the most common ways that we will use telemetry which is that we'll build a map we'll mark out things that we think are interesting and then we'll just link it to videos so if you are the guy who's in charge of cars and you just want to watch 500 videos of cars great i will make it easy for you to do that so you can figure out what's working and what's not so i think a fair question to ask at this point is wasn't this whole talk about methods and to a degree it was but a very very sort of high level and the reason i'm keeping it at this high level is because if you want to know what a good method is this is really the test you need to make sure that it follows this complete cycle right you should be able to generate a design theory you should do some falsification testing you should reject whatever doesn't actually work you should diagnose what's wrong and then you should be able to generate new design theories so you can go off and make new better versions of it so let me give you an example of how to

evaluate two methods say you have the following question where are players getting lost right you don't want anybody to get lost you want them to walk forward shoot the right things and never get lost in your level fine um there are two possible methods here one telemetry everybody loves telemetry it's the greatest the newest the wonderkin at the moment um i'm gonna compare that against a really old school method called asking people did you get lost yes where so this i did this um on the same level so these are examples from quantum that by the way is a great example of what getting lost looks like for telemetry what the hell is that blue dot these are all heartbeats of players what the hell is that blue dot out there it's somebody who has escaped the level they've jumped a fence run out of it and they just keep going until at some point they fall off a cliff and die figuring out where they got lost from this data however is very difficult and in comparison i'm going to give you the other side of the slide which is

that actually when we asked people 95 of people said that they got lost which i don't think is clear from the telemetry and we asked people where they got lost they gave us these beautiful explanations right and you looked at it you thought oh i know what i did wrong i got to fix this one right it was much much clearer to figure out where you needed to intervene but the other important thing about this circle is that i own the boxes in gray right as a user researcher i do falsification testing and i help diagnose problems but design owns the horizontal okay and if design doesn't keep up their half of it this whole system fails and to show you how that works i'm going to go back to richardson remember i think he is a very very very good designer i do need to feel the need to say that at the moment but as it turns out the epistolary style is not the reason his his books were very very popular instead what he had done is written a couple of books um where basically there's a young sort of young woman and then there's this

darkly romantic hero who does all sorts of really fundamentally inappropriate things he kidnaps her he like sorts through letters he does all sorts of weird things and this is considered very sexy and romantic right it wasn't that he had figured out the great next great sort of technological breakthrough it's that he'd essentially written the twilight of his day and age um and it turns out he figured this out at some point he was like this is not working the way i thought it was going on and what was interesting is he had the option was he going to reject his his design hypothesis which in his case was that people think about fiction just exactly the same way they think about non-fiction right it to do that he would have to say hey maybe people hold fictional characters to a different standard um or he wouldn't reject his hypothesis and he would have to believe that people think that the behaviors described in books like this are appropriate and okay to do in the real world and that's the interpretation he went with and he did try to solve this his solution was to to issue another copy of these books

with footnotes in explaining the moral degeneracy of his sort of romantic hero i have no idea if this was i don't think this is effective i do want to compare and contrast this with another designer a designer who did in fact reject in my opinion richardson's sort of core hypothesis because when she writes it is a truth universally acknowledged that a single man in possession of a good fortune must be in want of a wife this is the sort of statement that can only be made as fiction this is not true right and what's interesting about this is that you can debate the quality of a designer and you can debate the quality of their art which of course is a separate thing but i'm willing to bet there are more people in this room who know the name of this author and can tell me where this quote is from then can tell me the names of both books that richardson wrote and i told you that at the beginning of the hour it's jane austen pride and prejudice and it's interesting when you think about these two designers it's very easy at our

300 years distance right and our vast knowledge of how the novel actually works to think i am definitely austin right i'm i got this but here's the problem if you were to time lock the beginning of the novel and the beginning of video games it turns out that we're a lot closer to richardson than we are to austin okay and maybe you actually are like just vastly smarter than this designer who was the inventor of the modern novel but when i look at this i think well crap i have got to find a way to get a lot smarter than i am a lot faster and for me that solution is to embrace the following idea that daring ideas are like chess men moved forward they may be beaten but they may start a winning game thank you very much

i was curious in research uh usually you can find results that either confirm or refute a hypothesis from the game design team but i'm curious what happens when you get maybe a highly controversial hypothesis that in fact you find a null result no relationship or that something doesn't affect what you thought it did so uh typically it's actually harder to find null results in game design testing than it is in science because typically the hypothesis is i made this better did i make this better it doesn't necessarily mean that you haven't in fact added an improvement if you don't see that kind of jump because it turns out that you often times there's an idea in psychology known as the just noticeable difference where you actually need to make a sort of a large jump before people are willing to pay you compliments on it and so

unless you're actually adding a feature that's generating problems typically we're like yeah but that's probably a good idea or at least it's not bad let's keep moving right because um we're on a production schedule people we got to keep working on this so that tends to be how we handle the kind of null hypothesis yeah hi i'm bim fu poki um in one of your slides um i think it's a slide about the benchmarks showing uh fun as a metric i was wondering how do you measure fun as a metric we asked people how fun was this game uh in that particular case uh we have used uh microsoft uses a benchmarking process when we're actually establishing these kinds of things where we have asked these the same set of questions in the exact same way across a multitude of games to sort of control out any of the variants in terms of just how people interpret this obviously fun is not the only thing that you can use as your high-level metric it just happens to be a very common one because it is generally speaking a good high-level kpi

right all right thank you hi so uh when on the pm team you're trying to test quickly and not really engaging with you are how can you build this into like a b testing that you would show in the game um like for example if you're building a new fetui and then you put it in the game and hash it to a specific percent of people who can see it uh if the future doesn't work or it doesn't like test well then pm might look at that and say like oh okay we shouldn't have a fatoui but then another question could be just like was that the right for tui to show players is it that they don't want a fatoui or should you just continue to iterate on it so i think in that case my answer is going to be you probably shouldn't have production running your tests because um that doesn't sound like a test that actually tested the question that you were interested in which was

hey should we have a future right if you're just sort of throwing things out there and hoping data is going to tell you the right answer you're kind of not doing it right right there there are a few really catastrophic errors that designers can make when consuming data and seating design to players is one of them you are always a designer it is on you to find the solutions right players are going to tell you what they want sometimes but that's not necessarily what they need and it is on you to get that and that's part of why we really strongly recommend pairing with a researcher so you can actually design things to actually get the data back that you need that can be actionable for you okay thanks hi thank you for the talk um so do you have any thoughts about like psychological framing with your questions um the notion that if you ask some if someone asks their lover do you love me versus don't you love me that you might not get the same answer do you have any thoughts about the framing of the questions uh i don't just have thoughts about the framing of the questions i have thoughts about how you see people in the lab i care about that topic immensely um

there's lots and lots and lots of research um showing that basically um you can convince people that something is true or get them to say something that they know is untrue if you put them in the wrong sort of social group so the classic example of this is um there was a group where they basically imagine if everybody in the front row was a participant except for that guy and the row was actually full um and we went through and we asked everybody hey here are three lines tell me which one is longest um and every single person in the front row picked the wrong one by the time we got to him he'd probably pick the wrong one too and this is just a demonstrated effect and one of the things that we do is actually when we test people in large groups we put blinders on them essentially we put them in little cubicles so they can't see people and so there's there's no indication of whether or not you're progressing for instance slowly through things or whether because you should be going at your own rate and all of that kind of stuff so we definitely worry about that an enormous amount it's also why one of the most common things um that if you are for some reason running your own research and you are a

designer the number one recommendation is stop talking to people um and and while you are running things right and mike i jabber constantly during um usabilities but that's because i've been trained in the the right way to sort of ask questions and things like that i think i'm out of question oh you got another one um okay so when you're looking at data that you're getting from an experiment you're running uh sometimes you might or typically you would expect the data to follow a certain story so like you're increasing engagement and retention and then thus your average revenue per daily user goes up uh but oftentimes with experiments we see like maybe average revenue went up but then there might be like 8.3 reduction in your retention or revenue went up but engagement is down uh how can you kind of like filter through your data or then design more tests to get to like what the heart of is really happening through this test and what like their

real reaction is to it so it's fine to use those kind of high level metrics to sort of keep track of sort of your progression on something like i'm moving up or there's a red flag here but um in the same way that like the graph that i had around currency tracking if you're too high level it's just not actionable right you you have to get more context there um and frequently do it just through qualitative questions um when you made a choice why did you make that choice okay great explain it to me and then i'll go through and understand it that way so but i guess if we have data on a quantitative level it can be like representative but then how do you know that the user you would be talking to on a qualitative level would be representative of all your users well because you sample correctly is i guess the answer i'm i'm it's the sort of thing where you have to you sometimes people think that the best way to run studies is to basically run them like the census that's not true you don't need to get every single person in your in your database um to to answer a question to know whether it's representative instead

you correctly sample and find the right people okay hi um you're talking a lot about uh formative testing um do you have a process for uh looking at analytics post launch and what is that process like uh we we pull the data and we look at it um typically we however what we do is so you can't just hook everything in telemetry you have to actually go through this entire process to begin with if you want to have effective telemetry you need to start off with a design question that you're trying to answer you need to figure out how you're going to answer that with data you typically need to prototype this a couple of times to make sure that it actually works you need to get it hooked you need to then check your pipelines to make sure that they're hooked correctly and then things will come back and be informative i'll be honest i am mostly in the business of trying to prevent problems before they are released but we do definitely do ongoing engagement stuff and we certainly use telemetry i'm more likely to use it

in the lab than i am just post release them thank you all right this time for reals thank you very much everybody you