Playing the Middle: Balancing Trust, Creativity, and Business in the Science of User Experience

[Music] hello my name is Ian Livingstone thank you so much for joining us today today we're going to be discussing while UX we're gonna start by just doing a brief introduction of who we are actually the panel right so playing the middle this panel is about looking at how UX cross cuts all the different disciplines within our industry right so it's about balancing the creativity the design the business and the production as well as the the science of experience right so that's what we're gonna be discussing today and we have a number of great panelists which are going to be helping us with this so let's introduce that Who am I as I mentioned I'm Ian Livingstone I work at EA I'm the senior user experience researcher up at EA Canada in Vancouver I manage a team of researchers up there we take care of all of the projects that are that are coming out of out of Canada to my right we have Corey

may actually to start with with John sorry John is from where you know Warner Brothers right John has is a user researcher has a ton of experience and he's worked on a number of different platforms and genres in the many years that he's been here so you want to introduce yourself John yeah hello so yeah I'm currently working out of Warner Brothers Montreal for Warner Brothers interactive entertainment I did a quick single decade ubisoft before moving over to Warner working mostly I spent a long time on Assassin's Creed with some of the other fine gentlemen on this panel but did a couple of other interesting projects while I was there and now working out of Montreal and some other cool stuff and I'm here to represent the games user research side of this equation of the of the Iron Triangle that we're going to talk about yeah maybe it's worth mentioning that games you are I think as well as all the other

problems here they're not solved problems and so I'm not representing all games user researchers I think it's safe to say other people will have different opinions on how they would approach other problems but I'll do my best to make those of you who raised their hands when he asked to do so I'll do my best to make you guys proud she was right we have Cory make Cory May is a narrative director that's to K advocate for for the player experience Cory you want to introduce yourself sure so like Johnny I spent many years at Ubisoft I think 12 years then I moved to a place called certain affinity and then most recently I have started working at 2k up in Novato and Johnny knows I've always been a very big fan of user research of user experience in games I think it's the language that we developers use to communicate with the players and so I believe it you know serves a critical function in game development I think too often it is ignored and so I just want to let

everyone out there know that people on the production and creative side do very much care about what you specialize in and we want to find ways to work together better next we have Alex Hutchinson designer writer producer free lanes yeah also I'm at the moment of I guess as a Friday a freelance creative director which is one step up from homeless person before that I was crave directing for many years at ubisoft on farcry for Assassin's Creed 3 before that was lead designer of Maxis on spore The Sims 2 other games like that and yes now I have a PR rep from UB staring me right in the face from four feet away so I will speak very guardedly next we have Chris land Chris joins us from Bungie so yeah I'm the producer at Bungie I did a short stint at obscene

entertainment for a while then I transitioned to platforming services space at Microsoft on Xbox Live focusing on you uh you are in data telemetry and then I transition back to Bungie where now I'm focusing mostly on UI so I produce a few of the UI teams at Bungie great okay so just a quick overview of how we're gonna go through this this panel there will be time for questions at the end but there's a there's three topics that we want to we want to jump into and dig on first time we're gonna talk about user experience in games understanding the player and then finally feedback in the creative process so you can jump straight into that first topic so this is obviously gonna be kind of a general topic we're going to talk about what user experience is obviously there's a lot of people who know what that is here but when you're looking at it from a different perspective it can actually be quite interesting to see

what user experience means in people's day-to-day so our first question that I'm gonna pose to the panelists here and I think Alex you would be a good person to kick this off what does user experience mean to you yeah so I apologize invest is probably the most broad answer that will give off the hey what is the thing we're talking about but for me as a creative director it's really the culmination of all of the different disciplines that we have in the game so my job essentially is to try and get people who were much smarter than I am in different disciplines we have great levels of expertise in different disciplines to stay focused on the same idea and expressing the same idea throughout the game so that when the player touches it when they experience the game that it's consistent and it actually follows through on that idea so it's everything from you know if we do it perfectly which we never do but it's everything from you know you know the first boot up sound that you have in the game to the first button we ask you to press to the consistency of the controls - you know what actual you know interactions we give you in the game you

know something that always impressed me Bunji for example on the original halo was that sound that sort of you know operatic I'm not gonna embarrass myself by doing it but that sort of rising you know crescendo voices that starts at the start of the Halo games it meant you knew it was halo it gave you a sense of epicness a sense of all before you even started the game and try to get that feeling and whatever it is in your particular game and drive it through the whole experience so how important is feeling for user experience I think it's really important I like I like it much more than I mean we've got data people here so we should all be afraid but I like the soft data as much as the sort of really hard numbers the idea that you're driving for a human emotion and you want to get that back and it's very hard to prove that you have it with a graph but the you know if you can get it you can see it in people's faces and you can you can feel it in their responses and that that for me is the most important bit in terms of picking features and picking elements both are equally important right the the joke that I said you know some new people and

they were coming on the team that today was exactly just because people walk out of a movie in they're like oh yes I understood that plot doesn't mean that they enjoy the movie right so having both the heart and the soft the qualitative and the quantitative both contribute to the sense of validating your vision for sure I have don't pay me as the hard numbers I was just trying to get some you know your second is like justified okay well Johnny I have a question for you actually hilarious review faultless a lot of times what would you use your testing or play testing it comes down to numbers right they ask people on a scale of 1 to 5 what did you think and if I had my way I would take the numbers out and you can maybe keep them on your end but I think any client-facing paper like would you be down with trying to get rid of the numbers I feel like we have the same problem a lot of times with review scores right people just rush to the number and they don't necessarily read the review I think that can sometimes happen internally when you compile this very well considered well researched well put

together thoughtful documents and everyone races down to the number and what the [ __ ] do you do with the number it's just one point four what's a four point four I can no one can do anything with a four point four so I agree there's sort of an important difference treat formative testing and summative testing right the idea of like giving a score versus hey what's you know in the general sense how does this feel and where is it going and how do we move forward yeah but I think Chris would certainly argue that having a number moving you know it's important as you push a product through to be able to see that there's some sort of evolution and certainly for us an ROI on the service that we provide that says well this is moving up absolutely in a vacuum if you work very loosely with a research service and the only test once every six months and they show up and they go you've got a three point nine it should be a four point one there's minimal value there but having some sort of summative result is is useful particularly let's say for example if you're comparing across items then now you can say oh well maybe we need to invest more in this one or maybe we need to adjust that's it I understand your gut of like but if everybody just races

to that one fun number and then they throw the rest the report out that's wrong but also I thought that's you doing it what is what is think you're you're in-game economy has a three point eight like what that doesn't tell me anything right I want to look at people's actual responses so it's a point you in the right direction to where to look and say okay well this is the lowest score of all of my system yeah and then you go when you continue to dig deeper and you can look at metrics and say well they're not engaging out the system or you can look at the qualitative and say oh they hate the system and so at least it can give you some direction but you're right it shouldn't be a final answer nobody should walk around when my game is a three point nine I know but they do well the worst thing is they don't okay so don't do that good discussion we have other questions okay so Cory kind of building on that when you're in your opinion who do you think is ultimately responsible for providing a great user experience so originally my answer to this question was it was the creative directors responsibility well like I said I

I'm pulling back a little bit I think that you know since games are intensely collaborative or they should be it's really everyone's responsibility to share responsibilities look at the end of the day you have to accept some responsibilities accredit director you get like all the all the accolades you gotta like look at all that but I think that everyone needs to be working together to to ensure that the user experience is a is a smooth one but yeah you have to balance that against the enemy there needs to be some responsibility because if it's like a whole group of people there's only like 12 or in games that we've worked on like 40 [ __ ] leaves that if the buck has to stop some more right there has to be a person calling the shots and like making the decisions and assuming responsibility otherwise you run the risk of getting stuck with signs like designed by committee and I don't think anything good has ever come of that see it's it's a balance right there's no easy answers I think to any of this it was and like johnny was saying earlier I don't know that my answer is the right answer or the answer that everyone

should should take but I do think it's a shared responsibility by all the leads eventually ending with the creative director or if you work at other places maybe it's the producer actually because more and more these days and that's a topic for another day I suppose totally different yeah okay said somebody alright so actually kind of following up on on that one Cory Chris when you're when you're sitting down and you're you're looking at what's coming in how do you how do you go about deciding where you need to focus the efforts of your team and where you need to kind of start sacrificing that vision so I think like over the course of development there's always gonna be multiple things that are competing for attention and I mean really it's gonna come down to what are the problems that you're trying to solve and what are you trying to achieve with the product and I mean that can be defined that's usually defined by creative leadership in some capacity but the hard part is conveying that to the team and so part of my job is to work

with creative leadership or creative directors and understand what is you know what is the problem that we're trying to solve and what is the responsibility of this team in that space so when we so for instance like Bungie where we have a laundry list of features for the UI team and I spent you know a couple hours or multiple hours I should say talking with our creative director is like just getting them in a room and saying hey why is this thing important what is the why here right and that's gonna frame the sandbox that the team works in and so if you're having making a trade-off between two features or between two different systems you're gonna say which of these is more important for that ultimate goal and it's not always as black and white as that but really it's gonna come down to that there's gonna be some compromise but ultimately there's gonna be one that's gonna be more important to the experience so when you're looking at trying to figure out what's more important do you have things that you're using to try and identify absolutely so we're using we're looking at community research user research internal feedback

all of it also from a production standpoint risk associated with the different features like is this thing gonna ship on time right so all of those things play a part of the conversation and ultimately it's the responsibility of the creative leadership to make the decision given the data that we can present yeah and I think it's really interesting in the in the current climate where there are a lot of projects that have very long life cycles that are alive for many years after initial release that there's a different I think way to calibrate the features that you're building based on is it a first product you know are you making something new or the initial game or are you making keeping something alive you know like I mean there's been a lot of you know information say valve for example or Blizzard when you're maintaining a game over many years you deal I think with with feedback from users and the situation much differently you know sometimes on on a new game the the argument is I think we've had this conversation many times this we think this would be [ __ ] amazing you know that's that's the logic everything at the end of the day you don't have data yet because it doesn't exist and we'll use people like Danko you know Ian to go through it and try and get some sense of it before we put it in the box but a lot

of it you know having 20 people in a room every you know every every couple of weeks even is never gonna give us the same data and sometimes you have to back yourself to be like no I think this this will be amazing so again a good example from from Bungie is the controls for the further vehicles you know what I mean this was testing terribly you know three four years I remember like Hamilton back in the day the producer and Haley wanted to saying they hate it they hate it but it's right you know then everything in the game controls the same way it's consistent controls it's consistent patrols and it worked in the end but if you just go on based on user feedback then you would have you would have killed the feature I'm realizing we have a bit of a blind spot on mobile we're all console guys but mobile also again I'm sure anybody in the room that's working on when I was a gap course yeah like we you know we do most of our research in in soft-launched and that's where we get like you know and on mobile games that's where they get the bulk of the info I mean certainly they do usability ahead of time but there's a richness of data available to them where they're able to pin it very quickly once they're in soft launch that you know both products or software service can't

really can't track as quickly yeah cool all right moving on let's move on to our next topic understanding the player okay so as we know players are both are our biggest customers obviously but also our biggest critics and since I hope we can all agree that games are both art and entertainment the understanding that we develop of those players is absolutely crucial to the overall success of the games that we're making so that's because card is gonna answer this one first what right so let me ask the question yes so this is just an abbreviated version so here's the full question what is contributed most to your personal understanding of players that you feel others most often overlook misunderstand or misuse so I think I'm answering this question right so the

thing that's that's been big for me and yeah now Johnny will get mad is that while I appreciate the process of user testing and user research I feel like we are kept as developers away from the players and so we never get to communicate with them we get a documents that tells us what they thought it doesn't often do a very good job at getting to the why you'll get like a single sentence or but the number which and I think is not helpful and so if I had my way I would love to be able to communicate with the players after after the fact so I won't be poisoning your data but I'd like to sometimes get a better understanding of why why they're making the the selections that they're making or why they're responding to a feature or or an aspect of the game that the way they're doing and they absolutely won't let me do it and I don't understand why I mean maybe it's obvious these are researchers in the audience right now are cringing but why it's not eating me I mean I think what are you so if you don't if your instinct

is right your instinct is I want more context for the answer yeah I want to understand my audience the point is there is a certain amount of rigor that is involved in the process that needs to be maintained and then if you roll in and you're like hey I'm Corey man the guy that does all the cool stuff that you like and they go never mind you know actually I loved everything with that nature what though right they've already [ __ ] all over the thing in the report so our content yeah and you talked to me ahead of time like hey this is something that I really need to understand better or if you get preliminary data and you go alright dig deeper on this then we you know we can work together on doing it in a way that is properly structured and then because if you do it once with one night yeah that's a single data point then you fall into the trap of a question we're gonna get into later of like doing it wrong no and I know and you know how I feel about annek data then we can start to get okay well if this is the question that you want rather than one person we can formalize the process of gathering the data that you're interested in and get you more substantive and more useful and reliable information on the topic because don't you think I'd rather than you ruling at me like hey I'm the guy who made the think but I think that that

initial conversation could also benefit you as you like reshaped the mic why can't we invite I don't your answer is like not doing it for me this sort answer but they're gonna ruin everything researchers to do that I'm just trying to add an extra layer of data and your job is done at that point right I'm not poisoning you're still needing your context I would argue that my job is definitely not done if you're walking away from research results and going I still don't understand what my problem is yeah then I have failed it as a researcher and we need to have more interesting conversations about what you're looking for so that I can deliver results that are more useful to you but even more I could just talk to the people even if you're even if you're not engaging with the community and in a formal way or it's like hey like I am the creative director I'm the lead for this team right like engaging with the community in an informal way just playing your game right in case of like destiny right like playing destiny with everybody that's a multiplayer game like that provides a lot of insight that are in context that you don't get just by looking at

research results and obviously like the the idea of getting good yard is you have a lot of context in that space but I think there is something to be said for the organic nature of those of those interactions yeah we could all wear a fedora and they wouldn't know who we were but the know for me that one is love you from going on here yeah nobody to know you're gonna go so but also like you know another thing is that the thing that kills me in this context is even with that even we have a feature in test and you know and then we get data back and we can all can't speak to them think that kills me is most the time you're testing an unfinished thing you know so like you're taking something in that you know is broken and then they're saying this sucks and you're like bah yes you know yes it goes to find why it sucks well if we showed the Marvel movie sevens watch those terrible YouTube videos of the Marvel movies without the special effects of like some you know mediocre buff dude jumping off the back of a utility vehicle you know and then like without the effects and the sanno led to the stuff thinking oh this is the worst bit of cosplay that I've ever seen and and and you know like testing at that point is tear is me is terrible but we we often get stuck in that point of you have no choice I think I think

there's a happy medium there right - it's a both Carrie and Ron's point that there's this is this is why I think a lot of user researchers will argue that having a creative in the room well not in the room but there when when a session is happening so that they can riff back and forward with the researcher and you know point out the things that they're interested in is something that that is really important and I think a lot of times this story even the stuff not be interested in yeah yeah player gets stuck in there like no don't okay we'll just fix it yeah I think I think there's a lot more to be said about the overall collaboration on how it can be made more useful and still maintain that rigor yeah and I think one way to improve collaboration is to let me talk to the participants I feel very strongly about this okay okay moving on okay all right so this one was this one fourth this one's too Alex Alex should a game be built for a specific player base or should a player base be built for a specific game

I think you can discover new audiences and that's one of the joys of a game you know Spore we built swore we were in our hearts making a game for like adults doing adult things in adult appropriate environments and it hit really hard with kids you know and I guess that predate you know the preface like a lot of stuff that happened with would be with a younger audience later but like this sort of the 12 to 14 said so you can discover you know people as you go and that's amazing and I think that's one of the most rewarding parts of building a game you know finding any audience is super rewarding sometimes you do it but I think it's it's important having your head a clear target of who you want to enjoy it and I don't mean honor like you know males 18 to 35 which is horrendous but you know anything that's undirected or unfocused I think doesn't end up particularly going in New York or trying to you know finish an expression idea properly and we often get you know bad feedback from and we don't have any here so we can insult them from the marketing department say where you know you say who's our target audience you know and I'll say everybody you know and this is not useful information you know

it's a so I think yeah it's important to sort of know who you're early you know along with what idea you're trying to express who you think would enjoy that and then you can use you know a testing process to figure out if it if it works or maybe you found a new audience Chris yeah so I'm gonna take the cowardly middle-of-the-road approach so you know I think I think if you look at it from a business perspective like you're obviously going to want to attack a pre-existing player base because it's gonna provide minimal risk right you're gonna start making a first purchase your target first person shooter players right but I think the novelty of you know creative design is what allows us to define those new or create those new player bases that then lead to the longevity or the long lasting us of our game right I think if you went solely with the we're just going to cater to this one or this view play are these people it's really easy for a competitor to step in and say okay well we're gonna do the exact same thing just slightly

differently but it's the novel approaches that help you know turn your your new game or your game or convert move move that pre-existing player base into a new player base that then you don't have many competitors in that space and you can then ride that train for much longer you know I think I look at you know like dota from Warcraft 3 right like that spawned a whole new genre a whole new player base and now now there's more competitors in that space but for a long time like yea though it's the only competitor in the space so six of them successful business model and so I want to take like the idealistic / naive / fundamentalist approach which is never mind all of that stuff and like if you if you believe strongly in what you're creating it will find an audience and and your mention of dota is actually really interesting to me right because I don't think when when they were coming up with that there was n or any like consideration of you know play it was they were like this is a cool idea you're looking around in it and it it organically Greta that now you can argue that riot being the geniuses

that they are like took a look at that and then like shaved off the rough edges and so there is an argument and I [ __ ] love League of Legends but I feel like the genesis of that right well it came from the heart it didn't come from from a lab cynical consumer that's just saying that Dornan's is me you know see now right they still did it they just said they know what their brand dudes meet you know they did the lab strangely authenticating dude yeah but I mean it was specific to them right it was like what gets me excited this is getting me excited maybe it will it will excite other people but also audiences evolve right so I don't everyone these these days forgets but the Sims was bought by guys hardcore gamers first right we were not the first audience was hardcore gamers because they were following on SimCity they were super into it and then they always kind of interesting I kind of get it but it's not really something for me they passed it across and anybody who's went sort of what it was at the time with the Heike female audience at that time and then it migrated across from them over time I ended up going down to a very young majority female audience and that's where it sort of ended for a long time but that audience went through multiple

phases over its over its lifecycle it's also interesting talking about you know the role of defining your audience ahead of time or the creative I would also argue that that evolves throughout production right there are certain points where you want to have a clear definition because it'll help you in a different way rather than like later on where you know the use of personas is something I find interesting for example where it can be really useful very early in a project but I found most of the time when I've used it later on like when it's like to try to find test markets that it's like okay what you're just giving me a demo information like it's not it's not really a persona that you're using anymore and so certainly the the use of you defining your UX audience varies from preconception through to like gold or even beta cool all right next question this one's to you Johnny I like this questions like these are my favorite questions when has your knowledge failed

you causing you to miss something big or make any poor decision and how would you avoid that now right okay that's it I think I bring this one back to the idea of continuously maximizing context and this is a mistake I made on one of the games that I was working on back in the day and there was a system it was like a den defense type system and what game could that be Johnny Connors but the problem was I didn't I didn't fully understand the context of how it would fit within the game flow and I tested it on its own and that this may come as a shock to some people but it tested really well people enjoyed it and so you know it got the gur stamp of approval like this is a fun thing cool moving on and then it was only later when I realized exactly how it fit into the overall macro schematic of the systems where the way that it was sort of presented in the game broke that down a little bit in players it was the context of it broke it and so because you know if I had been testing it properly I

would have had them playing the regular game and even though that system wasn't integrated I would have done like a hard stop on what they were doing like do this now and force them that way because that was the way eventually it would be in the final product and so just by lacking even my own context and delivering that to the player and testing it appropriately I ended up telling everybody something was fine and I got a very angry phone call when the reviews for that game came out like hey why didn't you why didn't you spot this and so again yeah going back to making sure that you understand like the the larger vision and probably that was a you know a conversation that if I had had earlier with the creative director on that one where he explained to me what his end goal vision rather than just talking to the designer of the specific system who was like this is how it works and I'm not cool make sure that they understand how it works and I never took that step back and so that was one of that was one of my big like wit's coverage so let me

think I think that once upon a time I would get defensive with with sometimes with some of the results all right I would be like it's wrong or they don't know what they're talking about or and I think this can be this can be both a blessing and a curse like it's just not in yet they'll understand later when like everything's like online and the game is is more feature complete and I think that that was a mistake and so I learned my lesson which is to pay more attention to the data if you see a trend of people not liking something or not understanding something or not appreciating something it's very dangerous to put it off like it'll be better in the next test they just don't understand yet and to be a little bit more proactive to like get in front of red flags and and warning signs the other thing and I think Johnny would be ok with this that I've realized is that sometimes you want to be a little bit more selective in terms of who the audience is that you're putting together for testing so one of the things that

Jenna and Johnny and I had started developing at Ubisoft before we departed was something called a narrative review process because we found that user testing with kids off the street was not providing us with very useful feedback when it came to story testing we would basically get like a thumbs-up thumbs-down I like it I don't like it and and that was it and sometimes if you want to dig a little bit deeper it wasn't going to happen with these kids so we put together something with the other writers and narrative people inside the studio what we were doing user research but we were using people who specialized in the craft so that we could get more detailed feedback and so we had a very different looking survey that liked Tove a lot deeper Johnny actually let me talk to the participants in that particular instance but again that was very targeted in special circumstances so and we found that it was bid you forever from speaking to your colleagues yes so he he at least you know allowed me to do that

and it was really interesting and I I think we could have done even cooler stuff with it had we not departed but so that to me something that I think is at least worth exploring I don't know if it messes things up is to choose your audience for the test and I don't necessarily mean choose a sympathetic audience choose an audience that can speak to the stuff that you're testing you identified your target audience well not I I found people that could speak to the topic not necessarily people that would like the material right so it wasn't necessarily a sympathetic audience it was at least one with a deeper understanding of the subject matter for example to when you're working on a sequel you're obviously going to run some studies or people that are new to it yes and people that are returning from the previous yes and you're gonna so we can do the same right which is like people that that have experience with narrative design versus people that are again kids off the street cuz they're often kids off the

street aren't they not not literally like never mind all right next question yeah okay moving on just leave that one yeah okay so I'll click on through our next topic feedback in the creative process and I think this is I think I mean I think this is probably the most interesting one I think a lot of the conversations that we're having right now are kind of ending up on this on this topic like how does how does feedback work and I I would feel that I'm pretty safe to everybody would agree if I said that everybody on a production team has an opinion and there is a huge amount of feedback that that comes in for for a game while it's being worked on and this feedback can have an effect on the team's morale and it can also be really noisy it's it's hard to cut through and and get to what's actually important and useful so my first question and I think

this one's this one's going to Alex what is the impact what is the impact of and what is the best way of managing negative research or data or feedback results on that development I think it's it's central to the job of the whole leadership crew on the team it's probably the hardest thing that you can do because as Cory said you need to be open during this really vulnerable state right you need to be open to negative feedback and you really need to listen you can't let it destroy you because then you go home and you don't make a game you don't mean so but you have to be in this like horrible friction state of like listening and then going for a drink and yelling Cory that it's okay we're gonna get there we're gonna be we're gonna be okay and you know we would have throw down fights and and you know not even not in an aggressive way in a sort of upset way you don't even honor because you're really trying to wrestle with the idea of is it junk you know thing like have we made a really bad decision and we're making something poor or is it unfinished or is that is the audience like not you know understanding it correctly and so everything we're sort of talking about

these ways of wrestling with the problem of we're honest to god I know I mean this room is probably very sympathetic to this idea but honestly God you know it's always fascinates me that the audience doesn't think that a game team is a bunch of people trying with everything that they have to make something cool and that the only people you know more upset than the person yelling at you on Twitter if it isn't good is the people that spent three years of their lives making it you know but so you Susan puzzle when you see people I didn't see you in 2 a.m. last night but uh you know and it is it's horribly wounding and I'm you know I we've had we've had you know I mean I've seen a lot of people have really difficult times you know based afterafter not just the release of games but after reviews and features and that sort of stuff but you need to you need to still stay open to the fever because otherwise you fail in the other direction and you guarantee that it won't work at a more much more granular concrete level I think one of the ways that some of us had solved this was really integrating user research into the processes of the dev team making sure that you are completely integrated

it's not a pole for the Tina to use your results well that's actually for example we have we used here at UB and we would you know all the results would go straight in they would go to an approver who would look at the research feedback leave a comment put it into the next guy who was gonna have to write about it you know I have to fix it and then that would get sent to whoever and then would go back to the research team and there was it was a much more integrated process rather than the way when I first started doing research ten years ago which is here I wrote a thing that says your thing sucks bye I'll see you in a month but I think actually eating outside the second half of the point I wanted to make bit sidetracked by remembering people being upset is that you know that that's the job of this interaction with the user research group is to try and get some objectivity on it because it isn't just good if it's all emotion and all feelings and all response and your your personal belief about something versus another team members personal belief you know you want to get some distance from it so hopefully you can be like Cory said I feel very strongly about this and then you have to decide whether you keep pushing on it or maybe you bear out to the press or you know whatever feedback you're getting and and

modify it but that's that's the the job on the creative side I think of getting all this data is to try and bleed some of the emotion out of it so you can you can be objective and know where to make your stains and way it's a way to like you know make it easier I also have like a bit of practical advice for the people that have to put these things together so hopefully it will help is that if there's bad news like try and get it in front of people as quickly as possible if you hide that stuff and then it leaks out it [ __ ] poisons the team I've seen it less with with user research because they've been pretty open but mock reviews will happen at a lot of places these days and like no one's talked about it but I guarantee you they are going to find out and if they if it's being held from people it will start to feel like the the publisher the developer or whatever management has given up on the team so you may you may intend to be shielding them from something but you hide it and it leaks out it is interpreted as like you've abandoned us

and everyone gets very upset morale takes a terrible hit and people become non-functional the researchers realized a long time it was its best practice like that report goes to every yes as you know because it's the idea that you think you can pretend that the people working on your game don't know what the health and status of a game it's I mean it's dishonest and it's yeah don't they don't hide stuff it's it's it's a very very big mistake we can't we can't address issues if we don't know they exist and if they're just coming up in whispers and we don't have a way to validate it it's very bad so I don't know it's some day some of you may have the ability to decide like who gets to see what and I think Johnny's right err on the side of oversharing it's it's necessary well we ought about using research results as a platform for communication yeah like even bad bad results as much as good results one of the things that in it like I'll say the opposite of everything I just said but I'll give like a six hour head start to a creative director or producer and be like listen man because that's coming

out there that's it I want started thinking about it and I want you within an hour of me sending these results that you can send a bullet point like of an action item list like we got it like we know this didn't work we have an action plan we're going to go into this that's going to be part of our next sprint oh where this was great we expected that we work really hard thanks for this team and using research results as a platform for really open communication to drive the team can be super super helpful rather than the opposite where you office kata you try to pretend like it's not happening you know it quickly turns toxic and then you get a bad relation with the research team then you have less data and then everything you know it's very quickly for that thread of the the sort of internal consultancy relationship to the Sun disintegrate if you don't handle it nicely okay I wanna bring kristenjohn I was gonna say like do you mean you're saying you were advocating for transparency with research day like do you ever worry that like because that bad results do not always mean bad things right like do you ever worry that like transparently saying or transparently saying yeah okay like this there's this piece of data and

it appears bad and everybody's gonna interpret it bad that yet that's the wrong thing to do because really it's not bad if you would have more ever say new contextualized results by saying look ignore this one thing because it was like the build was buggy you know what I mean versus oh don't worry about it right ok this hasn't exist yeah that's the probe there like it so it's gonna be ok they just like waves everywhere lunch that's bad but if they say look this is a bad data point here's a very clear reasons why you'll see on the next test that it'll go back up like I think that's ok and again ideally you're working with people that are mature and responsible enough not to go into full-on meltdown mode if they see a negative piece of feedback which I know is is sometimes asking too much and I have been known to spiral out later all right no who'd of thought I mean obviously when that happens it's because I'm right Cory was that Cory was a fan of standing behind my desk and yelling at me while I thought that was that was none of my favorite fruit it was only

for about 3 years yeah if you were just get a drink in me usually if it would be ok afterwards that was just my way of saying I'm you did assassins don't ride on boats that's a pirate I still think it should have been a pirate game okay okay we're not gonna read this it was entertainment I'm sorry okay next question this honestly this one could go to literally any of you and I'm kind of scared who to ask now so let's go with Johnny what's what's your worst that's not how you use data moment the word and I get this short okay I mean I didn't see these published a couple of things on how not to do it of like you know cognitive biases that will impact the way that you interpret data and you know there's there's stuff like focusing on one bad data point which I think we've we've all seen several times

where somebody you know or they confirmation bias there's a lot of these things that can come in when you're looking at how like oh god no don't use that don't please don't do that or like you know you guys have mentioned like oh if the data doesn't look right throw it out there's another one yes you know if you don't agree with it that's you shouldn't throw it out you should look at it harder yeah sometimes the data will will say something that a person doesn't want to hear and rather than then accept it they're like oh you you did it wrong can I try it's a very vague respondent I know so what my favorite meeting of this and it involves dang cough was like no it's good it's a happy story has a happy ending okay is you know like one of the beauties of the franchise you know is that you have lots of data points right so for once using on a game it's the first edition of the game even if you get data or different what people like and don't like it's very hard to draw

you know long-term conclusions your sample size is small the you know what's up on gay on big franchises and UB hides a few you know we have many years of you know releases often with new evolving mechanics and different systems but one of the biggest challenges for me was when you know I saw a den cough present a slide where he'd gone through all the different mission types in a particular franchise and had gone over let's get ready good no it's not that good it's a you know over like five or six years I had seen you know the five or six different mission types that we often use and had pretty much you know a graph categorical evidence in one you know with enough sample size enough data to say this particular mission type in this job in this game in this franchise is despised by all players all of the time yes right and it was like a grandmother's curse away everyone thinks like you know we'll use the horrible numbers but one to fly like 4.1 made one right like across seven years yeah and it wasn't like a bullet no they're like Sarah Mary Kay we time every time and we're in the room and this is something that the teams because we were working at teams and data points but there's

teams and data points and marketing and publishing and all kinds of other fun people involved in this discussion and the management looked at the graph and there was silence in the room because everyone else was just like thank God we have the evidence you know it's like Watergate we have the tapes you know like we can tape it up and management leant forward tap the microphone and said this graph is outrageous it's amazing the graph can't be outrageous you don't even like the the interpretation of the data must be the sample is always all of the people all of the time but you know the mean like so even then even sometimes when you get great data like your ability to institute change even from within a development team is really hard when they're fixed special interests you know in the in the company I didn't using name okay moving on moving on we're not all right so I'm going to direct this one at Chris in the late stages of making a game how do you balance high priority feedback with the

need to ship on time I mean I think it's pretty much a just a straight you know trade-off process between risk reward and cost you're always going to get research at different stages of the game there are different stages of development and your tolerance for change is going to change as you move along some people are willing to gamble a little bit more than others at later stages of the game and that it's both good and bad but it a lot depends on you know who's driving the ship and what are the processes of the team like is the team set up in a way that allows them to respond to change quickly or is that a more costly process what is the state of other features that this thing this new feature this new ideas new UX might be interacting with is that going to create some you know unmitigated risk to those to those other things or is it going to be sort of a isolated isolated feature that's what I remember there was

a Twitter hashtag going around of like scariest than you did in game dev and he was one of the designers on Mass Effect it was like I secretly turned up all of the difficulty 20% like the day before we put it in the can and I know there was a bunch of games user researchers that just responded with like sweaty emoji like I can't believe we did that scariest thing I've ever heard and and then she you know she walked she was like if she walked it back was like well I had a really good plan and I knew what I was doing but yeah it was certainly one of those really Wow that is you know on the riskier end right okay we're getting close to the end so I just want to touch on the last question before we open it up for questions here this one I'm gonna get Sakaki oh how do you handle research that is completely against your design philosophy booze again this is just my answer it's not necessarily what you should do alright so no I think there's a way to be less fundamentalist about it it comes

back to I think you were saying this earlier is figuring out what's the actual problem right so is the issue that that people have a problem with the game itself or in fact asking you to make a different game or is there some other underlying issue that can be addressed to help solve the problem now if someone is turning around and you're you know testing gone home and they're asking for more guns I think at a certain point you need to have like the courage to say no right like this is wrong like we're not doing this so II you look and it's like is this issue something that that compromises the vision that compromises the experience in a fundamental way then then you need to you need to the courage to say I get it you don't like it this game's not for you if you fall into a oh well I kind of want this person to like it's maybe just like slipping one gun like you know put on her the pillow and you can maybe like go

target practice it's very dangerous to start playing the like compromise your vision game so I hope you'd agree with me right that like if it's if it's not the game you're making then it's it's the data is not that useful I like to think of games it's we're like because you eventually you know if you're working on the game you're gonna lose sight of the trees yeah and our goal is to be the lens that lets you see the whole forest again okay well this is how it works but I mean yeah you know but really for me too or so like I just popped into my head in but like we keep saying like you know as well and I don't that's the weird metric you know it's like it's like effective in a weird way you know is like sometimes you don't want someone to enjoy something it's a dark moment or a horrible thing or it's a prescient moment or something so they won't like it you know didn't he like your test is did you feel really under pressure you know the mean or did you did you feel the urgency of this situation or whatever it is that you know that you need them to feel is like effective well that we don't just an opposing thing where we saw a lot on a game that we've worked on where like this this the more story missions when we ask them like did

you like it they were never the highest score yeah but then when you get to the end of the game and you're like what did you like the most about this game you're like oh the story and so there was something wrong I think there was a flaw in our methodology if you asked somebody a story based mission was that fun well no it wasn't it was engaging or it was a motor generated emotional response but yeah it's certainly I think a big question of methodology and how are you approaching the situation what are you doing because I think like or fun or whatever it is whatever metric intervene uses yeah it can't be blanket applied no I I think you and I don't know you you would know better I think things like are you engaged right that this evoke an emotional response doesn't have to be a positive one but like do you keep you know does it keep you going forward well we all notice the steam reviews you know like you know thousand hours played I hate this game it's boring you know you're like I know in car in the auto industry they have it some form of like putting electrodes on people when they watch TV ads and then they're like I hated that ad and they're like no

you love I think I wanna be in your brain it can get very confusing I remember we were doing like a like a very early test on it on a project I think Alex was little maybe there's a guy in the room that was like I [ __ ] hate this franchise I hate this a new game that you're working on like this is so stupid like you've just completely ruined everything and then we're like so do you think he'll buy it means like what do you mean I already like pre-ordered the Electress Edition seven times now I can't wait to play in making I didn't know what to do everything you do but I own like nine copies of it and of course I'm buying and thank you so much I like don't I don't understand like I don't know what you do with that information he owned it more than we did I think that's what it was yeah he was more in saqqara that's where you need your user researcher to step in and say like this is what he really means yeah we just yell at the one-way glass though I just crashed the reader one you know that's not soundproof right I think that was a guy this is really horrible remember we were gonna put him in the game yeah for a while we're gonna put him in the game we started running down the court she's gonna live on using a live under a bridge and every time you were successful in something he was gonna come in and be like I said you're

terrible at yes he'll be talking about that TV stuff as it applies to video games maybe I don't know actually okay okay okay brain wait we're coming to the end for coming to the end so let's open it up to two questions we have about eight minutes I think Wow seven minutes so questions questions for our panelists oh and there's a mic just back there if you if you want ask questions just line up at the mic hi my name is Christina wood key I've been doing UX for the last 20 years and games a lot for the last few I'm an associate professor now I spoke last year on UX and I have a question why did you decide to have an

all-male panel for UX that's that's actually a very good question we didn't we didn't we didn't we actually had we approached a number of women we actually had a woman on the panel and she unfortunately had to back out only a couple of days before and we re struggled to find a replacement I have a website called women talk design call me can I can I get your card afterwards thank you hey Jess Jamesburg is a researcher i work for and so i have to be very careful to my question here so question from the production side what would you say is the biggest struggle or how do you approach the challenge of integrating the timing of research and how you get that into the game in an appropriate way so I think it depends on you know how well your studio or how well your team is set up to both gather data and respond to data I think

you know you can create a system that if you have the the resources and the time you can create an infrastructure that allows you to collect data frequently and accurately over time and that will give you the data points that you need toward to make pretty frequent decisions but if you're also over scoped in your project and you don't have a whole lot of flexibility then you might not be able to respond to that data more quickly so I think in the organization of your team you have to decide hey like how important is data going to be for us in our development process and there's a range there it's not going to be it's not or it is right but you can set up systems or processes that are more conducive to responding today than others hello my name is David hooker actually do have a question guys okay but is more for the research side how do you deal with our have any keys or tips for that kind of data where the consumer is obviously going to purchase the game like yes just address but their feedback

is still sort of negative so on one spectrum it's not gonna work out but on the other spectrum what the the metrics for how many sales are copies what have you is gonna provide the creation particularly in like pre-production sorry the question is how do you reconcile people that are still going to buy the game but have opinions that are like they're they want some improvements on some of the features and so I think that you know purchase intent is not always your most important metric when you're doing formative user research throughout a project so negative feedback is just another it's another interesting data point for a thing that you need to consider as you're making the game you know if that if that negative feedback would prevent them for purchasing it adds a little weight potentially but I don't think I don't think you know mashing those two together is necessarily something that you would do throughout particularly you know up until I would say very late into the project trying to douse purchase intent is is a mistake from the games user research perspective it's not a thing that you should be trying to get

out of that type of testing thank you here we out of time now we have a couple military we have four minutes hi guys how you doing I have a question about because I never worked in an actual games company so I guess I just have a question at the end of the year you know you go and you do your final report and you know you've done a good job and you know maybe there was some fighting between the designers and the user researchers but how do you say you've had a successful output like what is how does your boss say you're doing a good job because like just hearing how there's like global fighting and stuff like determined in the company you mean like on his like performance review well I write a lot no I know I mean for me I'll just like this objective metric is always that if I you know because I you know you've probably heard versions of

this already on the dev your job is to defend your vision and try and make sure that it's getting through and executed their job is to tell you when it's failing or when they have a better idea all that sort of stuff so if me the success is when we've heard something we like work through that problem and you can point to something like we had a lot on far cry 4 you know you know we're at the end I was like now that made it better you know the mean and that is a hardly subjective blow be metric for it in red you know today in terms of hard metrics if you're using some sort of if you're integrated with the team in using their bug tracking software you can extract everything that you ever said and look at you know did they take the feedback did they throw it out as will not fix war can't replicate or whatever or need more information the information and need more information is a super interesting one because that means you didn't do your job well enough and the date they saw what you're saying they're like I don't quite get it yet and you have to go back for that and so looking at the breakdown throughout the milestones of the project of all of the things that you told them and then tracking how those were resolved can be an interesting way to see like am I being effective as a researcher would you agree or disagree that at the end of

the day like the game if it reviews well and sells well you're given like a pat on the back and if it doesn't you're in trouble even even if you did even even if you're like look I showed you all this [ __ ] was wrong there like you it would be are interesting like armwrestling match of who reads reviews more carefully between creative directors and games user researchers because you guys I mean we're both biting at officer while we're fighting together but on things that are so kind of do my my experience like we go through with like a highlights and as soon as he sends them to me but physically because it isn't need to he just likes to I wrote I told you sale report once it was very cathartic but but I think you know we so so I from my position if you're going really high level I think I measure my personal success successes is when at the end of the day if I can talk to the team that I've been working with and say that you know they were not surprised by anything once the game was live then I've done a good job right if they know everything

that was wrong with the game before it went out and they accepted that and they made decisions on how they were going to to deal with those things I I did I did my job and you don't get a call going hey you said system was okay and it's not yeah I have sanics in my pack okay I think thank you do we have time for one more we're a squiz my okay mostly I what I hear represented here are triple a console type games and I know that one metric that's used a lot on the mobile side is just retainment and so I wonder if that's something you look at during a soft launch that's a lot more that even in consoles it's all retainment though yeah you Johnny used to have a thing you could see where people checked out like it was like this mission everyone was there they didn't play the game anymore after this mission or like that you'd see that you'd see the places where people pop out your attention in it you know that position and retention are the key metrics for everything so do you I then you have to wait until is

there a certain point where it's too soon to talk about that I mean you need to wait at a certain point for that it is tough to check retention in progress on like a story game cuz you know you can ask them like oh is this the point where you would think about maybe quitting and they're sitting in the lab and they're you know they're there till four o'clock and you're like would you maybe want to quit now and it's not the most reliable metric but it can be interesting haven't there been on occasion who's where like during the test someone's like you know what about I'm not doing this anymore they like I'm not going to have a cigarette they like don't come back like that could be interesting data that's alright it's happen usually usually you suddenly get about of sickness like you'll you'll have like 16 people and four of them will get sick there was something though I don't remember said there was something where the guy was like [ __ ] this and left like once not on anything I worked there's a few Weaver just to be very clear there's always the ones we kick out for being too stoned when they turn up all the time you look at me like you just like oh [ __ ] you're gonna [ __ ] up my daughter

but then again I always think though but then again that's your audio it's like 50 percent your audience like like maybe he should stay maybe should maybe the half of them should be stoned and then you know and then we would get really staccato maybe in the new world will be allowed to once mr. trudeau lets it happen once once marketing comes back to us and says that there's a stone a segment exactly they know it okay okay guys so we're out of time a couple of things please fill out the evaluation you got scans you should be getting an email with it good marks please and we'll be heading over to the wrap-up room which is in the West Hall level to room 2022 and wait twenty twenty to twenty twenty four oh okay they just listed off all the wrap-up rooms I'm sorry we're gonna be going to thirty twenty two so no no thirty we're going

to thirty twenty to twenty thirty twenty three which is over there somewhere just follow us after we leave if you haven't talked to us