Consumer Insights In Gaming | Blizzard, Zynga, Beta Hat, and Well Played

hey everybody and welcome today we'll be talking about how to most effectively utilize consumer insights in gaming from my own experience i've seen a lot of research studies kind of gone wrong and due to various problems and errors have led to false conclusions so today we will be discussing consumer insights with arguably the best ci practitioners in the entire industry can i say that guys yeah absolutely so first we have mark allen bach founder of well played research second stan kwan founder of beta hat third matt penfield vp of consumer insights from zynga and our final speaker mike swankowski head of global research and ci from blizzard welcome guys hey welcome thanks for having us thanks so just kind of jumping right in i thought we could start with kind of trying to understand what

exactly is meant by ci or consume consumer insights and maybe speaking to some of the primary applications that ci is used for in gaming maybe starting with you stan um yeah so i think there's um you know when it comes to consumer insights i think a lot of organizations can kind of view it a little differently depending on how it's defined [Music] generally like the way that i view consumer insights is really understanding like why people are doing or taking actions why people are behaving the way they are and in my own experience you know having you know led strategic planning teams at ea and ubisoft and um even on the media side uh there's always kind of this mix-up between consumer insights and ux and market research and so for me i kind of put them into

three different buckets i would say one is kind of market research which is understanding what is happening outside of the four walls and that's more from a market performance perspective like what games are selling how are they selling a lot of these things that we would see in kind of third-party syndicated data sources like npd or maybe like a sensor tower or app any and then you have like what is happening kind of internally within the organization which falls under kind of business intelligence or internal business analytics and these are all metrics driven uh and then consumer insights is really about trying to explain why a lot of these things are happening and so where kind of me and my firm sit is really helping to uncover uh why some trends are unfolding uh within the industry what people are thinking and what we can imply in terms of how they're thinking and how

we can you know apply that to either the marketing process or the game development process okay and maybe mike what what about for you guys yeah totally agree i i like to describe it to you as consumer insights being the group that answers the questions about our players or potential ones that we can't answer purely with sales or analytics data so understanding those those motivations um you know what what ultimately drives things and it helps pretty you can be used by pretty much any department i mean marketing is is a common user commercial teams others but uh development side events many different functions use consumer insights mike can you give us an example of like what would be one of the primary kinds of tests that you guys run at you know blizzard sure there's there's a wide variety of uses uh and it's very custom so some common things you will do whether it's you know kind

of brand health tracking understanding things about awareness consideration opportunities for your different games profiles of your players who they are what motivates them what they care about and understanding feedback on some potential ideas that that are out there whether it's you know new business models marketing ideas or even even games when they're in a certain level of development okay and what about you marquette well played and is there any specific application that you get more of than others yeah so we're a qualitative firm so we specialize in focus groups and relatively smaller samples but what that means is that we're digging a lot deeper into that why that stan was talking about before um so a lot of the work that we're doing i like to think about it the easiest way to sum it up is we're we're trying to be the voice of the player right so uh we talked to them and i think the thing that happens a lot across like stan mentioned these three

different buckets is that they're often integrated and used together it's usually when they're the most powerful is when you start to look at the why from a place like beta hat which is from a quantitative perspective and then we get to go a little bit deeper and really ask specific questions of of small groups of people about why they're doing the things that they're doing in their specific scenarios so the most common kind of questions that we get tend to focus more along those so it's things like concepts uh things like positioning where is this actually going to fit among the other games that these people are playing and then helping explain some of the anomalies that folks are seeing in their own data or in the kind of larger big data world by asking people directly why do you do this or why is this something that's important to you okay and then matt at zynga yeah you guys took all the easy answers um so i think what i would add at zynga we have a really player first uh philosophy

that management has kind of embraced and socialized and everybody down the line is bought into and one of the benefits of the consumer insights function at zynga that i really love is we answer the question qualitatively mark to your point what will players thank us for so in the dev process in the production process teams typically have to have some rationale for why are we making this investment why do we think this is going to be a good game or a good feature or why do we think we need to pay this debt off on our user experience and i partner with teams and with other you know groups within the org marketing or biz dev or even legal sometimes just to sanity check that this is actually something that players are going to be happy about and grateful

for and dig a little deeper into who those players are and why you know we believe that over serving them is going to really have a a great outcome for them and for us as a business so i that's my north star is can i help the business figure out what players are going to be grateful for and if i can then we're adding a lot of value right and matt stan kind of broke out the various types of consumer insights that he looks at in terms of like you know ux focus versus more in terms of the product and things like that do you guys have a similar breakout or how do you guys view things yeah it i mean it would be so boring if it's not your job and i'm sure it's pretty similar to mike's format actually so we align with kind of game development right so you have ideas and then you have execution and there's a continuum along the production timeline where you've you go from having a great idea for a game or a feature into really building out the experience for

players and ultimately putting it in software and we are available to support at any point in that process so we evaluate ideas and figure out what makes them great and then we try to track that all the way through to execution so consumer insights gets very involved at the beginning and especially when we don't have a benchmark set so mike touched on this a little bit but i think it's really important to call out that if your company is investing in a new business or you're trying a new feature for the first time then you probably don't have internal benchmarks that you can comp to and so i tend to get involved in those discussions about hey can we do some sort of competitive analysis or start to set some directional benchmarks around what we think good looks like and then we can measure those once the feature gets a little more mature and reaches software stage

so i think stan's rubric is right his his definition is right and we typically hand off along the way once we get closer to real software to other teams to do either you know i don't know market analysis or a ux testing battery all right and maybe focusing on the product itself maybe we could talk about like let's let's say at the early phase of development so concept testing and then maybe you know i know that some publishers or some game companies will also once a game is completed try to test whether they want to fully launch the game or not so maybe along those two types of like how would you test those things and what would be like common methodologies or approaches you could use to try and measure that again maybe starting with you stan yeah i think you know from my from my perspective every organization treats

that like what we'll call it the green lighting process like very differently and um i think there are some myths around the green lighting process um that there is a um that there is kind of a set structure that every single game goes through and i would say that some parts of that are true and then some parts of that i think are very sort of a do fall under the myth category um where you do have to have some sort of a process to understand well is this ready for prime time or not and if it is ready for prime time like are people going to play it because ultimately the question is are we creating something compelling enough that enough people are going to play it and as a business is this a worthwhile business decision for us um now each game from my perspective and even from my experience um having you know run like portfolio

planning is um every game has a different history uh every dev studio you know comes from a different background and then the sort of the the journey that each game goes through is also very different but we do want to get to that point of well how do we take all that context into account um and how do we make sure that we're supporting the creative process while at the same time understanding the business realities and bringing those two together um so it's it's always um a a process and i would say from my you know own experience it's been very much a um i would say a very bespoke uh kind of process and methodology for each type of situation and it's never really been like a one-size-fits-all type of approach okay what about you guys mike i would say a 100 agree with stan it it varies a lot

by company but that even within company where a game is or a franchise is in a situation say you're you know an unexpected franchise how you're going to handle it versus something brand new how expensive it is where what confidence measures you have and i like to view consumer insights as one of similar what matt was talking about how you have this path and we can help out in many different parts on that path and sometimes you need a lot of help in one part and sometimes you need nothing and it varies a lot by franchise and where uh where the game is in the path and a few consumer insights in the the best approaches it's additive like we're helping inform a decision or um information at when you need it sometimes like you're through a certain path and you don't you're good you have other measures that that you're fine with and maybe you don't need it as much and other times you need it a lot right and mike for you are you guys to

some degree like for concept testing i know a lot of companies like to do facebook ad tests but given you're coming from a incredibly well-known company that any kind of facebook ad test you put put out will immediately be picked up by everyone in the press so could you talk about some of the limitations there or do you guys not do like for for a concept the facebook ad test for example yeah i can't think of a situation where we've would have done it for for a concept okay uh in any sort of way the only time i've seen you know that type of work is done for advertising you know of course clicking through but a concept there's way more that has to come with it than than what it looks like as a banner uh just wait way more the decision points uh involved in that process um but they're they're they're all kind of kind of unique they're i understand that yeah maybe you're a more smaller cost quick hyper casual game just trying to

get out there maybe maybe it could work for that but you know for a company like us not not quite you guys are then doing more qualitative focus group types of stuff or well that's yeah that that that's part of part of the picture there's there's other things that that come into it um we have a large company where we can get a lot of feedback internally safely you know from uh employees that gets you you know things kind of move out of out of chunks right so say you have a development team and they're pressure testing some stuff among them and then it moves to a wider group and a wider group and a wider group and you can start getting feedback many different levels you know alpha tests beta tests you can you can do things on small scale it's no longer like we have to make this decision early on and then we launched this you know 100 million dollar endeavor there's so many steps where you can you can get feedback on the process um so yeah it's it's not like a one hurdle thing which is where

people might think it is okay and what about you mark are you doing a lot of qualitative stuff for concept or for yeah later phases of green light yeah a time at both really um the thing that's changed the most it's really interesting this used to be a very short answer you would get three paragraphs you would show a focus group three paragraphs and you would talk to them about which one they like the most and and why and each of those paragraphs might have a couple features under it in terms of a concept and feature set um and now that's changed drastically to become i would borderline say so custom that we almost never do even the same methodology for a concept uh we basically will design from the qualitative perspective methodologies around whatever that concept is so in the last year uh just to give a couple quick examples uh we've done uh for example seven fake announces of seven different games six of which were red herrings right

so getting around the issue you were talking about a minute ago uh but uh we've done soundscapes that go along with concepts to really help people get into the environment that that concept is supposed to feel like uh and then more recently with the kind of advent and accessibility of cloud gaming we've been rolling in more actual prototype play as early as the not the first concept of course but when you were talking about kind of that mid-stage when people are starting to have a first playable um and that's become something that's actually feasible to do online scary from a security perspective but feasible uh from a techno from a technical perspective um and so really we we it's probably the bulk of what we do at this point is is concepts either super early or kind of mid right at the beginning of of uh green light for games um and it just changes depending on basically every single game we do it for

the secure the the leak worthiness that mark talks about where you have to have some secure settings and situations i mean we we basically assume anything we put in a survey or anything like assume it's on the front page of every gaming publication within a minute of you putting it out how are you going to feel and you know how is how like how are all our executives going to feel about that and we work under that assumption because it happens often and i've seen it from our competitors in other places um but to mark's point not everything you put in a concept means it's going to happen it can be fake there's lots of things you can do so red herrings and blinding is very much a part of every process obviously the gaming audience loves to talk about games and there's always a chance that something ends up on reddit and you know i think we've gotten pretty savvy in terms of

being able to cloak um like what the kind of the actual intention is versus what we actually put in front of the consumer um and so we've um you know with a lot of those method especially with now everything going online like there's a lot more sensitivities but then at the same time we have to be a lot more creative and protecting at least for like you know my clients that i work with like the you know the products that they are working on um but we do have these constraints all right and matt zynga is pretty well known for being a highly data-driven company can you talk about some of the stuff you guys do yeah you also have like a lot of mobile game companies the fake llc's that you run your test through i know right so so god you guys are making me feel like maybe i'm out of step here because this whole notion of kind of bespoke projects for every single title um what i tried to do when i came to zingo was really put a format in place that wasn't one size

fits all it was scalable but we did have methodologies for every phase of production and so at concept we had kind of a qual and a quant menu selection that you could choose from to evaluate your idea or your ideas and then at design we had a set of qual and quant um test formats and then pre-production we do a ton of software testing through that entire cycle production continue the software testing and then once we get into alpha beta and obviously live you know you're getting real human metrics so you need a little bit less about um kind of what people think and feel and and you can see what they do at that point i usually pivot into uh what's your roadmap look like team and how are we gonna get you set up for a healthy business and you know for us a healthy business is like at least a hundred million a

year maybe you know ideally two two fifty um with a minimum five year life cycle right so there's a huge content investment that has to come after we've established market success so i'm not doing kind of one size fits all for every and to your question joe i'm really having to work super closely with our pm group and our analytics group frankly just to understand okay do we have a set of kpis that we believe are appropriate for this game and do we have an audience perspective or profile that we can draw on mike i bet you see this a lot too like um when it comes to sequels or flavors of games that we have expertise in like at zynga obviously we have kind of like ville based builder expertise we have you know word expertise um it's really easy for us to be like 90 confident

on a concept in those genres we already know who the customer is we've built games like that for a long time so i don't have to do a lot of work on those games they feel like sequels but on games where again we don't have a lot of experience it's really a ground-up process and and we don't necessarily have the benefit that mike has at blizzard of having a lot of internal subject matter expertise or or people who i would say would be the core target for the game so you know it i have to go outside and recruit those people and really figure out what's the right method to use to answer whatever the business question is and get the signal and i i'd be interested to hear from you guys there's always a tension between kind of size of audience right and stan and i have had this argument for literally for years about how you know what's a representative sample size look like and and on the quantitative side and on

the um analytic side obviously you want larger numbers and my experience is that if i get the right customer profile for a game idea i can make a lot of insight from a very small sample and i think mark you'd probably agree with me there right like if we get 10 people who are hyper qualified to give an opinion about a game type they're worth a lot of kind of perspective and insight and value to the business and so the tension that i have with my partners at zynga is there's a bias toward large numbers and a bias toward quantitative insight and i really kind of firmly believe that if you give me some money and i can go out and find 10 people who are perfect for your game and i know that there's a market on the global scale of you know 100 million people who look like those 10 people then those 10 people have value and we

can use the data we collect from them to project kind of market success so that it's it's kind of an ongoing discussion and it really you know depends on what type of game it is and what our internal level of expertise and comfort is but i'm pretty fluid on flipping back and forth between kuala quant and you know applying whatever's appropriate depending on the the game design or the moment in time to your point uh you know i think i don't think you're off base i think it's a constant balance of the process but not over process things right like you you have to have a kit you know a tool kit and how it goes and where it happens um but not create it so rigid that when the business industry changes that you can't move and to the sample size i i agree with you there i think the most important thing is qualification criteria

like not all samples of 500 gamers are equal in any sort of way you hear that stan yeah that's right buddy 500 well i think i'm you know i think i'm i'm pretty um aggressive with my screening criteria all right yeah look i'm a guy that has background in quant i i i love it and i i use it a lot but how you handle the 500 like i could go to 30 different firms gaming and have them go out and who they screen out who they screen in uh what you chose for your target audience if gamer like we're not all just games are going out to a target audience of general gamers like if i do that say for world of warcraft i would have all this audience of people that don't even play pc that would never play an mmo and like they are literally not relevant um so we you know you have to balance

all those those pieces together getting that qualification criteria is the most important thing to me i also i don't know about you guys like you know matt and mike i know from my time you know at um you know ea and even at cbs interactive it it's also dependent on like who the stakeholders are like and who the product leads are um like some of them do you have a a level of comfort with like having more numbers versus somebody who wants actually wants to hear the players talk about something and there's always that sort of balancing act of like we believe like this is the best methodology to get us from point a to point b and to help us with the right decision-making process but at the same time you know that decision-making process doesn't involve just me right it involves a lot of other people it involves a lot of alignment and i would say like within ci departments one of the kind of big value drivers of ci

uh when implemented well is um like you're essentially more of a cross-functional aligner than like you are just like the beaking of truth right it is well hey like we're going to make we're all going to hold hands and make decisions as an organization um you know do we have the right information and as a ci organization um do you have are you generating enough confidence within the organization that like the decisions that are made even if it's made on limited information that we feel good and that we feel confident about the decisions that we're making and sometimes in a creative industry obviously um you know you're taking a leap of faith with some studios or you're taking a leap of faith with some concepts and there's a part of like just believing that something is going to work even if you don't have the concrete data there but as we make decisions along the way

like where can we get some consumer input that is going to help you know with the right pivots when we need to make those pivots and if we need to make a big decision um you know that that it's not just a lot of infighting or whoever has the loudest voice within the room within an organization it is it is based on hey we're we're actually looking at the same sort of level of truth here and we're operating and having a discussion based on like numbers and data or conversations or quotes that like we're actually hearing that we can then all agree on and then make decisions off of so stan are you the way that you're talking kind of sounds like to some degree ci's being used to help make product decisions are you guys kind of used as political greece in that sense i mean there's always there's always political implications to uh data

right like at any software company in any entertainment company i think that stan's point is really great like the most effective um that my team and i are able to be at my company is when we're matrix integrated right so vertically we're talking to our stakeholders and then horizontally we're talking to our stakeholders our peers um and i'm talking to executives and my senior directors talking to gm's and my directors talking to you know product leads and if we all understand you know what we believe the truth to be from a consumer perspective and we go into a green light meeting and we're aligned chances are we're going to come out of the green light meeting with funding and staffing if we go into the green light meeting and you know the ceo looks at me and he says what do you think about this and i have to say gosh you know the data we have does not support the investment then things go badly so i think stan is right um there is a lot of consensus building

that happens and i i don't know about you mike but i've seen a really pronounced shift um in the area of focus for myself and the the ci function in general at the companies i've been at in the past uh 10 years from marketing which is where i started kind of doing market research on ads and you know just consumer facing content to product strategy which requires a very different perspective on the business and on on the relationships that i maintain so yeah i totally agree i i think that's actually just an evolution of the the discipline over time that the people that need consumer insights first in an organization as they're getting bigger is marketing like they they're like i need to figure out how to position this i need some advertising and those have the questions but as you get more mature as an organization and bigger there's other functions that need

you actually a lot more and i i would agree with you like the percentage of my team and my own work that is you know purely the advertising marketing side versus other functional areas particularly like product development commercialization other things uh the marketing share has gone down over time and other groups have seen the value and learned about it and i view that as the group's just getting stronger and more trusted and reviewed like hey these same tools can be used for so many things um and we're we're kind of a neutral party in this you know we just we just want to help people make more data informed smarter decisions yeah it's um what i found recently um and i think there's like one of the big shifts that are happening within the industry right now is this kind of shift to free to play which is like mobile is kind of dictated now like the business models of

the future for console in hd and one of the things that like mobile has done really well is you like like in the console hd world there is this handoff right of hey the uh development teams create the game and then okay here you go marketing here's the game now go sell this game um and there are in the past kind of two separate research streams that are happening one is the product development research stream and then there is the marketing research stream which is more about how do we convince players to download this game or buy this game and like what kind of words do we use and how do we get their emotions up and how do we make a trailer that just is going to make them feel like they're they feel badass and want to play this experience um but as we move more and more into the free-to-play

kind of um you know like the free-to-play business model um i'm from my from my vantage point i've actually seen um a kind of a convergence or more of a consolidation between product and marketing and where marketing needs to be brought kind of further into the process so that they're fully aligned on like what product features are going in and then at the same time a retroactive conversation from marketers saying hey when we run ua campaigns like this is what is performing this is what is not performing and someone on the dev side may not know what actually is performing in the market and what is not so i've seen a um kind of a i would say a a shift in in recent years of the two teams working more closer together and i would say even a lot of the work that i've been doing recently is um more

aligning kind of marketing and development together to be on the same page um i think in the traditional console hd space like they were very separate but now we're we're seeing a lot more alignment and um and i think that does have to do with like the mobile mentality and the free to play mentality so i do see that coming in right now okay and maybe matt one thing that you mentioned was that you you guys are testing all along the way i'm more used to again the perspective i come from is more mobile free to play but we would do initial concept tests usually through facebook ads and then once a playable is ready then there's some kind of testing to see whether we want to actually launch the game or not i know a lot of a lot of companies like to use nps as an indicator for that although i think that that's highly controversial but wondering what you guys think in terms of that end in terms of you know trying to determine for launch are there any other methodologies that you guys

have looked at and do you guys believe in mps i from my understanding ea and wargaming are really big on mps but yeah let me take this one for i may talk a little bit long on this one but um two questions right one is is this idea of concept testing through um facebook or some other kind of ad method to test for heat and when i started at zynga seven years ago that was what i walked into was here's how we concept test we put a bunch of ads on facebook and whichever one gets the most clicks that's the game we make or that's the feature we ship and what has changed over time with that method i in my experience is that um it's really easy to bias the result set that you get from that type of testing um just based on art that you show right like

that's not a test of a game that's a test of a picture in an ad and so it's a really really good way to test if you have a clickable ad it's not proven recently in any way to be a great way to test if you have a game that's going to make 300 million dollars in a year so the demands of the business have really scaled massively and the level of sophistication that we try to apply to kind of which ideas are we going to fund even if it's seed fund you know engineers are expensive um devs are expensive especially in san francisco do we want to throw you know a couple million dollars at this we need to see something that's a little bit more than you know we saw some basis points of uh performance that were better on uh this ad a versus this ad b so that's the concept test piece that's why we we invest a little bit more in qualitative and quantitative and really trying to understand who's the consumer and what's the idea great um nps so when i was at ea

we started the nps program and there were a couple of guys there who were really really smart quant guys and they started it and they got it up and running and one of the first very profound uh successes that i remember on nps city a was that we shipped madden i can't remember what year it was but we shipped a madden version and the madden version had like a direct tv package the the whatever the highest end price point uh retail sku was had a directv package and we had enabled nps um as an online tool that we could send out targeted uh crm surveys to people who had pre-ordered and so people pre-ordered we sent them a crm survey we said like hey how's your first day with madden going what's going on and all of a sudden we were all standing around this guy's terminal and the nps results were coming

in and they were getting word clouded so we were seeing the qual results and the scores and the scores were terrible and the qual results were all about the directv package not activating correctly people were losing their minds right because it was a huge selling point for the product and i think the team the the madden team is in florida so they're in tiburon and i think the team knew about it as well because they were monitoring cs channels it was coming and through customer service but because we were at hq and you know like a few doors down from peter moore's office who was running sports at that time it was really easy to take this information and be like knock knock so i we're on top of this and here's the triage plan and here are the next steps to make sure these customers get what they paid for and we make this good but it was a really really powerful moment for that process of

collecting feedback that was very timely and very relevant i came to zynga and i asked her out about nps and i thought that i was going to get thrown off the top of the building when i brought that up people were so allergic to it and what i found out after talking to a lot of internal team members was that on some teams they had had their bonuses tied to nps and so there was this really significant financial kind of incentive or penalty when it came to nps that made everyone very very reluctant to engage with it because they viewed it as unpredictable and so the the human response to that in the business was well how do we make this more predictable how do we game the nps process so that we can stay at a baseline and we don't have to look over our shoulder if if we have a release that isn't perfect and we're going to lose

our bonuses so people either didn't want it or whenever i would talk to them about it they were like here's how we do it it has to be exactly this way because if we don't do it this way we can't control it and if we can't control it we're not doing it right which obviously nullifies any value you're going to get out of nps so we just started my team just started an nps kind of it's not really even a reboot it's more of just a data collection process and i don't even want to call it nps because all what we're calling it as a player pulse survey it's really a satisfaction survey so we don't ask the same format that bain developed for nps which is like a 0 to 10 scale and a very specific and precise um coding of the results and and the coding of the language instead we just ask how satisfied are you with this experience like insert game name here um on a survey that pops up in a player inbox

and then a follow-up question that's optional that is can you tell us why you said that and that's it and that i feel really good about because i do feel like you know that ea example was just so amazing and that we could look at that and say like oh it's people on the east coast it looks like the demographic split is obviously kind of male but these are like 25 to 35 year olds who are having this problem like we were able to do a lot with the mps analytics that really helped us pinpoint the customers that were in distress which i think is really valuable and then from the qualitative we were able to diagnose what the problem was but i would never recommend that we tie it to a financial incentive or a performance incentive i think as a diagnostic tool satisfaction is important yeah i mean i've seen a religious war around nps right like some people are very for it some people are very against

it there seems like there's a lot of debate i i guess you're bringing up a point about the not only mps itself but the execution of mps if it's tied to financial results and things like that then that could be um but i know stan mike or mark do you guys have any thoughts on the use of nps itself or is there a better way yeah the mps metric like it's it's one question that got popularized from the vein study a while back i find people are on one end of the spectrum they have a strong opinion about it they hate it or they love it and my view in my team's view is like well it's it's it's one question you know we we ask a bunch of survey metrics it's one metric like you should never if if you're all in on one metric on on anything it gets dangerous like as soon as a company as soon as i've seen companies focus in on one metric whatever it is as soon as then incentives and other things get tied to it that metric stops being predictive as you want because everything changes around it

so my view is like we we occasionally will ask the likelihood to recommend question in certain studies but it's one question of many that that we would report on uh it would never be a standalone thing it's not we don't treat it any higher than than other things so if people want to ignore it they can and the calculation is also what comes up for debate as well and i feel like the easiest way to answer is just tell people what the percentage of the responses are on each and if you don't want to use the calculation you just want to use like the average or however you you you can do that you know we have no reason to hide it in this the magic of the calculation but some people really like the calculation so we'll give them that too it's um there's no reason to hide it but it's it's just one metric there's there's many others you can use yeah i think there's a lot of um i think there are there are a lot of sort of um

i think for those who are looking for some type of a holy grail like they they're just thinking nps can be that and i think for organizations who lean too heavily into nps it is like mps is an over simplification or they're seeking a simple solution to complex problems and that's where mps falls apart is it is singularly to be used to help assist and understand how the customer experience is um or how how customer experience is impacting your brand um and i've i've seen it used in a lot of um really strange ways where um it is used in a green lighting process to say would you recommend this game and then people kind of looking at that

and saying okay well we're going to make some decisions based on it is not for for a reason that is not meant to be used so um you know or they're looking at it towards as as if it's some sort of leading indicator um but mps was never designed to be a leading indicator it was always meant to be a lagging indicator but a sort of grounding point for better understanding why people may feel a certain way so um in matt's example of madden you know nps was highly successful there and that's the purpose of nps is to uncover why people may be really pissed off um or it's less so about being happy like if they're happy about something they're less likely to tell you that oh yeah i'm really happy but they're more willing to

tell you when they're really mad but then when it's used as a forward-looking metric like oh hey we want to run this concept test but hey like see if they'll recommend this yeah it's not a demand metric and and think about like our portfolio stand how many people do you think i can get to answer an nps question about a casino game effectively right like would you recommend this to your friends and family people are like uh under no circumstances would i ever tell my friends and family casino games on my mobile device right so so we get back these scores that are like negative 700 on on casino games and it's just not a useful benchmark because we know there's an audience for those games that love those games right so yeah i you're totally right it's not useful for demand in my experience i do want to bring up one anecdote and and i i think mark may um has probably seen this in the past he's been in the research industry for so long but i'm not going to name the publisher but um

you know just in the past i remember you know running some focus groups um for a new concept and um there was uh at the end of the focus group um the entire dev team was behind the the one-sided mirror and we had him run through a slice of the game and there was a lot of great feedback that was coming from the play sessions and everyone was really engaged and you know calling out things like bugs but at the same time like just how they felt as they were playing the game but then the the devs actually didn't seem to care all that much what they really cared about was they asked they asked me to have all of the players or all of the participants put on the back of their sheet if they were to give this a review score from 0 to 10 what would that review score be

and so this was their way of trying to forecast what the metacritic score was going to be for this game and they would take these six participants average their score together and uh they would omit the one guy who said gave it a zero because they were like ah well i don't know he's not a good participant anyway and then when we average the five together now we have an 8.1 like we're in business um and like that is where research really goes wrong and um and uh you know i'm hoping a lot of that does not happen today but uh it definitely did happen before in the past and and to you know matt's point about incentives being misaligned um when back then it was was pretty common for game development studios to be to be incented based on like what their metacritic score was or what certain review scores were going to be or some type of a metric um and and

that's when things can really go awry and that's when things get gamed but i haven't seen a lot of that in recent days thankfully so maybe speaking more about focus groups could we talk about maybe what are good ways of running a focus group or where where can it go too far where you can be focus group to death and mark maybe since you're the qualitative yeah you know guy can can you speak to that so i mean a couple people have already talked about the single biggest issue right which is stan left to step out of when they all did the numbers which is you have to take the two females out and then you have your whole female represented you know you split them all the way down that but like that's the single biggest error for sure uh is mistaking this for something representative uh looking at numbers uh starting to split numbers and putting too much weight behind the numbers um the other kind of most common ways that i think something gets focused grouped to death i i'm of the somewhat unpopular opinion

the game cannot be focused grouped to death because fundamentally when you break it down a focus group is talking to eight people uh who are players of your game so there is not any world in which that can be a bad thing it can be poorly done it can get weird results the way that a game is killed by focus groups is after the focus groups when the information is weaponized or taken wrong or not put into the proper context right well what about quality of stimuli though mark as i interrupt you i'm sure you're gonna say that next but like you know garbage in garbage out dude i'm sure we've all been in groups where we're like man do these guys really want this game to get made because i would not have shown those screenshots if i was gonna fund this but cool let's see what players say you know like there's definitely better ways to i guess hedge your um your process so that you get outcomes that are a little more realistic relative to market appetite right

well for sure but i think if you have garbage in and then garbage coming out that's understanding the context of what you really have there those are some tough conversations that you end up not having with the direct people that you're that you're working on a project with right uh and you know you end up having afterwards about well there are real reasons that these results happened i mean i think having been in the qualitative industry for so long one of the most important things is adding context and just accepting that there are very real biases to the way that we do things and there are very real biases to the answers that we get directly from participants and understanding some of that context and not being afraid to also be consultative in the end with the results right there's a lot of times where we come out of a focus group and we say hey the focus group said this like that's not a good idea or you know when you're actually in the room you could feel the vibe of they were really saying this because there was one groupthink leader who felt a certain way and he kind of led them in that direction

but when you really listen between the lines you get that way so you know not being afraid to hear the player's voice but then also really interpret that is really important because stan you give the example of asking for review scores you know you're asking players to play reviewer right we try not to ask players be anything but players you can't ask a player to be a reviewer you can't ask a player to be a developer you can't ask a player to be a marketer right you can ask them questions that surround those things but you then have to take those and give those to the marketer to the developer to the creative lead and let them make the decisions because you know players are great they're the lifeblood of literally everything that i do but they're not everything so you need to understand what they're saying and put into content amazing sound bite i'm totally going to use that probably tomorrow when i have to go into an exact meeting and remind everyone that players are just players guys they are not editors so let's refocus

yeah i don't want to go too far down the anecdote route but i mean i have two specific examples i can think of where we were doing focus groups one is very fast uh and the guy and the the person said obstacles the spectrum the person said you have a question in your discussion guide that says how would you make this game better of the players we can't we can't ask that that would offend the people who are making the game so don't ask them how they would make it better you have to find a different way to answer that question and then the other end of the spectrum both of these are bad by the way and then the other end of the spectrum which was somebody who said if you don't have them circle whether this should be real-time or turn-based combat how will i know what to do right like how will yeah that's a real anecdote by the way how will i know what to do and in both of those cases obviously those are pretty extreme examples both real examples but extreme examples of of what not to do right both huge successes i'm assuming yes yeah yeah uh actually let me think starcraft was one yeah played by millions guys

played by millions one a middling success one never happened okay so mark how what what's the best insight or what's the best way like when you conduct a focus group what should the takeaways be uh well really it depends on the focus group but that's a wholly unsatisfying answer so i think uh really the takeaways i i think about it as stan used a phrase earlier we use a lot you want to support whatever the teams are that you're working with so if that's marketing a lot of times that's creative if that's the development team a lot of times that's you know supporting the actual developers and basically the takeaway should be this is what the players said this is what they feel what are we going to do with it right and that third step is the one that really obviously a lot of times i bow out of that because as much as i'd love to be involved in it more that's kind of the next phase where they take the research off um but you really need to come out

understanding what they said and then using that we used to use the word ammunition right and then you get to use that how you like i think that's a little i think there's a little flaws of that metaphor so we ended up using the word support instead but basically we want to support the team so that they know how players are thinking and they can decide what to do more times than not they don't do what was said in the group you know every single cover is not a scantily clad woman with a gun over her shoulder right that probably would be the case if i only asked 14 year olds what should be on every single cover um so understand why have those conversations and may make informed decisions on your own yeah i like the way mark put it that it's the coming out of it that these are the observations we heard this is what hurt and now the players will jump into you this is what you should do you need to fix this and that's the stuff you kind of want to like because there may be very good reasons for why you wouldn't do that or you don't or they don't have all the contacts a great example i run to is

when people are like well ask them what they would want to do in the the next version of this game and they literally just take their favorite game or their two favorite games and combine them and that's what they say that you should do they're not like that helpful with that but getting the observations like i feel this this is where i'm confused this is the parts that's frustrating this is the parts that are exciting that's those are the observations that you can do a lot with um and it's really probably the best use of that work yeah mark were you uh you i think you were involved in this but another example mike that you just reminded me of where i think focus groups can be incredibly valuable for product development is just finding really jagged edges that are sticking out in a way that you don't expect about a product and you know examples that are kind of recent and relevant for me are we were in europe doing some work on a game that

is in development and we were just doing some character sketch um check-ins and there was one character in a group that players just rejected like violently rejected this character and we probed it and we got some responses and the team was a little shaken up about it and you know they were like well what do we do about this do we pull the character out do we like what do we do so we made some changes to the sketch for the next round of groups that we were going to do which were in another country and you know the understanding that we went into that second round with was look if we get the same feedback then you know there's a signal here and we should take it seriously and we got the same feedback uh even after the changes and so that character um you know underwent some pretty serious revision x post that feedback and um since then has been really really it's kind of emerged as a fan favorite

character and so i think that's an example of how to process even extremely explicit negative feedback that if you're not intentional and thoughtful about how you you know position that to a team you could just come back and be like well this character failed so take it out kill it and potentially you know the game is tainted from that feedback and that should never be the case right i think you need to be really thoughtful about the specific information that players are giving you and then also try to apply context to it that gives the team some options right like it sounds like mike you do this too part of my job is coming back with options so here's what we heard and here's what i believe that means based on experience and process and here's the three things that i think you can do about it and feel free to come back and give me like a more elegant solution

but we can move forward from this in these ways this is not a we're not at a terminal point in the product and that is a really healthy way to use focus group information i think um as opposed to just you know looking for the people who are going to tell you that they love it and throwing out the people who hate it or getting really obsessive about people who don't like things who you know don't always represent a majority perspective all right so if that's the healthy way of using ci maybe we could talk about the opposite the the unhealthy the kind of pitfalls like what are common mistakes that you know based upon your experience running ci like hey watch out let's not do this mark you you mentioned a couple of examples as well but like i don't know maybe starting with you stan what should people avoid in in terms of like whether it's methodology or how to approach ci hmm um you know the first thing that

comes to mind is um is is um [Music] so so i come from a from a background where um i think ci is most effective when you are able to ask the right questions that align with what you're trying to achieve and a lot of times there's and i don't know if it's like laziness or just hey this is the way we've been doing things it's um you know and i'm mainly a quantitative researcher so kind of more like large scale surveys and stuff like that where it's very easy to say hey we've been asking this type of question in this way with five points and so i'm just going to lean into that every single time i think one of the biggest i would say pitfalls with kind of on the quantitative research side is um relying a bit too on

like methodologies that you believe are tried and true and losing sight of like well what are the actual questions that align best to the objectives that you're trying to solve for and that really comes more from a perspective of like just needing to ask the right questions so um you know for me it's more i think even if you don't have the perfect methodology or it is not like a pure gallup poll um you know sponsored but if you ask the right questions based on whether it's market performance data that you're seeing or whether it is like uh hey like there's a gap of knowledge here and we really need to understand how we can get to an answer um like i think asking the right questions is important um and i've seen plenty of times where the intentions are right but the but the

way the questions are asked are are ineffective in in trying to answer those questions okay and maybe mike sure yeah um one of the things where i've seen some some pitfalls where people try and get players or potential ones to answer questions that they actually can't possibly answer or uh some other bad examples i've seen in in the past would be like well we went to this we we asked a bunch of potential players about their interest in the feature and be like oh that's interesting where did you do it well we went to the the event where people paid 200 to get into the event that loved the franchise and they think this feature is great it was like like that is not a good representation of your audience like if you want to get feedback on that event or you're you're couching in and they're like this is the super fan feedback like this is our vips or this is our how

you get like totally works for that but saying like this is representative of gamers overall um i've seen those kind of pitfalls happen that can be really dangerous and i i think it's our uh our job as both on the client side and and with partners that kind of wrangle be like this is what you're trying to answer this is what's possible to answer here's how you can go about doing it and putting it in the right context where um you know these things can be weaponized in dangerous ways with without that and mark i know you've already mentioned a couple things but do you have any others yeah just from a quick one that i can that comes to mind is the don't be too rigid in your definition of your audiences particularly for qualitative and then kind of a sub bullet of that is is don't use a firmly binary system to try to arrive to all of them there are some really good implementations of this around segmentation tools and around really like well-thought uh ways to use a binary system to get

down to a particular consumer like matt was talking about before finding the people that really matter you can get a much smaller number but i think all right mark what do you mean by a binary system sorry um and that actually made it technically not even be the right term i'm revealing myself to be a true qualitative person here but in my head when i say binary i mean choosing between two things so i think of a binary tool as i'm going to do like this or that or something else do you like either a direct choice it's more than two that's why i'm probably wrong to use the term binary but what i really mean is uh they need to be this age play this game and have 10 plus hours of playing this other game right i would consider that a binary kind of flow and what the mistake i see a lot is you end up losing someone who would be an amazing fit either because of their personality or some of the things that they do outside of those five questions and who play nine and a half hours of your game right um and really what that comes from is is

people who understand games so doing some level of fuzzy recruiting kind of like a fuzzy google search right where you give partners uh at least a little bit of leeway to say i've talked to this person you know they said they only played nine and a half hours of hearthstone um but you know they were playing uh they're a good example they only reached you know level five last year in hearts in last season in hearthstone uh and then when you actually talk to them it's because they only play off meta decks because they find decks that they find you know online they find net decks to be super boring so they play all these crazy decks but they actually paid 300 hours right and so have a little bit more of a flexible qualification system as well yeah understand what the spirit let me try and do it a little bit quicker understand what the spirit of your recruit is and recruit to that instead of being very rigid with three or four qualifications got it and matt those are all great examples stan you kind of under delivered i'm not gonna

lie i was expecting you to be like these are all the things that my clients do badly but i guess i'm not gonna i'm not gonna throw you under the bus i guess you're just trying to be polite given the audience uh so there's a couple of of very kind of consistent um areas that i try to encourage improvement one is coming to your ci partner or your ci function without a point of view so coming to me and asking me tell me what to do about this problem i have is really difficult it's difficult for me to boil the ocean for anyone i mean you can do it but it's not worth throwing 10 million dollars at some of these questions to answer them holistically it's much more rational to come to me and say uh you know we've taken a look at this in our game or in the marketplace and

and here's our best thinking and we were hoping that talking to players would give us a better perspective can you make some suggestions right and then that gives me a much narrower solution set to recommend from so that's really helpful is just have a point of view know your product know your know you know as much as you can about your industry or your business and then one of the other areas that i actually think comes up pretty consistently is um it's a a generic kind of presentation format where you guys probably all know this experientially and i see it so frequently you get the most value out of consumer feedback when you give consumers things that are really different to respond to right so if i show you i don't know a wow screenshot and then i

show you a battlefield screenshot like you're going to have an opinion that's pretty divergent about both of those even if you like them you're going to be able to tell me if you're qualified very specific things about those and that feedback will be differentiated um if i show you a battlefield screenshot and a call of duty screenshot that looks exactly like the battlefield screenshot it gets harder to really determine what the degree of difference is right so it requires more stimuli it requires more discussion it requires more energy and so i really try to push my teams to give me kind of boundary breaking or edge case examples that i can put in front of consumers so i ask them hey think about you know how far would you be comfortable taking this right like how bloody would you make this how you know i don't know how extreme how

difficult would you make this how confusing would you make this give me that example and then give me something that is um several degrees away from that on a spectrum and that can even be like an art test right like give me a palette test or give me a game board test um that's very kind of passive and neutral or pastely and then give me something that's very primary and always choose you know stimuli that's within reason that you would actually make or invest in but that's going to get you the best feedback every time like that's a i think we make that mistake a lot um i make it you know a few times a year where i'll put these i these um concepts in front of players and they're so similar that players are like i don't even know what you're talking about like what's the difference between these things i like this word in this one and i like that word and that one and i'm like okay well the stimuli wasn't different enough we need to you know we need to pause and rewrite it

in the back room and then try it again for the next group because everybody's just kind of they can't pick a winner and they can't pick a loser which means that there's nothing separating these got it so for a next question i was wondering if we could talk about changes and kind of what's coming up in terms of merging trends so in terms of like the consumer insights discipline and gaming can we talk about what has been some of the big changes over the years and how do you guys think it's going to evolve uh in in the near-term future maybe starting again with you stan um you know actually you know i would actually love to hear from mark on this because given everything that's happened you know obviously with kovid and everything it's it's really changed the way that we do research and um i know it's impacted you know mark obviously a lot more than it has myself so i would actually love to hear from mark about like how like the recent year um has actually changed uh the way we do research like online

is this so did you guys arrange this ahead of time this feels like uh guys i'm calling you out on this oh by the way mark after this i'd really like to hear about like how segmentations work and i mean i don't even know it oh surprise yeah no no no so i i promise i will be brief with this but stan's right i mean i think the metric that for people who maybe don't know as much about qualitative research is i spent probably on average over the last five years 100 to 125 nights uh on the road uh and this year has been zero since march right um so add to that the fact that we i actually built a facility about 10 feet to my left that also has yet to be used that was built in february so um i mean for us the the simple thing is that everything's moved online uh and from a qualitative standpoint that has some really really amazing things i've talked to people from west virginia

from alaska had our first non-continental in alaska last week i talked to people in montana regularly as part of our sample that has really helped us reach out to a lot more people because now the limiter is not how close you live to a facility in san jose chicago or new york it's now if you have a computer so that's one of the big changes uh we've been talking to a lot more people and then also the change that we mentioned earlier which has been the biggest challenge is security so the the technology we've we've invested in and has actually come works really well we we piece together different pieces of uh technology to make our own kind of online qualitative platform and that stuff actually has been time-consuming but delivers pretty much the same result the real really big change has been with security uh and being able to show some of these things and even let people play them at home in their own environment when you just lose a lot of control

over what happens i think if i had to pontificate on where this is going to go i get asked this question a lot and i would say i think from a qualitative standpoint probably as much as half of this will stay online from now on um as much as half of what what i'm doing will stay online there will always be a really big need for that security level of being in person matt you mentioned at zynga that one of the big changes was moving a little bit more qual instead of purely quant but it was was there anything else that zinger that you guys have changed or will be changing yeah i i think there's two things one is i completely agree with um these guys about kind of the evolution of interaction with players and the consumer insights framework i think that the tools and the technology have evolved and i would i would submit also that the player's level of sophistication has like massively evolved over the past seven years i mean i left i started at sony um

and then i went to ea and i left ea and went to zynga and when i went to zynga i was like okay all these mobile players are what at ea i would have considered to be more casual players meaning that their kind of level of exposure to gaming is the lifestyle and their level of commitment to the product and kind of their awareness of design tropes and systems and games is pretty limited uh they they just don't have a lot of experience and so you know they're seeing things um in candy crush like i don't know endowed progress and um kind of cascade content and they're seeing it for the first time and they're really loving it and responding to it in a way that um is interesting to me coming from a much more kind of sophisticated audience for bioware games or um kind of battlefield franchise games and over the

past seven years what i found is that the level of sophistication among those audiences has massively increased to the point where mike i would probably put some of the core blizzard fans in a room with the core awards with friends fans and those two groups of fans could endlessly debate all the reasons why developers are destroying their favorite franchise that they've devoted decade a decade plus of their life to um because the developers don't understand the game they don't have the same relationship that the players do and and these players are sophisticated and some of them are you know 75 year old women and some of them are you know men who are college professors like the the breadth of the audience is global to mike's point and very very diverse and now um thanks to you know i think some of the the mobile games that we've seen in the market and just the success of of gaming as a lifestyle vertical

people you know are getting into games that in the past they were so hardcore that even hardcore players wouldn't play them and i'm getting you know stay-at-home moms coming into focus groups or doing remote interviews who are like oh yeah no i lead a clan in um you know this this game this mobile game and it's really difficult to get everybody to do what i want but they'll do it anyway like we'll have you know two hour dialogues um about these incredibly sophisticated mechanics and and how they manage group dynamics and i think we started to see that in mmos so i'm sure mike's familiar with it but it's now it's everywhere like it's in our harry potter mobile game you know because there's guilds in that game and we have guild leaders in that game and they spend the same amount of time on discord that people are spending on discord and marvel strike force you know what i mean and so the the barrier

between the level of sophistication and the commitment has diminished significantly which is awesome because i think it means that as you know people who work with game makers there's so much more freedom and and the experiences that we can offer to players and as a researcher what it's pushed me to have to do is find new methods to kind of collect data and just look for signal and one thing that i've been super on the fringes about for years is the idea of biometric data as a useful tool for kind of collection and analysis of preference and over the past couple of years i've started to lean into it a little bit more and a little bit more and a little bit more and it's starting i think to become a really robust data collection tool for us that helps us to separate the distinction between kind of system one response to information right like

subconscious or unconscious response and then conscious response intentional response so if i ask somebody to look at a picture and tell me what they think i'm already kind of subverting the organic response to the picture because i'm asking them to be intentional about their feedback and they feel like they have to give me an answer but if i show them a picture i measure how much they're sweating and where their eyes are going and then i ask them afterwards so what did you think of that then i have two really really strong data points that are giving me a signal and if the signal converges i have a lot more confidence in the feedback that i'm going to give a studio partner or an executive and and what i found is that adding that layer of um technology to the conversation is very helpful in reassuring people right like people who are sub are concerned about ci as a function and and

worry about you know we don't have the right sample size or we don't have the right sample composition or qualitative versus quantitative i have a bias on on either of those methodologies just adding new ways to record consumer response helps to move conversations forward in a way that's really productive sorry for biometrics you said you're measuring where the eyes are going you're measuring sweat and are you measuring like brain waves or yeah yeah we do we do some neuro some eeg stuff as well i the trick with biometric is that it's expensive and it's kind of um it's very science-y but not in like market markets laughing right now not in like a way that like is easy to understand well dude mark you should laugh because i i resisted this for years like for a decade i was like no thank you i don't think this is going to work what i was actually laughing at was the expensive yeah because i i don't think expensive is a fair enough word like well listen man remarkably expensive

yeah here's the here's the math right you guys have heard me well mike you haven't but um mark and stan have heard me say this a lot of times if you want me to deliver you high confidence that we're gonna make more than a hundred million dollars it's gonna cost some money for me to give you an answer that we all believe in if it was easy to you know hedge that then everyone would make games that made 100 million dollars so it really depends on how big is the bet and how confident do we want to be you know ninety percent of my job is de-risking investment right ten ten percent of my job is finding opportunity and the ten percent it's only like once every five years that i find like a titanfall you know where i can't keep people from rushing the moderator because they want to steal the assets to take them home they love the game idea so much right that doesn't that happens almost never but

most of the time people are like oh yeah if you just change this one thing or these two things then i'd be way more into it and and figuring that out and figuring out like how are we making this the best game possible for the most people like i need as many tools as i can get to do that at this point because everybody's a gamer so yeah maybe we could also talk about the flip side of that like the not so expensive and for maybe like a small dev studio can you guys what recommendations would you guys have in terms of a small budget for ci what would you actually test what's a good use of deploying capital for a smaller studio and maybe i'll just open it up to anyone who can respond to that so i'm coming from a small studio so i'm very interested yeah i i and i can and this kind of goes back to the previous question of like one of the kind of the biggest changes and trends happening in consumer insights and um i one of the more exciting things and i can even argue that this is kind of detrimental to my own

business is that there is like uh research technologies have come so far that um it is very easy to deploy your own research and do it very cost effectively um really like the only reason why like firms like agencies like mine exist is hey i just have some institutional knowledge about like the gaming space that i can add value to the analysis or whatever but there are a lot of really smart people out there where now you have the tools to launch a survey to do some online focus group testing to do some user testing and like a lot of those technologies have really gotten to the point where it's so accessible for smaller organizations to take advantage of where i think that's really exciting um because now anyone can be a researcher but i think the kind of pitfall there also is okay well you want to make sure that

you're asking the right questions and you want to make sure you're using those tools properly and i think that's where you can lean into consultants to kind of help make sure that you're kind of asking the right questions or does designing an experiment in the appropriate way instead of like hiring a firm to do something but like a lot of these tools are now available for anyone okay i'd also say from that like there there are a lot of free tools you can start small you can you know you may not do the most sophisticated analysis in any sort of way say on a quan thing but say you have a game you have an email uh email addresses from your players you should you should be asking them some things ask that's some basic things also it's it's worth talking with you know a market research firm and seeing how they do a study once or twice you know early on take put that investment in even if you know you're not going to be

able to do a lot of it you're going to learn some best practices and things from that experience working with them and then you know as you scale up and then you you can you can do more um but keep you need to be talking to your players and your potential players out there in however you can if you have to do it you know in in hacky small ways that are not as representative uh you know it's still better than just talking to the people that's whole livelihood is dependent on this game you know right like this that's a that's a bias group you should and you don't want to just talk to your friends you need to be talking to somebody that has no reason to say anything you know positive or negative to you and it does need to come from a place of curiosity like it needs to come from like a purely like like i really do have a question that like i need some feedback on not just i need to validate what i'm thinking or because validation validation research is

is kind of a luxury for bigger organizations who really need some sort of a scapegoat if something doesn't go wrong from my perspective but when you're a smaller organization like it you know that type of research i think is a waste of time i think it's um you know now you're working towards something you've reached some bottlenecks in terms of decision making and there are you know some effective ways that you know it gives you enough information without you know going down the big agency sort of you know uh budgetary side of things where you can get the answers you need to give you the confidence um to make those decisions yeah there's also a lot of industry dialogue that happens right like i think this podcast is a really great resource for um you know just figuring out what trends are and what conversations are happening uh gdc is constantly releasing you know videos of past talks that are

available for like zero or near zero in cost so there's a lot of historical information that can help you make decisions about game design and game development that'll put you kind of further down the decision tree than you might have started and then you know i agree with mike and stan and mark like then just go find the people that you think are your players and talk to them if you have to talk to them in a coffee shop do it there if you have to talk to them on a survey do it there if you can get them on discord like do it there but their feedback is going to be valuable even if their feedback is critical that's that's the best place to start great just just to be super direct too reach directly out to me if you want to do something qualitative and you have zero budget i will help you figure out something to do small studios i mean you should talk mark i didn't realize that this is a gliding scale operation that you had running so i'll expect a refund

yeah that's right uh totally not no i should be but i should be clear where you know we're i'm happy to hop on calls with with anybody right and talk about some of this stuff and no you know there's a i think what i'm trying to encourage is i i think mike makes a great point to get out you'll learn a lot from doing an actual study but i think there's a lot of places that are probably pretty far from even that right uh even a small study of you know having a budget to do that so just to reach out and get in a quick phone call some moderation tips some sample bias tips some of the things that we think about listen we all play games like we our intellectual curiosity is probably strong one of those biggest reasons that we do this uh so we're we love talking about this stuff i'm happy to talk about it so i will put myself on the line here and say i will definitely respond to anybody who reaches out to us right and so maybe final question then along those lines and thanks guys for hanging on with me i know we're running a little bit long here is is this is how should people think about

in-house versus external ci so i guess working with you mark i mean it seems like there's you know you can kind of enter pretty quickly when should studios also think about building their own in-house ci and maybe starting with you stan in terms of maybe you know how what what are the situations where or what are the types of clients that would be best to work with you or how do you think about in-house versus external yeah no um so and i would just say like you know one of the reasons why i started uh my firm was um really just based out of my own experience on the client side um running research teams and you know i'm definitely on the side of like just roll up my sleeves and just do it myself and i came into kind of a a lot of situations where i would hire an external agency and i would feel very kind of unimpressed or would have to

spoon feed a lot of information before i would get to the result that i that i want and um so in terms of um in in terms of sort of external ci i think just um you know where there needs to be an area of specialization where they bring meaningful value to a specific area that you were trying to solve for um and i think this is a growing trend in kind of the research spaces uh kind of a move away from the one size fits all market research agency to ones that are a bit more boutique and um you know are that specialize in very one specific thing like you know matt mentioned there's a biometric firm you know mark there's a reason he just focuses on qualitative research i focus on quantitative research i think when we focus on one specific thing we can do things really really well um

the in-house ci functions you know i think are really important to make sure that like you know the product managers and the marketing um you know cert like marketing managers are not just going out and getting the answers that they want to hear um like there needs to be a kind of a culture of like the in-house ci teams really helping them be the purveyors of like hard truths and and it's not to give them bad news it's to say okay well um like kind of a um like to to to instill more of a like yes if culture within each studio of like yeah we can we can achieve these things but here are realistically all the barriers that you are going to run into if you want to do this and just having that sort of central voice and that objective voice is is very

valuable to to an organization mike yeah so i'd say that there is an evolution that a gaming company will go through on the the in-house day to uh outside so if you're small very very small you know you're you're under 50. you're not going to need a a full-time consumer insights person like you you just it's it's not a higher you do in your first 50 employees right you get to a certain scale they're like okay there's enough there's enough work that i need somebody on this and they're so so in the smaller smaller scale you're working exclusively with external people and maybe it's executive producers or marketing leads or whoever is is handling that or maybe you can't afford it at all you have to do some things on your own then you bring some some people in house and they start to manage some on their own and some with external partners to help them scale and that's where a lot of a lot of the gaming companies are in that place where they're getting to they they have a handful of ci people

and they're working with some external partners and they're doing some stuff in-house as well and then you can get to a certain bigger scale where like well actually we're running so much of this and we have so many brands we should we should bring a lot more of this full-time in-house like like the user research function is tends to be largely in in-house for instance uh and that's some things that happen so it's it's also a decision on how your org works like if you're a company that likes to have more variable things you can scale up and scale down like you're going to have a bunch of releases and when you're going to spend millions of dollars in research one year and the next year you're going to spend half that then maybe you don't have as many people full time in because you you need that variation um but that's how this is the evolution that i've seen it i've seen it work i don't know if matt you have a similar situation yeah i mean i was just thinking of when i started at sony it was me we

didn't have an insights function per se we had a bi guy who encouraged me to do consumer research and then i went to ea and we were small team internal assigned to different bus we had a bunch of redundancy like you know we were we were sports and sims and bioware and dice and so each of those businesses had one or two people who were attached to them and they managed the projects and some of them were internal and some of them were external agency and then you know what ea pivoted to i know was kind of a very significant investment in integrating ci and analytics right and so now they have probably 200 people in some function that touches what used to be ci and analytics as separate groups right and so i i agree with you i think there's a huge spectrum of of investment that you can make and

commitment that you can make the the area that i i really think is kind of most interesting that you guys talked about is um just figuring out at what point you need to scale up that investment and it for me it's always been at the point where there's a portfolio and you have to start making competing decisions right and it's very difficult to make informed competing decisions when the people who you're talking to about the information are executive producers or gms or svps and they're just fighting for funding and resources and it really helps to have a third party in the business that's neutral that can kind of help cut through some of that noise using you know consumer and and market feedback to help

the one thing i would say about external agencies and stan and mark you guys are so qualified uh within this dialogue if you're a small shop or you're starting out in ci and you want to know or even in marketing or product and you're like hey i don't know who i should hire to help me out with this project you should only hire people who have worked in games do not hire people who do not have experience in games games is a ubiquitous entertainment experience at this point but my experience is that it is still a very niche kind of coded industry and any time i've worked with a partner that is you know a aaa research agency that can do political research or packaged goods or entertainment and i talk to them about games and they haven't worked in games before they fail because they don't understand the coding and they don't understand the customer and they don't understand

the subtext and so i have had incredible success when i've hired people who've either worked in the industry or who are gamers or both right and and that's how i you know came to know stan and mark and i think the three of us or the four of us on this call have a common friend um nate nanzer who was in research for a while and has now gone on to to do some other things in the industry but like you know if you love games and you work in research um then i'm interested in having a conversation with you about being a research partner but if you haven't done it before it's really hard for me to ever you know retain those people um to give me feedback because they just don't know enough they they don't know what they don't know mark i forgot the question i was listening seriously just in terms of like uh in-house versus external ci sure yeah so i mean we're obviously external uh i

think uh the the thing that mike mentioned that we're starting to do some more of some of this stuff is starting to blend but most of what we see as being entirely internal is still ux uh in the user research side uh some of those things are starting to blend where like i mentioned before we're doing some ux as part of our work because people really want to get their hands on a build or something like that um but i again i i actually don't have a ton to add to this i think matt matt covered it well it's a little self-serving to say it as somebody who does work entirely in games but um you know making sure that the the person that you're hiring and the place that you're hiring who is external knows games and and and really does right ask the follow-up question beyond just do you have a games expert it's like yeah matt's the game's expert or i shouldn't use matt you know jerry's the game expert right he's down the hall you know every partner is going to start there but really ask for credentials and bonafides and projects i've worked on and stuff like that for extra all right well gentlemen thank you so

much for your time just to wrap this all up maybe if you have a final message for our audience or if there is a way for people to get in touch with you maybe you could let us know so starting with you stan yeah so um so like i mentioned you know i'm the ceo of um a boutique research agency called beta hot um you know to reach out to me my email is uh stan at betahatmr.com or our website is betahatmr.com so um you can reach out to me there mike let me unmute so yeah i talked before i lead lead consumer insights research at blizzard um yeah if you want to talk if you're on the side on the gaming side you want to talk shop or somebody interested to get into it uh you know please reach best way to get me is probably just uh connect with me on linkedin and send a message and uh the other thing i just leave people with is start if if you're not doing much just

start talking to your players and your potential ones however you can it's going to help you and know that not every project's going to lead to that incredible nugget but but some things will and it'll change how you operate mark yeah uh i don't know that i mentioned at the top but uh i run a firm called well played uh and you can reach me at mark well played llc.com uh website's the same well played llc.com either that email or there's a form on there if there's any questions and i guess the only thing i'll leave with is i'll double down if anybody wants uh some help doing some grassroots research i'm always happy to do a call and give you some tips all right and finally matt dude you are drawing fire from the rest of us nice work uh this was really great joe thanks so much for inviting me um i i think i really agree with mike i i love my job i love this industry i love making games

i love talking to gamers um i if i wasn't doing this as a career i would be doing this full-time anyway so you know like this is me getting to live my passion and i i'm grateful for that and i suggest that anybody you know who's in the industry in any discipline the more time you spend with your players uh the better your games are going to be really and and everybody should probably spend about 20 to 30 percent more time with their players than they're already spending even the people who are the most committed to kind of player feedback and and customer feedback we're not getting enough of it um they have great ideas they are much closer to the product than we are and you know i can't tell you the number of times that i've sat across the table from a guy who looks a lot like me like you know middle-aged white guy who's making games for um you know teenagers or women and i have to remind us both like

we're not the customer here so let's pause on our best thinking and let's go talk to some people who are actually touching this because they're the ones who are going to know better than we ever could what's going to make it better all right with that thanks once again everyone and um yeah bye everybody all right thank you for having us thank you guys [Music]