Losing Player Trust - The Data Dilemma - Extra Credits

so much of today's gaming is driven by hard data things that we developers can fix numbers to sales play length review scores but in focusing on that hard data we have at times lost focus on all the things we can't track by absolute numerical metrics things like how much you trust or respect a company and in the long run that does matter so here's something that happens way too often in this industry let's say a game comes out let's call it heck no man sir and it's great and it sells really well so the company behind it will crank out a sequel now maybe heck no man sir - isn't quite as good as the first one but because it's a sequel to a successful game it will have a much bigger marketing budget this time and that plus good residual word-of-mouth from the first game will lead to it selling more copies than the original now the review scores for this inferior sequel will be a little bit lower of course maybe not

as low as they should be because the hype is real and reviewers are human beings but lower all the same heck no man sir two's developers will see this dip in scores and somebody on the team will no doubt bring it up to the rest hey those scores are looking worse than the last time but then inevitably this sort of internal criticism of the game will get shot down when somebody points to the sales numbers because more people are buying this one they'll insist that the critics are just being elitist about it I mean clearly we're doing something right because people are into this look at those sales numbers and so critique gets dismissed as people flocked to the stores then heck no man cirthree revelations comes along with an even bigger marketing budget because now the thing is a proper franchise and because heck no man sir - was the game of the season last year even more people are gonna buy this one and once again those escalating sales numbers are pointed to as a defense against all internal critique and the company plows forward this is exactly how a great series gets

driven into the ground now obviously not all franchises are gonna see that sort of linear decline in quality but in this scenario where review scores are slowly falling despite rising profits if you only look at the things that you have hard data for skimming the surface level of that data rather than aggressively digging down into all of its root causes you won't see the real damage that diminishing quality is doing until it's too late because that slow deterioration and quality actually is having an impact on sales it's just buried and hard to see underneath all that other data fans do fall out of love with a series once it starts to wear thin and given enough time the number of fans lost will simply outweigh what marketing money and name recognition can do this is where the idea of soft data comes in you'll often hear that you can't objectively measure the quality of a game and that's true but we can measure something's relative quality we do that all the time here I'll show you

which is the better film Mad Max fury road or the emoji movie now I may not get a hundred percent agreement on the answer to that question but I will get pretty clear data it's relative data it doesn't give me a number representing the amount of quality each film contains it just tells me how the two relate to each other but that data is still valid the information it does convey is still helpful it still reflects the real world to the same degree that anything can and if we ask that question again with another data point like which is better spider-man homecoming or Mad Max fury road which is better spider-man homecoming or the emoji movie and so on then we can start building up an ordinal scale and that sort of relative or positional scale can give us data that is incredibly helpful for judging the actual quality of our games heck just do that scale for games in the Call of Duty series or all of the Assassin's Creed games and I think you'll learn a lot while you're at it throw shadow of Mordor into that

Assassin's Creed list as well because there's probably some very useful information to be gleaned from that comparison but this goes beyond just rating how well liked games are this extends to things like customer trust trust is something that we basically never take measures of as an industry but we probably should it's relevant to a lot of scenarios like here have you ever happened to see a company cram a bunch of particularly abusive micro transactions into a game you might otherwise love take your time I'm sure you can think of an example now throwing those microtransactions in almost certainly led to an increase in revenue especially in the short term and since all eyes are on that bottom line including them will be seen as the right decision even if players complain about it after all the industry is all too aware of the fact that players will very often continue to purchase and play the games we gripe about which fair we do that but here's the problem looking at just those hard metrics and the bottom line we basically have no way

of actually telling what damage we might have just done I mean maybe all of those complaints are just short-term griping from people who want everything for free and in this case players are generally fine with the microtransactions it could be or maybe putting those microtransactions in the game actually has eroded your players long-term trust and if so that is a huge deal even for the bottom line because people might not immediately quit or even change their playing patterns over one offense like this but there is a point past which if Trust dips low enough players will just give up on a game or stop buying sequels and stop giving you money and in either case don't you want to at least know you don't even have to get super sophisticated with it imagine if every three months you gave your players a survey asking them to rate how much they trusted your company or arguably even better imagine if we as an industry regularly asked players how much they trust a number of different game companies just like that

you would have a running timeline of your company's relative trust position it's not an absolute number but it does show your place relative to the rest of the industry and gives you a sense of the long-term impact of some of your decisions so if your company takes a sudden nosedive on that chart then you can review what you did last quarter and try to find the cause or maybe some other company just shot up the chart all of a sudden what did they do maybe you could take a closer look Krim some of their ideas when we talk about metrics and the use of metrics in this industry it is important that we look at all the hard data we've been gathering the things that are distinctly measurable with absolute non relative meaning but it's also important that start finding better ways to gather some of that soft data the things that we can't afix an absolute value to but that still have very real meaning because getting better at tracking that means happier customers better long-term health for our companies and hopefully better games for everybody see you next week

[Music]