Metrics - The Danger of Data-Driven Game Design - Extra Credits

All right, I know GDC was like a month ago for you guys, but when we actually wrote this episode, we had just come back home from it. As many of you probably know, GDC or the Game Developers Conference is probably the biggest industry facing conference of the year. This is where the whole game industry gets together to talk shop and figure out how to best approach the year to come. So, this week I thought we'd talk about one of the major themes at GDC this year, a trend which is changing our industry, metrics. This year everyone seemed to be talking about metrics and finally taking a scientific approach to AAA games. For those of you who aren't familiar with the term, metrics, also known as telemetry, both coming from the same Greek root metron, meaning to measure, are simply the tools we use to measure player behavior in our games. Most games these days contain hooks in the code to send data back to the game developer whenever the player performs specific actions. It can be as simple an action as pressing the start button or as complex as completing a specific quest with specific gear on a

specific day. Often we can measure how long you play and where you quit. For microtransaction games, we can measure when you buy and dredge up the actions you did right before to understand why. In sports games, we can track what teams are popular. In first-person shooters, what weapons. Any good shooter studio can tell you exactly how many people have been killed online with any weapon at any given time and then give you how that weapon compares to the other weapons in kill ratio. I could go on, but you get the idea. Basically, today if you're connected to a server, any developer worth their salt can learn something about the way you play if they care to put the time into it. We're getting better at thinking about what we want to measure from the outset when developing games and we're improving our tools for examining and relating this data, which means that designers can now look at real player data on a massive scale rather than just conjecturing in an office about what the players might be doing. Now, in many ways this is great. It lets developers tweak and improve their games long after launch. It's led to the culture of patching and updating games to increase their life cycle. It's allowed us to really dig into what features resonate most with the player base so we can focus on them when constructing sequels. But that said, sitting there this year listening to everybody talk about metrics, we started to see a downside. A downside

which I think could turn into a tragic mistake. Now, this is a bit of a generalization, but many of you may know that social games are often metrics-driven. This means that they simply launch a product at what they consider to be its minimum viable state and then watch the numbers, polishing it according to what the great reams of incoming data say. Many of you are also probably aware that this year, in terms of dollar-for-dollar profitability, the AAA industry got their asses handed to them by the social games crowd. This left a lot of people, especially on the business side of the AAA world, looking towards social games and saying, "Well, what did we do wrong?" And the answer they seem to have latched onto is metrics-driven design. The problem with metrics is that they can only look backwards. You can only take data about something that already exists. You know how the world of social games is so permeated with clones and knockoffs? This is why. Even though social games are far, far cheaper to make than AAA titles, somehow there's almost a greater degree of stagnation in the social games world than in the big budget side of the industry. Why? Because you can't use metrics to innovate. There's no way of measuring the response to something that no one's made or experienced yet. Instead, social games tend to find

things that already work and then try to use player feedback to polish them until, hopefully, they outshine the original. And that's not entirely a bad thing, don't get me wrong, but as an industry-wide trend, it is kind of destructive. Video games have always been a marriage of art and science. And now we're looking to use metrics not to place them firmly as a science, but rather to relegate them to a manufactured good. Data in to code out. And once this culture gets established in a company, it's very hard to turn around. Say you're a designer and you say, "Guys, awesome game idea. All right, a game where we give the player a gun that shoots two interconnected wormholes that let the player traverse space in interesting ways, with like a funny psychotic computer antagonizing him the entire time." And then someone responds, "All right, huh, but do you have the data to back that up?" It's very hard to argue with that. Soon, everything's safe and by the numbers, and your big company just has too many interlocking parts that depend on those numbers to disentangle your ideas from them. Now, hopefully, this will change as even the large social games companies are coming to realize that while metrics are a fantastic tool for understanding your player base, metrics alone don't create fun long-lasting games. It's also

important to remember that metrics are being used not only to shape games, but to guide advertising and hone brand image as well. But like everything else, metrics are only as good as the people building them. Very often games or brands have turned themselves into inaccessible niche products by listening only to the people currently using them or relying on a poorly constructed set of questions used to track your data. Now don't get me wrong, metrics aren't bad. They're actually very useful, but they aren't the answer either. Not alone anyway. There's no silver bullet to how to make the AAA industry compete with all the emerging forms of games and if we cling to some single nostrum as a cure-all, as the answer to all the woes of the industry today, we're not going to get anywhere. Thanks for watching. See you next week.