Design by the Numbers: Using Data for Good

foreign [Music] I'm a lead game designer at scopely uh mobile game developer and publisher and I'm here today to talk about data so analytics can get kind of a bad rap among game designers in the mobile industry it's the realm of product managers and analysts to a lot of people not designers trying to make the game fun but I have a confession to make I love data I secretly have an engineering background as well as a product one so I kind of come at this from a different angle than a lot of people and I've been talking about system design and mobile for a while now and what I want to see is more game designers embracing what we can learn from this kind of analytics I think that data can be extraordinarily useful in not only making games but in making games better so that's what I'm here to talk about today and I turned off the clicker so everyone

knows the value of play testing right and you've got a game and you're sure players are going to act in a particular way and then you throw real people at it and no one can find the right buttons people use everything backwards and it's Madness usability testing is now an important and kind of accepted part of development process and what analytics does is allows designers to do that on a mass level so every game designer has intuition about what players will do but there's nothing like actual Behavior to tell you what's really going on what levels do players get stuck on what are their favorite characters to use things like that and the best way to get answers to those questions is to see what players are actually doing there's a lot to learn from these methods that you just can't get from any other source whether that's a direct answer to something like what weapon is more popular or more effective or importantly just using data Trends to find uh red flags that indicate where

you should spend more of your time investigating either way it's vital information that you're just ignoring if you don't take advantage of it as a designer so now that everyone is Thoroughly convinced in order to demonstrate these principles I'm going to be using some examples from what I've actually done before and for that I'll be using my and scopely's latest game The Looney Tunes World Of Mayhem so this is an RPG battler where you collect characters level them up and battle other players and PVE you know enemies alike there's a ton of different materials that you need to tune up your characters and unlock better skills that's actually about all you need to know to follow along today but you know the game is a whole lot of fun so you should really all go try it as a bit of bookkeeping some of the specific numbers have been changed and the slides have been posted online so you don't have to draw this in your notes you can get it after the talk I'd also like to thank all the actual analysts who pull this out of the database for me

thank you and with that out of the way let's dig in so the first step to actually using data is acquiring it and the best way to do that in your own games is to track everything yes everything we're focusing on the design applications here so I'm not going to go into the nitty-gritty of actually programming all the logging but basically you want to track every time a player does anything in your game and record it in your database then most of what you'll use is going to be aggregated data what do most players do what's the average of some measurement things like that and we're going to talk about the three most important things to keep track of as we go forward and the simplest way to summarize everything is Taps we're going to be tracking where people actually tap in your game or clicks if you're not actually a mobile designer and sort of snuck in here anyway that's okay um at a base level anytime the player Taps anything in your game you want to track it you may not end up needing all of that like every motion of the

character in an adventure game or every slide in a match three but you want to make sure you're tracking what players are interacting with going one level deeper we have our Impressions which are the screens the players open the activities what players are actively choosing to do and the resources every time a player earns or spends anything it's actually a common mistake to just track premium currencies this way but you want to keep tabs on your entire economy you want to know where players are getting their stuff and where they spend it so if you're used to talking about analytics you're going to be familiar with all these terms but for those of you that aren't here's a very quick overview key metrics are the end results of pro of player Behavior not their specific actions so we're not only tracking player behavior for its own sake but we want to relate everything back to one of the game's key metrics at the end of day it's kind of the business reason we do the things we do to make the game better

the two most basic things we're tracking are retention and monetization where retention is how many players are still playing your game on day one day 7 30 Etc and conversion is sort of your percentage of players that are spending and from there you can track a field of acronyms of Revenue per user per daily active user so on but everything's actually summing up to LTV you're a lifetime value sort of the what you get from each player and this is definitely focused on the free to play aspect but if your premium game we're still tracking the player engagement and behavior metrics the same way no matter what this is the starting point so let's sort of go on to the really interesting bits the first thing to look at in regards to player behavior is engagement if retained players are the ones actually opening your game engage players are the ones doing something in it as designers we success we succeed when people you know play our game in this case we're going to look at how players interact with the game starting at the highest

level and working our way down this is when we start building questions to ask when we're looking at the data later first examine every screen the player opens where are they going in your game how are they navigating through it uh level select a store a help menu whatever it actually is after that how do players engage with the features in your game we want to track when they start a battle craft an item whatever it is for us in our game that's every time they do any kind of battle and whenever they upgrade or interact with their characters we actually want to get super granular with these actions which battles are they starting what's the team they're using who are they fighting what are they upgrading from what level to what level how often do they change their teams around you want to track all of those little details because when players make a decision you want to know what they're doing that all rolls up to tell you what players are actually doing in your game so we often want to see how players are

using the things we've built and right and how that impacts behavior and from there we can learn a lot especially when you start combining these things together if eighty percent of your players open the weapon Forge and only 20 of them are upgrading weapons you've got sort of a big Gulf between these two things to look at so here's a pretty clear example from early on and on games beta testing it's also probably the simplest chart you're going to see in any GDC this year we're looking at the percentage of players that are actually spending money in our game which is something that always comes up as something to be improved in Mobile we noticed the trend when we compared it to a particular action in this case viewing our store players who are shown to store more frequently are more likely to spend in the game that may sound obvious but it was the exact details of it that were interesting and how much it increased and so as a designer the question was how can I get people to view the store more frequently that is actually beneficial to them I don't just want to try to sell people more things

constantly so in our case we added a free prize to the game that could be claimed every three hours from the store it drove people to view it they could see what was available make their own decisions and it led to higher conversion by giving them free stuff constantly it was a pretty meaningful impact for the product that came directly from our Analytics Beyond looking at what players do you want to look at what they accomplish are players completing the things you set out for them to do goals are just the milestones in your game that you set for players these are usually visible in the game like beating a boss level or completing a literal achievement but they can be behind the scenes and be anything you need to track if you have any kind of level structure in your game at all that's definitely something you want to look at look for when they're completing a particular chapter beating a puzzle a fight whatever it is and importantly don't just look at first time completions you

want to keep an eye on healthy player Behavior every day these might be literal daily goals again or they could just be benchmarks that you want to see from players every day you know going through every part of your core Loop for example importantly you're often looking for what players aren't doing even more than what they are so here's an example from the daily quests in our game early on WE implemented Quests for players to do every day that take them through every part of our game battling other players upgrading their characters engaging with the city builders and so on and there was a large bonus for beating all of them thus purely a retention feature right we want players to be rewarded I want them to earn that every day and come back now this graph shows the completion rates of each of those over a period of about three weeks we saw that we had under 10 percent of players actually claiming that big bonus reward and that was a problem looking at the chart it was

obvious that two quests in particular were lower than everything else uh we had one for opening crates the sort of treasure chests of our game that you get after battles and one for playing the PVE battles that we call Invasion the crates was simply the effort you couldn't actually complete it in the first session you would go do a battle put them in your bank and you'd have to come back later and there was a lot of effort and we already had a daily goal that tracked the battles that gave you them in the first place so we simply cut it out it was redundant and the other battle goal was doing fine the PVE one was the interesting part where it led me to basically pitch a feature that let you auto play Odd battles you'd mastered before and here's what it looks like after we did that there was drastic improvement in players claiming all of the daily quests which is exactly the behavior that I wanted we were able to determine the need for that autoplay feature by looking at this that it was so much more of a burden than the

other tasks but players wanted to complete goals so you can often learn which parts of your game need the most improvement from these seemingly unrelated investigations there's usually something to learn whenever player behavior is lacking in one area compared to everything else in your game if something is getting less attention or people aren't completing it there's something wrong of the experience or the difficulty or the messaging or some aspect of it when you're looking at player behavior and tracking what you need to you've got these questions to start asking one of the biggest ones is what kinds of behavior lead to better players and that's where we tie into those key metrics from earlier a better player is one that is doing well on any of those things that's sticking around they have longer sessions more play time they're converting whatever it is we ask what is this player doing differently and is there something we can do to get every player to act like that and when you dig deeper is there anything that players are doing that you

didn't expect and what happens when they do so let's look at a couple more cases that show these sort of Trends again from when we were launching this is how many players came back to the game on the second day compared to how many battles they did on the first it increases with activity as you might expect but the super interesting thing was that you hit this relative plateau in the data as a designer this gives me the minimum number to hit if we can get players to that threshold they are twice as likely to come back the next day as players that did practically no battles at all so we changed the early game to really reward hitting that number the quests you get early the amount of energy you get where the rewards are for beating parts of the campaign those were the tools to try to just change player Behavior to get them up to that threshold you know give them time to realize what they like about your game and keep playing it on the other hand sometimes the

behaviors actually Peak this was an unexpected result we got early on energy in our game is used for those PVE battles and we compared that to the amount of characters that a player was earning in that first day from all the various game modes and rewards as players start to earn more characters they start spending more energy maybe they're having fun they want to try out the new toys whatever their motivations but there was a point when engagement with the campaign actually decreased they were probably overwhelmed by the amount of choice or they felt like they didn't need to earn anything new and it was critical that we paced our early game to try to hit that Peak as often as possible normalizing the early drop rates giving things from first time completions not repeats and things like that my goal was to get as many players into that sweet spot as we could and it worked really really well now that we're sort of moving past engagement the next thing and my personal favorite is tracking resources the base setup for this is a log of

every transaction for each player from here You observe all kinds of things for each material the average earned and spent per day whether it's over the whole population or just a segment such as new players or anyone over level 30. you can get the average carrying balance from a resource from day to day and the real trick is when you compare these resources to how much players need need you can derive from content demands and your desired pace of content if players have a level 7 unit and you want to get them to eight the resources needed to get to that level is exactly what you want to track here this could be materials currencies experience whatever it is and make sure you compare this to relevant player Behavior such as the characters they actually use instead of everything they own at all trying to keep this it in the mind at once is important but complicated so we've got a special complicated view for it that's the Christmas tree board

there's a bit to explain here this is a list of all the materials in our game that are earned and how much are used broken down by player level here the Green from the good point of the product means that not earning as much as they would like to and red means they're likely to uh you know they're earning as much as they're actually using so if you had a bunch of rank 5 characters in The Missing anvils to get to ring six then anvils would be green here so certain materials in this are never really in short supply you might have to work to get them like dynamite but you always have enough if you actually want it other ones like our fine materials are always in high demand giving players something to work for and others go through these really neat transitions where they're hard to acquire early and they're essentially free later when other materials have taken their place and you can catch up this view tells you what players have compared to what they need and it's almost exactly how I intended it as a

designer where there's always something important and players can still catch up on materials later and I had a lot of versions that didn't look how we wanted that we use while building it until it started to look like this but importantly every resource in your game should have a purpose and Views like this are good ways to make sure it's actually happening building dashboards like that is actually very helpful running ad hoc queries and throwing them into Excel to make a chart is good for a one-time question but what all the best mobile companies are doing is logging into those tables and Views every single day you get that high level view at a glance that you can check to see how it's changing checking them just becomes part of your routine and you can look for a sudden and unexpected changes a drop or spike in someone's material balance that they're carrying is a bad sign it could mean there's a bug that's live or there's an unexpected change in behavior and it's one of those red flags that leads you to looking at the parts of your game in more detail trying to

figure out what's happened slower changes on the other hand can be natural patterns of players learning the optimal paths in your game or The Meta changing and player shifting but the use of these dashboards is to discover problems before players even realize they exist in our case we had a particular unique material that was just always lower in drops compared to everything else and it turned out that it wasn't entirely my fault and there was a misconfiguration in a drop rate but we were able to find it and fix it before our community realized and getting ahead of that is always very good so that other view is great for tracking shortages and surpluses as players mature but sometimes you want to look at one particular inflow and outflow over time as I was saying so this is a view of a single material where each color is a different source of sync players for this one were earning a little bit every day from normal gameplay and then they had a periodic inflow from a daily challenge a sort of rotating Game

feature but players didn't spend the material on the same patterns so they were just always acquiring it and always had it on hand before they actually needed it after looking at this we reduced the inflow especially the size of the spikes and all of a sudden we had the desired Behavior players were making sure to log in on the right days to get the material when it was available they still had everything they needed it didn't actually change the economy or try to sell it to them but all of a sudden it was valuable it was something they cared about instead of forgettable you want to have those critical path resources and you can create one if there isn't one already in terms of the actual content in your game you're basically tracking how players actually use all the stuff in your game right in our game and many mobile titles characters or cards or units all what we're tracking here and there's usually three things that we are looking at ownership usage and win rates of the entire player base or a just us

segment again what percentage of players actually own the content out of them how many people are actively using it how many people win or succeed due to a task when they're using them tracking this kind of data can be the one of the most important tasks that a systems designer is doing on a mobile game so let's look at how I'm using this in practice this here is a bubble chart example with all three usage and win rates on the axes and ownership is the size of the circle it's kind of the dashboard of the character meta that's going on in the game and I start every day by looking at it this example is characters but as we said it would work just as well with cards or whatever else and what you're often looking for are the groupings and the extremes and how they can go into categories of which I've got sort of four of them first you have your stars these are your high performers of your game they're characters that are used frequently and win frequently and they'll make up sort of the meta of your game for the most

part having these is great you just want to make sure they aren't so good they're obsoleting the rest of your content for us Bugs Bunny is an epic character that you start the game with so he's always kind of an outlier anything that's up here just keep an eye out for them being too good and if they are consider introducing a counter to them late you sort of know what players are going to be using a lot and you want to counter it or God forbid Nerf it if we ever have to the next group are sleepers these are your characters that win a lot but aren't used as much and context is super important here some of these are Niche options that are only used sometimes in certain strategies but Excel when they are sometimes if you see a super low ownership rate they're just hard to acquire both of those kind of characters are probably great at doing exactly what they need to do in your game but if those aren't the cases then players may not understand how good the character is where to use it how to use it properly things like that which you want to try

to fix with sort of events and messaging your third category are underperformers these are characters that aren't very good and your players know that some of these characters might be decent and just waiting for that time to shine such as a specific event or the right team to be released but often they're just not good enough and this is where you're looking to buff the characters in your game not every character in a game needs to be amazing but none of them should be this bad I'm sorry Elmer the final category are fan favorites characters that are getting a fair amount of use but are still performing below average in this case we have Ralph the wolf who is not Wiley Coyote there's a difference um they're super interesting because people are still using them despite them not actually being all that great it might be aesthetic preference so the character or the weapon could be really fun to use either way this is your chance to learn what players actually like and make more of it and while

you're at it it doesn't hurt to make these options that are popular a little more competitive you don't have to make them all stars people already like them but it's nice to give the people what they want once in a while as a game designer just like with your economy having those dashboards that you can check regularly and see how they're changing can give you enormous insights you are looking for a sudden changes again and is a character suddenly winning much more or less than it was is there a spike in usage are there characters that are too strong or too weak for your costing of them in the game or their availability the best character in the game being an easy to earn common or if a super horrid Chase prize that takes months of commitment is kind of bad those are both bad situations for your game that you want to be able to spot in the data a subtle point but one not to miss is to look for how Trends will impact the other parts of your game if a rare fire monster is suddenly On The Rise as players discover a new team that means

players are going to be needing rare fire materials later on they're gonna want the counter to that monster if you can get ahead of those needs by looking at data Trends you're going to look like a genius in your game so now that you're tracking data what does good actually look like and here are some more you know best practices from what it does when you're doing it right first you want to see that players are actually playing the game that's both steady retention as well as people actually using every feature in your game you want to see healthy session times per day and everything should kind of get better as players age that's usually a consequence of less invested players churning out but if that's not true and you're seeing a downward slope then that's kind of a bad sign you also want to make sure that no one part of your game is neglected by players there's always going to be things that people interact with more or less than others but a healthy game is driving players through every piece it has to offer check that there isn't

something that players ignore because it's not really worth their time especially Elder players you also want to make sure that everything in the game is actually purchased or used that means core game economic items and as well as things that consume currencies premium or otherwise like before it's fine for some things to be Niche but if no one is buying a particular item that's a sign that something is off specifically be on the lookout for repeat purchases if someone buys something like a gotcha item or store SKU and then they buy it again that's a sign that they're really liking what they're getting and it's matching what you're selling it and you're merchandising and it's a good way of checking to see the real value when you're looking at the economy the last thing you want is for everyone to feel like they have everything they need it's the fight for limited resources that makes a game economy work and don't forget that as a designer you control both the economy outflow as well as what players needs are that's recipes as well as creating the obstacles that they have

to beat make sure you're using both parts but the Baseline here is to Source most resources just above what players need as your starting Place players we said need to always lack something so you can create pinches a bottlenecks in your economy those resources that players just don't get enough of you know fewer is better than many here if you have no pieces of a recipe to craft an item it makes it feel like you're not getting anywhere at all but if you're just missing one or two key items now you've got something to go after those economic dashboards are key to making sure that they're lacking what you want and have enough of everything else and if they're lacking something you're not trying to constrain you can drop it more freely and increase the availability but when they are lacking the materials you intend that's good limited and rare things create excitement that gives you a resource that you can give out as a reward for difficult things as something to grind over time or something to sell in your store ideally all three since the best monetization methods sell time

finally when it comes to balance you want to check the players actual experiences match what you expect them to be no one anything in your game should be much more effective than other things at the same cost if one comment is twice as good as another that's earned at the same time one of the two of them might have a problem we'll always make some characters that are intended for our specific strategies but you know it's a good rule of thumb and please don't just make anything powerful expensive players are so alert now to things that feel pay to win whether they actually are or not it's okay to have the occasional strong common as long as it's not dominating everything a powerful epic that can be found through luck or a sufficient time can be just as strong as one that comes out of a gotcha on events since you have to set the Baseline for what's strong somehow in short data is our friend monitor how each feature and piece of content in your game is used if only 20 of players

engage with a feature it's a good sign you have some work to do before it's perfect track your resource inflow and outflow creating dashboards to show you those unexpected changes shortages and surpluses create the demand for materials in your game and provide the supply accordingly and once you know what players need and don't have you can reward it or sell it that's actually the grand secret of monetization strategy and RPGs in a sentence so I hope that helps um I hope this talk helped some people out there uh we're a little low on time so I won't do questions here but I'll be out in the breakout area just outside um and if there's anything that you don't have time for please feel free to contact me anywhere I will literally always talk about game design and the slides in this talk can be found online as well as a reminder please fill out the session evaluations that you should have just received they're important to everyone running the conference and they are sort of ironically the data analytics of this conference so I would

feel really bad if you don't uh with that being said I hope you enjoy the rest of your week [Applause]