Data-Driven or Data-Blinded? Uses and Abuses of Analytics in Games

[Music] today's talk is that I'm is data-driven or data blinded the uses and abuses of analytics and games and I'm very excited to see this many people here very tired on a Friday to hear me rant about statistical significance because we're gonna get to that eventually but just to start with I want to say that data is awesome it is the backbone of science and progress and most of the large companies Google Google Amazon Facebook and most scientific undertaking but the thing that I want to focus on today is that data is also a hot mess data is a tool and like any tool it has uses and abuses and it also is something just that where it's very easy to make

straight-up errors how many conflicting health studies have you seen in the last 20 years I mean I'm at the point where if I see whatever I see from a like a nutrition test study in a journal I'm like whatever I'm just gonna eat green vegetables and whole grains and I'm sure that's the right answer and that's because of some of the problems with data you know Congress is the company that uses a lot of data and we have from our start but I never am completely sure whether we are this ship that is using data to chart a course off to success and treasure or where this ship and we're steering into a pile of rocks and disaster and the reason is because both have happened and both have happened a lot so Who am I to talk I'm Emily Greer and I am the co-founder and CEO of congregate and I am a shameless data geek I even put it on my Twitter profile

and I spend you know four five six hours a day and meetings but I always reserve some time to go into dashboards and read PM reports and data analysts at concrete know that they are always going to get questions from me on anything interesting that they write so I want to talk a little bit about the type of the sources of our data because that's gonna be relevant to this talk as I'm using examples throughout we have week on great started as an open platform for browser games concrete comm and we have averaged more than 1,000 games uploaded to congregate per month for the last decade plus so we are closing in on a hundred and twenty thousand games and that is a lot of data um back starting back in 2013 we launched a program where we do third-party publishing on mobile you may be familiar with some of our titles particularly adventure capitalists or

animation Throwdown but we've typed but we've published more than 40 titles on mobile and in the last year started publishing on steam consoles and doing premium titles as well and you may have seen the announcement that two weeks ago that coming this summer we are going to be launching a new PC platform called cartridge but that's not really the source of data or focus of this talk so I want to talk a little bit about how I got here and it's not because I studied econ and math and other things like that in school I actually studied Russian in the Eastern European studies so if you want to know about medieval Transylvania or the Ottoman invasion of Hungary I'm your woman but as you might imagine this was maybe the most useless knowledge that I could have on the job market and so I immediately did something else I went into book publishing but I didn't didn't like that very much and eventually I stumbled into data in the catalog business now i what i was

doing was called direct marketing but really it's with data science by the way that we we know it and I just fell in love with being able to test things and look at numbers and find the story in it and at that point because I love data so much I taught myself sequel because I hated waiting for IT to get me data and then I started taking classes at Community College and at UC Berkeley here on probability and statistics and a little extra math so that I would understand the theory of what I was doing a little more than than just doing it in practice after a while I got a little bored and I almost went to grad school and econ but then ended up co-founding concreate instead thinking that I was going to be doing something completely different but then wow we have all the data and I actually do more data and data science and it's become what I did before became really really really relevant and the reason why I wanted to talk about that is because I want it to make my first point

which is don't be intimidated you don't need an advanced degree to get statistics in statistics to get data right and just because you have one doesn't mean that you won't get it wrong I'm not saying that it doesn't help it does help but it it but for an organization to do data right it can't be just something that's thrown over the wall from quants it's something that needs to be that you need to have an intimate knowledge of the game and how games are developed and it needs to be something that everybody works on and everybody looks at and everybody participates in for for a company to do it right the other main other first point I want to make or a second point I want to make is that you should get paranoid and that is because under an apparently solid surface data is usually a pile of sometimes the issues are obvious when we first started launching games on mobile we had an endless-runner that wasn't

verifying receipts from from Apple and so all sorts of fraudulent purchases were coming through and so the average purchase was looking like five hundred dollars and that was immediately obvious like this is wrong oh and and we're able to isolate it and get rid of it but a lot of times data issues are much more subtle they're different things not firing in the tutorial sequence or weird client timestamps that are changing dates and others little subtle things so one of the things that I think is really important is that you should never rely on a single analytic system and you should never on an analytic system where you can't get in and inspect the individual records because oftentimes you need to see the individual little items to find problems and figure out what they are even when your data is good and when the data is correct it can be misleading this looks like a for like four separate pictures photos photoshopped together to

create an appealing color grid right wrong and data is like this it a lot of times something appears one way very straight forward when you look at it in one direction but if they change the angle you can see that it's really something completely different here's an example of that happening from from congregate comdata so these two these are two games in the same genre they have similar day 1 and day 7 retention game one slightly better they have similar life time buyer percentage game - it's slightly higher you can see from this chart that their daily are pupu average revenue per paying user is also very similar since ARPU average revenue per user is just buyer conversion times are poopoo they should have the same numbers right but of course you know that I'm setting you up so game one has an RP poo sorry

an AR poo of 227 and game two has an RPO of only 84 cents even though those other numbers looked really similar and the reason is because people are playing game one much longer than game two and it has many more transactions per buyer so d30 is higher transactions provider and thus lifetime are poo poo is much higher and if you but you if you only looked at daily sickness statistics you wouldn't realize quite how different these games are so one way to think of yourself is as a detective the witnesses may be lying or confused the crime scene may have been tampered with you can't trust any one piece of evidence and you need to constantly cross-check them against each other to validate what you're finding out so we do a lot of triangulating in truth and we do it by taking every source of data that we have and comparing against each other almost all the time so we've got client data from the congregate SDK we compare it to

adjust which is our mobile attribution provider that's also firing in our client we compare it to our server data we compare it to data from the platforms and then we compare it to benchmarks that we've built up over the years because there's always some variance between sources of data sometimes it's 1% some things could be as much as 5% or and that could be normal and tolerable but you need to understand what that is so we track everything all the time this is from some dashboards that we have where we're comparing four different pieces of data next to each other and and tracking that and then we track the variance over time so that we have we have a ceiling and a floor that's acceptable and anything beyond that we look beyond we check into and make sure that it's correct and you're not just a detective your see your CSI your goal should be to create a three-dimensional view of your players and how they're moving through your game games are very

complex and time-based systems and a flat view is never going to be enough another way to think about this and this comes from I should really not press against my chest my favorite while I majored in Russian Eastern your European studies my favorite class in college was actually astrophysics for non-science majors and I think there's a lot to learn from astrophysics to help us think about how we should be building up data and data for games so if you think about the universe we're sitting in one spot on on planet Earth and that's a really limited view and we have really limited tools to understand what this enormous thing is doing and for years millennia we didn't really understand it but gradually through first our visual observations with our eyes and then by building more and more tools and taking more and more observations from different places and then comparing those observations against each other

we've been able to build up a really beautiful and impressive model of the a 3d model and how everything works together is everything accurate yet it isn't it absolutely isn't but when you think about you know 2,000 years ago when we thought that the the Sun revolved around the earth it's really amazing how far we've been able to come want to talk about a lot of mistakes there's a series of classic data mistakes that almost everybody makes I make them even though I fight against them so I wanted to talk and take everybody through some examples of how they happen and how to what to watch out for and the first one and the first thing that you should always be thinking about whenever you look at numbers is audience mix so we tend to think of our player bases as sort of monolithic you know one individual millions of times but that's not really the way to think

about it it's really an aggregation of all kinds of kinds of subgroups of people from different demographics who have been in the game different amount of times who came from different sources who are playing on different devices and how those subgroups mingle and how they mix have a huge effect on our KPIs usually when KPIs for a game change it's more often because the mix of the audience has changed then the game has at itself has actually changed here's an example from one of our games it's the last 90 days or 180 days 180 days of retention for the same game you notice that it's mostly pretty steady but at the beginning there's a spike pretty high on d1 retention this is because we did we were experimenting with how chart position affects organics so we did just for a couple of days a really intense user acquisition burst only on us only

on iOS and these are the highest value customers and the highest value players that we could bring in and so they really spiked d-1 retention and if you look over time you can see that really valuable cohort move through the system so d7d 39 that you can it's immediately obvious where that one cohort is on the other hand you can see down here there's a really big dip and that's because we got a very nice feature from Google Play that included a lot of countries where the game wasn't localized where the game wasn't culturally appropriate and the retention for those players was really bad and again you can see them progress through the game d7d 30 D 90 affecting the metrics now this is a really dramatic example of high quality players versus low quality players and most audience mix issues are much more subtle but they're there and they're happening

all the time another example of this happening has to do with audience age this is from this is a chart an RBI chart from another game of ours pocket politics that has where you can see this hugely dramatic ARP dau decline since Christmas it's was 75 percent decline from around Christmas through the end of January and then a slight recovery thereafter and it would be very easy to look at this chart and think oh my god something disastrous has happened in the game why aren't people buying anymore but that's not really what's happening and you can see it from this chart we this game has been out for two years and we've been doing a very low level of user acquisition for for the last year or so because of fatigue related to the election but we were testing new creatives and we got a particularly awesome video creative that has aged trump-like character pressing a nuclear button again and again and again and

apparently people found that really funny and it drastically reduced the CPI that we needed to spend to bring users in and we were able to scale up our user acquisition really high so given the last slide you're gonna think okay so all of these new players are lower quality than the ones that were coming into the game before right but if you look at if you break your are the ARP tout chart out by d0 d1 and look at it over the same time period you don't see that drastic decline there's some ups and downs but it's relatively flat and in the same range similarly if you look at d2 to c7 there's a small decline but it's not huge much less than 75% d8 through D 30 it's a little more dramatic you can kind of see what's going on but D 30 again there's not much of a dramatic change so why was there this seventy five percent drop in ARP Dow and that was a player mix issue so

you can see that the D 30-plus ARP Dow is 25 to 35 20 20 to 25 30 cents at any given time but the new player ARP Dow anybody younger it tends to be around 10 cents so when the elder players were the majority of the game the ARP Dow is much higher as we brought a lot of good quality new players in even though they were equally as good as the ones coming in before the overall average for the game changed and overall we're making more money and it's great but that's something that you should look for a lot it's very common to see these kinds of patterns with any game where the average revenue per daily active user increases over time and that's most games that have a lot of depth in monetization usually the way you experience it is you have a big rush of players coming in when you launch the game from features from user acquisition and then gradually they churn out and only a small core base is left and over that time you'll see your ARP Dow rising and people will

think oh we're killing it on live operations we're doing really good but actually you you may not have made any provements the player base has check for your game has changed another point I want to make is that averages are average a1 dominant subgroups good numbers bad numbers for a dominant subgroup can mask what's happening with smaller groups here's a break out of tutorial completion for a game that we have that had a really bad I think it was an iPod crash and when we looked at the top-line numbers everything looked fine because it was only you know 5% of the total but when we broke it out by device you could see that there was something really dramatic going with with the iPod and then we were able to identify it fix it and everything went back to the range of everything else another thing to keep in mind though is that players are not average as I'm you know you saw that very

different art out between elder players in pocket politics and newer players and there's a lot of things that you can do that have very different impact on older players versus new players so this data is the PvP win rates in a collectible card game that we have that has a where there's a very dramatic advantage if you for being the aggressor for being the person who plays first you see that little bump there that's when we made a change to try and mitigate how how much of an advantage that you got from playing your cards first and it looks like it didn't change one rights much but if you break it out where you look at the max level players in this case that's 35 versus everybody else you can see that the change that we made was really advantageous to the highest level players and much less advantageous to the lower level players so even though the average looked fine what was

actually happening beneath that was not what we wanted to happen at all now I've just spent 10 minutes talking about how important looking at subgroups and looking at different player bases but there's a there problems with that too specifically the more that you slice and dice your base especially if you have a smaller game you can get into situations where you're looking at data where the sample size is small and that data might as well be garbage it's so erratic and and and misleading so you know looking at Android Australia or just your linux users or you know some other combination may not make sense so there's always this balancing act between slicing and dicing your your your players enough so that you're really seeing all the subgroups and you seeing sort of what's happening underneath the surface versus getting down to sample sizes where what you're looking at is useless so you just need to keep keep an eye on it and the folks

that congregate know that I I'm always going to ask them if they don't put sample size on a cohort chart or something else and it just means that I'm going to ask them as soon as I see it I don't think I want to talk about is the normal curve because this from my perspective is a lie nothing is normal it's not quite true and there are some things that fall to a normal curve but almost everything that's important in games revenue sessions battles anything else is a power curve distribution especially in free-to-play games but really all games are driven by intense activity from outliers and their presence or absence and any piece of data that you look at has dramatic impacts on what your what you're doing so here's an example from one of our games which is the lifetime spend of everybody who's spent within the last 20 days 28 days so they it's happened over

a year but it's people who have spent recently and you can see that the top spenders for this game are 45 have spent 40,000 45,000 30,000 and you can imagine especially in smaller sample sizes if somebody is present or somebody is absent that has a huge huge impact and one of the things to take from this is that any metric that you look at that includes something where one of one side of it is a power curve like revenue like sessions but the averages from those kinds of metrics are going to be much less stable and much more up-and-down than something like retention or tutorial completion which are based on binary yes-no yes-no metrics and you can look at those with smaller samples so in test markets I look where we we nearly always have small sample sizes I always look first at you know early retention and buyer conversion because those are more reliable than things like our poopoo

sorry another thing I want to talk about is cherry-picking this is something that sometimes we do consciously because we want to make a point with a boss or someone else but we also tend to do it unconsciously whenever I look at a group of cohorts or I look at a data the extreme positive values and the extreme net negative values always make a bigger impression on me than anything else so I'm always sort of fighting to make myself like check the average check the media and check the mean to keep that in check another example of where that this could bite you is say you've said in you know in a personal goal or an individual goal of we want you know I want this thing to hit this metric so say you think the retention of a game should be at least 50% if you're looking at the data and it hits 50% once you may remember oh that 50% and then ignore the 48 and the 47 and the 46 that come after that and think of your game as having

50% d1 retention and that's just a human bias that you have to that you have to fight and have to mentally fight and along with that come is in a similar vein is confirmation bias part of doing analytics building world's thinking through your games is having theories about how your players are behaving how things are working a lot of times you'll go into data and analytics looking to confirm a ticket particular theory if you're looking to confirm a particular theory you're probably going to find the data that confirms it and you may ignore the data that contradicts it or mud is that especially if you only look at data when you have a theory or a particular type of data when you have a theory you're likely to unintentionally unintentionally buy or to yourself when you're looking at it another thing to keep keep an eye on is the visualization of data because the visualization of data has a huge impact on how we perceive it here's an example of a

couple of charts that are our marketing team did for a partner that we were working on we were working with this is a game where ad revenue is very important and we keep very track careful track of what percentage of the daily active users are engaging with ads so you can see here that there was decline in the amount and the percentage of users who were engaging with ads and the team was concerned about it and put big red arrows on the analysis to show you know what the the rate of decline was but notice that these charts start at like 40 and 52 and I really really zoomed in on that data if you look at the exact same data but start at zero you can see that the amount of change in the engagement was actually pretty small and it probably wasn't that big a deal and it may have been just from audience mix I don't actually know what the answer is here but this shows you how you zoom in how you look at how you

create your axis how you perceive the data sorry how you present the data is going to have a big impact on on how you react to it the the other thing I want to talk about is correlation and causation now we've all heard a thousand times that coral is correlation and causation are not the same thing we keep hearing it because we keep needing to hear it I should be like a personal mantra every single day um you know an example of correlation and and and cuz I should not being the same would be you know ice cream and drowning deaths they both go up in hot weather but it's because both happen more in hot weather and they're not correlated the ice cream kills headline would be a really stupid headline but you could look at the data and and and and come to that conclusion and the other thing the reason why it's so hard for us is because as humans we are wired to look for meaning and we're little wired to

look for cause and that's why we didn't eat the poison berries the second time it's both - it's a superpower but the thing to keep in mind is that it's also a course in particular in games the curse that we need to watch out for is that absolutely everything correlates with engagement and sometimes that's meaningful but usually it isn't a classic example that I've seen is you know people saying players who connect to Facebook are twice as valuable as as everybody else so we should invent incentivize Facebook Facebook connections because then they'll be better maybe that's true but more likely it's that engaged players were the ones who were willing to take the risk that they're gonna spam their friends and and and make the connection so the true way to separate causation and correlation or correlation causation figure it out figure out what's really happening is a B tests and those are

really powerful um but it's not a magic bullet because nothing nothing in data is that easy testing has real costs engineering time overhead complexity confusing players divisions between your players and the more that your tests that you're running the worse that gets in general there that the test traps within a B tests are a lot of the things that I've been talking about like sample sizes audience mix cherry-picking confirmation bias power distributions but there are a few additional test traps that you need to watch out for mostly having to do with assignment so one thing when you're doing test assignments to keep in mind is that it's important to make sure that you split your test on what you're actually testing on so example here is a test in the store one way to assign people to the test is whenever they show up in the app they go in Group A or

Group B and then later maybe they will engage with it engage with the store and you'll you'll you'll see the results the problem with that is that you could very easily end up with a unequal distribution of people who interact with the store and therefore your test is biased that has something to do has buy something that has nothing to do with the thing you're actually testing the other thing is that if you're looking at multiple metrics like retention or anything like that then the noise from the people involved in the test may be drowned out by the larger sample so always always let once people have engaged with whatever you're testing um the even bigger assignment problem in tests to do with time and any anything we do to adjust when people are assigned to test so revenue is a is a power distribution so is engagement your best players always show up first they always show up earliest they don't show up off

honest and so if you only run a test for one day then your sample is gonna be heavily biased towards your best players if you run it for longer then you'll get a lot of less engaged players who come back and you'll get a more normal sample in terms of what your player base looks like similarly if you I've heard of people doing you know tests assigning people based on one day next day one day next day again you're gonna get a very biased sample that way now there's cases where you may want you may be forced into it and from like if you want to test a store on Steam or some other place that doesn't give you tools but even there you want to be very careful because people who show up on Tuesday may not be the same people who show up on Saturday so you want to do things that create an equal sample and an equal type of pre type of person interacting with a test the other thing to keep in another thing to keep in mind is downstream impacts a lot of commercial a

B test projects products will make you to make you pick just one one KPI that your your testing and that they're gonna you're supposed to just judge the tests on that particular metric and decide which is the winner they do that because they want to stop you from like fishing for a particular result and making sure you know and cherry-picking oh this was good so let's so a and I wanted B to win and this is good on B so B is the winner and that kind of fishing is bad but I think but what we've seen is that there's so many unintended consequences to different tests that it's always better to look at the full picture and look at everything that happened and then make a decision based on the full picture as to whether the test was a success or not so an example the screenshots you saw early of the store test a test that we ran recently on one of our games that is a very complicated

store with a lot of items and we had a theory that the number of items and how complicated it was was confusing players and reducing the number of people who were actually buying items so we did a simpler version of the store that focused on just a few items and tested them against each other so the main main thing that we were looking for was to see an increase in conversion and great we we increased it we increased the conversion with the simplified store by nine percent the problem is is that by removing some of the items where you remove some of the items that the most engaged players were spending on so even though we increased conversion the average revenue revenue per paying user dropped drastically and overall revenue was down 11% so conversion up revenue down oops that was maybe not the

greatest thing to have done so we were Verte into the control and are continuing to experiment along with not looking another flaw that you can get into with a/b tests is looking at results too early so here's another example from our game office space where we did some this is from a little while back where we did we were testing kind of a flash sale for individual items I think they popped in um and certain players they would pop in great deal short-term discount and you would expect it to increase conversion and reduce the are poopoo and hopefully increase revenue and when we looked at the tests after running it for 10 days that was exactly what we saw big increase in conversion small decrease in our poopoo 8% increase in average revenue per user yay but Oh since these are all the bad cases when we let the test run for 30 days and looked at the data again the basic

pattern of increased conversion and lower our pop who was still there but everything had moved downwards to the point where we were now at a negative on revenue revenue had decreased by 2% because the long-term effect on average revenue per paying user was so much larger than on conversion and part of the reason why that's important is because of statistical significance and the question of what's the biggest big enough sample size I have a particular point of view on this which is that you know you don't need you don't want just the minimum a number of players in a particular test you don't want just enough to pass it's just statistically significance test you want really big sample sizes and this comes from I think a from a misinterpretation that I think most people have on how statistical significance works and I'm gonna stumble over statistical significance a bunch of times because it's really hard to say so

and it's really hard to say fast so you'll hear hear things people will say things like we had a statistically significant 5 5 percent lift from this test but that's not how statistical significance works so the way statistical significant works significance works is it assumes that there's a true underlying lift between the two populations and that if you run the test again and again and again the results will come back in a in a normal curve of results where the true mean is the the underlying lift so if you have if you run a test it passes statistical significance it's 5% it could be from around the mean it's probably 4045 percent chance that it's in that range but it also could come from the other part of the curve in this case 5% the underlying lift would be something like 10% or it could come from the far upper

end of the curve in which case there might only be a 1% or 2% lift and that's important in terms of how you think about what statistical significance means when you're looking at results and this is why I really like large sample sizes because if you run if you have sort of double the sample size that you need then it's like you've run the test twice and average the results and over time that's going to become the results there are going to come closer and closer to that mean so big test sizes whenever possible but there are costs to that so there's always trade-offs now you know that I think this is a lie so I want to talk about abnormal testing because a lot of what we're looking at our engagement related metrics or revenue related metrics and that our power distributions and those need to be looked at differently so this is our old friend the power curve and of revenue with buyers who spent forty five fifty

thousand dollars he if you think about what that means here's a sample of a potential distribution in a game it's not uncommon in free-to-play games to have 70 or even 80 percent of revenue come from a very small percent of buyers who spent more than five hundred dollars these to a B tests have exactly the same distribution except for one has an $1,000 buyer and one has a $500 buyer and the net impact of that is in this small sample on a negative forty percent on the average revenue per user even the distribution isn't very significant that still happens when you have a larger test and that's something that you need to really need to look out for and watch out for because a few big spenders can have a very big impact how do you deal with that um it's really important to look at medians and other thing and other in any kind of distribution of how revenue and

other things are coming in so that previous example the average was drastically different but if you look at the medians they're both 15 and that helps show you as an analyst that the differences are happening just way at the top and not through the whole distribution another made-up example is this one where there's actually a big difference in the distribution everything is doubled on the second one but has a slightly lower buyer and so the averages are very similar but if you look at the the median it's double the other one and that shows that probably that's the better outcome of the test versus the other so that simple store test that I talked about before where you're looking at revenue and that was what we were judging on this chart helped us figure out what we what was going on so this chart breaks out all the different purchase sizes and does it by day and

when we looked at the same chart for the simple store version versus the control there was much less of that bright pink one hundred dollar purchases so we were able to sort of confirm by looking at the distribution what had really happened the other thing is that when you're looking at something like revenue revenue or you know some kind of economy sync instead of using a normal P or Z test you should use something called a wilcoxon rank-sum test which which compares the medians rather than the means um this isn't something that's easy to run by hand but most statistical packages like are include it as an option and that's what we use now I want to talk about testings dirty secret which is that most tests don't show a significant change and this can be really frustrating for a team because you've put a lot of work into creating two version different versions of the game and you want to have some you want to have the work that you're doing have meaning you want to have you want to be

improving things you want to have numbers go up but the one of the things to remember is that a test that doesn't show a result is still meaningful and still interesting here's an example from butter shop butterscotch and they get shenanigans who made the game crash-lands they were kind enough to share this particular example they wrote for the game a huge long very detailed description and we're wondering how much of an impact is that having on conversion so they decided using Google's AV testing tools and Android to test the most extreme possible version of that which was no description at all just there for accolades four or five accolade that we're talking about how awesome a game it is so they ran it for four months and it shows absolutely zero difference between having this like thousand word description and zero now on the one hand that can be frustrating all that work waste wasted on writing

description but in all game companies and all companies really a time and effort is really valuable so anything that shows you what is not worth doing and what is not making a big difference is something that you can reduce your focus on this result where description has very little impact is exactly what we've seen in terms of our testing using the a beat or a B toolset that that Google gives us which is really nice at the high end icon has a big impact video has some impact screenshots a little less in the description not too much and overall this kind of testing has been really useful for us we used to art spent a lot of time arguing about game names but then we did a lot of tests a B testing of game names against each other never saw a difference in result and now we're much much Chillar about choosing a game name and yep so so we can we can let some of those arguments die a little faster because we've had data that tells us this isn't that important but when

you're when you're testing it's also very important not to extrapolate too much from a result as you've heard we're big fans of Google's a/b testing store tools on Android we're such big fans they actually did a case study on our use of it and came to our offices and you know took a video of all of our a bunch of employees talking about particular tests here's our senior producer peter Eichmann's talking about an a/b take on tests we did for a game called global salt and we took we had been using this girl with a gun icon as our main one and tested a helicopter against it and got a 92% increase in conversions 92% this is a really big amount and had an impact of hundreds of thousands probably millions of installs on the game so really exciting great yay testing except where there's a funny thing the reason that we were using that girl

with a gun icon testing is because we had tested various different potential icons on Kongregate comm and that game that's the one that one so the helicopter said the girl with a gun had a point to eight percent conversion rate on Kongregate and a point one nine percent the helicopter had a point one nine percent so the results were almost exactly flipped and that's why you need to be really careful when you're testing to think about context and make sure that you repeat any test and that not everything is going to work the same place same way in the same place in different places another thing to keep in mind is that not everything is testable specifically late game stuff late game and late game players it's very can be very difficult to test this is a really nice quote from Daniel cook of spry spry Fox spry Fox and lost garden that I saw on Twitter recently he was talking about YouTube and how the

algorithm changes the audience but I think that it's a really good way of thinking of thinking about things particularly that the game also changes the audience the longer somebody is in the game you're setting their expectations you're training them to behave in a certain way and the later you get in a game they're not going to react to something the same way they might have on day one or day four and day eight players talk to each other that makes it harder to test as well so there's a lot of things happening at the end of a game that make it difficult to test so I've created because I like frameworks kind of my hierarchy of testing from critical to difficult and it basically follows the funnel of players into a game so starting with advertising there it's really critical and there's tons and tons of tools almost any app any ad network that you use is going to expect you to a be testing and this is important because the same creatives that work on Congress

calm versus Facebook versus a video ad network they're not people are not going to respond to them the same way so you wanted to have a ton of creatives that are constantly testing and this is very important to get advertising to work it's also really important wherever players are converting to actually play your game whether it's a landing page or a store page on a platform the problem with this is that outside of Andrew and Android there's not really great test tools but you can use things like store maven and others other types of services to sort of fake test those and whenever you have a platform that does allow you to test make sure that you do it it's still pretty easy to test on the initial experience of a player you know the getting through tutorials getting through the first day of a first day there's no real big gotchas in that but there is one thing that's important which is that though testing on the first day generally doesn't have huge

shows you show very strong results usually what you're looking for there is how long do players retain how deep do they get into the game how much do they play and in the end how fun your core loop is is going to determine those things and when you're testing the initial experience you're really just testing that presentation and the presentation of something has some impact but doesn't usually have as much impact as the game itself um where we've seen more meaningful testing on our side is in the first few weeks that somebody is in the game this where we've seen we've seen good luck with tests where we're testing out how we unlock features and good results on store testing and other things this is sort of a sweet spot for testing in a way that's meaningful in a game because players are still new enough to a game that that you haven't kind of set wired them in into a particular type of behavior but you it's deep enough that it can be meaningful and you can test really substantial

features but then once you get to late-game it's pretty difficult it's not impossible but you should tread carefully there's a lot of problems with payers talking to each other there's a lot of problems with sample sizes audience mix etc so it's not impossible but given how much effort testing is I wouldn't recommend focusing on that so besides that sort of there's a hierarchy of testing I also like to think that there's a life cycle for data and games and the tape it's and it kind of follows the stages of the game creation and the life cycle of creation of the game and that can be loosely split into two sides one is the creation concept in production production versus optimization and in my view there the optimization phase where you're trying to answer the questions of what's

working what's not working what's breaking it's really crucial for all games free-to-play premium whatever you're doing every game can be made better by looking at at data for any game as a service that LiveOps looking at data it's absolutely critical we'll only make games better as long as you don't as you don't pull into a lot of the classic blunders where things get more interesting and from my perspective and where I don't think I have it figured out at all is on the creation side now production I don't think there's really a role for data in it you just need to make the game you're not really answering questions you're just trying to to get to the fun but there's a really big when you're in in concept and pre-production you're asking a really big question which is what should we make and this is the question always for for game companies this is deciding the fate of your company again and again you're committing enormous amounts of resources you're potentially committing years to a project it's what's defining

your brand it's setting you up for both success and failure and this is the most important question that you can ask we've done a lot of you know being a data-driven company we've tried to pull data into some of that creation process and we've had luck with a particular series of games of collectible card games that we've made starting with Tyrone Unleashed which is a very hardcore gritty a sci-fi themed collectible card game that had really great monetization and long-term retention but very high CPI it was very hard to market so we looked at that game and said we want to make it more marketable so we reworked the game the team synapse in Chicago is the one who makes it and they did a brightly-coloured fantasy cartoony themed version of it with to make it more accessible and that did lower the CPI it didn't really make the game more successful but it did it it did make progress along those lines then

we were able to get the to license from Fox a bunch of animated IPs and were able to create animation Throwdown fundamentally the same game but we worked a lot on the accessibility of the early part and these characters made the same gameplay very very accessible so that was a big success for us so then we went to we were like okay let's test test art before we even start making the game so again with synapse they were making a game that was gonna be an idle game so we tested a bunch of different creatives against each other yeonhee rose relic relic Raiders and castaway Cove and the most casual one castaway Cove had a really good put through rate and we decided to go with that but the funny thing happened is that as we got to the to the two test markets that data didn't hold up particularly well we went through three rounds of testing of castaway Cove in each case really good click-through

rates as you can see here but once we got to test markets and a really brought in a broader sample they really took a nosedive and then on top of that we had we finally got to a beliefs either conversion on the store page we usually target 30% or higher and this one it was really disappointing it was below it was 22% and overall it's not that the the CPI is for this game are bad but the pretty average and we're really trying to make something better than average part of it I think is that the gameplay and the creative didn't really work together and that conversion piece which we weren't able to test earlier was important I also think it's just some of the limitations of testing there's probably another five or ten things that we did wrong but it shows that you can't just necessarily use science to figure out what you should make the other thing to keep in mind when using data to figure out what to make is that data Maps the known world this is a map from 1457 made

by the Genoese map maker and at the time it was the best map that had been made in several thousand several thousand years it was the first map in a long time that had the relative proportions of the continents correct and if you look at the Mediterranean did the and anything near Genoa it's actually a pretty good map but the farther you get out into Africa into Asia into the Baltic it gets you know more and more crappy and our view of the world is a little bit like that and even more there was a huge amount of the world that was completely unknown to that Genoese map maker and that's really important and games and data are like that if you're using data to decide what you're gonna make that data is always going to tell you to make a slightly better version of something that was already a success so you know Madden every year for for 20 or 30 years but it's not gonna tell you to make something like journey which is very different from that anything that's been made before and is a wonderful

experience I don't think there's a particularly right answer um again I like frameworks so the the what I've put together is kind of from the left side familiarity to the right side you have innovation and from the bottom you have unknown newcomer to at the top establish an incumbent most game developer oh sorry EA sports it would is a good example of what you would find in the top left because it's a huge company making a familiar game of a familiar game again and again and again the other end would be indicate where it's you know unknown team's making tremendously innovative work most game developers are going to fall into this yellow band and that's because when the bigger you are the more your established you are your problem there's more at risk usually your budgets are huge you've got big expectations from your investors from your shareholders

even from your existing player base so every decision that you make is high risk so you're probably going to put your investments towards something that's more familiar and safe and stay around your familiar territory when you're an unknown developer your biggest risk is standing out in the crowd so innovation becomes a less risky to you and in fact it can be an advantage because it'll help you stand out but that doesn't mean that successful games are necessarily going to come from this yellow yellow band example would be star do you value valley which is a you know better or a very polished Reaver new version of Harvest Moon from a totally unknown developer and that's been a huge success and certain incumbents cannot with the right process are more risk tolerant so supercell has a process where they're very comfortable with larger wrists and going going farther and because they have a lot of profits

they can do it and so they were able to release something like kasha rap royale which is truly an innovative game even though it came from a very established incumbent so um final words is that whether you know you're a detective a CSI an astronomer astronomer a katara cartographer or Explorer keep your eyes and minds open stay a little paranoid and try and find your wall to recreate so thank you I don't know that there's very much time off but in the five minutes hopefully I can take some questions um special thanks to a bunch of people who helped me with data in the talk and also we're hiring [Applause] any questions nope okay hi so I was

particularly interested in your list of things where you found that the testing was useful because when we were running a bunch of a/b tests we often found as you found that you know eight or nine times out of ten it wouldn't make a difference at all or difference in the sub population and so could you go into a little bit more detail about the pests that you found did work so you mentioned store feature unlocking what and were there any common features between things that tended to get statistically significant results I think they were features that were large enough and to have an impact a lot of testing a lot of times people will test things that are really too small to have an impact so the kinds of things that have worked are significant testing on major features something that significantly changes the experience there's a pretty broad range of things that have been significant for

us on animation Throwdown for example we did a test on how long research timer's should be so how long you wait until you get to unlock the combo that you're discovering that was a very significant test and had a big impact on retention so does that answer the question it's you know I think a lot of things can have an impact on your game and it's hard to say exactly what I think you know testing the color of you know of two buttons is not a very big question and you're not gonna get a very big answer so you have to have a very substantial question before you're going to get a significant right so the the timer especially economy-based timers then can really dig into your economy like in that game so that's that's a perfect examples okay yeah I had a question a lot of entry is changing with more and more games depending on ad revenue which is something congregate it really sets the bar on and you were showing us how the map of the known were

change the light mm-hmm and something that perhaps at Bannion sensor tower don't cover yet yeah is the ad revenue market I was wondering how hunger gate the analyzes and views the market in that context so one thing that's tricky and that is trickier about ad revenue versus other revenue is that it's much harder to pull into game and attribute at a at a player a level so what we tend to you do is look at proxies like percent engagement with ads that we can fire on our side and if and then we can make a good guess at what kind of revenue we're gonna get by geo based on that one thing when to understand with ads is that there is our old friend the power curve in terms of revenue the first ad that somebody's shown is much more valuable than the ninth or tenth so ad engagement is a good proxy for ad revenue even though you don't have it

there exactly does that design answer your question thank you right so if so you're a data analyst a large company and you're saying that it's hard to convince people to make new games using data because dates are only map to name world what tips might you give to data analysts large companies who might want to encourage executives to make those new games example um I think fake thing oh I hope this is apparently my last last question I can answer if they want data if I think think about what their motivation is and think about and use that to finally try and find things that will satisfy their motivation so it might be clumps of different types of things or looking at

something that's radically different or yes something like that the other thing I would say is that the culture of a company and the culture of risk in their executives is something that's extremely hard to change and so if you really want to do something innovative and different then you may need to go to a different company yeah the larger the larger the company the harder it is to influence ok go um thank you everybody