The unreasonable effectiveness of data for better monetisation

[Music] [Music] and good to be here always good to be here for a pop gamer Ted Ronny had a business development for scientific revenue and I'm going to talk about the unreasonable effectiveness of data for improving your monetization so last four years we've been around we've you know built some pretty robust machine learning technology but also a body of knowledge that we use primarily for optimizing in-app purchases and increasing revenue you know for four companies we create as much as twenty forty percent revenue or the regular pricing already existing in the game but in the last year and really the topic we're going to talk about now we've been trying to take the same data

same information we're already capturing in figuring out how can we use that to improve other aspects of mobile game performance and really specifically talked about and we're been working as sum and user acquisition since we call the Morpheus moment what if I told you this really is actually the thesis of the whole paper and discussion we'll put a paper out on this as evidence is now clear that machine learning can dramatically improve your revenue right now again that's what we've been doing you know for worth a net purchase revenue but also again the same data same information that you'd be capturing that we're doing this right now is going to help you to acquire higher value users help you make decisions about store design you know figuring out do you want to show ads this player do you want to show monetize them with I APs and the list goes on and on improving retention all this and the main point here is the same data same information can be helpful useful across all answering all of these questions a little more about the pricing aspect of

things and will go on just really because it sort of sets the stage and the framework for where we love us discussion it's active revenue again we do dynamic pricing for in-app purchase revenue and what we're trying to do here is transition game developers away from one-size-fits-all approach the pricing - I'm segmenting your players in your game and really trying to deliver optimal pricing for different groups of players right so now on your own you might have you know figured out you know maybe this may be testing and that's good and you figured out maybe the best price possibly would be the p2 and you've seen that you're going to capture all of the utility or the revenue underneath the demand curve at that point that's great we also want to capture B as well and we will offer p2 but there might be a group of players within your game that you know the actual willin spend more you know for that purchase and so in that case maybe we want to have them a higher price maybe they're from you know a higher earning income country like United Kingdom we're in right now we're here in London versus in the outside a

whole bunch of other reasons engagement and spending parents too and we raise the price them or more likely than not and this is more the more common use case is you've probably overpriced your net purchase revenue to begin with and there's me a whole bunch of people out there who actually like your game and we're willing to spend on it they just thought it was too expensive and we want to lower price for them to capture that and the way we're doing is we're doing machine learning on the data that we take posts install data information but and create payroll patterns and profiles and that allows us to do this that's about pricing but if you do want a little more information about pricing this talk was given by Bill who was posting through this one back at Pocky gamer connects and Helsinki if you really deep discussion around behavioral patterns and data science machine learning and how we use it for pricing this deck actually is a lot of words on it so you can actually get a lot of information from that this is a SlideShare it's already up there available for you to check it out also our website and resources section down on the bottom and even has a video from

the Pocket Gamer talk in Helsinki enough about that talk and poem so we're gonna talk about data science now we can use it for user acquisition and ads and this is not necessary part of our business right now so this is more of a you know where we're going and we're doing somewhat of it now for some partners but also what you can do yourself but first let's introduce machine learning pyramid and you may be wondering what is that actually it's a brand new idea thing we just kind of came up with it but we do think illustrates nicely what we're talking about fact this is unveiling it for the first time here at POC gamer London and it's also so it begins the foundation of is really just about all about gathering the data and store that data that's why it's the base of the pyramid it's rather large amount of work and then reporting for human consumption really this is going to be those you know KPI reports that you may already be building for your boss or for your team or whomever you know or hopefully building point point is the gathering data and the reporting of the data this is all you know the basis of machine

learning that also all very easily understood something you're probably already doing right now for I mean it's also the pyramid is proportional for the amount of work involved so two-thirds of your work it will actually be spent working on the gathering of the data and trying to put it into some sort of usable format going up there predictive analytics now we're getting a little more complicated not you know but also very well well understand becoming more well understood talk a little bit more about that in terms of the literature out there and to help you get along and then now once you've got the data if you've you know you've made some predictions based on how people might behave based on that now you can start with changing system behavior this is really where it gets interesting right and this is where you're actually now with the information you're gonna make some decisions on what how what do I want to price you know this player how do I when do I want to serve an ad for them what type of that you know positioning within your game you again some of the predictions are set about predict you know the gather data reporting all well-understood predictions becoming more understood

certainly for example the churn prediction is good sorry it's very well solved highlighting that you know a couple articles here to the term prediction article to say similar things into highly recommend you to check out those articles because this is really how you can sort of figure term prediction yourself they are again the SlideShare site will be available put it on our website so you can find it there or it's gonna fight your account but nice thing here for this is that you can have your data scientists if you have one read these articles all the informations in there then transform the information directly into code and then now you will also be able to figure out when the players gonna leave your game and then the decision is up to you what do you wanna do with information in our case we might say hey I want to get one last big spend out of you while you're you know if you already wore a pair in the game before you leave you know the game altogether so that's pretty valuable prediction questions you know this is note that there's a mixture of

true/false you know in this case is this user about the churn true/false with mixed with numerical yeah you know how many more minutes will this user play right that's important to the distinction and again you know the first part of this whether or not the churn is solvable that is available in the literature as we just you know pointed out to you but moving on to you know in-app purchase revenue and then add control you know this is you know will this user buy an IEP for instance this is also savile right this is what we're doing is other people are also doing this just means taking more work involved and having data science approach not the literature is not yet quite available on that out there or maybe there's some literature out there but it's not as well completely understood in the same way that term prediction is but it's coming and it's also again very much solvable for yourself you'll notice also that there goes up sets of questions now MIT which money will use or spend you know within that when you start kind of subsets those questions we can start assigning probabilities to those right in

different outcomes coming out and then again using putting into a prediction framework to determine how we want to move with it so intuitively you know these are solvable questions right you know you have a question there's an answer for each one of these right you know so what is the number one predictor of spin that's engagement you know over time what is the number two predictors spend history previous bet what's number three predictors spend you know interaction virtual economy goes on and on and that questions keep going on you start even having other types of questions with you know are you know is you know is it more valuable for an Android user an iPhone yeah well spend more well that one actually depends you know we talk about a high-end Android device for low-end Android device right answer that subset question helps inform the other you know bigger question too we could keep pulling on more and more questions ideas essentially the main point here is that it's a ton of data information and this is why you need to have machine learning and data strong data science approach to

get through all of this because this exhaustive amount of detail information at high speed to lie you make predictions on that and if you can be quite effective in our case what we do is we call these pricing triggers we have an SDK will literally capture hundreds of pricing triggers within the game and we will put well almost center out of pricing engine in the cloud there were doing machine learning on the data we're doing behavioral analysis and then in real time based on this information that we've already determined about your game and players we're playing in your game we segment players in real time and we use the information to price but it can be again used for a whole host of other benefits within your game the you not be perfect you know this is illustrate this point here is that you know like you know so you already are doing this within your game they you know that's gonna be the random guest line right so you know so whenever you already making predictions and someone comes in your game and you're making the

decision on you know are they purchase what's the right price you ask yourself if you're using data or not then you will you know you're just making a educated guess you know at best in this case the you know what we're doing for tech machine learning what you should be doing yourself that's the black line that's the prediction line you know note that there is going to be false positives you know you are we are gonna make mistakes you know you're gonna make mistakes out there um you're gonna have miscalculations that's okay you know you by using machine learning and you know using computers to make these decisions you're gonna do better than guessing and that's why you should do this right so don't expect to be perfect right out of the gates you know don't suspect oh I got to wait until I answer these questions so I'm perfect you can do this sooner than you think and how machining works learning works in terms of what you know or you know we're using us to define distinct classes of users in the game right so for instance blue is the good as you can

see labeled in this line so what's a good player in this game well that's we've defined that it's gonna be someone who who stays around a long time and they spend repeatedly we can we all agree that's probably pretty good we want to have that player we want to keep them as long as we can we also have bad players that maybe some fair combat but the red group but these are people who turn quickly and they won't spend at all when they're in the game right so not really you know certainly want to spend any money to acquire these type of you know players come in their game but we're gonna classify them as as bad and this is where the machine learning is going to the date and we're making those tag ins and definitions related to that and now taking it to a practical use of user acquisition the first thing in facebook you know you're probably using that for acquire users already some second most people are it's quite effective you know generally you would give and here's a list of players I think are valuable Facebook go forth and bring me more just like a write look-alike targeting in you

know they throw it out across a social graph and be on the you have no idea how it works and won't tell you but it's somehow it works and you bring back players and you know you do it again if it's it's been effective for you middle last year then they introduced value based look like targeting right and this is pretty cool like there's a nice enhancement of that what we're doing what they're doing where they're doing here is really not providing adding a numerical value on those players right so zero to one so going back to our earlier example of the good and the bad right you know with a case bad you know we want to assign a value of zero so actually want to give this information to Facebook and we're saying do not buy these players at all that avoid them as much as you can and then we take the good we also deliver there again good and people who do spend repeatedly within your game they can stick around for a while absolutely what these players you know please go forth and buy them but the nice thing is with the predictive lifetime value that we're signing to it we can actually they can you know use that information as more of a ranking within that and go after inquire that so we moved pretty please the results and

it's been working really well so we've been working with partner on that couple partners actually we don't talk about unfortunately can't really you know name who it is because we do have confidentiality agreements with Vernon you know we're doing pricing it's a fairly sensitive stuff so if you understand but with couple partners we've been doing this and this examples from one in particular is we've been working with them we've only I've been doing Facebook ad campaigns sending both the good and the bad to Facebook and having them acquire users since we have attribution I DFA we're able to track the results as it comes back the game in terms of results the organic users that they had coming the game that actually they wouldn't they I had a lot of organic users going game which is great there's free you know coming in we all like that but obviously they had the lowest value that's okay it's free they were working with an outsourced user acquisition partner who did a pretty good job they were actually you know to X the value of their organic users so that's good you know you're spending you'd hope so but

then the day didn't we work with their partner on this as well is the data that we were captioned for the lookalike users so it actually cost one and a half times what they already were spending on you know for user acquisition but the kicker is that the value is much more right you know in fact it was six the ARPU for the people that data that we identified was 6x the ARPU for the with already requiring from the user acquisition so that's over a 45 day average a 400 percent increase so we're blown away so it works so you data science take it seriously you'll have a competitive advantage you know if you do and then you know absolutely using machine learning and you'll monetize much better and retain users much longer okay so we talked about the good and the bad of the main how we use that for users acquisition you know again sort of a thunder lying thesis of this whole presentation and why we say that you know is this work is that is that we want to reuse the same data over and

over and over again right you know because there is a bit of work the captioned to do it properly but the really important point is you know all those seven things we do have a cumulative effect that's quite substantial so we did use acquisition moving on to add control you know within it mostly net purchase games you know what you have ads in there at some point and you know industry average might be wait twenty days and if they haven't spent blasting with ads right and they're ones a little bit different on that and of you know they have your own decisions on when to serve ads and where to serve ads and what level within the game etc out there but you know an average the idea is wait till they spend then blasting with ads if they don't right get what we can out of them well what we're going to do is we want to try to accelerate when we show ads right you know again same data information those bad people those are the ones who turn quickly they'll never spend if we can identify by them sooner and we can then for instance we have a good idea that a player will in three days will never spend well the great idea in seven days of data from the player that they'll

never spend so in this case let's start showing ads sooner because I'm not going to Smith they're still around or at that point let's show some ads you miles get some money from them anyway you know while they're playing and enjoying the game going forth so that's what the predictor models become much more certain about a user and we can turn on ads sooner for customers using the same data so that's the you know to summarize the points the you know main issue its main point here is that there's sedimentation relations best practices that are susceptible machine learning right I listed some of them and you know each one of these different you know they all rely on the same basic rules of thumb out there the same data our pyramid remember we're gonna do the data collection we're gonna do the your cleaning accurate you know scrubbing of that data we're gonna report that that's actually again that's two-thirds of the work anyway you know you're already doing that anyhow you're well on your way to having a machine learning approach to a modernization then the reporting in the predictive analytics and they should start making some decisions you know again each practice we talked about a lot of different little things you can look each one of

those things they may not seem like they're gonna have a huge amount of impact in a game maybe 10 20 percent but you add them all up together it's a pretty substantial increase in the revenue you know for you I'm so accumulative impact then is potentially enormous and it works so we have the slide share of this presentation actually this must be the slightly wrong deck the present we already also have this presentation on our SlideShare account so if you'd like to see it have this information we will also post this presentation on our website and a resource is probably in a couple days time thank you right that's pretty good yeah yes I think this whole machine machine learning aspect I think is something a lot of people kind of a bit frightened of in some ways not quite not in sense of but nothing any was expecting Skynet's take over just yet but there is that kind of question about how much of this analyst analysis is something that is tangible it's something that you know that we can get under the bonnet of and

feel like we owe and whereas if we have a data analyst we can ask him where if we looking it up it's all about the setup isn't it with the machine learning is the data and you're having the data first of all I mean there's you know in the reporting that Delta DNA was before me you know they have some day they give you a lot of the data you know so they forgot how to slice it they don't tell you what to do with it you know that's kind of the next level yourself the terms of the predictive framework and this is what we're doing we're not finishing zone is a pricing engine with a game on top of it absolutely no they're doing that spiking a lot of top game so the information is out there I talked about the churn and Ellis one is absolute something you do now you do need to have data scientists so yes this is not like getting out Excel and sort of just you know clunking this out you can't take my Excel from me yeah I don't think actually there's correct vase no I mean went to my old school like me I was trained in telcos to get data analysis and so when I came to games like how no one was doing it at all so it's really weird and you know what do you mean

you're not checking the users and what they're doing and over time I mean how do you do this and now we're getting to the stage where we're actually going so far beyond what I was doing if this idea of actually being to take okay here are some things we know here are predictable right if we may not know what the predictable factor is but these are the factors we see tell us what we can predict yeah and that I think it's a magical time it's you know look and we see it around us all the time Amazon changes the price a million times a day right we all know that you know so we don't you know so this is happening it is happening out its beginning you know not quite that level within gaming but as it says what we're doing right now certainly some other people are doing to some larger games do companies have armies of data scientists or doing this right now we said one of the other issues I think it comes up and somebody asked me this yesterday actually is what do we think about people who price different job yeah are in app purchase items so the same item with different prices for different people how do we deal with people's anxiety over that we do you have any thoughts on

that or is it yeah what a local woman we use we thought a lot about this obviously persons are the core of our business and you know every single media habits in first question so I mean there's a lot of ways you can slice it up you know what do you do you know so one you know what would you do about it is you know what we try to make sure people do not see that just a price you know so when you get annoyed if you were midway through a game you just wait a minute the price just change we're never gonna do that you know and we're very careful but the main point is that you know you may have overpriced it anyway so there's people playing your game your conversion is you know hopefully over 1% you know and you're doing well that's what you're doing Wow 1% of people in your game would have purchased Wow you know how much other people playing free and you know the right price honest my opinion is what someone's willing to spend for it you know and and if they're getting value you know for it's a digital good and they're making a fast decision when they go to the store front to say whether or not I want to buy coins and the right price is what the woman spent for it and you know okay sometimes a question comes up in the notion of fairness right well you might set your prices you know what we were in the UK you know pounds but maybe you're you know on Euros right you're in the

continent there and I think it's unfair to charge Greeks the same and trying Germans right they have much less money you know so you know unfairness is interesting question it can be sliced up many different ways yeah and also at the marketing guy I'm used to pricing strategy predatory means we bundle differently we put things in different orders and under finish you planning exists people are willing you're so workforce know so it's yeah and it's also how you present it to so but it's a good question it could be a whole entire topic and happy to talking when later afterwards if it weren't you we probably should at some point I think there's a question that other okay it's actually a very good question it's tinier but it's we need for us it's like a minimum of hundred thousand daily active users a 1% conversion and actually that of the hundred thousand we'd like to see 10% of those being daily new users right we need new people coming in right you know we're trying to you know we're doing predictions and a they were trying to analyze the data as it comes in before we actually do prices or before you do user acquisition you know in this example then we like to see one percent

conversion you know means people have to be spending if people aren't spending anything at all it's not a pricing problem and so you know it's but so you need some people already spending and then at that point then we can absolute tune it to increase in app purchase revenue for us it would just take a lot longer to run and it would be a lot higher variance you know we would be we would be less accurate in that front so it would take a stronger stomach because we measure itself versus a control group so you see on a day whoa huge swings and you know the problems the smaller number of users and you have a quail coming your game that could land in pricing group or into the control group and you know that's just luck where they landed you more data than that you know smooths out you'll have even distribution of whales and control and pricing group and that's how we can see the effectiveness [Music]

[Music]