Through the looking glass – Following the rabbit down the data… Whole

[Music] [Music] so we're going to talk about something that you guys have probably heard a lot about data so we're gonna talk about an aspect of data that you've probably never heard about and so when we say the title through the looking-glass it really is a play on Lewis Carroll and it's what did Alice find when Alice went down the rabbit hole we're gonna follow the data through a to a level that you've never been able to see before so this is all about transparent UA and so we actually ran some camp we've run a number of campaigns we've been doing this now for about well we've been doing it for five years but only the data has been able to be used in the last 12

months more recently probably July first we started running some campaigns first with a developer on Facebook we're not going to talk about that we have a case study we're gonna talk about ones that we've been running on an RT B platform a fully transparent RTV platform our goal is to help eliminate wasted ad spend anti increased performance so who wants to save money and that's what it's really all about and the goal is to do it as quickly as possible so the way that you do this is you identify what's not working as fast as possible and stop it so it comes down to a couple of things so the goal of any campaign I think we've outlined them here is there anything that we've missed I mean do you want to improve optimization check you want to minimize your costs check going to meet your performance objectives check but what you really want is users who what we'd like to still acronym who create value for you they stay they play and they pay that's really what you want

so this is one app that we've been doing we started in December this is the d30 retention for an app we started in December first we actually stopped seven days there at 20% for their d30 retention how many of you have apps that have 20% retention at d30 or less anybody have less than three at 30% there are no developers in here there's another app same thing same time period within a few days December probably one of the most expensive months to run campaigns over 40% so this is what transparency overcomes it overcomes the marketplace and efficiencies anybody want to throw just shout out what the marketplace inefficient sir cuz I really don't know what they are I thought it sounded good on the slide anybody want to shout something out what are the challenges that you guys are having when it comes to doing you a fraud that's a good one

anything else that's all we got okay good scalability anything else for targeting that's a good one what else okay so what we did was we went through four different phases in each phase we focused on optimizing a certain thing in the first phase we looked at creatives and the second phase we focused on bidding because we are doing RTB bidding in the third phase we focused on app targeting we're not focused on devices here we're focused on apps we'll talk about that separately too long of a conversation for this forum and the time frame which I've got eight minutes left and then phase four is event based targeting that's where you drive value so in phase one we looked at creatives and what we did was we looked at our daily budgets and our daily budgets were equal this was the goal we're going to spend an equal amount of money across the creatives that were available to see how they see how people resonated with

the creatives anybody do this now I think this is kind of a standard practice right okay on the top we have impressions on the bottom we have clicks now they're color coded to make it easy number 11 you can see that people started clicking on number 11 so what would be the general hypothesis that you would draw about number 11 I'm sorry I can't hear it's engaging that there's a user people are engaging with that so then the traditional school of thought is we need to really focus on throwing money at that campaign and just drop the rest right are we all in agreement with that okay who said no do you know where I'm going with this I want to hear what you think first before I go what are you looking for

okay right we want we want to go further don't we so this is another way to look at it - this is a clip to rate right blue is the same one we just looked at but that's still not enough that just is a different way of looking at the same thing right oh so we looked at clicks again this actually was ten times the other clicks all right but from an install which is where you were going this one actually had twice the installs that the other one did and it had one-tenth of the clicks so what would you say about that then highly engaging waste of money people were really installing off of this one and you're paying based on an impression base with RTB right so that means this is a waste the blue so what

we wanted to figure out then don't thank you we're going to figure out how to drive those together right so that here they are side by side you guys are clapping already thank you that's I didn't expect that phase two bidding again we change the daily budgets but now we're focused on the campaigns and so what we're doing now is we're not looking at the creatives anymore but we're looking at segmenting on price and so again we're taking the app targets and I got to give you a little background here so we do 20 more than 24 trillion calculations every day based on every pair of apps all I'm going to tell you but each one has a different value so for this specific campaign there were over 4,000 targets way to throw out all 4000 targets in the first phase right back here to get to this to know which segments we would use and how we would price each segment so

all of these are different segments with different values and then we wanted to see how do they perform so now it's about pricing now we have a predictable stream of impressions and clicks all throughout the day and you can see this is like seven days you can see when people are awaking and when they're going to bed right and you can see how we're kind of evenly split between the two creatives we're trying to drive this and combine this engaging creative with the one that drives installs to see what we can make happen so the CTR rates of each one of these you can see something here anybody guess what happened at that point right there after three days we figured out they weren't working so we shut them off we want to shut them off quick make so buddies call me right now nope it was my wife I can't answer but we let the ones that are going continue

from here we've made a transition but this is again clicks and installs we drove the impressions down drove the click-through rate up a little bit right so that's a result now we're gonna go faster now we're going into the specific app targets now we're targeting on a one-to-one basis apps now we have more control over the cost you guys anybody old enough to know what a graphic equalizer is you yeah I know you are yeah so when you get the music that you want you can dial it in right so it sounds really sweet this is like a graphic equalizer so here we have the budgets app specific this was a mistake I can talk about that later we let logic rule instead of data because we wanted to test something but you can see we actually spent less money our impressions actually went up click-through rate went up it was pretty good apps again good performance we shut

them down again oops data science is machine learning in Reverse if you guys didn't know this who knows that did exactly do not somebody told me that this is what it looked like throughout the entire time putting it all together this was creatives spent a lot of money bids on the segmentation this was only focused on the apps so we spent less money here take these out okay the reason that we're taking those out is because we targeted a game that we thought had an app that had really good synergy with the one we were focused on the CPM was really high we were wrong logic versus data it didn't work we shut it out so again if you look at everything it's creatives first section bids I call the volatility in the first part so model so that is

impressions are higher clicks are higher CTR you can see the clear stages we took a break to analyze the data right there the second change was right here very sharp demarcation line but we were able to move right into the apps and keep going clips and installs obviously we went up but this is an overview of all three and you can see what happened the installs went up and we spent less over time and that was the goal phase for event targeting so we had about a five-hour discussion and - cause of the developers about events because we go from this to driving events because events drive value in your app what are the events that drive the most revenue I'm sorry the drive the most value I gave the answer okay it's those that generate revenue right there could be rewarded video it could be ads that people watch that you get paid for for

monetization and it could be I ap right so again this was the retention people who stay play right and pay this is the event so we had four engagement events two revenue events and then one based on game mechanics that was important to the developer these are the events visualized based upon the install date of the user these were the sessions per da you starting with one we started you can see that the daily sessions increased this is the events by the data of the event different than it was based on the install date so here we tied them to the install date of the app here we tied it to the event date that it actually occurred so that you can see when it took place but all of these are tied to a very specific device we know the

device number we know the city in the state and we know how many times they spent money this is the rewarded video per da you the number of sessions that they play this is the daily sessions per paying user now this one actually surprised me and we had to talk with the developers about this it was a lot lower than I expected it to be if any of you seen this in any of your games where the people who pay are actually playing less than what you would expect anybody seeing this or just an anomaly I'm gonna pick on the lady right there are you seeing this I'm sorry all right they don't have to grind as much exactly and then that's when you think it's like oh yeah that makes sense so the next phase is scaling so the only way to do this at scale is to use

machine learning and so how many of you would like to have 80% of your workload taken away just one two three four somebody write these names off for me five six so when you integrate machine learning into this process the only hard part becomes uploading your creatives the data takes care of the rest what campaign should I run in what country with which apps should i target and how do you price it because that's the data that we have so you use data data science and machine learning together to save yourself a lot of time and money that's really the key questions did we finish in time yeah oh good I was afraid we're gonna have enough time so I really hate to rush no that's great I can I can run over with the microphone if anyone else so all that was just BS anyway you guys so I made it all up the whole point was

to like baffle you guys with BS so you couldn't even ask questions disruptive exactly so so now I'll tell you how we do this no since I guess at this point we have how much time do we have a couple of days maybe yes a couple minutes we should have actually 15 after that one guy I'm just kidding so on a daily basis we've been detecting apps on the device for five years does that spinner persist it's only with opt-in right on a nightly basis we calculate the the presence of all the apps and then we calculate them using consumer purchase intent data and so we know the impact that one app has on the in the influence on the other in other words whether or not it's gonna be downloaded the app on the device so we do last time I checked it was a couple years ago a year and a half ago more than 24 trillion calculations at night there's a lot of data buying this I think in one data set there's over 12 billion rows we have the device data we've got the targeting data this can be used without an SDK people who are

interested in using this for you I spawned you should talk to me because we can do this without an SDK but you got to be committed to actually growing value in your business you don't do this for a week you need to do this for at least a few months to understand how it works can i plug myself oh yeah that's that's my okay how can people catch up are you here at a stand up the show or are you know we're never booth I don't like boost they're sort of tying it down so if you want to ping me just I'm Bret Fowler app fluence or you can find me someplace I'll be around [Music] [Music]