From missile defense to user acquisition – How advanced AI models predict performance

[Music] [Music] okay so although everyone my name is Rory I'm very excited to be here thank you very much for attending and I'm going to talk to you about machine learning so let's start well we're going to see three different applications of a technology that is called machine learning I'm going to tell you a little bit about what it means and how we used it the first one is going to be in from the video editing domain this is this Obama over there after that about missile defense I'll explain why and then user acquisition which I'll refer to as the UA from now on I hope that's okay so before we start I want to show you a

couple of videos one of them is real and one of them is fake and I know you need to move your hands and it'll be hard but I want you to engage a little bit and tell me which one of these are real and which one is fake okay are you ready it works especially our friends who are lesbian gay bisexual or transgender I visited with the families many of the victims on 3rd ok so please raise your hand if you think the right one your right hand side is the real one this is this one yeah and the other one okay that worked better in previous times but cool cool cool that's also good and you're right actually this is a fake one the sorry this is the fake one this is the real one and it's important to say that although it didn't fool most of you you're probably very very talented and sharp eyed people but but usually and I want to illustrate that this videos were

created solely by a machine learning algorithm no manual editing of the lips or anything like that was done here okay so how was it done basically the algorithm was trained using 14 hours of data of Obama videos of Obama giving lectures and and whatever and the algorithm learned found similarities across videos found patterns found modifications that are relevant that look good and then it applied the audio line onto the video onto the target video and then created the fake video and just to show you and I hope these will fool you a little bit but how robust are now amazing this can dull this technology can be so basically we can take any target video and you can take any audio sample of Obama both should be of Obama but with some modifications it can be applied to anything and this is something you can

make to help families refinance their homes to invest in things like high-tech manufacturing clean energy and the infrastructure that creates good new jobs not to mention the job training that helps folks earn new skills to fill those jobs ok so as I said before all of this was done without a manual editing no a video or any after effect kind of work here just like this so this is the first example of application of machine learning in the video editing domain now the next one is going to be about missile defense I just want to tell you a little bit about myself in my previous job I served for four years in the Israel military in the cyber security units and along the way we've dealt with various challenges one of them was trying to help build the algorithms behind a missile defense system ok and this is what I'm going to talk about now and we use machine learning that's the key so as you can see here before I

start missile defense system must know what kind of missile what type of missile the system is targeting ok various missiles as you can see look differently behave differently they have different sizes and altitudes and diameter and what we call drag or the friction with air and because it's very very different the system must know what it's targeting so that's the challenge here the challenge is to try and classify what type of missile it is using the data now what is the data the rather we used is a very simple location data of the location of the missile in space while it's lie okay so it's only XY and z coordinates of the missile in its flight now I know this is not very clear so we used our very talented creative team to build the next illustration to show you exactly how the missile flies on this diagram are you ready yeah like this okay so that's

basically what we have it's a very very kind of poor data and not much details not a lot of features what we called but still the idea is to try and classify the missile how can this be done so the algorithm takes these dots and create the whole trajectory the whole path of the missile in the air in order to kind of try and predict how it's going to look and now that the algorithm created these trajectories or the whole path it can say okay the larger missiles fly higher the smaller missiles fly sharper and a lot of kind of similarities patterns stuff like that and using these kind of insights the algorithm can predict what type of missile the system is now targeting and this is the goal does this make sense so far excellent so basically the two main principles of machine learning you don't tell the model how to work you give it a lot of data it trains and then it gives

predictions and everything can be done very very very fast very accurately this example of missiles this system or or similar systems can save thousands or millions of lives using the ability to make predictions very very fast and very very accurately how is this related to user acquisition so we covered these two pretty fast now before I show you what we did with user acquisition and how we did it I want to tell you a little like what I mean by user acquisition because if you ask five people you'll get ten different answers I guess so basically as I see it at least user acquisition has to based parameters you need to have a very good creative a kind of video or an ad or an image that will that will take your user to say mmm that's that's nice I want more of that I want to click I want to buy I want to go forward and do whatever you want them they to do

sorry and you need some technology need technology in order to have a full inside of the funnel you need to know what is going on before the install and after the install you need to track result you need to optimize results etc so you need to have all this kind of infrastructure in order to start once you have that you just need to throw your ads to different channels and optimize it and that can be done using Google AdWords an amazing platform Facebook or Pinterest Twitter whatever you like any social platform out there various media buying is a lot of Representatives of them are here and also search engine optimization and App Store optimization the new kind of buzzing word can you see that okay now you can see observed emulation and surprisingly enough this is exactly how yellow where the company I come from is built we have a very strong creative in-house studio we have a tech team and we have different departments that

manage each of these channels now regarding observed emulation we actually have the biggest or the most successful App Store optimization department of the world we're covering apps like tinder and and Airbnb and we even do the App Store optimization you can come in guys if you're like crowded or not we're doing the Google app in play store which is kind of awesome because they can do it because of like policy or stuff so now I want to tell you what we do regarding machine learning and user acquisition in order to do so I'll use a little better video that we made and I'm going to introduce you to Allison what you could know exactly how your campaign is going to perform well they know it's not just our kick-ass UA experts amazing creative studio or our fully managed solution

meets Alison yellow heads groundbreaking predictive mastermind engineered by top university math professors specializing in machine learning allison analyzes dozens of parameters and draws actionable conclusions based on millions of data points with advanced statistical model she will predict your campaign performance allowing us to optimize faster and better to get you greater results stop wasting your ad spend yellow head sir did you see the missile we have something with me something so how is allison so good and how as i'm going to show you well first thing the big advantage that a machine learning algorithm has on a human being is the ability to look at much more parameters the person before me just talk on how many parameters you need to see you need

to optimize accordingly you have the whole forest of parameters in his example and so basically this is a list a short list of what usually a campaign manager looks at when they try to optimize a campaign so you have the amount of spend you had and the CPI the cost per install and the FDD not everyone calls it like this but this is the first time depositor the first the amount of users that did any deposits in the app and ROI and not very long list but usually that's enough allison looks at this as you see it's not only that allison looks at more kind of more parameters it also looks at the greater level of detail per each parameter if you look at R or Y and that's what most people look at so Allison can look at the ROI after one day of install 2 3 4 5 6 etc and various chunks of ROI and stuff like that and that kind of gives

Allison the insights or the ability to cover and give more information when it gives its prediction now before I move forward I'll summarize Allison in one sentence Allison has the ability to take our data a lot of data we have in our database two years of tracked data something like hundred like sorry ten million dollars of ad spend from what we gathered along the years and Allison predicts the our yd7 or the future return on an investment of a certain ad in seven days time let's say you're opening an ad today you'll wait a day or something to see how it's working and tomorrow you'll be able to know how the ROI is going to look with greater level of accuracy in seven days time does this make sense I hope so great I see nothing fazes so how we build Allison okay Allison is looking at this but how we started and how we realized that this is what we need to look so basically the first part when

you have that much data is to try and analyze it not a very simple task usually things are not very clean as you want it to be but with a lot of work and a lot of process we've managed to find some great analysis and we did a lot of analysis so I'm going to show you very quickly a few so this one it's kind of a situation Soria plot of month and day of the week to see how weekends weekends during the summer look a little different and stuff this one shows okay it's a bit tricky what percentage of our Y is being deposited in each day after the install as you can see after the first day roughly 30 percent is being accumulated or already deposited and then it drops and an each app and each and each account has slightly different curve like this and knowing that has a

lot of importance when you try to predict the future this one shows various bins of installs and how they behave according to frequency and min cost and stuff like that this one this one okay this one is one I really like this one it was kind of a haha moment for us this one shows the correlation between CPI cost-per-acquisition over here and ROI and you can see that when you have low CPI when your CPI is still kind of low we'll use well view when you'll grow your CPI when you'll pay more for each install your our Y is going to increase okay that makes sense and you'll get better quality users your users will be more valuable they'll deposit more and although they cost more they'll be more valuable and that works up to a point but from that point on you're paying too much for your users and although they might be a little bit better they're not

good enough to improve your total ROI that means that your ROI decreases so making it to summarize this graph shows that when you grow and grow your CPI in the beginning it's actually a good thing to do it will increase your ROI but after a certain point it will start dropping and knowing what this point is or what is your optimum CPI is very very valuable and we know that for each app and that helps us find better results and Allisyn obviously knows that and better predict the future so this is just a few examples now this is the interesting part the results or what else and can do this this graph is from the last month I'm obviously hided the names of the app to manage but you can see that it's a bit tricky so I'll explain what we see here if a certain account spends a million dollar a month okay now listen tried and predicted every day for the next week what will be

the ROI and you can see here that in most apps or on average we're in about 80% accuracy Allisyn predicts 80% accurate with 80% accuracy the future ROI of current ad campaigns today think of the significance this kind of information will have now it's yeah now this is cool and we were very happy when we produced this graph and then we said okay that's not enough why isn't that enough because two reasons first of all maybe that's not very high maybe our account manager give 85% accuracy in their decisions so 82 is nice it's great it's perhaps it's better than some competitors but it's not good enough that's one reason and the second reason maybe we're testing Allison on a very very very kind of stable month machine learning algorithm is like stability so if this week we look is the next week and then one after Allison will be it will be easier to kind of predict what

will be the future just go with the line and it will probably be the same so that wasn't good enough so we started thinking okay how can we really isolate Alison's contribution to our campaigns how can we measure how good Alison is and or how much it improves us and then we started thinking okay an account manager or the campaign manager takes dozens or hundreds of decisions every day they can increase the ad spend for a specific ad they can open a new audience or change an audience they can change the creative they can increase the bids or reduce the bit that cetera et cetera etc hundreds of decisions so we started calculating which of these decisions looking retro actively were correct and which were wrong which should have been taken and which shouldn't okay and now we got to a certain number for each app we know what is the rate of correct decisions that is being made now obviously a more experienced campaign manager has less error rate and so on

but what we wanted to see is how Alison changed changes this okay what is the difference between an account manager without Alison and with Alison and this is this graph it's normalized to 100% this is the the blue green whatever color you see the this one is what alice is what account managers did before or the rate of correct decisions made by account manager before they used Alison or in apps that don't use out so for certain reason and in the orange you can see that in some apps first of all in all apps Alice improves the rate of correct decision which is wonderful because then there is no reason whatsoever not to use Allison and the second part is that you can see that in several apps Allison almost doubled the rate of correct decisions that mean that when an account manager thought hmm I think this ad is not going to perform very very well I think I'll close it and Allison told him no no no no no don't

keep it it's awesome just wait have some patience and the account manager did that that was right the ROI of that ad a week later was great and to the other side around okay I hope this all makes sense so basically we covered machine learning in video editing and we've seen how similarities can produce cool and a little bit creepy results we've seen now in missile defense this technology can be applied very very very fast and very accurately and we've seen how using user acquisition we can dramatically improve that are the ability of our professional team of our account managers in order to produce higher results better results thank you very much and you can all meet us at booth 117 to hear more about all of this and more thank you we sneak in one quick question if anyone has it I

don't want to do the one quick question I will ask a quick question if not and I noticed you've got your apps ABC et cetera yeah but both charts were there the same app so it was a at a in the same in both slides you know intentional but they think I sorted it by the names so I think it is but I noticed the one that had the least impact from Allison have it have actually the lower corner of retention no kind of fun yeah it only makes sense I haven't tested it but yeah but make sense it's always interesting okay thank you [Music] [Music]