Demystifying Machine Learning for Programmatic UA & Retargeting - Joseph Iris, Persona.ly.

foreign [Music] [Applause] Joseph Iris please demystify machine learning for programmatic UA and retargeted you rock my assignments must be much better than the other ones so I don't know I'll let some people kill them hey everyone good to see everyone I hope everyone flew in well if they flew in I guess there was a lot of locals uh we tried to build this to be as unbiased and possible and not a push to use our

services I hope it works that well get some feedback offline from you guys later but yeah anyway before we start I wanted to give you like a data oriented intro to myself because we're talking about data about machine learning so I guess that would kind of work so this is kind of my like my my life in numbers you can see the experience in my hairline obviously ethic is like basically a nightmare we all like Love and Hate I have one lovely life that's also surprising to some of you I know based on you know how much how little you know of me so far two adorable Kids One turns four in a couple of days two dogs that it's really not fun having most of the time recently uh way too many pets in general uh the the one cat mentioned here is not really necessarily the worst one the other one had this point in time where it just came up stood on my chest and sneezed in my face to wake me up I don't

know why uh and yeah a team of very Russian and very you know Eastern European opinionated data scientists that I really like fighting with way too many hours playing Apex I don't know if there's any people from respawn here it's not the mobile version but it's a lot and yeah all of this comes up into one big adventure um we're gonna talk about a few things here basically the main concept around the world the mystifying is really to get your people acquainted with like the underlying principles of machine learning why you shouldn't really be scared of it because it's not really that complex when you get to the bottom of it and to explain how it applies to our use case with some examples of how we built our Solutions and eventually obviously talk about the elephant in all of these rooms about the loss of signal for IOS and Android and some practical advice about running with programmatic

dsps that actually use machine learning again don't just say they do so when we were talking about programmatic generally speaking uh just to give you like uh a big big picture idea of how this works generally speaking in the infrastructure you should have some sort of solution to start off a campaign for the cold style problem where you don't really know anything about the audience that you're targeting other than very general things I it can be some segment generator or something building cohorts that you would Target first or something demographic uh then you have the videos that actually bid on all the auctions we have like basically 300 100 to 300 milliseconds where we can bid on an auction uh you log all of that data fit it into a model which is basically just an image of yesterday if you think about a machine learning model right there's no magic there under the hood it's math statistics that you probably all learned

in like a couple of courses in your degrees or whatever or you just know the general ID around it it's just basically how well can you remember yesterday in the right dimensions right so when when someone says a model that's what it means it's not like uh I don't know there's nothing crazy that the last thing that happens there you can see this looping back from the model into the videos this is basically the actual learning concept every day every few hours depending on how you build your Tech stack you would update your model with new observations uh New Impressions new clicks new installs and this way you should be able to uh learn that's the actual learning yeah seeing new things uh yeah so inside programmatic in the 100 to 300 milliseconds we have a lot of stuff needs to happen uh there's a bunch of predictions uh a lot of them obviously around the user engagement predicting a click giving an impression predicting an install given a

click predicting an event given an install there's also an install prediction based on The View only obviously because view attribution is a thing and last but not least is actually what we call bid shading in the industry because everything all the auctions in programmatic went to first place auctions meaning that the price you pay is actually the CPM that you pay you have to predict the marketplace to not pay over it so you shade your bits kind of like in 300 we fight in the shade if you know that reference so yeah all of that gets us just to the impression in terms of like the dsps they try to you know manage all of these things within the time limitations you have two types uh it's kind of simplistic but this is kind of the best way I think to look at it on the one hand you have the ml based ones uh when we mean when we say custom predictive modeling we basically mean a set of rules that it that is

determined over time but it pronounces really complex relationships for example an exchange can realize if a specific cohort sorry a model can understand if a specific cohort enjoys seeing ads at the beginning of those sessions or the end of their sessions so it really depends on which Dimensions you build how detailed you go and that way you can get like very very specific outputs the other end will be more rule based you will have this sort of like line item structure where you can set up line items based on like different dimensions like the example here where it's like the store category the ad placement and and the device type and you can set bits over this you can set all sorts of rules to basically stop buying your performance is ineffective you can set up a role where you're saying okay if I'm getting more than 10 CPI just stop buying them so this kind of it lives but it's not learning There are rules and it adheres to them you can add

those you can change them and it's very specific and takes a lot of time um the learning aspect of machine learning is something that's very difficult to visualize and it was very important for us to do that anyway when we came into this uh we didn't know one thing about machine learning so we wanted to make sure we can visualize it and explain it because you can if you get an example so if one of your partners says it's a black box his either being lazy or lying because eventually in order to run within 100 to 300 milliseconds you need something that's not that much of a black box otherwise you can't really do much so basically I want to start with with this visualization what you're seeing here the dates basically represent the same model being trained day after day the x-axis is the probability of an event happening in this case an install you see it's in groups of five percent

and the y-axis is the percent of users from the targeted population within that probability group you can see here when it starts off again that at the beginning most of the users are basically at zero yeah because the model doesn't know how to discriminate between them how to classify between them over time you can see the distribution normalized and basically all the users on this end are users that are going to get higher bits because we know there's a higher probability of them converting so this is basically how learning looks like in terms of like the users that you target um basically one pain point for programmatic DSP is getting uh um basically exploration done in an effective way you don't want to spend 25 000 with the DSP to get one paying user right we heard that happening so yeah basically in order to start off effectively you need some sort of cold start solution uh we have one for iOS we have one for Android basically some sort of General

segmentation or some sort of uh approach that can help us start from something where we don't need to learn something we already know if you're promoting a match free game there's no reason to Target a 12 year old boy because he only cares about fortnite right it's still popular I think fortnite with 12 year olds so yeah we need to avoid that the last thing is with your cold start approach you want to be careful about building something too specific that would be considered over fitting basically you would be learning only a Target population that converts at a very high rate obviously most users don't convert at that rate if you feed your machine learning only that it's going to overfit it's going to think all the users are going to convert amazingly and then when you try to scale it fails miserably we we were there and we had to figure that out over time uh the other thing which I think is the most important thing about working with a DSP that runs machine learning and the Machine learning and doesn't just say it is to be transparent and to explain

what's happening under the hood because you can do that there are all sorts of tools generally speaking based on our experience and all the literature that we read as well uh solutions that are too complex to understand even if they're considered to be the state of the art usually don't outperform the very let's call them Common Sense approaches so that's again when someone says it's a black box I couldn't understand if I want don't buy it doesn't really make sense in 99 of cases definitely in our case because of the time limitations that you have like 300 milliseconds you can't compute you can't like split atoms in 300 milliseconds right maybe you can I don't know like all of that physics stuff but you get my point I guess um so for example this is like a visualization of like the the graphs that we show our partners about feature importance right so in this one if I can explain it briefly the grain line stands

for zero no impact on the probability of something happening everything in the left increases it everything and the right increases it you have the features listed in order of importance red stands for high value when blue stands follow that's a lot but like you get used to it very quickly so an example here we see that rewarded placements increase the probability and that The Closer the context is uh the performance improved so these are things that can be explained basically with any Machine model machine learning model that isn't like considered a full black box but even the more complex ones support something like this so we believe this should be the minimum in terms of transparency uh yeah so the the hard thing and loving scan that's kind of a lie it's more of a love hate thing where I love hating it but yeah we scan is like you know not great in terms of like what we can do with it as an attribution solution

but eventually obviously there are ways to work around it right basically we're losing context so we we're keeping only context and we're losing everything else right we only know where we're buying the ad and we can relate it to the ad we're promoting um other than this obviously the the changes with scan we lose signal but the prices of the of the auctions decrease accordingly So eventually machine learning can still be applied it just has less signal learns a bit slower but the decreasing prices help with that now the way to work with context in order to work either inside scan limitations or generally speaking how to Target an iOS uh the way we see it you have three main Solutions the first one is actually using store categories for whitelists and basically seeing if your app works well on one ASO makes this a nightmare because people change their categories every couple of weeks to increase the ranks uh the other one which takes a lot more time and maintenance is manual

tagging creating your own categories that's something pretty common in the industry and in any industry it's also called The Mechanical Turk that's like the service it works but it's not that scalable every time you have a new app you need to tag it every time you have a new words you need to consider them and NLP which is basically natural language processing not the avocado that's you know used for coaching lets you do this automatically basically you can represent an embedding of an app that captures its content basically inside app descriptions there's mostly the truth about what's happening inside the game uh you can see like the mechanics the theme and if you can capture it in a way that machine learning can understand it you can do some nice stuff right uh so the way we did it we're using a model called Elmo because machine learning guys love obscure references to the Sesame Street for some reason but

basically it lets you take the words inside the app description and represent them based on the weight uh weight being the like Rarity basically the moral world is real the more it's meaningful for the content of the app and basically using vectomf that I don't know much about but it works you create a representation of of an app that's a vector and when you have vectors you can measure distances between them so you can pronounce the contextual relationship between two apps in a number that ranges between one to zero if you choose to normalize it and then you can build Dynamic white list off of this using normal structures of scan campaigns or whatever you do but it's much much more dynamic because each new app just gets its embedding you don't need to worry about maintaining it uh another huge part of machine learning and applying it to programmatic is creative selection

you can do really amazing stuff and improve your performance when you do it correctly uh in terms of like the methods that exist we all know a b testing obviously we wrote instant regret there because of the associated regret cost right you're running an A B for a week you're regretting all the costs it went to be that sucked so when you're building uh mobile solution like What's called the motorhome Bandit that's basically coming from the allegories for like I want to unbend it it's a slot machine it's basically an algorithm that's designed to find the best slot machine that's gonna win that's why they call it that way but basically as you can see in this image you can see this from the start everything is split evenly between all of your creatives in real time you capture the best performing one in terms of Click rate or IPM whichever one makes more sense for you for your use case and straight from the first Target that you get the first install it starts getting more and more inventory so this is like an real-time a B test that feeds uh with

each Target that you get so it's really really useful and much faster to converge the next one we have is contextual Bandits which is basically the same thing only with more than one Champion yeah so the best example I can give here let's say you have a game this is when you're using the multi-ambend its fingers I described before you have the champion it has the highest IPM it gets 90 of the inventory the other ones get split according to their performances you can see the IPM is decreasing in reality you're targeting much more than one group of users yeah even when your game is a niche you have more than one Persona that's just the reality it doesn't have to be a game but you get my drift so instead of just selecting a single Champion you pronounce the context of the users to the model uh basically saying uh all sorts of demographic features about them and you

choose a creative based on those uh yeah attributions of the attributes of the user so this way you can as the example shows get higher IPM overall because you're targeting the right creative to the right person like the classic example is the difference in like age or gender basically older people care about one thing younger people care about the other and yeah last thing I wanted to show is you have some actual practical tips for you guys when you're about to work with the DSP so the first thing is making sure it's an actual DSP uh when my father described our business of ethic to all the Jewish people rabbis and stuff like this about like what we're doing he used the Yiddish word that says business in the air right so basically people can just follow around I'm doing machine learning I'm a DSP do your due diligence ask them the right questions see that there are more than

three people on LinkedIn and yeah question those stuff uh data sharing is is critical obviously both to avoid targeting existing users but also to help the the machine learn faster so after you do your due diligence make sure there's an even playing Ground because social has all of this data so you want to make sure your dsps have the same opportunity to one at the same performance be clear about your goals you'll be surprised how many cases we get kpis that are basically I had one campaign in 2019 that had this amazing performance and this very very tiny bit of audience that's not a realistic API give us an average give us something that makes sense build it in a way that's like gradual you know because basically as the next Point says there has to be some patience for the machine to learn obviously you don't want to go too far because at that point you start being frustrated and money is just being built but be careful I repeated this many times throughout

the the presentation feel free to ask questions and feel free to ask full time for the right people to ask the questions you're spending millions of dollars a year on these dsps you can ask for a half an hour with a product manager they'll come out of their cave they'll shave or something they'll babe once that month and you can ask them the questions and yeah the the first original uh writing for like the the phrasing for the last point was if it quacks like a duck it's click spamming I like that joke better but other people don't but really I mean if something really looks good or good to be true it's basically organic being stolen again this still happens it's sad but it still happens so just be aware and yeah thank you [Music] questions and make it a good one

for Joseph did anybody have a burning question Joseph you are free he just answered everything so well you're full of like machine learning answers that's why thank you thank you very much indeed cheers thank you another round of applause please for Joseph [Music] [Applause]