Winning the Game: Leveraging Contextual Bandits for Creative Optimization - Joseph Iris, Persona.ly.

[Music] [Music] [Music] we're going to look at winning the game which is leveraging contextual Bandits for Creative optimization with the wonderful Joseph iris from [Applause] personally chck so first I want to thank everyone for drinking responsibly yesterday

seems like everyone else doesn't care about creative selection as much as you guys so I appreciate it I'm Joseph I'm the director of machine learning products as personally uh we're a programmatic DSP uh providing UA and uh retargeting Services based in Israel and yeah here's my clicker we're going to talk about creative selection I'm going to try and give you a lay of the land of how machine learning is used to make that happen where in platforms that actually can do these things that are more impressive than AB testing show you what goes into it uh the major tool we're using and we believe most of the industry is uh which is called Bandits that it's a type of algorithm uh from the subsection of machine learning called reinforcement learning I'll get into that and we'll talk about the future uh of when they're taking an additional identifier away from us and we still need to optimize campaigns on Android with the Privacy sandbox um AB tests are obviously a

necessary evil uh in any industry ours is no exception uh when you don't control uh the buying fully when you're using a separate platform you have to make all sorts of uh workarounds in order to make your tests work because there's another algorithm in your way it takes time resources uh and ideally you'd want to have a bever method overall I can the list is longer obviously it's again a necessary evil but with machine learning we can do other things than just generate stuff like everyone's usually talking about for creatives we can select them better um when you're approaching designing a system that does this better using machine learning you need to consider what the U UA manager cares about obviously performance but you want to make sure that each and every creative day generated because obviously work hours going into that is tested and one thing that is a major

paino is fatigue so you want to play around that that's like a confusing point in this because sometimes you have these Evergreens that just last forever and you can't explain them so you have to sometimes Force the machine to not get rid of some variations um and it's a question of risk and reward right you want to check everything but you don't want to spend too much money doing that uh bending appear to be the ideal solution for that because they take that into account and they try to minimize your exploration and allow you to exploit it as much as possible according to your goals um so Bandits come from the term one arm bandit which is a weird nickname for slot machines uh and from the theoretical question of Which slot machine should you use on the casino floor in order to maximize your reward obviously you can't know that and it's manipulated in many different ways but

they play around with these things being scientists uh they the it allows you to test as many variations as you want as you can see like the octopus the idea is like having basically as many arms as you want and uh it's unbiased which is much more important than any other I don't know about any other uh Advantage here but that's critical people get married to the creatives they generate together with their creative team and uh sometimes it just doesn't work you don't want to force it um another very important thing is the recret cost associated with running two or more variations that some of them are bad and not Champion ones using classic AB testing you have a pretty long period of time as we can see in the first uh chart here where you're giving everything equal parts of your traffic even though they're not necessarily performing the best one uh when you're using Bandits

think of it as real time AB testing with all the metrics available in memory that's the difficult part about implementing it because you need to hold up a lot of information in memory and serve it very quickly and we're already Limited in programmatic where we have like 200 milliseconds to bid uh but in order to enjoy uh your exploration from the first positive example uh positive sample you want to use it directly so as you can see in the second chart very quickly the green variation starts getting more and more traffic the other ones get eliminated over time uh the third example here uh is another layer of Bandits that we've implemented recently which feels kind of like the end game of this thing where instead of having a single Champion for an entire campaign or campaign group depending on how you build your system you'll have one for different audience

cohorts uh based on a bunch of features I'll get into that later um the great thing about this is that you can constantly add new variations inside obviously the studios we work with generate a lot of them some on weekly basis they want to test crazy stuff they want to make sure they test them quickly and get the insights quickly and we help a lot of our partners with those tests uh I'm going to show you three different approaches uh that play around with this like exploit explore uh concept and how they work briefly so we have funny gifts that help you understand the concept here we have a greedy is that a that's a hamster yeah basically uh the hamster found out that the carrot is the best creative variation that allegory doesn't exactly work but he learns that pretty quickly he just keeps exploiting it uh so you'll have uh smaller and smaller percentage

of your uh budget or your impressions going towards the non- champion and you're going to be very exploitative like the danger here is that you might land on a local Optimum meaning you have a creative that was better initially but it wasn't statistical and you'll keep putting all your eggs in that Bas basket so that's not ideal we do use it for the start of campaigns where we really need to get some uh install going uh more than anything else the two other methods Thon sampling is the next one um they divide the traffic uh according to the performance in relation to the other creatives so if we have one creative outperforming everything it's going to get let's say 90% of the pi the other ones are going to get distributed according to their performance and nice thing about this one is when you get a new arm it aggressively samples it because it doesn't know how it looks like it doesn't know its performance it doesn't know it's world so it samples it very

quickly and this way you can get insights pretty quickly and see if we have actual new champions the last one is quite similar uh it just uses different math like uh the Thompson sampling approach is more probabilistic the upper confidence Mount is more grounded in rules depending on like the size of the data set and everything but basically it tries to minimize exploration on new arms but still make sure they reach this confidence bound that tells you okay I know enough about them in order to see where they stack up against the rest uh we employ all of these in different uh types of campaigns depending on the lifetime of the campaign and like the the app promoting in general uh so here's like a quick uh example of how Thompson sampling looks like as I said the champion would get 90% The rest would get like distributed according to the ranking uh in IPM if it's for videos uh

the problem is that inside this single group you're targeting there are different personas different cohorts even when it's not with Device IDs and it's just different context in the apps they coming from so you're some sub optimal when you're doing this I'll get into contextual Bandits and how they work next uh the idea is to use the same algorithms and just add another layer of data uh that is considered as part of this selection so basically all the features that we use in our classification for installs or for po install events we use them in the creative selection as well so these are things around the other predictions that we have the probability of install the probability of inner purchases uh features about the creative that we extract objects that appear inside it its length its format as well as the publisher so if someone is in a context that's similar to your game if for example you're promoting a match free game you want to treat them one way if they're

playing something completely different or reading the news you want to treat it differently uh it's just a different mindset even if it's the same person um other things we do with machine learning to improve our creative selection and maximize what we can is as I said earlier we enrich the feature set for the creative selection data sets with actual objects from inside the ads so if it's static it's pretty easy to get the objects if it's a video you uh you have like we have like open CV which is like a library that enables you to split it into frames and then extract the things so if there's a sword in it and stuff like that um we don't rely on categories I said it's a lot of times before we can't so because of ASO and switching categories for fun so we extract the store texts and this way with uh vectorizing them we have an embedding that represents each app and we can check the contextual relationship between two apps so we'll have have like different ranges and those are uh also

another part of our features and for the future again on the Privacy sandbox uh we're going to have topics uh and we use those to represent like where the users are coming from some of the topics they're providing are going to be very in depth so this can become pretty close to user representation but obviously much more privacy oriented this is just to highlight how the image extraction works we find the objects that we know are meaningful uh and we enrich it it's pretty pretty useful like the the bump we got from this for Relevant titles where some objects or some um I don't know items are important like midcore RPGs it was significant uh I want to show you a couple of case studies that highlight how this works um so one of our biggest Partners is retargeting is papaya work very closely with them uh and uh when it

comes to real money gaming there's a lot that goes into the users uh recency and frequency like how heavily uses the app and not necessarily just how much money he deposits but when he less deposited uh if he deposited a lot lately etc etc so for these kind of apps that are heavily um affected by this like user Behavior we create a Time series analysis to represent where the user is in terms of his usual Behavior Uh and it effectively splits them into several groups and for each one we selected different creatives uh and what you can see here is that's the feedback we get from our clients uh a lot for retargeting is that we get significantly lower cost per reactivation while meeting the in this case very aggressive kpis because it's RMG they're very aggressive uh another example is from Alien Invasion

ID uh that was recently purchased by crazy Labs uh in this case uh there are mechanics we want to express inside uh the creative selection and uh different mechanics appeal to different users uh and this isn't like the way we designed it it's just kind of how it landed automatically we feed the machine these are the different creatives these are the categories users are coming from and this is kind of what came out you can see it for the IP casual audience they get a simple mechanic for the idle and RPG Corey they get actual highlights that are pretty similar to what's happening in the game it was really nice to see this happening because again it's not done deliberately it's just what the model learned and uh same here we recouped within 3 months and uh yeah the future privacy you can't get away of of this subject in any literally any session you do uh but we've tried already to simulate

how that reality is going to look like an Android an iOS we already have the store description stuff and it's working pretty well um but here we basically have topics that are going to be as I said pretty representative and you still have some way to distinguish between uh users even though you don't know their you know device graph uh the device features if it's the strongest device the most expensive device it's also a huge part of it but other things that you know like the state of mind they're in uh if someone's playing a midcore RPG at 4:00 a.m. it's different from someone playing a midcore RPG at 8:00 p.m. on a Sunday so giving this to your model can still give you some distinction and improve your creative selection um we did some tests retroactively because it's not exactly still available and we tied topics to categories as much as we could and we

saw this distinction happening in a pretty similar manner to the way it does on device IDs we couldn't fully test it yet but it seems to cover 60% of what you would get normally uh we still need to see how it's going to look like but as usual the same way we saw it scan as soon as the signal decreases the prices decrease so everything balances out so we're not too worried about it um a couple practical tips um for working with these sort of platforms that run these test so obviously all the closed gun Gardens run this in one way or another you're not necessarily going to have much control over it unless you're really close to them uh but when it comes to dsps I think it's good to first ask how what sort of uh thing they're using how they're uh what they're doing with new arms or creatives when they need to explore them just to try and make sure

that your things don't get exhausted too quickly um at the beginning I do recommend to provide as many variations as possible these algorithms can handle it and they'll find like the champion that suits their traffic if they're buying inapp in games it's going to figure itself out pretty quickly you want to add new ones progressively and not in bulk because at that point you already have established uh the ipms or the ctrs for everything you don't want to distribute too much and please don't kill old Champions even if you really want to it it really it can mess things up you just want to avoid it and that's it thank you questions if anyone has questions feel free yeah hello yeah um uh nice presentation thanks thank you uh I've used uh uh bandits in the past for incentives in writing haling because it's very

challenging to find for each driver the right incentive um you're tagging the frames uh in the creative videos are you doing any uh meta classification of the whole like you know like a story classification no no too hard honestly we're sticking to what we can do with open source stuff right now uh we delved into manually tagging like the sentiment as much as we can yeah yeah it wasn't ever scalable and whenever we tried to do it with mechanical Turks it failed miserably like you see that the feedback you get from people is mixed even when it's so obvious even when you have those like mobile ads that are obviously appealing to your human emotion of like you know the ones where the window is broken and you need to fix it so you're supposed to be sad you just don't get the feedback you'd expect I guess that's the end game but we're going to have to wait for jna to be able to detect emotion at that point we should probably be scared from gen anyway right yeah I mean but you could do a second level model right you you have all the tags of the frames and then you have to build a

story I used to work for a TV company we wanted to do this but then I left so I was hoping you had done it so not yet it definitely went on the drawing board but didn't really PASS testing okay thank you sure thanks you hi um and good morning and thank you for the presentation thank you I I have one question because you also presented and also showed like it's it's great to optimize creative towards different audiences but how you ensure then also that your your algorithm is not optimizing towards an yeah to have an average best creative for all audiences but really optimizes towards the different audiences because like you showed that the champion is not the average Champion for all audiences but you have really in Creative um tailored Audi so it depends on how confident we are about the user's classification uh if this is a user where again when it comes to device craft stuff where we

actually have more information we know if this is someone we touched in the past and we can affect in one way and we can classify him as let's say as a mid core player right in this case he's going to get the champion that's relevant for that uh in other cases when we're not sure you do get just the average best one it's just the way you set it up uh and uh the thing is this goes hand in hand with the overall classification and there's also rotation between the C creatives naturally at least on rdsp that the same person won't see the same creatives two times in a row so um we're using like we're spending millions of dollars you know on these campaigns we're trying to utilize any scent we spent to get as much signal as possible so obviously when we're exploring a new audience deliberately or if it happens just you know uh implicitly the machine considers that so it it always goes hand in hand with the classification so it's kind of hard to

separate it but I have to do it for these sessions thanks great thank you thank you very [Applause] much