UA Creatives Testing | How Traplight Games Optimizes Mobile Ads for Battle Legion

all right hello everybody and welcome to ua coffee talk given changes in the mobile user acquisition space creatives have become increasingly important to drive competitive advantage and so to speak with us about their methodology on doing creatives testing we have with us today a few folks from trap light axelly vertanen elena and philippa derose who will be presenting actually the first time we have kind of a presentation uh in in this in this podcast but presenting how they actually conducted a creative test and also with us today is john lau from scopely to also ask some questions and to give a little bit of feedback on what they've been doing but maybe we could start uh elena and axelli if you guys could tell us a little bit about yourself and flippo i mean you've been on this podcast a bunch of times before but you

can go ahead and introduce yourself as well but go for it maybe starting with you elena sure yeah hey everybody my name is elena wattenan and yeah i'm working as a ua analyst at trap lights and uh been here for a year pretty much exactly a year so yeah this is the type of stuff that i do there that i'm really excited to show with you as well like what kind of what kind of insights did we find with this study so yeah exactly hi i'm actually i'm the lead marketing artist at trap light games and i've been with the company about one and a half years now and for a little bit over a year we have been working on the current life title battle ranger all right go ahead philippo filippo hello uh chief growth but really just make sure that uh elena and axley

are are happy people and they feel immensely gratified for doing all this really cool stuff that they're presenting okay great well excellent elena go take it away let's uh it'd be great to see what you guys have done on a trap light and this is for battle legion right that for that game that's right yeah right so we we set out to [Music] understand the impact of a creative duration uh in the performance of our marketing um and indeed uh we are working with battle legion at the moment it's a mid-core free-to-play mass battle title on android and ios and uh our core audience is uh around 25 to 45 year old nails and uh here are some of our key geos the the usual suspects so little bit of background um in this uh ever-changing landscape with

uh creatives uh slowly taking uh the lead role in performance marketing you know tracking is going down automation taking over with various level of success um we really wanted to uh understand uh creative performance in a uh very specific and deeply regarding the duration so we developed a modular creative production pipeline in order to do comparable creatives and we wanted to understand the role of duration and resolution uh in different placements and of course uh discover how creatives can be used to influence ad placements so we created a total of 64 different creatives in four different aspect ratios and 16 different durations and here is a example of the shortest 7.5 second creative it has the standard beginning of five

seconds and the standard ending of 2.5 seconds and they together form the the shortest version and we did them up to 45 seconds of length and we always added a 2.5 second content block between the standard and standard beginning and ending to form the next length and all the way up to 45 seconds when it uh already involves plenty of different value propositions and as we know that the people watch videos whenever it's a pretty short time we think that the amount of value propositions didn't really impact the outcome that much and we will go to the details with a little bit later so the campaign setup um we ran all of these creatives on automatic placements across all of facebook inventory it took about three months we gathered

almost 3 million impressions uh 3.6k downloads and the total spend was 44k so uh you can briefly calculate our cost per install if you wish i'm not going to say it out loud and test run in the audience of a male 25 plus worldwide on android only um you know they value a very base to work like one percent from our uh previous learnings uh with value optimization and in a a b testing set up five creatives at the time so we ran like a multiple of uh tests back to back and uh with the highlights of high level results um these uh uh studies um revealed some interesting new stuff but also confirmed our previous learnings that we have been uh able to identify just looking at our legacy data set

so with this uh we focused on share of spend ibm and watch time among ad placements across different resolutions alterations and resolutions and from here i will give it to elena for for some more detailed results any questions at this point sure actually maybe i could ask a few questions so like in terms of like the rationale behind testing the duration and aspect ratio specifically what was kind of like the motivation behind testing these two specific aspects of creatives and what what are some other things that you could have tested well of course you could test every aspect of of the creatives in in detail that's given but um as i was hinting earlier we saw some interesting grouping of data in uh in our legacy data set that alien was looking at yeah and we saw that uh some

uh durations were outperforming some facebook's uh so-called best practice durations uh in certain uh placements uh and therefore aspect ratios so uh we then uh thought that there's uh there is some learnings to be done with this specifically and uh and went for it so this kind of came from elena seeing something in the data and then based upon that you had a hypothesis that hey maybe if we actually try to test more the kind of duration variation because of what you saw in the data then maybe you could drive higher performance for example and then yeah yeah sorry yeah that's exactly it and uh what happened was uh of course the date that i was looking at was really messy because it had very as a very uh large scale of different creative concepts so it wasn't really comparable so that's why we really wanted to make this test so the only thing that we're looking at

is duration and resolution so we can get some clear data so that's the background right and then in terms of testing i noted that you guys did value optimization on the android side we don't have that on the apple side anymore but is there like how do you guys think about these creatives on the apple side do you think that testing on android you could use like the the similar effects of what you're seeing on android would then replicate on the android or on the apple side as well well for our product um precisely we see that the the performance is pretty comparable between the platforms or like we we saw before the changes happened and of course like these uh specific results they really like originate from the from the user behavior level that the the user like faces these uh creatives in a certain uh place in the facebook uh

uh like platform uh there are certain limitations uh regarding skipability etc and this are all shared across the uh platforms okay so i think these are pretty comparable yeah yeah okay so talking about our results at a very high level still i want to take a look at ipm at first so it was chosen as one of the key components of the analysis as a as was mentioned previously and uh this graph shows how our ipms j changed by each of our durations in all of our four resolutions so that's what the lines represent and we can see here that ipm increased as the video duration grew in all of the four resolutions that we tested and uh but this is of course looking at all of the placements combined so looking a little bit deeper we could see that the ipm actually varied very heavily between ad placement so next i want to talk about ad placements

so so yeah this is a look into how our spend was divided in the test based on uh based on the uh different placements so this is of course looking at all of the resolutions combined and here we can see that our top three placements were audience network rewarded video facebook feed and facebook in-stream video this is very much aligned with what we see in our regular campaigns as well but uh looking at each of the each of the resolutions in more detail looking at the data in more detail in the next slide here we can see that there are some differences so for example i've chosen some few i've chosen a few highlights here to uh to show you and uh for example we can see that uh the four to five format or also called the instagram format had the smallest share spent going to audience network out of all the

resolutions resolutions that we tested but it was the best for a facebook feed and these differences are explained very easily by just how much of the screen does the ad cover and just how natural it is for the viewer and uh that's also what explains how the portrait resolution got the largest share spent going into instagram stories again very uh easily explained by the uh nature of the format because it covers the whole screen in instagram stories and that's why it was the best best format for that and we didn't really see much spent on instagram other than the portrait format for landscape here we can see that audience network really dominated the share of spend and uh that's because it's really the only placement where landscape is suitable or where it's optimized for displacement so so it actually attracted over 60 of the spend

of the entire landscape format okay so looking at how the share of cost was different in each placement here we can see how the video duration affected that share of spend so looking at the square format here we can see in the graph that the share of spam that went to audience network grew as the video duration grew so similar trend to the ipms that's what i'll talk about a little bit later but yeah here we can see that especially like longer videos we're much stronger on audience network and for facebook feed we could see the opposite trend where shorter videos got a lot a lot more spent and uh longer videos did not much not did not get much spent on facebook feed at all and then in terms of our third biggest placement in the test facebook in-stream video we can see that its share peaked at around 15 seconds or 17 and a half seconds and this is because again the nature of

the placement facebook in-stream video is a mid-roll ad and it's unskippable until the 15 seconds so clearly we can see that that is what the algorithm is optimizing for and after that as video duration grew that doesn't get much spent anymore so for comparison we can look at the landscape format so here we can see the difference in the share of spent so audience network really dominated here we can see that the share spend that went to it grew really uh really quickly and uh we could still see the similar trend with facebook feed that shorter videos got a lot more spent and the same trend for facebook in-stream video as well where it peaked at around 15 seconds so so overall like we could see the similar trends based on each resolution but they were individual differences and we could also identify some thresholds

like so uh where like we could clearly see that there's a rule in the algorithm that oh this resolution in this duration no longer gets spent at this placement so it was interesting to find out these little data points but yeah we can talk about the next topic and rpm which is what i already talked about previously we could notice that ipm grew as video duration grew but actually looking at it in more detail we could only see this in one placement in audience network actually so so we're wondering so why is that and this would be because this is first of all the longest unskippable placement at 30 seconds and after that the user can close it but also viewing the ad is voluntary because you want it to get a reward so what this means is that most people who actually watched the ad they actually stayed for most of the ad

what's what this leads us to thinking is that longer you have to watch the more likely you are to install and the reason i say have to watch is because uh it's a really big difference in if the ad is voluntary viewing or if it's unskippable because if it's voluntary viewing people only watch around that one or two or three seconds so that's really why why audience network is the only placement where we can see these increasing rpms because it's the only network where people really watch so uh yeah quick question and for comparison so if i'm looking at the landscape line the yellow line um skip the uh ability to skip starts at 30 seconds and that's probably where we see that drop off right around like 32 when they realize they can skip and then interestingly enough your ipm starts to recover and go back up for those who actually

end up watching this for the full length of the video i'm assuming that then yields for a much higher intent user by that point yeah yeah that's an interesting finding and uh next i can show you a little bit about our watch time we could see that actually quite a large portion of people actually stayed for the entire ad even if it was over 30 seconds and they could have closed it that's that's a good question but yeah uh in the next slide we can see a comparison of the ipms uh in the different placements so our top three placements and audience network is the only one where we can see the increasing ipms so the ipms remain pretty stable throughout all of the durations on both facebook feed and facebook in-stream video and this is of course due to watch time so looking at watch time we can really see what happens here so in this graph i have put the average

watch time by placement in seconds so the gray line is the actual video duration and then the other lines represent the average watch time in each placement so i think for example the the voluntary viewing placement so what we have like facebook feed and and other feed type placements we can see that the average watch time is only around one to three seconds and uh it's very different from for example audience network where we can see that the average watch time follows the video iteration really closely until the 30 second mark where people start to drop yeah the outlier here is the facebook in-stream video and which we can hear here see that uh the average watch time for it is around 8 seconds out of the 15 second which is the unskippable amount so a lot of people leave before the the unskippable ad even ends and another way to look at it and see exactly where people start to drop is the share of completed views in

this next graph here we can see that for example for audience network in the blue line we can see that ninety percent of viewers watch the full ad on this network if it was under 30 seconds but after that this share starts to drop pretty heavily but still even if our ad was the longest which was 45 seconds still 25 percent of people watch the full thing which is significantly higher than in any other placement that was interesting to notice so you know for comparison you can really see that basically all other placements uh the light is at the very bottom meaning that very few or not no people watch the full ad and we can see that for the facebook instagram video that after 15 seconds is when the untip unskippable video ends people also leave important thing to remember here is of course that watch time doesn't

directly lead to profit we're not optimizing our campaigns based on watch time either because a user can watch the full video and still not be interested or then we can get a really valuable user by only watching like a few seconds so that's not like uh one of the kps we were looking for here but watch time is able to tell us about the background for some of the other kpis for example that i showed previously with the ipm and how that contributes to watch time as well yeah then moving on from watch time i've compiled some of our main findings here really summarized all of the main points into the three main findings so first of all this study gave us some insight into facebook algorithms mysterious secrets so as i was saying we detected some thresholds first spent on ad placements based on resolution and duration and uh i could say an example of this is that

we noticed that the 45 formats the instagram format gets like no spend at all on audience network or very little spend until 30 seconds and uh after that it starts to get a lot of spend on audience networks so it was a really clear indicator of like a threshold in the algorithm of how the spend is uh distributed and that's an example of some of the stuff we found out and secondly which is probably the most important finding for us as a company is our preferred placements and durations so we found these ideal sweet spots of combinations of duration resolution and placement and uh this can we this can be used in our creative production and campaign management and uh thirdly what i wanna bring up is uh we got some detailed data on ad viewing behavior that we haven't really looked

at before in this detail at this level of detail so founding out that watch time is not dependent on resolution at all but uh but also it was interesting to find out exactly the extent to which it's depended on the placements and how that can contribute to performance so so what can we do with these findings how what can we learn so i've compiled like a three top three things that we can use these learnings for first of all i want to go back to the creative modularity that actually was talking about this new modular creative production system that we developed it really helps us to improve our speed of things it really allows for quick iterations and overall expert at the creative production process greatly but also learning about this uh

learning about the facebook algorithm secrets and how it was influenced by our duration resolution testing we can really use this data to leverage facebook manual placements in the future but also steer and manipulate automated campaigns with the help of creative production so this is really the top three findings summarized so yeah if you have any questions then sure elena i actually have a few questions uh could you could you dive a little bit deeper into the creative modularity system that you guys have like how does that work and and you know what are the if you can dive a little bit deeper into like the benefits of that yeah yeah i think actually it's the better one to reply to this one that yeah i think he can explain yeah sure so that was the um what i was like in the beginning of this presentation showing briefly uh and here we have on the screen these uh modules so we have these uh standard beginnings

and endings which are just a single uh already rendered out a couple of seconds long video files and then we have a bunch of different so-called content blocks and these are also already rendered in the sound transitions fades are rendered in and there is also a fade in opacity transition rendered in so uh these are extremely easy to use you just drop them into the timeline shuffle them around and uh exports got it so basically if i'm understanding you correctly the way that you've developed your your content video is to create specific sections and then you you you modify the duration of the video just by picking specific you know sections to include and then you can remove sections to make it shorter add more sections to make it longer is that correct yeah exactly okay

great and elena maybe another question i i have for both of you is that so you had a number uh in terms of some of the graphs and some of the data that you collected around you know so some of the front-end metrics like ipm and wondering you know still on the goo on the on the android side whether you looked at more downstream metrics and whether you saw any differences relative to like you know the front end versus the back end metrics yeah definitely uh we took a look at all of the regular kps we also look at in our regular campaigns so of course the russ has an uh imported role in that but also updale and we actually even looked at retention based on this but without no like significant differences there but uh we used all the other kpis even the later funnel kpis to really determine our like best best durations and resolutions and uh that's really the

part that you really don't see in the presentation but we created another like uh entire table for that about other like best durations and best resolutions so for that analysis all of the other kpis really mattered as well of course profit is uh every question about uh how you guys intend to use this moving forward uh as we all know like the your ecosystem is like getting more and more privacy focused and so getting a creative level reporting is largely gone depending on what type of optimization you're using for facebook specifically if you guys are let's say trying to tailor your creative mix based on what you believe historically performs you may or may not be able to actually get that reporting post installed data so how will that impact your ability to use this kind of methodology moving forward

well yeah it's a good question it definitely is just the one sad truth about our field is that things are changing so fast and things that were were still uh great month ago may already be changed so that's just always something to keep in mind but i think uh just comparing the data we got from this test with the data we got from the previous analysis looking at our legacy data we could see a lot of similarities so we believe there are some like truth to this that will stand the test of time but still of course things are changing and uh it's more difficult with the law to summarize uh my understanding of what you guys are saying is that you guys will continue to run this based on your confidence based on legacy or historical data um to a certain point right like but but for the time being you will continue to leverage kind of your past data to dictate how you guys will actually

um influence the creative mix for example if you were to run like a chipboard yeah yeah might not have that visibility okay yeah yeah no that's true and um i think uh the findings from this this studies especially we're going to be using in the near future but also this allows us especially the creative modular system allows us to really uh test more concepts like even faster than before and we can test like different beginnings for our creatives and stuff like that so it's uh the testing is ongoing it's not like no we found out everything about our creative creatives right now and this is the truth no it's still of course it's ongoing but john maybe i could ask you like what what would you recommend in terms of like as as you know we we get less tracking capability do you focus more on front-end metrics

or maybe do you try and with your non-lap traffic just kind of assume like based upon a cohorted day that that's kind of like and if i guess you'd have to run you know you'd have to you might have to run your campaigns kind of serially or one at a time or something and then try to estimate what the downstream stuff is or like what would you recommend can i can i just quickly echo in before john i'd love to hear john so what we've learned sort of from a more distant perspective high level perspective is um also from like previous experiences with other games there seems to be um a truth in the fact that the more core the audience the more core the product being advertised the more consistent behavior is on a channel like facebook or even uac between the two platforms so uh there is

that's a bit like what actually was saying before right that there's actually more difference in behavior of the users between platforms the more casual the gang is um so we have an advantage because obviously with trap light we focus on mid core for a certain kind of audience yeah um so we're able to kind of bank on uh legacy knowledge more um unless of course we're treating a new channel let's say for example tick tock that's a whole different user interaction um but we're collecting as much data as we can with this kind of experimentation to sort of gain the confidence and infer that you know even as facebook privacy evolves or other aspects like idfa et cetera um we are sort of inferring that you know things will sort of stay the course uh unless of course we suddenly see the

performance uh crippled and tanking then obviously something has changed but we've seen a certain consistency right which is ultimately we in in the first set of data that elena was observing oh look there's a pattern here let's just check that this pattern actually does exist and we went a little bit more deep dive with this work to validate that there is a pattern so um it's it's kind of you know i guess sort of saving the coins for the rainy days of further privacy changes that will also happen on google et cetera and again i think it's more of a function of the product and what is being advertised to what audience more than you know giving us that confidence to also go into triple a and use certain kind of durations and formats

yeah i i think i think this is very smart what you guys are doing because you essentially you guys are utilizing the fact that you guys can still access custom audiences to benchmark um and i think that's great because as as felipe said like in the near future like as things get more uh privacy implementation where might potentially remove the ability to for you to target custom audiences or say the cost of running custom audiences are going to be increasingly more expensive as we already seeing when people run you know one percent lal you know purchasers so so and so forth you might have to be at the point where you're relying on creatives and creative themes and durations and orientations to do the targeting for you right so what i find interesting about what you guys did is that since you guys ran this experiment you now have

benchmark data so let's say in the future where you know you can do certain levels of targeting even on android um you can say okay i know that my my audience will respond to certain types of themes certain type durations and you're going to utilize that to dictate who will get at exposure because the algorithm will always optimize for either engagement or ipm or conversion events right triple a you could still run conversion events so um i think this is very intelligent what you guys are doing and uh hopefully when you guys are thinking about like when you guys are running let's say a scan campaign or when you guys are running a um when android you know becomes more private uh you guys can actually take these learnings and apply them assume that's what you guys are doing yes that is 100 spot on and it kind of connects the dots and echoes a

discussion we had a while ago with mr joe kim about like you know potential impacts of idfa changes before it even happened um the the creative is ultimate the ultimate entity that we control less and less everything else right but how can we control it that's that's the question right aside from obviously varying colors and things like that are there more intelligent ways of learning how to manipulate creative to you know to the best of our performance basically so that's that's what's really important and i understand okay like we have control over duration let's let's learn more about durations uh we have control on like sizes and shapes let's learn more about those specifically so it kind of ties in with the high level strategy of okay we can only control our destiny with creative now how do we go about

that because indeed the the original sort of hypothesis is it looks like we're headed for a world where ultimately creative will be uh filtering your audience and i went as far as saying actually potentially maybe in the future due to you know how restrictive things are uh that the more sort of eye-catching loud creatives which generate a lot of attention and get a lot of installs might not be ideal anymore in the future because you can't target those and so all you're getting is just masses of volume but not necessarily the kind of quality that you need to perform so there might be a change also in how you know there is the whole topic of fake ads uh but we won't have them anymore in the future because you have to really filter

your audience by using a good creative strategy maybe one other question i have is elena and axelli when you think about the specific kind of content do you think that it because the way like the approach and kind of assumption here is that you've got these different content blocks that you can kind of put in and modify stuff but do you think that based upon a different theme or different kind of content that you could expect these kind of results around duration and placement to be kind of uniform and then the one kind of content i do worry about a little bit that may not fit this model would be something that's more narrative driven so like you know like some of these creatives that have become viral like you know oh am i pregnant or you know you that the but there's like a story line behind you up to the point about you know wondering whether you're pregnant or not but like i don't know what are your thoughts in terms of the content impact on some of these results

well we saw some um uh like anyways explaining uh on some similar results compared to our legacy data set which includes includes like uh tons of different uh ad ideas and concepts and uh by just that we're able to prove that there is some like a universal trueness to the results basically um but do i go to your next point you are exactly right that the this kind of production methodology doesn't uh apply to all types of uh ad videos especially the like uh narrative driven or like your a little bit fancier more expensive uh trailers or also uh cannot be produced this way um but this is what kind of like sets the foundation all right and uh is the bread and butter to to the ua activity in a sense right and maybe right now accely maybe to your point one

of the things you could potentially do is to start with this kind of approach to kind of see what the optimal content duration is and and kind of format and placement and then based upon that then you could say okay actually if this is the right if this is the optimal duration and placement like landscape on whatever then you create your narrative driven content against that optimization for example so it could kind of be a two-step process potentially yeah yeah precisely we identified those sweet spots and are now already almost exclusively using those uh learnings and uh positive results from from that so i i wonder how this will actually change the way you guys approach um how you guys tap in different sources of inventory so using facebook as an example majority of advertisers are getting their inventory on certain placements right so we're talking primarily

ig feed news feed and audience network will probably make up like close to 60 70 percent of most advertisers inventory now when you're thinking about how you're doing this modulation on the creative side are you guys giving thought on like okay maybe this can open the door for me to tap into other forms of inventory that i historically hasn't really bought a lot from but maybe if i create the right ad for the right placement i can actually get the results that i'm looking for yeah that's a that's a very good point and uh um definitely like this can be used uh to find uh such a new placements or like you start going deeper into some uh placements that where we didn't attract much spend with this test uh by just ruling out uh these uh that we know that the this uh creative uh now works for for instagram

um like uh stories for example uh we can rule out that the uh it probably doesn't work in messenger so the other way around then uh we can rule out like the vertical aspect ratio and uh and go for for all new ideas uh so yeah it's a it is a process and a long one are you guys considering running similar tests on other platforms such as like on uac or app campaigns yeah i actually um before we ran this test when i took a look at our legacy data i did it both on facebook and uac so uh there is some bit of background already for the google ads test but we haven't run a test this extensive there yet but i think that's something that we definitely should do i think actually

google ads is our best uh best network at the moment so there's really definitely good reasons for us to investigate this there as well because of course the same creative doesn't work on facebook and there it's very different and uh that is something that we want to test in the future and that's a great point you bring out just to echo elena and answer your question uh john and the previous question as well um it is giving us numbers and data to kind of re-prioritize activities on testing more video networks uh in a certain way uh which we had kind of postponed uh but we are definitely kind of reprioritizing it and testing on those as well as soon as we have bandwidth because it's given us enough reason okay

so i just have one last question if you ask elena and xle if you were to go back and rerun this test is there anything you would have changed or basically did is this the way you would approach it again well from my side uh i think we did a pretty good job for the uh for the resources uh that we have at hand and uh i don't see much uh uh places where to improve at uh at least on the creative side like uh of course like uh uh now knowing um uh with a bit of a longer uh cohorts uh we we might have considered google first but uh of course facebook is a great option there's so much uh inventory available that uh i'm very satisfied with this okay great yeah i think what what comes to my mind of course i would

love to run this test on ios as well but at the moment that's not really uh useful let's say because we really need the visibility we want to get those later funnel metrics that just aren't possible for ios anymore so uh that's unfor unfortunately something that uh we weren't able to do but ideally we would want to test this on ios as well even though we have noticed that our creatives work uh quite similarly on android and ios but still this is something that i would like to test but yeah at the moment not possible maybe not ever but we are testing an incrementality tool um and perhaps uh we will consider expanding the use of it for for this for purpose as well uh because we seem to be getting some pretty uh good insights from you know looking at ios more as just an incremental um

mechanism i guess can't talk about which uh tool we're using though that would be promotion okay one last question for me then is uh you ran this test on android how would this or does it impact the way you approach ios in this new ecosystem yeah that's a good that's a good question um i suppose uh not to not much like uh we we see and then we believe that these results are are very useful for ios as well so uh we have rolled out uh some of the learnings to our ios uh facing campaigns uh in terms of the creative duration for example um i'm not seeing like a good results but then if we would run the test of course uh fully again on ios side we would need to then uh

just to rely on the very early metrics like the picture rate is a something uh predetermined for creatives ibm and then compare compare that to to this uh data set and see see what is what is what okay great well i think those are all the questions right so definitely traplight philippo elena excelli thank you very much for your time and for sharing your results filippo you know if you guys run any more tests let us know it'd be great to have you guys back on and with that uh for the audience we'll catch you next time very and again thanks so much for sharing your results thanks everybody thanks for having us i vote that next thank you [Music]

[Music] you