Panel: Ad Creative Testing for UA – A Look Into the Future

[Music] thank you [Music] we're going to be having a really good panel coming our way um which is talking all about your creative testing for your UA look into the future which will be interesting so we have Stephen Bose from Unity Technologies who's going to be our moderator uh then we have Auntie pacquela from small Giant games Misha sirotuke from huge games and Tom Collard from King

so while they're getting themselves all set up please remember to put your headphones on because you won't be able to hear the presentations if you put your headphones on you'll hear my wonderful sweet voice saying sweet nothings to you and it's quite a lot of nothing going on here um so um just while they're setting up as well I might as well just reinforce the fact that this is day one of Pocket Gamer connect which is the best conference for mobile games and we have some amazing sponsors and I do encourage you to go and take a look at all the different stands everybody's there to talk to you hopefully not too salesy but definitely it's a big education even just walking up and seeing everybody at their different stands and booths um so our Platinum sponsors today are Agora apps flyer and app gallery and then we also have our gold sponsors which is share it transperfect gaming apps flyer app loving Unity miloco meta

free verse audio mob quality liftoff agile exola data.ai pangol and digital turbine are we all ready no we're just we're just getting there the most we've got our moderator ready now say Stephen I'm going to hand over to you to kick off our panel session everyone this is not a questions at the end but I'm sure the gentleman if you see them in the uh in the the room today just go and talk to them everybody's um here to share and educate so kick it off guys wonderful thank you thanks Jackie guys thank you for joining us here today um just to introduce myself first I'm Stephen Bose I'm a senior senior client partner here at Unity looking after our user acquisition clients and we're here I've got Tom from King Auntie from a small Giant and Misha from huge so thank you guys for joining us um do I have a clicker for the

sorry there are some slides that are supposed to be on there nothing like Monday morning technical issues oh my God perfect uh maybe I need some more coffee okay is it working yeah no worries so well I guess I've done the intro so I can just discuss what we're gonna talk about first I'm going to discuss the pillars of creative testing um with unity things that we find important and then following that I'm going to discuss um our growth and monetization report at Unity which we published for 2022 and then after that I'm going to use my steam panel here to discuss some topics on ad creative testing how device one

has evolved over the years um we're going to look at some blind testing at blind spots for Creative testing and then discuss some privacy changes and how we're testing with iOS 14 changes and then we're going to look at what the future holds for UA creative testing so I guess I'll just show you my little cue cards everyone I know here we go perfect great so there are lovely lovely faces such in details cool so pictures of creative art testing so focusing on three pillars which is important for testing your creatives um for UA is finding the best testing method so you want to make sure that you're working with the right partner in order to test properly and understanding what audience you should go for um I know I'm biased but I'd recommend Unity ads to do that with um you should also pick the right testing tools so considering whether you should do split testing or creative a b testing and then you also should look at

different optimization events in which you should reference to using adult budgets or fast or slow paced testing methods and it's quite important second pillar is focusing on the data and so picking the right kpis to track for the most part I think for you guys you look at IPM but it's also important to look at CPI for each creative to get a full holistic View and it's also important to look at ensuring and looking at your data and ensuring that you're recording this properly and sharing this with your with your data team as well as the creative artists and in my opinion the most important is iterating with performance artists so there's no point making putting all of this effort and resources and Financial Resources into testing your creatives if you're not going to record this data properly share it with your creative team so that they can then iterate your creators and make new ones built off on that data cool so we published our mobile insights growth and monetization report this year and it's I've pulled out a few important points for us uh or a few important

points in terms of creative testing and for us to discuss so the first one here is on what is the most effective creative characteristics that you should include within your creatives we have found that over 86 of the games that are that are advertised on Unity show True game content in their creatives this is important and we find that it has a has resulted in a higher click-through rate as well as conversion rate because advertisers or sorry viewers of the creatives are more trusting of the creatives because they show actual gameplay we found that game creators that show false gameplay or exaggerated gameplay and often have lower click-through rate and conversion rate because users do usually respond and haste-based to base to switch creatives the second one is that we found that with midcore and hardcore games we found that power-ups and upgrades are super important for that for that genre they result in a much higher conversion rate and click-through rate and then this is something that I guess we can all kind of assume to be true but we wanted to test it loud music was particularly

heavy beat is um has become very prevalent in casual and Hyper casual games and but now it's kind of spreading across all genres and that is associated with a significantly higher conversion rate um what is the most effective way to display game instructions on your creatives so on the one hand while on-screen texts kind of makes sense for you to it cognitively stands for you to like understand how you should play a game on a creative or analysis on video creatives or analysis found that voiceover is far more important and has a much higher conversion rate this is most likely due to the users having a much lower cognitive load and when they're watching the video created themselves now I'm sure you've all seen the real money incentive and where somebody's like jumping around like loads of cash flooding around them or when you got into their PayPal account we found that this that those type of creators have a significantly lower conversion rate and click-through rate it's a 60 lower click-through rate and 24 lower conversion rate and this isn't just for

Social Casino ads we did see it kind of growing in other genres but overall it has had negative effects on those creatives showing gameplay and so we looked I had this assumption and I'm sure you guys know that I had this assumption that showing failed gameplay when a user like usually fails a level typically has a higher conversion rate or a click-through rate we wanted to test that assumption to see whether it is true or not and we found that failed gameplay where the user usually fails or it fails to pass the level and results in a 60 higher click-through rate and a 50 higher conversion rate those numbers are really Stark um and we think this is usually down to why most users of those viewers of those creatives have a takeover mentality where you want to like grab the phone and improve the and show that you can do a better job okay do not show a real person reacting so similarly with the money creatives you probably have seen the ads where somebody's jumping around or like when they've got really upset when they've either fail they've died and stuff but our analysis from the report has found

that those type of ads have have a much lower click through and conversion too okay I'm gonna stop speaking at you but before I do that um along with our report we also um show our findings in soft launching and campaign trends ad campaign creatives and best practices and some advice on in-app purchase and in-app advertising mix along with ad implementation tactics a monetization and performance you want to scan the QR code there you can download the report I'll also have my email at the end if you want me to reach reach out to me and send it to me okay so now I turn to my panel Tom auntie and Misha um first question so leading on your years of experience um I would like to just start off with getting an understanding of how creative testing has evolved over time so Auntie I'm going to go to you first for that one sure uh hey everyone nice to nice to be on the on the panel here with the esteemed host and my esteemed colleagues

um so for me I started doing UA in 2015 um and way back then obviously the industry was in a bit different place overall in terms of what kind of formats we had available what kind of AD placements we had available and that dictated a lot of uh what exactly it is that we were able to do so for example in 2015 uh in like browser Facebook was still very much a thing uh the Facebook app was made was like largely the de facto place to go to also not only buy UA but there's creatives and that was basically on necessity based on there just wasn't that many options available I mean video ads were definitely coming up there were a lot of good vendors working in that space already back then uh but on Facebook you do video you do Banner you test these types of things really based on what kind of inventory you had available and then as few years passed we had the ad event of playable ads definitely changed things a lot in terms of also having to test a lot of different things

and on Facebook you couldn't obviously do playable ads back then so then you had to really go to different networks and go to the place where the inventory was available and whatever the capabilities of said network was you basically took what was given to you uh that didn't mean always that the metrics were accurate or it didn't mean that you were given all the data possible but you kind of had to do with with what you had and then in more recent years not only video has has risen up but then video plus playable as a as a format either in forms of an end card or done just playable as a standalone experience after the video as it is depending on on obviously which network you work with so I'd say the testing itself the foundations haven't necessarily changed but you have to be very mindful of the formats that you have available and that's maybe the the biggest thing has been changed that we have to kind of evolve with with what what is given to us but like you like I said in the beginning IPM obviously has been available for us because we've always had Impressions we've always had clicks

we've always had installs so that hasn't really changed but uh in terms of the form factor uh it has definitely impacted the big big scheme of things for sure Tom does that mirror what your experience yeah yeah very similar um and just from the creative production side I'd say one thing that's changed is that we focus a lot more on optimizing assets like optimizing top performers and creating iterations of those top performers as well as just coming out of new Concepts as well um but yeah certainly from the the actual products used yeah very similar cool cool thank you for sharing that moving to our second topic um Misha I'm going to go to you from this one um is the UA team involved in the creative generation process at huge

sorry all right oh how about now yes wow he's giving all the juicy gossip and only only we heard all right so we got stressed um so yes are you a team is involved in uh in a creative generation process but not so we have few teams that are involved in the creative generation process first team is the creative team itself so those are artists and the creative strategies that analyze their creative performance from the past different look at different tagging but also play games themselves Etc and then we have the the product team and product team may may suggest that you create this based on the product features themselves and then we have the UA team and UA team often suggests ideas based

on exam on events like this right so we have people present here as well where we get inspired we also get informed we have a chance to connect with our peers with our vendors and and then bring this knowledge to to our team for example if we hear that there is a new format coming up we heard that for example Unity is doing video plus playable and this is important information to bring to our team and make sure that we have um ad format created that will support this new new features cool thank you Misha Tom do you think that the UA team need to be more involved in the creation of performance art yeah I mean definitely um like like me as well we have a we have a large team we have a you know uh creative team we have an insights team who sit between uh us and the games team and we've also got uh creative analysts but they're always looking for feedback from the buying team because yeah I mean we've got the most knowledge on what the you know the the industry Trends are and what the partners are doing um so yeah yeah super important for the buying team to be involved too yeah

thank you okay moving on to our favorite topic um iOS privacy changes so how has your creative strategy changed after iOS 14.5 Misha I'm going to start off with you again on this one yes just want to make sure that the work it's working good so for us and I'm sure for majority of other gaming company iOS is a probably majority of Revenue so it's really important to test creatives on iOS unfortunately due to the Privacy changes we have less information that we used to have so for the Simplicity of things we actually test our creatives on Entry today and we will apply the knowledge on iOS unless there is a very specific case maybe involved so answer your question the creative strategy change that we don't really test creatives on iOS at the big scale but we rather focus on Android only and apply those learnings into our iOS campaigns cool thank you yeah yeah I know if I may uh we have pretty much the same approach so we um

we do of course run a lot of ads on on iOS still and and do try to scale there as much as possible but in terms of creative testing getting the learnings Android is definitely um uh easier in a couple of different ways one we can actually rely on all of the data in terms of having accurate or accurate enough attribution um that's all different different conversation but uh having Africa accurate enough data being able to get decent sample sizes with uh affordable enough CPI at least in our our particular case um and uh just generally having the ability to to um still run campaigns at scale and worry about not worry about losing a lot of the installs but yeah or a lot of games though uh that have for example the luxury of sample sizes and and uh are able to get uh installs and cohorts that are large enough uh I'd still argue and that monetize early enough so certain genres for example like Harper casual or or the companies that have

really been able to make hyper casual succeed at scale then iOS testing is very much a possibility for other developers in other genres where they don't monetize super heavily in the first hour of in the first day or two and it has definitely become a challenge okay cool and from Tom from King's perspective how the new regulations change in which the creators are produced I mean like the others we we do the the majority of like the core creative testing on Android so it's not not impacted that um but like you said as well it's important to repeat tests on iOS and that's become a lot more difficult with the the campaign and creative number limitations on the partner um but yeah I would say with uh the scam 4.0 update coming at some point um the four digit campaign identifier that will get and that might allow some some more creative testing um at the very least it should allow a little bit more breaking down of campaigns so yeah hopefully that helps okay we live and hope okay

um looking at what are the blind spots in Creative testing these days so I guess Tom we're going to start with you again on this one I mean we've talked about iOS so I won't know yeah I won't get too much more into that one um but one thing that um that I've sort of noticed as well is that you know one problem of having a very formalized creative testing process is that you're you're effectively optimizing um your assets to whatever you know placement and location is that you're running those tests in um so something that I would you know definitely recommend is repeating tests in different locations on different partners and making sure that those results are indicative of your activity as a whole um so yeah I mean if they are then it's great but if not then you might need to be running more creative tests in sort of live environments yeah that's quite interesting is that something that you do at huge yeah but also to add on this is from theoretically the creative testing process should answer the question whether or not the creative is effective right so you look maybe at IPM or CPI or any other metric that you feel comfortable so if it's effective then

you should push it to your regular campaigns and you should scale and you should bring you the users that attract the right users right but I think the blind spot here is that you still need to re-verify the theory so the ideal process should be that you know it should be effective to attract you users on a high IPM and then it also should attractive attract users that would be very bringing Revenue to your company but it's not always happens and also for example if you go to another country it may be completely different so the blind spot here I would say is that you shouldn't really trust 100 that this creative even though it's effective it will be effective at the high scale or effective in different country or yeah Channel then you tested it on yeah it's always hard to extrapolate to find the inside once you once you move it to existing campaigns cool so uh guys I'm gonna go to all of you on this because I think it's an important question what tools do you use for running your creative tests I'll go this way so Tom um I mean personally I use iron sources e-compression shared product quite a lot

um I think it tends to be you know one of the the strongest performing ones on the market for um actually delivering equal Impressions on a on a app and some level within our campaigns cool um but yeah and I do um we've we've been using a few so um obviously we have we use our MMP which is um in our case it's apps wire but but what we use them for is obviously to rely on the data that exists it's like an obvious thing of course you have a tracking partner but it warrants nothing that you do need to also check your creative level information on that and verify that at the end of the day the retention also works on whatever personal metrics you might be looking at uh we're utilizing two to three different networks um and trying to get an accurate mix of what our live traffic would look like but still in a way that where where the networks in question have the tools in place to actually measure that and then uh finally we are are trying to find the right mix of Partners to allow us to

measure not only the data that we get from an MMP or there we get from a network but then the other creative attributes that are involved in in various sort of basically using tagging resources to make sure that we're also looking at not only which individual file ended up winning but also what are the elements within that file so basically attacking service for that cool is what we use yeah so on top of what guys already mentioned we we test creatives currently on two networks so it's unity and Facebook and uh we either test directly on the networks themself or we use campaign management tools so we use uh two different campaign management tools for our Facebook campaigns and one uh different campaign management tool for Unity so we can do both on the platforms themselves or we can we we use campaign management tools to do that in a way effectively but also sometimes there are automation options that help us to speed up the system cool

okay full disclosure Unity is about to release a creative a b testing product in which you'll be able to a B test your creatives um and all three of the panel here have had Early Access Alpha and beta testers for this product uh so Tom if you could just run through some of the advantages as well as the challenges that you've had with that with this product yeah so far yeah really happy of it so far yeah um we've had a lot of cases where we've wanted to run you know equal impression share on uh on on Unity campaigns um you know not just for sort of you know early creative testing but also on our you know uh audience pinpointer campaign to try and seeing those environments uh whether creatives that aren't picking up are just uh you know objectively worse or whether the algorithm is favoring older creatives um so yeah it's been really really good for us to to actually test this I'm sure products are very happy to hear that yeah I definitely agree with uh with that in terms of the challenge of um because we've been using Unity as well in Creative testing as well as in live

campaigns for a few years now and um the challenge is exactly the the quote-unquote Legacy creative that keeps on winning and keeps on winning and just is at the top of every single campaign but what the new product allows is is for a lot more diversity to the emerging creatives in live campaigns but also in Creative testing where the sort of initial front runner creative doesn't just win out and take 80 of all of the spend where instead we have um good testing results while somewhat still mirroring the actual Market of what are the things that the market demands so it's definitely been a been a a great enhancement of of the live campaigns and of testing and base of ours brilliant cool so if I may add only um I will not be repeating all the all the benefits that there are but I think the things that um well you guys will add eventually but we we don't have this campaign in our third party campaign management tool because the the API is not yet open right so this is something that you guys will add

for sure but as of today we can only use this a b testing campaign in unity dashboard and for some sometimes we would like to use it in in our partners dashboards as well so that's just something that is not yet available I assume for sure yeah I'm sure it will be soon yeah definitely I'll make sure I bring that back to the product team I think some of them might have heard it already I know uh thank you guys okay let's look at the future um how would you like creative testing to to evolve to in the future sorry um you want to say that one we should I can take that so the way I would like to evolve is is quite big and I will do my best to make it work but basically uh my ideal creative testing process would be that the artist the main goal for artists is to generate a great looking creative that will obviously attract the right users and and the goal of UA manager is to scale the campaign at the right budget to make sure that they are profitable right but the creative testing is this thing in between that

not always people like to do because it's manual Etc so what my my vision in a way of how I would like to eat to look is to make it automatic automatic in a way that when creative art is finish the piece of creative could be the the video or the playable both they drop it on some sort of Drive let's say it's Google Drive and then the system picks it up automatically and either create the campaign automatically in unity Facebook or any other tools that you choose could be also DSP or Tick Tock uh create the campaign automatically or adds this creative to existing campaign automatically and then pauses it after a certain amount of criteria yes could be a 10 000 impression or fifty dollar spent or seven days running you you pick it up and then sends automatically email to your your peers right he says like this is a new creative that has been developed this is the link on the drive and these are the initial creative testing results on Unity on Facebook on tick tock on anything else and then a media buyer can look and say oh I managed Tick Tock I'm going to grab this creative because it has the ipms that I needed to have or I will actually not

touch it because it's not good but maybe for the unity campaign management people is good so that's how I would like it to look and um and I'm like I'm gonna start working with my team to to see if we can push it that far Auntie would you like more Automation in the test first of all I'd like to sign up for the SAS tool that you're going to make in a few years uh no that sounds really good um so what I would maybe want to see and I I don't know why I just keep talking about the mmps on this channel panel but uh what I would like to see is maybe um just more tag related elements and more things that are that each and every MMP could pass to each and every client uh just as a default so not only looking at the campaign at group creative levels but also adding more Rich data in terms of video length video orientation um for example uh music no music are they using audio or not uh just very

foundational pieces that we're all sort of tracking individually but it's very hacky in a sense that everyone has their own little Google Sheets set up where they are putting the things that are attached to the file names and all of that but that needs to be part of um yeah of more and more Studios just coming in out of the box so I really think that that could be there could be networking leadership there could be MMP leadership in terms of just allowing more and more Studios to to get a hold of that data in a more approachable way which currently we're extremely limited so far but that's maybe from an industry-wide point of view that's what I'd like to see cool is that something that you would do a king or or what are your thoughts yeah right or automation would also be great uh make your life easier yeah yeah and also sort of you know more data as well from the networks on uh you know on our live campaigns things like you know IPM data can be misleading if you're looking at it in just a in a sort of you know a normal live campaign um receiving sort of more feedback from the partner from the networks about you

know what the algorithm's estimating for each for each creative this uh because we may see one credit for the higher IPM much worse delivery and it turns out it's only because of the sources it's been serving on all the types of users that have been running on so yeah more more data from the networks on that side would be great cool thank you and I had to finish um if you have to finish out finish us out on a nice question uh if I can go this one here so looking back on your wealth of experience we're in UA campaigns and if you could offer a bit of advice to a new UA manager or somebody a little bit less experienced who could be in the audience what would that bit of advice be I'm gonna go to Auntie first because you're looking at me I feel like you're gonna have just been courteous Oh I thought you'd be like you calling me old no well that I had that in the back of my mind but I decided not to verbalize it uh no um basically to piggyback on on the

point that I just made um make sure that you have your you trust your analytics and you trust your game team in terms of having a data set up that allows you to confidently test and measure and do whatever it needs to not only scale your game but also make sure that retention doesn't drop monetization doesn't drop and that General and prices don't go Sky High those are the main things and then simply enough use creative tagging as much as possible because it's free for you to do to follow all of the elements that are within your creative uh it's just a simple setup in in basically any any sheets based Tool uh do that tomorrow if you're not already doing it today Tom sure um yeah I mean it's an obvious one but I would just say streamline your processes as much as possible because you're going to be doing this a lot um you know investing a bit of time early on in making future processes as quick as possible is very very worth it

um and then also you know um think about you know why you're seeing the results you're seeing you know and trying to understand you know what's causing different creatives to perform at different levels can and that that means that you can be much more involved in the uh in the Improvement of those assets and the conversations of your your creative team and those producing the assets so leaning back what you're saying earlier how the UA team should be as involved as possible in the in the creative uh creative generation process I'm gonna have a less process oriented answer but I think my advice would be make sure you have a budget for Creative testing uh I'll I'll tell why because creative testing is costs money right and if you have a budget General budget for for the user acquisition campaigns and it will come probably from this budget and it will obviously not help you to generate a good Roi if that's your main goal right so if you test creatives in let's say tier three country then it may cost you anywhere from 20 to 50 dollars per creative at

least but if you touch creatives in tier one countries could be few hundred dollars right so then if you have five creatives that's already probably goes into thousands per month so make sure that you actually have a budget that and it's clear whether when where is it coming from if it's separate creative testing budget that you agreed on uh or it's coming from the UI because if UA budget changes then your creatives process could suddenly be gone if it's if your if your budget for this specific Network would be lowered for example next month right so um prior to build all the processes make sure that you will actually have money to spend on Creative testing because you don't have to have creative testing actually creative testing is a tool that may help you achieve your goals but you may as well achieve it in in a different way so that would be my advice cool thank you and I because you said process oriented I'd also like to throw in one thing because we are talking about ads that actual people see on their actual devices at least I hope in most cases uh don't be afraid to also test not only

iteratively but test completely new Concepts and just take basically don't just copy what the what the competitors are doing uh feel free to of course look at the unity competitive insights that I mentioned there in the beginning send Stephen emo who sent it to you um but also uh come up with new Concepts please do because I'm I'm at least done with just seeing the same and same ad frame by frame done by different uh different companies just in a slightly different way and you might just see that that being Diversified also from an artistic point of view really makes a difference I love the fact that you said performance artists in the very beginning in terms of the people that that actually make the ads and it is exactly that it's jumping through different types of Hoops and really trying to understand that what are the things that could make people click on this ad well at least think it should be being about being creative and and really grabbing people and grabbing people's attention not just doing the

same old thing that everyone else is doing for sure yeah I totally agree with you you cool I think that brings us to the end so I want to First say thank you to my panel um my email is on the screen there if you guys want to reach out to me if you've got any questions on unity's Creative IB testing products I just want to talk in general I'll also be around the conference if you want to tap me on the shoulder I'm sure these guys also won't mind you tapping them on the shoulder and if you scan the query code there you'll get access to um unity's resources so perfect well thank you guys and thank you all for joining us here thanks everyone thank you thank you thank you everyone that was really really interesting I'm sorry we're rushing through we don't have a lot of time for questions but I will say try to grab them at some stage if you've got any burning issues that you want to discuss especially the new testing for immunity I think it would be quite interesting [Laughter]