Outfit7’s Nina Oblak & Kayzen’s Ryan Goeden On Harnessing Split Testing To Develop Winning Campaigns

[Music] hello world this is pg. Biz of course and a big welcome to as my co-host Brian baglo would put it our lovely listeners trying to get my Scottish accent in there but he does it so well and yes he's not here but I'm not lonely in fact I'm far from lonely because I have a team team of experts today I have two unique perspectives on how you can harness AB testing to develop winning campaigns and game Concepts so let me introduce them because I'm excited we have Nina Oak Nina hello hi great to have you and your first podcast so this is exciting for both of us you are an ASO team lead at outfit 7 and for people who don't know Nina you started off your career at wuga in gaming then joined

outfit 7 more than 5 years ago started as a UA manager and then moved to ASO so you've seen it from a number of perspectives and we all know UA and ASO are so intertwined and what you don't know about Nina though is that she wanted to be a sports commentator and call the shots in her team now she's got team of ASO experts so I think Nina we can say you have achieved that yeah thanks it was quite the right right from going from this idea of what I want to do to where I'm now and you say it on your website as well a dynamic force in Mobile gaming with a portfolio of chart topping games 20 plus games now downloaded worldwide over 20 billion times we have a lot to learn from you and we're going to hear another perspective because we have Ryan gayon he is business development lead at Kaizen great to have you yeah likewise

nice to be here Peggy uh this is also my first podcast so quite exciting oh wow I would have never thought that Ryan because I've run into you with so many shows and I have to say that you have it you have the ability to communicate great to have you a little bit about you as well you have been in Mobile advertising for over a decade previously working in aobi kachava and hey a little bit about you when you're not working when you're not doing what you're doing which is working with major app advertisers on UA and retargeting well you're giving yoga lessons and giving good energy as well yep I'd say so or at least I try to it's the aspiration um but yeah I'm a qualified yoga instructor and I teach a class every Saturday so you have good energy and we want good energy so it's a good setup it feels to me like you're setting off some Sparks so that's it we're fired up we're going to dive in I want to start with you Nina

what took you on this route when did you say to yourself that's it AB testing is going to give us more than just a decision on creatives it's going to give us much more about our strategy yeah so I would say we first started using this approach when we had a question of what store assets to use going into soft launch for one of our games so the store assets these are like icons screenshots it can also be a um name of the app um and we know that although like native tests on the stores these are Google experiments and apple Apple's ppos they're great but you cannot do them if the app is not live on the store yet so and we had this problem already in the production uh phase um so what we did is we created these fake Pages internally um in outfit 7 um that essentially they look and feel like the real Pages um but they are not you know

you open them in browser so and we um connected them with the ad campaigns and this is how we drived traffic to them and then we just observed what players are doing on those pages and we also observed like the first part of funnel but maybe I will explain that a little more into details later um and because we are very lucky and we have like a big um user base we have a lot of games we have a lot of diu Miu so we we were able to use use this internal cross promotion Network to drive this traffic um but we have a limit here right because this is only like cross promotion traffic is only like a part of distribution but we could not cover organic traffic or not even the players who are playing other games beside ours so here is where Kaizen came into play as well because they helped us reach those users um and yeah like you said it's really it helped us answer so many questions not just from store optimization but

also like for verifying Concepts or defining what kind of art we use and so on there are so many use cases we were able to cover with that that's what's interesting about this is that you're able to stretch this model this approach to AB testing you set it yourself testing content creatives concept so many different use cases maybe even a couple that you came up with that we're not always thinking about yeah it's true I mean we we really we fastly recognize that this is not just something that we could use for the distribution purpose and for the optimization of the store assets and the conversion rate of dis imization there but it can be used in any cycle of the game production as early as the idea for for example when you get the idea of how the game mechanic should look like or anything like that you can just um test that or for example the art of the game is it 2D is it 3D is it more real or more cartoony and then um Market ility of the

app will the users like when we present them the ad and then take them to the stores will they even react to it will they like the ad will they download the game so this is everything we can test very early on with with this setup and then later in production we can see like for example which which features are more like um attractive or like the which worlds if we have multiple worlds in the game our users like best and this is like the information that I think it's very beneficial also for the production team for the game team and I think a part of it is also because this is like a two-step funnel so in the first step we have the ad so the creative so this is where we like first hook hook the players and then drive them to the fake storefront and here we observe like how do they react to this game like the players um do they do they watch the video um how what's the duration how many how many seconds do

they watch it um do they click on it and the most important kpi here I would say is Click through rate and then um then we have the second part of the funnel and this is the the store right but in our case the fake store not the real store so once the player lands on the store we can see what he or she does there like um scrolling through the screenshots like which one do they pay more attention to or enlarge do they read ratings and reviews and description because you know we have a heat map behind it so we can really see like where they where they touch and what they do and so on and in this part of funnel like conversion rate is the most important metric what we're hearing when you explain all of the different things you you can do and address and even model here is what's possible when a marketer has insight into the data and really owns the approach really has control here Ryan you work with a lot of

companies not just here with outfit 7 I mean how do marketers need to approach this how do they make it their own approach their own AB testing basically owning this really you mentioned the word control and I think that that's really an important important concept here um the we we all know that um uh it's quite a struggle for a lot of advertisers to really control all aspects of their campaign um everything down to like you know bidding logic and optimization decisions a lot of that gets delegated to various ad platforms to do that that work themselves and so it's that lack of control here um I think we all know that outside of the wall Gardens most of the mobile app advertising is still dominated by sort of blackbox AD platforms um and of course blackbox is a bit of a kind of catchall term that means a number of different things from like lack of transparency over campaign data uh using managed Service uh us using kind of set

and forget algorithms that do all the bidding for you um and then I think as well really kind of a lack of an intuitive UI that's going to help you kind of get into the weeds and Tinker around with what's going on in your campaign and making your own sort of bid uh bidding logic um and so really marketers don't really know what's happening on their campaigns before the user installs now with that said they know a lot about what happens after the user installs um you know we we all know that device IDs have been kind of normal for the ecosystem for well over a decade now and while that's going away um eventually what that has done is it's allowed marketers to really uh one do deterministic attribution and measure long-term value of the users that they get and so they really focus their attention on what's Happening after the user installs and being very precise and sophisticated in that which is great but what it does is it creates a little bit at least what I see is a little bit of a skill Gap where when it comes time to start going and really being precise

with the campaigns you're running you don't really know how to do that what vendors am I going to use what are the features that I need to be able to do this there's a little bit of a skill Gap there and so I think that that's really inhibiting you uh advertisers from like I said taking control of their campaigns and really getting into the weeds of of how those campaigns are run and how things ultimately bid at the end of the day bring up a great point because that's just it we're doing this very differently you talk about a skill Scout let me ask you Nina you know it's a team effort you have the team you have the tech execution matters um what is required in the team to do this really well to get that you know wash rinse repeat effort and outcome yeah so we have this system really in place um internally in house though we have on one hand in the distribution we have product marketing manager and ASO manager they are the ones responsible for defining the test so what are the goals what is the timeline um the

directions that we want to test so which which will be the testing groups and then we of course need the 2D artist of or the video production team to take care of the assets both for the store and the ads then we have it managers who create those fake pages and they have to put the system for tracking the store events also there right so that we can like really get all the data then to analyze later um and then of of course like the data scientist like this is the person who is um involved from a very start to help up to set up the test um to make sure to um check it all the time you know that it's everything is um doing great um to collect the data to analyze the data and of course make some conclusions and I forgot in between is also of course the UR manager and that's the person who is like who knows the campaign part in and out and and with that I mean like cross promotion so in

our ecosystem but also Paya um so they know the targeting they know the reach and it's really a group effort and we try to sit down at least in the beginning and at the end of each test together um to discuss and then to check the results and everyone is contributing their knowledge and their feedback um and we adjust as we go for example I mentioned heat map before we haven't always had that but then we found like really useful to see what where they actually users click on the store or for example we changed like not a long time ago also like the method on the analytical side of how to determine which group that we are testing is the winning one because now we also um take into consideration the markets of where the users are coming and also like the size of the revenue they represent in our whole Revenue share so for example if we get like a lot of installs from India but there are maybe not those users or players are maybe not worth that much so that we don't base you know

our conclusions just because India is such a big share in the test so we try to balance this thing out these things out and yeah it's like you know it it adapts through time and and that's it maybe one thing also that I think it's worth mentioning is that we constantly check you know question ourself are we on the right path because Market changes a lot and like technology changes keeping in mind that we also have some Indi and our audience who might not have that kind of a resource if you had to describe your routine for this approach to split testing because I can start small and I can get big results we'll get your results in a moment you know massive but you have to start somewhere what would you say where smaller teams could start let's say if the team is really small and they don't have resources to do that there are always like um um third party options you know a ailable so there are a lot of

companies out there who offer that to do that for you um to build this kind of fake pages and also to analyze the results and so on and then maybe when you expand and you have knowledge on how to do it you you can go and start building inh housee just to uh kind of Riff Off of that idea too because I think this is something that's worth it for Indie developers to keep in mind is that um uh uh doing this kind of ab split testing um over over programmatic advertising actually delivers quite a lot of Bank for your buck um we're not talking massive user acquisition budgets that have tens or hundreds of thousands of dollars behind them we're talking hundreds to low thousands of dollars to actually invest into this and get really uh really valuable data from it you don't have to go and you know take out a fundraising round just to be able to do something like this it's actually uh quite simple that can be done with uh relatively limited resources yeah and that is very true also because you know we started with our cross promotion network but not everyone has that luxury of having their own like ecosystem and

this is where like programmatic really comes into play because with that you will be able like Ryan said like reach a lot of players and also very targeted players which is a really really important thing as well if you're trying something testing for something that it's really Niche you know you don't have to spend thousands and thousands for for this that's always always good news I'm sure people are like yeah that was already the highlight of the show but of course there's there's more than that and you brought it up yourself Nina talking about how you can reach and Target players and some in new and different ways because you're able to see the data differently tell me something about what you were able to do differently maybe even um some surprises because it's always when you can see all aspects of the data that you find some amazing consistencies or correlations yeah so I think one thing that was very important for us is was like going from just testing on our

existing user base to expand to the you to the players that are playing other games so I don't know if you if the listeners how well they know outfit 7 but we have this very well-known IP it's called talking Comm and friends and we have bunch of apps in in that ecosystem so but the player base it's more casual it's younger audience and so on and then for example when we were developing a different game that was a strategy r RPG game of course if we would test on the on this kind of players we would not get relevant results but because this is not the target audience so it wouldn't really help us with the help of Kaizen we actually um managed to reach the players who are playing in like other games as well like through because they they can um they can get through them through different networks and so on and they even offer like a targetting that it's like more Niche like for example for for the strategy games and that's like yeah that was a game changer there are not many things on the market

available for that but because for for the ab test to be successful and the results to be reliable you need this control environment this is very very important so and you need this equal split of traffic which maybe sounds a little abstract but but I can explain a little bit more in detail so when you have like different groups it's very important that everything is the same for them that you have one variable that it's different but everything else it's kind of the same so they have to have the same um number of users so the same sample size that get into them um you need the same traffic mix so if you have multiple platforms and multiple countries and even multiple games that are coming from you really try to aim that they are kind of the same for you know the same mix so that it's not biased so you don't want to have group a coming just from strategy games and Group B just players from puzzle games games because then the results you know

you they're not really helpful and we really managed to achieve that with the help of Ryan in the team I'm hearing something also in here that's very intriguing about not targeting your user base and it's actually allowing you you said a game changer I mean like quite literally if you were saying oh you know we're in casual but we want to go into strategy or we're seeing all this mixing and melding between the genres anyway you know hyper casuals hyb casual and Casual is looking more like strategy and everybody has a story narrative or something I mean Ryan is there something here about also allowing you to test the waters a little bit if you're saying oh yeah we've got a whole casual audience but we really want to look at this other genre this sort of allows you to test it without having to take the plunge yeah y I think that's a a great point to make here and and what I what I see here is that there's really a real opport Unity here to do this very kind

of detailed granular uh targeting that allows you to create you know two audiences that are similar with each other in programmatic advertising um there are very rich data signals that come through the bid stream sorry to use kind of whatever esoteric language here but um you know when we get these sort of bid requests coming from different Publishers and exchanges it's actually chocked full with a lot of very rich contextual signals um you've got things that kind of device level stuff like what's the uh what's the make and model what's the operating system and the version um you've got more creative signals around ad formats and sizes um but you also have like Supply and inventory signals like what is the publisher that's actually showing this ad what exchange did it come through um you know I'd say reckon there's probably a few dozen signals that comes through every single bid request um now that does not mean that every ad platform out there is taking advantage of this and using it or say opening up a user interface that allows an Advertiser to do this kind of targeting but it does

exist and there are ad platforms that go and and create this kind of ability here if if you're working with say an ad platform that doesn't expose uh uh all or most of these types of signals you might actually be tricked into thinking that you're doing an AB test on an audience that appears to be similar like let's say you want to go and and and Target iOS users in the United States and you want it to be say an iPhone 12 or higher um you might be able to find a platform that would allow to Target to that level and it might make you think that hey I am in fact targeting the same audience here uh with two different two different things that I want to test but what you're not what realizing is that there's also what is the publisher behind that we talked about maybe uh you know wanting to Target Publishers that uh uh have a particular that are are of a particular genre that maybe matches the audience you're trying to go after well if you don't know what publisher you're targeting that campaign that you think is actually uh um targeting the same audience might be distributed across a wide range of Publishers and so

having a A system that would allow you to say Target specific Publishers or maybe a genre of Publishers uh um is is fundamental to this you know building up that science to make this similar to what our other data sets were you know there may be incomplete data sets but they are nonetheless important it would be fascinating maybe I'm giving you guys an idea you know get together and come up with some sort of resource that says here's like six or seven signals that able to get but you didn't know what to do with them well here's what you can do you know get to use cases which brings me to my next area which is exactly that setting up a campaign setting up a campaign like a pro Nina because you obviously know what you're doing and I'll call you a pro even if you probably won't do it yourself you're probably too modest um give me a tip or a best practice that is something that's really crushing it for you we really try when with setting up the campaigns for test to mimic the real life you know how how

the campaign will go in real life or let's say maybe with different words after we launch a game where do we expect to get the traffic from and with that I mean which Goos uh which platform even which network and so on and try to really find those users so I think it's very very important that you know beforehand who are you who your users are where you can find them um and then really try to mimic as much as possible possible these kind of campaigns that you will you will have later on also for the testing um be kind of mindful of costs you know if you for example know that you know us is your most important Market but it's it can get pretty pretty expensive find a Geo that will behave similarly like in um in the in the campaigns but maybe even later on for the Post install data like for example take Canada for us or I don't know there are like multiple combinations that you

can use you know as a proxy for for a certain market and for us like equal speed is very very important so that everyone gets um an equal share Ryan what about yourself you're not a marketer well you are you know all about Market but you're working with so many different games Studios companies maybe there's a best practice something that you're seeing them do that's like hey this is it this is how to set up your campaign like a pro there's two things I've been thinking about here um one I think is quite high level um and it almost seems silly to say but I think it's it's worth saying here um and that is that um uh before you even think about what is the right sort of uh uh tactic for a given AB test just do the ab test uh go and find a way to do this um like I said I talked about that skill Gap and I think that that can be a little bit of an inhibitor to taking the plunge and doing something like this that's very true don't yeah exactly yeah and this is and this is where I have to give kudos

to outfit 7 because um as much as you know I do sales and I want to take credit for that it wasn't like I came to them with this idea they actually went and did the research across different vendors they came to us I spoke to numerous people there we spoke about all kinds of different use cases before sort of honing in on this they were willing to take the plunge and then did their due diligence to understand what was out there um and so I would encourage everybody first before anything just do that get yourself in that mindset that I'm going to take this plunge and do it then a second thing that I think is is much more detailed but it's worth mentioning here and that is that if you're going to do any kind of AB testing and you want to try to hit the same audience with two different ideas uh make sure that you're paying attention as well to The Exchange through which this ad is being served um so you know you've got the Publishers that are showing the app uh the the ads you've got the advertisers like outfit 7 and then you've got all these technology providers you got these exchanges in the middle that are um you know managing these auctions and each one of them has

their own sort of nuanced way of getting that ad into a publisher app um they all have usually typically have an SDK that goes ahead and and serves it and I'll just say there is a lot of nuance from One exchange to the other in how those ads get rendered and if you're not factoring in exchange as part of your targeting criteria uh you might introduce some some bias or some variability that you didn't really anticipate just do due to the way that exchanges SDK works just at a high level paying attention to how they're rendered is a big difference yeah yeah and even things like the size of the clickable area around a call the action button some of them you know are take the whole screen and other ones are actually just the button so it's a lot of these Nuance here that can impact you know your your click-through or conversion rates well Nina you made the point yourself earlier about your team you know it's aligning marketing UA everything all about optimizing thanks to the granular data

that you are getting so tell us just exactly that so what have you actually been able to do because you have this visibility although the idea of the spit testing was initially for the distribution optimization especially for the icons and screenshots and like the assets that are on the store but we found out that it's like beneficial also for the games team you know or for example when we were launching my Talkington Friends app um where there are like six different character s we created a test like where one each group so we had like six groups in the test um featured only one character on every screenshot on the icon on the ads so the whole funnel was just like one character was the star and then we compared like we created the order on on in which like we defined um which one users like the best and or which one they like the least and that then impacted like what we put on the icons but also other promo materials and so on so so who are the

best loved characters um oh it's the Tom it's Tom it's always Tom and Angela so there's great ways to use this in different ways right not just thinking about soft launch or not just thinking about stages but but all of it what are some of the marketers doing that you're working with or you're just seeing them where there like hey they're pushing the envelope this is really really cool right yeah so if you imagine like um everything that happens up to the install thus far that's been kind of something that the ad platforms own they're focusing on Impressions and clicks and clickthrough rates and stuff trying to get you to the install um and then it's it's the game developers like outfit 7 that are looking at everything from the install onwards tracking the installs and then inap purchases trying to relate that back to marketing channels but actually for those that really want to get into the weeds here and and and own this data and kind of start looking at it um if you can get your hands on on the full data set which

you can do by doing programmatic advertising uh you can actually get a full uh full funnel data set from impression to click to install to inapp purchase um uh that's that's specific for your your app or your campaign now uh I've seen advertisers that will actually take this once they get that get the data from the ad platform get their their their install and event data combine that all together they can actually start developing their own algorith rithms to drive their own user acquisition so it's basically taking that that same kind of transparency over data but now how do we turn that into some type of buying Logic for a UA campaign so when I do an interview for pg. Biz we also say hey we've got so and so guests coming up we've got something in the works you know what would you like to hear what would you like to ask so I've picked a few questions that we got in and I'll start off with you Nina biggest personal mistake there's a lot of trial and error here to what you're

talking about share one yeah I would say the biggest mistake and this is like it can still you know happen is when we don't Define hypothesis on the test very well so then then it's just a waste of test you know if it's not a well setup or defined test of of what exactly we want to test in each group then even if we get like a best performer we are not able to go from there because we don't know what what was working or if it's if it's something that creates the bias or if it's not just one variable that is different again you cannot pinpoint of what what was working what what was not and it's not just and it's actually then it's a waste of resources because everyone invests the time and effort it's a waste of money because tests are not for free anyway you know a waste of time because you're losing a week or two or even more to set up run the test analyze and so on you Ryan you do know even from your own life and your other day job in a sense well your once a week

job how to keep energy levels High what's a tip to keep teams on track to keep the momentum because it's going to be like Dina said trial and error and you got to have some confidence you know test your test what's a good piece of advice from that level um okay that's that's a good good question here um I'm going to riff off of what uh what you said Nina about the um the the degree of collaboration that that exist that existed for you guys to put all this together um and I would just uh connect that with what I see motivates us as a company at Kaizen and it's the spirit of collaboration um I think the more eyeballs that are on it the more people in the room talking about this together the more excited people get and when the going gets tough or a test fails or you make a bad hypothesis or you screw something up uh you've got everybody to kind of uh uh put your arms around each other and and you know uh Soldier on um and so I say just you know make it a collaborative effort um and just have

everybody have like you know like the test of the test just a spirit of experimentation we're all in this together if mistakes are made hey we can accept that because we're all in the same same Mission here exactly that always on experimentation you know you you go to conferences I go to them all the time what do you need to be to be a great marketer you need to be curious which is last question to you exactly that Nina what's a soft skill that you need to have to have solid results now in marketing in ASO and what you're doing oh that's a good question um I would definitely say that you have to be curious like you did because the things are changing so fast so that you're trying different things out um also we see that the player base They do change you know the TA the The Taste are changing through years and maybe something that did not work last year maybe it will work year so be brave and go sometimes you know out of the box and

try new things you know don't be afraid I want to thank you both so much for sharing so much good information there good advice best practice one amazing show thank you so much Ryan thank you so much Nina thank you Peggy it was it was a pleasure this show is all about how to do your job better how to make an amazing game how to Market it and you have a say so if you have a story or know someone we need to shine a light on then we would love to hear from you we want to hear from you we want to reflect the reality of the mobile games market and all its wonderful complexity and strangeness so if you have any suggestions for us if you have any feedback for us you can always get in touch you can email us at podcast pocketgamer.biz you can find us on Twitter @ PG Biz and you can reach out to us through the pocketgamer.biz website if you're interested in

listening to all of our podcasts you can find them at pocketgamer.biz podcast and we would love to hear your thoughts on future shows and we've got you covered on all the major platforms so subscribe to the audio podcast as Brian said look for us on YouTube if you want to read it hey you can do that too because we have a companion post for you as well on the pocketgamer.biz website tune in again for the next edition of the pocketgamer.biz podcast and we look forward to speaking to you in the near future until then I'm Brian Bago I'm Peggy and saltz and that's a wrap till next week