Session: Automating App Store Optimization With Generative AI - Andy Carvell & Livio Marcheschi

[Music] so next up we are going to be talking about automating App Store optimization and we're going to be using generative AI to do that and we've got Andy andio from Fe to come up and tell us all about that so guys come on up and tell us about automating Appstore optimization hi everyone hello um yeah I'm Andy this is liio uh and today we're going to talk uh for about 20 minutes about what we really believe is going to be the future of App Store optimization uh just a quick show of hands I can just about see you under these lights um anybody here actively

involved in appstore optimization or your company is is doing Appstore Appstore optimization for any of your titles yep I'm seeing a few hands go up okay so some of you at least are familiar with with abst optimization already um and there we go yeah um so before we get into into the detail just a quick if you can step aside from there there we go um yeah quick introduction to ourselves so uh this is me I'm I'm Andy I uh I grew up in the UK where we had a very strong home computer scene I think that's one of the things that's kind of uh similar with Finland uh there's like a strong home brew development scene there my my father brought a home computer uh for me and my brother to T taught us how to code it back in uh back in the early 80s um and uh ever since I first got hold of a computer all I wanted to do was make games uh and actually my my first job after I graduated computer science was with a a small Finnish company called Nokia uh

it's a joke um of course they were they were huge um even bigger than uh than these days back then uh and yeah I uh I made a game called space impact it was one of the the early arcade shooting games on a on a mobile phone one of the the sort of like early arcade games uh it was embedded first on the uh the Nokia 3310 and on a few few subsequent uh devices as well I really had a lot of fun making that game it was like a real exercise in resource optimization we had to squeeze the whole code uh level data uh sound effects everything into uh and the graphics of course into about 16 kilobytes uh which was a lot of fun uh after after being in the games industry for about 10 years I went to Business School uh ended up in Berlin after that uh and where I joined SoundCloud uh and I was leading mobile growth there for about uh four and a half years it's also there that I published a framework called the the mobile growth stack um anyway uh after after being a SoundCloud for about four and a half years years I left to set up

feature which is uh the mobile growth consultancy that I've been running for the last eight years with my colleague Moritz who's unfortunately not here today but I do have here today Livio hi everyone um this is liio I work in as product lead at feature um I've been working in product growth for most of my career the last decade or so uh mostly scale ups and startups um including the Y combinator portfolio and my experien is mostly in conversion rate optimization marketing tools and the like and so I'm responsible for plus play at feature which we're going to introduce to you today and so Andy please great so yeah just very briefly about feature we're a mobile growth consultancy we tried to make a kind of a kind of a new style of uh of agency we're very much focused on driving business impact uh very much focused on data driven experimentation and we're working with a lot of global

Brands both in the apps and game Space so uh appst optimization few of you already are very familiar with the topic um obviously the main challenge is really getting your app or game discovered in a very busy category hell of a lot of competition standing out from the crowd so first of all just visibility discoverability and then once somebody's actually landed on your store Page it becomes about conversion rate optimization right so this is what we call the the C Loop um if I just going to run over here try and make it a bit more interactive starting off with a lot of research to understand deeply the audience and the users and how they're thinking about the game and you know how they're how you can can react to how you can effectively Target them with with like winning creative build some hypotheses around that prioritize those based on impact or expected impact create some new assets so things like icon screenshots um feature graphic um descriptions Etc test some of those

assets run those on an experiment we tend to do a lot of testing on the Google Play uh experimental framework because it's a bit more robust than what Apple currently allows in terms of native testing of course there are other testing Solutions on the market but however you do it you're um you know evaluating those those results uh think if it's a win or a loss applying the win hopefully getting a sustained conversion rate up increase on your your store Page listing and then you go around that loop again and again and that's basically that's AB up to my ation or at least the conversion rate optimization part of it um and you know the Big Challenge there is getting to the winds right even if you're doing good research and you're creating great um creative which hopefully you are um there's still you still need to go around that Loop multiple times probably before you're going to get a standout win right it's a lot about the Cadence and volume of testing that you can do um of course you need to be good at these other things as well but if you're good at these other things which I would argue is table Stakes um really what you need to be able to do is go around this loop as

many times as possible to get to the big wins and that's exactly what uh what we've developed at feature um from having done this manually and and we we're doing abs optimization manually for many many clients across you know for the last eight years we understand the process very well we understand the Google Play Store very well uh and we've been able to build something which gets to things like 322 deployed experiments in 8 weeks for wildlife Studios that's significantly faster uh and with a better win rate actually than uh than we've seen uh in in manual ASO optimization pretty often uh we've been able to repeat that with other Studios like line Studios loid publishing and a bunch more we've got some more case studies coming out you can check out these case studies on our on our website by the way yeah thanks Andy so let me introduce you to press play at press play we do accelerated uh Play Store experiment testing so to go through the loops uh the loop and they just describe much

faster and in automated way uh we use generative Ai and we fully automate AB testing on the platform um making the process quicker and more impactful for four different type of assets uh for icons uh for the order of screenshots uh for feature Graphics in case the game has a video on the store um for and for short descriptions that can be easily localized through generative Ai and and built on SC scale and also there's a fifth type of um assets which is the splash screen which we are also supporting since uh last week the green one green green button here yeah all right so at the cor of PR play with the creation of gen of assets using generative AI so what we do is that we combine the best practices we've developed uh since the feature was created uh in ASO and up store optimization and we take inputs from the

game in terms of which type of guidelines for Styles branding uh which experiments have worked in the past learning from user acquisition campaigns and the like um in order not to repeat mistakes and to Leverage What has worked um we combine this with game assets we receive uh for instance in in the image here you can see the same character from multiple angles we can take in in order to produce as a result um a virtual and unlimited number of assets um thanks to the combination of of of these these two type of inputs um at the core and so what makes this possible is the so-called AI model fine-tuning which is a f a fency way to call the customization of uh artificial intelligence models for the purpose of creating uh icons specifically and for a specific game um as some of you might be aware of there's a lot of AI models to

create images like mid Journey Dolly and they've been they've been trained with the millions of images um however they're not the best if you're trying to reproduce a specific IP a specific style a specific game character and what we do is then we train specifically the model for the game um even just five to 10 images can be enough to have a good result and and that's what these slides show so in this case we receive um the character from multiple angle we took it as an input did some magic trained the model and we're able then to uh reproduce the same character on a horse or different settings um some some real examples so thanks to this process we able to create um icons uh which are on brand also feature Graphics let's focus on icons now which are on brand so these are three examples um this is um avakin live from Lockwood studio um and the

character is an NPC of the game the name is Nadia Fox and in this case we fine tune the model using um for a specific character for for based on Nadia Fox and as you can see then we can reproduce her in different poses uh context uh settings and like and another example here is harer Clash by Voodoo in this case again we we focused on on a specific character which was what they had uh on the icon um represented on the icon and in this case we decided to stay closer to the original style and background and rather change the position of the Archer um this St are all live by the way so if you check online they might find a different icon than this and last case is very interesting because it's a fine tuning but not for a character but for a gameplay this is hexa sort by Lion Studio and Lovin and in this case we

reproduce different asset um representing the game plan of the game which is characterized by this uh exagon rather than just a character and yeah yeah yeah and feel free to uh to test us out on this by the way we feel like we've got pretty good at training models to work with different game companies IP by now um so if you have some IP that you're thinking well yeah we'd love to try this out but we're not sure if actually this is going to work with Gen then uh then come and chat with us got a booth out there um we can we'll be happy to actually run a test and see if we can actually work with your IP and get them into some some new gen gen AI novel situations we're we're very confident that we'll succeed with that um so yeah bit more about how how it actually works so Olio sort of talked about how we sort of ingest a whole bunch of information into the model in order to actually get it get it started um kind of seed it with enough information and understanding about your IP and your um past experiments Etc that

it it can start to generate hypotheses and creative briefs it's essentially briefing its own creative Department um you could actually take those creative briefs and give them to your own um art team for example if you wanted to um you don't need to use the Gen component but most most folks do want to use it because quite quickly it's able to get to really good win so how does it do that right the first test that it does it's going to generate some new icons for example this's an icon test um icon generally is where we see double digit increases in conver rate optimization um so it's a great asset to start with um the first the first set that it generates they almost always lose usually really badly right look at this like minus 51% conversion like you'd be like oh okay this is um but it's learning every time it runs these it Compares these it does an analysis of like okay this was the um asset that I generated this is the original asset and and the AI tries to understand like what was you know what was losing about this right what it it does an image analysis

right reverse image analysis to actually look at um you know what were the components of these images why why did this one lose why is this a bad hypoth bad hypothesis and because it's generating these more or less instantly able to run these in parallel uh we can run like sort of upwards of 50 experiments in parallel using custom store listings all automated so like it it's it's handling a lot of the complexity around deploying those um which means that it can very quickly learn from the mistakes and quickly get to you know we're seeing here like round two already with uh Sky Warriors it was getting to like you know fairly High single digit increases in conversion rate um and then after a couple more iterations it's getting to 16.5% and that's actually not even the highest conversion rate increase we've seen we've seen like double- digit increases closer to 50% in some cases uh ah yeah what we see here with avakin life um so this was an example where once we introduced fine tuning it really kicked it into high gear um So within a

couple of months we were seeing a 57% increase in conversion rate um again all through iterative AI automated testing yeah and just another couple of um product features of of Press Play Because press play is like the core is is generative AI but it's not just that so Press Play It's scale in its Automation and it's reporting so with scale we we mean that we we can run multiple up to 50 concurrent experiment per game title at the same time thanks to um unique segmentation of the audience we we do uh through custom store listings the automation I'm going to I'm going to tell a bit more in a second but in general uh we can use Smart rules to uh for instance kill the experiments which are underperforming as early as one day after they started as a consequence the speed of testing Can can go faster and faster the loop can repeat and like these 50 experiments which are live always at the same time can just

continue to produce impact and finally for what concern reporting we we give an overview of all experiments that has have been done in the past that are currently live and their performance I'm going to show you in a sec um so um this is the this is about the automation so press play can automate and schedule experiments first the left side of the screen uh so it's possible to set up um the general configuration for the experiment so what the confidence interval we are targeting clearly like the basics minimum detectable effects but also how many days mean and Max we want to run the experiment for and what's even maybe more interesting is these rules I was mentioning before either to kill the experiment earlier if they're not doing well for instance we can say that if a title has reached a thousand of installs and U and is still below 15% um we we want to kill it right away another interesting rule which works on on the other in the other case so if we

do have WIS is that they can be iMed they can be immediately applied um and they be they become the new control and the testing which the testing can continue the loop can can continue and the testing can continue without any manual intervention um on the other side of the slide you can see the experiment Q which is actually the core of press play so in the experiment Q shows all experiments that are ready to go live and it's possible to assign them a priority and um and to schedule them so that they can be deployed even at night in the weekend when no one is watching uh without again a manual intervention and finally to close up on on on features um with press plate this is very interesting for whoever Works frequently doing experiments on Google Play whose interface is not is not very helpful and having aggregated data is is is tough um we offer the possibility to collect all

historical experiments in one place and they can be filtered grouped order for instance in this case uh the experiments have been ordered by the best performing one um and then it's possible to click on them see which V variant was tested um what was the icon or the feature graphic of the short description um see what the learnings are um and then know take it from there in at in the new round of testing great so like the results really speak for themselves um you I think I already already presented these these results so we got you can read more about these on the on the case study we got a bunch of other games Publishers and uh developers in well we we're still basically in early beta um but we have some open slots so if you're interested to try this out come talk to us later we have a booth outside um you know I think like just to sort of sum up it's really about getting that scale running thousands of experiments massive

parallelization of experiments um High experimentation Cadence um automation being able to deploy and kill and scale like scale winners kill kill losers and do that all automatically and then also automatically gather and consolidate all the reporting and learnings from that so that all of that learning not only goes back into the engine and helps the machine learn better but it can also go back into your organization help your organization learn um you know maybe what's working which you can also apply to some other product whether or not you're using press play for them so yeah just finally a little bit about feature um we're a mobile growth consultancy we essentially uh work as a as an extension of our client teams we can help with kind of all of the stuff that you see here so basically full funnel Plus data AI art creative um more or less a One-Stop shop um to help with like you know kind of plug into your team help with strategy and execution on any areas where you need extra support

and uh yeah feel free we'd love to to connect on LinkedIn uh and and our chat at our booth we've got a couple of other folks from the team here uh we're going to be at our booth for the next few hours so would be uh would be really great to have some more conversations thank you very much great stuff thank you very much I think we have still like about an about a minute left for for questions it's all good yeah so I'm just about to see if anyone want does anyone have a question they want to ask we have a question right here yeah hello thank you really good presentation sounds a really good tool I I get a question regarding the screenshot because I've been trying to use Al AI for the custom product page for screenshot and I had a lot of rejection with Apple so did you find the tricks or something it's a really really good question on the screenshot so right now we're not generating screenshots automatically right we're doing screenshot or aut testing so we can test permutations of your existing screenshots like every permutation in in all markets to find the right the right

uh value prop to put in front of people we're working on some stuff to be able to put like gen like backgrounds and and like banners on top of your screenshots but we don't do the generation of the screenshots themselves we work on right now uh feature graphic icon short description and what's the other one I forgot one another another asset um and oh yeah it's screenshot aut testing yeah and um and uh yeah screenshot generation will come but we're not doing it yet for exactly the reason you described because like if you just use gen to create a you know a random image of what's supposed to be your game it's probably not going to be representative of the game Cool any other questions question we got one over here how's it looking like on the custom product pages side for that matter so uh do you you mentioned it briefly like cpps and csls uh can you leverage um yeah the tool for it uh maybe uh liio

can uh can speak to that one so um you're you're curious to know how the csls are set up for is so leveraging the uh this tool for CSL so so basically the Tool uh automatically is using csls to get this like parallelization effect so we we're heavily leveraging custom store listings to be able to run multiple experiments and speed up the experimentation testing I think your question is can we use this to optimize a specific custom store listing yeah that's essentially what we do so we tend to run uh experiments in specific custom store listing so combination between a store listing and a language and then we leverage we leverage that learning on other markets because what we see typically from clients is that they want they rather have similar assets AC globally apart from the ones which are localized so that's normally how we work but I can tell you more like later after yeah good question thank you great stuff I think

we probably should stop there but so thank you very much guys it's really interesting stuff thank you [Music] [Music] oh [Music]