Session: App Store Optimization For Games With PressPlay's Autonomous Agents. Moritz Daan, Phiture.

So, the next talk is App Store Optimization for games with Press Play's autonomous agents. And can we have a big round of applause, please? Thank you so much. Hey guys, I'm I'm Moritz. I'm co-founder at Feature. I'm leading the AI strategy over at Feature. Um, I founded and sold two online gaming companies. And I also led the international growth team at SoundCloud. Uh, today I'm going to talk about App Store Optimization for games and how to automate that with the help of AI.

So, Feature, we're a mobile growth consultancy and agency. We're working with a lot of games publishers, but also non-games apps. We We're based out of Berlin, but have a team all across the globe. At the very end of 2023 when Sorry. I think it was 2022 when ChatGPT 4 came out ChatGPT came out, we We had already done a lot with AI within our consultancy. Um, we have a bunch of engineers, data scientists, data analysts who are working on things like predictive analytics, which Kamsut talked about a lot in the previous session. Um, but we've never worked with generative AI. And we saw that as a major opportunity to overhaul the business, re-look at how we're doing things, and actually drive more impact for our clients, and so that's why we set up an incubator called Feature AI Labs.

And basically together with the whole um engineering and data science team as well as consultants, we started that incubator with the idea to launch seven different proof of concepts within within the realm of couple months. Uh and out of that, of course, we had to kill a lot of those concepts because they they didn't they didn't there was not enough maturity in inside of the concept to actually drive it forward based on the APIs and the models that were available. Um but there were actually quite a few that that were super super interesting. And so I'll call out a few a few, and I will be talking mostly today about this guy, Press Play. Um but I'll I'll call out a few. So we have launched experiment insights at scale, where we track all of the AB tests that games publisher are running in the Google Play Store or the Apple App Store globally. So we're

monitoring that ongoingly if they have a test running, and then we see what whether or not that test has been applied after it's been running for a while. And based on that, we can make predictions on what conversion tactics are working in a specific market for hyper-casual games or within what within kind of the the genre that that you're playing in. Uh AISA, I'll also say a few words about that in a second, but then we also have ASO keyword automation. So can we predict the perfect kind of keyword metadata and iterate on that automatically without uh human involvement. Um and then Press Play finally. So AISA is our bid reinforcement learning model. So it's a a self-learning model, which the way it works, we we look at when a client comes on board, we pull all of their historical bid changes in from Apple Search Ads. We look at what the client's objective

was, and then we look at what impact those bid changes actually had on that objective. Um and we over the course of 2 3 days, we train a model specifically for that client to understand what to basically understand like what bid changes have resulted in which results. And then we train it train that model on millions of simulations where we basically every time that that it predicts a a bid change right, we give it a cookie. And so that model within the course of a couple of days becomes really good at predicting what the outcome of a certain bid change on a certain keyword will be. And that led to a lot of clients really seeing massive um We just heard Consu saying like the CPIs have doubled in in gaming last year. And so it was really fantastic to see that we were able to we we now just launched a case study where we were able to

decrease the cost per cost per acquisition by 39% but this one is is still a little bit older where we actually were able to to drive up the the number of installs by quite a lot at a significantly reduced cost per acquisition. Um the other the fourth one that you saw on the on the slide is Press Play. Um at Feature we've been doing App Store Optimization for over the last 10 years almost. Um and we we basically are the the question that we ask ourselves with App Store Optimization is with that with the App Stores with so many apps out there, how can you make sure that people will be able to find your app, but then also if they see your assets that they actually download your app, right? Conversion rate optimization is that second part. And that's the part that we focus on in our regular uh ASO engagements. We have a framework,

tried and tested framework where where we basically go in manually, look for each client that you know, what are popular features, what is uh uh what have you found on in your user testing that people really like about the game, etc., etc. Uh so basically doing the research uh and then hypothesizing on like the findings of that research, uh prioritizing the different hypotheses, the different experiments, if you will, and then creating those assets with uh with our uh creative team, and then testing the assets either in the Google Play Store listing experiments or on the App Store. And then every time we run a test, we want to learn and iterate on the learnings. Now we with Press Play, we really wanted to figure out how can we create a new paradigm that does a lot of this automatically. Um and so with Wildlife Studios uh in October of 2023,

we started a 8-week pilot period where we generated all of the assets uh and deployed them automatically inside of the Google Play Store listing experiments. So within 8 weeks, we had launched 322 variants. We killed a lot of them, but we also had quite a few wins that really moved the needle for them. Uh of course, we wanted to make sure that we can repeat that success, and we did that with Lion Studios from AppLovin uh where we in 6 weeks deployed 200 experiments uh and drove 600k in projected installs, as well as Lockwood Publishing. They have a game called Avakin Life where we um drove 1.5 million increase in projected installs. So, the way it works are the the assets that we can test with with it are the icon, the screenshot order. I'll come back to that, but basically like in which order should you push position of the eight

beautiful screenshots that you have for your game, and the feature graphic, which is basically shown when the video hasn't started yet. Also are very prominent, and then the short description. And so, for each of these asset types, we want to generate with GenAI automatically the best performing assets and then deploy them automatically in the in the Google Play Store. So, when we started with Wildlife, the first round of the first batch that we had generated was really performing super bad. And we had a little bit of a tough client conversation. I mean, they they knew it was a pilot. It was like experimental software, but like we we were pretty shocked by the results. And so, we quickly, like after 2 weeks of running the first batch of experiments, we iterated on on the way that we were generating these experiments by basically feeding back the results into the assistant that we had built, which generates these experiments.

And so, in round two, you already see that we got a lot better results in the Play Store experiments. And then in round three, we finally got that first big win of around 13% average conversion rate increase. And that's really what the intelligent assistant that we built is doing. We basically initialize it. We tell it everything that we know about the game, branding guidelines, what can you do in the game? We feed it all of the metadata that you have already in the store. We feed it with all of the reviews. what are people raving about in the reviews? Uh all these data points basically go into the assistant. And then we uh start generating having it generate hypotheses. And then goes to the testing backlog. There's still an approval process, but it's for the rest it's all automated. It's just pressing play. That's also why we call it press play. Uh to deploy uh these automated uh

uh creatives that we're providing. Uh and then we have rule-based uh intelligent rule-based uh applying and killing rules in there as well. And for each of the experiments, we also ask the assistant, "What did you learn about this experiment based on the results?" Um so with African Life, uh we um when we started with them, uh the best available model with a proper API available was DALL-E 3. Um and so we we started with that model, uh but you can can see that it's not really like it's really this typical DALL-E 3 style. And so it was not really even though it was generating already a massive win over the original app icon, uh we felt like, together with the client, like this is not really in the style of of the game. And so we asked them for a bunch of assets uh from from this lady over here. Um and started fine-tuning models. Uh in

this case, we fine-tuned a stable stable diffusion XL model. Um and we did that by, yeah, feeding it all these different characters that you see to the left. And then we're suddenly able to position that character wherever in on a horse, in the desert, uh and we started to run like hundreds of experiments on on that particular icon. And that that led to massive gain in in conversion rate over already the So basically the compounding growth that you can get from really iterating on these icons. Um the screenshot order experiment, so you have probably with your um your creative team, your artists, put a lot of effort into creating eight beautiful screenshots. You might have put them to an AB B test already in the Google Play Store. You might have found that that test actually uh exceeded the the conversion rate of of your current

listing and have applied it, but you haven't probably tested like are these actually in the order of uh of importance to the user. And so, if you want to do that, uh it takes quite a bit of testing because like eight to the eighth is uh is quite a few permutations, so you will if you would test each permutation for 1 week, you would end up with 773 years of testing. Um that doesn't work, isn't really scalable, uh and uh and so, we uh we figured out a way how to kind of predict based on very short rapid tests in the um in the Play Store, really predict like what the perfect combination is of all of those 880,000 uh different permutations of the eight different screenshots. And based on that, we're then running uh a final AB test where we are able to um yeah, to to get the win that we had

predicted. So, we typically run the the top three um uh outcomes that the prediction has yielded and then see which one is indeed outperforming the current current listing. Well, what's interesting, by the way, to see is that a lot of these these screenshot tests that we were running actually two screenshots oftentimes outperformed the eight screenshots that our clients had in in there. Uh so, there's a lot of gain that you can make in terms of like really predicting what the right setup is. Yeah, just So, we were initially we were struggling with struggling with getting the characters right. Later on to the left you see basically the input imagery and to the right all the different outputs that we can generate. But basically we we once we had fixed that issue we we run into a a new issue with hexasword really had a hard time in getting the hexagons right. And so, we also did a

bunch of fine-tuning on different models stable diffusion XL and later in the year flux. I don't know if you you guys have used flux and flux pro but it's a really like it's really outperforming like the results from a stable diffusion XL which you can also see here. Like this was something that we hadn't noticed on a very small like the basically the size of an icon in the Play Store. We weren't able to kind of spot that there was something weird with this this image both of these images actually but the community of this game actually they they they were a little bit more vigilant and actually saw even though that these these icons were outperforming the current icon there was quite a bit of backlash so you still need to really like make sure that whatever you're generating is really doesn't have an extra leg or an extra arm. This was especially the case with stable diffusion XL. So, actually throughout last year when we were developing

press play we switched I think like four or five times in terms of the image model that we're using. Cuz that is like going at such a crazy speed and in in which these image models are adhering to the brief better that we're sending over adhering to like putting the not no extra legs on the uh on the characters that we're putting out. And so we we made significant improvements already just by switching like one line of code and using a different model that just came out. So these days we're using flux for this, but there's there's tons of other models that are getting really really good. Um we also for for feature graphics and for splash screens we're we're also working with a third party considered a sort of an Adobe Photoshop where you have all your layers or your Figma, but where where we are able to replace those layers with

the the copy and the the background imagery that we that we generate. This works really well for for games where we can also upload your custom font. Um and where we can really align kind of the the experiment hypothesis with both the copy that we're showing and the and the imagery that we're showing on the screenshot or on the feature graphic. And then finally as sky is the limit like you you don't have to work with hundreds of freelance translators anymore. You can just push this out into um tons of different localizations. And that's also what we're doing. So we we have typically with a client we would have somewhere from 10 to 50 concurrent it depends a little bit of on your traffic, but 10 to 50 concurrent experiments running at the same time. Technically if you're Google Play partner you can go up to 505, but that starts making no sense anymore because

then you're touching on really low traffic countries. So the way that we're doing that is we're setting up custom store listings for all of the top markets and are basically testing in those markets. Um and we have the smart early kill rules as well as like the the AI assistant that really learns about like the um what works and what didn't work. So, these are the smart early kill rules. So, basically all the automation is running 24/7, looks at the data of each experiment, sees if there's experiment slots available where we need to deploy a new experiment. Uh but also kills experiment that have exceeded the maximum number of days or that uh are already showing, even though Google doesn't call it out yet, they're showing us already a pretty negative conversion and we just want to move on from that experiment. So, we do kill kill quite a few experiments um in the process. Really all to find that big breakout

winner that we we did that we can discover much quicker by scaling those experiments very quickly. Um yeah, and then um finally we built a a web app around this. This is not a client of ours, but like uh the way that the work is that you can as a client just approve these experiments by pressing play. Every experiment comes with a hypothesis and then experiment title, so that you also have a bit of an idea of where the where this is coming from. Uh and that the system can also learn about uh whether or not that hypothesis was the right one to go after. Um and you get also all of the the results of your With some clients we have eight years of uh experiment data and so that muscle memory that might leave the company if an ASO practitioner or so leaves leaves your game uh studio, but here you can actually have everything in one

view. We're also working on kind of a synthesized learning overview that you can then access um if you ever need to look at, okay, what were the the big wins that we had in the past and how can we build on top of that. What we're work working on now is really going from this one single agent assistant idea to to a um multi-agents autonomous system where we have one agent overseeing basically the whole strategy and then we have specialized agents for each of the the bits and bobs that they need to review or they need to do user uh do certain research like for example scraping reviews from the Google Play Store and then presenting them back to the ASO strategist as here's like what people are raving about in the reviews. Uh You have a learning agent that collects all the learnings that we have from the historical experiments and then we also have these QA agents

that make sure that all of the experiments are um you know on par. There's no extra limbs in there. It follows brand compliance and it also follows any legal compliance that your company might have or or at least uh the the Google Play guidelines. And we find that putting everything into one agent in one one assistant uh waters down a little bit the capabilities of that assistant and therefore I I truly believe that this is the the future for um and that future is here today already but uh we're now testing this out for for the first time. Yeah, to recap uh we can scale thousands of have run thousands of experiments with many different clients um we can automate all those experiments very quickly and we can also uh report back to the clients uh having presenting that in kind of an

all-in-one view of uh of the portfolio of of apps. And some of these studios they might have you know 10, 20, 30 different apps. And so, having that all in one place is uh is also a big benefit for keeping sanity and not having to click through all of the different Google Play Store experiments um in all the different consoles. So, yeah, that's that's uh in a nutshell Press Play. I think I might have a minute or so for I don't know what time it is, but Any questions? Yeah. Uh I guess a natural limitation of running a lot of AB tests um is like the sample size, like how many downloads you have as a company, like at what kind of scale of a company would you say it makes sense to Yeah, we That's a really good question. So, we typically suggest uh it's a recommended uh value of 100,000 installs

a month uh in in your in your uh for one title one title. And then we break it down by all of the different countries in which it makes sense to run a test. Of course, if you have like 10 downloads a month in a specific markets, it doesn't make sense to run an AB test there. But for many clients, actually, the way we're we're doing it as well is like they have a SO team running in the in the core three GOs or so, core five GOs, and we're just taking kind of the the long tail, getting learnings also back into the SO team that is running kind of the manual AB test, getting the learnings back into their hands so that they can also apply that uh or test that in in the United States, for example. But the the lion share of of clients that we have on Press Play, we run everything uh basically all the markets. Thank you. Thank you. Thank you. THANKS. THANKS.

MHM.