8 Reasons Most UA Strategies Fail as They Scale

Welcome back to the Deconstructor of Fun podcast. I'm your host, Mishki Katkov, and I'm here with Nle Patel, head of product at Cohort, and Bruno Sista, head of analytics at the same company, Cohort. Two of the sharpest minds in predictive modeling, signal engineering, and all the messy realities of postAT user acquisition. Welcome, gentlemen.
Thank you. Thanks for having us. So in today's episode, uh this is for everyone and anyone who owns or aspires to own a P&L, essentially a rorowass curve, a person who fights with attribution or has ever wondered why the hell their campaigns fall apart right when they start scaling. So let's kick it off with the the question that I think we had before we started recording and that is about Roas iceberg. So Nle, you have a thesis of Roas icebergs arguing that the biggest risks are invisible. They're below the surface. So

my question is like what's the most common invisible killer you see studios underestimate today?
Yeah, I say the the top one that we see like the most visible part of the iceberg which you may be already addressing is the difference between paid and blended. So like let's define your paid channels. That's quite well known. You'll then have your organics and the question is do those organics count? They obviously count to your P&L but how do you work out which networks are generating organics? Uh which ones are not? Because if you think about the rorowass curve on a paid channel and you then have the additional influence of the organics that paid channel generates, some might argue you should uplift that rorowass curve for the paid channel by the organics it kind of generates. And so the reason why this is part of our like what kills rorowass iceberg is because you know in a studio uh you may just look at your historical

organics you may look at all your networks and then kind of partition them based on some sort of logic. Uh, and the reason this can kill your rorowass is because like how many games out there have really strong IP and like obviously you got the Monopoly goes of the world IP driven. Uh, but the chances are for you know the the the median studio that you don't have strong IP. Uh, you may not be focused on viral loops, you may not have gills and you know all these kinds of features. You may not have social play. So one thing when we work with studios uh is we ask ourselves should they be going for a blended rorowass target? Should they be looking at blended rorowass or should they be looking at at paid paid paid rorowass only?
And so just to to basically dumb it down to my level you we're talking about K factor here, right? A K factor meaning like for each download the K factor is like for you buy a download and you get 1.1 because your K factor is 0.1 and

that kind of gets added to it. So one out of 10 brings a friend or somebody else and that is the uh the the sort of a um or paid plus organic.
Yeah the organic uplift as we call it. So it's about like first of all it's quite easy to measure your organic uplift on a country platform basis. you kind of know iOS USA, I've got so many organics, right? But then how do you weaponize that? Should you weaponize that? Should you try and lean into the fact that maybe meta is highly organic generating, but then your rewarded networks where you have a user that goes in, wants to complete five or six different actions and then get out. Uh, you know, how do you split your organics between those two different channels? Should you, right? And we work with studios to kind of break down your baseline organics. These are your true organics just by participating in the app store or maybe getting the lucky feature. These are those organics. And

then using regression analysis, you can work out how do different paid channels lead to organics. Uh and I think that's how you don't kill your ROS. That's how you kind of battle this part of the ROS iceberg, which is by being smart at what is your incremental organics and what are your true organics. So basically the iceberg you're talking about here is that studios bet the pay [snorts] a larger CPI because they overestimate their organics.
Precisely. They misattribute organics to paid channels.
Perfect. All right. Well, Bruno, let's talk about the the sort of a the paid and blended trap that that was also some of your thesis. So, uh studios allocate too many organics into paid channels just like Neil said. How do you diagnose when a team is lying to itself about its blended rorowass?
Yeah, obviously, you know, these days incrementality testing is on the rise.

There's numerous solutions popping over to understand like what is the incrementality of your UA campaigns, right? In all of those experiments, um there is an element of looking not only at the paid installs that are reporting reported by your MMP because they are based on like very strict tracking links etc. and focusing more so on causal effect, right? You are looking at what happens to your total insolves in a geo compared to a different go. You test state by state or you test, you know, you do an AB test with a set of audience or 50% of audience that is targeted versus, you know, 50% that isn't. So that just exposes sort of the blended or the gray area between paid and organic and that there is always a correlation, right? There are certain installs that are influenced by paid but just aren't reported by your MMP. And so one of the ways in which we're helping companies tackle this is by looking at blended in

that holistic manner, right? What is the percentage of organics that are truly organics because they're based on ASO featuring like social pay etc. And what is the percentage of organics that can be directly correlated to your paid activity? And we find that by unlocking the value of those incremental organics, we're just unlocking more marketing budgets for UA teams because suddenly their rorowass actually increases versus what they see on the MMP.
Um that just helps growth. Yeah. Exactly.
There's an incentive to to basically see the organic higher in order to unlock bigger budgets.
Exactly. Right. You ultimately have to justify your UA budgets, right? And if you're looking at just paid rowass as it's reported by MMP, you're potentially just, you know, capping yourself.
Yeah. Yeah. I I I have some experience of this. I remember at Roio, we had very different curves for Angry Birds games and non- Angry Birds games. And I think

it's also so so clear why Angry Birds games made were so much more successful on average because of the blended roas actually being more right in the sense that there was much more organic traffic because of the IP and also could be that the team was just much more careful of overinvesting into something doesn't have an IP and essentially putting it the K factor at zero.
Exactly right. Like if you are a smaller indie dev, you're not realistically going to hit those elements of virality as often, right? So it is truly credible to believe that all organics are paid or paid driven or almost all of them. Uh so understanding what is your true organic versus paid influence organic, you know, it just unlocks your growth.
In the game of growth, everyone is looking for that power up. Vibes is exactly that. It's built [music] by industry people, so they actually get the challenges you're dealing with. As a result, they [music] become the fastest growing gamified rewards platform out

there. And the results are kind of nuts. With their AI powered targeting, you're not [music] just buying installs, you're bringing in the right players. And now they've got this new rorowass optimization engine, [music] so your campaigns basically level up automatically. Vibes is built different. It's designed for growth. And I guess you could say it's powered by good vibes. Check out [music] their DOF exclusive case studies by following the link in the description and go to vbs.co. [music] That's vybs.co. Share your email and join the weightless list. Yeah, more accurate. Okay. Well, nille Bruno warns against assuming that rorowass curves behave similarly across geos and platforms. Do do teams still model them this way?

I think so. Like I think you only have so much data to base your curves on and like let's take let's take growth for example, right? You're always wanting to increase revenue, unlock new geos, new channels. Uh you're doing everything in your power to to work in that direction, right? So it's almost like the moment where you are most vulnerable as you go into a new network that's when you have the least amount of data right and so when you are most vulnerable you're taking the most risk as well with your your measurement and and you know how you model out that network like extreme example last couple of years last couple of months rewarded networks doing great great things if it is your first time on a rewarded network and your entire data set and let's say you're quite data savvy you're you're building some sort of predictive modeling, uh you have a data scientist or two working on the problem, the only data set they have is on SDK networks or similar, you're going to make a materially wrong decision because you're going to vastly overestimate the tail tail endors,

right? The rewarded ROAS curve is probably going to peak a lot earlier depending on kind of like what actions you're you're rewarding. And so we you do see studios make that mistake and it can be shortlived mistake. It's a hard-earned mistake. You tend to only make it once or twice. Uh but at the same time the big big question then is how do you scale successfully new networks, new geos, maybe even a new campaign type when your data set is not applicable in that area.
That's that's interesting. And is there sort of a like a minimum framework for AC accurate curve prediction?
I think yeah it really does depends. I think like it depends on the the confidence uh you can get into that data. So for example if you have a lot of very stable traffic then obviously if you know it's going to be stable that that's great. on the other hand, you have quite a low confidence uh interval

on it when it's going up and down a lot or maybe you just came out of like the Black Friday uh Cyber Monday Christmas period and your data is I don't want to say tainted but it's heavily influenced by these seasonal factors. So this is the deep gray area. I still think we're at the tip of the iceberg. Uh but you know how do you transfer learning uh to use the machine learning term, right? How do you use transfer learning to go from area you know to start you know thinking outside the training data as it were?
I think I got it. So I I want to move on and talk about gross versus net rorowass and the misalignment that comes to that. So I've you you've talked about you've noticed that many teams actually model rorowass in gross but operate their business in net. Now what's the situation like when does this misalignment blows up?
Yeah. So I'd say you can go to an ad

network and you typically have to say like on day seven I want to earn 12% rorowass. Right now chances are you're sending your MMP data in a mixture of gross and net and that's being sent onto the ad network. So basically you have gross or net data depending if it's AP which comes in gross or IAA which comes to you net. You're sending out to the MMP that they're getting a set of signals. You're then telling the ad network I want 12% at day 7. Is that number gross or is that net? When the difference between gross and net can be your app store fee you know 15 30%. Also, your VAT rates, there can be a lot of misalignment and you're basically putting the ad network in a in a at a disadvantage to optimize your campaigns. They don't know what your requirements actually are. They just have a different basis of measurement. On the other hand, if you go to your CFO and say, "Hey,

100% rorowass, it's great, but that's a gross number." and then your CFO is like, "Yeah, but now you're, you know, now you're actually 63 cent on the dollar because you know all these people are taking a cut." You're in a bad situation at the end of the quarter, right? So the question is how do you maintain signal? And this is where the concept of like you can give one signal to the ad network via the web app that that meta provider or whoever provide. How do you make sure you give them the correct signal? But how do you make sure you're planning internally as a company uh to to make sure you're covering all your UA costs and and and other costs as it were as well. Game devs, here at Heroic Labs, we know that great gameplay is what players love, but integrating metagame features like challenges, progression, and rewards is crucial for retention and monetization. But coding them is a tedious and timeconuming [music] process. That's why top studios have turned to Hero from Heroic Clouds. Hero

provides a complete suite of battle tested metagame features that are quick to [music] implement, letting you spend more time on the core gameplay experience. Ready to ship fully featured engaging games? Check out Hero today at heroiclabs.com. [music]
How would you enforce consistency over this?
It's relatively difficult. There's there's really only two options, right? you you either change all your MMP integrations to use net revenue instead of gross revenue, which is a colossal task, right? You're asking uh an SDK engineer to work with real time currency conversions and tax conversions and calculations. So, it's it's it's realistically like not not a simple task to do. Um, and furthermore, if you're transitioning from a state where you're looking at growth to a state where you're looking at net, you're breaking all of your campaigns in the meanwhile because you're going to have to change all of the targets. So, the only other viable

option is to build sort of a data layer between your UA stack and your internal analytics where you're doing all of that conversion. And so, you know, you start tracking your real rorowass targets, net rorowass targets, campaign by campaign, and just using that insight to optimize your campaigns, which again is also not a perfect solution. Um, but it just allows you to align your objectives from the UA team to the finance team and make sure that everyone's talking profits instead of talking different flavors of robots.
Makes sense. um talking about overinvesting especially in a single channel. So scaling spent usually lowers LTV while raising CPI just because you're acquiring users that are not in the core audience and that has its own effect. Now I'm curious how do you teach the team to see the CPI LTV elasticity before it's too late?

Yeah, I think one one one moment we come into a lot of studios is you know they're making that big budget for the year and every budget has to have higher revenue. How do you do that? You spend more and the question is what is that elasticity? I think it it's a really tough tough question because I think the flip side of it is as CPI increases LTV can also increase because they are higher quality users the ones who are more likely to convert right and so I think one way we've dealt with the situation is by having that database so I think like we've had $6 billion of UA spend funneled through our our our machine learning and our and and our engine and so you can then at that scale start to see the dark spot spots and the light spots and CPI and how it scales and I think the scaling also changes based on whether you're an IAA or AP app uh whether you're um you know changing geos even campaign type right like a CPI

campaign you know rest in peace they can have a very different scaling curve to like a troaz campaign uh so I think it is being mindful of all the different segments within the market and exactly how CPI and LTV are going to interact but also taking that win which is higher CPI doesn't necessarily mean lower rorowass if you can make that kind of arbitrage with higher quality users and so our solution to this is basically a data plate right having that data set leveraging that data set and then on a per campaign basis understanding what elasticity looks like and then you can go to the CFO and say like look if you want this revenue and you want this profit like you could out of the kind of triangle of profit uh profit, revenue and cost. You can't control all three, right? Uh because as you try and control two of them, the other one is going to be this loose radical that kind of gets set by the market for you. You can say I want to spend this much, but then you can't guarantee uh you know a set of profit

because you don't know what kind of quality users you'll be getting into. So being mindful really and having that data to back up the decisions you make.
Yeah. Yeah. I've seen I've seen that misalignment kind of tear the campaigns apart. Um, yeah. So, I wanted to talk about the uh the new UA meta. So, before the recording, I think Bruno, you mentioned that at networks need a UA manager to feed them the right signal. So, what's the most common wrong signal the teams send and how does that tank their performance? So one of the main challenges is I believe that you know most companies are setting their base targets at a very aggregate global level right um when you know heads of UA are sitting down and trying to figure out again as we were saying during their budgeting processes what is the row they need to target they do that for the you know the complete

portfolio and they do that for specific you know points in time we want break even by day 90, which means we need to hit 50% rorowass by day 7. And that applies uniformly to almost all of their networks. But we also talked earlier that different types of networks, different types of uh traffic, even different types of campaigns within the same network have very very different user behaviors. They're targeting different audiences and their ROS curves are therefore very different. Right? So I don't think it's necessarily fair to make all campaigns fit the same day seven rorowass target. I don't think it's necessarily fair to have all campaigns fit the same day seven target even if the percentage is different. So one of the main issues is trying to fit the all of the campaigns into the same DSIS for example. a day since install. That is one of the reasons why Applov and other ad networks are moving towards day 28 campaigns and pushing them with everything they have because they do

recognize there are some campaigns that do require more time to learn and they focus on more long-term revenue. So, I think being very mindful of the different LTV and rorowas between your campaigns is extremely important. And then obviously you know we're mostly talking about rorowass campaigns but you know signal engineering is also all the rage right now and understanding your user behaviors and what are the kinds of events that mark the signal of a good user is also something that's very important to do like look into your internal data and understand like what are the early signals that's that tell you this is a good user and use those as event signals for your campaign. campaigns like don't shy away from doing CPE CPA campaigns or even rewarded campaigns. If you feel very confident that if you can get a user to let's say log on day three then they're going to stay for the long run. So look into your internal analytics, understand your users, understand your rowass curves for

each of the audiences that you're targeting and make sure that your targets or your optimization targets are fully aligned. So spotting a good user meaning you start segmenting your users and in you'll notice that certain segments have the highest LTV and those highest LTV what is the uh the user path for highest LTV segment and after that you're essentially taking the key events from that and putting it as events for your campaigns.
Exactly right. You need to start mapping out your ideal user journey. Right. what what is the path that leads to a really good user? Identify all of those milestone events and turn them into MMP events. Like go for custom events. You don't need to stick just to the default pre uh preset events. Just if you find that, you know, a user that logs in on a blue moon and purchases item number 23

on the menu inevitably ends up being a really good user. Leverage that. who owns this this part of segmenting
that's the biggest challenge it is mostly owned by product teams uh and one of the challenges is to have you know user acquisition people feeling empowered enough to also own that data um so I think there has to be a tighter collaboration with UA teams and product teams to identify what marks a good user
and then making sure the product team doesn't AB test away that kind of nice that that nice signal that you have, right? Like in the AB testing first kind of mobile gaming world we live in, it is too easy for the UA team to like the product team not to be like we're we're AB testing flows now. That that campaign's going to go to zero because that signal that you're sending to the ad network no longer means, you know, what it used to mean a week ago. You guys both worked at at various

different studios with millions of segments and multiple products. How do you solve this organizational issue?
It is a tough one. It is 100% a tough one. User acquisition people, even marketing people, aren't typically invited to product conversations. So I think it's important to break down those silos by making sure the product team is sharing all of the big wins whenever they're doing AB testing, improving products. Um you know all of those insights are shared with the UA team so that we know any changes that happen to user behavior they inevitably show up in our rorowaz. So it is important that we stay on top of any changes uh or any permanent changes to the way to the product. How do you see this evolving with live ops tools moving from sort of a se AB testing just AB testing to segmentation and now personalization at the most sophisticated companies where it's machine learning or AI same

thing uh basically personalizes them into specific segments without product actually having the input with that how do you match that with MMP signals That is an extremely good question. Um, ultimately you just have to make sure that there is sort of this almost like you know you what some companies are calling sort of flywheel teams that look at the different levels of interaction with your users. There's, you know, product is represented, live ops is represented, marketing is represented because they're often the first and the last uh touch point with the user. So, making sure that all of these teams send their envoys and share all of their most important insights is important to just keep the train running. Attention app and game developers. Are you ready to unlock the secrets behind the world's top performing apps and

games? With Sensor Tower, you get the ultimate competitive edge. Whether it's breaking into new markets, fine-tuning your UA strategies, or boosting monetization. Dive into data like never before. Track global trends across app stores. Analyze market leading inapp purchase and inapp ad strategies and benchmark your performance with realtime insight. The best part, Sensor Tower doesn't just deliver data, it helps you turn it into action with intuitive tools and expert analysis. It's everything you need to make smarter, faster decisions. Thousands of app and game studios, from indie creators to global giants, trust Sensor Tower to help them win. So why not you? Start your free trial today at sensortowwer.com. That is sensortow.com.

again you worked at various different companies as do I. Um there are some companies that are operating live ops in I would say like a 15-year old model where the team is operating one game and they're thinking very player first and it's a you know proven approach. They're working on that game and then there are more I wouldn't say sophisticated but a new way of operating it. It's not even that new. It's been for 10 years. Chinese already started doing it decade ago. But basically there's a live ops team that operates multiple products and in that type of a live ops setup it's not the teams are owning the game and are wearing the t-shirts and they work for this game. Now they work for live operations of the company and they generate cash and in that live operations they are very tied to marketing. It's a whole different team. They didn't build it but they're making money out of it and growing it. What's the uh what's the pros and cons of those two different organizational models when

thinking about signal engineering?
So I'd say yeah it probably changes by by organization size too. Like you can have a small organization is doing very modern live ops. Uh I think like a while back you had Voodoo on your podcast and they were speaking about how they isolate and silo these teams and they are the folks who are great at doing that part of basically the the the life cycle of the product. I think for the larger organizations you can have very segmented teams it is easier because you have P&L responsibility in those situations right so each AB well each uh segment each kind of test or hypothesis you're changing in the game I think that's tied to someone's P&L and so then you do have a holistic like general manager who is saying okay I have to kind of like weigh up the costs of uh losing this signal as it were to the benefits of having higher top Dow because uh you know this AB test is oh this this test is going to put higher

shop packages in in the shop right so I think in the the largest scale you have P&L responsibility you have general managers the second part is like for for the the for other studios who are maybe smaller but equally have a game which is like you know 30 to to to 50 million annual revenue I think that's where it gets really it can get really uh difficult because they probably have a lot more shared resources right like the development team, some of those folks then when put on a live ops hat. Uh I think there probably will be modern tooling that comes out that kind of streamlines uh live ops and multi-segments and I think there are there are great ones out there already. Uh but at the same time I think as we move into what I call the probabilistic marketing stack and that is from Gen AI creatives like dynamic creative optimization on steroids with with Gen AI all the way to having infinite segments where the the you know everything about the app is tailored to each and every individual. In this

probabilistic world, we do need a new approach, right? Maybe it's some sort of I don't know MCP server that has all the relevant information and it can tell you exactly what segment is going to get which creative and which in like inapp experience and optimize for that. So yeah, it's we're just starting this kind of like probabilistic world uh when it comes to like digital assets end to end like this. So a lot of growth in that space I'm sure in the next you know couple of months and years. Yeah, I'm just thinking about like everybody's, you know, talking about how AI changes, the new tools change the ways we're working, but we haven't really seen a major change, especially in European or Western game companies. They still approach freeto play almost like they're shipping DLCs at just faster pace than before. It's like next update, then the next update, and there's events in between to keep players engaged, but it is a DLC approach. And and again looking at some of the more sophisticated or not again different uh monetizationheavy

uh publishers, let's put it that way. They are not thinking that way. They are the the learnings they're taking immediately and they might be collecting learnings from multiple different projects at the same time and then picking up the best one and then putting it into different games without having a meeting with the next team. Hey guys, we learned this thing. You might want to try. It's like no, the same team makes the same decisions. across the portfolio and the portfolio sees the the overall growth.
Absolutely. In situations like that, the bigger the portfolio, the more learnings you're going to get, the more impact you're going to get by unleashing one learning onto the entire portfolio. So it does kind of like make you wonder whether like in the same way Microsoft and Google are kind of all of the benefits of AI is being credited to them. Could it be similar in gaming and what does it look like for the midmarket uh or the indies as as Bruno mentioned? How do they compete when they don't have

150 games in their portfolio? Um yeah, interesting times.
All right. Well, let's talk about MMPPs and especially this single source of truth. So, you note that MMPs can't catch re-engagement, at least not all of them. Uh, scan selfattribution or postd 180 behavior. Uh, what does a real single source of truth really looks like in 2025 or in 2026? It's it's close to the end. And who should own it, data or UI teams? So I think Bruno this is let's start with you or delay which one
well this is a subject that is really truly near and dear to my heart. Um working at Runescape I launched the mobile clients of one of the first truly clo crossplatform MMO RPGs. uh Runescap being a game that already had a 20-year-old desktop version, actually two 20-year-old desktop versions and

then suddenly going into the world of mobile. So, you know, obviously MMPs can capture most of what happens in a mobile environment, they can capture nothing of what happens outside of it. If you have a web store like most bigger developers are now um trying to do then you know MMPs are currently not yet capturing that sort of activity if you have you know obviously re-engagement and retargeting efforts. MMPs don't really have a way to deal with it in an effective manner. So building an SSOT ultimately means treating your MMP as one source of data, the source of data for everything that happens on mobile. But then the data that you have to enrich with data that happens on other platforms, other revenue streams, um, etc. Even other marketing interactions that might not lead to an install. So

um,
yeah,
from my perspective, I'd say like think about the iceberg. I think we're well and truly deep into it. But at the same time, like I'd say just focus on scan with your single source of truth, right? Like it can be that simple. Like the biggest platform has just destroyed visibility and solve for that in the first instance and then build out your single source of truth. Right? If you could have a single source of truth that allows you to match your Android spend on iOS and have the same kind of visibility on it, that's gamechanging for any studio. So I think the best single source of truths we work with have done that. They focused on scan. How are we going to solve it? How are we going to start working out more and more about every campaign? Is that using the scan schema? Is it using optins and then finding like doing like similarity based on nonoptins to those optins? So then you can say, hey, this is clearly a meta

user. The rorowass curve and the the metrics tell me that this is clearly a Almeida uh media uh user who who who looks like that kind of rewarded curve. And so focus on scan. That's the biggest problem for many studios. Build out from there. I'd say allocating a re-engaged user back to its user ID once you lose MMP visibility. There's probably like what 10 20 10% right there's a 10% uplift there. So focus on the big fish scan. [music] This podcast is brought to you by Appsflyer. In today's digital world, understanding your app's journey from [music] discovery to download is more than just insightful. It's essential. Enter Appslier, [music] the leader in mobile attribution and marketing analytics that allows you to measure the full potential of your [music] marketing efforts, making every ad dollar work smarter. With Appsier, that's your new reality. Dive deep into datadriven insights that reveal exactly

where your users come from [music] and how they'll interact with your app. Appslier, where your app's potential meets performance.
Right. makes that's a very simple simple um advice easy to follow. Um one case or or one challenge is that the bad metrics create bad confidence. So you've said that one metric alone can mislead an entire UA organization. So if you had a if you had to pick a single accuracy metric for an early stage studio, which one and why? Bruno, metrics are your thing? That is a good question. Um, you know, ultimately you do want to continue, you want to do a level of prediction, right? You, as you're launching new campaigns, as you're estimating the long-term value of your current campaigns, you do want to be forwardlooking rather than past looking.

So, you want to have an element of prediction of your rorowass and your LTV that isn't completely tied down to the ratios that you saw 90 days ago. that just means leaving in the living in the past and just having this you know these delayed signals of performance. Um so one thing that's important as you're doing predictions is to also keep track of the accuracy of your predictions right how far off were you when you said that your rorowass was going to grow by 3x um you know between day seven and day 90. Um there's you know a few different uh metrics that you could use. One that we do tend to use is uh mappy. So you know tracking your mean average predictive error between you know what you predicted 7 days ago and what you observe 90 days later based on the real day 90 to day 7 ratio for example. U do you do you want to add here? Yeah, I'd say I wouldn't

want to I I actually can't limit it to to one because I think accuracy definitely gets the majority of the focus. But you have to think about confidence as well, right? Like you got one network that always has like, you know, between $10 and $15 LTV, another one between $1 and $100 LTV, right? you could have the same level of accuracy, but one of them is going to have a much bigger confidence interval, right? And so I think when it comes to like really great studios who want to be at the state-of-the-art it's like yeah look using Mappy to constantly back test do blind back tests on their own data review how good they are forecasting review how good they're predicting but then is also to arm the UA managers with confidence intervals because your predictions are always just predictions if you're a UA manager who gets a bit of information from your data scientist they can say It's very accurate, but a

lot of noise recently, right? A lot of noise. Take this with a little low lower confidence than you're you're used to. Maybe let it mature for another a week and then make that change to the campaign that you're, you know, expecting to make.
And how do you use main average predictive error? Like is this a UA team metric or a product team metric? Like who uses it and why?
Yeah. So, so mappy or like mean absolute precision error would be you take a set of data you train a model you make the prediction for some you know with some new data for the future period and then you compare that to the actual number and then you sum the difference for every single day in your prediction range and you say you know that that's the it's basically like the back test that you're doing and saying I predicted x it was y what's x - y across all your segments across multiple days. So I'd say that lives with the data science team. A data science team may also want to use you know like just MAE and median

app and wappy or weighted app to basically counter for small things like hey in this segment there's 10 million users and so it's overweing this kind of low LTV high volume segment. So I think like yeah Mappy is the one that we go to when we're we we're chatting with customers but having a data science team that's well verssed in all of them can look at the data and say look we need to wait it here we don't want to wait it here that gives the UA manager the most confidence to use those predictions
mean absolute precision error interesting so it's almost like you're batting average before you take a
exactly
the data team comes in and says hey um here's what we suggest.
Yeah, exactly. Right. So, we're seeing the sort of user acquisition all of your campaigns, all of your segments and geos within a campaign as sort of, you know, a bet realistically, right? You do have some information on how they've

performed in the past, but you don't necessarily know how they're performing in the future. You can make an informed guess. So, you're constantly exactly doing those guesses and taking actions and optimizations based on the recent trends that you're seeing. You should have your data science teams involved in trying to estimate what the future ROAS is going to be. And so basically, you're just judging them by how precise their their recommendations were.
Web stores, you've seen what good looks like, but have you seen great? App charge is the direct to consumer platform backed by the likes of Supercell and built by monization pros from Moon Active, Platika, Huge Games, and others. People who know the difference. App charge powers over half a billion dollars in transaction a year for top studios optimizing every detail. Deeply personalized web store offers, custommade design, industryleading monetization features, in-game payments

SDK, and more. 70% of top grossing games already have a web store. But the one scaling fastest, they're doing it with AppCharge. Learn more at appcharge.com. Never heard about that but about that specific metric. Interesting. So, um I wanted to ask about teams over bidding uh and not actually not over bidding over bidding under bidding but really just changing their bids and their row as targets too quickly because they're not confident of what they're doing and maybe the early signals are not quite there. I know uh Bruno know you worked at Misplay so that's another one where it takes time before you understand how the loyalty platform is needs to be optimized so you could be changing really quickly because it's a new channel you're just not accustomed to I'm not sure and then you start changing metrics so what are the overall the pacing rule you wish every UA manager tattooed on their forearm

look um everyone's on the same boat we're all working with networks that are intentionally opaque about how they recommend you run you run campaigns. All they're telling you is don't change things too quickly and you know that is both in their interest because you know they're worried about maintaining a certain stability of their revenue but also to your interest because realistically yes we live in a world of machine learning rorowass algorithms that do take some time to learn right. Um my rule of thumb is first of all understand what is the conversion window that you're optimizing towards right if you're working towards a day seven rorowass or if you're working towards a CP event that typically happens on a day seven mark that as your conversion window. Don't ever make or avoid making more than one change within a conversion window. Right? If you're optimizing towards the day seven rorowass, then if

you make a change, wait at least 7 days until you make the next change because that gives the algorithm time to figure out the best performance um to meet your day seven target rows, right? So wait at least one to two conversion windows. And I think that is generally what Google's advice is. if you can walk talk to one of their most competent account managers. The other thing is yes, avoid making too drastic a change. My rule of thumb is 20%. I will not change a troas target by more than 20% up or down relative to the previous target. I will not change a budget by more than 20% up or down from the latest revision because again if you are making too drastic of a change you're sort of forcing your rorowass campaign even your CPI campaign to explore new audiences and to understand okay I'm delivering to your target right now who else can I serve your ads to

that will also be able to hit your target and that takes time right you're going to be spending a lot of inefficient ads spend money or a lot of ineffic inefficient UA budget testing campaigns that aren't going to be profitable. So my general advice is make changes less than 20% up or down to budgets or troas targets and make them at least one or two conversion windows apart. And I know that that is so exactly so I know that that's a daunting thing, right? You want immediate results. You don't want to wait a week or two if you see that your campaign is underperforming. But realistically, if you make a change today, the campaign is still not performing tomorrow, you sort of have a knee-jerk reaction to already change it again. That's just going to hurt the campaign. It's going to make that learning process even longer. And that's just going to ultimately like impact negatively impact your

ability to spend. It's just going to stunt your growth.
That makes sense. It's a It's a hard one to keep though. 720.
It is extremely hard, especially with newer campaigns where you don't know what, you know, good performance looks like.
You know, ultimately, it just comes down to the same question, right?
Wait until you're confident enough about your results, right? It goes back to that level of confidence.
20% cut to something that looks bad seems stupid, but that's the advice. Let let the algorithms learn. Okay, very clear. Well, final question. Let's talk about the future and the future of predictive roast. So, um you know, honestly speaking, most predictive models uh today are spreadsheets with lipsticks. Um where do you see this space going and what will ROS's prediction look like in three years and what do you think will come obsolete? We can start with just

the first like where do you think the space is going? I'd say the space is getting more and more automated. I do believe on the ad network side they will basically only have ver campaigns in the future. I do believe the ad networks will take more and more risk and offer later and later windows. So app loav day 7 to 28 in a UA manager's ideal world they get like a day 365 100% right that's the number you go to app lobbying. So I think the UA the ad networks will get better as well and they'll take on more risk by offering longer and long windows. I think you then like as I was saying generative creatives dynamic creative optimization will kind of reduce the lever of having creatives being changed in a campaign by a UA manager. And so where does that leave predictive you know predicting rorowass I think first of all will be around auditing or or or managing these different ad networks

right they have the curve that's to their their best interest they're running auctions of you against your competitors so I think it's about having that auditor in the room who can who can say this is what an independent data point is on your rorowass and that allows those conversations with the ad networks or whatever tuning parameters they allow UA managers to have tomorrow to to make the changes there. Um, yeah, I think it's going to become a lot more probabilistic and a lot more a lot more automated in the in the near future.
More automation, more probabilistic later. Windows dynamic creative optimization and UI managers become auditors.
Exactly.
What becomes obsolete then, Bruno? Um well hopefully spreadsheets uh will become obsolete right we've all spent days and days and weeks slaving over the same spreadsheets but you know as an as an was saying it's all going to

ultimately become more predictive a lot smarter uh we're going to make predictive LTV more democratic so that not only uh bigger companies that have data science teams dedicated data science teams can access that's sort of predictive LTV. We want to make it available for everyone. As ad networks are becoming more opaque and more like black boxes, we're trying to sort of shed some light into what might be happening under the hood, right? Um, you know, in the future, I foresee the work of a UA manager as being, you know, you do make changes to the campaigns, you still optimize the campaigns, but you're sort of building predictive models or predictive algorithms that are also informed by the changes that you've made. So you know if you've changed the target on day X or you've changed the budget on day Y or you've added new creative on day Z that all feeds your prediction algorithm for how you think campaigns are going to perform in the future. So all of those become inputs

more than just outputs.
Fascinating predictions. Well, fellas, um, this was a master class because you're, if you're listening to this and you want to understand whether your your rorowass is above the water, below the water, or at the water, you should definitely replay this episode, share it with your UI lead, share it with your CFO. And um Nle Bruno, thanks for for stopping by and and shining a flashlight into the deep sea of user acquisition and teaching dumb people like myself to understand a a little bit more about this uh this craft.
Pleasure as always, Mishka. Always.
All right, everybody. And check out the link in below. Cohort is a fa fabulous platform to be honest for for user acquisition. We've used it with some of our clients and uh can't recommend them enough. So on that note guys, thanks

again
and thank you everybody for listening.
You did it. You made it to the end of the episode. As a fan of the show, it would help us out if you'd subscribe [music] and leave us a review on the podcast service of your choice. Those stars and likes, while a small action on your part, mean a lot to us and help to grow the show. More importantly, are you a member of the Deconstructor of Fund Slack group? If you have 5 years or more of games industry experience, go to deconstructorofund.com/lad and apply to [music] join. Join the games industry best professional community filled with peers always willing to lend a hand. [music] Or subscribe to our newsletter to get all the latest insights from the deconstructor of fun content creators. Thanks for listening.
[music]