LTV Prediction for Mobile Apps with Hybrid Monetization: Tenjin’s pLTV

I want to say that our accuracy and Roman you know you emphasize the word average which is a you know the right way of talking about model accuracy really well um is actually really really good right 90% average accur accuracy is is fantastic and something that is really rare in this industry. Hi everyone, welcome to another video about the tangent update. Today with Jaspit from our product team. Hi J.
Hello Roman.
We'll talk about LTV prediction. What it is, why it matters and how can you access it within Tangent. Um yeah, let's start with a few words about yourself, Jazz.
Cool. Thanks, Roman. Hey everyone, I'm Jazz or Jazz Preit and I am the staff product manager at Tenzen. Um I usually work on data and dashboard side of

things and we have built predicted LTV for all of you uh folks out there.
Exactly. Uh we prepared for this video a couple of slides. I'll just uh share my screen and just we'll go through them and we'll have this sort of product conversation. Sharing my screen let me switch to slideshow mode. So yeah, LTV prediction tedd for apps with hybrid monetization. Hybrid is a new hyper now. there is a lot of hype but we felt like there is not enough of tools built specifically for that subset of apps um and OTV predictions is kind of an essential tool um when you run user acquisition uh for a hybrid game

let's see why you need to uh do the LTV prediction want to comment on that just
yeah for sure um I I think Roman that's that's an excellent point we need more tools for hybrid that are you know lacking uh out there but at tension we are super focused on on this um just to like give a small um intro kind of thing why hybrid specifically so for apps that or pure inapp ads um monetization model, your revenue will most likely mature a lot faster, right? So on for instance on day three or seven whereas for hybrid apps uh or apps using a lot of like inapp purchases um your your IAA revenue

might mature faster but your inapp purchase revenue may take weeks or even months to mature which is why we have PLTV um for hybrid apps. So this way you don't have to wait for your m for your cohorts to mature. Um and like make you know campaign optimization decisions after day 30 or day 60 or or longer. Um you can make your campaign optimization decisions a lot um earlier, right? Instead of waiting waiting for your cohorts to mature like on day two for instance.
Exactly. And here I have this hypothetical example of campaign A, campaign B. As we can see here on day zero, it's like campaign B is a clear winner here. However, it might not be the truth for hybrid app because a user

might come in, make a purchase, and it will overweight the all of the revenue that been uh gained through showing ads. And here is how it would look like if there would be uh prediction for day 14. Right.
Right. Um now uh we already worked with a couple of companies. Here is a quote for a company called Tapped. Uh just uh work closely with James and the team to work on the whole predictional TV project. And uh yeah, let me just read the quote and this is like pretty typical um quote that we see from other folks that are using the tool. We use tangent predictive TV metrics to monitor the performance of new cohorts in real time without having to wait for uh long-term

data. This allow us to make user acquisition much faster. Just exactly what just said and right.
Yeah. Um you know Roman so when we were doing the product development for predicted LTV we worked with a lot of customers right um closely and uh TAP was one of the customers even in like the alpha stage uh they used our predictions from much earlier and now that you know this product is out uh for everyone and is available to everyone to use we have gotten very similar feedback from almost all our customers um how you know they're using this and they're able to make user acquisition decisions much much faster
right um and here is again like emphasis on the speed so if I'm waiting for 14 days then I need to wait to see

what campaign actually uh performs better if I use prediction LTE blue line Here I could see the LTV before the day 14 or whatever day it is and reinvest money quicker and as you can see here you will just get get more money at the end. Um maybe just for someone who doesn't know what hybrid app is uh let's talk about that just what is a hybrid app?
Cool. Yeah. So when we say hybrid apps, we mean an app that uses the two revenue business models that I quickly introduced you to in the beginning of this video, right? So one of them is um inapp advertising, which basically means your app is making revenue by showing

ads to your users. And the other business model is AP which stands for inapp purchases. So inapp purchases could be either you are selling products one time, it could be subscriptions um it could be other things but you are also making money by selling a product within your app. So hybrid apps do both of these things right? They make money by showing ads and also your users can purchase your products within your app. So you're able to make money by both ads and purchases. And when an app uses both these models in tandem, we call them as hybrid apps. Now you can see this is an app an example of the distribution of inapp ads versus inapp purchases. It can vary. It can be like I don't know 20% ads and 80% inapp purchases. Um but this

is a great example there Roman of what a hybrid app looks like.
Exactly. And like seems the future is in uh hybrid apps and that's what we also see in our
100%.
Right. Uh one of the things but it's a challenge right because it's new. So it's a challenge uh to predict an LTV for hybrid apps. In the past you might find um LTV predictions for app purchases for inapp advertising based apps. Uh but so far no one has done the predictions for hybrid apps. So and here we have highlighted the word like neuronet network. Maybe I'll ask just for you to comment on that and talk a little bit more how you approach this challenge.
Cool. Yeah. So Roman is absolutely right that Tenzen is solving this problem in a very

unique manner where because there's a lack of tools out there supporting hybrid apps we're focusing on that right and and he's absolutely right that you know Roman you're absolutely right that that is the future and that's what we are seeing uh from an industry standpoint of view um and obviously when it comes to user acquisition decisions or campaign optimization um however you want to call it um using we're using ad mediation data. We'll talk more about that later on. Um, but we're using a lot of, you know, highly granular data available um, at ad impression, user level. Um, and and that goes the same with inapp purchase data. And when you're using a lot of this data, you want a machine learning algorithm that can handle it really well and it can

handle this unique challenge, right, that you're talking about, which is to support hybrid apps. Um, and and neural networks are a natural choice for it. I don't want to go too deep into the technical aspect of what neural networks are. But you can think of neural networks as um these are actually called artificial neural networks. They are a machine learning like um techn one one of the machine learning technologies where you can train your um model and then train your model using historical data and that model is able to learn from historical patterns and trends and then make predictions. Uh I don't know Roman if you have any more questions I can keep talking about this. Um one thing why we used a neural network versus other things. So in our product development cycle we did look into using simpler models. We did you know use many

other bench like we had a lot of benchmarks. We've done a lot of benchmarks for us. neural networks were the absolute correct choice because a they're able to handle large volumes of data really well which is exactly what this unique challenge um uh proposes and b um they're able to find patterns they're able to learn from trends really well so you don't have to do a lot of like feature processing um why you know anyone should use tangent predictions versus you know doing your own thing. We'll talk more about that. But from a neural network perspective, we want this model to learn from data across all apps and all organization and find patterns. So even if your campaign is new, let's say, and it doesn't have any data in the past, our model will learn

from similar apps and similar orgs and make accurate predictions for your campaign.
Yeah, that was my question actually. Why why should I use your
Yes. Okay.
Yeah, you got me. Uh yeah, let's go through like some challenges for the hybrid apps. Uh you actually mentioned this, right?
Yeah, I did. Yeah.
Uh a lot of signals and we use impressions, add impressions.
Mhm.
Um and we've deliver maybe we can talk about the cohorts. The predictions are available on the day zero, right?
Yes. So uh let me rephrase that actually. Um so our model learns using day zero cohort
whatever data is available. Um and using day zero cohort it will make predictions

for all the cohorts in the future
until day 30. So we we focused on day 30 right now and we're going to add we are in the middle of adding support for longer cohorts until day 365.
Um but that's what we're doing right now. like it will take your day zero data, learn patterns from it and they make predictions for cohorts until day 30,
right? And I guess one of the most important one if we're showing if we're talking about the apps showing ads is like add impression, right?
Yes. Yeah. Because this is campaign the the use case is you know user acquisition um and campaign optimization. We are using very granular ad impressions as one of the features. Um we have a lot of features that we're using like around 100 um and this is one of them.

Yeah. Yeah. Yeah. This is like just to answer this is also one of questions uh that I had uh internally like how do we make sure that the early predictions because from the hyper casual days I know this is one of the super important factors to understand how is the campaign performing early like you need both like early and late and uh this is why and how we we do Yes. Um, next challenge. This was in my example like in the one of the first slides, we had two campaigns and uh at some point campaign uh A actually got better than campaign B presumably thanks to purchase, right? Um and we update a prediction every two

hours. Can you explain what does it mean? Yeah. So we are using a lot of your data like I mentioned like hundreds of features right and then the distributions of your ad impressions how your campaign is performing based on the changes you've made you know to your campaign like budget bids and etc. um and and other a and other you know real things happening right outside of your whole entire like UA strategy. So all that data goes into the model and then the model will make new predictions based on it. Especially when your cohorts are not mature, right? Once your cohorts are mature, um the model has a lot of that data already and has made like very good and accurate predictions.

Um but especially when you're making any changes, you know, any changes are happening to your campaigns. This is why we make these predictions like this quickly to factor all the real world changes happening or distribution changes that are happening and then we give you like fresh predictions
that reflect that reflect the reality of you know that world now
is it the same for the first like two hours so I've started a campaign I'm starting seeing the data for the campaign does Does it mean that the next 2 hours after I see it like I should get some TV prediction or I need to wait longer?
So, uh, Roman, my understanding is that we need a mature like day zero data um, and then we're able to make

predictions for 1 to 30. So, let's say that you come back on day one, right? So on day one and to day 30 obviously your cohorts haven't matured. We use the immature cohorts also as one of the features and we add that data to the model and then the model will predict based on it. Right? So every two hours your your cohorts are also maturing and the model takes that into account and it will give you new predictions you know for like day one to day 30. Um so so that's what we are basically doing.
Super. Gotcha. Uh challenge number three. As I mentioned like there were already products that can do uh prediction for IIA or IIP.

Mhm. We decided to do it prediction in one metric for both and we made it available in the dashboard. Maybe Jess you can take us through some of the thinking when this was decided why we do that.
Yeah. So for product management development right like the way we wanted to approach this problem was let's build you know an alpha really quickly um and let have some of our customers drive you know the feedback and and then and then evolve the product organically. Um and while we were doing this we received a lot of the early feedback and that early feedback is very critical for making you know a good product for LTV uh or prediction specifically

um we want to build something quick that solves you know the use case for hybrid. So we went for this one metric. So um our our existing customers know this that we have LTV um a combined LTV metric right which is the actual LTV um and that is available after your cohorts have matured or it is also available for uh immature cohort data right um and we know that hybrid apps use that metric a lot um so we wanted to provide um a similar metric that combines both inapp ads and IP. And for tension specifically, our reporting data and infrastructure is built in a way where we already have this this historical or actual um all LTV available. So we ended up making predictions for that metric. So think of

that metric as your y axis in machine learning terminology and then you will use all the historical trends and then make a prediction for that right and and that's what we are doing. So a it was a decision from a product perspective because we know our customers use this and this is going to be very impactful for them and then using this um using this metric we can now provide uh derivative predicted metrics like P roaz and P ROI. Um but obviously our infrastructure reporting infrastructure and data infrastructure is in built in such a way where making predictions for this actual LTV is a lot easier.
Gotcha.
Uh now to the last challenge for hybrid apps is accuracy. We have the stunning accuracy average accuracy of 90%.

Yes. Um was it always like that? Did we had to work on that? Maybe you have some benchmark on my industry.
Yeah. Um, so this is this is a great question and um and I want to say that our accuracy and Roman you know you emphasize the word average which is a you know the right way of talking about model accuracy really well um is actually really really good right 90% average accur accuracy is is fantastic and something that is really rare in this industry. Yes, this wasn't always the case. Um like I said we did I I mentioned briefly in at the beginning of this video right like how we did product development right we started with alpha beta production um obviously for alpha the accuracy was um not as good and

before alpha was even started we did do internal benchmarking right like we um our GFS had uh predictions and we bench benchmarked with them and then we did our own internal benchmarking with other machine learning models, right? Um and then we picked after and then the way machine learning uh development goes is you want to look at different features, see which one is giving you good accuracy, um which one ruins your accuracy, find you know do some um feature engineering, understand your data really well, understand your business really well and then this is a very organic way of how we did things, right? Like okay, how do customers make decisions? talk to experts for instance. So a lot of that has gone into building predicted LTV and which is why I think this 90% accuracy that we received uh was made possible is because we added a

lot of that expert insight into this and and approach this problem from the customer perspective right how do they make decisions involve them early and so on so it was a long process it wasn't an easy process to achieve that um we work you know very hard to get that 90% accuracy. That said, I I do want to mention although our average performance of the model is 90% right, the accuracy varies from campaign to campaign, app to app and country to country. Uh I I think I forgot to mention this but our predictions are available at app campaign and country level right. So you can you know you can allocate budget based on different countries for your campaigns, different campaigns, different channels, right? Like if your channel A is not performing for this

country, you can do all sorts of things for all sorts of like countries and channels and campaigns. Um when it comes to accuracy of a machine learning model, it can vary from different campaigns to different countries, right? 90% is the average. So far, you know, our customers have said they've seen 90% accuracy, which is great. But I do want to you know uh add this caveat that your accuracy can def depend on your country and your own like campaign and channels which is why we're thinking of adding confidence intervals uh in the future. So this will help you you know make your decisions a lot more confidently.
Right. Right. Right. Right. Yeah. I think this is an important point to emphasize again like it's an average even though like it's great result. Uh but it really depends on a lot of

factors. It's still a prediction.
Cool. Uh I think that was the last challenge. Now the main question is how to get started available on all paid plans. So now we actually we also uh started with cancel anytime feature. So you can start with us pay 200 bucks and get the access to predicted LTV. If you're already attention customer then you already have an access to it. Just click on the edit metric on the dashboard find predicted LTV metrics uh predict LTV predictive errors. Uh what else is there? like at least at least at least three good guy. Yeah. Predicted LTV, predicted ROAZ and predicted ROI.
Yeah. Yeah. Yeah. So, no feature gates. Just try to see how it works. If you already tried it, leave a us a comment.

If you have any questions, also leave us a comment. Any last thoughts, Jess?
Yeah. I I just want to say Roman and thank you for highlighting that. So using Tangen is now easier than ever, right? You just need to go on our dashboard. You just need to subscribe to our $200 plan and you can cancel it whenever you want. Um so if you're interested, try this out. Um Yep. And I we'd love to get your feedback um on this or anything else.
Super. Thanks a lot, Jess. Give us a like if you like this video. Uh we can do more videos. That's only one use case for predicted LTV. There are more. We can also show a demo. Uh yeah. So there is a lot of content to explore here. So um
yeah, there is.
Yeah. I I can endlessly talk about this

I think for three four hours straight. Like we can talk about technical stuff, we can talk about the fraud use case that I didn't mention, right? And how to set it up. So yeah, u stay tuned for more content.
Exactly. All right then. Thanks a lot, Jasp. And thanks for
Thank you so much, Roman. Yeah. Cheers. Bye.