Game Analytics 03 - Retaining Players

[music] Nikolai Alber Peterson, co-founder and chief product officer at Game Analytics since 2012. What has changed in these few years?
It was a very exciting time back then. We were about to democratize analytics for everybody. Everybody was using Sinka and wanted to be professional. So we started this company. We were very game agnostic in the beginning. uh and we got a huge traction by mobile very quickly. Since then we've been growing especially with the hyper casual wave and now we have like a vast uh feature portfolio supporting all platforms PC and uh mobile as well. So it's been quite a journey.
Yeah. Well, let's get into this journey and let's talk about retention which is you know the king metric of any game. So how is retention actually calculated because there are different ways and the numbers look quite different when you do these different ways.
Yeah, it is the amount of players installing on a specific day that

returns a certain amount of days later. So for example, users playing on a certain day um a thousand players return uh 7 days later. the percentage uh is then uh the retention on that specific day and you can report it by day 1, day 2, day three all the way up to day thousand if you want to. Um so it is an indicator of um how people are engaging with the game coming back playing it more and more.
Yeah,
there's the uh the gold standard of retention and that is the the 40 2010. So day one 40% of users are coming back, day seven 20% of users are coming back and day 30 10% of users coming back. So that initial funnel is high and it kind of creates that high level of stickiness towards the end. If you fall outside of this, is it an automatic kill switch?
It is not like we see different uh curves and they fit different genres, different ways of content being consumed, um different ways of making money. That makes sense. But what you

really need to do is to take into account how you will make your money back, your investment in making the game, uh to see if that fits with your retention curve. But it is a good baseline to go for 40 20110 and just see.
Well, let's say you are interested in making it into a business case. Do you see a lot of studios making the math?
We see them trying to do this where they try to map to like a very high level and then many studios do not. And what we really recommend them to do is to keep tracking retention very early and then using retention as I mentioned together with how would you make your money? How would users actually pay for a product via ads or inapp purchases? And then also uh what is your perceived scale? And what we are seeing sadly in in many cases is that games are they are somebody's baby or pet project or like they really invest a lot of time in it and it's very hard for them to to stop making it.
Okay. So treat it as a business as well as a craft.
Exactly. Yeah.
Okay. And how do you know that your

retention metrics are good enough to move forward? I think you have to treat it uh a little bit differently as you say like early stage right because initially you might have a lot of bugs you might have a lot of UX issues you also need to have players coming in because retention is from install and onwards so you can't keep testing the same users they will be buyers or it's not the same experience anymore so you need to have an influx of users testing out your retention curve and then if you can't really move that curve and you can't really have a business case where this makes sense um then you have a bad curve right later on you already established hopefully that you have a good game, you have it out, people are playing, you have people come in from different angles like user acquisition sources, organic, you need to measure if the users you acquired are actually good retention, like do they retain and make money for me? And then you need to check retention like a health metric. You need to check if it fluctuates. And if it fluctuates, it's a symptom of something not working well. So it could be that

people are getting tired of your game. It could also be that you simply have a buck like you can't progress through the third level in the tutorial and then at one point as with all games over time they will reach a point where it's not sustainable anymore to continue working on it. There's not enough players coming in, not enough money being made and then you have to make the decision to then kill it and then stop supporting it basically. Right.
Yeah. So basically early on when you're moving forward you were suggesting through all the stages take the cohorted approach. So every cohort, let's say it's a monthly cohort of users
and then next month comes a next batch of users and you're doing a next batch of test rather than treating users as this one blob of people that you're throwing different experiments on. We have this in in our cohorts feature where you can go watch it per day like the users installing in this day and then moving on and and per week.
Uh and we can report um the sum of the revenue that these players generated over time what you would also call LTV
for a specific week. For example, you

can see these users that you acquired for the new build, the new season pass or whatever. And then 30 days later, okay, how much did they actually make me? And if that's way lower than your previous season pass or way higher, then you make decisions based on that.
And you can also dissect that down based on the channel where the users came from, whether it's Tik Tok or YouTube or other ones.
Yeah, exactly. So you invest in the right channels. the simplicity. That's how you start testing different things and seeing who is your core audience, where are they coming from, what can we do to improve them. And then once you start finding good results on those cohorts, whether they're weekly, bi-weekly, monthly, you start doubling down on the channels and the features that seem to be performing.
Yeah. And then you have all these other variables like how do you do creatives like the the store presence? um people that experiment with fake ads. You know that that's not the game that you're seeing in the ad.
And the question is like when do I stop this experimentation? So you are testing these cohorts. Some cohorts are going to

be better than the other ones. Some are going to be worse when you're trying different things.
At what point do you know you should stop experimenting. This is not going to be good enough to launch
before launching the game.
Yeah.
Yeah. Because it also takes time to test. It also takes time to to keep the team on waiting, right? So,
it's very important to to get some kind of a baseline um figured out for your game. And if you're an experienced studio, you will know from previous titles or you would know from previous season passes, what can I expect for retention, right? Or LTV um forecasting. If it's a totally new game and a totally new game mechanic, then that's a different story. But uh if you don't have any signals early on, it's very very difficult. And what do you mean exactly by a retention baseline?
If you're in the market of of building match 3 games or other games, you you have to establish this baseline yourself. Of course, you can use data points shared on communities via voodoo and other things to get like this good retention curve for a game, but it is quite hard when you're starting out.

So, the first one is a mostly an ad- driven game. 80% ads, 20% inner purchase. It's quite popular. So, this has day one 40%, day seven 9%. So dropping quite quickly and day 30 1.8.
So this is a hyper casual game.
Yeah. [clears throat] This is what you would call hybrid casual,
a bubble gum.
Yeah. It is lots of users make the revenue on ads within the first 30 days, right? Then you have a a popular word game which is challenging your friends to to play like a word game scramble or something.
Day one is 28%. It's way lower than 40, right?
Yeah.
And then day seven is 15% but that's way higher than the previous 9%, right? And then day 30 is 10%.
Wow.
Which is way higher than 1.8
which is the target goal.
Yeah. Which is a very good goal, right?
What do you think about payer retention?
Yeah. Payer retention is uh of course the goal that you want. You want to retain your paying players because they will pay more and more. Right? There is also the players who have not paid yet. They are the ones who might convert. So

they are also very important. Right? So retention in general is important. So I would kind of look at how good are you at converting your users? The goal is having good payer retention, but how good are you at converting your players? Are you improving your conversion over time? Can you put in some things to make more people more enticed to pay? The more people who pay, of course, uh the better, right? And then for the paying retention, as I mentioned before, you can look at it as these install cohorts installing for a week and then how they accumulate revenue over time. So normally studios have like a day 30 total revenue per user. We also report the total sum of course but the per user is very comparable through like uh fluctuations of uh active users and so on.
And how important it is to look at your cohorts based on LTV.
It is very important. We're also working on predictive LTV to kind of very early on to see uh what happens because uh when you acquire a user that user can make revenue for many many um days or

even years right so if you have a big ad campaign quite often the return on investment on that campaign shows like almost a year later for many of these bigger games right uh so many of these games if they stop making ad campaigns they would actually still make revenue but it would just dry up right so it's super important to look at the LTV of these different cohorts to see is it going down, is it going up, am I doing something right or wrong?
Yeah, because I I vividly remember um several years ago u game lead for Angry Birds 2 which is uh in a purchase driven game quite large hundreds of millions if not billion in revenue lifetime. The game lead was very keen on following payer retention because essentially that is your customer retention. And it kind of makes sense because as you stack up your payers, so basically each cohort retains longer of payers and you're keeping your payers and they're doing subsequent purchases, that means your LTV curve per cohort keeps on growing.

And that in turn means you can push your customer acquisition cost higher and through that get more players. because their LTV and their return on ad spend is going to be guaranteed not only after a month, two, three, but even further on.
Yeah. So, if you see this LTV curve and it's kind of flattening for users like which means that at a certain point in time they stop buying anything
or churn
or churn. Yeah. That means that you don't have enough content for them to keep playing and buy things or it just means that they just lost interest and and so on. So you need to kind of figure out how can we keep them interested? What are new things to buy?
Novelty of a game. Does that have a an element?
I think retention u generally is like you have a good time and you keep coming back. So I think novelty mostly is about finding an audience that finds your game fresh, right? Um and also perhaps it is on the user acquisition side with the creatives like this is a new idea,

right? But to me it is very risky also to do the novelty like you can do both. You can copy match three games or you can do like a novelty thing. If you do a novelty thing, I think it's very important you test your metrics very early on
because you can feel like you have a great idea, but then you figure out as you scale there's no audience. Nobody likes kind of this underwater match three role playing game. But isn't it also dangerous not to have novelty? Because you can make exactly the same game as exists in the market, which many developers do. And if you copy something that is already very successful and dominant, you're kind of capped.
That is an issue. But sometimes you take like 90% and then you take 10% novelty and then you push that as the big kind of novelty idea. But it really isn't the selling point.
The bones are kind of the same, right? basically derisk by having a lot of proven elements. Like if you're making a puzzle game, it's still match three or match two or whatever it is. You don't have to create a new puzzle experience.

Yeah. and then find that novelty whether it's in the theme uh whether it's in the progression mechanic or some customization mechanic or so forth that allows you to advertise it something different and allows you to change the first time flow of the users
to tell them that hey this is kind of like what you play before but new
yeah exactly
same but new
what's again then the marketing's role in early retention
sometimes the goal of the UA department is to bring in a lot of players and bring them in cheap but then they might turn 3 days after and not make any revenue. So there's a combinational effort between the product side and you know the UA side are these players valuable and that's also where retention comes in like this campaign they were not really retaining that well does the CPI still make sense so there's definitely some math there in in the LTV that you generate from these players and then how much they cost
so basically marketing and product should work together
which is of course not surprising because they are working on on on the

same thing to bring the type of players players that actually affect positively the KPIs of the game. And then again, the game team's work is around retention of those players, making sure that they have the retention of not only, you know, majority of players, but the payers so they can afford paying to get those quality users into the game.
Exactly. We already talked about fake ads.
Uh I went to this Google conference a few years ago where a gaming studio presented um their test on fake ads. M
they also mentioned how they it was hard to to convince the development team to do this but they tried it anyway in a in a region and retention did drop a lot because it was these fake ads with a minute or you have to pull these strings and I think we all played them
but actually the revenue was the same a lot more players a lot more ad revenue but they churned uh right away so they decided not to continue with that experiment I think others might have different experiences but we see them all the time so it must makes sense for

some games and some audiences,
but it also leaves a lot of bad reviews and uh on stores. So, it could be if your audience is people who never read kind of these reviews or not affected by it like it it could be very specific games that that really cater to this.
Yeah, it it is interesting because a lot of sophisticated studios are running those campaigns
uh and they wouldn't do it on a whim.
No.
And how about IPs? uh we're seeing now different kinds of IPs whether we're talking about toys or uh celebrities or um movie show IPs. What is their effect on early retention given that they bring the value of familiarity?
I think it's a very interesting topic and I think it's actually becoming more and more prevalent with brand branding. It used to be that when you had a branded game, the goal was to make the revenue when the movie came out for example. But right now, brands are working with studios to create this branded experience. We're seeing it a lot on Roblox, Fortnite. And the goal here is just to create um a good playing

experience. So, it promotes an ambassador of the game instead of making revenue. So, the brands actually pay the game creators, not the other way around like it used to be, right?
So, retention is is super important for these brands, right? If you look at it on just like a very monetization way where you add a brand on something and make money, then I think there is a lot of audience to be drawn in by that. But you can also get a lot of like uh fringe fans that come and play. Oh, this this was Ronaldo or whatever. I installed it and I don't really like the game. I'm going to churn like after a day. And then you have the Ronaldo super fan who buys everything, right?
Can an IP a familiar IP buy you more time if you have a weak gameplay? I think it definitely is is a bonus in some cases, but they're really thinking about it like is there synergy here? Are we damaging our own brand? Uh, so it it is a very difficult exercise, but I do think you get a lot more players. Many of them turn because they're not really, [clears throat] for example, Marvel,

they're not really big Marvel fans. They just want to try it out. And then you get the occasional super fan and they are actually big splenders, right? And those are the ones you want to retain. Uh well, let's move on from early retention into late game retention. Well, let's say day 30 and forward. What are the KPIs that matter the most when you're trying to fix your late game retention?
It is trying to figure out if these players are, you know, turnurning out of the content. Uh some games they have content where you can keep playing the same levels, right? And others have a season pass where you have a specific amount of levels to churn through. your most active users are they done playing these things or have they played all the heroes you have so to make sure that they they have content basically and also that they have things to buy that they find interesting
it is basically content of the game I would say
for increasing it long term
do you measure different social interactions because in the late game in big portion of the games social becomes more and more important because you

become part of a community whether it's a guild or other type of an element or a competing element or a leaderboard board or something and you can't really disengage because if you disengage you get kicked out of that guild or you drop down in your status. So those are kind of like the driving methods.
Playing with each other is definitely a sticky factor that we see a lot and we see u games investing a lot more in it. Not necessarily direct multiplayer but async where you can kind of play with a friend in turnbased and so on becomes more and more prevalent. Yeah. So that that is a way to increase stickiness. I would definitely do that.
The second part is AB testing. So typically this is the one where you try different variations and through that try to make the decision. Is there a point where AB testing kind of becomes testing for the tasting sake a little bit of a data theater or is this something where what you just have to do all the time?
Mostly like 99% of our our users just

want real fast results. They just want to put out two changes, you know, measure very simple even though it it might not be, you know, probabilistic enough, right? And we do also see people doing a lot of testing and then making uh without these these math to really know if it's the right decision and then make decisions based on one or two% differences. That's way within the era of probability, right? So, I do think that you can go overboard, but then again, we also see very positive uh results for some studios. An example is just changing the whole color scheme from purple to blue.
Super simple change, right? And it had a massive impact on retention. One thing that we also saw a lot people do is change the ordering of how the packages in the shop is shown. Do you show the largest pack first and then downwards or the smallest? Things like that just unexpectedly have a huge impact.
There's one part that it takes you time and it does cost money. The other part is that the results are never as good as they are with the test groups. You would

see an uplift of let's say 24% in revenue when in the shop it's put in the biggest product first versus the smallest product that it was before.
And once you implement that test to the uh the main branch, you're not going to see a 25 4% increase in revenue.
Probably not.
And it's going to be actually significantly smaller. And that is the type of a difficult part that studios kind of often grapple with, especially executives because they see all these positive numbers, all these big figures and all of them being put back into the main branch and still kind of looks the same.
It is a little bit of zero sometimes. Some AB testing actually gives us such massive changes that you just this is an obvious change, right? So we do see many positive effects of it actually.
Yeah. So over a longer period of time you start seeing a lot more positive effects.
But if you see results that are very small in differences, I would very much uh recommend not taking that as a hint to change things because

it needs to be significant.
Yeah. So from AB test to segmentation to what we're now seeing is personalization. Again, behind personalization, there's AI components. Are we still in in sort of a hype cycle when it comes to AI being able to personalize experiences for different type of user segments?
How would that translate to multiplayer games and what would you actually adjust? It's very hard to test and measure, but uh I think it's coming. But it would probably be more about the gameplay experience of you kind of coming into a game and it learns about you. But in terms of this personalized monetization approach,
I'm not sure it works in terms of multiplayer games and so on would be very specific to single player games.
I don't know what you think or if you have concrete examples, but it is very
difficult topic. We've seen with multiplayer games the way there components of the game that are being personalized. Whether that is for example in-game store uh it changes from a typical store that is the same for

everybody to almost like an inventory that is available for you for the next 3 days or 7 days and the store will totally change after the seven days and it just lives based on the progress and your activity.
We do already see personalization in terms of difficulty. So I guess you know that right? If you die a few times, then the next thing it's going to be way easier, right? So, all these things are a kind of a personalized uh progression uh adjustment.
Um going back to like full circle when thinking about kill switches and you know, you mentioned Woo in that sense, what do you think is the best strategy for launching new games? Is it putting something out, iterating it cohort after cohort, or is it only releasing something if the early retention metrics are strong enough?
I think it's it depends on what platform. Like for PC is very different. Uh you have early access period and different ways of iterating, right? But still, you need to have a good experience when it comes out just like a

Kickstarter. Like you can't have just bare bone stuff that doesn't work. If it's a mobile game, it's it's much easier to test and hide under the radar. You could iterate on that if you wanted to. So you could have a game very early, simple mechanics. Of course, it needs to not be program art or anything like that, launch it in the Philippines or some other country only and then iterate, you know, 1% as you mentioned on that all the time based on player feedback. You get an influx of of players and you iterate and you figure it out, right? I was a big fan of Plants Versus Zombies and I saw that they launched Plant Versus Zombies 3.
Mhm.
But I couldn't play it for like two years because it was in I think it was Australia and Bulgaria or something.
They iterated for 2 years.
Yeah. And they kept iterating and they kept iterating and then at one point they made it like a semi- global. So it was UK and then apparently the numbers were not good. So they removed it globally. So they went back to the drawing board. I would recommend for mobile games trying to launch in different smaller areas and iterate on your game. It does complicate things. We

talked about novelty because mobile games are often ripped off and you can sometimes see your stats for that specific game on sensor tower and other things. So kind of summarizing it, there's a huge opportunity cost with iterating over cohort because you could be building something new that actually is good and can only get better rather than something mediocre that could get good. So if you'd have to choose one approach, would you only put stuff out that is good and basically kill anything mediocre? or would you still approach in a way where you put stuff out that is mediocre and work it up until it's great
on Steam platforms? I I would probably have a more polished experience because first impressions matter uh and it's out for everybody in early access, right? So you you need to have
it's technically a launch.
It's technically a launch. Yeah. But if it's a mobile game, if it's a game that you put out on socials and you kind of have people play through Discord channels and so on. I think Minecraft

was put out on a web page, right? Very basic. that played kind of one of the first versions many years ago.
Still not much more than a basic,
but back then it was crashing. Performance was horrible, but you could you could pay like $5 and download it, right? A Java app,
whether you can afford it or not.
Yeah. Also, of course, if you're working with a with a brand, you can't put out something that is terrible, right? So, but sometimes, well, most of the time people uh make the game and then they put the brand on later when when they know it's good, right?
All right. Many things depend on the data and that's why you have data analytics.
Yeah, true. All right.
Thank you so much, Nikolai.
Thank you. Was fun.
[music]