two & a half gamers session #49 - How to leverage User Churn Prediction? Why is it important?

[Music] welcome everybody this is no gaming podcast two and a half gamer session number 49 we're sharing actionable insights dropping Knowledge from our day-to-day UA game design and AD monetization drops definitely not discussing the lightest industry news but we are having so much fun let's not forget this is still 400 a.m. conference discussion Vibe so let's not take it too seriously today it's a special episode so we are not talking about ad monetization so Felix is not here but we have uh yeah no decimals no decimals this time we have clemon and and Cameron from adictive and the usual suspect yakob and myself mate so before we dive into different topics but well there is only one topic today which is user churn prediction so come on cam can you can can you give us a little bit of intro

from from your side clam let's start with you sure thanks for for having me so I'm I'm Clen CL favier I work for Addictive company that just celebrated it 10 year anniversary couple of weeks ago actually um what we do we focus on app retargeting as well as cross promotion for apps and games yeah and my name is Cam I been with a company for about five years working with clemont uh first on retargeting but then on our newer products around uh cross promotion but how you guys get got into the into the games and then addictive afterwards that's that's interesting part also let's let's uh bring that in as well for for the audience I'm interested as well we know addictive so now how about yourself clim how did you get into games

um well by joining additive which was uh more than seven years ago now um definitely wanted to work in uh in startups and uh was uh yeah the first one I had the chance to to join and uh it's been a great ride since then joined as a more on the data side so that's why I so interested user Chan in general uh and then able to Value roll and now I'm uh Co nice cam how about yourself I was uh doing sales in the US healthcare system which depending on where you're from um you might know as like the most horrific thing ever and so uh it was uh quite depressing to be into uh and ultimately I needed something that was a little bit uh lighter in tone and also um you know living in San Francisco that the cost of living is

quite high so I looked around and was able to connect with the VP of sales at Loff so it was one of their first um hires for sales and you know there weren't a lot of mobile marketing sellers just walking around at the time so I started with another guy who came from outside the industry as well and it was probably the most difficult six-month ramp I've ever had just having having to go to conferences and talk to people like yourself who had been in the industry a while and start to disperse some of the uh companies that say they do everything and really understand what they ultimately do and and start to kind of get the lay of the land with the industry but was really taken I think my first conference was uh GDC in San Francisco I bought my own ticket and went just to network and talk to people but really just uh how how wonderful and sharing a lot of folks in the industry are with knowledge and expertise and so have been kind of hooked ever sense and

uh had my first child and so couldn't do the commute anymore and uh ultimately was looking for a company as good as liftoff and their DNA and kind of core values and what they were doing and liftoff was more focused on UA at the time and AD dict presented A new challenge with retargeting and so was really attracted to what they were doing and came on board about five years ago nice also we back from PGC Kim you're already home or you're still in Europe I'm home I it was a quick trip I got in over the weekend just experienced London in a few days and then flew back on uh on Wednesday so I it still doesn't seem real but um the San Francisco weather is very akin to London so I think it's all it's all good so we're back yeah any any any quick comments from from your end before I start uh um kind of ranting about the PGC it was it was a good conference though it was a good conference I mean I I I'm sure you had a ton of insights

from the content and everything um My overall impression were just like people are very hungry to reconnect I haven't been to too many imperson conferences and I think there were over 2,000 people there which was pretty crazy and then uh you know seeing the Dance Floor on Monday night particularly with so many countries and dancing Styles represented was uh amazing to see I was just in awe and then uh I think from the conference level just like m&a and web 3 are are very much in full gear despite the economy I mean a lot of people are finding out about Job losses and things so a lot of networking around that which was great to see people reconnecting and finding new opportunities but um definitely web 3 was still a huge part of what people were talking about um as far as what I saw yeah it was good to good to meet people face to face it was really really like overcrowded there's almost uh nowhere to to move uh the venue yeah the the talks um I guess it

was okay but the level of um know developers versus service providers or the well the percentage or the ratio is uh definitely way more skewed towards the the sales provid service providers uh I I saw I saw one panel where there's like one developer surrounded by five s service providers and it felt really interesting so definitely hunting him down um so I guess like that would be my my feedback to to PGC when I I talk to them to actually try to change this a little bit but on the other hand well it was great the the Dance Floor on on Monday night especially after the the official Afterparty closed and we just went to a random bar where there was nobody pretty empty and then in 30 minutes it was like packed like 100 people it was incredible came came there that great to

speak about the service provider set of things too it's like you see everybody having to deal with Co and being totally overwhelmed and then they're thrown to the wolves with like five service providers I feel like half the mission for us is just to to let the the folks know we're not serial killers and to be really respectful of them getting hunted by everyone you know so I can I can definitely see it from both sides no I I am in a very nice position because well I'm the the UA consultant so I'm not being hunted that hunted that much but I know like some of my former colleagues they said oh when yeah you get the you see the glaze look on their eye and you're like kind of flinching from you approaching so it's uh it's good to be respectful of knowing what they have to deal with for sure and anything the the the conference providers can do to help support that um is good too like the meet to match system is a little bit tricky the online system to create meetings more of like a cultivated approach to maybe help match

up those folks for what they're looking for could be good in the future yeah it's like meet and match system I tried to open it once or twice and then no it's not happening it's not happening okay well yob you weren't not there clim I I didn't see you as well in The Addictive crew so I guess uh next time yeah next time yeah kids du to this time Clen was steering the ship while all of us were dancing yeah the I was answering him on slack to make sure he knew I was still alive but that was about it nice all right anything interesting happened in the last uh week for for you not so much regarding the news or regarding the developments guess like your life come on my life like uh everything's hype up about the new Harry Potter game I'm really looking forward next month so that's that's that's still on the shopping block I know you hate open world but it's going to be pretty big

it's the most wishlisted game on Steam currently and it's going to be very very big okay very very big let's see let's see okay let's let's talk about the the user TR then uh I think everybody knows uh what's user turn but as a reminder um so I think like there are like two two perspectives one from the from the Game Dev side and then one from let's say the marketing point of view so yob can you can you tell like um the audience like what is your def definition of of user CH and like how you kind of take a look at uh like look at that usually when we were looking into it it also boils down uh how you define it because I know that we were talking about before and I know that let's say on the mat three project I work before we would count something as turn where users didn't log in for two days some people are more lenient and they

stretch it to something like five days even maybe a week depends on the level of Engagement some games has just different rhythms than the others it's it's normal but what ultimately you want to get is some kind of a formula prediction or let's say A system that will pretty much tell you that okay this is a player that's entering this kind of a risk group what can we do with him before he actually leaves because that's what the prediction is all about of course and and the other problem is there that uh like for instance one of the best predictors Fortune for me uh during the years were uh lower and lower session counts rather than not being online because not being online can mean that let's say you were in a holiday or there was some kind of event in your life that happened and you didn't have time for a game but you log in so there conference yeah a conference so like three sessions three sessions three sessions zero sessions zero session and then suddenly again three sessions that and but you already counted this turn if you you

know your your model is kind of not accustomed to that but rather than for instance the typical turn for me would be something like three sessions three sessions two sessions two Session One sessions zero sessions where you can clearly see the lowering engagement going forward where that was your time that you were supposed to do something between that time so that would be some kind of let's say basic definition how user turn I was like en countering this throughout the years okay okay thanks Al but how about the like the different perspective actually the marketing perspective so guys how are you looking at at user turn and uh and then not only the the user turn but then actually like the prediction that yaku mentioned honestly when I was doing all the UA stuff then uh well it wasn't really a part of the the daily daily U

yeah for sure I mean on on our side it's pretty much of course a data science job to to be uh predicting that and we consider user churn basically as the probability of a user to not come back to the app um and that's basically what we're trying to assess with the highest accuracy possible uh with our data science team so that's that's really the that's really the goal okay uh and with the data science team so are you are you building uh or obviously with the prediction so so how does that work so do you have uh machine learning models or or what kind of uh like how how is it work how does it work AI please tell me AI yeah yeah AI obviously AI what else yeah yeah we just write the prompt and then it's all comes out very nicely uh no but I think the the data

science process is is very uh very interesting um basically always starts as any project with the the business need um so why are are you doing what you're doing so in this case it's in the end ultimately it's about increasing LTV of users and the retention of users within either your app or your studio basically how do you want to do that well by identifying users who are at risk of living basically to uh in order to perform some marketing actions to keep them within your ecosystem and what what do you do to do so well you build a predictive model that's trying to predict if users are going to that's sort of what it is about the second step is you define the data science goal so basically it's the metric you are looking to optimize so it's really a data sense metric could be accuracy the one we we use a bit technical it's called Au Rock it's called area under the curve um and it's very important because a machine Lear model can be very

good at one objective but very bad at another one so you really need to say okay what is the metric you are going to optimize so I said in our case it's a area under the the curve it's basically a metric that actually checks if we're ranking the users in the right order so basically if we compare two users one who CH one who didn't we want that we predicted a higher churn score for the one who actually churned basically that we're ranking users in the right order that's really what we're aiming to do the next thing and it's super important because that's what's going to allow you to compare all the differ models that you're going to build uh the third step is you need to define a baseline so what sort of a basic model that you could do and what is the the the accuracy of this of this model so if you were for instance to do something random like fli a coin it would be 50% accuracy um that's super important because when you do your models you want to compare them to this Bas Baseline and

just to give you an example for instance for user charm if you were to do a simple model that would look at you know the past activity for the last seven days and say okay you were active in the last seven days you're not going to churn you are not active in the last seven days you are going to churn this model on average would be 75% accurate or even 80% accurate in some cases so if your data science team comes with a model say oh awesome we need a model 70% accurate well in the end it's total crap just because you know basic rule of some would have given you would have given you an 80% accuracy so that's very important to have a baseline that's basically what would the marketer think in terms of yeah a basic rule of some to do so um based on that you build your model so usually they read you know a scientific literature try a couple of different uh machine learning approach I yeah the word and

uh and finally you compare uh that to your goal so in this case we compare the Au the accuracy let's say and finally once you've something that is satisfying you implement it in production and you monitor with a couple of kpi just to check you know if it's running um well on a daily basis and if the accuracy is stable over time okay so do I do I get that right that if I build something at home with my random data scientist and we get out with 70% accuracy we are considered Baseline crap yeah for model don't okay I don't want to have any problem with it all that I S Ste in the gaming industry please no course don't worry yeah yeah this is no gam podcast nothing else for us positive thinkers out there it's better than better than random so it's glass full thinking yeah oh yeah okay okay but then uh two questions so what's the like the

minimum amount of data to have at least like some statistically significance and then if 70% it's crap then what's it should the best case scenario is obviously 100% I guess right so but any yeah um so in terms of uh statical significance uh B Bas a number of users so I would say um even see that 100K ma even a little lower could already give you some uh interesting results what you're checking basically here is is your accuracy good enough with the volume you have and um is it sort of stable over time that sort of the the two things you're going to look at to know if you have enough data to do so and uh in terms of a good accuracy what we seem to be reaching on a regular basis would be around 95 Au Rock So 95 %

accuracy 95 okay how how reliable would you say then are these models regarding different sources of traffic because if you train into of one source of traffic and then suddenly you know whatever new creative comes in or whatever ma starts to do crap with his job as usual look at this guy on the horizon there's pretty much new new kind of traffic going into and is it going to you know destroy the model or how how reliable it can fit with it yeah yeah I would say the model is doing uh predictions as if you were to do nothing and then every action every marketing action you you do is basically to sort of counter that so the output is the model if yeah if you do nothing this is sort of what would happen um and then any articulating action are going to are going to help you so it's going to improve if I understood exactly it's it's going to improve and exactly and just so so people understand really the this

Au measure it's basically we check if we rank users in the right order so basically we're going to compare uh two users one who churned one who didn't and our goal is that the user who actually churned had a higher churn score than the one that didn't and we do that for all the combinations that are possible and in 95% of the cases were correct so there's only 5% of the cases where we said okay this user turn and this one didn't and the scores was actually in the opposite direction that sounds like the usual kind of a scientific uh significant statistic like the what's the U you know the the pp value yeah sorry um okay uh but then you you mentioned uh obviously kpi so so what what are usually the the things that our company is looking for I I would guess

retention but uh well retention is nice but then I eventually well aiming for getting Ras after like out of different things so which way retention monetization how how think about it how can this kind of like improve these uh these kis yes I guess the another way to do to to phrase the question would be yeah you've got this nice CH prediction model it seems to be accurate and uh what what do you do with it right yeah yeah yeah okay yeah yeah that would be exactly and why defitely the good question right exactly because yeah we have this nice model we had fun building it but what do we do with that um so yeah there's definitely a lot you can uh you can do um you can leverage it for I think three sort of categories for

product in the product perspective uh for owned media such as you know SMS emailing push notification and for paid media such as um retargeting and and cross promotion and uh we can we can deep dive into into those three if You' like yeah of course because I I'm pretty interested to to know like how do we keep the the the players active actually and yeah you me mentioned push notifications I would say well there are other things like email marketing retargeting so like uh which way to go uh because obviously yeah we have model but then that's only the part of the of the job yeah and you have to take into account the LTV of the user right and kind of the genre of the game that you're looking at so for hyper casual retargeting doesn't totally make sense all the time um user LTV is quite low and that turn is naturally going to happen super super quickly so you can look at Cross promotion is a solution for that because you're just keeping

that user within the the ecosystem but then for more of a mid to hardcore game that you get you know typically a large percentage of the revenue coming from a small amount of purchasers it's looking at the LTV of the users within the churn model and kind of predicting how to avoid cannibalization um you know before moving that user to another game or keeping them in that existing game through a number of mechanisms like um in G events or you know uh other incentives to to keep them coming back over time would you consider would you consider being like the model being used more in a way that it tries to kind of increase the LTV before the US a churn and it will eventually churn or rather tries to prolong the retention in a way that it significantly increases the player kind of Lifetime spent time within the or a game sorry this time I mean like

those are kind of very big trados if I understand correctly that one thing is we're trying to squeeze more money as much as we POS as possible in a way that probably at some point the model will tell us like this user will turn anyway so that that's one way so let's throw him a discount that's like completely no-brain 500% discount let's cut the spending dep in half whatever this is the last resort we can throw at you or rather whatever let's make the game much more easier rather for him to kind of keep him within the app probably like that pesky level in Candy Crush that have you know blocked you for three months finally it's very easy to do yeah I think it can be used in both scenarios right uh concurrently so you had the product team really focused on eliminating those roadblocks and then the marketing team going after those incentives to to maximize LTV and then ultimately having to do that trade-off between cutting their losses with uh retargeting and looking at Cross promotion particularly

with what seems like a more steady release schedule of games coming out this next year it seems like a lot of studios are focused on being more aggressive and uh I think more timely in their release schedule and so uh predictably moving those users along if they're stagnant or or much likely to turn um from the statistical perspective any interesting uh interesting numbers that oh sorry G no like just want to hang a little bit more within this cross promotion team because I think it's kind of important here uh like in my days or let's say in in my eyes if I look at the market I think there's like very very few companies or even genres that can do this would you say this kind of limited per genre because if you let's say whatever AFK Arena you don't have another AFK Arena sitting somewhere in your portfolio that you can move the users to but if you're King you have 50 other match threee games or play RS so I guess that's much better but rather than the question

would be would you still consider it be effective within current days of like the new market Paradigm of idfa and everything being cut down L like pretty much being it the focus currently that people kind of try new paradigm my God where the I am seriously keep if if you start talking about tectonic shifts and I'm I'm just you know hanging out immediately content fortresses yeah course yeah I mean so so one thing that we do specifically is use idfv um in lie of idfa to have a way to to to connect with those users but you're exactly right I mean if you have a couple M to hardcore games there's not a huge use case for it unless you start to launch that release schedule and I think with the Paradigm Shift of how how how games are some good mirroring for you uh yeah how games are changing their monetization strategy there's a lot of um I think changes in how that model is

going to look hyper casual with uh you know what's happening to ads and other other genres having to be more aggressive to have more games out there there becomes a larger use case for it how quickly that gets adopted is is TBD Eric CER and his predictions has you know recommended everybody to do cross promotion but it kind of feels honestly in conversations like the early days of retargeting where maybe some had tried it I know mat has had some successes with it in the past but um you know it's specific to the company and and a lot of these companies are set up to run UA specific to each title and so those teams coming up warming up to the idea of sharing their best users is not totally a a happy one so it relies a lot on the politics of individual companies and their adoption rate of new technologies and their willing willingness to test I think that's always kind of the litmus test for how how these things can be adopted and and

also how they get measured you know I think some of the attribution companies have done a good job of starting to set this up for the future where it's very similar to UA in terms of How It's attributed but cost um just without the cost because it's not you know you're not buying a user from the open market you're kind of moving them along but generating installs so looking at kind of LTV over time as they go throughout the portfolio and looking at portfolio left over time but there are still like the I think there's a sorry sorry just one one comment there's still discussion about uh like like you said if there is no like another AFK arena in the portfolio then well you shouldn't do the cross promotion because it doesn't make sense i' I've seen it actually does make sense just needs like a proper segmentation uh and and you need to understand like what the target audience is and if if there is at least like a small overlap then it can actually create a lot of synergies

yeah I was thinking that sorry I was thinking a scenario where you don't have another title um but in that scenario you're talking about absolutely and if you have couple of uh of apps within your your portfolio what you can do also is uh build some Affinity score which you know would be okay what is the other title that is the most likely to uh be of interest for my user and then you're just going to basically recommend uh that one specifically with your with your cross promotion I think I like even our favorite data platforms would have the statistics back in the days I don't know if they are still available that you can see like a game and then there's some kind of affinity score of that game like this user plays Clash of Clans so he also plays this other game yeah usually they would provide you a overlap of your

title with other titles which is sort of the level one of that and then you can run some let's say a bit more sophisticated model which would take into account a bit more data points um to build a model that would recommend okay what's the the most likely uh app you're going to you're going to be able to download at the user and that will interest you any other interesting uh data points you can uh you can actually share about the the user turn that uh you know we didn't didn't touch still um indeed indeed I think uh gaming is is actually quite interesting when it comes to uh user CH in in general because uh I mean those are apps you're supposed to be using every day at least for for most of the games and so basically as soon as a user doesn't open the app for one or two days uh that's already a big a big warning while for instance for shopping app or

an e-commerce app once you made a purchase most likely you won't open the app for a couple of days right unless you're a a very very heavy purchaser but then at some point your credit card might might uh you know limit that um so uh so that's actually very interesting and we do run a lot of interesting analysis on um yeah inactivity window of the user so how long the user has been inactive and what's the likelihood of the users to come back and uh basically if you wait already two days of inactivity uh those users have only 25% chance of coming back if you wait seven days then this probability is going to fall below 5% so coming back to jakob's comment at the very beginning I think that's why a lot of gaming company would say Okay a user did not open the app for seven days I consider this user charm just because the likelihood that they're actually going to come back by themselves is super is super low so

that's why in gaming it's very important to try to predict that and to act really early um and not wait too long because otherwise it's basically like if you wait too long it's a bit like doing UA all over again on your on your users that that actually CH basically how how early can you well let's say the model that you have been working on act upon like when's the fastest time in way that it tells you like this user is looking like it's going to turn like two days before he would actually turn three days or like what's the time between him actually churning and you are determining the you know risk group I would have to check I think we leave them a s day window you know to come back and and check if we if those users actually actually Trump um and we can start predicting basically right after the right after the install where it's actually a bit more

difficult to predict is for users um that have been here for a long time you know because you don't have the the starting point you don't have their their first um yeah their first touch point with the app for those users usually it's a bit it's a bit harder uh to to model also mentioned 100,000 Theus but yeah go go ahead yeah no like for me like this would be very very kind of let's say rewarding in a way that if you can actually like bring back those whales or like prevent the whales or very heavy Spenders or power users for going out because those I guess are the most kind of I always so like I have like this one other game we were working on and I know like we were making content for it like crazy and just 6% of the player base was at the end of the content it was eating it like crazy and that that was the power users because they were responsible for like I don't know 50 60% of overall spend or

something and and those are exactly the ones that that they are there for years and it really pays off if you can prevent their turn so that would be something that I think is kind of important like if you say you start ranking the user if I understand from the beginning so yeah would it would would it seem something like like 100 100 100 for like one to two years and then suddenly the score would go down that's how I would kind of imagine it so you mean in terms of the distributions of the of the score yeah you would have honestly quite a big chunk of users with very very high scores so very very high probability of of Ching I mean I guess it's not a surprise right we we we know the the stats of the the the few users that actually keep playing the apps after let's say 30 days po install definitely definitely super low so obviously no no difference in the in the CH prediction you have a big bunch of

users with very high likelihood to to Chun for sure and then it sorts of um smoo out decreas yeah smoo out exactly and I think the when you talk about waiting seven days the probability goes down to 5% that kind of speaks to the the rule based or B the Baseline model and you know you can be losing your really valuable whales unless you have that more specific model to your business and I've invested there where it's cl to the 90% range so you're more AC accurate in your projections going after those those users given so much revenue usually is calling coming from those whales and so really being as accurate as possible in leveraging that versus kind of the the rule of the mob and that you know it's going to go down to 5% after seven days on average but you're not not speaking to the specific behavior patterns of the whales who are highly engaged over time so you know it's it's worth

kind of uh dividing between those two things okay now I'm getting back to the 100K daus that I wanted to ask didn't let me but like it's like not every game actually out there is able to get 100K daus so if I have a small game then what it means like I'm pretty much and then then I can just say goodbye to all of my you well CH users and I can't yeah I can't use the the prediction no I don't think so at all I think it'ss yeah go ahead it speaks to the different tools that you have at your disposal to reach users so in-game messaging things like that you know uh are very easy to reach those users and and get them on their on their Journey um for cross promotion reaching you know your active users per day to drive meaningful scale and rationalizing the investment and a product around that or an investment in that um it can require

that 100K da threshold just to see the minimum amount of installs necessary to rationalize that investment but there are a number of ways that you can also reach those users outside of that um at times in terms of those who have opted out that can be a challenge because reaching those users is is getting more and more difficult so I think that's where the investment becomes rationalized and being able to reach even those users who have opted CL your your comments yeah um I would say otherwise your your back with the with the Baseline model which is okay so he's not completely dictated gu okay okay um no no but you can still try out uh because speaking to our data sence team it's there actually said you could try actually below um but I think it's about trying and checking again this accuracy to see if if that makes

sense for for you basically yeah because uh in that case like what kind of uplift can let's say on the retention side can I can I actually get in terms of the prediction for for the user turn because it would be definitely like uh rationalizing that uh that I mean if you have those 100K users and you start even low even low L okay even lower of course because for those like smaller companies uh you you know they're very much you know about uh the margins and uh yeah 100K users the is very very high even if we are talking something like midcore because midcore games are kind of very very uh user spend heavy but regarding users themselves it's kind of low that's why you have those crazy rpu because that's how it works and that's that's connected to to my comment because uh they need to know if if this is actually worth it so if you know you say there's very nice

uplift uh that they can actually expect uh then obviously that's point for the for using these type of things so any any any data points you can you can share or anything yeah indeed um so again what's tricky here is that the the uplift the sorry the the Chun prediction model per se doesn't do anything it's really about what what you're making out of it so uh um yeah it all depends Gage garbage outties CRM and and prod team okay but yeah like there's some couple of you know interesting appr pretention Benchmark and that abers or things like that which give you some sort of overall numbers which are very interesting and um basically they compare you know in terms of games um companies who run uh owned media remarketing versus those who don't and in this case you get a desert

more than 200% uplift so it's really amazing and meaningful obviously and for games who run paid media remarketing so typically app retargeting then uh they have a 123 desert retention uplift so that's sort of what you could get and if you apply a CH prediction model on that you can I guess easily double the double the results so that would be a a bpark estimate for for what you can get okay yeah that's that's pretty nice uplift or well yeah really nice number y from your point of view if you have a when you had the the the prediction J prediction model in plays in terms of the you know the match three game that you worked before uh what kind of activities you could done based on those number uh numbers and I would say it's it's again dependent on what we are when which genre we are in because for instance a match three game I would say this is

like bread and butter of the whole game yeah like this whole just trade off between retention and monetization and this kind of frustration tolerance at the end of the level like how much losses a user can take before he converts or leaves and you are pretty much AB testing all the levels all the time so I would say this is some kind of a special scenario here because match three game is in the end a churn model within itself that's like how the whole thing is based and that's why it's so difficult to build and everybody thinks like oh it's just dropping candies will make a match three game that's why there's thousands of them but actually the machine under it is the real magic behind it so that's that's very specific but if we take something like a midcore RPG for instance uh that I work on before I would say it's more of a the difficulty difficulty is definitely one thing and we know for instance we were having a different matchmaking uh offsets within the game so let's say a

PVP uh element was there and we were definitely skewing the some kind of a win rate that we wanted to get out of the user even though like it it seems like it should be 50 50% but if no if if you are kind of winning losing too much you're definitely more prone to churn and winning is always the best like we talk about it on the podcast numerous times for inst Marvel snap does it like you're playing Bots uh in in the beginning of the game because the developers wanted you to win because 50 50% win rate isn't even considered fair by humans to be honest like there were studies that people don't consider 50 50% Fair they consider something like 70 to 30 so it's just as you go and yeah I think all of these things can be utilized but again it's it's heavily dependent on how do you know your genre and your game and then when I have this information that like okay this user is let's say his you turn score is going very very low so let's do something and that's the that's that's the product

expertise that you need to do like let's offset his uh met match matching ELO so we can increase his win rate let's give him some kind of more resources out of nowhere so let's say suddenly we increase his um daily economy resource game because that can be done for instance I've seen that in U random dice game uh if you're losing you're getting these kind of consolation prices so they're actually bombing you even even with rewards after you lose and then there's lots of these like I even seen that this kind of lower activity event that can be triggered where I actually didn't log into the game for like two days or one day they automatically triggered this event that oh log in every day and you get this reward and that at the end of the seven day there's this big reward and it's it's completely custom made to the US it there's no like Global liveops running so they have it just set up like that that it's probably triggered by some kind of over activity and then it immediately gives the uses some kind of a prevention mechanism nice okay so that's from the

from the game side okay any so I guess like uh we can agree on like this is kind of important from both marketing and then the the game design side any like final advice for for companies who want to kind of like Leverage the user and prediction what they should do and what they should definitely not do and uh try to avoid the the mistakes that you've seen that could happen potentially yeah I mean for one it's to to look at if retarding makes sense leverage you know as much as you can of owned and operated channels but um you know work with a good partner to run a pre-launch analysis and look at user behavior and see if retargeting makes any kind of sense for you monetarily and then also start to structure your

company away to allow for testing of cross promotion and the proof is in the puddings to see if it see if it actually works and start to to build around tests around that even you know at that da threshold we discussed it could work just depending on your game and so um have an openness to testing mindset I would say or would be two of the keys uh along with really understanding if retargeting works for your your game or not nice Perfect all right thanks uh thanks guys also thank you very much for listening uh definitely subscribe I'll put some some information show notes uh if you want to reach out to CL or Cam and uh yeah see you next time thanks byebye y [Music]