Superstar Session: Hypercasual Live Ops- Building a Data-driven Growth Machine

next though as promised we have mikael from crazylabs who once again i'd like to thank as our sponsors of the hypercasual and social games track who we we can't do this without so hi welcome thank you for being here everyone it's good to have you here yeah as uh as uh rotten alluded to before you're gonna be uh talking about hyper casual live ops and building a data driven growth machine so we're really fascinated and before you start though i'll let you start getting there set up providing you're all good and feeling comfortable share my screen yeah just share your screen i'll let you know that it's all up and working and while you're doing that though i'm gonna let the chat know in the audience you've been very good with the questions today it's been very engaged um so yeah keep

keep talking in the chat and keep throwing those questions into the q a because uh once the talk's over i'll be back and we'll be going through some questions and answers but uh providing you're all okay feeling comfortable we'll get started okay yeah so here we go see it's crazy labs okay so thank you sophia i'm happy to be here hello everyone thank you all for joining us today my name is mikhail sakhari and i'm crazy labs vp live ops today i'll talk about the strategy behind liveops what we do in live ops and how we bring value to our games and to the company remember that paper plan we used to play when we were kids we took a piece of paper we followed it we threw it on our best friend and sometimes it hit him sometimes it hit the teacher in class sometimes it hits the window it was fun and amusing and it was a nice game but when dealing with 50 games we can't take a chance we

need to create a value fast and accurately we don't have time to see if our future hit by mistake and maybe will improve the game we are committed to provide value for our partners developers and studios around the world so we create a method the method enables us to take any hyper casual game and bring value enhance performance prolong the user value and the usage and increase revenue for the developers and this is the nice battle that you see here but this is not enough for us we want to do it fast accurate and in scale we took our method and create a technology that will expedite the whole process and take the game to the next level this is the missile you see here although it looks scary you will see soon that is not rocket science and it's really possible to get there uh what we are going to cover today i'll

talk a little bit about crazy labs about our method about our kpis and let you in on some of our secrets if you have any questions during my talk feel free to write it in to write it in the chat so crazy loves is a casual and casual mobile game developer and publisher of number one hits games with over 3.5 billion downloads our recent games success include acrylic nails icon paint tie-dye html slicing a maze and many other modes as you can see here we have a casual games lab a hyper-casual game lapse and a crazy lab well not really crazy just a top secret but i'm sure you will hear about it soon first of all what is live ops in crazy lab live oops defined as the stage where the game was proven as one that has valid business potential around one month after scaling up

so the whole presentation is referring only to games that pass some certain kpis and we know that they have a real business potential our goal is to increase the game's lifetime make sure it will be sustainable over time and will gain profit for the developers and for the company from now to the horizon so basically there are some well-known way to do it first of all is content everyone knows content is the king we need to make sure um the game has enough levels that the the gld is good we need to make sure we have a rich start that the item in the souls give value to the player and that they will excite him or her second in hyper casual we need to keep our players in a good mood make sure the levels are interesting enough but not too hard we need to create a balance between challenging and satisfying in the player and we look into the game's psychology we integrate into the game

design the principle of progression achievement and rewards let's see some examples um so in this slide you can see two of our writing styles tie-dye and food clinic in both game we haven't reached the shop and the variety of tools the player can use every time we release the new version and with additional content we saw an increase of performance of more than five percent in tidal you can see here where the battles types water tax and tie types so there are endless options for the player it keeps the game interesting which increases the playing time and therefore the revenue of course when you have interest in content the player will be happy to pay for a to pay with rv as well uh in hyper-casual games we want to keep our players satisfied and positively excited we encourage them we'll train them and congratulate them when they pass another level we make

sure they receive positive feedback throughout the gameplay so seems like awesome congratulations you made it confetti flying in there fireworks and overall spirit of happy happy happy fun fun fun joy joy joy always although hyper casual games consider this short in terms of lifetime the psychology behind players motivation is the same as in other genres we know that players like to feel a sense of accomplishment and a sense of progression they would like to know that they achieved something and they rewarded for that we can see in this specific example how these principles implemented in food clinic one of our games we show the player a before and after photo so they can see their progress and they know they did good we showed them a progress bar which always worked that pushed them toward getting

the reward of this new cleaning tool and we celebrate the achievement between confetti and relevant text the graph you see here the columns the bars is a real graph that describes our user value we had a constant positive trends by the way you can see that we have some not so good versions as well but we monitor it but the feature i mentioned in the last couple of slides improved the engagement and the user value significantly so now after revealing all these successful tests i would like to share with you a personal confession there are some people are gifted and they know exactly which feature can hit the spot and make all the difference for their game they know exactly what to do in order to create number one hit game and they have all the answers i am not like this that's not me to be honest i don't have all these secrets in my pocket i have no idea what's going to work in

advance no clue as to which feature will increase the revenue or i will double the ltv that's why my team and i test as many features as possible we try everything and focus on creating real value that will improve the game's performance and when i say everything i really do mean everything it doesn't matter how much experience you have i recommend listening to your colleagues and your team i take the time to hear them out and collect their ideas because they believe the right way is to literally test everything if i learn something in my experience as a growth architect it's that if you follow the method if you test everything if you measure it measure it very very carefully you will get to the desired result in this case the operational processes and operational efficiency is really important and i'll show you how we do it in a couple of months

here you can see the many tests we've done in order to get the optimized look and feel of amaze look at the screen which one of this design do you think will be the most engaging one so you can uh you can choose your own preferred one but after around 100 of tests we realized that this one is the natural look and feel this perform the best we would never have guessed that before actually conducting this test we encourage you and encourage the design team to improve the ux and the design but eventually we got the surprising result therefore i encourage you to test everything and assume nothing only the data can tell you what works best for you and for your audience let me give you a sneak peek of what happens under the hood here at crazylabs after talking so much about the method

let's see the real thing we have a dashboard to supervise all running tests and it's provide us real-time information and deep analysis for each of our unintended tests let's look at the numbers um i can talk hours about a b testing and how to do it properly this is not the topic of today's talk but i really want to give you some useful tips first of all it's crucial to know that abby test is the fundamental of all product improvement testing features in a row instead of a proper a b test the trans grouping parallel run groups at the same time if you don't do like this there it can cause wrong product decisions so in this specific example you can see five features that we we've tried in parallel and for one of our games group a is the control group

always always always run test versus your current status i stopped counting how many times something gets wrong with the build and without a control group that keeps the current status we would have never guest it out uh you see here some more groups a group b the new rv locations see the bonus level these another arvie text you see some features that we've tried out and first of all the first thing that i look when i see result of a b test uh i see that the audience is well splitted we have the same amount of user in all groups that means that probably a proper a b test in addition you can see that we measure revenue kpis and engagement kpis on one hand all our tests are carefully analyzed we would like to make sure that we provide value for our players and create a better game than the the

previous version our internal system gives a higher score for tests that increase game time content consumption engagement and retention then feature that just provide more ads without re-improving for the game we have an internal scoring system and it takes the kpis under the consideration all these decisions are being made by our algorithm and the machine gets the decision for us automatically so the last column is actually the ranking the ranking that we get from our growth machine but in order to get an optimized ecosystem you need to run such tests with within the funnel starting from creative throughout the onboarding process advanced level all the way to the retention phase monetization aspect and many many other

phases in the in the funnel all these tests need to run in two stores 20 different countries and the results can be different for organic and non-organic players we need to consider different traffic sources and player segments and the list goes on and on and on and on so the only way to do it is to be systematic automatic and data-driven systematic means create a method of reading and understanding the result fast we can't invest five days in discussion per each test results automatic we have tons of automation open groups automatically close including groups apply tests with the winning feature on different audience automatically automatic qa many many many automa automation spreaded wherever we can whenever we can and of course data driven as said before

all our decisions are data-driven and based on real experiments this is the real secret weapon of our liveops there is no single game changer we've built a clever operation that can push our games above and beyond we all in parallel we hold in parallel hundreds of tests spread across different parts of the machine touching different aspects of the game this way all the pieces come together and create this giant well-optimized infrastructure that can simultaneously run hundred and thousand tests in parallel that's what and that's what eventually leads to better performance to better user acquisition capabilities and ultimately to better gains that can hit the number one spot if you look at the numbers you will see that every increase conduct an exponential growth thanks to our horse

machine you can see here that if we take five percent increase and add another five percent increase and add to it another five percent increase you get a result of almost 1.8 increase which is almost double the lpp but even if you're small studio and you can't run so many tests in a small studio or a single developer you can create your own mini goals machine choose three games aspect you would like to improve and make a change even there even with these three types of successful tests you can see an increase in your user value three tests means more than 15 increase in the revenue that will allow you to pay 15 percent more on your cpi and therefore you will be able to acquire more players and expand your audience sometimes that alone can make all the

difference for a single game maybe whether it will live or die so i think that's it for today uh just before i finish i just want to remind you the three significant things that you need to take um in just a second the three uh i want you to remember this three simple steps one create your data driven method and work systematically second test everything assume nothing and third one don't look for a single game changer go for an accumulative increase in multiple games aspect so that's it for today i hope you enjoyed this information uh if you have any question or you want to contact me in person uh you can see my email and content please directly and now do we have any question we do

we've got uh quite a few questions so i think we'll dive straight in and see how many we can get through so the first question from psy is uh what tools do you use for setup um for setting up a b testing and checking the data how do you track um you know the impressions per user in an a b test um so basically when you conduct an a b test you have two steps you have two parts of the a b test first you need to split the audience in even way as i described it before and then you need to track the events accordingly or specifically specifically for each group so basically we use a firebase and with the remote config and this works for us really really good we integrate our internal events into a big query in firebase and that's and all our ebitdas methodologic and algorithms based on that infrastructure

infrastructure fantastic thank you and next question is what tools do you use for analytics and automation um so we have different first of all we have the we have also that's going to be like one how one hour uh talk but basically we have our the event so we have uh bigquery events we calculated and we aggregated them into our data warehouse and on top of that we have many many statistical models written in python that integrate statistical models that calculate exactly what are the results what are the kpis they check statistical significance for each test so we develop in-house tools within a node i think but like our web developers our backend developers uh developed for us uh internal dashboard

that's built on this stack fantastic and next up is uh do you consider lives ops alive ops as a temporary uh short-term event or a permanent or about permanent updates to a live game okay it's a good question um first of all the hyper casual is um it's a genre that i think like two or three years old so it's still evolving i know it was considered as a really really short 10 term janelle and each game considered to to be live for a couple of months and then die from the test we've done in the last couple of months of the test we've done in the last six months i see that the lifetime of a game can be really really long girl it can last six months or one month or even one and a half months if you keep the game updated and you

integrate all the time good and valuable feature you can uh prolong the game lifetime so fantastic i know that it's not that the common i know a lot of people think that it will die really really soon but i see that valuable content can increase the lifetime of the game yeah and then just a few more quickfire questions which we've got is um so i'm all mainly about a b testing so the first one is how many players are enough for a b testing oh it's a good question so um so it depends because uh when we implement the test if our machine or our algorithm see a huge difference between the groups so we need less players for that yeah so maybe sometimes only 5 000 per group can be enough but if the changes and the differences are really really small we need to collect more data to accumulate more data in order to have statistical significance and then we can

sometimes get to 20 or 30 000 players per group fantastic and and then another one on this sort of uh area is how do you handle ua and game feature a b testing in parallel ua and um game feature so um if you how do you effectively test um ua and game feature sort of analysis a b testing but at the same time so testing the user analysis and the game features yes so as i saw as i show you in this giant infrastructure yeah all piece of the system is actually act separately so we have creative tests and we have testing the stalls and we have testosterone the game and we have tests on our monetization and everyone can really really work independently and we make sure that the tests are not hurting each other but our system know how to isolate it

and just uh work simultaneously so this is exactly it's a great question and this is exactly the machine that we have yeah and another question is um just so they're specifically focusing on the prototype stage do you still use um methods such as a b testing during the prototype stage or is it different than liveops yeah so it's a little bit different because in the prototype test sometimes like the incremental improvement that i show you it's not enough in the prototype you need to bring like 30 improvement of 40 sometimes you need really to double it in and you don't have time to run all these tests so actually in the prototypes we do run sometimes everything from the creative but the game needs the a b test is just too slow and you actually there you need to implement significant changes or feature really really fast so i will not recommend this methodologic

for the prototype phase that's great and then one more that we've got in here is um what time frame do you use for reviewing a b tests uh it's a good question uh we are aim to have final result um in two weeks uh actually um it's it's about we have some parameters because we we can check out after one day with value or after seven days or after 14 days so usually we try to get result after seven till ten days but actually well today we try to shorten this even more so it's working progress we've got a few more questions coming in so if you've got a minute or two so we've got um do hyper casual games allow live ops events as often as casual games like how do you plan for those is it different or is it the same

uh it's a good question i must admit that today uh um in crazy labs we don't act in specific way as casual games we uh we don't do perio we do really low amount of periodical promotion like spring promotion or things like that or daily so we took some of the methodologies uh that works in social and we implement it but i think it's work in progress and this is the way the hyper casual will evolve too i think it can work yes still something that needs to be proven um and then um we've got if the same a b test in two different conditions for example two different uh geographical areas uh give opposite results what do you do there a lot of time yeah and actually we have a lot of this internal discussion uh because sometimes if we have different results it means

that we need to hold two different configuration of the game so if it's important enough we do it but we understand the operational consequences because then you need to hold this separation for the longer term and q8 twice all the time but sometimes the 12th the effort and we do hold different configurations of the game for different audience and to be honest i'm glad someone asked that because that's one of the things i was thinking about when uh when listening to the talks i was hoping someone would ask that so yeah glad to see that um and then um so we'll do one more question and then i'll address the fact that something that someone's put in twice and so the last question here is uh do you take liveops into consideration when analyzing which games from external developers you want to work with if you take liveops like if they if they've i i think what the question the person asking the question is is trying to ask is if the if you're working with it looking to work with a

developer to publish their game and they've already done some live ops would you want to look at their data or would you want to just do your own data from scratch um first of all me personally i always challenge everything i see so there are some things that i know that already worked and they are good sometimes i see a feature and said okay only for me let's test it again because i want to validate the result and the improvement and in general we're really really happy to integrate the knowledge and the experience that external developers already have and in addition we have our own methodology so it's i think it's a combination yeah and so i know frankly uh this last one that there's someone has asked twice for more information on it's not really a specific question and um i don't know about you but i i you know it's probably a talk in and of itself but someone's really wondering if you can share any further insights into doing multiple a b testing at the same time in parallel so maybe within the same game if you're testing various different things at the

same time is there any lessons or takeaways you could quickly share um i can tell you one specific example for uh because if we do a monetization tense test and we try to uh to tweak the monetization parameter how many ads to show and which with uh which network tool and things like that and in addition we do game change and we introduce new rv new content so if with this specific two parallel tests they can cause like an issue because they um affect each other so we we have a dashboard and we see all the open tests any moment and the analyst and the machine gets is resolved but we our eyes are open and we see that maybe it can cause something but if it's like a gld or the speed of the knife or we change the color of the background so it's not related to the monetization so such

tests there is no harm in running in parallel so our system allows that and we monitor every time the the result and make sure that we understand what we see and and there is not a conflict between the tests fantastic well thank you yeah and i was going to say to whoever it was that i asked the question because it came through is anonymous um i'm sure you guys are going to be you know all of you from crazy labs will be in the discord or people can reach out to you if they want to carry on asking some questions yeah feel free to contact me in this card or my email is here and feel free i will be happy to talk to anyone fantastic well yeah like you say actually yeah your emails behind you so yeah please anyone if you've got any more questions because obviously we've sadly run out of time but you know the questions kept flying in so very popular oh someone's saying the email is slightly hidden so if you oh sorry the other way oh there you go that's it we can see it now so i at there you go um so yeah and then we'll try and get

someone to um uh write it into the um the chat as well for for everyone who's still here after you go okay thank you yeah because it's a mirror one so maybe maybe i'll write it now for everyone okay there we go and then everyone can reach out crazylabs.com thank you thank you for that there you go um so yeah so um that's that's oh i've only shared that to panelists but i will share that to everyone now um so just give that to everyone there we go everyone so there you go so yeah thank you fantastic talk and thank you so much for staying with us for all the questions as well yeah so um sure no problem i'm happy to share uh whatever we do here yes but obviously a fascinating talk that's really ignited our audience and we've had a very engaged audience today so thank you to you for not just the talk but to crazy labs in general okay thank you so much i was happy to be here thank you for the fantastic event really interesting thank you and thank you for letting me speak here

oh no and thank you for sponsoring this track as well you know so thank you very much okay thank you sophia bye bye bye bye everyone you