The Science of Decision Making: How to use Data to Make Better Decisions | PGC Digital #1

thanks everyone for joining what what I want to discuss today if how do we use data and supersonic in order to build the best games and to actually squeeze the lemon and make them the best business yeah out there it's a topic that we can cover in a couple of hours so I'll try to minimize everything to the next 20 minutes and I'll touch the main highlights of each area but but of course that afterward and later we'll be happy to deep dive into many of these different topics in more detail the way which which takes the course much longer and just just to slide the bar supersonic so so you understand the background so we actually founded the company in the beginning of this year in January we are owned by IRA source but we're an independent company a gaming company Erin sauce if you don't know is today probably the biggest SDK Network

and closed platform for games we're based in tel-aviv and we're doing both publishing of third parties of developer louder and in-house development of games we actually launched three games in in the past six weeks and ultimately sought it in jointless and all of them got to number one or two in the US and gained more than 50 million installs already John which is relatively new it's our latest lunch from a week ago so it's still around four million installs but but no doubt all of the HP we'll get to to few dozens of millions away installed and what I want to touch is today is how do we pick the game how do we manage it as a business how do we scale it to the level that we managed to scale did three games in the past two months and I'll start with the short intro before we deep dive into the live option and what happens after the game is out I'll start with with the first step which probably

is how to evaluate the game potential based on their of course how do you decide whether to launch a game or not and what to do so in general of course what we are all looking for is a game that can have LTV that is higher than P I in skill right if the this is the balance that we are all looking for for every game for every genre and I think that is here and and I think that in hyper casual ad based games what what is crucial always is to start with CPI in order to once failed two quick second because it's very easy with minimum investment to get data from the match read which from the market to get actual feedback whether users liked the game or not considering successful trends and of course testing few different videos to make sure that the feedback that you get from the market is really upon the gameplay itself and not upon specific creating that works better or not what

we always do is we are using several campaigns with several different videos to make sure that again we are actually trying to understand the real market ability of a game only or is very very early stage we're looking mainly on CPI CTR and CVR and we're looking for everything that fills us that it can be marketable in scale in general of course hyper casual we are looking at the range of 30 cents CPI in the US iOS to get a good indication that users actually liked it we were also looking at the city of something we start with the city and we are looking at the percentage of the future hopefully to be more than 5 etcetera back but my message here is that always start with the CPI have several creatives to make sure that the data is not biased through toward a specific video or specific edited video and look at all metrics which are many CPR City FC we are to understand the marketability perspective of the game

then before we are deciding whether we continue to continue this game or not we are also looking at and trying to predict based on the data that we have what will be the LTV of such game right we are looking at the play time we're looking with the retention we're trying to evaluate why does the beacon we achieved after adding ads after adding different mechanic in order to monetize the users and we're building our own formula this is one example but we're actually looking it from the few different end we are looking at play time looking at retention and try to evaluate with the reward video with interstitial with banners with IEP which LTV can we get and after we have this and we can kind of kind of predict based on the data of their very initial test whether we are going to get to a place where L TVs can be higher than the CPI on scale then we move into the publishing of such game and distribute it globally and this is well data becomes even more important because in the first phase you have

minimum amount of data and you try and you try to predict well but this is what you have you have some data and a lot of predictions and a lot of models in order to understand what will be after we launch but once we are launching the game then we have real amount big amount of data and this is where optimizing and using the data in order to make it a great business becomes super super super important so let's start with with with how do we optimize the operations of a live game on both fronts both motivation and user acquisition so first of all say that in order to optimize monetization specifically specifically I'm speaking about Ed based game we are looking at two steps step number one is how you implement the ad units within the game and how you are artificially convert users into impressions and step two is how well are you managing your waterfall in order to convert these impressions into dollars and then this is how you

actually build the LTV in ad based game so I'll start with the first first step of how to implement the ad units and to generate impressions out of active users so we have two big families of ad units you know them probably system initiated ads were that the app is presenting an ad depends on the definition that we have in the app itself and we have user initial ads was the user decides when and how often to watch ads in order to get something back usually and there are two big differences between these two families because as you can see here we're looking at ALP though average revenue per daily active users that this adding is generating and we are looking at the impact that it has on retention because the mixture of retention and Aldo this is what builds

the actual LTV on day 30 day 45 day 60 whatever you guys measure and what we usually see is that for reward video because it's an opt-in because it's upon user request we say that of course it's generating revenue and up low because user engage with agent and this is generating revenue but also we see that in many many cases it actually benefits with a retention as well because usually it adds more content to the game and it helps the player to progress in the game to enjoy more parts of the game and therefore we can say that in general for ad based games you want to have as many rewards video as possible because it's only benefits with your LT fees increases in your retention is increasing your opt out and if you if you will have better engagement through the world with the word video your LTP will be higher the challenge here of course is because it's an opt-in how do you actually make the right interest for the players to decide to engage so much with the word video the second family on

the left side is interstitial or 50 minutes well of course interstitial as well have a significant contribution to the alp dog that we're generating but we also know that when we are showing a lot and a lot of interstitial retention goes down and here the question is obvious what how many interstitial should I show how often what is actually maximizing my LTV and and and in order to analyze everything that's related to implementation of ad units we're actually looking at few different metrics we are analyzing the impression first down right right when we are looking on both engagement rate meaning what share of your users choose to watch at least one impressions per day usage rate which is the average amount of impressions per day and the multi this too is generating the your impressions per Dow right right in interstitial it will usually be according to your decision whether you

show an interstitial every 30 40 60 seconds and according to your play time in a world video it's much different story because here again the impressions per Dow rate of the engagement depends on how well you will implement these idiotic in the game and what we always do is we are analyzing it forwards which is specifically per placement mean that if we have several placements and we should have several placement of your video in the game we want to look at them per placement to look at the engagement rate to look at the usage rate to understand what works and what even maybe we should be increase the record in some places in order to drive more engagement maybe we need to change the visibility of the place but what we are looking is analyzing these two these elements for each placement and placement within our app in order to maximize the engagement through the world video which as I said is crucial to the success of the game because it benefits both with healthy without down and retention another thing that we always do is we compare the

engagement with it we see to what we know that the industry benchmarks in and it's different between genre and janna and another important thing that we always do if you also look at the engagement that the user ages right what we know is that many many times you have a very good implementation of ad units for the first 10 levels but then in day 2 we see a big drop in the engagement of users with old video and therefore will always look to maintain the engagement throughout the user age and 12 different levels in the game and to make sure that always the reward is valuable for the user and is interesting for him to engage with and and in the bottom line and this is the bottom line for the monetization what we are always looking to achieve is not better retention it's not better albe down and is not better engagement with adding this what you want to build them is a better LTV and we always every change that we do it whether we are

doing it in a be testing simultanously and testing two different strategies or if we're looking at a twelve times we always look at LT little court to understand which setup is maximizing the healthy B and if you remember when we spoke about interstitial we say that you need to find the sweet spot between generating as much out though as possible and not hurting your attention too much and finally a sweet spot is only by a be testing analyzing the LDV and decide what which which implementation is the best by the way it's not only the interstitial frequency it's also when do you show expires interstitial immediately after three seconds after 10 seconds as you go more aggressive I hope that would be higher retention we below would setup maximize your ante be there is no one answer for that because every game has his own behavior and a be testing and looking at healthy beyond different setups is the

only way for us to decide how to implement the head units well how to generate the maximum LTV by engagement and by a maximum impressions per user throughout the lifetime and not specifically then after you you are looking at an app and you are analyzing everything and you are deciding how to implement the edges within your game now you have let's say eight impressions per day per user both little video and official but how am I making sure these eight impressions per day are generating maximum revenue for me and this is what waterful if getting into the picture and it's significant because if you will optimize the waterfall you is Sapir an average eCPM will be higher and then revenue per impression is higher and your LTV is higher and vice-versa if you don't and and and the thing that I can say about waterfall is it in general here the balance is between on one hand to make it as granular as possible on the other hand you don't want to

generate too much latency which will harm your feelings generate latency means when the mediation is to go instance by instant by instant by instant you're wasting time and you are missing impressions if it takes too long so you want to make it granular so you optimize through it maximum but you don't want to make it too long so first I recommend to allow in a bidding which solves this cumulative verses latency balance then I would say that you should look at at waterfall per country or group of countries separately they were the right waterfall to maximize eCPM in the u.s. is not the same waterfall in the UK and it's not the same waterfall in Japan or China different ECP Emperor's different networks perform better and worst in different countries my personal recommendation is to stay around 40 instances this is what we find the right balance to avoid too much of latency and to optimize the instances on

a weekly basis according to filler eight this is how we do it in order to make sure that you are maximizing the eCPM and generating as much revenue as possible per the impressions that we have managed to generate and everything as I said is done by AP testing we are reading a few different tools on a be testing 1 2 list for the game itself and the second tool is for the monetization itself but for everything we're testing well usually distributing 20% of our users to add new setup and we're looking what happens to revenue though of course but out the retention LTV how does it impact it per ad unit in order to understand what is the setup that maximizes our LTV this was called the monetization side but I think that data is not less important or even more complicated on the user acquisition side because I'm the monetization side everything happens within your own aim on the user acquisition side you need to digest data from several external sources not your neck and you need to

decide how to take the right business decision in order to optimize your user acquisition so here as well and we won't have time to dip dial to dive deep into each and group but I can tell you that I also divided into two different words first thing is you want to optimize you IPM your CPI your conversion you ads you creatives themselves which will help you to get more scale and the second one is that you want to optimize your CPI how much are you paying per install you know make sure that you are doing it right for the left side for the creatives run over it very quickly whoever's recommend to run with several creative types we play Abele's videos interactive and card we know that many times in rewards video clip which works better interstitial not so if you are going live only with the video you're missing part of the inventory if program like Hollywood playable you're missing part as well so important to have all of them diversity always challenge your

current creatives what we are doing with already's were upload at least for creative sets we see how they behave we look at the win one with rock the losing one and then we are called nutri creative to challenge this as well and every time if we are doing it after a few tries we find something it's improve our marketability better and third I will say that localization make sure to localize your creatives if you want to actually good getting a good conversion I also W asked in the UK and Australia and and everything is English speaking countries because we see that in in Europe or even in or in a book we see that localized queries generate much better performance the first tip will be make sure to refresh your creatives because after users whatchu creatives so many times that offense will stop delivered is created because the conversion is too low until you will offer a nucleating that will increase the probability to make an install as

well and then you are able to constantly refresh you can even reach those users that still didn't choose to install to play your game and in the fifth point which is the most important in my opinion is that you have to treat your creatives the same way that you treat your game and if you look we're looking at the past we see that in the past clay it was very simple you had impressions tap and install and this is it but it is not what happens today with advanced creative like playable like interactive and card well there is a full world between the impressions and the top and in this world there are so many input right when you will showing you play your playable to a user display the different game what is what get you a better City I'll stay player wins or loses do you want him to win the flammable or loser player built what is generating better completion rate what is generating a first faster time to engage these those are networks that we

are analyzing we have a platform that is built but we are using for this and and and this way we can understand exactly the funnel of different creative here is an example of the same playable some of it is but for different versions of the same playable long-short difficult easy you can see the different methods that we see play play but playable in the Delta and then at the end of the day we understand the funnel and we understand for instance the diffuser have a higher CTR if he loses the playable we probably should make the play bell a bit harder so more users will lose and more users will click and this is how we improve the creatives only based on data just like will improve our games we look at the engagement of different users we don't create him and we adjust our creative into maximum engagement in order to drive more a user acquisition and you need to understand it today creatives is part of your own boarding

finalists the first engagement of the player with your game and how well the creative will be will impact his behavior in the game as well because this is the first engagement that you have with the players this was so the creative side as for the CPL shadow how much are you paying per user first of all I'm saying that always optimized world out don't increase your bid in sources where is the retention is in higher or the playtime is higher or I don't know what what because there are so many methods that impacted at the end of the day it's business you want to maximize your revenue and income you want to pay more well then out with higher and pay less well the output is lower so choose the right KPI to optimize a third it second is go gondola if you are just averaging everything and say hey my quality from iron so these eggs so I'm going to pay X - 25 % you are missing big part because when you look at I will source and you look at

lotty of iron sauce then you will see that you have you get users from several different apps some of them has a great date we output some of them have a low date we are true you shouldn't pay the same CPI for all of them so you need to bid the same same for Facebook maybe you should bid per Instagram and Facebook the same same for all other SDK networks for me from Google and YouTube and for any other source that have granularity of traffic within it and what I'm saying here is that look at ult be as granular as you can pesos per device per country of course and beat the right CPI according to the Galant 11 that you get and update it constantly because the conditions in the markets are changing all the time meaning that if two weeks ago this specific source has a great quality for you maybe today it's not the same it's the same situation you need to adjust you simply it was a very quick one run about how we do it to summarize it maybe I'll just say that

in general we truly believe the data is the only thing that the only wanted to dictate our business improves both when picking the day the game decides how to implement ads and building the right LTP and to maximize it and to manage our user acquisition on scale as efficient and accurate as possible we are building all the technology tools that we are building internally in order to digest all this data and to get even recommendation for business divisions for my own in our technology in order to act right and we believe that it is the only way to actually succeed on scale with several games and to squeeze the lemon as much as possible per game thank you very much for listening well thank you no doubt I'm sure if everyone could they be giving you a round of applause that was an excellent talk so thank you very much also a really popular as well numbers really increasing we've got quite a few questions so if you have some time for some questions I'd love to ask you some of them oh sure

everything great so we've got rockin asked what's the average arp da you you see on your games yeah so it's a good question again you cannot look at average ALP though though because it depends in country I would say that in there are us us which is probably the leading segment that we are all looking at so hyper casual you want to have out there Eastern stands up to 17 cents this would be a good metal again don't forget it up though is also an averaging KPI if you have more UAE and you have more users in there say zero day one then out though is higher if you are slowing down on your way and you have more users in day 20 you have the with no else so don't look on the average out though look at the revenue per cohort and four and compare between the same day if you actually want to make the right decisions yeah okay great another question is for interstitials someone's

asking how should they set it up either with a variable for every level to show the ad or not and also should be sure like what's the ad frequency sure how many minutes do you find is most appropriate yes so this is a great game that we are facing there with every game there and we had this question for the three game that will announce this year and all of them are helping millions of thousands so making the right decision can impact a lot our revenue and unfortunately there is no one truth for one game specifically we started showing in the middle of the level every 40 seconds this was the maximum LTV that you could generate for the other game it was only end of level every 30 seconds this maximizes our LTV so I wish there was a simple answer that to this and this but think about when just like I said think about how often and then a bit of different setups to find each one of the maximizing of LT before this specific game which is dev very

different between different games okay yeah another attendee as asked so someone has said I calculate that I need to show around 17 interstitials per day per da you to get a positive ROI do you think that's possible I'm not 17 interested Jabra day I think it's not really you know session and I think it's not very good yeah and one of there is how do you suggest Indies do the initial setup and optimize the waterfall sector so this is it is probably a big question I think if you are playing to self-publish you have to make sure that you have the right resources like if you're Indian you want to publish by your own and management division by your own then you need to have someone that is doing it full-time job that is within the water for this optimizing Trudy so so this is one option the second option is to work with a publisher that will support you this but but in general you can do it by your

own if you have the time and the people and just a structure of waterfalls give the good networks good place using the waterfall start getting traffic and as you go you will understand who performs better who performs not as good and you can optimize and change the waterfall are going to the actual performance that you see Kenny and we've got two other quick questions I think we've got time for if that's okay I fear what is your experience with their noob / Pro ads with a weather for you for this creative so so we know that it works at least it worked a few months ago it works better than well today it's a little bit less common but it still can walk it's only the headline that you're changing what we recommend is after you build your video test few different headlines and check which one is generating the best conversion again it won't be the same answer for every creative and I think this is what nice and makes our industry so interesting because if all games were the same and all the solutions were the

same everything was easy but what nice is that every game is really its own world and for every game there are other and different metrics or decision to take and it's what makes this industry so complex but so interesting yeah I mean someone else's a sport is a good number of interstitials to show per day per da you I mean I'm wondering a bit like overs it's gonna be varying anywhere between four hybrid casual game anywhere between four to ten well four five six seven eight or ten depends on you actually testing depends on your session land and play time and depends on many many metrics that are for your own game so photo tend to have different setups and see what maximizes your LTV yeah and someone else is per what is your revenue split with banners interstitials and rewarded videos yeah so I would say that in general what what in hyper casual water would say that the interstitials

should be around 60 percent I'll be 30 percent and minus 10% what we really try to do is to increase the show of re where when believers in it and we believe with the right implementation and we can be more significant than 30 percent and we see the good in for the details so we're investing in this a lot hopefully to get to a place where we can get and 50 40 50 interstitial for the Arabic 10 banners this will be asserted that we are happy about but but usually 60 30 them and I think we've got I know this to mourn here but the questions keep coming here very very populist speaker but I think will round off with a nice one which is what's coming next for supersonic great so so we're currently testing several different games some of them are looking in a very good direction if you remember this is where I started my talk with we have a very interesting CP Ivan and very interesting play time and retention that indicates a good LTP and we're playing

to lunch probably every two well two weeks a successful game it is the goal that we put ourselves by end of this year and which really believe dependency we should be the biggest publisher for hyper casual games in the industry and this is where we are aiming to be with a nice pace of releases but let's see release ready and entertain people at home it's our main mission country definitely and I think it's obviously more important than ever at the moment so thank you with the questions do keep coming but I'm sure a view will be in the meeting system and and obviously you'll be open in other areas such as social media and everything like where five employees of supersonic barbg contact which you can approach any of them and we'll be happy to take it off the world and see how can we walk with you and disagree the best games in the world owner fantastic so yeah so I'm really sorry to everyone that we can't get through all the questions this has been such a popular talk and such a popular speaker and but yeah I wanted to say a massive thank you to you Nadab not

just for a really fascinating talk but obviously to supersonic in general for you know sponsoring this track and helping bring all of this these these wonderful insights to the industry happy to do it and happy to keep pushing English before they're well here for the ofin thank you very much applause