Marketing Science w/Tenjin

hi everyone and welcome to the first tangent data scientists webinar uh we are happy today to have three very interesting speakers we have julia from popcor you from greenpanda um sophie from juicy publishing so for this first edition we wanted to create this webinar to give the data scientists a place to come and learn from each other and this session is definitely about data scientist of epa casual uh so first yeah we'll start with a small round of introduction uh julia can you tell us more about you yes i'm i'm a data scientist at popcorn or i was just finished my last day on friday um i've been i was at pogba for about one and a half years and i was the first data scientist there so i really got to see the entire data science journey of popcor and before that i've worked for five years as a data

scientist in different companies in different settings what kind of industry it's really been a mix consultancy and retail and banking so a wide variety no gaming before uh and can i ask you what will you do after popcorn yes so i'm taking a few months off just to focus on my family at the moment and then uh we'll see what happens after that it's it's freedom and adventure yeah a year like 20 20 i think it's good to take some time for yourself with your family yes exactly thanks uh you what about you can you tell us more yes sir hi everyone so my name is new mon i see i'm from green panda games which is based in paris now i see from 2000 i think well now we are part of like ubisoft mobile like you know big family which is like there's also like half and calibrary and like ovens and several like studios yeah and for yeah exactly yeah we're on the

big family yeah and so like for grim panda actually like we are a company which is like today like focus on the high per casual game and then like like little by little like emerge and also to the casual game yeah and about myself i flee i have joined grandpanda like two years before as the first data person in the team so which means like i have like you know viewed this like the data team from scratch and like which means like i'm not like limited to data funds but also still like you know data engineering and also like you know data analysis and all of this yeah and for my background i have a background with both business and like technique yeah and i have worked in their advertising marketing and also like online dating yeah and like i just actually started the gaming industry of two years before okay fairly similar to julia how about you sophie you're joining us from lyon right

yeah exactly uh i am currently the data scientist at juicy publishing which is based in lyon we are doing uh we are publishing games for hyper casual only for now uh i've been in the company for like two years in march it will be three years uh and i am the first data scientist of the team uh properly speaking because there was one employee before that was doing everything so he he put the basis of the data science at juicy and i continued the project i've been before that i've been tool developers for the industry for piping industry especially especially specifically but i love data so i specialized in data analyzing science okay cool so it's very interesting that you all

free have in common that you were the first data scientist in the company you were at or you are currently at um so maybe let's start with you sophie how was the data stack when you started and did you feel like you inherited things that needed to be completely changed or how was the process um actually it was the basis were were there so the business was already targeting what what we wanted to to to spot and to watch uh as metrics so we had uh basis uh with your help actually with the engine um and we we had saw the data boards from tenjin and we had the visualization on modern n6 uh which is a platform that um is easy to to to to query sql and to to to code a little of python so yeah

that was the basis and we worked uh after that to improve all the metrics and all the process that are going with the matrix uh yeah that's okay um yes maybe just for the listener datavault is a product from the engine that put the attribution and cost and other revenue data in a database so that effectively data scientists like us can quickly just query it um what about you what about you how was it like when you started and what changed like a lot of changes yeah yeah sure actually when i joined to be honest we are really on the ground zero yeah like at that time we don't have like like a real for example data warehouse it's a truck so we really did like a benchmark to look at like the different products and solutions in the market and then we just finally decided to choose like google cloud which includes like bigquery because it's that time which we believe is the best adapt our resources and our like

needs yeah so like currently we are still like based on google and we are quite happy like with like the google solution especially you can see like google has like you know just released a lot of like new features in the last like two years and we still like really like the products and we enriched a lot and we were and especially like first it's like we are also using firebase so links between five days and google is really like as apply we just really happy about like our choice and the solution yeah was it more a question of cost of feature who convinced you to go with google cloud rather than yeah at that time it's really about like the resource because you know when i first started i was like the first person so my first job is to build out data warehouse to centralize all the data like no different sources for example at that time from tension from like you know librarian which is exam today so and first it's like we don't at that time we only have one person we didn't have like too much support from data engineering and you know if we start with for example like aws or like

snowflake yes we need to have like definitely need to have data engineer and also like very professional like this engineer to start yeah so but google actually the integration with google is much easier even though you don't have like very strong data engineering background you can just start like noticing quite quickly and smoothly so that's why at that time we decided to choose like google yeah yeah okay cool very interesting uh what about you julia how was the stack at the beginning um yeah it was quite there wasn't a lot uh built up there wasn't a lot there but i came just before i joined popcor i'd had my own startup and we didn't even have a date engineering and engineer so that was completely i am not i'm not expert in the back-end stuff at all so it had been completely wild west and like you know these solutions are really just made out of duct tape so compared to that just like having actually back-end engineers on the team who were really super skilled um that just felt like you know a luxury

after that experience and also comparing to some of the other companies i've been at who are you know like more legacy they would have they would have data structures they would have databases but they would be really old and they wouldn't you know work properly so compared to that as well being able to build something up from scratch that actually serves your purpose was a great like a huge difference and it was really positive um and i think when we joined fairly when i joined fairly soon afterwards we started using snowflake which is a tool that i really like using i think it's super it's really like for me as an end user with data like it's super easy to use uh and i was i was really impressed and happy with that and i know they use amazon aws as well i don't have the technical i don't know the details of the technical background but i don't have any complaints it's been really easy to work with the data well you mean uh it's the backend engineer who took the decision but to go with snowflake

yes it was completely there yeah completely their decision you were happy with the capacity yes exactly exactly yeah just compared to like working at these uh like more legacy companies where you don't you don't even know who set up the system or why it works as it does um it was so much easier like even when a problem comes up to be able to work that closely with the backend team was a huge improvement yeah well yeah i can imagine two heavy is gonna help while when you're effectively the first one and you need to learn on the spots and build things you don't know you could yeah it's very cool good um so what's your opinion for for the listener like let's say we are a small per casual team and we're thinking to scale start launching more games and start investing in our data infrastructure and also what is interesting i think is that different uh of you have different attribution provider in terms of how you you receive attribution data

or even cost data and ad revenue data what would be your advice for someone who needs to build this from scratch someone wanna yeah um yeah for me the the first question to ask is what to what do you want to drive your business and what are the data that you want to to see so if you are a airwise driver driven or antiviration driven so and then build your tools in in in that order to see this these metrics uh that was our first step at juicy and then we added more uh tools to improve the process of ua particularly uh for for monitoring and um and processing uh changing the beads and etc so we were we wanted to

sorry we we built an internal tool to do that but you have some in the in the industry like um big check or some something like that um you have to the choice of the mmp are very difficult so we we went for for fort engine but uh we are constantly uh looking uh if uh there is uh new updates on other and etc so we are actually with engine for now um and uh we are glad for without us but i think it depends on the business and how you you handle it uh yeah how easy was the project of building this internal building tool i know that's a project that a lot of your casual want to pick up and most of them are a bit fearful of how difficult it's going to be um actually we have uh we have the

we we are lucky to have one a very good uh tool developer actually in our team so uh it was quite easy for him to to develop something and to call the different api and change the bids etc but we first asked if we preferred to develop the tools or just find one in the industry like i said a big check do do the same but yeah it's a cost after that so it's a it's a choice okay so the main reason to believe in us was the costs and the fact that you had the ability in-house yeah and yeah and the the fact that uh i prefer developing tools in intel because you have what you want exactly so it fits to your needs and it's very very important i think yeah that's very true yeah uh you

what do you advise to like companies get started with building a stack without the first thing you should tackle yeah a sleeve for me like the first thing is always back to your data infrastructure you know like actually all the things for example like the we can say automation didn't or like data funds is always made based on like if you're like solid robots like no data infrastructure yeah so my like my first like no suggestion is always to choose it and choose it to like properly first based on your resources you know if you are like you have like more for example the engineers like background or like in your teams more like analytics like background and these like different resources good like to make you like start in the different approaches for example if you have more like engineers of course you can go in more into for example aws or like you know snowflake or azure but if you're in your team they are more like on the business on and off expert maybe google may be like you're like better choice yeah and the second part they also look at

like what you already have today for example if all your you have you're using gmail you're using like firebase and this play like choosing a google which is google which will give you like a very similar slide you know just correlation between all your services but if you're using outlook in this way maybe like as worried it will be easier for you yeah and then it's yeah and then except that like for me one choices also to really look at like what's your needs and what are your use cases yeah because in my point of view like data science or data it's not something it's not a utopia it's not just about like you know some like you know super cool like guys which are just a writing code and algorithms you should really think about okay what the data science could bring to me and what my my use case is how it could help me to solve my issues of my like you know actual like practical issues yeah because i think usually like high perpetual gains were not in we're not like a huge huge scale usually it's like less than 100 people so you have to use like all your resources like smartly and however you

should deploy for example your data science like those data finders and your data science resources yeah good thanks very much julia what about you what would be your advice um yeah actually i think you've uh what both of you have said is makes a lot of sense but one thing that i would also really like to add is um accuracy to make sure that the data you have um if you're getting data from different sources do they actually tell you the same thing like if there's a way to double check the numbers that you get i would really advise that because especially i think a lot of people that are outside of data science they still view data science as you know a bit of a black box they're not sure they can trust it and if you start getting results that are really goes against people's common sense or their expectations people are going to be very suspicious of those numbers and i think like that would be one of the first things i would advise a data science team is yeah just or like any data team just

like make sure that the data that you have uh that it is truthful like it's so easy like especially when you're in a startup it's so easy to accidentally like you're setting up an etl tunnel for the first time it's so easy that something goes wrong and you get numbers that don't really make sense and the sooner you can catch that the better and it sounds like such maybe like a very basic advice but like it's it's better to worry about that in the beginning rather than later when you've already like set up your experiments and your algorithms and so on so even if that means kind of doubling the work finding two source of the same data well maybe not um like maybe not for everything because like of course you don't want to bring in maybe two third-party providers like that's you know like that's a lot of work but i feel like there's a lot of times where you could get the same numbers like you could check number of installs both from google play store and from your tracking provider for instance uh and just like double checking that it

sort of makes sense and then would you advise to do this at least at the beginning to make sure all the data is good in place and then you could give up this monitoring or would you keep keeping this monitoring in place yes i would i would advise to set up some monitoring uh on that just like yeah just check those numbers every once in a while to make sure uh they're in line yeah yeah it is actually a very common issue that a new app is getting launched and the monetization is not what the customer expected and it's just because one api was failing or one monetization channel was not reporting the data correctly so yeah it makes it helpful to have even if it's a different matrix that you you have as a benchmark like ad impression or something else in the revenue that show you that oh no there shouldn't be such a drop exactly and especially for hyper casual as well if you're publishing many games you want to and you want to compare it to a baseline with your other games

um if that bass line is suddenly not not not right anymore you have to figure out where that comes from that helps um so in your career in epocho what was one of the most interesting projects you had or one you learned the most from anything you would want to share with us yeah there's been so many uh and it's um but obviously like the ltv project has been like that's huge and that's very central and i'm sure we'll talk about that more later as well uh we had one really interesting project that was i learned a lot from which is using machine learning to decide which level the user should see next so you know simpler level for users who might not be very good at the game or harder levels for users who want like more of a challenge and that was for me that was really interesting it's the first time i used like unsupervised learning and so on and it was a really fascinating project uh how fast that's been really cool how fast could you recommend which level

to to show to you uh we we tried a bunch of different like many different setups like how how quickly should we introduce like the algorithm is choosing next level i think the earliest we used was five but we also experimented up to like we don't use the recommendation engine until like after level 30. so it was yeah yeah it was we tried a lot of different variations is there anything you can tell us about the tool you use for this machine learning project or even what programmation language or in oh yes yeah this was all uh done in python and set up on aws yes yes exactly uh you do you have a project uh at greenpanda that you found particularly interesting or that was very critical in the growth of the company yeah sure of course like um i thought

like you know examines a lot during my past two years like the first year is really about like you know build like the data infrastructure the data warehouse centralize all the data and also break like you know the bricks between like each different data like courses and then create the dashboard and record and etc yeah and then like on the second half especially since last year we are really step into life so we try to like step into the casual game like industry which is like more complicated and which like really needs like more invest and like more customized data analysis and like a data time so we did like with our last game we really did a lot of like customized like you know just data analysis from at the very beginning we are really like integrate into the like production process of like the game yeah so which like starting from design all the event tracking yeah and all like the parameters and how then and then like decided how we use like this event tracking to set our analysis and then like you know we

start then we just conduct this analysis and based on this analysis like you know we just like send our feedback to our like producers and game designers to help them like to optimize the game and to like understand the user behavior and also to analyze like the new features etc yeah so for me that's what's like really as we can say really like an exciting live project and also set up a very good foundation for our future like casual game like no development yeah and apparently now we're just trying to sum up all the best practices and set up like a standard like process and like and guidelines for the future games yeah so that's like the data and on the data part will be like more smooth much more smoothly like for the new game yeah so this is like one part and and the other part which i feel like very interesting is that the changes on ios parts yeah with the ios 14 with like definition of idfa and ski ad network especially like how we could since we don't have user level data

how could we like you know continue like for example the ltd production user segmentation and also if they try we fly you know the sks network and that's a very interesting part like work to do and we are still like you know try to like you know investigate anonymous so are you that's a very interesting topic of course ios 14 that's i think if i would have asked this question at the end of 2021 everybody's most interesting project would have been to adapt to ios 14. uh so today what are you preparing for like mainly just receiving a new type of data with a sk a network postback or you're even changing already like the granularity at which you're looking at the metrics like lte and roi yeah so for expression for that's like of course we did like av tests to see like you know what will be like user consent in different countries yeah and regardless and different types of games and between what for example old users and users for this part it was already done and also like we have

concrete data based on our games you know because really it really depends like based on like companies and your games yeah so on this part at least we have some concrete like numbers yeah and then um yeah well i know that in the industry we talk about the consent will be about 80 percent of the user would opt out can you tell us is it more or less what you got as well or did you get better results yeah i think like it's around that yeah it's i think it's around that and i'd say we really need to be like you know like maybe bit more conservative than optimistic on this one especially in the u.s market yeah definitely yeah yeah and then for the skm network for this part yeah we are just like we have started like our tests like you know with afghan networks especially like you know to test on their like conversion values because like you know you can only choose one conversion value which is like from zero to 33 so you really need to like choose the smartly and also like about the which and

different campaigns which like conversion value you want to send and we are like testing on this yeah yeah if your newer game are casual i'm guessing deciding what event for the conversion value is a bit more complex because it wouldn't be simply ltv prediction or ad impression would be more custom events yeah exactly since especially there's the timer so which means if we don't like standard license and it will freeze 24 hours then it will expire and you cannot send it anymore yeah in fact f challenge yeah definitely that's also exciting yeah that will be the future yes that's coming up in the next few weeks or more uh what about you sophie in juicy publishing i'm sure you've seen a lot of different projects and a lot of different change yeah yeah uh true and then the project i want to talk today uh is the data vault that we are creating now so

it's a project uh in in in in course uh but uh yeah the we we want to have the data on our we want to to have the data on our side so we are building a data vault that will uh let us have the aggregated level that we want uh based on your data and uh yeah with the as you said um ios 14 uh it's a it's a bit uh a challenge today so we want to have um the most flexible that's possible to to to which could talk with all our different tools so the engine uh the api firebase etc so yeah this is our biggest challenge today and this is a very interesting project right did you choose which data storage solution yeah we choose uh as you actually

we choose google uh we choose bigquery because we already have a database on bigquery for for the data from firebase so we want to centralize all our data on the same technology so yeah be query for for that um then i would say my last question this time is uh there's two main projects i would say uh the upper casual industry often talk about and we mentioned both of them already like ltv prediction uh internal bidding tool um what would be your recommendation is there an order between these two that people should do or is there any pitfall people should watch after if they want to start for me there is no uh i think the priority is to decide uh it's different from one company to another uh for example for us we work

for with ltv prediction so we had to work on that first and then build a product on it but some companies i know are working with uh rose eris d0 or d3 or d7 so you don't need prediction and in this case you can focus on the tools how far do you predict day 16. pretty cool yeah you what do you think ltv prediction or bid management well i think it's like same as coffee's uh like coffee said like from firstly you decided to start with ltv and i'll give you like first they have you must still like make sure you already have like correct historical like ltd and that's where you start to build like your ltv like prediction yeah and also like ltv production is also based on games yeah because the um first is super complicated we even though we are like in our portfolio

the most of them are like you know high perfection we also have classic games which is like political we also have like you know some like new casual games yeah you see like these games actually they are very different like from each other so which means you cannot have like a model which applies to all of them yeah and you have to really like you know design like the model based on like which game that yeah so i'd say like start from the ltv and then for the video actually it's the same i think like today a lot of the beating tools actually they are not that like you can see like advanced or predictive yet they're still like using historical data yeah so like i think first is try to make sure you have like your correct type of data then the next episode is like prediction and then about like the video and i think there's also like tours and even like alphavin or even like iron stuff it's also trying to like offer this course and so you have to like choose wisely between like a third-party tour or you want to have like your internal like no kind of solution because if you want to have your internal solution if it's very important just like like to like to make sure like the volume and

also like the speed and the like the validity of the like your calculation computation and how you like you know send or share like you know these data for example into your internal life process or sent to a third-party platform yeah so i'd say like on this part there's still like locked ways to do oh and also you have to have your resources if on your background like backhand part it's not they like you know capable to do it or not if they are not maybe to choose the third party to work if you are really very confident with like your internal like you know developing skills then of course like no internal like throwing the better but you have like the control of like each part yeah exactly like sophie mentioned earlier uh julia what do you think doing ltv prediction and bid management um well i would say just for like specifically for hyper casual um so much of the the revenue and the profit is determined on how much

you spend on your bids or like the the accuracy of your bids so to speak so i would say that that i would probably i would personally prioritize that slightly higher however i'm not necessarily sure that is something that a data scientist or like that would be a data scientist full-time project like for instance at popcor that was in the domain of the ua team and they were super skilled uh guys who really you know like they were really good at that and that's not necessarily like you know it's rule based rather than like machine learning prediction um so with that said the biggest project for like the data science team was still ltd prediction rather than bid management but yeah it depends on what the structure of the company is like esther ua team uh who have done this before and have the skills necessary to make accurate bids or is it like you're starting completely from scratch and you only have data scientists well then they should probably focus

their energy on uh bits this is my advice but yeah it's really bad it's easier for the data scientists to jump on this project if you already have some i have a specific rule they want just to automate and then it's just about automating these things or even the machine yeah exactly well thanks a lot for sharing your knowledge today i'm sure the listener learn a lot about what it is to be a data scientist in either casual if they want to move to the gaming and ever casual industry or even just start it there thanks a lot and we hope to catch up with you later thank you and thanks a lot thank you my pleasure