Driving ROI with Data Analysis – Part 1

[Music] welcome to the navit gaming podcast where it is our mission to explore the business and future of video games we bring together the industry's brightest Builders investors and thinkers to keep a pulse on current events dissect emerging Trends and games share Lessons Learned and have a great time this podcast is also part of novic growing ecosystem which ranges from free and premium research to Consulting and advice services for more information visit [Music] www.nic.edu service solution providing an Enterprise grade data platform and services for gaming Studios on all platforms Dives bi tools and service of data experts replace the need for a full in-house bi team saving Studios hundreds of thousands of dollars yearly if you're

interested in learning how dive can Unleash the Power of data in your game's business and save money doing so simply head to dive. games or check out the details in the show notes and with that let's jump into the episode all right so welcome everybody to another episode of our data Corner Series today we have uh two uh levies on the on the podcast so elad Levy who's the founder and CEO of dive he's back on and uh we also have the second Levy whose voice you might recognize from our round tables uh Tammy Levy um and since um yes this is this is also kind of like the first time uh you know uh all three of us are kind of together on uh on on a virtual meeting I guess we can jump into intros very quickly um so yeah maybe we can just start with elad so elad you know just as a quick refresher for our listeners maybe you could just quickly

tell us about yourself uh what dive does and you know how it's kind of helping game developers make more successful games uh we are I'm a 20 years entrepreneur in the games industry with multiple exits and uh last company that I've started 5 years ago dive is providing many people call us bi as a service which is basically we build custom Data Solutions for uh different companies and different Industries and over time we added a whole set of liveop stools to help game studios maximize their ltvs which include Integrations with CRM segmentation AB testing and a lot of other goodies that is it in a in a nutshell awesome and uh and dami yeah it's it's kind of been a while since you know you told listeners about your background and what you're doing today

so maybe uh maybe today is uh today is the day to refresh everybody about uh about you and what you're doing in the industry yeah so uh I am part of the of the Round Table career which is always super fun um to to record uh I am currently the chief games officer at a startup called Captain TV where we're making um games for community so basically games that uh communities can really come together and play together and not only that but play it with uh streamers in a way that is kind of like a very authentic interaction between streamers and their communities and really help streamers grow their Community Foster their Community uh before that uh I spent many years at congr and um touched a lot of different things at congr but always with a with a good serving of data uh even you know at some point I spent a few years of um

focused just on our data Pipeline and kind of rebuilding our infrastructure and making sure that that was uh robust and uh very much alongside with what lad was saying right like just making sure that we're capturing the right data that we have um that it's also some in in a way that it can be repeatable so as we launch other games you know we can take what we've already built from one game to another so think about it as much more as the syst them then a one of um and really expand our understanding on how what to measure how to measure optimize our ltbs and Empower PMS um UA managers Etc so when myu you said let's let's talk data I was like sign me up I'm I'm ready yeah you said I I didn't even read the outline and I'm I'm ready for this episode so that was already a good sign and uh and yeah I remember I think on a on a very old round table

when Anthony also joined us Anthony Pella who you worked with that congregate uh I his one line description of Tammy was uh she likes to measure everything I still remember that line so so yeah you're you're the perfect uh guest uh for today you know with alongside elot um great um so I guess yeah maybe maybe with that you know we can kind of um jump into today's topic of discussion which is essentially D driving Roi with the data analysis and you know it's a it's a it's a very broad topic that um super broad I think yeah many understand yeah super broad topic I think many in the industry also understand quite instantly you know that it has a lot of value um but not so many people maybe understand like how to engineer it uh you know in their gaming Studios so we'll be trying to like you know dig into that a little bit more and

you know a mix of strategic and you know tactical uh discussions today um at least for me uh the kind of the way I think about Roi from data analysis it you know it just goes beyond simply analyzing the data but really you know ends up covering the entire activity funnel so for example you know it wouldn't be Roi effective uh when product teams don't know how to talk to the data team or data teams don't know how to Showcase results to the marketing team or another example would be you know how data an analysis cannot occur if budgets aren't allocated properly or the proper data tooling doesn't exist uh across various teams so yeah it's it's a really like bigger Topic in my eyes and in uh in today's episode we'll just try to understand like you know various parts of this entire value chain but uh yeah to kind of uh kick it off um I'll start with a very broad question so feel free to answer it you know how how you guys guys uh see fit uh we can dig into

like separate parts but yeah the first question is you know from or something that I just think about a lot is how should game teams really think about setting up their data pipelines teams and budgets so yeah like I said broad question can take it one by one uh but yeah maybe elad if you'd like to go first yeah I I am a huge fan of keeping things lean as lean as possible uh because um it can be small game studios it can be big game studios some years are just they just suck and uh the only way to survive them is to keeping things lean so I would start with whatever solution is out there like making it simple and easy to just get the first basic Ki because that is all need that that's

basically all is needed when the game is in is starting and then it unfolds over time you know and LTV live Ops and progression systems chair monetizing things further so this is it's it's a process it's an iterative process that just takes time but I I personally like keeping everything lean and like to stay on the safe side because there's a lot of buzzwords and noise noise out there and to be honest every company ends up with their own kind of flavor of data with their own tools and their own people and different people has different opinions it's like it's art in a way everyone Paints in a different uh style and color so I just you you know would approach it lean and

slowly and uh you know as the game evolves and more budget is available then you can start considering adding more I know people tools complexity yeah exactly yes right yeah and I think that one the jump jump of that uh one of the things that you mentioned is like it's an iterative process and that's always how I've thought about it you know when in the early days uh of like developing what would we called our or kind of standardized schema back at congr we thought about it in even that way of having teers of analytics that were tracking and having something that's like very easy to understand in terms of you know early on what do we actually need to measure in the game Let's only set up analytics for that piece so talking about you know kind think about it as a funnel or you have to have like

a real goal of what you want to measure at What stages and that really is going to be a good guiding principle on how much complexity to add like you don't need to have hooks for all of your game economy when you're first launching the game and just seeing our player is going to come back for another session like if that's a question they trying to answer you don't need you know a bajillion different Hooks and tracking and granularity and that's so true I yeah Le less is more yes yeah and I think that part part of it it also comes with like being true um it's kind of like a little bit of a chicken and egg with the things that you asked man you of your team versus your budget uh but you have to be true to yourself about am I going to act on this data like do I have people who can look at this this data understand it and act

on it if not don't collect it don't even bother I mean yes you can collect and have it for a later dat but you're you're um you have to introduce yourself to the the size of the team what your capabilities are how much are you going to be able to act on that data and from there decide what tool should you use to instrument you know your your data pipeline I think that teams that have gone through the process multiple times will probably have a stronger feeling of implementing and instrumenting their own data pipeline for newer teams I would always or even with your startup or early on I would always recommend uh even if you've been around in in experience like find a tool that helps you capture that data very easily very quickly um because you'll have the data to empower you and continue iterating

quickly you can make things complex from there as much as you want right like you can add complexity and layers of complexity at any time no I love what Tammy said because that's I had we had several cases like that that we meet clients and the developers decided to track I know hundreds of events now more events is more complexity because it's more data it's more storage it's more QA because someone should actually validate that this is correct so every event that you at is more QA even though a lot of people are ignoring it so it's very often for us to just delete everything and say okay let's take I know 20 events the basic stuff you know the really the basic things just a few like a handful of events in in our case in the case of dive the SDK and that happens also with many other uh companies Firebase also

does it out of box they they track the sessions and engagement and retention so it's not you don't even need to like send any custom events but I I really that that's that's a great point that Tammy mentioned like keeping it like to the bare minimum at least in the beginning is is best practice yeah yeah and I mean I've also like had you know um yeah many many conversations with other developers in the industry who um sometimes they also kind of uh they they would wear it as a badge of honor when they say when the question is asked okay like you know what are you tracking on the data and the answer is we track everything but and you know it's it's with lot of Pride horrible I don't like please don't say that it's it's it's hurting my ears know the we track everything it's and another thing that is um is I see

that is happening very often is that developers think they understand data and there is a huge difference between someone that works with data and is a data engineer and knows how to properly collect it and clean it and normalize it and the developer that just sends event and say yeah it's data is easy you know we just send all the events and see what happens and then you know then upper management relies on that data and one of the you know they wake up one morning and the dashboards are incorrect or or they cannot rely on it or the kpi are wrong and then they go to you know they make a PowerPoint out of that and go to a VC and try to raise funds on on something that is a complete mess and I've seen that happening over and over also in m&as ER they when they do due diligence

and they start digging into the company's financials nothing makes sense like not even the Apple store purchases do not make sense with the basic reports so it's it's really really recommended to take it step by step and slowly yep yep yep yep I think I think that's great advice you know uh depending on uh the team that you have uh the money that you have and the measurement goals that you have you know uh essentially track what you need uh and then as the game if the game like you know crosses the next gate and as it evolves keep tracking more of what you need and increase complexity slowly so I think I think that makes sense and probably it's a piece of advice that you know applies to both startups and you know the bigger companies uh instead of you know this problem may actually occur more in big companies where it's just you know the same we did this on the old game so we should just probably track everything on the new game but you know know if it's starting with a small team

then you know yeah people don't have the time to look at all that data so anyway yeah sound sound advice um so I guess yeah so you know that that that's like really like the you know the top of the funnel question for me uh you know how teams kind of think how team should think about the data pipelines teams and uh budgets uh but you know once let's say that okay now the team has kind of set up everything in in this you know lean fashion um how the next question is kind of okay you know the processes uh itself and like how teams are actually functioning with all of this so what what maybe there's a question for Tammy first um but you know how what would your advice or thinking be around you know day to day week to week month to month quarter to quarter data processes for you know teams of different sizes that's that's a great question and I think that it um so first of all it definitely depends on the stage of your

game those processes like yes there's going to be like some good routines that you set up in place and processes that probably apply to any type of uh kind of games at at all stages but it's very different if you're testing say test Market Stu launch early on like those routines will look very different than when you're first launching the game trying to get you know potentially grow the game um start to get into kind of like your liveops Cadence and then once you get into liveops Cadence like that's that gets into a much more like routine day-to-day week to week month to month like predictable routine um you you effectively should get get to that but it goes I think it also goes back to um the question of like what what are the goals right like what what are your

goals every every day and what can you act act on right so you're not going to be checking metrics on a daily basis like you could you could get obsessed refreshing dashboards but looking at metrics that uh you can't act you know today if if it's something that it's not going to give me information that I can act on today it should not be part of like my my daily routine right and of course there's like some best-in-class um I wouldn't even say best-in-class but like standardized uh metrics that you look at and dashboards that you would look at these days um in most companies right like you want to every day you just want to be monitoring the health of your game you don't want to come you know wake up one morning and see you know DA has tanked uh and not you know have it go for like two 3 days and not not have a flag or not realize that that's

happening right so you want to want to look at like your daily daily metric and uh make sure that your your game is doing well it's it's stable it's like you know you're able to identify any odd behavior uh from the Baseline that you're expecting um from there when once you go into like week to week month to month it really depends on the Cadence of the game right so on a week basis like I think that that's where like the unique live Ops for for each game start hitting right so if you if you launch weekly events you want to make sure that your weekly events are performing as as you expect um and here it's also where you start tying user acquisition right like user acquisition has to be part of those routines uh because know i' I've seen it many times where you have your PMS that have like a a very good like routine they come in they take their dashboards you know they

see let's do arbow as as like a a blanket metric that's can be controversial because some people love it some people hate it but let's say your AR out tanks and you know they're freaking out right like they triggered an event uh or a or um you know they did they had something scheduled in their livs arbed Out Tank oh my God their hair hairs on fire uh and what happened was that you know y us team was trying some new campaigns and you know they brought in a bunch of kind of low quality unqualified cord of of users uh and like the answer would be very simple if if you're like have the right processes and and communicating between you know the different folks that are like really servicing the game um yeah so I'd say like that looked like at your weekly and and starts to expand to month I've had yeah I I've had that uh issue happen to me many times where

my entire day or even two days sometimes even three days completely blew up because you know just the interact like you know something like this abdow tanked or some Metric uh tanked and you know uh a full you know what's the word like a full wides scale investigation was you know initiated and uh but if the teams just talked a little bit more uh you know could have maybe figured it out in a few hours uh and what is like 3 days so but but yeah that that makes sense you did bring up one interesting point about you know um I think it's a very good distinction that you made about uh how to think about these processes you know during uh soft launch versus you know during the first 3 months of launch versus like you know the more Cadence live Ops uh we can come back to that but um um but yeah El uh what what what's your thought on on on you know this uh setting up data

processes I think different stages is the like key here because you know early stage games I you know da is not even relevant I prefer looking at the wow with weekly active users and Mau and Ma Wu and Ma because they're there is not enough volume for the daily Trends so it's it's I just prefer seeing it on a weekly level and then as the yeah as you start pumping more traffic you can look at the daily Trends obviously I like a simple PDF Excel and table that shows numbers and trends like the last for example uh two weeks on a daily chart you can see that without even a graph just to see and then a small indicator saying if you're going up up if you're going down against the average so that

you can see if something happened because what tamy mentioned happens a lot in bigger companies because the UA team is completely disconnected from the other teams or I know one of the developers decided that he wants to push something to production because they have this cool new process where you can push to production whenever you want and all of a sudden some something is not working or the churn is spiking but those are a ton of metrics that you add in like later stages because if if you have a you know sourcing money inone money out a mimo dashboard that you can see if for example there's a I know you you Source too many coins because some crazy bug filtered in in the back end or the new version then Revenue drops all of a sudden or yes if there is a marketing

campaign sending their own traffic obviously it will affect it's very funny you know like if if you send um like horrible traffic the retention is horrible and if you send quality traffic the retention is amazing so I I really like looking at retention in in a segmented way according to marketing channels or according even to a specific feature because sometimes I know people who finished an achievement the first day or played a specific game mode in the first day will lead to way higher retention than people who have not done so so it's it's all um coming down to the the stage and the metrics that you add over time going back to the iterative process that we mentioned over and over you cannot just throw all the kpis in one time because M no one will look at I

there was I used to write I'm sorry for the parenthesis I used to write uh but you said it's a casual conversation so I'm I'm letting myself be I used to write um I actually started in cyber security many years ago and I used to write a lot of procedures and standards and documentation and one of the thing I've learned about was that the the goal of a document is to be R and I think I think that this is correct for a dashboard as well because if you build a dashboard and no one is looking at it then it's kind of useless useless yeah yeah got it got it all right so can I can I jump real quick um I think that there's like three three elements there that um I think are super important uh from from my experience that I think tou on a bunch of the things that Eli was saying but um one is

understanding your Trends uh two is understanding how you segment your um your players and the metrics that you're looking at and this can be for like all the way to the top for like just executive reporting and you know going super granular into the game specifically and the third one is um data that is good like you start segmenting so weic like um cohort sizes like that's that's one example of like data that is good and like actually will give you information uh yeah you can segment players you can have uh be looking at like say weekly Trends and then you you look under the hood of like how how the dashboard was built and you're going off of like 50 players and you're seeing like these these bumps and drops and bumps and

drops and trying to act on that and it's like you know what you're not you're not getting any significant data here so um back to the back to the less is more uh you also have to understand what that means in in terms of like just good data hygiene and good data practices makes sense makes sense so yeah I I I want to take it back to like one um one thing you mentioned Tammy uh you know when thinking about data processes apart from you know being um yeah having the lessers more mindset or also kind of you know starting lean and then you know adding complexity over time depending on the goals and what you want to measure and the stage of the game um the stage of the game uh specifically was quite an interesting you know delineation that you made and it sounded like you know um there's a chance of more chaos existing

in a soft during the soft launch period versus you know a live op period where you know people have you know learned about the game maybe the team has expanded you know uh processes have kind of evolved learnings have been disseminated and it's just more of um you're more on a treadmill and versus like a soft launch where you're kind more mature the game is more mature yeah more mature yeah exactly the is more stable the of course everything is more the kpis are more so so yeah so like I mean on the soft laun side of it you know if if this chaos or this imminent chaos is kind of foreseeable um how do you for soft launches specifically during that period how how what what would you kind of think about to you know not result in wastage of time for you know when looking at the data given your example which was perfect which is you know a cohort size of 50 people

resulting in you know all these kinds of bumps uh and you know uh essentially teams and you know leadership kind of pulling their hair out because of it so how yeah during soft launch process how would you think about like the data process setup I think that this um there's actually like when we talk about like even self launch here's especially actually especially during self lunch there has to be like a very strong delation between a big team with a big budget and a small team with not a lot of budget uh cuz that's you know it's going to give you completely different capabilities of of what you can measure how many players can you drive into the game and just starting from that very kind of key point of like how many players can you drive into the game how many players can you acquire that will inform you know how how you operate during your soft lunch how much can you

measure um also in terms of you know your own resources uh team budget team pressures Etc it's like how quickly will you go from soft launch to uh Global launch right like do do you have the resources to just keep iterating to measure day seven day 30 like really get into kind of deeper into you know the funnel which a lot of teams are advocating for these days because you know we've as an industry we've been too focused on like optimize day one optimize day one optimize day one is like and then what happens right so I agree with like the sentiment of like hey we need to look beyond that but that's like a luxury that big teams with big budgets have Runway you need runway for that yeah you yeah you need Runway right it's like if you're if you're have six to 12 months Runway um just to to throw a number out there uh

you're not going to be able to get that deep into a funnel before you want to start you know actually like launching game getting you know monetizing the game Etc so I think like that delineation is like super important um but even with you know taking a step back away from that is what are the goals of of it goes back to goals always for me it's like what are the goals of your soft lunch what do you want to measure what do you want to answer what you want to prove and have a plan ahead of time like actually plan have the questions that you want to answered uh understand what data do you need to answer those questions and uh can you actually act on it right it's like if if you're not going to iterate on the tutorial like don't fool yourself in collecting every single step of the tutorial uh funel if you're like if you're like no my tutorial is good we're just not going to change or for whatever reason right but

uh so have a very clear goal in mind of like what are you going to measure can you actually iterate on it and act on the the data that you gather from you know that the part of the game or you know like have a a strategy start having a strategy um having your goals setting targets and back to like the good data practices how many players do you need to be able to get a significant result for whatever you're trying to measure um which is why for example like monetization becomes like pretty hard to measure for a lot of teams during soft launch because if you think about I think yes it's a mistake to look at monetization at the beginning it's like is you're you need like a huge huge huge user pool for it to be like a significant measure uh so

I've always you know my My Philosophy around meditation especially like during early like when you first put into the game and like early self launch it's always been like if there's purchase happening that's a good that's happiness for a soft launch that's happiness right it's like yeah like I'm not going to look like the number won't tell me anything right like but people are purchasing yes they're interested in the yeah I I want to add a few things here I would actually want to like bring it down to actual kpis I love looking at volum retention in the beginning because sometimes people do not necessarily come back the next day so it's easier to it's an easier kpi to explain to VCS for your next round and um uh engagement session length those are things that you do not need D28 forget about LTV and Roi and this is

like not the time monetization definitely not the time uh I would focus on session length and you know rolling retention engagement and maybe no more than D7 you know because it all takes time you know every UA burst that you do takes time to see how it affects like people's stay they do not stay and then there's a huge feedback maybe not for this topic or maybe not for this um podcast but there's a huge feedback that comes from the community yep a huge feedback because sometimes a handful of players can give you a ton of information that data will not because data will probably need more volume so if you have a I know a Discord Channel a Facebook fan page somewhere somehow you you communicate with those players you will get a ton of

insight for a for a very like for a fraction of the price of having a huge team so um I would probably you know focus on those metrics at least in the beginning yep yep make total sense and one one thing that I would C there with at least in in how you know from from conversations with teams in general is like when you think about retention like just know what kind of retention cuz you said like like oh I would focus on like measuring rolling retention um you know there's different points of view like how should you even measure retention rolling hard you know all sorts of different uh formulas around it you just need to understand what it's like the way that you're calculating why is it what is it telling you um so braing retention for example like it it helps you more understand churn right like I'm

not like specifically measuring like the the day one day seven da d da that a lot of teams post and like you're aiming for like that that gold standard of like oh they hit that that retention but like oh I'm just like I want to understand how um where turn is and like that type of measurement will give you that uh but kind of yeah also putting it in like more concrete as as you were saying that um other pieces that I would add in there um also look at you know crash rates like we're talking about like deeper in the game metrics you need to have metrics to know that players can load the game players can start the tutorial right so from load to tutorial nothing is like blowing up yeah non BL da I call it yeah so

Au okay okay got it no no it's it's a very important metric like how many people like are like logging and doing nothing in the game you would be surprised by the huge amount of people that just you know open the game and then do nothing and that means that something like they don't integr like engage with the core Loop so non playing da is is a excellent metric and crash rates obviously please begging to everyone to not use the free crash tools they suck and the ones that are paid they're not that expensive but you can hook up the player ID there and you can get way more information about the the like the the timeline of what happened exactly in the game before it crashed so yep makes sense it's really it's worth it all right cool yeah so I guess yeah so don't um I guess yeah

don't don't focus on the metric itself but focus more on the actual purpose behind the metric instead or understand that first and then and then choose your metric list I guess that would be a key takeaway then um but yeah let's uh let's kind of then move on from you know the processes so you know um there are after these processes are kind of laid out uh the teams essentially need to work with them and and yeah you know data teams usually have or they I would say should have you know a one to many relationship within a company um and you know where they're talking to uh you know the designers and PMs and US specialist leadership etc etc so um it's not uh it's not always been perfect I would say I'm not even sure if it's reached a great state today versus 10 years back but you know if how how do you guys kind of think about you know

how these relationships should be set up how should they be set up and what should they kind of look like uh in you know these one to many relationships between the data team and the rest of the departments in the company uh maybe elad you can take that first I can give you a an example of the experience I had with the house of fun and Pla because I had the luxury of see how it games grows from zero to top grossing hundreds of millions of dollars annual and we used to separate that to two main functions ba and bi ba would be business analysts data analysts data scientists Etc bi would be data engineering maybe reporting like doing all the bi is the ugly side of data data Engineering in general is the ugly side of data so product

managers you like marketing guys upper management everyone would W would speak with the ba team with the data analyst with the business analysts and um usually if they started poking them every day for the same report then the ba would ask the bi guys The Dana engineering guys hey can you please automate this report for me because they Engineers are kind of like developers that understand data but data analysts are more connected to the business side so um I actually think that product manager and data analyst is like a is a very healthy relationship that the he that the product manager drives him insane every day about I know extract this population I want to see a correlation to this feature or that feature I want to see we launch that let's see how it work with it there's like constant communication and questions that are going back and forth

and I think it's a healthy relationship but that's like the main functions was was this delineation it's a very interesting structure um I don't think we like back at Zinga at least it it never used to be like that uh or any other the companies I worked at to be honest um but is like apart from you know playtika or you know uh these roles being seen slightly differently in terms of what their purpose is did the ba team also kind of act as a filter to the bi team you know as as almost like you know managing managing the bi team and um not managing but it's a different skill set I mean like the the most annoying thing that I get is we are going to hire a data guy that's like the most annoying title I've heard in the games industry yeah we have have a data Guide what what what is a data Guide like data Engineers are

developers in away that write code all day and their career is completely different from a data analyst who went in the University through I know he studied statistics or maybe they have a math background if they're doing data science but let's say that developers are not connect Ed to the business usually at all I mean they they need like uh they're isolated and they focus on what they need to do like they need to move this from point A to point B they need to compress that they need to normalize that those are all like stuff that request that are coming from normally data analysts because data analysts are not developers so they are like they say Okay I I I need you to prepare me this table the snapshot of this table if every morning okay fine we can do that or I need you to automate this

report but they're not the skill set is different and they're not it's not a a business function but I think that I like grouping all the let's say business analysis ba in one group let's say data analyst data scientists because that group also marketing data analysis you know obviously the the guys that work with the UA managers every like they will understand the business better and then separate the data engineering bi to another group that will focus on the bcks and shovels of you know the tools the database size the compressions the backups the maintenance the waking up at night when the process fails Etc so that's my from my experience that works really well yeah yeah there there's there's overlap there and and I think that that's why also people get get confused right like uh someone who a

data scientist like kind of lives somewhere in this this where they can do some like engineering work like they could potentially do some like a if it's not like super complex and you're building a whole data pipeline they could do a lot of like the data engineering work yes but I I I wouldn't waake them up at night that's like that's how I see it I mean I'm I'm I was a CTO all my life so for me like the I I always try to aim to waking up as less as possible at night and you know analysts and scientists are not the one are not the one that I would wake up at night data engineer on the other side yes if the if an ETL fails if a process fails and someone needs to wake up he he will be the one that we will need to wake up because that the you know the the is not flowing yeah yeah exactly yeah and I think yeah absolutely yeah I think it's like what what is is the core responsibility between like a data engineer versus a data scientist versus a data analyst

versus a business kind of like a bi or ba analyst um the the other thing and I think it's like a one one piece that I've noticed as well is that even when we talk about data analyst so let's say like you have your whole pipeline data is ready for use and consumption right like it it all works well over here like the technical pieces are are working well so your you have your data in a way that can be manipulated consumed and utilized even then there's different uh flavors of data analysts and I think that back to what you're were saying that of like oh I just have a data person um it's it's it's so hard to say that because it's data the data yeah yeah it's like you even your your data analysts have like different strengths like I've I've work with data analysts who are really

really good at dashboarding and like they actually love it right so they're live and and breathe you know building dashboards that can be consumed by others and you know making sure that that works beautifully versus data analyst that like doing deep dive analysis and like you come with like a very complex question and like they like going in and you know pulling the data and man how do I answer it's another it's another Talent you know building amazing dashboards is another Talent it's a completely different Talent reporting is like another world yes but there's so many so roles in data then that it's definitely not a data guy it's I I think that's the key it's not a your one data your one data dat guy not it's a myth that one guy can do it all no it's not working like that even the quality even

the QA no one is paying attention to to QA I mean it took us months to educate the QA teams how to finish a session and again then go over the the events to see if the data is correct because someone needs to do QA of the of the events that that you're sending so data QA data quality Integrity this is another hugee problem and something and a role you can say you know in I remember we had just we had we had two people just doing QA like full time and uh in dive we automated this we we added like a data validation function in our dashboard because it's really annoying to dat today but yeah and I mean I'll I'll pla I mean just kind of also goes to show uh you

know where uh where the reputation that they have in terms of you know um yeah just being very particular about uh their data and you know applying that data to drive uh results uh this reputation that they have built and it's a great great one you know where where this reputation is rooted so there this very interesting story and good to understand um but yeah I think the data guy myth is is also kind of [Laughter] interesting yeah um cool all right we we're actually like running up on time and there there there are a bunch of questions actually to go through maybe we should actually plan a part two also of this discussion to continue you know down the I enjoying the the talk with Tammy I would just add another you know to the listeners there's another thing that uh really people keeps missing Roi and roas going back to the

topic of the conversation ER you have to when you measure it you have to take out the the store fees first like a lot of game studios look at the gross revenue and for me doesn't make sense I mean it's that it's not the money that is left in your pocket so you know they they don't deduct that when they calculate the ROI and so please when you do that just you know take out at least the store tax fees and then calculate the proper Roi otherwise you know it's just money that is not really there yep I think yeah related related to that I went uh at congre I went into this whole quest to change how we talked about LTV um exactly for that reason um because it was like LTV what's LTV you just throw in all the revenue it's a time it's it has a factor

of time there is no not life I can say it's like you need a yeah it's you need a factor of time and you need to have it be net net revenue what's coming in yes so I went on this whole uh Quest and got everyone to call it something different that that I've never seen in the in the industry but um so we call it like oh yeah yeah you have to tell us yeah yeah yeah so it's it's PV so it's net player value and it always has to have a a number next to it so it's like MPV 30 it's like your net play value by day 30 mhm net player value okay yep kind of like that maybe maybe we'll start using that term in our in our reports going forward the quest the quest continues Tammy the quest continues the quest it's such it's such because it's so so easy to um just put like it can be the

difference between you know it being a good campaign and a bad campaign like that's it's it's that easy right it's like that minor thing which is not minor can screw you over uh first of all second of all are you talking about like a projected number or an actual number that's that's whole that projected that's a great point oh my God that you you should have done this like it should have been a one and a half hour talk the projected thing you know I speak with companies sometimes next time and I I speak with company sometimes and they say yeah we have like I know we have a we get a I know 100% Roi on I know 18 months and I'm like seriously are you running are you do you even exist for 18 months because right building prediction models need historic data you cannot invent data out of thin air so you must run for a very long time so that you can build a

predictive model so so you know if you take the the line and you continue drawing it in you know in air and supposedly the aroo will continue growing yes that will probably get there but I've seen so many dashboards that the arooo just goes flat after I know 60 days for example and that's it and it never stops growing because you know because of live Ops or the content Trad meal is not enough or some other reason but it's you know kind of makes sense yeah yes yes of course yeah I'm just I had to I had to like really laugh a lot at that statement yeah can imagine a conversation El and a potential client do you even exist for 18 months but anyway let's let's let's come back uh yeah I think yeah we should definitely like plan a part two we can kind of you know BL has a projected line that it it continues like you know you look at the

dashboard and when you activate the projection they just continue Drawing the Line in in air but it's not real you know it's not really it's not it's you can even wideboard it in you can wideboard like yes exactly and to make it I'm just gonna flatten it out a little bit towards the end all right so I guess yeah in part two uh you know we'll probably like jump into um yeah some of the other topics you know even further down the funnel one of my favorite ones is you know how designers PMs and UA specialist should think about formulating the right hypotheses or questions that they would then go to the you know uh the the data guy with um and uh and yeah you know other topics like you know once the data is in how should game teams think about taking action on it etc etc so we'll do all of that in part two but as kind of a final

question to wrap up this discussion um yeah Tammy you know if there would be one piece of advice that you would give game teams to kind of gain outsized value from data what what would it be oh man just just one piece of advice um I mean I think it goes back to what we were saying which is start simple like start with you know just understanding what you need to measure and why you need to measure it so just measuring for the sake of measuring is not enough I would say that that's the that's the advice that I would give um and don't yeah don't over complicate things that would be that would be really my top top advice keep it keep it simple at first don't over complicate

and don't measure for the sake of measuring like have a have a goal R makes sense elad what what about you I will rephrase that a little bit because Tammy mentioned it a few times and we have a slogan in a in the in our company that data starts with questions we we love repeating that phrase over and over and over data starts with questions you first create like first of all you know write the questions what do you want to measure exactly write those questions on features on you know what kpis do you want to look at and then take it from there and you will find out you would probably just need a handful of events instead of the whole we track everything and don't hire data guy please no specific then let's let's let's triple underline that all right great yeah I think that's a that's a great way to then end it for today you know the first question of the

next of part two of this discussion will be about you know asking the right questions uh so I think that'll be a good good opening discussion but but yeah thanks again elad and Tammy uh for coming on today um if yeah no worries uh if listeners want to get in touch with you and you know learn more about dive uh how how should they do so it's easy it's just a lot my first name at dive. games and my LinkedIn obviously is a i i really take time every day to answer people in LinkedIn like from even from other Industries it's it's it's fun I I like that so I like helping out people it's kind of like my thing so I'm super happy to help email LinkedIn whatever awesome and damy uh how about you if people want to learn about you what you're doing right now uh or even maybe

you know have a discussion with you and about data and like what your GDC talks and such yeah uh yeah yeah I have I have a few uh especially like on LTV like before I went on my quest actually I think I reamed it for the the the GDC talk just so I wouldn't confuse people but yeah I have a couple of uh data I think it's like two or three data driven talks um in the jdc vault uh I also had did a few back in the days of of casual connect uh around ads as well very very data driven too and I think that they're more uh you know even though these were like 2014 2015 they're like so relevant today as well like just think about it in like a framework of like how how can you actually uh Drive healthy um ad monetization for for your games and and as uh elad said I'm also in LinkedIn I'm not as good at being a prompt responder

but I will respond uh I do I do love uh you know helping and chatting with people um so even if it takes me a little bit of uh time to get back to you I will I will definitely do so awesome any your company's website is uh www. captain. TV righttv if you if you want to find out more about what uh streamer hosted Community games are all about super exciting that should actually even be like another episode entirely you know we should even talk about talk about that side of gaming at some point but uh but okay yeah I think we'll wrap it up um but yeah thanks again to both of you for coming on and yeah looking forward to continuing uh the discussion in part two if you enjoyed today's episode whether on YouTube or your favorite podcast app make sure to like subscribe comment or give a five-star review and

if you want to reach out or provide feedback shoot us a note at podcast nov. or find us on Twitter and Linkedin plus if you want to learn more about what navic has to offer make sure to check out our website www.nic.edu and advisories Services again that is www.nic.edu [Music]