Driving ROI with Data Analysis – Part 2

[Music] welcome to the Novi gaming podcast where it is our mission to explore the business and future of video games we bring together the industry's brightest Builders investors and thinkers to keep a pulse on current events dissect emerging Trends and games share Lessons Learned and have a great time this podcast is also part of novic growing ecosystem which ranges from free and premium research to Consulting and advice ADV services for more information visit [Music] www.nic.edu built the platforms for some of the largest live service games including League of Legends fortnite Destiny 2 and Plants versus Zombies 2 pragma Powers services like accounts matchmaking and player data for the world's most ambitious live service games the pragma

back-end game engine is the only solution that is truly extensible so that game designers aren't blocked by clumsy blackbox designs with pragma Studios no longer need to hire a large backend team and get the ultimate peace of mind that their game will always be ready to scale to learn more simply head to pragma Dogg or check out the link in the show notes and with that let's jump into the episode so welcome welcome welcome everybody to another episode of our data Corner series we're joined Again by elad Levy founder and CEO of tive and also Tammy Levy uh Chief gaming officer at Captain TV so let's as always you know let's start with some really quick intros and then jump in uh elad as a very quick refresher for our listeners uh could you briefly uh intro yourself and what dive does I'm a 20 years entrepreneur in the games industry with several exits and the last company

that I've started 5 years ago dive focuses on providing data and bi custom solutions for uh companies different companies different Industries as well but games are really a beautiful challenge because they're so complex to manage makes sense and dami uh could you quickly tell uh listeners about your background and what you're doing today yeah know it's uh great to be here again and dive into more data stuff uh so you might know me from uh the um Regular as a regular panelist from naic's round table or some of my GDC talks uh right now as Manny mentioned I'm at Captain TV uh and we're a startup where we're making Community games uh and it's really games for uh streamers or creators in their communities to play together in a in a very authentic way uh but I always say that I live in the

intersection of product data and game design and uh that you know I've jumped around a bunch of uh places from user acquisition to analytics uh to product so before Captain TV I was at uh congre as a chief product officer but also um before that head of insights and analytics uh and before that I was at uh singga um so I'm I'm always ready to jump in and talk data awesome thanks uh thanks for those uh intros elad and Tammy but before we jump into today's discussion um elad wants to just share a few quick words thank you Manu so for those of you who do not know I am Israeli and uh I just wanted to tell my friends and family and everyone in the Israeli Defense Force that I send my love and prayers uh please be safe and stay strong and back to you Manu great thanks El lot and yeah uh I guess yeah Tammy and me definitely Echo

the feeling and um and yeah we're here here to support in any way we can but but yeah thanks for addressing that um but yeah I guess we can uh with that we can jump into today's discussion um as a reminder this is part two of driving Roi with data analysis which is a broad topic that many understand has value but you know a lot can be learned on how to engineer it and in the last episode we talked about how game team should think about setting up their data pipeline data teams and budgets then we hit on what Cadence data processes should look like and how they should evolve over time and then we ended with how the on to many relationships between data teams and the rest of the organizational departments could be structured to drive positive results I'll put a link to part one in the show notes but today uh we're going to dive a little bit deeper into some more tactical questions uh around initiating and conducting data analysis

and that should no pun intended no pun intended yeah every every time I say di the crew here Smiles a bit but but yeah I think a lot of these tactical questions should result in some actionable takeaways for listeners so we can pick things up from how elad ended the last conversation uh he mentioned that one of Dave's internal taglines is data analysis starts with questions and I heavily subscribe to this line of thinking so elad uh I'd like to dig into that a bit further um how should data consumers in game teams think about formulating the right hypothesis or questions before approaching the data team with various requests well from my experience ER if you have an amazing product manager on the other side like a ninja product manager on the other side that is understands data he will know

what to ask and how to ask it but if you don't it's something that uh can't be learned over time as an part of the iterative process of data you start with simple questions obviously of installs da Revenue session length basic stuff and then curiosity starts you know picking up and you're like what if we I know if we give like a different balance at the beginning of the session will it affect the session length or something like that or will it affect engagement like looking for correlation so it's like it's it's a process that evolves over time and every team have different questions like product managers game designers like game Economist for example focus on progression systems and churn developers focus on crash crashes or crash reports or types of crashes per

day or errors per day marketing people obviously focus on the UA and campaigns and Roi so it's like every team has their own set of questions and you just need to centralize everything properly and start knocking them off one by one got it what what do you think tamy uh yeah I Echo a lot of that um I think one of the things that is super important that I always try to emphasize to to folks is um as part of that Curiosity and exploration you also have to suspend a little bit of your own bias um so you have to be ready to be wrong uh and be open to to surprises um with that you also want to make sure that you're coming as as eled said with questions and not uh chart requests and this is it it sounds like a

like a silly piece of advice but I see it time and time again um a lot of times you see people coming over to your data team and asking hey can you make a chart that looks like this on the x axis like that on the Y AIS with these segments and by the time you get there you already have uh you know so many assumptions baked in like it's is it are you even asking a question is it the right question like that is not that is not an um exploration in in your data so you have to take a step back and really think about like what is my goal like what what question back to the questions what question am I trying to answer or what goal am I trying to achieve am I trying to increase retention increase repeat spend sure you have to understand and identify which kpi are relevant for that uh specific goal that you have but you don't come with like the exact chart

and then from there you also have to understand your audience right I'm already talking about new players Elder players some specific segment and now you have kind of like all of that information you can you're ready to formulate a hypothesis which is an educated guess on what might happen an example here like super simple is hey if I increase the rewards that I give out um to players for watching videos um talking about like reward ad ad monetization it will boost engagement with that placement um that unit and therefore increase ad Revenue now you have like the whole kind of like idea there so I think it is it's it's very important to kind of come at it from that point of view make sure it's measurable make sure that you prioritize the most important questions the most impactful ones the critical ones and then again be open to your prices you are going to be wrong um so be ready for

that got it makes sense makes sense so I mean you already hit on some examples which is great because I would like to make this a little bit more concrete so I mean we have like a lot of uh you know War Stories battles cars on on this uh question itself but maybe elad could you maybe share an example from your experience when you know asking the right data question led to needle moving results how how did that situation play out uh you know at at which point of the game like do we have the highest churn like can we or yes I I will have to obviously create a dashboard for that but where do we lose the most players at which point of the game and that will probably if you structure the dashboard properly it will expose the the problems with the progression system in the game and if you work on it

properly with you know the economy game designer and live op teams that will move the needle instantly because all of a sudden you you lose a ton of another great example is funnels like all the questions that are around the funnels and loading funnels if you break that into pieces I had great experiences that when in games that are really mature and you move a lot of traffic sometimes you improve something in the loading process and you know you just it's like something really small that affects like half% or 1% in the in the loading uh funnel and you you see like the needle moving instantly after like couple of weeks you see like more players flowing in and and you know kpis is improve so I that's uh I think it's a couple of basic questions that I can think of okay

yeah and I can I can jump in I can jump in with um another example as well that um as elad was talking I I was thinking about so back to my kind of comment about uh ad monetization let's let's use that uh as a starting point um that was something that we actually rated on a lot at congrate um over a bunch of different Idol games so this was of like an iteration through like within a single game but also across games um and as we were going through it we were basically coming with questions of is this placement uh going to improve either uh percentage of players engaging with our ad units or number of ad um or number of views that we get per player on a daily basis so those were the two metrics that we were trying to move um and it was a

combination of both iterating on the specific ad units but also um so and that that means making the ad units that we were presenting either more compelling surfacing them better um one example that we worked on with um a team in in Canada called hyper hippo uh is we we just made the button for hey engage with ads a little bit more prominent it was not even like a huge change um we just made it slightly bigger and that I I forget the exact number but it was like a 20% increase on on engagement or something um pretty significant um and it was just like hey the button was hard to to to tap cuz it was too little in the corner just make it easy to tap and people were engaging with it um so that's a kind of like a very specific example then over the course of you know

many iterations like that we were able to get from our early um implementations we had like 40% of Da engaging with ads we were able to get to like over 70% of our Dau engaging with ads uh and all without you know hurting retention so that kind of like that concrete example is something that took many iterations many years many games but but um you know it was just coming with very clear goals and questions and um kind of refining that process we have a we have a card game that has a like card mode uh there are two types of card modes in the game and obviously you know we did like the retention report classic rolling whatever to see how it looks like but then we added the card mode as a segment like as a filter in the report to see if

it's somehow correlating to retention a very simple like question like do people that play specific card mode are are more like like have have better engagement and the average spiked from like 30% D1 to 70 it's insane because when when you hit when you hit something you just like you see things like move like crazy and they and I think that when games are still they start those movements are really aggressive and after like when they're super mature you're fighting over the half perc and the 1% really small changes but I I think that both were like really good examples and yeah that's the kind of stuff that sometimes is just you know the kind of upside that's hidden in plain sight you know but as long as you're asking the right questions uh you can kind of uh discovered that upside but I mean since

these were like slightly more positive examples um maybe Tammy you could give us a slightly more negative one uh you know when maybe um again like sometime in your experience when maybe asking a poor or the wrong question caused wastage of Team resources or you know adverse player results uh yeah so there's there's a few examples that come to mind um there's one example where um we had a um a CCD a card card battler game um it was asynchronous PVP so you know just not not um not real time you know you have your your deck you're battling other people that have their deck set up and we had um kind of like this very minor change it's like a change that you're you're like oh what

can happen nothing can happen right what was the change we changed the order of who plays first so which card gets dropped first and uh you know our we were just like cool that that's going to change a little bit uh win rates so we had some hypothesis like team team was thinking it like okay win rates are going to going to change a little bit but didn't think much about it after that uh and we started seeing a community get very upset about like hey the game all all of a sudden got super hard like this change is brutal like you can't win like there's like very real disadvantage on going second and we were looking at um at the dashboards and the data and we're like oh there was like a little drop on on the win rates like what they're crazy nothing's

happening and you know it took a lot of Cycles to like dig in and and figure out what was happening but it was just like one of those things where it's like oh the community just like doesn't like the change right uh but it it it just got escalated more and more and as we started digging um a little bit deeper there were there were two pieces to it one was that we were um the the dashboards actually had like some sequel mistakes like the the sequel was had errors uh so there there's also that piece of hey you know you're you're going to if you don't have like a solid um QA on on your data and your dashboards you might be looking at the wrong data so that's that's lesson number one there lesson number two was even when we fixed that the the dip also looked you know okay it was more

substantial but it was not as um prominent but once we started segmenting by certain types of players specifically player age um and uh yeah it was like Mosley mosle player age there was this huge difference like new players just based on how matchmaking worked they were fine but the hit that Elder players were taking like our most vocal players are players who monetize the most um etc etc the hit was like they had gone from like let's say something like I forget the specific numbers but it was like something like from like 70% win rates to 20% win rates and that is like a brutal you know hit to the experience of of the player and like

their progression and you know of course they were going to be in an upar for like this uh having happened and it was just like a lot of Cycles to get to the bottom of it um and I think it's it's it it's this was not specifically not asking the right question but looking at the wrong at the data in a wrong kind of from a wrong lens I would say um so that was that's kind of like one of the big examples that comes to mind on like rabbit holes that you chase that uh you know it takes a lot of time to really uncover the right um answer if you're not coming you know we didn't really have a a you know question when we made the change of like changing the order of who goes first or a goal right yeah I can I can completely relate to that uh just like the example or just like the

situation of who is the start player and changing that because I mean I come from a family my my wife's side is a heavy heavy board game uh like playing family and um yeah the biggest discussion for every board game uh and you know now we have kind of formed like fully randomized solutions for figuring out who the start player is but the biggest discussion is always who is the start player because it comes with a lot of advantages and disadvantages but I mean that's that's just like a random Side Story but but yeah elad uh do you also have like an example to share a slightly more you know negative one yeah I do have actually a couple of examples that come to my mind first is um I saw a lot of time and effort waste getting wasted on exess events I think Tommy said we track everything in the previous episode that did that the we track everything is a real dangerous thing

because it complicates everything it creates more work for QA more of work for the data Engineers the analysts are just overwhelmed and they completely at some point they start losing ER like the focus of what were what was the original question they just send events of everything and just a pile of garbage that no one is validating and cleaning and this is excess events is something that I really recommend avoiding at least in the beginning and then another thing that is uh missed not like not not the wrong questions but not asking that question is um building Source sync dashboards in the beginning in gambl lingu they call them mimo money in Money out because quite often you release something in the game or you fix a bug or you create a new bug bug where you start sourcing coins and that instantly affects Revenue

because Revenue just starts dropping out of the blue and chasing that is is insane it takes a long time you know so unless you have a a proper uh dashboard for it that you can monitor daily it's really really risky yep makes makes total sense awesome uh so let's uh move on to the next step of the data analysis um you know the right questions have now been asked uh the data is in the analysis has been done uh and conclusions have been made um this is kind of the point you know when game teams need to take action uh and a key part of that is getting various stakeholders bu in uh so Tammy what would your thoughts be around how game team should you know present uh the data results and proposed next steps to stakeholders in different teams and also stakeholders of different seniority levels uh yeah that's that's a place

where I think that we don't always focus our conversations about data and it's it's crucial it's like what are you going to do next with the data right and I'm a I'm a big fan of thinking about it as storytelling with data like you have to really think about you're telling a story you have to make sure that it you know has a beginning a middle and end and it's it's a compelling beginning middle and end um so think about it as as that you're you're telling a story your data has to back up that story your takeaways have to back up that story um but kind of going into more specifics um you know you mentioned different stakeholders I think at the very beginning like once you're putting together a presentation a product review a deep dive an analysis whatever presentation you're you're putting together you need to think about your audience who am I talking to am I

talking to um the game team and am I talking to designers am I talking to marketing am I talking to leads the exx they will have different priorities and care about different things so um you know it's it's super important you're not going to include the same data for an exec report than for a a deep dive for the team that's like working hands on on like the nitty-gritty of the game right um so that's that's one piece two is really think about how you visualize the data uh I think it's you know I see so many charts and graphs and that just don't tell me anything um so you want to make sure that you know when you're presenting a visual it's I understand what you're communicating and um you know just for like more

specific examples mistake there's so many mistakes on data V uh visualization that I see out there from your axis like your retention for example let's say you're showing retention and you don't your ax is like you have super sumed in instead of having it like from zero to 100 and it looks like hey the game took a deep Di right just like retention took a like a huge drop and if you look at it it's like oh it went from 40 to 38% and just from like okay that's pretty zoomed in graph yeah based on your your AES of course it's going to look like a disaster but is it uh so that's that's one example um another example of like how how I see just data mistakes out there

is um you know learn when to use decimal points right like I don't need to know that 3,675 32 uh dollars were spent etc etc etc like that again you're not telling me a compelling story like you're distracting me with details that um might not be necessary specifically for you communicating results uh another piece of advice when are you talking about absolute numbers versus percentages and when you're talking about percentages do you have a significant like a sizable group of players that you're talking about because you can't tell me that 50% of players out of a cohort of 10 did x y and z that again is like very misleading so there's like all these pieces where

you know when you're talking about data you can be very misleading so make sure that when you're thinking about the data that you're presenting and telling this story uh again suspend your bias make sure that the data is like you're you're being objective with the data that you're presenting and that you're following you know good practices on how you show that data um have key insights and take takeways that's another piece right it's like open with the key insights close with your key takeaways uh don't make me read a 20page report before I even understand what you're talking about and what you want me to take away from it and this is regardless of your audience like the data that you include on those key insights and takeaways will change based on your audience but don't make folks read 20 Pages before they know what what you're trying to say um and only show

data that's impactful that that we can act on um and provide the context right like try to provide context once you're going into the details provide the context and finally have actionable recommendations like make sure that there's like okay we can act on it in this way but if you've done everything up to that point it should be fairly straightforward for you to come up with those actionable um recommendations yep yep completely agree and I can um kind of personally attest to you know uh the data visualization Point especially like that uh does take like significant time and thinking to get just right um you know I kind of see it even in the articles that I write on uh on on on navic digest you know just getting the right basically this line of you know picture can say a thousand words applies two graphs also and if you get it right

you don't have to like you know write three paragraphs or like tell a very long story but just show the picture and the entire story is kind of captured in the picture itself um so but yeah that's that's that's all like great advice uh Tammy um and it kind of like dovetails nicely into the next question um you know there's this learn measure iterate cycle is a very popular one uh in product development but many teams you know both early and late stage are overwhelmingly stuck on the day-to-day treadmill and in such a way that they forget or don't have time to perform one or more steps in that cycle Tammy tamy what do you think in terms of you know game themes kind of building momentum and sustaining this learn measure it trate cycle yeah I think that um it it starts with trying to integrate those measure practices kind of like that flywheel getting your flywheel going very early even you know before your

life while you're in early development if possible so one of the things that um we've really kind of iterated on in our processes so we want to bake it into your process basically from the very beginning one of the things that we've uh really tried to bake into your processes at at Captain T for example is for new games like we are going through that flywheel of um measure like learn measure iterate and and begging it to a product road map with internal even internal surveys like how you know let's measure internally with the game team um you know what's the level of conviction that it's you know we're building the right game and be able to measure that how that changes Milestone over Milestone that gives us a lot of information at least internally and conviction and then as we start open up to new players um and testing

externally that you know early on again we won't have necessarily you know retention that matters for us us to measure but we can similarly have um you know do a little bit more qualitative uh analysis and and data Gathering from you know getting this kind of poll survey what I what I call the poll survey with both the internal team and the external testers and kind of start that flywheel very early so by the time you're Gathering data it's baked into your processes that everyone's like looking like okay now like first of all what were what was our goal at the beginning of the Milestone like what are we going to measure um it starts with that right it's like hey this Milestone ARG go is to test uh with these many players and we want to see uh either you know positive sentiment in this way or retention in this way you know have very

specific right so it goes back to what we've been talking about um so what happens is when you start like that by the end of it like or even when we're testing like the team like everyone in the team starts asking like okay are we tracking towards our our goals like are we hitting them but even then like even before then it just gives everyone a very um clear way of prioritizing their time and their work as well yeah um I because once you're like about to hit that deadline of like Hey we're going to hit the date that we want to go live um you know as a product manager you start throwing things at people was like hey can we do this thing can we do that thing and they'll turn like a Savvy team will turn around and be like does that help our um day seven retention test or like uh right no maybe next few these maybe next

one right so it's just like building those processes and like educating the team to be Savvy in that way I think like part of it is like educating it and you know if you did not start that early you can always start like you can always start getting your team moving towards that place but you need the team to care about like the whole team to care about that flywheel and you need you know your leads in the team to be driving that as well and integrating into every part of the process of what you're doing yes I I used to I'm sorry Tommy for yes I I really agree with what Tommy says I I used to be I worked as a CTO for many years and I managed bi teams and I used to argue with the product managers I used to force them to add in the product spec the the questions and everything so we would bake it into like jira subtasks

it was part of the fature development so that when when everything goes live we already know what we track and how it's it's being tracked already on production because otherwise you start running around you know like headless chickens and like oh we don't have this event we don't have that you know the question was not there in the beginning and it's like yeah but you can't do that and you can't change this and it's just it BEC everyone like pointing fingers at each other and it becomes really messy so wasting time you really have to like that's a that's a key role I think of you know tech leads or CTO or that to bridge the graap the gap between product managers and and development so that when we execute what they ask what the de product manager ask us for so you know it's executed properly makes sense all right so uh we're coming up on time uh so you know let's end with uh getting your quick thoughts on a slightly adjust adjacent topic um it's also a long-standing industry debate uh AV

testing versus community so I've generally found that you know there are two types of game teams uh one that loves AB testing everything and are and are also like very transparent with their Community about running AB tests on them uh the other uh the other uh type of game team essentially aborts AB testing uh because it doesn't like to split its community's game experience and thereby prevent any backlash um so Tammy which school of thought do you subscribe to uh and why that's that's a great question uh so I I started my product management uh career at a place that tested everything um so zingga is notorious tting s notar is for for testing everything um and I saw I honestly saw The Good the Bad and the and the Ugly about doing

that I saw a lot of benefits from you know AB testing uh at the same time I also saw and I I had to at some point do this like try to untangle this AB T that had like branches upon branches and like so many variants and you got to a place where it's like can't even make sense of like this test like it's not an Navy test it's like an ABC d e FG use the whole alphabet to test um so I'm I'm honestly somewhere in the middle uh because you have to be smart about what you test when you test and you have to understand your capabilities one thing that you have to remember is that um I actually think it's a bit of a futi question until you have a big audience why because you need enough player years and it depends on like which which metrics you're you're going

to measure against but sample size Matters like you need to have a big enough group on like your a and your B and your whatever else you're going to do to even have a good result so this really is a question for like big teams and at that point I think it really is depends on how you've built your community um so if you build your community and I'm I'm I am very much uh proponent of just like having a very open healthy communication with your community if you build that type of relationship with your community you can explain hey we're going to run a test uh you know this is how you know how we're going to do it and you know if you some of you see a little bit of a different different experience like these are the reasons why we you know we want to just just learn from uh you know this this test so I

think it's it's being transparent with your community understand your capabilities be smart about what you test uh I don't think you should AB test everything um because it's you know you can get yourself into terrible places of uh you know optimizing in all sorts of weird weird ways yep and also there's like a little bit of a culture impact because sometimes you know let's like a big a big discussion in a in a design design meeting or a big you know argument or a point of difference in a design meeting sometimes gets concluded with okay let's AB test it but you know uh and it but on the other end of that statement is a lot of effort going into the ab test a lot of you know there's an attached cost to it the result results takes time and basically a lot of things get slowed down so uh at least back in my

experience at Zinga you know uh even after coming up with this feature spec and you know do uh creating like an AB testing plan there would be another meeting between the PMs and the lead PMS where we would go over the ab test design itself because in that meeting we would be able to eliminate a lot of the obvious things that don't need to be tested so your ab c d e FG till Zed test can actually then just be a ABC test you know if you just talk talk through a lot of the uh variants and just eliminate them and save time save cost and you know just make the analysis for the eventual PM who would do the analysis a little bit easier the other the other piece and what you just I think kind of touched on a little bit is like test things that are going to be impactful test big changes um you know if you're testing two variants that you know there's like a 10% difference between uh how many

coins we give to these players versus these players is that actually going to make a significant difference and impact to your game I don't know if that's going to make a difference to your game I assume no but you know test big big changes um that are actually going to be impactful on on the game and uh back to one of your previous questions Manu tests are used to viol validate hypothesis so you have to have a a hypothesis you have to have a theory of you know what what it's an educated guess of what might happen and what metrics will improve um and I see a lot of times like kind of what you were saying just like being used as a copout of like I don't want to make a decision on this so let me just AB tested and see what the the data says and it's it's a little bit of like a cup out by data yeah but yeah elad what about you which

which school of thought do you subscribe to and why I so first of all I love AB testing obviously and uh I really think you have to take it slow volume obviously like Tammy said is critical but you know you have to take it slow like the whole ABCD like gazillion tests at the same time is very confusing you have a ton of kpis moving and doing at the same times it's a bit um it's too ambitious in my taste I prefer taking things slow you know few more week one weekend you test that another weekend you can test that it's okay I've had cases that even before like getting into Mobile and social games I worked in online gambling and I remember that something simple as testing AB testing the color of like a button that we changed it from red to Green green Spike the CTR like like crazy so you you never really know what

will happen and you also never really know what it will affect you have a theory that it will affect I don't know click through rate Revenue whatever but I I like er you know that's why I'm saying finish the test let let's let the data sink in and then see maybe it affected turn maybe it affected engagement the session link they're like a lot of kpis right when we develop dive the dashboard we have AB testing in our dashboard and many companies do ab test by gold which is a nice idea in theory but every company every game Studio we ever worked with they ask for so many kpis after the test that you cannot really Define a goal beforehand so in in many um cases what we've done is we built bu the dashboards with like 50 kpis or like something insane like 20 30

K like a ton of kpis and then you know uh check post AB test what happened in that in a in B in control group Etc so because you don't you don't always know what will happen as a result of that test um regarding Community I love Community feedback I think it's critical I think it's one of the biggest input for like a because data tells a story but Community can explain things that data will never ever be able to explain so a proper Community Management has a ton of insight to a game but your question specifically I'm not a big fan about that transparency part because it depends on the game to be honest because games that are kind of a single player let's call it like I know match three or you know in Social Casino

you don't interact with other players so if you run an AB test no one really knows about it I mean unless they I know Post in the fan page or Facebook that they got this or that multiplayer games are most social by Nature so they will probably speak more between them and then if you try to do like you give more coins or like something to one group or the other group then they will start arguing or coming complaining so it depends I would be more cautious in like multiplayer games very social games you know versus single player once yep makes sense yeah I mean yeah it um I guess yeah with many things in life you know the answer is it depends and but it sounds like uh it it sounds like both of you are kind of uh somewhere in between these schools of thought basically to kind of summarize it in one line it would be I guess what your takes would be is there's just a

time and place for either approach um you can't don't go on either end of the spectrum don't like live on one end of the spectrum all the time but you know just kind of pick and choose depending on the situation whether you should really a test something uh or whether you know just go live the change and and the numbers will kind of make it obvious um even without a Navy test but um but great I think I think that's a great way to end it for today uh thank you again elad and Tammy for coming on uh elad if listeners want to get in touch with you uh and learn more about dive how should they do so so it's super easy to connect us our website is dive. games and my email is elad dive. games you can find me in LinkedIn as well and again thank you so much Manu for this uh great episode and bringing on Tomy as well it's really fun to you know speak with

other Professionals in the industry yep I think we could have probably do 10 more parts of this series but we'll we'll stop you and how about you Tammy how how can people get in touch with you uh yeah so you can uh ping me on LinkedIn I I am slow to reply but I will reply I promise uh my my LinkedIn hygiene is not uh the best but I will uh I will always be happy to connect chat um and even jump on a on a call in and exchange stories I I love connecting with people and I think that I'm I'm very much about uh if we share our learnings uh across the industry will just be a better gaming industry as a whole so uh definitely reach out awesome great then thanks again to the both of you and and yeah we'll be back with episode number four soon enough uh and

probably in the next episode we can touch more on where data analysis is going uh in the future uh you know how how how teams processes uh and everything around this topic is changing going forward um but yeah uh I guess we'll end it there and see you in episode 4 if you enjoyed today's episode whether on YouTube or your favorite podcast app make sure to like subscribe comment or give a festar review and if you want to reach out or provide feedback shoot us a note at podcast nov. or find us on Twitter and Linkedin plus if you want to learn more about what novic has to offer make sure to check out our website www.nic.edu and advisory services again that is www.nic.edu [Music]