Data Science at Games Studios | Andre Cohen Head of Data Science at Tilting Point

you hey everybody today we are going to be talking about the role of data science within game studios and more specifically one what is the role of data science within the game studio - what are the primary applications data science is used for three what does the data science team look like in terms of structure and team size four how do you hire the data science team five the biggest mistakes made with data science teams and six the future of data science and Game Studios and here with me to talk through all these issues is andre cohen head of data science and mobile games publisher tilting point welcome Andre hey hey Joe glad to be here hey yeah it's been a little while since we last met but yeah it's really great to see you on video and really great to reconnect so I thought I thought I drew we could go ahead and start just by

establishing who you are and why you are actually a really great person to be talking about data science in game studios so maybe we can start there in terms of like you know your background and career and data science gaming as well and before you start just want to let the audience know how we met so previously when I was at Sega and then NBC Universal as a working in games publishing you know I had the opportunity to meet with a lot of different vendors a lot of different tools providers and you at that time started this company gondola and I have to say that you guys were really impressive in terms of just the thoughtfulness of your solutions just how well you guys understood the various problems and the technology that you guys had so just wanted us to say that you know from that from that moment that we met I was definitely very impressed with you yes yeah no thank you yeah that meeting was was like a lot oh we had a lot of practice before he got to knock on Sega's door and he didn't even see the the before old lady after

but yeah - Terry a question right so Dana science that I have to be really clear didn't exist when I went to school that's a new thing that's a that's a hipster San Francisco uber Airbnb Facebook kind of thing I actually went to to a PhD in computer science I was doing machine learning my my goal was always just to be a professor you know a nice cushy job and I realized halfway down through my PhD that I never had the access to funding and I never would be able to be to see how it gets done right like how do you get money because at the end of the day when I went to school that was the time when it became obvious that professors were getting their own funds like a startup having their labs and doing their research and it wasn't just free money being coming in it was really a business so and one summer I ran out of money

like a lot of PhD students run out of money or they don't find the funds to continue working and I met Nicholas the the guy from gondola my co-founder and he had a game he was working on this game on his own and needed a developer just to remove like I was like AdMob from the game and put moat pub in like really early like this is iPhone one right this is there is no freemium there is no in-app purchase this is hardcore $0.99 game so I did that for the summer I never work Billy with an iPhone but how hard could it be to work with phones it worked out pretty well and at that point I realized you know what I want to work on a startup this is a this is a hot market freemium had just been announced like the concept of in-app purchase by Apple this was like it was like the beginning of a gold

Gold Rush and Nikolas and I joined forces and we started making a game from scratch and I think this is where I glad I learned a lot about the industry and maybe how I was able to relate to you at Sega was making a game from scratch I'm talking about no unity at the time right there is no there is nothing so I did all of it from coding the game engine to talking to the graphics guy getting the audio into the game connecting all the different SDKs at the time like OpenFeint whatever it became later Game Center and and analytics there was this concept that you might just want to track your players and and understand their behavior like there was nothing it was no game analytics at the time so built it from scratch this whole back-end for the for the game there's also this concept of like wow wouldn't

it be great to do an ad test in a game so there I go coding this thing like completely blind to what else is happening right this is completely done in New York just the two of us being Nicolas and as you would expect with a lot of people that freemium game we made never really took off at the time wrote was it yeah Angry Birds had come up with this like great publicity story like how Angry Birds took ten games to hit success right and so Nicolas and I looked at each other like wow some basic data science tells us that this is our second game yeah the eight were to go so we cut it short we cut it short the game making but we figured out that you know a beat us what did work and we doubled our revenue in our game by just tweaking numbers in the game literally numbers in a JSON file and so that's part our imagination to

pursue this data science route to change from being a game company to being a service provider and that's seven years ago and we started by just focusing on analytics with the idea that hey maybe if we give people data they will make decisions took two years to figure out that's not a good idea because data by itself is there's this concept like is it actionable data right then we took two more years so you know exploring dynamic pricing this is how you met us dynamic pricing for games yep we realize that that itself is not good for every game or applies for every game and then we took another two years to figure out you know there is live ops optimization the whole player journey is an optimization in itself and there's bigger things you can do and that's the story we got acquired by tilting point and now we're talking

grant and said just talking like starting with the role of data science could you elaborate a little bit more that in terms of like you know on the general level what is the mission or the primary objective of data science or what should it be within game science and within game Studios and just to actually kind of keep it real for a minute here I will say that in my own experience just like it a lot of companies they're like the higher up you go in terms of like executive level they're all saying yes we need data science data science is so important but then we have some like what does data science do they literally have no idea what the data science guys are actually working on so it'd be great to to kind of clear that up for everybody especially if you're a high-level executive and here's the deal they're not completely wrong so broadly speaking data science to me what it means is we are basically exploiting data in search

of making money we there's the belief that you know data is as there's money to be made with the data there's insights that you can learn from it you can make decisions for your business based on data that's not an old that's an old concept nothing's changed I think so much so that I would argue that you know in a big publisher if you make revenues from in-app purchases ad ad monetization right I would argue data science is the third pillar like that's how you make money and so that's how big of a chunk of your revenue it could potentially be now the problem is like as you're saying that's keeping it real and you know the confusion is that people think that oh man we're not making money though those first two pillars not working out data science let's get the third pillar in they'll they'll turn the company around and that's where I think a lot of things you know hit the wall hit the Fed and don't

work off for anyone so there's a lot of roles in data science that's the other problem like who do you get for data science and people talk of roles and it's not a role it's a way of thinking people saw you know Airbnb and all these you know unicorn almost companies seven years ago seven eight ten when they got started and they saw the data scientist as being these rock stars they're like we find value in data and transform the company around and what they fail to see is how big of a chain of things were happening at the same time in the company in the organization it wasn't that they hired data scientists is that they hired a whole bunch of people along the road that fall within the data science world and that's usually the thing that I have to educate a lot in terms of like specific applications could you speak to what are the specific things or specific

applications of problems that you guys are getting it goes back to this concept that you know data science is a whole system in the company it's not just one thing in many ways it's like you know product in a publishing company is not just the PM right you have live ops people you have creative people you have artists you have developers that work on unity and data science is no different so data science starts with collecting the data what events do you have in your game like that that's a very important part of the business how do you store that data like where is it going does it make sense the way it's being stored will it be usable tomorrow right or is it just being dumped into Amazon or a cloud service provider and hoping no one has to look at it again what questions to ask the data that that often is it was so sad I remember we

were naive ten years ago me and Nikolas and so I wrote this analytic service and I told him necklace we have real-time analytics okay okay the game just got launched okay what do you want to ask I'm ready like switching hats right from developer to VI and I was like I'm ready to write some sequel Niklas tell me what you want how many players in Japan it's like dude that's that's the best you could ask like two months of work and all this you know PhD down the tubes and now we're checking player counts so it's hard and I see this all the time in game companies like the questions rarely are really well formed for our actionable questions ad testing is the other part of data science it's a whole world of you know testing and it's not just for the game it's also for creatives anyway right and finally there's data science as you know the the one that people think of

data science as the guys who do that make pricing do longer Studies on the entire game like LTV projections row as projections looking into the market to see what games would be good to build next and that's a whole other subject but so it's a really broad spectrum as you can see right like if you were to try and categorize the biggest areas whether it's like you know you a relative to you were manifesting some form of concept testing I mean do you even do things like game balancing and things like that or like if you if you were to categorize how much time you spend with and on the different problems and then I assume because it's your point it cuts across so many different teams you're working with all these different teams as well right like so how are you splitting up your time amongst some of those applications our teams so I like to split it in two in two ways the first one is it depends on the company by the way like if you're if you don't do ad monetization half of

this conversation goes out the window or if you don't do marketing for your game because you believe your game is so good that it speaks for itself power to you you do that so I spent 50% of the time anyway matters like can we can we do better EA can we optimize the bids do are we doing the best we can with the services that the network's provide like Google proxy events and things like that so that's the 2% the other 25% are definitely in what could data are we collecting and data quality it takes depending on the size of the company the bigger you get it gets harder is just managing data is a full-time occupation making sure that we have the synergy and all the data so that we can track a player from add to attribution through in gameplay to churn through the push notification trying to revive the guy you know all of that

should be consistent and then the last bunch of the percentage the last 25% is you know long-term projects that that's I think what people remember all the time it's like ah he's doing the the super fancy multi-armed bandit algorithm where it's like PhD work like don't ask but it's good stuff it makes the company look valuable and that's the other 25% maybe this is another follow-up question like one of the things you mentioned is like managing the data now one of the things that I've heard is that data scientists hate cleaning data but it takes a whole lot of time a lot of science does get get kind of lazy about that like can you speak to like why why why do you data science teams have to spend so much time clean data or managing data so tilting point from tilting points experience and I've heard this from other places that it's not that they don't like cleaning up data

it's not their job and I said and I don't think it's our expertise because data science is I think often like someone that did a computer science machine learning course and they're interesting in models and making models and get doing that part of the chain of events there's a whole world of you know DevOps and making sure the servers are running and there's the other amazing position of data engineers they are the the stuff that more people should be thinking of I mean like they are the that here is at the end of the day of getting the data and cleaning the data and they do a fantastic job like a tilting point I see should be their job basically it's their job and you know what it hurts to see it the wrong way I was doing interviews this couple months and at first I was trying to find data scientists for this exact purpose they just said I'm cleaning up the data and I would tell them upfront like transparency right like man listen tilting point you know well we have some

data to clean still like we're still working on that I mean there's data science projects to but you might have to clean up data sometimes and the good ones the ones that were more veterans and the topic would look at it and be like wow not for me not doing that there was through disappointment in their eyes because all their dreams like just evaporated and so I realized like no you have to hire the right person for the job okay and then you mentioned that you spend you know half your time on you and so just to give our audience kind of a more concrete understanding of what it is that you guys do could you talk about like let's say for you a what is some of the deliverables look like so there's a specific problem and what is that deliverable what do you guys actually deliver to the u18 it's just if you can think of any specific examples oh there's a ton a ton so there's two ways there's there's optimization and there is other other things so the there's the

optimization route of things that is things like can we do better bids right and it looks at looking every day at the bids that we're doing at different networks and optimizing them to be competitive right oftentimes I see that you can over bid on something like and and you don't know that like the next person bidding for that target audience you know is is half the amount and you're completely overshooting and obviously Facebook and Google gladly take your money and but you don't have to you can get the same result for less so it's it's a basic you know optimization there is projects like Google proxy which are really fun because Google proxy events for those that don't know it's a system where Google in order to find you your audience for your campaigns they they try to learn about what is your perfect player based on some criteria like do they spend money in your game but that happens very very early right

spending money hep is like one in a hundred if you're lucky a hundred plays you require spend money so one of the things they allow you to do is send additional events and signals to Google to to train their algorithm for other things other than just spending money and one of those things could be the propensity of spending okay this player hasn't spent yet but it's looking like you will like Jesus gotta wait like 70% chance at spending and I can make a real big difference because it makes Google learn faster about your target audience basically just trying to find you know kind of upstream events that will be very you know highly correlated or potentially it can predict a purchase event essentially exactly it can be things as dumb as you know do they finish the tutorial you know do they come back you know five sessions in a row or how many days have they played you know these are all week indicators and what's really sexy about this is

that there's a lot of synergy between this kind of project with a lot of other projects you might want to do because prediction is really powerful and we could talk about that in another part but so predicting you know that if they will spend or how willing they are to spend is can impact live ops and targeting of promotions and etc and then the other part in that I spent a lot of time thinking about is can we do better ads because one of the easiest ways to improve your you way spend is having a better ad obviously you can't just be chasing the the mythical ad as you know a ridiculous click-through rate but you can learn you know you can learn about rather you should be using a green button that says download now versus the red button that says free now you can learn about character choices some countries have preferences for specific character in the game and with data you

can actually see that and you can help artists make better decisions about ads and creatives just so our audience can kind of and what potential impact data science can have because you talk about what kind of bill says using so whether it's you know in bidding or finding proxy events for Google but you know how what kind of a performance improvement or a cost improvement are you generally able to drive using you know whatever data science methodologies you guys are using yeah so you way has a lot of variability sometimes nothing happens and sometimes you see great results and partially I attribute that to the fact that you're optimizing already a machine learning algorithm that is a complete black box and in some ways they would rather you not optimize it at all right because they're doing the business for you of optimizing it so if there is it you can see for instance like in the bidding example like you can see a laughing you

know the bid or improving by twice or spend just by tweaking the bids google proxy same thing you can easily have a 20% uplift in your campaign if you have the right events and if you have the right predictor model you can see a bump there the and by the way I always like to be super conservative because there's nothing worse than like over-promising because that's the other downfall of data science is that for years we've been like Oh 5x hundred percent uplift and it doesn't do anyone good it dry it inflates the whole thing to a degree that can be met and for product you know we're looking a little less it's more consistent like so you know in game author optimization dynamic pricing what offer to show a player how many ads to show in a session you know those who are looking at you

know 10 to 20 percent up lifts but they're much more consistent they don't vary as much okay great and then in terms of the team itself how our data science teams typically structured and maybe you could talk about if there are different kinds that you know whether it's five game or whether there's a data science team that you know is organized around applications how do you think organizing your data science team and how how do you figure out how many data science scientists that you need and what were the typical types of things that they're doing yeah so this varies a lot based on what business you're running right it bases off of are you a game developer or your game publisher do you have is it one game that you're you you're putting all your eggs into or do you are you distributing it across a whole library and and oftentimes like and as I said data science is a

collection of different roles data size just being one of them and so if you talk about like the game developer route right you don't actually need data size necessarily I'll say that you can do great things with less and what you need is you know good control of the data and someone that is eager to exploit the data for profit essentially it's someone that sees an opportunity in in the game and says wait you know why do we show always the $7.99 offer when the game starts like we know the attribution of the player we know that they've been acquired for $10 like the CPI was 10 like why are we showing $10 here or vice-versa why are we showing the $10 offer for an organic player like this guy doesn't even know what the game is about and we're already like trying to make him into a whale and so they're you

know it's not Sara Lee data scientists it could be a data engineer with a with interest in that I've seen physicists with it doing that too it's more about having a strong base a strong foundation and controlling the data and having having them one or two people to exploit the data if you're a game publisher stuff changes a lot though for publisher you really have to work this there's this thing called the pyramid data science pyramid that's like how it's usually conveyed nowadays and it's it's basically from the bait foundation being ingestion of the data going through cleaning up the data analytics baby testing data science and so for when you're talking about that pyramid is a nice way to think about it because it also kind of represents how many people you need so if you basically it's like almost double per layer that's

what I've seen it tilting point working out and I would see this growing if you have one data scientist you probably need to be AI analysts if you have to be AI analysts you probably need four or five data engineers that's just a you know it just multiplies itself and it's it takes more commitment it definitely takes commitment to do that because you have to hire a lot of people to start the engine so let's say you're you're a small game studio you've got one game product and so for that kind of a team what would you would you suggest they not have a data science team initially or and if you were to get a data science some type of data science team or one data scientist earlier what kind of a team would you hire and when if you're like a new game studio just kind of starting to get a launch a game so if I'm the Bucks lunch a game and I need and I believe that there's like pillars

of revenue and one of them is data science I think it's the best thing you can do is is get someone that is as close to a full stack engineer as possible to be honest someone that is not afraid to clean up the data someone that is not afraid to do data engineering do DevOps someone that's not afraid to look at data and graphs and and find the because here's the thing even if you were a data scientist right this is the what I keep doing most of the time is even if I find something that makes sense such as what offer to show now you have the whole problem of how do you get that into the game like and this is where full-stack news comes in you have to have the know-how of how do a games work are using the employ more swerve or Delta DNA like how are you going to push that knowledge to have an impact so when you hire someone that's very focused it like in a small studio you risk the the likely problem

that they're too focused and they'll hit like walls all the time is like ah man I don't know unity you know or live up side I don't I just do I just threw models and machine learning so you have to think broader that is when that kind of person be pretty rare or somebody that knows the data science but can also you know it has a C sharp unity and pipe super rare super rare but they happen they happen I I just spent my last six months you're doing this kind of work and you have to keep looking there are eager talented engineers out there they're not fairly rare you can spot them and like pretty easily after you say hi to them you know okay this guy this look guy just like stacking is passionate about technology they're not focused as far as the data science that part of their background is concerned

what would you be looking for I mean is it you know someone who's just taking a random being a science class in college or are you looking for a PhD or you know yeah how much how much before they're dangerous well that's the thing I hate to do this but it's like I don't have been being real here and I don't want to like these ridiculous expectations you know like oh man you need a PhD and I better have a data science degree because those things don't really exist okay not really and they're very new and data science without experience is also not particularly the best bang for the buck if you're you know a small studio with the game because you have to learn on the job and it's part of the that's part of the role so the the best people like if you're looking for data sites like if you because it's probably easiest it's someone that has done actionable things that's what I would say I would

focus on action it's it doesn't matter if their project was you know finding the best best time to call an uber for for JFK and just that was there you know hackathon project and they they actually show that you know if you do it at 5:46 that's the best time to press the button versus you know five minutes before if I've minutes later that's the guy you want you want someone that just likes to do things and always is producing actionable stuff as far as their degree I am biased to computer science that's just my background personally so there's a certain amount of knowledge that comes from going through the emotions of a four year program it makes you think of problems differently but that's I'm surprised all the time without computer science degrees so it doesn't sound a requirement and maybe just going back to that new studio like the game startup situation if we can't

find that uber data science programmer DevOps yeah my god what should we expect in terms of like for more traditional type of scenario like how much of a budget how many people would you think is you know kind of a practical rough range you know just start building applications I mean yeah I mean I don't want to be a lawyer here right also be like well it depends but I have to say one thing what the biggest factor I consider when when thinking of these roles is how long do you want to live with the product that they generate and how long is this game going to last if your game is if it's a game that has a license with a special IP and it's you know sunsetting after a year anyways because you didn't pay for the extension or you know the extension will be ridiculous that means you only have a year to to live with the the work they produce if if it's a game that you're hoping will be you know a chain of games

or it's like this is the thing then it's things change a lot so to your question right short-term hyper casual you know for hire I don't know something like that you know you can do something with much less I would say you know 70k data engineer I would focus on data engineers to be honest because they actually don't complain when they go into data science they actually see that as an opportunity to learn something new and improve their resume but the other way around not so good data scientist doing data engineering they always feel like it's a one step down so you want to start there and and yeah the the you know it does have to be a rockstar because it's only a year who you know as long as the data was dumped somewhere it's cool and as long as there's movement in the product that's fine and you have a team of two or three engineers and it'll be peachy if if this thing is for a long like a

long haul like you're gonna build on this game and you have a two year you know roadmap already of features and PvP modes to be paid you can't you can't you get go you can't cut corners that's you should never cut corners the decisions early on that you make and the decisions the engineers will make by themselves because there will happen right they will make decisions on their own especially at the beginning of a team they will have profound like directional changes two years down the road that are that are very painful and I would never I would never sacrifice the salary for the team so yes you probably need seven people to operate this thing the data science team but if you don't have that budget I would start from the bottom and move up so I would still get the best the best you can possibly get you know data engineer or DevOps person I would still get the best bi person in the

world and if the budget doesn't get to data science you run short just wait a year it's fine because the quality of the data will allow at the very least it will allow the executives to access the data and they will be able to make something out of it because I've seen so often you know you a managers who opened the raw data in Excel couple pivot tables couple graphs you know they love this stuff to like they do all the shortcut keys in Excel and they do all the bidding optimization by hand and like paper and pencil style every morning at 9:00 a.m. that's fine that will produce the revenue to hire the data scientist later so I thought we can now talk about mistakes I mean you know everybody's got mistakes and I would imagine just because the role of data science is often misunderstood within a lot of organizations and game studios I wanted to ask you what some of the biggest mistakes you've seen are in working with

data science teams but maybe I can also preface this with just my own experience which is that you know and kind of to your point about how early decisions kind of impact your data science team so one of the things that I've seen is that that like some of the biggest mistakes were just setting the mission of the data science team and hiring that right initial lead and and so what I've seen at times it's just like data science kind of gone wild where you know we've got these guys and it's really more about like these really esoteric weird problems that have nothing to do with the business case right and so they're out and windmills I don't know so I so given that would be great to kind of hear your opinion and maybe your own thoughts on that a bit that kind of situation yeah now the the PhD is running you know going crazy going wild in their own department and their own floor right because the other thing about teach these is that you don't want

to have them like with the other people like you're gonna like separate floor they do the crazy stuff you visit once a month maybe and that's a common case that's a common case that if you have funding to because that's not cheap it comes on very high cost and usually what that means is like you've got some cash to burn and good for you because at the end of the day when you hire a very large pool of PhD talent first of all there's great work that they do for themselves like the value that they're generating for their academic life is you're gold because gaming has the most amount of data and the most interesting kind of data out there so you know it's great that that that will exists but it's hard to direct them into the real world very hard it takes a very special person to to focus PhDs to to real life problems and by the way Pacey like this whole idea of a PhD and the company

makes for great it's a value of the company based on the hires you've made right it is not transferable which is kind of sad because that was probably the idea was I will hire PCs to create value tech value right so that you can sell the company off or be acquired but it turns out that there's very little as far as like mistakes I see a lot there's a couple so like the first one and we just talked about earlier is the budget business PA data science is inflated in salaries across the board I mean it's in the u.s. since late it's in New York inflated you go to Ukraine it's insulated and you're talking about 20% inflation at least so it's very likely that you go out looking for a higher that you think is you know being 70 Kaling should be decent like you know not not complete junior not out of school yesterday and it turns out that that not necessarily the case you you can go really far a data science without

having that experience and you're still paying a large amount and that's why I say always go back to this concept of like you don't necessarily need data science you need someone that gets the job done because also like the UH flipside mentioned earlier they don't require a PhD they're not PhD level optimizations you are now doing deep learning you know genetic algorithms with you know tensorflow it's just basic optimizations the other mistake I see a lot and this is actually the one I am more passionate about because this is what I really learned from gondola and I see it happening all the time and it hurts because it's hard to communicate specially with data scientists is that they have to run with small projects in mind you can data is talents in a company is not a it's not a long-term sabbatical project of 12 months where you you sail out into into the void and you hope to find the

2x improvement in LTV predictions within that year and nothing happens because what happens every time and we did this at gondola all the time I did it is you become further and further away from your customer and by the time you you find the optimization you were looking for or the model that delivers you might have totally missed the mark on what they wanted and then what do you do right I mean that kind of happened with the dynamic pricing because you can have the best dynamic pricing solution it doesn't mean that every game is suitable for it like if it's a PvP game won't my players talk up to each other like is it fair to have two players with different prices right that happens when you go to the PhD routers studying a lot and not talking to people the other mistake that happens from the same purpose is that easily nine out of ten the project fails

like catastrophic Lee it's like not good like oh we were doing this project optimization model and yeah yeah now good 5050 best better accuracy is fifty percent accurate or less like I just don't know I don't know what to do and and it hurts a lot and it hurts morale for the engineer because they they're suddenly looking at their you know end of your review and they're like gosh all I have to show is this horrible model that failed and on the other side it hurts the the company because they're like what are these guys doing like is it are they just bad or then we do something wrong is it that hard why is it so hard or does it like that's also though happens like uber does and what could it be so what I the way to solve that and I keep trying to do that all the time is fail fast fail fast like how

can we get this model or this data science project to sell within I don't know a month month is already too long I want to like if this is going to fill it fells usually really fast it's not something that takes a long time to fail and it's fine Phil it's good it's good it saves everyone a lot of brief and it moves us forward so last question I have is really about where the future of like data science or whether it's applications team team structure of that kind of thing where the market is headed and certainly you know you're talking about a lot of changes that happen and just as a specific example we have Facebook's new their updated data use terms for mobile app measurement not able to get like device user level data back and things like that but just given how dynamic the industry is and things like that just wanted to get your current sense of given all these changes happening where were we headed yeah I mean data science has changed a lot when I started write real-time

analytics was the thing and that's all everyone did then there was a be testing craze everyone just did AV tests and data scientists were part of that business then we moved on right like we know we don't talk about a/b testing we that's that's live offs people talk now right we don't touch that I touch multi-armed bandit testing now maybe maybe in five years I won't touch that either right things change and so the first part is of the industry is is letting go you got to keep your letting go of things as they become commodities real-time analytics is nice controlling the events through sending is nice but at some point you gotta let go there's people like amplitude that do a fantastic job you don't need to do that as the myth the industry matures I see that it's all about the end-to-end player journey optimization that is the new Holy Grail it was always somewhat of a Holy Grail since the beginning because we could sense the the fact that there was a

continuous length of data from creating the ad to doing the UA through the player starting playing the game the first you know fatu e experience through the whole lifecycle they churn you try to revive them you showed them hats like there's a whole chain of events and getting that under control is I think the new playing field that no one has control over yet there's no service that can do it yet there is it's something that requires a lot of orchestration and that's where I think the the good companies will show up and will pop up in there's the companies that can do this from end to end as far as you know and then that goes into like well how do you change it right how do you get to that point of having this end-to-end optimization of the player journey and that's tricky I often think that from tilting points perspective you know data science has to be in every department to

get this done right right because even business development the people that bring the game into a publishing company it's data science to be made they're like what games are you looking at well who are you targeting getting the games that make most sense to the company so that you can publish it successfully they decides they're - and so in some ways I often have this dream that you know maybe data science should be something where it's distributed across the whole organization and have you know people specialized in finding these opportunities in their data and bubbling it up but it's hard it's hard to get them to to work like that it doesn't make sense unless you're a very large company and what happens at the end of the day I think is that you you just have to be moving around and you have to consistently move around and find the where the next bottleneck is because that's ultimately what data science is about is you know if you look at the chain I mean that's what I do it tilting point I look at this full end-to-end chain I find the next broken link that is the biggest broken link and you try

to retool it try to like fix it somehow make the data go from Beadie to you way okay okay we got that link down okay let's find another problem oh it's a ad testing data now going through the MailChimp email okay let's figure that out though and keep doing that all the time it's a full-time job well so that was basically my last question and so like maybe for our audience do you have any any message for our audience for is there any way that people who might be contacting you can I leave a link to your LinkedIn or something like that or you know or something sure I mean the best way to I'm old school very old school the best way is on Draco and a gmail like then like then I I do that too yeah last words is I think is data science is a fascinating topic that requires it to be very realistic if you're realistic things can really happen and you should you should focus

on on the people it's about the people that you work with it's not about their degrees or their passions necessarily because that's where things go wrong it's really about their character and what they're interested in working in and makes a difference cool all right Andre thank you very much for your time and that's the wrap thank you very much hunter thank you Joe [Music] [Applause] [Music] [Music]