Panel: Your new BFF: Data As A Game Development Tool.

[Music] thank you [Music] we're gonna have an amazing panel about that as a tree a drive for development and I will let Charlie do the dirty work on everything so please give a warm welcome to Charlie and the panelists hello everybody uh uh welcome to uh PG connect Seattle we're very thrilled to be here uh we are on of course as Pat said on the last session of the data Deep dive track uh we are doing a panel with uh Maya two lovely panelists here about how data can be

used as a game development Tool uh so as we kick off the panel I would like to introduce my panelists so if you'd like to say your name who you work for what your role is and maybe like a little fun fact yeah hi everyone my name is Dale Lawrence I work for Bango audiences and I'm the head of our Client Solutions team so I helped sort of manage onboarding and managing the relationships with all of the clients that we have um you've thrown me under the bus with an interesting fact there um the most interesting one is that I was a budding child actor when I was when I was very very young um I don't do it anymore it was a it was a horrible industry to be in but it was an interesting fact for you to know hi I'm Shirley Roberson and I'm an attorney with Hughes media Law Group we represent a lot of gaming clients and I get involved with the privacy and the data control issues from a legal perspective I'm not here to give legal advice but if you have general questions um I just have to give that disclaimer I'm not your attorney today I can be if

you want me to be um I got two interesting facts first of all I didn't come prepared with a sexy accent like the two guys did so I apologize I did grow up in Georgia so I can fake it if you give me a beer and I have been bitten by a shark so there you go it's a lot more interesting than any of my facts okay all right so we're going to start off with a fairly I'd like to say a fairly like a nice introduction question to the panel so uh my first question I just wanted to ask uh how important is data analysis in your everyday tasks who wants to start now yeah yeah um for us at bank audiences it's incredibly important so just to sort of set the scene for Bango and what we do is we are predominantly a payments processing business so we process in-app purchases across the major app stores we process subscriptions and you know a whole wide range of different payments so data analysis is incredibly important for us to understand what the users that are paying through our platform are

doing across a wider ecosystem so we're constantly analyzing the data to segment our users into usable audiences that can be targeted through social media campaigns so making sure that the segments that we're producing are having the highest quality possible whether that's based on the similarities between the users the amount that they're spending the frequency of their spending you know the time of day whatever it might be we're constantly analyzing and trying to produce new algorithms that can segment the data in the best ways possible so it's sort of an ongoing process for us and is a yeah incredibly important um for me I'll give you two perspectives because I can give you the perspective for my clients of what data they're analyzing and looking at and why it's important to them so my clients who are gaming developers and Publishers they're of course looking at what people are buying what's motivating players to play and end users to participate and what trends are coming in and out of those different divisions several of my clients also do a lot of merchandising

behind their games and so I I'll say Cut the Rope is one of our clients and has a phenomenal International merchandising program and so they also use that data for their ancillary lines of monetization for me I'm looking at what my clients are looking at from data control for primarily the privacy laws that are starting to really impact Gamers and and you know web Industries in general you know what data are they collecting are they allowed to collect like that data in the regions that they're collecting it and are they using that data appropriately or do we need to maybe encourage them to save it a different way put it all together and you know take out the private information parts of it but just to make sure that everybody's compliant with gdpr California Iowa Nevada Korea Australia everybody has their own

new privacy laws and they're all slightly different and to be honest if anybody tells you that you can be compliant with all of them by doing a single thing I would check out that source of information so lovely thank you very much that's a nice introduction that is cool uh so so my next question is um of course this is a panel about how data can be used as a game development tool and of course Bango especially like what you partner with the likes of Microsoft and you have a huge you know game companies uh so my next question is uh so what are some of the effective methods of collecting and analyzing player data and how can this data be used to inform game design decisions yeah so I think everyone is aware that over the last few years it's become incredibly hard to get as much data about the users that you're acquiring the data that you've got on your Platforms in your games so I think people have had to start to get a bit more creative in the way they do these things so I think you know what we're trying to do now is help people to understand as much as

they can about the users that they are acquiring you know everyone knows about their users and what they do within their app or their game but we're trying to help people understand what they are doing across a whole ecosystem so if they are making a payment in Candy Crush for example King might be looking at this and saying okay we know what they're doing within our games and our games only and what we can try and help them to do is say if you look outside of your own games what other games are they playing what other payments are they making uh how frequently you know what genres are they using do they like and that sort of helps them to map out the user profile a bit more outside of their own games so I think that's the biggest area that we're trying to sort of combat those those data changes that have been happening that makes it a lot harder for people to you know understand the users that they've got um I would also I would agree with that I think that's exactly what um our clients come to us okay you're in the industry what do you do um we also help our clients interact with each other and share information when they can um when it makes sense for them the

other side of it is to find out the new technology and stay on the envelope of all the ways to collect data from your end users and from other resources uh you know there are so many people using AI Now to create their own art and their own artworks you know how can you collect that data about what's going on in AI to help Define your game or your next production or your next development project so that it is stimulating and interesting to those people who are playing with the new technology and can they get their new technology and their new artwork or their sound system into the game because anybody who is a user wants to come in and be a participant and we have this thing in the law called your your moral rights and those are things that in a lot of countries around the world you don't get to give up when you create something artistic you always have the right to claim I'm the person who created that and so that that is a huge uh negotiating tool and and very

nice carrot to put in in front of somebody when you're working with Cutting Edge technology and they're doing the new development it's very fun funny you mentioned about AI I do have a question about that well but we'll come back to that later uh so uh yeah thank you for your answers uh so my uh my next question um so I wanted to ask uh how can data be used to improve improve player engagement and retention and what are some of the best practices to do so yeah so I think there's been a big shift in the industry over the last few years of people looking solely on trying to get that immediate roast return you know everyone wants 100 of us as quickly as they possibly can and I think over time you know everyone's spending all of their money on performance campaigns and trying to get as much return instantly and I think naturally that's become a lot harder to do so people are now having to look at retention as more of a okay we need to keep these users for longer periods of time it's great if they are spending on day one or day seven but we're not going to get our Us

return until maybe day 30 day you know some of these Tales of how long it takes to get your money back is really long so increasing those retention numbers is incredibly important and something that I think has got a lot more Focus recently and what we're trying to help our partners to do again is look into the data and say are there certain users within certain genres that do retain for longer are users downloading an app in a certain category like hyper casual is one example where you know it's very much in-app advertising driven the only payments tend to be to remove ads that's just a one-off payment once they've made it they're never going to make a payment in that game again so what we're trying to do is focus on the the more High retention categories like the RPGs the strategies and trying to acquire those users that have those abilities and sort of show characteristics of retaining for longer so when you get those even if you are a casual developer but if you're acquiring a user from a strategy game they have those characteristics of retaining for longer so I think understanding that balance between the two is really important and um is a big

Focus for all of our clients at the moment now that is a challenging balance to to go between those two um what I would add on to that is to really diagnose what it is that you want to get out of the data before you start collecting it um are you spending time collecting data that isn't going to be essential to your root cause of what you're trying to either identify or solve and are you not collecting data that you need to get to that resource and don't just go by the status quo of oh I worked at you know this gaming company and we collected this data and so we must do that really look at your game as an individual if you're passionate about your games if you're passionate about the product that you're putting out you're going to inherently know a lot of the answers but you're still going to have questions about who does this and why aren't the other people doing it and that's really the advantage or the the line of thought that we try to get a lot of our gaming clients to look at is okay you've got

your user base you can find out what you need about those guys fairly easily it's the people who aren't engaging with your game why are they not engaging and what is it that you need to get in front of them to tempt them into the engagement and I think just to add to that I think everyone needs to realize that obviously it's not a one-size-fits-all industry every genre is completely different every Market every country you know uh against like Android versus iOS they all communicate and work completely differently so having sort of tailored strategies for those and understanding what your pain points are within those areas is really important to make sure that you can overcome those problems and sort of put yourself in a position for long-term success brilliant lovely so what are some of the uh ethical considerations and legal considerations when it comes to collecting and utilizing player data in game development actually I think you may yeah so do you want to start this off yeah I took a whole class on this in law school um now I think the the primary things that Gamers need to think about

um first of all is really knowing what data you're collecting from your end users and from any other resources that you're getting whether you're buying lists from data sources from other resources but also understanding what your privacy policy says I mean how many out there have actually read the privacy policy that is attached to the game that you work with or the games that you work with okay so do you actually know what you have contracted with your end users that you're going to do with the data that you're collecting from them so that's the first thing that you need to know is what has my contract which is your privacy policy said to my user that I'm going to be using this data for and then the second is make sure you adhere to that one of the the primary things that I do with a new client with a gaming client or anybody who has a web internet interface is pull up their terms of use in their privacy policy and I'd say

probably about 70 percent have copied them from another website or uh service provider who is in the similar business they do and then just pasted it into their play and then I walk with them and I said do you really do this do you do this and they're like oh no we don't do that or oh well we do it this way instead and I'm like but you're telling your clients in writing that you're going to do it this way and now you've committed fraud now I'm not here to scare you because I'm not that attorney that's my sister who works at Perkins kui um but um you know I'm here to help you not get into that and if you're into that don't listen to people who say you have to change it immediately you've got time just figure out what your business is doing what information are you collecting and are you sharing with the public the accurate and true communication and being transparent with them about what you're doing with your their data and I think to add to that as well I think it's it's quite a difficult industry obviously with all of the changes to understand what data you can use and and the laws are constantly

changing you know I think the amount of conversations we have with our legal team about can we do this can we do that and you know there's always going to be new hurdles that you have to overcome and when it comes to the data you know understanding what you can and can't do when it comes to third-party data which is you know a lot of clients that we work with we have our own first party data which puts us in quite a unique position but there are lots of third-party Brokers out there who are pulling data from multiple different sources so understanding what you can do from you know what they can do what you can do and how you can work together I think is really important that's great uh so uh you know what are some of the challenges outside of the legal and ethical considerations uh in uh utilizing data effectively in game development yeah I think a lot of it comes down to the the Privacy changes I mean it falls within legal but it's slightly different you know people had these really strong identifiers in in idfa for Apple users which they based a lot of their development on you know understanding

what users are doing tracking them and with that overnight basically disappearing you know there's more changes coming cookies are going to be deprecated Google ad IDs are going to be deprecated so I think there's constantly you know privacy problems coming up that people have to overcome and it's about understanding you know where you can get good data from and you know one of the big big things that we try and say is that good data is is better than bad data and no data is better than bad data so it's so easy for people to try and pull different like nuggets of information from different places but is it accurate is it going to give a true representation of the users and you know it's difficult because everyone wants to get as much data as they can you know data is King understanding your users what they can do what you can do with them is so important to everyone especially when it comes to game development because you want to know exactly what's going to resonate with those users what's going to make them retain what's going to make them spend money what's going to keep them engaged and you know the industry seems like it's constantly fighting against that and it wants people to have less less data at once the the end users to have

more of the control which is completely fair enough you know we don't none of us want all of our data being sold to everyone all around the world but it's understanding what you can use and what is the truthful data source for when it comes to development um I you know what I had a different answer until Dale gave his um and I would concur that you know knowing what you can do with the data but also um is the data real because it just reminded me of something I saw in the news yesterday on geekwire that uh there is a scam going around and starting in the Tacoma area of people who are using AI to replicate other people's voices and then calling members of their family and then scamming them out of money saying send me money and it's actually a replicated Voice through AI so um when you mention cookies okay that's an electronic thing you can easily see how that does it but again going back to the AI and I know that's not the topic of our conversation today but um it it you know those opportunities that are

out there and I didn't make that connection until Dale said that um but my my other answer would be the cost um you know there is a cost of collecting and storing data and I see my clients constantly looking at that cost not only the fine Financial cost but the environmental cost you know cloud storage and things like that and and servers um what the impact on the environment and a lot of our clients are very concerned about that and so um that's just sort of the general cost of collecting and using data and I think also you know from a user acquisition standpoint you find we see with lots of our clients and then generally within the industry if you're running advertising on Facebook and you're using interest targeting which is derived from what Facebook defined as as an interest you know so if if we're friends on Facebook and I like something to do with football suddenly you like football you know and and that's the way Facebook defined what an interest is which is a very loose and vague term and you've got some app developers that are running user acquisition based on interest targeting and you know it can

help you gain users but if you're using this as insights for development it's a really inaccurate tool if you're running an interest campaign based on casual games based on strategy games and based on RPG games side by side what you might find is that none of them are very accurate but one of them is going to perform the best if you sort of go gung-ho and say you know the RPG ad set or campaign perform the best so we're going to implement some RPG features into our game you know that's a really bad bit of data to sort of frame your development but there are lots of people that are relying on these platforms so I think it's sort of trying to question question where your data is coming from you know try and be as critical as possible and make sure that you're not sort of putting all of your eggs in one basket when it comes to you know measuring the data and using it for development are there any other uh expert sort of advice to overcome some of those challenges uh you know of course we have plenty of game developers that are you know yeah the best the best you can I know maybe that maybe there's some uh you know expert uh insights that bang I've seen

of course you can't if you can't share too much that's fine yeah and I think you know the bank of audiences business was originally built around providing audiences that could be targeted on social media which we still do and is a huge part of our business but what we know we can do from an anonymized and legal and privacy safe standpoint is help people to understand what what users are doing and because like I said we have first party data so all of the data we have is only from the payments that we've processed so we can tell you what users are doing across an ecosystem it's not based on an algorithm it's not based on modeling to say well we think this group of cohort is similar to this one and they did this and they did that you know it's all based on factual data and I think that's one thing that is it's not easy to get is it is available out there some algorithmic models work really well some don't and I think it's about making sure that you've got a good spread of data and not relying too much on one data source um I I would also add um just a knowledge again going to the the root of the data source of what information is

being collected and what is represented by that data um there are three states right now um here in the U.S that have very severe biometric collection laws um Illinois Texas and Washington um about how you cannot use a data collection that includes biometric measurements without the person's permission and you have to get the individual users permission so if you're doing data collection based on game development and you know some VR maybe maybe applications of games or representations how you want to enhance the artwork and things like that and you want to use biometric data to make those changes you also want to make sure that the data sources didn't include those three states and any other countries around the world have excluded that so that you're not then you know inadvertently breaching um an area of law that those states are very passionate about and are very active about you know making sure that you understand what you can and can't do

there was actually an interesting sort of legal case with Facebook recently with a gambling company in the UK where obviously gambling is in the UK you have to be above 18 to do so and there was a situation where the advertising authorities basically said that Facebook's um the way that they Define or prove that a user's gender age isn't strong enough you know so if you're a gambling developer and you're targeting over 18s on Facebook you know it was basically ruled that it's not an accurate enough measurement and we can't prove that they are actually over 18 you know which sort of opens up a whole new can of worms about understanding you know are we targeting the right users are they actually above 18 are they using their parents cards you know are they using a credit card or something to try and fake their age so you know these are always going to be problems that come up from like a legal standpoint but you know it's sort of interesting to show that you can't always rely on what you think you're targeting or you know the users that you are acquiring moral of the story is be careful so so

um Shirley mentioned about artificial intelligence earlier on this is the time that we're going to dive into that just a little bit of course uh this this event is uh having a focus on AI of course everyone's been talking about it lately and we're going to talk about it now so uh so what are some of the ways that machine learning and artificial intelligence can be used to analyze player data and inform monetization strategies yeah I think there's obviously some really good things about Ai and I think everyone knows that you know in the future there's going to be so many day-to-day things that we do that are going to be completely automated but I think we're in a stage now where it's it's progressing and you know the the rate at which technology progresses now is huge you know so overnight these things can change massively but we're still very early on in what's happening so I think there's going to be companies out there trying to get ahead of the curve and making sure that they've got as many of these new implementations or new strategies involved as early as possible so they can make the most of it but I think there's obviously going to be

questions around how accurate is it going to be working you know are there going to be any legal changes to AI there's a lot of conversation at the minute about trying to restrict what you can use AI for you know chat gbts in the news every single day I think I read something new about it how it's being used in good ways and also how it's being used in bad ways as you mentioned so it's an interesting one and I think there's going to be a lot of developments over the next year or so to find out how it can be used in good ways but I think there's also going to be problems that come with naturally um I agree and I think probably the thing that I would mention first and everybody's aware of it is the opportunity for AI to take one small piece of information and diverge your analysis onto a completely uh different or inappropriate or incorrect assumptive stream and also the you know inclusionary issues having to do with AI and making making generalizations based on automated information uh you know I

that's something that's very easily discussed every day within the industry I think also that it we need to pay attention to the licensing and the intellectual property of what is being used to generate the AI in the first place what is the source of that I've already had a lot of clients who are artists um we represent a lot of artists who are saying hey I've got this stuff on my website and somebody scraped it and used it to create this AI picture of somebody else and it's my picture that's my intellectual property and they don't have a right to use that well the laws haven't caught up with that they're fighting about that right now but what I would advise if you want to do that that's great you know that's the purpose of sharing art is to encourage people and to stimulate them and Inspire them to create their own unique art but if there is a chance what I would recommend is go to the source person and say look we we've got this artwork we created it

we'd like to use it in a game we sort we sort of used yours with some AI processes can we license your original stuff from you so that we can use this derivative work from it and then you're safe and you don't have to worry about anything well that didn't come from a lawyer so actually on in the main Expo area we actually have a booth where you can take a picture there and they can AI yourself up so uh if you have any free time during the conference be sure to check that out thank you there you go from a lawyer so switching back over to a little more General data questions um so what are the best practices of how data can be used to inform user acquisition strategies yeah I think um sort of going with a multi-pronged approach and I've actually got to talk directly after this in the other room to talk a bit more about banggun yeah a brief spoiler here but there's like a multi-pronged approach so I think everyone will always Target

their the obvious categories if you're a casual game developer you'll Target people that are interesting casual games or your competitors so that's sort of the the obvious answer but then what we're trying to do now is understand what they did what are those users doing in a wider ecosystem so are there any really high crossover categories you know of 100 of the payments that we process in casual games which percentage of those are also making payments in strategy games and of those in strategy which are making payments in word games you know and understanding what users are doing across the ecosystem so that's sort of the second approach the third it comes from targeting users based on their payment behaviors so we can give you an audience of people that made a payment but you might be an RPG game and you monetize off really high value payments you know you might want people that are going to be spending rather than one dollar at a time they might be spending 10 50 100 in some cases maybe even more so understanding and layering all of those strategies across your whole user acquisition strategy will really help you to make sure that you're getting the most from the spend that you're putting out there you know we're we've seen over

the last few years is that a lot of people are now relying on automated targeting products I guess it falls within AI of like AAA on Facebook you know the the spend within Tripoli on Facebook is just massively massively increased over the last few years and although it's great in getting you additional scale and low cost the problems that come with that is that you don't really understand anything about those users you're saying to Facebook go and spend my money here's my budget I want I want users from America that's all you know if you acquire a thousand users in that campaign you've got a thousand users that's all you know about them you can you can maybe do some creative analysis and find out what creatives worked well but actually what those users are interested in where they've come from you you know nothing so trying to frame your future development around those users becomes impossible because you don't know what they're interested in and you know that's sort of the the the the cards that are being dealt by the platforms you know costs are increasing to Target for advertising so naturally you're forced to go broader or use automated products and again they do have their place as a

part of your user acquisition strategy but sort of relying on them fully but can become really sort of harmful for future development and I think over the next few years I think people are going to start to realize that because within the last 12 months people spend on AAA has massively increased they're not doing they're not doing much segmentation they're not breaking down their campaigns as much so I think very quickly it's going to catch up with them and they're going to say actually we don't know anything about the users we've acquired in the last 12 months we've just got low cost scale whereas what you need is you know what are they interested in what is going to make them retain what's going to make them spend and I think that's going to be a big change that's going to come up yeah and I think just tying on to that maybe saying it a different way is having a good postmortem after you've done some kind of analysis go back and really look at the data and what did you get out of it did you achieve what you intended to achieve when you went in with your investigative plan and also um did you did you ask the right questions or do you need to modify your questions um I think that Dale you've talked about

you know the the monetization of the games themselves and how to use that monetization data I'll look a little bit more on the creative side of the game development itself and what data are you using to decide what colors to make your characters or what um you know facial construction what music are you going to lay behind it and what dialogue those kinds of things a completely different type of data set collection and what's really available to you in a cost-effective manner do you do it in-house do you Source it out to General licensing collectors and so those are things that you need to think about going into the data collection before but then also just making sure you do a post-mortem after the fact to make sure that um you know the next time you're developing a game maybe you tweak your questions a little bit differently or you go back to the same development phase and and ask different questions and then finally what I see some developers do especially small startup bootstrapper companies is they have an

idea of what the data is supposed to tell them before they go into the research and they're going to pursue the data until they get the results that they expect and not really looking at the data itself and that can be sort of fatal to your company if you're not really looking at what the data is saying and just waiting for the story to to read the way you want it to yeah I think the post-mortem analysis is really important because a lot of people if something doesn't work they're very quick to just say we'll just forget about it you know we'll just put our attention somewhere else we'll focus on on a completely different area whereas what you should be saying is Why didn't it work or questioning why it didn't work was it the creatives was it a feature within the game was it something to do with the way that you've acquired the users or the way it's monetized and you know being critical of yourself is really important and it's very easy to not want to admit that you did something wrong but sometimes you have to if you want to progress a bit further you know it might be a case of you just got it slightly wrong and if you tweak something slightly you'd see massive success so you can't just forget about things that don't necessarily go the way

you wanted them to and it's really important to you know put a bit more emphasis on those different areas and do a bit of a post-mortem okay brilliant so um one thing one thing I wanted to ask was um of course these uh you know big game companies developers like you know the likes of supercell and like King Rovio they you know they all have uh dedicated teams for data analysis uh to inform their own you know growth strategies and parking all that sort of stuff so um so so one thing I wanted to know was um how would you advise Indie developers of course who don't have that budget and you know they don't have multiple teams dedicated to data analysis how would you um advise them that they go about data analysis and to inform their own growth strategies without breaking the bank of course with the cost of living Etc yeah I think it's important to fight like to understand as a business what it is you're trying to find out you know I think some people go in with sort of a bit of a blindfold and thinking we just

want to know as much as we possibly can you know the more data we've got the better which in a lot of cases is true but you can very easily get overwhelmed and you know lose sight of what it is you're trying to achieve so I think really being clear on what it is you'll you want to or you need to know to succeed is it a case of you focusing on the monetization is it a case of you focusing on you know like you were saying like the the design of the game or the colors within the game you know and picking an area that you think is your weak spot and trying to overcome that before trying to do everything at once I think is yeah probably the most important thing to do if you are a sort of a bit more budget constrained and you don't have as much freedom to explore every possible corner of the uh opportunity and I think one of the things that you can do is find unique data sources um you know friends and family that's great when you're bootstrapping go out to colleges who have classes built about the topics that maybe your game is employing and can you have a class project be the the class plays your game for an hour a day

or a half hour a day and gives you feedback or can you go to local organizations like I work with Rain City Rock camp for girls and if you've got a new gaming program that's based on you know writing songs or doing music uh come to our organization let our members and and participants in our community tested out and so there are lots of alternative sort of pool data pool sources that you can use that don't cost you the money that the bit that the big companies can afford to pay really I just realized I've been hogging the questions also all the sessions so but we've got about like seven minutes left so is there any questions from the audience yeah one at the back and then we'll come there what do you think of a tool like game analytics which does provide you um you know a viable solution but in the back end there is a company leveraging your games that are right for advertising targeting and so on and so

forth so it's like a trade-off right you're trading semi like relatively cheap good product but of course there is some strings attached I think if it works for you then it's fine I think you know certain gaming companies are very very secretive about their own data and some are a lot more willing to share things and I think we've noticed that as a business you know we have a certain way of working or that we'd like to work is to understand as much about the way people are using our products as possible but some people just don't want to share that you know they want to utilize the product that we have but they don't want to they don't want us to know anything about it they want to keep it to themselves and they want it to be like their unique information so I think it completely depends on you know what your stance is certain industries or genres of gaming are very very different to each other I think Social Casino as an example is you know highly secretive about what they do and how they get their returns and it's because it's really really competitive market you know it's sort of dominated by a few of the big guys and if you can get something that's going to give you the Competitive Edge then you know that

you're going to do that we notice this a lot when it comes down to case studies you know as a business we always want case studies and when we're going to people saying you know you've told us you've seen great results here let's let's talk about it quickly the immediate answer is no no we don't want anyone to know how we've done this how we've got here so I think it completely depends on your own stance of you know what you're trying to achieve and what's important to you at that time um and I would say um to add on to that because that's exactly exactly right is have a set standard don't let that decision um what you're basing your decision on what's important to you fluctuate from circumstance to circumstance and the best way to do that is have a Mission Vision Values for your company you know what is your mission statement what is your vision for your company and what are the values that the company stands for and if you take that source of information and bump it up against the question that you're asking you know nine times out of ten the answer is going to be really obvious any more questions

hi um I wanted to ask a question specifically around timelines for getting data um like for instance my company when we put out new features they kind of expect us to action on things within a couple of weeks but you know what is a kind of a timeline for data to come in that makes sense for a feature to be able to have enough data to action on just kind of curious on your thoughts around that yeah I think you'll always find pressures from above you know the people at the top want to know what's going to happen straight away you know has this had a direct impact is this going to help us and I think a lot of it comes down to sort of managing upwards like managing those expectations and saying you know we could make a sort of rash decision and say you know it's been a week we've got this data let's make a big change but you might find that over time whether it be two weeks a month or three months you know depending on your game and the style of game that it's not beneficial to make a really quick decision or a quick change you know the more data you can get and you know it's about setting what you're expecting it

comes back to setting the expectation before you run these tests what is it that you're trying to achieve if it is trying to achieve a longer string of monetization or people making payments over a longer period of time you're not going to get that in a week or two you know you're going to need months to be able to measure that if it's about instant retention then maybe you can understand like day one retention maybe a quick change can be done so it depends what you're trying to achieve but I think it's definitely about managing expectations upwards and I think that's not always easy to do but it's yeah really important yeah absolutely setting those expectations and understanding what data you're trying to investigate because um yeah a price cut can give you an immediate response and you'll have that data the next day how many people bought or didn't buy um but you add a new character to a game you're not going to know for months whether people like that character or not or responded to it so it really depends on what you're looking at so there was a one question at the front here

yeah hi uh you mentioned the the social casinos but I'm wondering if you've done anything with land-based games and and that's an even more restrictive space but um is there is there like ways to get data from like player player cards or loyalty points or other things that that can uh analyze sort of um the land-based games regulated spaces yeah I think it's obviously a bit harder to measure when it comes to Land Based um but then there are lots of companies that sort of uh diverge between the two so one of the areas that we work within as a business is Casino Cruise companies you know they have land-based casinos on their cruises and they're saying to us we really want to understand you know find the relevant users in the first place then also being able to track and measure what they're doing once they come on board a cruise are they in the casino how much are they spending and they've got some quite interesting ways that they track this you know based on the person that makes a booking they get a certain card that they use within those casinos and they can measure how much they're spending um but it does obviously become a bit harder measuring the footfall from

digital advertising to going into land-based casinos it's not always as easy to marry those two things together and say we know that we've got this return from this or this user saw this advert and went here so I think it can be done but it just depends on on how you're you're measuring it and tracking it internally if you have the the means to track it then there's no reason why you can't yeah and I think that goes also into the type of gaming that is going on where you can publicly see the results of a lot of sports gambling and sports gaming um things that are got you know you see the results and who had the first pitch and you know where the quarter fell but um you know when with card games and other maybe um you know player based games you're not going to have that cumulative data of the results of who won how much they were gambling how much they won so I think you also have to think about segmenting what kind of gaming you're asking about brilliant I think that's uh pretty much

time so uh can we give a round of applause for our lovely panelists [Applause] [Music] [Applause] [Music]