Analytics from a Live Ops Perspective | PGC Digital #1

so I was asked to talk about analytics kind of taking the life of perspective on this and I think I have a lot of things for his team in it so let's try to see how it goes so a bit about me and my gonna change the slides here no I got it yeah yeah so my gaming industry experience is about five years long so I started at the a Helsinki working with SimCity build-it and now I've been close to two years now delete data analyze the small giant games having worked with with of course empires and puzzles as well as puzzle compact which is our soft large title I was pretty lucky at during my time at EA I also got to collaborate with a lot of great folks from PopCap fire monkey monkeys and all those great DEA teams but really I think like the reason why why I might be talking here is for

example so I'm working with small giant games and one thing that happened last year which we won the Pocket Gamer best LiveOps Award for 2020 and this is this is actually interesting in the sense that we don't really have a dedicated live ops team most small Giants work with with live ops in one way or another and even the analytics team is tiny compared to the most other Studios but but I'm in that sense we do some things actually quite differently from others but hopefully you're gonna kind of get some ideas from this as well so because everyone is working with live ops everyone also has access to our homegrown reporting and analytics tool set and really the main use case for for data and analytics for us is to have really create this data in from discussions around design marketing and Finance I think we're quite different from many other free-to-play studios and

that we actually don't do a lot of things a lot of a/b testing we don't really do segmented targeting on the game side like obviously user acquisition is it's a different beast but coming into these conferences I see a lot of talks about analytics focusing at the analytics talks focus focus on topics like a be testing advanced a/b testing methodologies segmentation and so on however I think there there are some kind of foundational topics that are talk less about and yet they are important for live ups games and really helped to create the ideas on what should we improve in this in the first place so what is live ups I guess in terms of this talk I'm just talking about keeping the players engaged by continuously adding new content new features running events running offers and really just iterating and improving on the on the existing gameplay I tried to divide to talk in the end kind of

three quick parts so first of all what does the game team need from analytics when it comes to live ops looking into the game more specifically how is it working and then kind of going into really to the players and what is it in the game that they value so let's start with the first part what does the game they need from analytics I think there's two kind of the most important parts or data that is timely and especially correct and really what can you do when you're releasing these features out maybe every month and of course before you go live you don't really have the data so what do you do in that case the first one is data that is correct and and it's mainly about SEO reporting right reliable so let's say you like doing a live ops you move so fast you

can have errors in the tracking or the data collection you can have errors in processing you can especially have players who have hacked the clients and sending this all really really weird data about the game economy and stuff and really what you do not want to do is having your marketing and Finance teams making decisions based on faulty data and also of course as an analyst and especially for our speaker so we don't really have a lot of folks working with with analytics the more you spend on analyzing anomalies in the data the last time you're actually providing valuable insights for the team and not to mention building any kind of automation or prediction modeling on top of all the data is just a disaster waiting to happen so getting the foundations of data quality writing is extremely important and we should discuss more about that so one way we deal with this small giant games is that we actually have most of

the tracking it comes from the server so these way we're less prone through for example the hack clients and the Foley data that come from from those players but I think another important part with with the server-side tracking is that let's say you're running a lot of game and you you basically have an issue for example the players can't I don't know login or whatnot but for some reason you're not tracking the correct part of the game that could help you inform the decision on what should we do about this and so instead of doing the doing the client update dance which kind of can take days in the worst case it's usually a lot faster to change the tracking or add tracking on the server side but that that really is like what to do when you're alive with the feature and you can actually collect data what do you do when a feature is not out yet so then there's probably a lot of tons of things you can do about the one thing

I'm more familiar with this is simulating simulating data for designers people in the team so for example you can think about simulating long term reward distributions in the game and kind of once you have that data it can act as a benchmark against live data once you collide once you go live with the feature and verify that everything is acting as kind of planned as designed basically and I think it's really important to build some kind of simulations las testing framework for any random game components already during the soft launch of the game and and I guess like the more randomness you have in the game the more the players will actually have doubts so it really helps to have multiple checks in place that everything is working and it's important to give the players trust on this of course simulation can also be also be used for things like progression analysis estimating the difficulty and so on but again kind of trying to trying

to loot take the lives perspective here I think it's important that you do not feel the simulations in a vacuum you need to take into account that there are levels components like offers in the game that will have significant impact on on progression for example and I personally made this error and try to learn from that so then going forward looking into like how is the game working and of course you like the most important part parts are the economy and how is it working like how is the other currencies are you seeing inflation how many players have reached the end of content or your crafting components coming up as you're supposed to or other inventories that are piling up and of course the changes that you're making you're introducing new features offers and balance change things balance changes how do all this affect player currency wallets for example spending

habits and so on and I really think like the important part here is is having of having focus or creating focus so do you probably all know the the Pareto principle but but it's not really wrong to say that like in the in high level you can think like eighty percent of hard currency spent goes into twenty percent of things eighty percent of complaints where complaints come come from 20% of game features and really the key is is finding the focus areas finding the levers that will make a difference when when you improve and change them during that kind of climb up space another thing is is feedback and I have an example here from our game obviously we're running a puzzle RPG we have heroes they go to battle they have been raised and the example here is that we basically had multiple heroes that we changed at the same time and you can see here the other buff

actually made a good impact on on doing red of that particle hero whereas the other one didn't change at all so these kinds of things can be a really good feedback to the designers to better understand what are the levers that will actually create impact that the players will know this and then what are a kind of changes that will not create such a big impact so this is another key point for live ups it's a better understanding of the impact of the changes that you make allows the team to do more with with less iterations so then going on what do the players value for example this can be topics like what content presents with players what offers are working or are not working can you spot specific content or price points that might be problematic and it's especially important to contrast this this actual player behavior data to the community sentiment that you're seeing for example on Facebook or your forums or wherever

you're having that those discussions so trying to come up with some examples again we're running possible RPG you have heroes and for us the heroes have these special skills that range in speed from very slow to very fast so I had this graph where basically you we're just loading beam rate versus hero use rate and the red dots here are the kind of the faster heroes and the windows are slower heroes and you can see kind of looking at the trend you can see that overall players have a pretty good grasp of of hero performance but they they have a tendency to undervalue particular slower heroes kind of you can see there's a group of heroes there that are actually in this in this game mode have basically the best win rates in the game but they're not really used all that much so creating these kinds of insights

can help the designers create content that players will will value more and of course be willing to buy that but also I think this kind of data is very valuable for for example come the management and we've seen that it really helps them engage and excite the players and have discussions around around this kind of data the one more example I have is is against on price point bias so one thing we found for example was that the 490 and price point was performing performing significantly actually I think this was like the $3.00 price point just locally it's 490 and it was performing significantly worse in Japan globally I'm kind of looking into this I noticed that that for and actually known as well are both considered kind of unlucky numbers in Japan and so this particular price point combined stock and of both which then reduces apparently its desirability no

matter what the actual quantity it offers is and really understanding player preferences player biases are these kind of you can use to some as tactical tools that you can leverage in the design again to create content that no first of the players will value more and and in some sense one of the aims for live webs is that while you when you have the correct focus and you understand player behavior and you understand the impact that your changes will make that will really help you and even if you're running for example AP that's it becomes less important what kind of a be that methodology you use for the analysis because you can you should basically see the impact coming mile away if you have this correct so going into the summary I kind of touched on three points so first of all data data quality once you have that correct you're definitely going to spend less

time analyzing invalid data and really the main point is maintaining trust for the game team when they go and see the reports and make decisions based on the numbers they know that they can trust that that everything is working as suspected and for us one of the one of the key things that we're doing is using server validated data to on this point another part was controlling the economy and balance and really the key points are finding finding the areas on where you should focus the effort giving feedback to designers and really having like giving the the game team opportunities to create more impact with with less iterations and then understanding player bias is understanding the preferences and really just creating content that the players will value and allow you the price price that content according to their preferences and the final point I

hope to make is that even though I've discussed this as a separate topics optimizing on on looking at specific metrics on the economy or wherever it's really important to have kind of a holistic look on the numbers when you try to optimize optimize the numbers individually because a lot of the time the KPIs are kind of pulling rope so you get you can kind of easily move one thing up but then move move another thing down and really it's it's it's extremely important to take a holistic look when you when you try to optimize optimize these things that's really all I had hopefully just 15 minutes so any questions I would be happy to to answer that was perfect Hana me I'm very impressed actually I think you know what you try to get across here I think the idea of actually thinking about how you look under the data to find the causes of whatever you're trying to investigate

that that for 90 yen thing yeah I knew about four I kind of think I remember of nine being an issue but it wouldn't have occurred to me to think about Japanese pricing at that point being a problem and fact that you kind of like and look under the dating kind of get beyond it I think that's your choir kind of cool thing and we have a bunch of questions for you you'll be glad to hear and this one is from psy premies I'm gonna probably be terrible the chastise for pronouncing that wrong but hoped I didn't do too and how do you identify events to send to Google or Facebook for better you a for better you a yes yeah that's what that one well as always like coming from the analyst the first thing you do is actually you talk to the UI team and try to figure out what did they already know

leverage the understanding that that they definitely have on their topic on you a and then here build on top of that and kind of interview them what are things that we're not sending it what are things that we can try and then kind of finally like after you've exhausted all that knowledge then you go out or your own and try to build methods on how do we how do we expand the things that we're not sending it and then you Diane is saying do you build your own maybe testing tools well do you use a third-party tool we have our own tools you know so why use your own rather than something external in that case I think historically we found that like there's a lot of good tools out there but for our particular use cases we always need something extra and over time it has

been easier just to build one that our one that we basically know that works for us and of course also in some sense also gives us a bit of competitive advance yeah I mean so things I've been through investigating is configurations within the backend effectively deliver an a/b test without having to go through a lot of the processes if your data model is robust enough that theoretically should help but suppose at all it's not much more than that but anyway it's just different mindsets isn't it so next up how close does your team work with you a team are they separate you know sodium analytics guys over there yeah because over there or are they kind of more connected organizationally they are different and of course like but the dingiest the UFOs are constantly looking in the data so it's not like it's on us definitely like the US you a team is very very very

much on top of the data and a lot in that sense we are mostly providing tools of course if needed we can definitely pitch in and help and look at kind of on a more granular level when the tools are not kind of its kind of obviously it's important impossible to have everything on the reporting stack so once we go we can basically we can definitely do a deep dive but but really UN team is also very knowledgeable Steve do it differently I mean I kind of like because I've traditionally been a product owner I've had responsibilities both in terms of user base and also you know kind of gameplay experience or at least experience ignorant of any on the project and so it was quite good having an having the analytics folks working directly with me but I've seen other teams that work very effectively by having you know a separate team that they've called on to answer specific questions and I think that's interesting is because if you're being asked to set

up questions from a game design team they're coming from a different angle than the guys who are asking from you a point of view and they may end up needing some some similar insight and you kind of need both mindsets to be able to uncover the depth of what the real problem is and what the real opportunity is that may be hidden in the numbers feeling cool thinking about and how do you suggest to do initial design level of difficulty curve and to improve the progression in LiveOps Brigade that is how do you suggest to do the initial design of level difficulty curve and how do to improve it I heard you try to improve the progression in live ops I think we're trying to say is basically you know wedgie star in terms of working out your level design difficulty and how do you deal with that as the lifecycle

of the game continues and you need to have the progression of a player over an extended with the Benz punctuating the activity yeah well I mean we've obviously we obviously have designers who would be way more capable to understand sir these questions but I think I guess like two things that that we we tried to do is first of all having the difficulty set up so that it's easily tunable by bias basically but as little manual changes as possible so if possible you could basically doing it just by changing one one single parameter then again of course the the really the first pass that you should do is just like play the hell out of it see how it feels and then go from there and I think that for me is what I was traditionally in my way which is basically I make a gut feel of it I mock it up and then I I play it through either myself with lots of other uses

and and and slowly but surely increasing them reviews they give access to but recently I've got more into the whole use of Monte Carlo simulations which I still don't quite understand but I have a guy who does and so that has made a lot of difference because it allows you to see where the way you've got your positive or negative feedback loops in the in the in the process they can disproportionately send resources or power-ups or whatever it may also you in the rewards that come through play yeah unbalanced the difficulty curve dramatically and you may not notice that when you're doing it manually yeah though to me yeah yeah yeah I mean obviously like just having a simulation on top of it will give you basically the reins of outcomes where as kind of once it when you play it by yourself you're kind of having this a very particular segment of players that are very knowledgeable about how they will proceed what is the range of outcomes but the simulation will kind of give you

a wider picture about who she is exactly and then moving on from that slice enough do you use machine learning and if so for what purpose I hated their to say that we used mashing learning because we do very simple we actually like try to keep models as simple as possible but having said that I think they work extremely well for our case so we kind of managed to find a nice a point where it's kind of just simple enough but still works for us and mainly we we basically do in well of course like I think our main use cases is user acquisition for for prediction models yeah I was very lucky because then I knew Dennis deepmind before they were being bought I was doing some work with them on a game which never got released because they got bought and other things became a party it was really eye-opening the guys

that they had working thinking about how machine learning can affect game design you know really just fascinating and the fact that we can like save machine learning is so different ways of looking at it you know AI so we didn't always looking at it they're really just coding solutions to problems and so you know whether it's a simple you know solution because it's manageable or whether you're creating a whole entity that can play the game without instruction that's a different feat I think most of us I'm going to be able to do the latter but maybe some of us can use the former to create you know exercises and really test our own assumptions yeah yeah definitely how do you track sources and sinks in in-game currency do you use your own or an industry standard tool for that and will you also be prepared to share the analytics tools that you use so again we're basically running mmm

pretty much everything on on custom analytics and the custom collection yeah yeah so no real nor we're basically not using a lot of third-party tools out there and again we're basically collecting pretty much what what the the data that the game uses on the server side so we try to keep again the collection we start with the simple things collecting what they're what what the game has and then try to enrich it if we find out that we need some else so presumably you do things like you have an event trigger an event trigger delivers you know something value pair combinations that define what's happening that you may have some reference data so you know to post that every time but you can produce a picture platic heat map staff or other framework type stuff I mean I say this out loud because I think it's quite often that people don't talk about the detail of that I have a particular interest in a

method called hasip hazard analysis and critical control points which is a method that Pillsbury and NASA developed to make sure they didn't poison the apollo astronauts because you can't break a package up you know you're gonna make sure that the package you send up to the you know in the Apollo 11 you know was going to be open about and was going to be safe so how do you make sure the thing is safe because you can't test it so what they did is they look at what's what the expected behavior is what acceptable was and they look for variations on that throughout the whole process and I think if you apply that mindset to games it allows you to really think about what do you expect behavior to be in your game what are the ways we can measure that effectively and then use that to create patterns and funnels that allow us to see what's the best way to improve the experience of the game in general yeah definitely a creating like

so many things can be thought of as funnels and they are extremely like it is actually surprisingly easy to like if you have a problem you will spot that in a funnel so like just creating analytics funnels or even funnels for anything that you can can be very very helpful it's just a shorthand way of getting people with not data-literate to understand what's going on pretty quickly I think yeah and I mean pointing like this this is where the problem is yeah exactly exactly or at least where the problem is exhibiting is that one problem hand I had a game where people were founding a particular level but it was actually like previous two levels where they'd missed something and I work I could see it in the funnel where where it was happening they were failing and dropping out the game but it took quite a lot of digging to work out what it was having that problem yeah yeah that's good yeah so next up do you a be test price points

for our Peas not really like we actually like on on price points we we will try to so the same offers for all the players yeah what why wouldn't you tape it I've got the thoughts but I'd love to know what you think in terms of why do you not just side to do them mainly because we have the games PvP for Enda and we have a very active community that will just raise hell if they notice that we would maybe does these things so I think it's way more helpful just to keep things straight for the players and kind of show everything do all all buyers but isn't there an opportunity particularly had save consumables in the game where you can offer price value bundles on limited offers that would not necessarily break that and I can

completely get when you're talking about kind of character shifting stuff that's a different beast and particularly you want to make sure you got a level playing field but you know potentially couldn't you read deal with that with the yellow star rating of the player versus other players maybe but again like another way just to do it it's basically having variety in the offers themselves are just letting players to us what they want yeah exactly all right the problem with my logical argument is that like you said earlier there's actually an emotional response from a lot of players to the very idea of what is perceived as paid play to pay to win even if it's not necessarily real the perception of it is yes quite daunting should we say so next up we have so many questions I hope you are okay with this and yeah I think well I think we'll have to make this the last one because they've run out of time what are

the important high-level KPIs that you focus on to improve light ops for us it is mainly well the first of course is retention and like especially for RPGs the one thing that you want to have is that long tail of retention and that really has been adding a significant driver for the success of our game then other thing as conversion in the sense that like if you can well of course first of all if you have players that are willing to spend some money on the game they are more likely to stay as well but then there's my god I hope I discussed this that there's kind of this rope pulling between the numbers sometimes like you're gonna you can basically increase conversion but you might decrease your poopoo but but if you're getting the same amount of money

from a wider use space I think it will always create a healthier game in terms of long-term monetization long term sustainability for the game so I like the it's really it's not that that it's a basic stuff and the body depends on the game and this already I think that's the answer to always me it's like it's fine I think we had another question as well asking about is there a minimum viable analytics I mean it's it's basically finding what is the most effective thing that you can can measure to set yourself goals and then over time increase the complexity because in the early days we may not have critical mass in terms of the data significance so yeah we have to just do the best we can until we can get the point where we can actually ask more detailed refined questions yeah and that's that's again because like retention and conversion ask a binomial metrics have way less

variation than other kind of are poopoo and this kind of continuous numbers absolutely I'm saying I know that note thank you so much that was great thank you