Game Analytics 01 - Building

Allison Bus. So you started your career as a game analyst and now you're chief operating officer of game analytics. So I'm really curious about like how has the role of game analyst changed since you started.
The function has become more specialized. You know at the beginning it was Swiss Army knife. We were covering everything from you know marketing to live ops and then it's just become kind of different buckets of how data is being used. The role of the analyst has just maybe become more specific for different teams.
Do you feel analysts are now more respected or is the role just more complex?
Yeah, that more integrated I would guess respected for sure like kind of part of the process for for many studios especially as they get more sophisticated and and have greater need. They work much more closely with the teams than they were in the beginning. At first, it was kind of like, "Hi, I'm here and I can [laughter] help, but now it's it's very much even the teams

looking for that support, you know, as soon as they can get somebody specialized, they're they're into it."
Yeah. You mentioned sophistication and so there's two terms. There's data sophistication and data maturity and we hear a lot about those two. So can you define the difference between the two and does it matter for a studio that is just setting up their analytics?
Yeah, data maturity is all about the infrastructure uh that you have to handle data uh all in all. So that's the maturity factor. Uh sophistication is more around moving from a reactive use of data to a proactive use of data. So that's how I see the the terms being different.
And early on does it matter? Do you need to understand between the two? Yeah, I think you need both actually. Like you need to set up your infrastructure of course, but then be considering how you're going to be using it overall. I think I have definitely seen studios that are very mature in their data setup. They're tracking everything, but they're not necessarily using it in

their decision-m processes at all. I've been there for sure in studios, and it's frustrating to kind of say we have all of this that we're sitting on. It's a gold mine to be able to uh facilitate decisions. One interesting thing about data sophistication, I think, is that it kind of doesn't matter the size of the studio per se. A lot of the very small teams are quite sophisticated and how they want to use data and integrate it into their decision-m like who is our ideal profile using game analytics. It's kind of everyone in that [laughter] sense. Uh and in in terms of sophistication, it it can come uh with any size of team. So talking about team sizes, what is the uh the difference in how they approach analytics between let's say you have an indie studio of 10 maybe 15 people to a midsize studio of 200 300 people.
Yeah. I mean small teams it's usually a generalist working with the data. You know it maybe not even a data analyst or data person and so you typically have just a small team. It's about resources

and and what's available. indies are having to move super fast and maybe focus on more high level. Whereas once you get into like a big enterprise setup, you break into specific functions in terms of like infrastructure, what you're tracking. It's also about completeness like making sure that you have all of the data sources available to you so that you can use them for your analysis and for your work.
And you talked about different sides of studios and game analytics works with thousands of different size studios. Do you see any patterns in how the different size studios evolve their data stack from going from that small indie studio to a midsize and maybe even a large one?
Yeah, the small studios start with maybe your highle metrics, dashboards, um kind of those KPIs you need for for publishing and getting your game out the door. I would say at the very beginning it's kind of about your gameplay signals to support those highle metrics of engagement and monetization. Then in

terms of your data stack and in terms of the analysis you're doing, it evolves into more of your AB testing, your segmentation and your personalization, which I know we'll talk about, but it also really has to do with game success. So oftent times you start really early with the high level stuff and then once their game has success, they really go deep into utilizing the data to operate their game.
Yeah. Uh so before we go to the uh highest level of of data usage, what are the bare minimums? Like when you're launching your game and you're a small studio, what are the things you're tracking day one?
There's the highle metrics that I think everyone knows around engagement monetization metrics as well. Um maybe if you're doing ad monetization, those types of things. So very high level. I think what's important in that is to know why something is happening with your retention. So tracking more than just your sessions and your users, but these

gameplay signals as well. Game analytics does have a a very cool feature around our engagement matrix where you can see for any particular component of your game, how many users are engaging with that and then how often they engage with that as well. So that's very cool to see if you have a feature that has high usage but they don't do it very much kind of what's going on there or or the opposite
maps.
Yeah, exactly. You can kind of find your your core features and the core items within your game that people are engaging with and kind of double down on that.
Are they teams that underdo their early metrics or not metrics but analytics? Yeah, I mean there's the analysis paralysis of course, so you can kind of overdo it in that sense where you kind of get paralyzed around it. I think kind of pairing data with some decision making processes is really important for teams so that they don't go too deep. Just focus on the decisions you're trying to make from the early part of what am I going to track? Well, what

questions do you have about your game and what data do you need to answer those questions? H we talked about the uh the tech stack. Let's talk a little bit about the people. Do you think studios need both analyst and a scientist and what's the difference between the two? Is it the same job, different title?
It depends again on the size and capacity of the team. Um it's great if you can get someone to do both. How I see an analyst is that they are somebody who helps facilitate decision-m with game teams through the data that they have. A data scientist on the other hand is more about your kind of processes and prediction models and and things like that. They're a different kind of function less around decision-m and more around automation maybe
you know when I started at PopCap we had both an analyst and a data scientist and we worked together really well in terms of you know facilitating the game teams. Yeah, I've just of my own perspective. So, correct me if I'm wrong. When I worked with analysts, they helped to

analyze the data to be honest. Just just something that I already knew how to do, but they would do it faster and I was able to concentrate on other things and they would kind of dig deeper, but then I would have to make the decision based on the data. And then our work with data scientists, they looked at the data in a totally different way
and they brought insights that sometimes weren't actionable and sometimes really changed fundamentally on how we thought about player progression and other elements in the game. So they kind of were going on their own journeys and coming back with information versus analysts were sent on a specific mission that they came back and
they're helping you make a specific decision versus maybe more holistically um looking at patterns and anomalies and things like that. Yeah, true.
A lot of the talk about AI is around analysts of course.
So with AI does it make it easier to get the insights or is it just more noise because you're getting more insights? I think what AI can do is help with that anomaly detection. It can help analyze

as you said you would go to somebody and and ask the question. What's really important in that is that your data is structured and kind of ready to be read and understood by an AI and that's proving to be difficult quite honestly. [laughter] So somebody has to in still kind of interpret the data, feed that into the AI um and then ask the right questions around it. I think really great analysts, by the way, are able to like build context around what they're analyzing. AI is a good like co-pilot as opposed to being the pilot, you know, in the seat themselves.
So, you're not seeing the world where AI replaces game analysts or game scientists.
Yeah. It could make them faster, more efficient. Maybe teams can go directly to an AI and not not need that person who's in charge of bringing the insights over, but as you said, somebody's always in charge of making that decision. Kind of providing here's your options in terms of some of those decisions and here's how I think they could impact the

rest of your KPIs or your performance at the game.
Yeah. Just turning data is not actually making a decision.
Yeah. Yeah. Yeah.
Talking about different strategies and the evolution of strategies. When I started, which seems like 200 years ago, we did AB testing and that was like the cutting edge of everything and then it slowly evolved into segmentation and now companies are talking about personalization.
So, what are these things and what's driving that evolution?
Yeah, the game industry is in a interesting place right now. I've also been around a long time. And I think you've seen this evolution of at the beginning when I was at PopCap and and beyond it was kind of it almost feels like we're back to some of those roots like really trying to understand which groups of players you know like different aspects of the game. That's what I was doing early on anyway and then
and then things kind of changed with the hyper casual revolution and then we're kind of back to some of those basics with these things around AB testing segmentation. Um, I'm excited about the

personalization being much more automated.
AB testing gives you like a yes no answer. Segmentation then gives you some more nuance in terms of these groups of players that are doing uh, you know, taking certain action. And then your personalization comes in having different treatments on those segments. So, it's one thing to kind of analyze different segments of and groups of players and know who's engaging or who's monetizing. But then the very important part is closing that loop and taking some action on it. And that's all around the personalization part.
Can I test a thesis of this one?
Yeah.
So I have this very dumb down view which is ice cream sales. And basically if you have one kiosk and you're selling strawberry ice cream, chocolate, and vanilla, AB testing would work in a way that I would sell either one of them in different places. And because I would assume vanilla is the most popular. I would turn all my kiosks to sell only

vanilla.
Yeah.
And when I see the numbers, it would actually not be that big of an uplift because actually 60% prefer vanilla and the 40% don't want any of that. Yeah. And and so what happens with segmentation personalization is
especially with personalization, you increase more different ice cream tastes and you're able to serve everybody the type of taste they want rather than narrowing it down to what is the most popular and cutting everything else away.
Yeah. Yeah. I think that's clever. You know, you could again kind of find those groups of players but then taking action on it and there's more functionality that allows us to do it. I think it's been a lot of manual work for game teams. If you look at live ops systems right now, it's a lot of manual work.
100% with with AB testing was a lot with segmentation. It felt good in the beginning because we would put people in different segments by analyzing how they perform and then based on the decision they would go towards that segment and be treated certain ways. But then
as games get older, you start creating segments and suddenly you have 600

segments and the player base is changing and that becomes a circuit board. And so that now we're kind of entering this personalization where AI is helping to basically optimize the game for wide variety of user base that is constantly changing.
Yeah. Yeah. That's wild to think about when the mechanics of games are the same in a way, right?
Difficulty playing patterns.
Uh even in a simple looking puzzle game, it's like how do you progress through it? And it's not all puzzle companies that do it. There are those who focus on personalization and there are those who are equally successful that focus on just perfecting the one approach and they are winning as well. So
there's no one right way but you have to be good in whatever you choose.
Yeah for sure
another interesting part is when it comes to analytics or it comes to any third party service is whether you do it inhouse or you take that third party service. Uh what's the mental model for that type of a decision making? I'm obviously very biased on this question,

by the way. Um, and I've been part of game teams that have, you know, built data pipelines from scratch. Um, one in particular, we spend a year redoing their data stack and it took us a year before we had a DAU dashboard. So, it's very complex work building up a data pipeline. I don't recommend that really. I unless you have the time and resources to go all in on something like that. Studios who choose game analytics often they are moving fast
maybe the system that they have built if they're starting to have some success maybe it's super expensive to run I mean our specialization is to drive down costs of a massive massive data stack that we have it's like 400 million daily active users that we're piping through game analytics and we do that at scale at a very low cost but it's very personal as well to to studios what they're choosing Well, when we talk to studios, it's like the mental model for this type of decision-m is

studios tend to think about only building the product. They don't think about maintaining and evolving it, which takes a lot of work
over time.
The cost advantages that they might be
projecting is perhaps not there because usually third party services provide that service to tens, hundreds if not thousand studios. So they're able to push down the cost compared to your studio that would be providing this service only to one client. So those are the type of mental models but usually people don't think further than building.
Sure.
You don't think the evolution.
Yeah. And your game starts small anyway, right? So like you have you're not tracking that much and it's easy to to set up and get going, but it is maybe more about thinking about the long term for your game having success and scale and what happens then essentially. I'm of course pro third party services but if you're a big massive studio then perhaps you can build it for your large portfolio.
How our perspective on that as well by the way at game analytics is just that [clears throat] we we've tried to build our product in such a way that studios

can evolve with us but then we also understand that we don't track everything and so we've kind of planned to become part of a larger data stack.
We have a product called pipeline IQ that is data infrastructure. It's a data warehouse event export. So once you're at that stage, then you don't have to go and kind of rip out everything that you've already built maybe with game analytics. It can be additive to the rest of your data stack.
So supports the studios that kind of go into that higher
when they get there when they get exactly when they have a large portfolio. Makes total sense. Very customer focused. Uh well talking about some of the pitfalls, what are the sort of a common pitfalls when studios are setting up their analytics stack? Is it more around the tooling, the tracking or even the uh the team culture?
There's pitfalls in in tracking like too much tracking um creates a ton of noise. We see that a lot.
Yeah,
there's something about team as well. We

were talking about decision- making and and how that works. So, just kind of considering
not only what are you going to track, but how are you going to integrate this decision- making into your team?
Correct. Uh actually, what do you think about this? So in some of the studios that I worked with, they've gone between datadriven and data. And the datadriven studios were the ones that were aggregating data from various sources. So they could be community data. It's definitely game analytics that they're getting, but it's also data that they have through their gut feeling and experience that that drove their decision-m.
And then there were studios that were large and they collected all possible data and they were very much data. They they wouldn't make any decision until the analytics of this AB test showed something.
And interestingly enough, the data companies both were successful in their own terms,
but the data companies were a little bit slower and required more manpower
and every time they got these results

from AB tests once they would put it back to the general audience, the uplift wasn't quite there as they projected.
Yeah, I'm pretty flexy on that. like uh I think that the decision- making and how you do that um I think it's it the an Amazon thing where is is it a one-way decision where if I make this decision there's no going back and there are large consequences or is it two-way like I can make this decision and then turn around and and it's okay. So if it's a larger decision that you think is going to break something have major consequences you should apply more data and insights to that. But if it's something smaller, a lot of AB tests are very small decisions. Yes. You know, where you could just make the change in your game and see how it
5% of the audience, new users get this. If it's good, it's good. If it's bad, it's bad. Doesn't matter.
Yeah. Yeah. And maybe you could revert or change um something that you've put out that is having an impact. And and so do you feel that that separates those companies that are the best in the

business when it comes to analytics is that they are contextual in how much they lean on data?
Yeah, I think the the context is super important. I think it's about accelerating decision making and kind of using it as that. Game analytics as an example, we don't make games, but it's kind of the same idea and kind of eat our own
dog food.
Dog food. I was going to say soup. [laughter] Uh but in terms of like using data to run our business, it's the same kind of thing. And and we have a lot of intuition around what people need with analytics, we have a lot of conversations with our studios. So in terms of our product roadmap, you know, a lot of it's driven by intuition, but we also try to apply that data to it. There's a speed that we need to keep up on as well. So balancing all of those factors is is important. And yeah, I see the successful studios doing that. Finding balance with all of these things.
That's the hardest part is finding the balance, knowing what what's important, what's not, especially as you're getting access to so much data.

Yeah. Yeah. For sure.
So, if you could give one advice to product lead or a founder who are setting up their analytics today, what would that be?
Get started quickly with a third party tool. like focus instead your energy on uh making your game the best that it can be and using that creative energy to to build the best game that you can as opposed to putting that energy into uh getting your data in order. Obviously, Game Analytics um as a a partner to help you get started with data, and that can evolve over time as their games have success. But we're here to help studios get started quickly to get the insights that they need so that they can have the success with their games that they're building.
Makes total sense.
Yeah.
Thank you, Alison.
Yeah. Thanks, [music]