Session: Game Data for All: How AI Is Reinventing Game Analytics - Orly Shoavi, ClarityQ.

[music]
So, in that case, let's move on. We've got a a talking owl. Game data for AI, how AI is reinventing game analytics. We've got Ollie from Clarity Queue. Come on up and introduce yourself.
So, hi everyone. Let me start. Thank you for being here. My name is Ollie Shuali. I'm the co-founder and CEO of Clarity Queue and we provide an end-to-end AI platform for game analytics. I previously co-founded a company that was acquired by AppLovin and then I was at AppLovin for 3 and 1/2 years. So, the gaming industry is super familiar very familiar with the gaming industry. Um, and now I'll be focusing on

on on in-depth examination of every game analytics solution, the must-haves in every each and every such solution, and of course we'll dive into real-life examples from our customers that use our solution. Um, so first of all, the problem. These kind of questions are being asked on a daily basis on or even an hourly basis in every gaming company. Cohort analysis, A/B test analysis, root cause analysis, engagement, retention questions are being asked all the time and the problem is that the only ones that can actually answer these questions are the game analysts. They are the ones that knows SQL. They know how the data is built. They know the bits and bytes inside a data, the events, the parameters, etc. Uh, the problem is that they are always busy, swamped with other questions from coming from all around the company. Um, and on the other hand, they don't like to um, just answer question and and be like a sequel monkeys uh, for the rest of the company. Not to

mention there are tons of petabytes of data coming from the gaming every day, so it's really hard to provide insights from this amount of data. And then came GenAI, and the promise was just chat with your data, connect your favorite model, your cloud, your Gemini to your data, and then you can ask anything, and you can get real-time insights. But then reality hits. Hits. The gap between the promise and reality is really really really enormous. Why? Because of context. Without understanding your game, understanding your own business, the agent could not actually uh answer any data question accurately. Um the agent the AI agent needs to actually understand your specific way of metrics calculation, how you in your game calculate even the simplest metrics like daily active users. So, [snorts] different companies actually calculate uh the daily active users metrics in a different ways, and not to talk about churn or D7 retention analysis, etc. And

the agent needs to understand these kind of things, and your internal glossary, what is levels, what is missions, etc. etc. So, only once the context is uh properly built, then the AI can really uh answer questions accurately. So, for every uh such a solution, for every AI analytics solution, in general, in a very high level, there are two parts. The first part is what we call internally the context builder agent. It actually builds the context such as that I was just talking about. Um so, the context builder is actually um connecting to all your data sources, to of course to your database, to the data warehouse, the BigQuery, the Snowflake, the Databricks of the world, and to your BI tool, your favorite BI tool, Tableau, Looker, um or or a Power BI, for example. It also can be connected to your Confluence, to game research, game design documents, etc. So, the more sources you provide

the context builder agent, the more accurate will be the other agents that I will talk about in a second. So, the builder agent is building the context, and then we have the conversational agent, which actually answer questions using this context. Um, now it's a feedback loop. You see the arrows goes both ways. Why? Because once an analyst um, ask a question, and maybe uh, it's the analyst can see that uh, the context needs to be updated. So, it's a constant feedback loop from the conversational agent to the context builder agent. So, what is a context in terms of game? So, the most the most important building blocks in context layer for gaming is the events. I'm sure you all know the concept of in-app events, tracking events. It tracks the user behavior, the user journey inside the game. So, it's uh, the agent really needs to know for each event exactly what is the meaning of the event, and each event can have dozens of parameters, and the agent

needs to also know exactly the meaning of each and every such a parameter. Let's take a look from real customers of ours. This is a an event called RV watched. You can say, "Wow, it's easy. Every game expert agent will know that RV watched is an, of course, uh, an event that is fired whenever a user uh, watched an rewarded video, right? But, let's look at the parameters of this event. SC balance and HC balance. What's What is SC? What is HC? It needs to understand the glossary of the specific game. SC means soft currency, HC means hard currency. Uh, so the agent needs to know in order to answer correctly questions about the currency. Uh, you can also look at the mission ID. Mission in this case, in this game, equals level. So, a user a business user will probably ask about levels and not mission it the business user doesn't really know that in in the data level is called mission. So, it will ask questions about level

and the agent will have to answer accurately and look at this parameter which called which is called mission ID and not level ID. So, this was about the events. Your data is probably composed of tables and again the agent needs to understand for each table the meaning of each table and there can be dozens of columns for each table. So, it needs to understand each table meaning. For example, you see here again real table from one of our real customers a table called user metrics live. The agent the builder agent understands from the data itself from past queries from all the data sources you provided understand that this table holds user user profile metrics. So, whenever a user will ask about it, it can go to this table and answer accurately. The most important parts to my in my opinion in every context layer of gaming agent is actually the semantic understanding of your metrics. How are you calculate churn? How are you calculate CPI? Let's see ARPU average

revenue per user. So, here you can see the sequel that actually represents how in this game they calculate the ARPU the average revenue per user. And again, it it is done differently between companies and between games. This sequel is generated by our builder agent. Again using all the data sources using for example the sequel that are building your dashboards in Tableau or in Looker. Now, we have two more components relatively new for the context layer. You probably know the concepts from if you are using cloud. The fourth one is skills and the last one is memory. So, skills are actually predefined actions that you are doing in your company whenever something is happening. For example, if you want to examine new release, you have a new game release, you always look at the revenue metrics, engagement metrics, retention metrics of the new release. You do a set of

operations, all the always the same operations. Why not define it as a skill, and then whenever you need to do release performance analysis, you can just trigger the skill, and you have it as you used to it as you used to. So, this is also part of the context layer. By the way, this is relatively new because I don't know if you know the term semantic layer. Semantic layer used to include only the semantics of your data, meaning the metrics, the segments, etc. Now you have the skills, so you actually do root cause analysis or cohort retention analysis in your company. The last part of the context layer is memory, things you always want in your agent context. You have We have personal memory, I want you to remember my name, I want you to remember my role, and product memory or team memory, things that you always want whenever someone from your team ask data question, you always want the agent to know about. Exclude always exclude users from this and this country, it's not an interesting company country, for example.

So, this is it. We built the context layer, the context builder finished the onboarding. Of course, as I said before, it's ongoingly updated, the context is ongoingly updated. Um but now you can the sky's the limit. You can ask anything about AB testing, root cause analysis, engagement, etc., etc., whatever you would ask your analyst, you can do it by yourself. For analysts, it just make them much more productive. Um um two more very nice things you can do when you have the context. First of all, you can create automatic reports. I want every Monday to see a retention report of my game. To my Slack, to my email. You no longer needs to wait for the analyst to provide your build you these reports. Um and the nicest things, this is again relatively new and we're very proud of it, are the dashboards. Remember you used to go to the our analyst and ask for dashboards? This is no longer the case. You can actually have an uh question, make a dashboard out of it. If you want it to be private, it's okay. If you want it to be public, you can

publish it. And the most important part is no longer you no longer needs analyst in order to ask deeper questions on the dashboard. You have the dashboards and you can ask it, "Why do I have a drop in revenue? Why do I have a an increase in my churn?" Um so you can just chat with your data because your context is already built and maintained. Let's look at real-world example. This is coming from one of our real customers. This is real data. Let's see. So, um we're asking the drop, the leak in revenue and you see the agent in action. We always We believe in trust and transparency, so we see we show each and every step of the agent. You see the thinking process, which is super important. You see all the components in the context layer that the agent is actually using in order to answer accurately. For every analysis that you see here, you see some numbers. We always show the sequel behind the query. It's very very important, by the way, to do so if you're using a solution that is

not that transparent, something is wrong. So, everything is here. Um let's see. And you'll get a nice visualization, which you can share, download, make a slide out of it, make a PDF out of it, whatever you like. And of course, you can see here there's a problem in mission 15. Level 15 is somewhat problematic. Um you can see all the graphs. And now I want to make it a report. I want to see if I have a revenue leak um every week to my as a as I said before, to my Slack or to my email. Not a problem. I can just write it down in natural language and I get everything um, to my email. You can see here. Here's the prompt it generated uh, the prompt. Super easy and I get it. Now I want to make a dashboard out of it, right? I actually I asked ad hoc question. I don't want to ask the same question again and again. I want it to to be a dashboard that I can go into every day or every week or whatever. So I can make a dashboard out of it. Uh,

here's the dashboard. Now it's private. As I said before, I can publish it. I can even send it to analyst for review. Make sure everything is correct. Um, and then I can go to this dashboard. Here's the dashboard and I can just click ask and I can ask any questions. For example, I can ask why do I have a problem in mission 15? What's the problem in mission 15 in level 15? And let's look at the root cause analysis that the um, actually the um, uh, agency is doing right now. You can see it really goes deeper. By the way, this in this game they have a skill for root cause analysis in order to actually understand the churn cause for each level. And here you can find that the level is more difficult than other levels that they had before. Uh, it takes time for the users that do complete level 14 the um, the duration of the level is much much much much much higher. And also, um, this level requires currency uh, average

currency that is much much more than the average user has when it comes to level 14. So you have here all the reasons why do we have churn in level 14. Um, this is just a sneak peek. Uh, to be more juicy, we've analyzed thousands of questions from our customers to see what users are actually answers they uh, they ask about. Only for gaming companies. We serve also um, more B2C verticals, but uh, focus on gaming. So, not a surprise, a third of our users of our questions, sorry, are are all about revenue and monetization, money in general. Then we have also about retention and churn and about funnels and progression. You can see it here. Super interesting. Now, we didn't analyze only the subjects the the subject of the questions, but also how what is the way that people are asking. So, 27% of the questions are

actually exploratory questions, open questions. Segment my high highest LTV players or segment my churn my churners. So, it's something that you can see in dashboard. Dashboard we don't want to help you in this kind of analysis. The second the second most largest party here is descriptive questions, which are maybe they can be answered using dashboards, maybe not, but it's very nice to see. Also here you can see AB test analysis, root cause analysis, which are very very popular in our platform as well. What about the future? So, what we see as the future of game analytics and AI game analytics is first of all, why not ask questions and surface insights for 24/7? Why just ask for users to help to ask questions in order to surface insights? So, we want to proactive game analytics. We want the insights to be in

our slack every morning without us even need to ask anything. And also the second point is very important, closing the loop. If the recommendation of the agent is conduct an AB test analysis in order to check this and this, I want him to press a button and then it will be connected to my AB test platform and then I can very very easily just generate this AB test. Same goes if I want to change some resource or change some screen etc. Make it actionable insights. The third one is something that we constantly been asked. When I ask a question about my KPIs or about my metrics, I don't want only to get an answer but I want it also to be compared to the benchmark of the industry. So tell me I'm above industry benchmark, below industry benchmark etc. This is us party queue. We have a booth here. Thank you so much for listening. Come get soccer ball. It's really nice. Happy to be here. Thank you.
Thank you.

[music] [music]