Session: Game Data For All: How AI Is Democratizing Game Analytics - Orly Shoavi, ClarityQ.

Our next talk is from Ollie. Ollie Ollie at Clarity Q and we're going to be talking about how AI is democratizing game analytics. So, round of applause, please. [applause] I'm Ollie Shavei. I'm the co-founder and CEO of Clarity Q. Uh I used to be the co-founder and CEO of a company called safe decay which was acquired by Applavin and then I was uh general manager at Applavin for three and a half years and then I left in order to found Clarity Q and Clarity Q revolutionizing the way game analytics is done of course

by using Genai and in this talk I'm going to talk about how every Genai solution for game analytics should be the musthaves and I I'm also going to provide some real life examples from uh gaming companies that are our customers. ers today. Uh but first let's dive into the pain. So questions like this are being asked on a daily or even a hourly day basis in every gaming company. Uh cohort analysis, segment analysis, AB testing analysis, user behavior, user journey all the time. You're asking these questions. Now the problem is the only ones that can actually answer these questions are the analysts. They know SQL. They know the bits and bites in the data, the bugs that are hidden inside the data. And no matter how brilliant the analysts are, they are mostly most of the time they're super overloaded. They need to answer other questions coming from across the company from marketing and growth and ops and and management of course. Um so decisions are actually made in delay of hours,

weeks or even made without uh supportive data which is the worst case possible. Now on the other hand you have the analysts and they don't like to be like SQL monkeys or report monkeys just answering uh uh ad hoc questions all day long. They want to do some in-depth analysis. They want to bring um uh meaningful insight that really really move the needle. Uh so there's frustrations from that side as well. And in gaming companies we know we all know there are tons of data pabytes of daily data billions of events. uh and these things need to be solved in our uh in our era. So when then came Genai and everyone was saying perfect now I can chat with my data data I can ask anything I want I can connect my favorite LLM to the data via MCP but that really doesn't work like magic you can ask anything you can even get answers that sound smart but then you see they're inaccurate so there are more uh more to this you have to understand

the gaming the game context the business context the metrics the events the meaning of the underlying data before an agent can answer uh data questions accurately. So this is in a high level for every geni solution for data analytics for game analytics. This is in a high level the the the structure. Uh the first thing is the context builder agent. I will call it the builder agent which builds the context. It connects to the data sources of the company, your data warehouse, the BI tools, the Tableau, Looker of the worlds, to any documentation you might have, uh to your SQL query history, etc., etc. So, the more um the more sources you provide to the to the context engine, the more accurate will be the context that it actually builds. Uh and then once everything is built, once it's truly understand the semantics behind your events, data tables, uh and your calculation, your metric, your segments, etc., Then you have the

conversational agent which uses this basis which uses this context in order to accurately answer the questions. Let's see how we build context accurately uh for gaming companies. So first the row data the events the ones that tracks your user behavior the user journey what's the users are actually doing in your application in your game. So um for each event as you all know there can be for each game there can be hundreds of events or even thousands of events and events type I mean and for each event you might have um dozens of parameters and each parameters might have uh hundreds of or even thousands of values for the uh parameters and you need to understand all this in order for the conversational agent to be able to answer accurately your questions. So this is why the context builder uh agent has to understand really each and every event and data points that I just mentioned. Uh you can see here for example this is from one of our customers event called an event called

RV watched which is fired whenever a user watches rewarded videos. So you can say this is easy to understand right it's called RV watched RV every by the way every simple agent probably know that RV stands for rewarded videos. But let's look at the parameters. The parameters are SC balance. What is SC? It's something that is in the internal glossery of this specific gaming company. What is what is SHC balance? And more importantly, mission ID. You can see the the um semantic of mission ID. The description is the mission level number. So in this specific gaming, mission equals level. Without understanding this, the conversational agent will not know how to answer a very simple question such as how many users completed level five. So you have to explain somehow to the conversational agent what that mission equals level in these specific games. And this is exactly what the uh converser agent needs to do. By the way, spoiler

alert, we also need human in the loop always. We all know by now AI doesn't work like magic. Okay, so we have the builder agent as automatic as possible and the more data sources you connect the more accurate it will be. But you have to have the analyst who knows who know the data go over everything and approve it. We'll talk about it later on. Um okay so after we understood the events let's talk about the tables also we need the aggregated tables understanding. So you need to understand each and every table meaning each and every column of tables. there can be hundreds thousands of tables, thousands of columns and you need to understand everything in order for the conversational agent to be able to answer it accurately. Uh in here you can see for example a a table called user metrics live and this table includes metadata on users. So the conversational AI needs to know that whenever it is being asked about user details it needs to go to this table in order to answer

the question. super super important for all the context building to be as automated as possible and as accurate as possible. Okay. And la now let's look at the most important part in understanding the semantics of your data. This is what we call the semantic catalog. Maybe some of you have heard of the term semantic layer. It's not exactly the same uh but it's very similar concept where you understand not only the row data the the building blocks the events and the tables but also how this specific company calculates each and every metrics. So in each gaming companies you calculate um churn differently retention differently even a simple basic term such as daily active users is probably calculates differently in different gaming company. Even in the same gaming company, some people can calculate differently the the metrics called daily active users. So it's important for the uh conversational agent to understand exactly how it is supposed to answer questions about daily active users. Um let's look at a metrics called ARPO.

I'm sure everyone here knows this average revenue per user. What you see here is the SQL query that represents how this specific gaming company um calculates ARPU. Okay. So in another gaming company probably different SQL. Now this is built automatically by the way. It should be built automatically from the data sources that are connected to the agent. If it's not built automatically by the way that it's a problem because there's tons of manual work in here. Um again human in the loop is needed. They need the analyst needs to approve everything here. But you need to provide super easy to use UI in order for them to approve it. And this is what we do. Perfect. Now another thing that the context uh builder agent needs to do is actually protocols. You probably have in your gaming company uh predefined set of questions for some purpose. For example, I want a root cause analysis for um to to examine the the drop that I

had in revenue. Uh so there are some predefined uh series of of actions that I need to do that need to be done in order to uh analyze this uh scenario. Uh another scenario possible is for new release. Okay, you have a new release and you want to understand you have a series of of steps you probably do for every new release in order to analyze the performance of the new release. This is what you see here. Here you see actually the the output of the conversational AI uh agent uh for this specific for the other gaming company. You see here um the summary. They look at retention, usage, etc. Retention and quality. This is some of the protocol. This is part of the protocols that this specific gaming company uses in order to analyze the new version, monetization, adoption trend, etc. So a simple question can be asked analyze the performance of this version versus the old version. No need to provide the entire process for the agent. So once the uh builder agent built this

context now the conversational and by the way the human in the loop uh is is completed by the analyst approved by the analyst. Then you have the conversational agent it can answer any question in a very accurate accurate way. engagement, retention, adoption, anything you like. In-depth analysis are being done every day on a narrow basis. Once you go uh you just can't uh you just can't afford going back to just writing SQL manually or being totally dependent on the analysts. Let's see some real life examples from customers of ours. Let's see if it's working. So, okay. So, we have the AB testing analysis. Um we kept we kept the the typos everything it's really really the the analysis made for AB testing in the US show them the level funnel for first 50 levels including level completion rate RV per user interstial per user on different graphs we believe in transparency so the agent always will

show you the exact steps that is going to do and this is how you know that the agent actually understood your question right if not you can just fix it you see all the um steps you can see the sequence This is for analysts. We're also co-pilot for analyst. So they can see the SQL behind every everything. And you see the final analysis. Super easy. You don't have to understand anything about SQL or data in order to get this analysis. And you see here the completion rate, RV per user, intel per user, and the summary table. Now they want to know why the drop in completion rate for group ENG in level 34 to 36. You can see here the there is indeed drop in the completion rate. Super important. Let's see. So the agent first want to make sure that there there is um a drop in completion rate and there is and there's also a lot of RV per user and inter interstitial per user in this specific um group E and in G. So you can see it here and also it asked why users

aboning due to ad frustration or some technical issues and see what it's uh they found out that the restarts per user at level 35 look at group ENG of course the completion rate rate was dropping because there are tons of restarts probably because of the ads something is truly wrong with group ENG anal an analysis made in 10 minutes instead of days um you can see here And let's see the next question was can you compare the revenue of groups ENG okay maybe the user saw so many ads that it somehow compensates uh the the completion rate the drop in completion rate but this is not the case the revenue stay the same for group ENG so probably you don't want to go for group ENG in this AB testing analysis let's go for the second example I wanted to show you the LTV prediction No.
Any suggestion how I uh operate this?

Ah, perfect. Please create the most common purchase chain per purchasing bucket. Example, players with LTV of $100 to $200. This is the first purchase in $10 and then $17, etc. They want to predict whales. How can I identify whale in a very early stage in the game? So you can see here the analysis again this is game owner asking it's super super easy for them to ask and they want to understand the patterns let's see so the key insight is a player second purchase price is highly predictive of the user being a whale but the user insist I want to know is there any indicator to differentiate between low and midval players before the second player and again there's a strong uh you can see the first purchase price over 20 and depositor depositing uh money before playing it's a good signal for being a mid mid uh uh player not a whale maybe

but some some someone that you want to invest in. So these are all real examples some takeaways after two years of uh implementing developing and working with gaming companies that work with our solution. The first one and we talked about is context is the key. Without accurate context and understanding your own data, it won't work. It it won't be uh as accurate as you want it to be. Second takeaway is no solution will be 100% accurate without human in the loop. Analysts are super essential for these kinds of solutions. Okay, we have to have the approval for the uh semantic understanding and we also need the you the analyst to be the first users of any solution like like uh for game analytics. Why? Because as we said before analysts know the bits and bites the bugs the small bugs in the data the filter that you need to add only in this version etc. So they need to be the first users and only once we get the green green light from the analysts then you can spread it around the company and

everyone can use it. Uh third, no one likes on boarding. So no, everyone hates on boarding even though it's super important for this specific use case. Um but our job is actually to be doing all the onboarding as automatic as possible. And as I said before, we have to have human in the loop. So it's our responsibility to provide to analysts the best and easiest to use tools in order to make the approval uh process. uh force, explanability and transparency are a must. Not just because it's actually uh making the agent much more accurate, but also for you the users to understand if the agent understood correctly your question, if the direction it takes is it's the right direction. Sometimes you need to fix it. Sometimes you need you want to direct it to other metrics to look at. Super important. Um and my favorite takeaway, users will indeed ask anything. So in clarity Q the first version that we made

was excellent in generated in generating accurate and very fast SQL but once we provide clarity Q what we call internally clarity Q 1.0 Oh uh to users then we learned that they indeed ask asking anything from change the bar bar colors to our company logo colors or summarize all the analysis uh into mail for the management uh or suggest ways for me to be the first one in my category in the play store. How can I increase revenue things that requires much more in-depth analysis than just one SQL? So yes, we need to um to uh make sure our agent knows not only to answer questions accurately but also knowing to also answer I don't know I I I don't have enough data to answer your question or I need more clarification in order to understand your question. So what's next for uh for geni for the game analytics in our opinion first uh add market aware analytics you have

sensor you have you have external benchmarks we want to connect it to uh to the game analytics so you can ask questions like uh how how my retention is doing compared to um um games like me like in my category uh predictive analytics is super important also there are already models that do it very well we need to combine it with uh LLMs and um so you can ask questions like I want to increase the AP uh price how would it affect my overall revenue and real-time proactive insights this is the holy grail actually uh it's going to it's going to be a reality very very soon but it's important not to um not to uh develop blindness to the insight so we need really meaningful significant uh uh insight that will really really impact uh the conversion revenue, retention, all the KPIs. That's it. Thank you very much. Something is wrong here, but um you can

ask any questions. We're we have a booth out there and I would be happy to talk. Thank you so much. [applause]