Listening to Players at Scale

[music] Welcome to the Navit Gaming Podcast, where it is our mission to explore the business and future of video games. We bring together the industry's brightest builders, investors, and thinkers to keep a pulse on current events, dissect emerging trends and games, share lessons learned, and have a great time. This podcast is also part of Navik's growing ecosystem which ranges from free and premium research to consulting and advisory services. For more information, visit www.navic.co. This episode is brought to you by Heroic Labs, which powers some of the world's biggest games with its heroic game stack. Its composable products are Nakama, the open- source back-end game server, Hero, the game development framework for all your live service game features, including the complexity of managing offline mode, Sattorii, the

worldclass live ops product for maximizing monetization and retention. And lastly, [music] the Heroic Cloud to provide massive scalability. After nearly 10 years of operation, thousands of studios trust Heroic Labs to help them focus on their games and not worry about game tech or scaling for success. To learn more and reach out, make sure to visit heroiclabs.com or check out the link in the show notes. This episode is brought to you by Neon. For the first time in our industry, developers can finally have their own DTOC channels to save on app store fees and build closer relationships with players. But there's a catch. Selling direct means either building complex infrastructure yourself, or choosing among legacy providers whose inexperience in games or questionable reputations pose real risk to your business. Luckily, there's Neon, the DTOC platform built for games. As a merchant of record with customizable web

shops optimized for conversion and modern APIs that let you use your existing live ops tools, Neon is trusted by some of the biggest names in gaming to help you sell direct without the typical overhead. To start selling direct and finally take control over your full player experience, simply head to neonpay.com or check out the link in the show notes. And with that, let's jump into the episode. Hello everyone. I'm your host Evan Becker and today I'm de delighted to be joined by Tom Gainer, CEO of Leveler. Leveler is an enterprisegrade intelligence system built for community live ops product and research teams. So today we're going to discuss the business of bridging the gap between players and game developers, especially around player sentiment. So just to start things off, Tom, can you give a quick overview of your background and ideally the origins of Leveler?
Sure. Uh and thanks for having me on the pod. So um yeah, we launched Leveler about uh 5 years ago and um I think if

both myself and my co-founder and CTO Ben kind of look back at the the common threads in our careers, it's been how do we bring users together around a shared passion? How do we fuel that passion and how do we turn that into insight and how do we turn that into into revenue? So I started my career working actually in interpublic group agencies working with blue chip brands like Mastercard, Nike, AB&B and helping them build community around gaming, sports and and music properties. I then moved into the startup world. I worked for a live streaming platform thinker Twitch but with quite a lot of uh live sports content building community around live content. Uh that then had an exit in 2019. Um and then I started working on Leveler. Ben my co-founder and CTO was at YouTube for seven or eight years. was also really thinking about community there. And I think what both Ben and I were seeing five, six years ago was that, you know, traditional social media platforms, which is where we used to come and talk about things that we loved, about games that we loved or

sports or music, um whether that be Facebook, whether that be what was called Twitter back in the day. Um they used to be conversational, but they were really shifting. they were shifting to become a a broadcast media format, short form content, algorithmic scrolling, very valuable top of funnel, but just not the place for conversation and connection anymore. And we were starting to kind of play about with platforms like Discord um as kind of naturally curious people that have been kind of around the community space. And I think we had a hypothesis that as kind of Tik Tok and Instagram and these um short form content platforms were really shifting in that direction that you know as humans we've always wanted to gather around a campfire and that's that's not going to change. And so our kind of hypothesis is was that we were going to see growth on communitydriven platforms like like Discord and and Reddit and that businesses and brands were going to need a set of tools to really help them get the most value out of those those platforms. I think it we looked at the early days of social media

and we saw growth of businesses like Sprinkler, Sprout, Hootsweet, Brandwatch, Meltwater, Chorus. There are a lot of companies that grew to to pretty large revenue sizes and and we felt that there was going to be an opportunity to help do the same for the shift to these new community platforms. And so that was the that was the original kind of kernel that led to us building Leveler a few years ago.
Well, seems like it's definitely turned into quite a bit of a thing. uh and and feel free to go into as much detail as you feel like is appropriate here, but how does Leveler work to actually gather player sentiment?
Yeah. So, um so, as I said, we're kind of focused on on two platforms today in in Discord and Reddit where we really tend to see the highest quality and and signal. Um, so the first thing that we do is begin by taking sort of the fire hose of messages that are coming in from our customers kind of discord servers and and also subreddits. And the first step to kind of really understand sentiment is is to understand the conversations that are going on. Um,

and of course in a high volume Discord server, I mean some of our customers are getting a million messages a month, right? I mean it's just it's just constant noise. Um and there's a lot of conversations that are happening simultaneously often in the same channel or the same thread. Um which just from a human basis, right? I think if we look at the tools that I just mentioned some of those those companies there some of the tools that have been built for kind of social media and digital teams um have had a huge amount of investment and yet community which is where we now see you know our customers Devon are telling us all the time that Discord is now their most important platform we hear that all the time from the games industry and yet the tools haven't matched the expectations for community teams right if you actually work as a community manager and you're having to bring back insight into your business from Discord, from Reddit, from Steam, it it's just thousands and thousands of messages, right? And manually that

becomes very hard. So, we've got to bring together that fire hose. And what we then do is we bring a combination of kind of statistical analysis, uristics, and LMS to help group and then understand how those messages actually fit into individual conversations. So then within the context of that conversation, we can actually kind of carry on to process and understand what is the sentiment of the user within that conversation. What is a summary of that conversation and what topics are being discussed? And so once we've done that work, the goal is then to get that insight back into the customer in the format that they need it. So whether that is coming to into our dashboards and coming and looking through topics, whether that is actually just coming into our level AI solution where they can ask a question around their data and we'll have agents go and find that response, whether that is actually converting that data into automated email reporting that lands in inboxes, whether it is uh anomaly kind of slack detection because people often want to know has there been a big shift in the number of messages or the shift in

sentiment. I want to know that in real time so we can connect that into your internal systems or or maybe it's via API because data teams want that data. So the kind of end point will vary depending upon which person in the organization wants that data but the upfront work is where we've really invested a huge amount of our kind of time and and engineering resource because you know ultimately if you're going to use that player data that player voice data internally you have to trust it and you have to trust the LLM. We've got to make sure that's not hallucinating, that it's really high quality. And you know, we've heard Devon from so many customers that they've tried to kind of take raw Discord data and throw it into LLMs and just haven't got the results they've wanted. So I think that's that's where a lot of our the magic in our product is.
Yeah, it's definitely a tricky process and I I'm sure there's been a lot of learnings over time in terms of how to actually massage a lot of that, how to deal with a lot of those edge cases and problems that come up, especially when it comes to, as you mentioned, LLMs and AI and hallucinations. last thing you want to be doing is passing on bad data to uh to these customers

and and there's a lot of uniqueness as well depending upon the the the game and a company. So some of the things that we hear a lot are hey like my player base has you know a very unique set of of words of phrases of language right um we've run into problems in the past where uh you know the word die which doesn't mean die in German for example uh like that could suddenly be perceived as as negative by the LLM and so therefore we're not getting the right results out of this how can we make sure that the LM M understands our games um nuance and so we've really made sure that we can kind of be that human level accuracy for customers and so we can prompt the LLM to understand the nuance of of your game and your language. Um, and then we've also got things like translation. So, you know, the the beauty of a platform like Discord is that, you know, you can have a truly kind of a global community, right? Um, that also creates challenges

if you've got people sending messages 24 hours of the day, but it does mean that you can be getting feedback in from players in Japan, North America, Germany, all in one place. Um, and also speaking in their local language. But we then have to make sure that if you know it's a Japanese publisher that's using us or a North American publisher that we're then translating all of that different uh feedback properly and and and turning that into into insight for them. So there there's a there's a lot of nuance in getting that piece in getting that piece right. And I think one of the good things for us Devon was you know games publishers have come up against these issues for a long time. So they have a they have a real opinion on this and they'll let you know that and that's been very helpful for us as we've as we've kind of thought about what we need to build in our product. I've had very kind of open transparent feedback. Yeah, I can imagine like you some of the examples you brought up being like a real challenge like die for example singular for dice and being in games obviously that's a probably pretty

common thing to come up right but it the context matters a lot and that's one of the things I think is a challenge when it comes to processing lots of text data is probably getting in uh the context like for example lots of talking about different languages and stuff like that not only do you have slang in these different languages but you've also different got colloquialisms uh slang that gets developed you know across genres or specific games or referring to things within the game as opposed to uh you know something that sounds like it might be talking about outside of the game. I got to imagine that it's got to be a process even on a per game basis to continually evolve that as opposed to just being like okay let's set you up and then you're good to go. You've got to keep coming back and be like well oh we got a we got an anomaly here because it turns out it misunderstood what the players were talking about.
I mean let's let's think about what a healthy community is. it it's a place where people build relationships, right? And they build their own kind of sarcasms and and slang and language and that, you know, like I'm sure you have with your group of friends, right? Like

if people came into your conversation and you guys were all sat at a bar together, there' probably be stuff that I wouldn't have context for. That that that is what a community is like. And that changes and evolves over time as people new people come into the community and people leave. So, as you say, it's it's a constant iteration. Um, but that's something that we've really got to get right for customers because that they want that that level of accuracy. And I think we'll talk a bit more later on about some of the the challenges within customers, but you know, one of the things we see a lot is kind of uh people in the org haven't always necessarily given full weight or trust to some of the community data that comes in. And so we've got to make sure that, you know, if a community team is is is using us or a live ops team or um a game designer that that they have genuine confidence in the data that they're working with um because they're making some very very big decisions on on on on this insight.
Well, it's a good thing that you guys do

like citing of the data and stuff like that to help them double check, feel some confidence, right? I imagine they look back occasionally on those things. let me make sure that that means what they what it they interpreted and then at some point they start to feel like okay maybe we can trust what what level is sending us. Uh, a lot of those are challenges, right? And I'm sure we'll get into more of the challenges, but on the flip side of that, what are some of the biggest uh, benefits or advantages that you've seen coming out from this approach, from this data for, you know, the these game developers? I mean, they're they're doing this for a reason, right?
Right. I mean, I I think the place to start here is actually talking about um the biggest challenges that that that we've that we've heard from customers and and and that will then lead to to what the ultimate benefit is. The the first one that just consistently came up was um we have you know customers saying we've got this problem internally around kind of signal to noise prioritization right so discord and reddit have become our most important platforms they are super valuable to us but we have so much

feedback from players right and it's coming all the time that it's quite challenging for us to know what to actually act on you know what data should we bring back? If we listened to every piece of feedback that our Discord community giving us, our road map would go around the world three times, right? And we hear this story all the time of, you know, someone in the exec team seeing a post on Discord or seeing a post on Reddit and sharing it internally and everyone starts running around because somebody's found this post on Reddit. And yet the people that then have to act on that data are like, "But I don't know who this player is. Is this a Is this a new player? Is this a legacy player?
Is it even a player?
Is it even a player?" Right? Do we know what market they're from? Um it's really hard to then act on that data. And and additionally, if you're a community manager, right, and and kind of pre-leveler, their workflow was I have to go into these platforms every day. I

have to scroll through. I then have to manually kind of screenshot things that I think are interesting. I then have to bring that back. But the problem that they were facing was, okay, well, thanks for the feedback, but how many people care about this, right? And and what players care about this? Because I can't I can't just take your word for it. I I I want some data to help back this up. And so we've done a lot of work to make sure that our customers get a real understanding of kind of waiting, contextualization, um, citation, and what type of users. And maybe we'll talk a little bit more about kind of some of the segmentation work that we've done later on, but we've we've got to get that piece right. When you bring that data back internally, we have much more context in understanding whether we should act on that data or not. Right. And and maybe just one last thing on that challenge there, Devon, is we've kind of learned about how the community manager does play a very kind of important role of um uh kind of sort of passing some of

the information that comes internally, right? And and actually if they did pass on every piece of feedback from players, the engineering team, you know, product, game design would be overwhelmed. So there's too much for them to work on. So you know the the community team do build a six sense about their community and there is value in that when they can combine that their knowledge with our with our data um that becomes much easier for the business to act on. So so that was the first that was the first big challenge and so you know in terms of a benefit teams are are listening to their players more than ever right and ultimately the goal is we want to build great great games and great products for for our customers. we want them to to buy more of our game um or buy our new game and and drive revenue and we're starting to see that come to life. So I think that that one's great. I think the second thing the second big challenge that came up a lot was this issue around reporting latency and I talked about that that kind of way of working of community managers. I mean literally

being tasked with you got to scroll through these platforms. Um, but we, as I said, we have global communities, right? Live issues can appear at 1:00 a.m. on a Tuesday night if you're an American publisher, but your players are are in kind of Asia. Um, and and systems today aren't kind of typically set up to to handle them. And so what we found was community managers doing hours and hours of work, lot of energy, kind of pulling together reports, then sharing that back internally, but that report might not come three, four, five days later after either an issue or an opportunity has arisen, right? Because we're not just hearing about bugs or frustrations players have with games. they're also talking about, oh, I'd love it if they did this partnership or there's a meme taking off or, you know, marketing teams need to know if there's there's things that they can jump on. Um, and so, you know, we've got to make sure that that that the right insight and the right data lands with the right user in the organization in the place that they

want that to land in. So, whether that is I'm in the exact team, I I want to have a daily email, right? Uh, you know, for many of our customers, we've got daily emails going out in multiple different languages to different personas in the organization. Whether that's a Slack alert, um, whether that's an API feed, whether that's coming into the dashboards, um, you've got to make sure that that you've really got the structure set up with that with our customer for for the data to land in the right place. And so, what we're helping do there is just minimize community risk. I mean, community risk is real, right? Um, you know, our customers have said we can we can burn millions of dollars in a day if we post the wrong thing into a community if we don't react fast enough. Um, and so they want to know they they want to know if if things are going either off or if there's opportunity in the community. And then I think the third thing that kept on coming up was um, yeah, Discord is is super important to us and Reddit's really important to us, but it's hard for us to understand what the value of those users are and of those platforms. I think around the traditional social

media platforms we kind of built um you know customer acquisition costs and lifetime values and really understanding the value of those users and of those platforms particularly advertising driven platforms but that's not been done in such detail within community and I think that's frankly why some community teams are still undervalued and and underserved within organizations. I think what we've seen is, you know, a Discord user is typically at least kind of six times more value valuable is what we've seen with customers. We've seen customers uh and help them prove that their users can be up to 90 times more valuable on Discord. So these are players that are spending on your game. If you think about the effort to go into a Discord community to give feedback, they're highly engaged. And so for for customers that have 50,000, 100,000, half a million, a million users in their community, I mean, they literally can be sat on millions of dollars in there. Um, but but particularly sometimes at exact level, they just might not, you know, if they're not nent users of those

platforms quite realize how valuable they are. And so, you know, we're doing a lot of work with customers to help them understand um, you know, kind of ROI, kind of competitive insights, player churn, how do we kind of give them more insight into their funnel because this is happening on these platforms. It just hasn't been highlighted well enough yet.
Yeah, that makes a lot of sense. And it's interesting you mentioned, you know, the the value of the players between the two different platforms in that like the the hurdles to jump to be on part of the platform. When it comes to Reddit, you you could just post with a Reddit username that doesn't have to be connected in any way to that uh that particular subreddit or anything like that, right? You don't have to have any level of commitment to participate in the community. Whereas a Discord, you may have to join the server specifically. You have to go through oftentimes verification steps depending on how they do it for anti-bot and other things. There's usually a lot of hurdles to actually get to the point of sharing some information that usually like indicates a certain level of commitment. And these sound like the two main platforms you're doing. I I'm really

interested in that like the sort of difference in approach partially because of you know the the thing you said about the value but also because uh they seem to be very different types of communities different styles especially like levels of sarcasm and negativity versus positivity like these two different platforms. How are you kind of approaching them differently from each other?
Yeah. Well, I mean you made an interesting point there about kind of the I suppose the steps that one has to go through with Discord versus you could just post on Reddit. But um to come back to that like uh we actually we we had our we had our company uh we had our company offsite last week and um I kind of pulled up this slide where I showed an example actually with we we work with Google Gemini and their Discord community where they've really lent into kind of customer feedback as they're rapidly iterating that product. And I I pulled up a photo from first of all from uh from Tik Tok of Google Gemini posting about uh a voice intelligence solution and the comments underneath were

basically can I get Gemini for free and your competitors are better and there was just no insight or value there. I pulled up the same post from Instagram and in the comments all it was was flame emojis, love heart emojis. It was just emojis. I mean, there was just like no nothing that people could take out. I then jumped onto Reddit and yes, it, you know, it's it's it's there's still more steps with Discord, but Devon, people will go on to Reddit and post paragraphs, right? So, there's still a there's different there's a different level of intent. I'm going to go onto that platform and give you that feedback. I'm not just posting an emoji or asking for the product for free. I'm doing it because I care, right? And so on Reddit, you're going to get kind of that you might get those deeper threads. And then within Discord, it is that kind of, you know, people sat at a at a bar, a restaurant having that real-time conversation together and and real gold can can fall can fall out of that because people are there to talk about your product. And it's interesting

having, you know, both kind of spent some time in the games industry early in my career and then kind of done some work in sport and music and and then come back to gaming. One of the things that I think the gaming industry kind of uh undervalues and maybe that's because some, you know, there are a lot of people that spend their whole lives in in the gaming industry because of their passion, but having thousands of users in a community talking about your product all day every day is not normal. Like most industries do not have that happen, right? You cannot imagine what a L'Oreal or a PepsiCo or a large fashion brand would do to have that many users come together talking about that product all the time. Like it is it is so incredible that we can access that level of of like pure passion and want to talk about our products in the games industry. Um and yet it's it's kind of you know not always been kind of synthesized in the right way to work with. And I think, you

know, we're in a really interesting position, Devon, where customers are getting so much value from our data that they're often saying, well, you know, what about Instagram or what about Twitter? Like, do you work with those those other platforms? And and actually what we're saying is, well, like what value are you really getting from them there now when it comes to player voice, when it comes to sentiment? Sure, you need to understand from a reporting perspective what's happening on those platforms. Are we seeing a shift in users? And you know I think people have got addicted to you know growth that the last decade as marketeteers we've been taught you know numbers go up. Um but actually you know how we look at the world is I if you want to really listen to your players and your customers you don't do that on on those platforms anymore. You do that on Reddit. You do that on Discord. And yes there's going to be some nuance on on Reddit and and Discord. And I think, you know, if we were being kind of really having to summarize it simply, often we hear that Reddit can be a bit more negative and Discord can kind of feel a bit more

homey, but it's actually unique per customer. You know, it depends who's do you own your subreddit, your main subreddit, or has that been set up by a by a player or or a creator? What what's their culture like? How have they tried to set up that subreddit? And it's exactly the same with Discord, right? like these are communities and and the values that you set up for that community will bleed into how people engage and talk and and react in them. And and so I think yes, there are some nuances between, you know, Discord is the real time bang bang bang players come and tell you first. Reddit is moreformational and and and threaded, but it's also going to be unique per per per subreddit, per per Discord server, and and by game.
Well, that makes sense. It's definitely like a lot of context to process for both you and and the the customers and of course players themselves even sometimes to process all that context. Uh it brings me the next question which is like how involved are game developers in this process because you know you're

talking about the reporting that you're giving them all the stuff that you know and that you do but like where's where's the game devel like obviously game developers have the community managers but where else are they involved in this process other than maybe receiving the data.
Yeah. I mean I I I think I mean just first of all from an actual community perspective something that gaming does do incredibly well uh and others are following suit I think we're seeing technology AI SAS are learning from gaming actually and following this is often we do have developers and the team in communities directly talking to to customers and and and to players which uh is something uh that that isn't done in in every vertical and really humanizes the the product and the and the brand and brings closer to customers. So like just the amount of our customers and very you know we some large you know large customers with with big games like in there talking to

players is is amazing. Um I think the from our perspective from the product perspective one of the things that we heard a lot Devon which was a challenge was product teams and engineers and live ops really kind of focusing on qualitative data right we've got telemetry data I can see what's happening in the game I can see what's happening in the product uh that's data I can trust I'm going to rely on that and so they were they were kind of undervaluing some of the qualitative player voice community data and I've kind of talked about some of the ways that we're that we're fixing that and giving them more confidence in in the waiting and the context of of of that data so that so they can feel that they can work with it. But I think where we're seeing really exciting evolution within studios and publishers is when they're is when they're bringing those together. Is when they're bringing qualitative and quantitative data sets, combining them, and then internally they

need to work on how they're going to weigh both of those different data sources and what they're going to what they're going to listen to. And obviously, you know, telemetry will will tell you kind of um kind of what's happened, but it won't always tell you why. won't it won't always give you the context for why did we see that shift in the data in how people were playing the game and often you can find that if you go and find it in that player voice data and so for us I think um you know we kind of expected that it would be mostly community teams that were kind of using our product um we've actually made a purposeful decision to not operate on a kind of seats first model um because we felt as though there was so much value sat in community data that we wanted to try and democratize that and get it into more hands in the organization. I think for us seeing you know typically we have somewhere from like 15 to 20 people within within an organization kind of logging in and using our dashboard and they will be across kind of you know um player experience UX QA um uh game

design often exec teams directly in there and that that's really exciting for us because and for community teams because they're now getting those insights further into the business and people are really acting on this is what our players care about. this is what's important to them and that's getting into the road map and and ultimately building better games for for their players.
I mean, that's great that it's that's hitting more of the people in the company because a lot of times when you're working on a game uh especially in some of those more siloed departments, you really don't have a lot of idea how it's being received, right? You may be like, "Oh, I came up with this great interface." If you're in UI and you're like, I don't really know if people are liking what I did with this interface, appreciating it or this is totally the wrong direction and they kind of rely on a lot of like game of telephone to get that feedback back to them or going on social media themselves, which I don't always recommend doing because sometimes that could really be disheartening for for a lot of game developers to really be like, "Oh no, everyone's just slamming my stuff." especially if they don't have

the sort of like in-depth context that you're building around this stuff to like shape that sort of narrative that comes back to be a little more useful to people. So that does sound like a good approach
and you want to know that and you want to know that pretty quick and not necessarily have to wait for someone to do right. And so that's where not only can you come in and literally ask a question within our within our product and we're going to pull up a response and give you that data. Um what we're also you know spending time working with customers on is you know if you can provide us with context of what's on your road mapap what you're shipping what's coming next understanding the context of your business we can start to just in an agentic world we can start to automate those processes. So, you know, hey, like product team that just, you know, came out with a patch note, we can just automate that you're going to get a report on day one, day three, day five with the feedback and the data that you want. And you know, it's such an exciting time to be building a kind of software business for for enterprise

customers because that type of personalization was pretty hard 5 years ago, you know, and you know, if you're an enterprise business, you don't really want to buy software off a shelf [snorts] in all honesty. like you want uh a solution that understands your ways of working as a business, your processes that is optimized to your business and uh I think in in an agentic world a lot of this becomes possible and customers are customers are really leaning in as well
when it comes to like that sort of context of personalization. um when you guys are taking in this data and you you mentioned stuff around like context around the individual players and things like that. Are you doing stuff around and you did mention segmentation actually uh sort of understanding players maybe uh putting them in sort of segments or buckets or something to be like this is that player that's always super negative. This is that player that's just the cheerleader player. Uh so that there's always like that additional bit of context around the

comments. or this is the sarcastic player really is a great one that you brought up earlier so that you know how to take that particular piece of feedback.
Yeah, totally. Um this is something that comes up all the time as well, Devon. And um we we've done a lot of work on on enabling segmentation within within the product and being able to come in and ask a question about a specific segment and we'll and we'll find you that data. So some of the things that we hear quite a lot are, hey, if we just listen to what our hardcore players, our legacy players are saying, are we going to build the wrong thing? What do the new users think about this? And so um because we're sat on um such a a kind of a large warehouse of data, we can start to segment things like uh players that join the community in the last 30 days versus 12 months ago. Uh, we can pull game data from Discord. So, people that played your game or a competitor game, what do they say about this? Uh, can we only listen

to what people have sent more than, you know, 20 messages and are in the top 10% of most engaged users in the last month? What have what have they said about this? Um, and so that kind of starts to give the customer better context to to understand what they should and shouldn't act on. Um, and then additionally, as you say, we can start to kind of user level start to better understand how individuals within our organization are and and kind of we're building out that kind of CRM layer. Um, so again, we want to give the customer as much context so they can decide if they then want to act on that or not.
Nice. I And you've talked a lot about how to process a lot of this volume of information and stuff, but [snorts] what about game companies that are really lacking discussion around it? Is there is there ways that you're sort of trying to get players to be proactive about discussions? So, for example, like you know, a lot of um I get a lot of surveys from games that are like quantri or whatever and stuff like that where they're surveys or other things trying to sort of like extract sentiment where

that's a very obvious like focus group kind of style of doing things. Is there ways you're able to be proactive to then sort of encourage sharing of information for especially games that are not having the level of discussion that they would want?
Yeah, totally. I mean, you know, if our customers don't have an engaged community and they're not getting a lot of conversation, then, you know, they're not going to get full value out of out of the insights and and so that's something that we definitely think about. And we do have um some engagement tooling within our product. So, we've got customized leveling and we can connect that to to game play. So, we can reward you for playing games and and driving more engagement. So, you know, in the Fortnite community, we've got seasonal based leveling in there. It gives people a reason to come back, gives people a reason to have conversation. And so, we are trying to stimulate that conversation. In terms of kind of automation there, um, it's definitely something that we're actively kind of looking into and developing. I think that we we care deeply about the

community experience as a community first business. And so, uh, we think that there is a huge opportunity to do more user research and to better understand how people are feeling about things. And, you know, a wellargeted message in response to things that players have said or complained about or or kind of raised to get that product feedback. If you can if you can get that to land with a user at that exact moment, I think there's a lot of there's a lot of value in that. Um I think the challenge here is is kind of you know both Discord are very clear on the um the experience they want to keep on their platform and create for their users which I think they do very well as a platform and and so there's a balancing with kind of terms of service and and and their experience but also how how are the community going to react to this. So it's something that we're we're actively thinking about and and kind of talking to customers about. Um but ultimately we've got to make sure that um the the kind of quality of conversation in that environment isn't isn't isn't kind of degraded. I mean on

that topic then like obviously focus groups like serve a purpose. So does surveys and things like that? What type of information or like sentiment in general like makes more sense to be gathered the way you're gathering it versus the way sort of these survey groups and these focus groups gather where they're sort of like structured a certain way. They're kind of gathering certain types of information. And there's certain types of context that comes with that as sort of a thing uh that just you know it kind of shapes sort of what people say and how they say it. Like if you're if you're like I say I'm a game developer and I'm like which which approach should I go with? What types of things makes the most sense to go with your approach versus these other approaches that are more like quantized data or intimate focus group data?
Yeah, I mean I think you're still going to want to have a combination of different data sources ultimately depending upon what you're trying to get back. Um, so you know, if you're trying to do deeper research on a topic that might not be top of- mind for the community, um, or you don't have a convenient way to make it top of mind, right? Maybe, uh, you don't want to share certain

information yet. You've got something coming up that you don't want to bring out, but you want to try and start getting getting feedback on that. Um, you might want to do that in a in a smaller environment. Um, or you might want to talk about something sensitive. you might want to not pour fuel on the fire or bring that to the to the larger community. So, I think that there's still value in um in some of those kind of different um research formats depending upon the insight that you're trying to get out. I think the other thing with some of our customers is, you know, um, you know, in game you might be able to access millions of players, you know, a which some are lucky enough to have on Discord, but you're probably going to have a larger um number of users uh kind of within game stuff. Now, you're not going to get as deep a level of of feedback, but if you're looking for just a hey, can we get a topline kind of AB test here? There's still going to be some value in in kind of using that that larger audience set. So, um, you know, I think just as we talk to our

customers about Discord and Reddit, these aren't a a replacement of social media channels, right? You still need a top of funnel, right? This is this is where you then convert people that want to be part of your of your journey, that want to be part of your game, that want to be part of your product. It's where you start to convert people from fans into into super fans. Similarly, from a kind of insights and research perspective, you're still going to want to have a few different things that you can pull levers on depending upon the the right insight you're trying to get out.
Makes sense. Um, one area that that you haven't talked about that I'm kind of curious if it's an area you guys have explored or you're looking at or or have the tools to at least be able to approach is in-game chat, right? You know, we we have like obviously these external social media platforms, but there are a lot of games with in-game chat of different types. Sometimes it's like multiple channels where it's like like the MMO style of like global versus like guild versus you know individual player. There there's a lot of different approaches there that like you know different types of data and context. Is that something you guys have explored or

interested in?
Yeah, we've actually had a few customers come come to us about this. So, um, we've we've done some some pretty cool product stuff around this, um, and and have a couple of customers that are that are also using our our solution for that and and wanting to start kind of bringing more data into into our ecosystem so they can start to combine that together. I think what we found with in-game chat is there's a huge volume of of data that you're sitting on because often people are are kind of, you know, meet me here or do this or do that. Um and
yeah chat
there's a lot of in-game chat and look there's still some value to be had in there. Um but uh on the whole the kind of more kind of either thoughtful feedback which you might see on Reddit or the I need to come to Discord very quickly to talk about a thing um might not happen as naturally within within game because people aren't coming to a place to have that conversation. people are being thought of often thoughtful when they turn up in Reddit or Discord,

they they they kind of want to share. Um but but definitely there's still signal there and so if we can find if we can find player voice signal, we're going to help you find it.
Nice. Um when it comes to sort of tying that to like different things over time, right? You mentioned seasonal seasonal stuff, for example. Um, like for example, I I I I often will have to be looking at reviews over time and looking at correlating those to patch dates and things like that to kind of understand how things are taken when it comes to context for game reviews. But you guys have like obviously different ways of of collecting that data. Are you doing anything around like obviously it sounds like probably around time based stuff and like events, but what about like things like AB testing? That sounds like maybe a trickier problem where like you don't know which players necessarily are seeing the A or the B and things like that because obviously that's traditionally tied to metrics based stuff, but you're trying to say like there is actual value in the sort of qualitative uh stuff that you get from these player feedbacks. Is there anything you could do around that sort of AB testing, especially like maybe what it's a little

bit longer window because obviously like if it's too short maybe no one says anything.
Yeah, totally. I think what we're trying to do is help make it easier for for customers to do that. So, we've built, for example, a kind of best-in-class scheduler um which allows customers to uh kind of go and post announcements into the community or post polls into the community um and then we bring that data back into um into our dashboards and and into our reporting so it's easy to kind of then highlight and and bring that data to life. Um we also kind of automate um updates and events that are happening in the community onto kind of all of our calendars uh and all of our data analytics. And so you know if we see a spike around a certain event uh or a certain day, why did that happen? We can we can actually click into that data. We can go another layer deeper and find that and find that insight. And so typically what we're seeing more and more is that customers are coming and starting to to do some of those kind of

polls and you know very direct kind of you know we want a response from you community. Um and what we're trying to do is just make it a easier for them to do that and b make sure that data is then being kind of considered in in in the wider aggregate that they can then bring back and work with. I imagine then there's also even like the context of like uh if you're talking about they're being more proactive and looking for feedback. A lot of times you have some of these discords where they're sharing like patch notes, but sometimes those are prior to something happening as opposed to being after, right? Like are you looking at some of these situations where it's like how are people responding to the announcement versus how are people responding to the actual thing in the game and sort of like providing that cuz a lot of times people overreact especially to the new content good or bad react very strongly and then when it actually happens and then over time it may also be a different trend and you may have to kind of silo that individual like feature to having those sort of three different phases and tracking that over time. Is that something you guys deal with or
Yeah, definitely. And this is, you know,

we're trying to approach this in a number of ways. A, that segmentation piece, so we can decide if we want to segment out certain topics, certain users. Um, so we're only listening to to certain to certain folks. Um, in terms of being able to come back and see, you know, let's take that patch notes example, like we're probably going to get the the most emotional reaction within 24 hours, but what's happened on day three? What's happened on day five? And so we actually can now analyze our data down to the hour within our within our product because we need that level of granularity for a customer to help them to really understand how something like a patch note is is is affecting um the the community. Um and then the third thing here is I talked earlier about bringing together that qualitative and quantitive data. So you know players might say something but what's actually happening back in game? Uh, and so we we're working really hard on um helping our customers to to actually start to better understand who are those players

in our community. Are they playing the game? What did they do in the game versus what they said? And kind of bring together that telemetry and uh that player voice data that's going to give you the highest quality signal and insight.
That makes a lot of sense. And that was one of the definitely the things I keep thinking about is like how are you even, you know, validating that these players even play the game? you have so many people that say one thing and do another or say one thing and don't actually play the game or all these things where like there's a certain amount of context for what they're saying that has to kind of come from game metrics about that individual player that you have to enrich. So are you then tying into pulling some of the metrics towards your direction before passing it back as opposed to passing it to them and being like you figure out if this is a real player? It ultimately depends upon the customer and how they work with uh work with data internally. Um so we we kind of can do that we can do that both ways. I mean I think that you know first

things first discord tends to give you kind of a large enough data set that you can you can make some action with. But what we see is that you're then going to have users in in the organization that do want to get really granular, right? You might have someone in your product team or an analyst that that just wants to dig further and really get that next insight. So they do want to be able to connect that player and understand that they can connect. So, you know, we we can do that in different ways for for customers, but ultimately if we can get that account connection going, you can you can really add value to the player, right? Because there's a world in which you can start to reward them and engage them and connect them back in game and Discord has done some really cool stuff with their SDK around this and we've seen Rivals and others doing some really interesting stuff here. Um, but alongside the engagement piece, you're also just going to get a more granular level of insight on what a player's saying versus what a player's doing. Well, it seems like with Discord, it might be easier to do that because of

the amount of stuff that that goes around like connecting accounts seems to be more of a thing on Discord or or different ways to do that, different validation systems around these bots, things like that that support that. Whereas Reddit doesn't seem like there's any mechanisms that I've seen at least to help connect my social Reddit account, which is not going to be game specific at all,
to this game. Is there any way you found to kind of like approach that? I mean, obviously you don't have to get into trade secret details or something, but I'm just curious and like how you're approaching that from the the Reddit side since that seems like the more difficult one to connect those two things. Well, most of those Reddit users are likely going to be in in Discord um or in your kind of email CRM or in other places where you can have a conversation with them and um and look to get those connections going. So, you you know, if you've got your channel set up properly, um you might not have to do the kind of connection on Reddit specifically.
Makes sense. I mean, hopefully you get if if not, I guess the community manager

can always DM them and be like, "Hey, I want to help you with your issue. What's your in-game name?" That sort of thing. like it's kind of
that that doesn't scale that doesn't scale if you've got 500,000 users in your
absolutely well it it does use AI I suppose but uh yeah that's it definitely sounds like a tricky problem so I I wait to see how that gets solved more broadly over time with scale uh but on that topic in general like what do you see as kind of the future over say the next 3 to 5 years barring AI just becoming completely censient or something and and doing it completely without you uh where do you see this this whole business of what you're doing player sentiment, qualitative analysis, social media poll, all this stuff going over the next 3 to 5 years.
Yeah, I mean, I think there's a few things here. I mean, player, you know, player feedback and sent analysis has kind of been extremely man manual without a tool like Leveler. Um, but I think even if you're using a platform like ours, there's still a need for the expertise of a product manager or the user researcher or the community manager to know what it is they're trying to

deeply investigate um, and what is happening in in their community. You know, users are always going to want to be able to kind of dive as deep as possible to really understand the nuance and the detail and to kind of slice and and dice that data. Um, and so you still need a kind of best-in-class analytics tool. But I think what we're really focused on is, you know, the kind of future facing product really understands context. You know, context of what's happening internally within your business. What's on your road map um that needs to be taken into account, what's already shipped that might need to be taken into account. um understanding, you know, how your reporting works internally, what's the culture of your business, um you know, what what products do you use? Do you do you use Jura for your bug reporting? And and once we've got that understanding and that context, we can really kind of map that to connect that data into your systems and your processes in in the way that you want to work here. So I think you know the key theme is having broader kind of

awareness of of what's happening within the organization means that we can we can kind of better build the flows to remove a lot of that manual work and make sure that the right insight the right data is turning up at the at the right time. Um and look these are jobs that people have been doing painstakingly for like for many years. So you know if you sit down with a customer and you say hey like just how does reporting work today? When does reporting go out? Who does it go to? What does it look like? you can really start to understand some of the the the the problems and challenges in that organization, but how you can kind of uh affect that and and building agents is going to remove a lot of that and make you know it's not going to remove the roles. We don't see it in that way. We actually see it as making somebody more of an orchestrator, more of an orchestrator and spending more time where they can get more value. If we can get, you know, one of our um customers at PUBG said the other day that, you know, we're saving them about seven, eight hours a week on what they were spending on reporting before, that that that's a day's work that they can now spend on driving engagement in their

community instead. I mean, that's really high value. That's where they're going to get the most leverage. So, we want to remove a lot of that kind of that painstaking um that painstaking work. Um, I think also, you know, there's kind of an opportunity to better interact with the player base, um, and to get kind of those deeper insights. And we talked earlier about, um, you know, asking players for for their thoughts, for survey responses, for comments at a time where they've just discussed something that matters to them. Um, I think that we are we're really thinking about how we can automate some of that process in a way that still is kind of valuable within um within the community. [snorts] Um, and I think that secondly, it's about it's about speed. Um, speed is such an advantage um for any business, but I think particularly in the games industry. Um, and we've got to make sure that you know in real time, not every piece of of feedback or player voice because you just can't work with that much data. But if we are able to

find the valuable insight and get that to you where you need it in real time, we're going to give your business an advantage. And so those are some of the things that we're super focused on.
Nice. Next, you'll also have to figure out how to get AI to understand memes for for processing those as as I imagine
image generation will will ramp up, but also image recognition. So, that should be really interesting. Uh, I just wanted to finish off with a with a couple quick, you know, rapid fire questions and just whatever whatever uh you think would be good. Keep them short, but uh just fun stuff. Uh, just starting off with like what's the funniest thing you've seen from player sentiment or funnest some something that that stands out there?
All right. Well, we've seen this one a few times now, which is we give a new organization access to our dashboard and a community manager comes in and the first question they ask Lever AI is what does the community think about insert their name?

Um, which
like googling yourself sort of.
Totally. I love that. That's one of the first questions that community managers ask. What do my community really think about me? Um, we'll spit them out a whole answer. We'll give them graphs. We'll give them visualization. They'll get positive, negative sentiment. We'll call out quotes. So, uh, yeah, they can very quickly get a sense of what the community really thinks about them, which is cool.
You just have to put put that right into the onboarding, so they don't even have to type it. Like, we know what you're really going to ask.
We know your first question. There you go.
Yeah. Done. Now, you can move on.
I'll bring that one back to the to the product team. That's a good one. Thanks, D. [laughter] uh what's the the biggest issue on the flip side of that that you've seen across like multiple games that you're helping to solve or just like that you just see coming up pretty frequently? Yeah, I mean, do you know what I think one of the challenges still is I still think there's a bit of a disconnect within this is certainly not all organizations and it is it is changing and I think it's changing fast but there is still a disconnect in in some pretty large organizations around how kind of executive teams are thinking

about the growth of these and the value of these platforms um versus the teams that are actually having to work on them. And you know, by that I I I don't mean that, you know, they're not necessarily giving community teams enough like headcount. I I think it's actually thinking about are we setting up our organization to not only kind of drive the the revenue opportunity that platforms like Discord provide um but are we bringing that that insight into the organization the right way and are we working with that properly? And I think that, you know, in in some orgs, you know, if you're not a nent Discord user and kind of used to some of the traditional platforms, it's it's remarkably, despite being the most important platform to the games industry, it still feels like we're we're potentially early on the bell curve, at least from a kind of really investing into the platform. Um, that feels both kind of exciting to us,

but I can also see how, you know, for some teams it's it's kind of frustrating that they're not necessarily getting that kind of full lean in yet on what they know is is such a core place for listening to and engaging with their players.
Makes sense. Uh, lastly, uh, like, and this is meant to be a positive call out, is uh, what game do you think does a stellar job of listening to players? And it doesn't have to be a current customer of yours. I just mean in general, uh, you think it's a a good role model for a company listening to players?
Um, I think probably there's a there's a couple to call out here. I think just as an organization as a whole, I think that um, Croftton are doing some pretty amazing work um, with with players. I think they've made a very conscious effort to signal that they're going to be an AI first business, but they are really kind of leaning in as an org. I mean, we see their their folks are in our products kind of all day, every day bringing back these insights. And so, what we've just seen is just such a kind

of um intent to want to understand deeply what their players are thinking about and bring that back into the business. And I think that we've just seen that across our company and been super impressed. I think another another business, not a direct game actually, but kind of Curse Forge, which is an Overwolf business, um, again, just have such a deep care for their community. And we see this in the way that, uh, often, and, you know, we've spoken to so many people across their their team there that aren't directly in the community, but just they want to know, they want to be connected to this data, they want to bring it in. I think that's cool. And then and then maybe just a third one like not from gaming because I think we can always look at gaming examples but it's also good to kind of just keep an eye on what's happening in in other spaces. We work with with a number of Google teams and um it's pretty cool to see what Google Pixel have been doing. Google Pixel community for their super fans. You know this is a product that's you know there's 40 million people in the world with Pixel phones. Uh huge huge product.

um they had somebody in their community the other day that kind of gave frustration around what the homepage setup was like on a Pixel phone and within a few weeks Pixel had actually shipped an update to that homepage and could come back into the Discord community and tell a user that they use their feedback to make that change. For for a product and business of Google size to be really listening and tending to a customer like that, it massively humanizes the brand. And then guess what happens? users go, "Wow, they're listening. Let's give them more feedback." And that creates this really powerful kind of loop. And so, those are just a few examples of of kind of games and brands that I think are doing a great job of kind of working with and listen to their communities.
I never thought I would heard that about Google, but I'll tell you what, I'll have to connect you with Samsung so I could give some feedback on my phone.
There we go. We can make it
right to them.
I also I'm kind of curious. Uh just this is just just as a last little thing. When are you going to start uh putting putting bots in internal Slack for your

customers so you can see how they really feel about your product? [laughter]
Good [snorts] question. Um I don't know. We'll ask a few question we'll ask a few customers what they think about that and I'll let you know if they agree.
Right. Right. Well then well then of course you know use your product to analyze what they answer with.
Well you know again we're doing a lot of that with just all the feedback that's coming in. we're not sat within their slacks. Uh I don't know if that'll allow allow that one. We can only ask.
You got to find some way to dog food it, right? But uh I do I do want to thank you for for stopping by today. I really appreciate all the insight. This is definitely like a newer industry, I think, a newer type of product. uh and can't come at a better time when there's just seemingly like a lot of fragmentation and and sort of discourses all over the place to be able to actually get that especially as we move past the phase in gaming where like it seems like quantitative has kind of hit some limits uh and and get this qualitative data I think is is super important. So really appreciate the work you're doing over there and coming in and sharing some some knowledge here. Uh

so awesome. Keep keep it up.
Thanks to you uh thanks to you and the Navic team for having me on.
Awesome. Uh thanks to all the listeners as well. Of course, uh make sure to check out leveler if you can. Uh just so you're you know on the spelling it's it's uh you're you're going to you know level and then LR at the end just so you don't get confused because the spelling is a little bit different than you'd expect. But you can check them out at leveler.com and uh and see what you think. But in the meantime, thanks everyone for tuning in. Hope you had a good time and we'll catch you on the next episode. If you enjoyed today's episode, whether on YouTube or your favorite podcast app, make sure to like, subscribe, comment, or give a five-star review. And if you want to reach out, or provide feedback, shoot us a note at [email protected] or find us on Twitter and LinkedIn. Plus, if you want to learn more about what Navic has to offer, make sure to check out our website, www.navic.co. There you can sign up for the number one

games industry newsletter, Navic Digest, or contact us to learn about our wide-ranging consulting and advisory services. Again, that is www.navic.co. Thanks for listening and we'll catch you in the next episode.