Session: Player Value At Scale: Automating Replies That Actually Help - Jonny Hair & William Smith.

Okay, next we're moving On moving on fast to um a talk about player value at scale and this is from uh Johnny and William from Taco Baka which is one of the greatest companies obviously who make fantastic uh games. So um yeah welcome to the stage guys.
Yeah hello everyone welcome to our talk today on the practical AI track player value at scale automating replies that actually help. There we go. I'm Johnny here. I'm the

senior customer experience manager at Tokaboka. I've been with Tokaboka for five plus years and prior to that working uh with Mojang and Minecraft in support.
And I'm William Smith. Uh I've been at Tokaboka for over four years now. Prior to that studied game development. Uh today I'm the live ops marketing manager, but throughout 2025 I was the customer operations specialist and in that role conducted what we'll talk about today. So, if you've never heard about Tokaboka before, based in Stockholm, Sweden, we're one of the leading mobile gaming studios for kids. We create playful spaces, playful spaces for kids, for them to have fun. And our promise to players is to create a safe, fun place where players can discover who they are in the world. So, we have a 15 year history. We've developed up over 50 apps and games, but our most popular game by far is Taco Baka World, where we can have up to 60 million monthly active users in a given month. And between 2022 and 2024, we saw 75,000 reviews. So,

most of our players are kids, but we have players of all ages. So, and we have a really strong social media following. And all of the collaborations you can see here, they're from the last year. So for players, it they use Tokaboka world almost like a creative tool and there's some fantastic storytelling and role playing. Uh now our talk today will focus on the following question of which we used appollow to explore which is how do we manage reviews of scale and maximize value uh for our players. Now we've learned a lot throughout the year and I hope to share some of it with you all here today. So we're going to go through what our main challenges were. the inherent value of reviews, detail the 2025 strategy and execution. Uh go through some top tips that we accumulated throughout the year and finally some key takeaways. But first, for a bit of fun, does anybody here happen to know when the first review in history was written? Johnny, do you happen to conveniently have this information?
Conveniently, yes. Uh the first

complaint in the world, the review was actually way back in 1750 BCE when it was written on a clay tablet. And you can actually see this today in the British Museum. But fast forward to now and the tablets, they're a little bit different, right? And for kids, app store platforms are an incredibly empowering space for them as consumers. For many of them, it's the first place they'll get a chance to leave a review. So, it's almost as if they have a pocket full of stars and they can distribute it to their friends however they choose. So what we'd like to do now is just go over the context of our main challenges that we've faced. So as mentioned with Tokaboka World, we see up to 60 million monthly active users in a given month. And between 2022 and 24, as mentioned, 75,000 monthly reviews on average. So we're talking about a really big scale here. And between 2022 and 2024, we have dealt with some difficult

technical almost architecture issues and we've really seen that impact our three star rate reviews and ratings and below as you can see here. So for us also we we've discovered that due to this a reply to review strategy manually it doesn't really scale well and the biggest most important thing it doesn't provide the most value for our players. Now even if you are able to reach all of your reviews under four stars which is extremely difficult to do at scale but let's imagine uh the platforms today still won't give you the insight to understand what happens to that review after your reply what your impact actually is but appall does and you can see here looking at our uh semantic uh sentiment tags uh the impact that our replies were having. The issue that we was facing was uh we only had partial solutions in place for the affformentioned architecture issues and players will speak with their stars. I mean, matter of fact, you can even see one of these. We were losing a whole star with that automation, which is not

great. So, it's a bit of a conundrum. We knew we only had partial solutions for our players that required much more complex long-term solutions. So, if we can see that our replies are actually damaging our rating, should we even do replies in the first place? But let's zoom out a bit just to understand the value of reviews. Hopefully, it's pretty obvious to this room, but monitoring app store reviews, they're incredibly important. Monitoring them is non-negotiable. I mean, that's how you understand how well your activations are doing, how successful you are. But failing to engage with user reviews, that shouldn't be taken lightly because that can affect how you are featured on App Store platforms. So, for us, this conundrum, it was a bit of a no-brainer. We needed to find a way a path forward because our players needed the help. And for us and for our players, making them feel seen and heard is incredibly [snorts] important. So to summarize, of course, listening to

player sentiment, that's a key to measuring your activations and understand how you're going. But increasing your reply rate at scale, monitoring the reply effect, that is the key to really maximizing the value for your players with your reply to review strategy. Now, we're still on this journey to this day, but we'd like to share with you how we adjusted our focus heading into 2025. Uh, so again, our objective was to maximize the value for our players with automating replying to reviews. And there were four core pillars that we identified where we really needed to challenge ourselves in order to achieve this. First of which being we needed to fully commit. Previously our focus had mostly been on increasing our tagging and improving our reply effect. But this time around we really wanted to go for reply rate which is the actual quantity of re reviews that we're automating and reaching in the first place. Uh now this does open up a risk though, right? What if we're responding to tens of thousands of reviews every month only to find that we're having a negative impact? And again this is something we had already seen before. But instead of letting this

make us more riskaverse, we wanted to to challenge ourel and try to find a solution that satisfies both requirements. And that takes us to the next pillar which is granular monitoring. Uh reply effect and rate goes hand in hand. So we quickly realized that we needed to implement an experiment-based approach where we could implement automations one by one uh through care and close uh monitoring and monitoring that would allow us to see the effect of even the most subtle changes on a per copy automation and category basis. Um and then that takes us to the third pillar which is high accuracy. This was the greatest strength that we had with our manual uh reply strategy probably the the the one the only one and uh this is where if you don't address this the wheels can come off the wagon very quickly. You can blink and suddenly find you're not helping your players anymore. Uh so for us we need to assess are we tagging correctly? Are we assessing and identifying issues accurately? Uh do use approval mode where required. Uh what was the next one

and then we can get on to only evergreen when would we actually yeah
uh yeah so for evergreen when would we actually consider an automation to be evergreen to be done and finished? Well really we would never actually set and forget anything. Uh but you can easily get stuck optimizing something repeatedly over and over again until you just have diminishing returns. Uh and so we set uh for all of our automations a set of parameters where we would consider an automation to be of high reply rate, effect and accuracy. Uh and then once that was the sheet, we could move it out of our closed monitoring and move on to the next uh automation, the next experiment. But it is worth addressing here though there are some automations that you'll have in place. We'll find there's nothing that you can actually do to attain a positive reply effect. This might be issues where you have only partial solutions or fivestar reviews. And so it becomes kind of a balancing act between your reply effect and rate or even versus the two. So these were our targets for 2025. Now your reviews are intrinsically an extension of the game and the game

experience. And so for us we decided to look at the product road map and combine that with our experiment based approach to set the year ahead. So for Q1 and Q2 we had fairly modest humble targets. uh the product roadmap at this stage wasn't addressing the same uh pain points that we wanted to address within reviews and so we decided to instead focus this period on just setting a solid foundation uh to get the rest of the year running. And then Q3 though here those improvements would start to happen where we would start to see that uh uh those improvements for the more negative one to two star reviews uh be put into the game and uh we could start applying those automations to to kind of go with that momentum. And then Q4 here at this stage uh we would address uh some of the more player wants and wishes to introduce new features that would improve the more three to four star range reviews uh some of the more already positive experiences basically. So what were our results? Well for Q1 and Q2 it held up true to our predictions. We were able to achieve our

targets primarily through improved tagging and uh specifically through semantic tagging. Appfollow provides this machine learning based approach to tagging which uh basically just nets you the perfect uh deal of both greater reply both a greater reach but also greater accuracy which is kind of a good deal. So we took it going into Q3. Uh here small changes to the product roadmap uh made it so that we it diverted from what we had initially made our bets on. And so we decided to adapt though we had a good agile system in place. And so uh we instead decided to just address and look at okay well what is the live service right now? What are the positive and negative player experiences at the moment? And how can we address them? And through iteration of copying qualifiers we were able to attain some pretty great results. a over 20% jump in fact which is uh pretty nice. We take that. And then Q4 at this stage we actually had a pretty solid setup. I mean we were able to reach the majority of our reviews. Uh we were identifying issues accurately getting players relevant direct responses. So

what's left? Well, we wanted that 70%. We wanted to hit that target. And so we identified what the last missing link really was, which for us was uh low confidence and low accuracy reviews where we couldn't quite assess what is actually happening or what's the issue that the player is experiencing. Before this is something that we would have ran away from in fear of having a negative reply effect. But this time around, we decided to instead adjust our content strategy to allow for workarounds where we could still help the player and get the help that they need even though we can't identify what they're experiencing. And that was enough. That was enough to get us through that final threshold and to reach our targets. So some top tips then uh from this year which might be relevant to you especially if you're already managing reviews or indeed are considering specifically appollow. So first off get your tag coverage in place. Uh your tag coverage is I mean it's what you're doing is ultimately identifying reviews right you're trying to identify issues and you want to identify accurately because that's what enables the rest of your automation to really perform. Uh semantic tagging

again is really worth shouting out here. Being able to take your more rudimentary keyword-based tags which are typically a pain to try to iterate on and instead make them into these custom data sets to to identify and cast the net much wider and more accurately. Then once you have your automations out there in the wild, extract everything that you can out of them. Each and every little one is essentially a a little puzzle piece that eventually over time will hopefully form a really nice beautiful painting, you know. And so you want to extract everything you can out of them, especially over time as a uh status quo builds for your reply effect and rate. And then we still do manual work, but it's not the same. It's much more productive having your agents manually review automations as opposed to manually uh reply reviews one by one. All the great data that we've had through Raffle has been super, you know, that's that's what's enabled a lot of this. But quantitative data is always just going to be one piece of the puzzle. So having that other side of the coin has really given us that peace of mind and helped us nudge that reply effect especially in the polishing stages.

But what about copy? I mean ultimately really all that we're doing we're sending a reply to a player. So it's pretty important right now at Tokaboka we have some pretty uh strong tone of voice guidelines. So for us what made sense was to uh implement custom human authored responses and only create a handful for each automation. Now you might ask what about repetition? What about localization? Very astute. Uh, that's where AI comes in. That's where Appfollow's functionality in terms of rephrasing and translation has really helped us out so that we can send these fine-tuned, polished responses and have it reach a player in their language without repetition. And yet again, copy is as essential a component as anything else here. Keep monitoring and iterating because this is, you know, this can single-handedly make or break your automation. But uh what's really enabled us uh to do that to monitor and iterate so effectively is Appfollow. So some shout outs are due to Appfollow. They've been an incredibly helpful vendor uh

throughout this entire process from day one uh getting us that granular data and tool sets that we need to be able to achieve this. You know other you know other vendors you know you might make a feature request maybe hear back in half year hopefully but they were always there for us in day one helping us in the here and now to get to that next level. Now, we mentioned earlier that we would only run an automation if we're really confident about the data. But we actually remove this as a parameter because one, we found we really needed to get out of our comfort zone and this was one of our biggest learnings. So instead, what we did, we leaned into the inconclusive data and we used more general responses, more general copy. And by doing so, we could cast the net much much wider and we could increase our reply rate dramatically. And as you can see, we saw great effect with our reply effect. Of course, with one-star ratings and reviews, they can only go up, but we saw very, very good results

with up to three stars. And most importantly, too, it's channeling your users to getting the help they need. Uh, one thing too connected to it, put yourself in your users shoes. So this is a and think about how you can synergize your support resources. So this is a pretty simple example but you can direct someone to a landing page of all your articles which is fine but it can become a bit overwhelming for the user. So instead you can consider having a master article latest bugs latest things they should be informed of. Then they feel like they're in the right space. They're going to get the help that they need. Now we're just going to quickly coming to the end summarize our key takeaways and key results for 2025. Over to you.
Yes. So some big numbers. Uh from the beginning of 20 2025 until now we saw a 95% increase in reply rate plus45 increase in reply effect which is almost half a star. 681% increase in total reviews replied to year-over-year and almost 27,000 stars added to ratings after review reply in the second half.

Great. And to summarize and add to these uh key takeaways, firstly get your house in order with your keyword tracking because that's going to be what unlocks automations and helps you drill down in your analysis. Now, and for us, I think the hypothesis proved correct that automating replies, it did assist us in scaling, but also more importantly, it provided much more value for our players. And of course, get yourself out of your comfort zone. really push yourself on the reply rate and it's about scaffolding AI on top of your human processes and monitor monitor monitor can continue iterating because it's that uh human touch iterated with data and with an AI that just it's a beautiful combination and it works really well. And lastly, app reviews and ratings. They're not just about your app reputation, particularly for kids. They see it as a direct channel to communicating with your game and brand.

So for us, we're really proud that we really have maximized the value for our players here. And for us in the future, the next steps to explore some features that we haven't really unlocked yet, uh AI replies and really getting us out of our comfort zone even more and our knowledge knowledgebased feature with app follow. So really synergizing our resources even more and just continuing to build and maximizing that value for our players. So, finally, last message, you take it from here. Please reach out to us. We're very, very easy to find. Uh, very easy names to find on LinkedIn as well. Um, we would love to hear what you're doing. We would love to hear if you're using AppFollow, how you're using AppFollow, and uh, yeah, let's keep up the conversation. Trade notes. So, thank you all very much. And, uh, yeah, have a great day.
Thank you. [applause]
[music]