Session: AI & Automation Hacks For Ratings & Reviews - Anatoly Sharifulin, AppFollow.

[Music] We've got uh up here um from uh Appfollow to talk about AI and automation hacks for ratings and reviews. Kicking us off and giving us intro to the NextG UA track, it's Anatoli Sh, excuse me, Sheriff Fulin from Appall. Yeah.
Okay. Hello everyone. I'm Anatoli uh from AO. I live in Finland almost nine years but didn't speak in Finnish. Sorry, but I speak in English. Uh in the past I was ASO expert. Now I'm focus my focus on AI and

automation especially for uh players uh experience and it's in better how to get uh and extract more uh actionable insights uh and today I'd like to talk more about ratings and reviews as I see uh not everyone care about uh ratings and reviews and how to get real actionable insights for your product for your game and I want to fix this gap if I can. Uh okay let's start. Uh you could see some popular games uh and see first line it is uh star rating and you could see that uh Android versus iOS like quite difference uh and rating it is visible for everyone even like my mom can go to store and see average rating and it's better is it like 4.0 zero below or 4 4.5 as a best-in-class and make a decision about install right and the thing that uh we as experts know that uh average rating by stars it is not the same what you could see in like some uh

platform as the average rating by reviews and if is difference more than one star it is something uh crucial something important because it looks like product know so well uh retention know so good and need to be uh care about this stuff. If it is like once again more than 4.0 that sounds good. Your game will be perfect. Uh and you could uh acquire more and more uh users. Uh this is example of uh easy brain uh games. You could see that different games even like really popular and legacy ones with high volume of ratings could have 4.7 even uh in Google Play. And this sounds really amazing. They like best-in-class. But I what I really like that they are new games last line uh 4.7 as well that they really manage and know how to cook I would say uh like quality and uh it will help them uh to acquire more uh users. If we speak about

reviews and content in this reviews first of all about volume uh I was surprised when I see that uh Call of Duty War Zone in like months of worldwide launch they have 240k uh new reviews. It sounds a lot of insights. If we remove all positive stuff and keep only for uh like team but everything else really important for player experience support community even marketing teams and the thing uh if you uh use WhatsApp and JPT of course we use uh they have 300k every month reviews this is like really extreme uh number of uh reviews potentially your game could have severals or several uh 100 1,000 reviews But doesn't matter because uh this review could be analyzed and this our first stop uh when AI could help and understand better okay what is what they speaking about summarize try to

categorize uh translate to English if we don't know all other languages and this is a real example how you could uh get insights and don't spend 40 hours to read all 4,000 reviews just for months or just for week uh and get insights for your product team or for uh someone who make a decision about changes an idea to improve quality of the game and make it uh really uh good uh and taking stuff. If you have uh AI and say okay please detect better monetization issues, ads issues, feature request uh or something like that, it will be easy to uh split all reviews by right topics for you and get the same stuff with AI summarization. Okay, next and this is some our research what we made that uh when you uh or like we leave a rating we could uh also leave some comment and if five stars more than

50% in sounds really good quality of the game or app uh if it is less and this example of new games by cyber that they have uh a new launch and at the moment now 40 it is like uh close to be best-in-class but the thing that uh if uh you have not 50% please check it out right now what users uh write in reviews and potentially it is really good ideas for your next releases and uh where should we focus especially if you have quite high volume of reviews in some uh review segments how we call it for example feature it what visible for everyone uh or like when ratings drop for example first of uh I like the game at five stars but after one month or after release of after some problem with inapp uh purchase I like drop it for one stars and this type of reviews really important and if you care about it it is

really really crucial for uh your like game future and if we compare app store versus Google play and it's better what is behavior uh of users yeah we found some differences for example It is real examples of one title by App Store and Google Play and like green one you could see like more uh like one word in green like in positive sentiment but in iOS you could see like yellow some mix it or negative and the thing that in Google Play how we discussed before more volume but also more short and positive reviews if we compare to app store but in app store you could see really long reviews with a lot of insight Right? And this is what sometimes really interesting and need to uh focus on not just like uh skip all positive one word uh reviews after ratings and reviews we can go to

replies and I understand should we reply to everything where AI could help and first of all uh current benchmarks uh what we know from US market that 20% in average current reply rate so not all reviews require ires to your attention. Uh if you like based on benchmarks but if customers start to reply 40 50% it is really good uh uh threshold and some our customers could have 100% and you could see on this chart that doesn't matter what the size of your portfolio or volume of reviews uh you could achieve roughly 100%. Yeah and it could be automatically or it could be manual but we will discuss a bit later but the thing where we see now higher reply rate in such of categories as like board kazina puzzles and so on. So if you have now this type of game potentially you need to uh be more focused on replies uh and try to reply faster to this type of

comments. What about exactly replies and how you could forward because to the community or to help center the thing? Yeah, we highlighted two examples that you could say okay uh it is not possible like discuss here but please go to our discord community uh where we potentially have some live op stuff or what will be interested for you. So this is exactly time when you could talk uh to your player to your customer and share some interesting information what potentially he or she didn't know. And second like really simple stuff just forward everything or like negative to your health center. It could be zenesk or help shift when you or like support team community team work with all tickets. And the thing that now we see that some players and potentially you uh has complaints concerns why like templated reply. It is not really what I ask or what I shared in review but for

me it is please go to our help center. One thing is this link not clickable. Another thing that it is not what I ask and how to solve it and this is our next stop about uh for AI. AI really could help even on automatically or manually to rephrase easily. Yeah. How JGPT now can provide this capability. Please rephrase please improve uh message or or translate to uh user language. It is easy. Sometimes it could be in one click or automatically. If you speak about automation and aftermated replies, yeah, based once again on some review segments, four stars only positive or like less contain some words. Please reply uh this type of uh messages, replies with some AI instructions. And the thing that now AI could really replace templates. If you're familiar with like support community stuff, they use some

templates, but now you could have custom instructions how you want AI should uh replace in in the end human support. And what we see at the moment after several years when we go deeper in automation that uh now hybrid approach really works work works at the moment. So you try to automate everything what easy one I would say. Yeah. Positive stuff or some one topic or one tech reviews uh using automation in instructions. Uh but everything else when really complicated cases for where you want to have human touch for your agents and uh AI could help. And second thing if you have some automation and especially with EI instructions uh you should have some like premoderation stuff when uh reply generated but your support team or community team could check everything and okay this is like

positive this we could improve because not uh super clear because AI uh really good but we'll be good uh next year and next years and the thing that now approval mode could help you to manage as a human really high volume of uh reviews and tickets. And one more thing about automation what really need to mention if we speak about reviews that some reviews could be with like if you know a lot of images with like some poo or some other stuff or it could be offensive or real spam. It is not super visible but sometimes it is visible for your team for you and you want to remove it as like hygiene stuff. So uh feature what available in consoles in app store and Google play rep of concern could be automated as well and this is the best uh uh result that 75% deleted from 1,000 reviews uh that if you really know that for example kids

could use this type of uh uh text comments for for your game you could easily remove it and stores could confirm it. And last thing what I want to uh share with you how to measure all this EI automation or hybrid uh approach thing first of all it's part of uh support KPIs that how your fast to reply what is reply rate what I already mentioned but the thing that yeah you could u measure uh after reply rate uh or you could measure what we discussed in the first slides about average rating because idea Ideally, especially if you're on soft launch or you could you have small game at the moment just a growing bit step by step you could change rating because from one star to five stars you could improve it when you react faster reply okay it is our bug but we will fix and you will share with your product team

this issue or product engineering uh team and uh the fix will be in next release. So this is overall thing. Yeah. If we measure success of the game and potentially uh like revenue, how we uh faster react and how we could improve uh reply rate and uh average rating uh it really help and some some numbers about uh uh benefits. Yeah. What we see now that first of all you could optimize time of like human time of your team. Second, you you uh can uh save some money because potentially you don't need 10 or like 20 support agents uh or someone who work with this insights and share the everything automatically or semi-automatically uh with product team. So key takeaways uh collect all player uh feedback including uh reviews, rating and reviews

from stores. Focus on actionable insights. uh prioritize uh manual replies especially for complicated cases after as much as you can uh and ideally yeah you could uh get better customer experience and more revenue that's all thank you so much for your attention any questions if you have any really good thank you very cooperative overview so do we have any questions from the audience about uh about Anatoli's talk or what he said about uh about UA and analysis Yeah, one here. Maybe we can bring a mic. Thank you very much.
Uh, thank you. I saw one bullet in there about how uh you prevent repetitive replies and that there's something that is somehow paraphrasing the concern that the complaint has. How does that work? And I guess in practice like how do you actually prevent like someone seeing a bunch of reviews or yeah someone going to the page seeing a bunch of reviews

and like the a very similar reply to every review.
Yeah. Uh if we speak about technically solution uh this thing that if we in one batch try to work and reply to I don't know like 10 or 100 replies the thing that in this batch reply should be unique. Yeah. or as a like one idea potentially you could have like I don't know 10 20 different variations on one template or templated instructions what will generate really unique thing so the message will be how yeah we shared before uh oops one sec yeah this one this slide uh please go to for example to uh support yeah but phrases how you appreciate why it is important or what you could do it could be differently because rephrasing in AI really good at the moment and the thing that you really need to manage not to like one reply and uh it will be unique in one review but

for set off absolutely if you have like few reviews and one per day for example potentially it could repeatable yeah and potentially you need to like solve it differently but in that case as I know sometimes manually uh manual agents works better we don't need no time to spend And because when you have uh less reviews, every review matter.
Thank you.
Very good. Thank you. Oh, one down here.
Yes, please.
Thank you very much.
Thank you for the talk. Um, is there any relation between uh the ratings and reviews and the app store optimization
and ranking of the app or things like this?
Yeah. Yeah. Good. Uh first of all uh average rating how I mentioned on the beginning slide. Uh one sec. Yeah. Where's this benchmark? Yeah. Yeah. Uh this is what already several uh times approved uh and confirmed by different players and platforms not only

we what we do uh that uh 4.0 it is really threshold because it is just on like human and saying oh it 3.9 or less. this is what I don't want to spend time because I can choose other competitors what have like really best quality from others and this is already these users already approved it and this is the thing what could impact but I don't have information how much conversion rate could increased but absolutely if it is like less than for for example two stars dramatically down yeah uh one more thing what I want to share that it is not in current slides but in appendix example of real case of art puzzle that uh how they change average rating because if you think that 4.5 really what we want to have
so how we need to work with reviews and insights and react quickly so they see here that uh something like dropped

and when they start to uh quickly uh iterate and send better all issues and several updates what they like two two weeks or two months they to understand the problem. After that they every month improve improve improve and what we can see this is like Google Play because Google Play rating have like really complicated for formula what includes new uh ratings and look how the rating increase after several months and this exactly uh example how you could uh potentially improve your like conversion rate and be care that uh quality of uh uh game is like enough and marketing team could uh generate more and more for example paid traffic. Yeah, because here they they lose some traffic because some other puzzle games uh with higher rating on the top. And last but not least thing, this is what Google Play confirmed several times in their documentation that uh they count uh

crashes in error sorry some other metric what related to crashes. Yeah. And the thing that if you in reviews in especially red see some crashes, you don't need to wait when Google play depitize it. Yeah. And react faster. And this is the thing that because without the if you don't have uh issues in your game uh also you could be feature it because sometimes for them it is like quality threshold. after that okay you will feature it more on some uh category or uh games uh placement
so this is not pure about ESO but overall traffic what exists in the stores
that's perfect I think we that was a great question and also yours as well thank you we have to leave it there because our next speaker is ready but a round of applause please for Anatoli for Yeah. Thank you. [Music]