From sentiments to insights

[Music] [Music] I'm honored to have this last session for today my name is Marco Swatara I'm from quantum shake mobile developer that was founded in 2015 and today I'm here to talk to you about data analysis and the insights that we've been getting at quantum shake while working on our titles and also in the mean time we also have built an in-house tool for doing analysis of top grossing apps just to gain and market insight and understand

what's going on on the market the current situation and also to see what we can learn from other teams what is great on their games and what is not and today I'm not going to talk to you too much about details of sentiment analysis of course afterwards we can discuss about the finer details about topic extraction entity segmentation clustering all these but here I'm talking about more about the insights very quick intro about myself I come from a marketing background I used to do business consultancy sales I didn't see two games in 2012 in 2012 was a turning point in my life I went to the Brazil eventually to study games and on the mean time I started working on a consultancy work for the private and public sector and even nowadays I work

as a data specialist for the public sector but I focus on my games company and that's my passion so talking about the timer should it be running at the bottom it's not running so I was thinking thanks I to present to you one question while paying attention to the presentation think and ask to yourself to positive and negative reviews have an equal wave I want to use the challenge your mind set about it actually to try to push yourself to think both sides and the context as well because in some cases there might be a teen line firstly I want to present a little bit about top crossing today what's out there

top crossing 200 category top ten games in that are responsible for 50% of the top 200 crossing revenue it's a huge portion it's really really lucrative and how some games are so really well performing while others are not what is they're so special about them that's actually what really really made us as well that quantum shake to be more data-driven and want to analyze what's out there to understand the market because if we want to perform well in the market we have to understand it well so we started to check it around what's out there and we found out that the main of the game there they are for just rated four stars of course their quality products it's obvious that they should be aiming for that they should even go past the market but on average the median is four star rating and another thing there is

truly obvious that they are social by nature nowadays there's not even a slight chance almost to get to the top crossing if you are not social social features are so important the best forming games become best performing games because people talk about them they want to share about them and oftentimes they are in fact programmed and created to be enjoyed in a larger scale than just for single player experience so they're socially engaging entertainment experiences interestingly though if we think about the marketplace 67% of the top crossing list our games actually so games are ruling and if we think about the history of top crossing list early days Apple actually distributed a list of top crossing for

each category we analyzed it back then but recently in the I think it was 16 or 17 June this year Apple pulled out that so now they only provide a listing for the 200 general listing for 200 applications but majority of that are games on the earliest title actually dates back to 2010 that's 70 years on the top crossing list that's crazy and that's really interesting as well and the most recent one was exactly three weeks released three weeks ago released on its Rovio's new title and we first match that means that basically depending to get to the list it's really really difficult but you can get there but it requires a lot of effort thinking about the revenue on the App Store's available nowadays on Google it's 90

percent games on App Store it's 75 percent games so games aren't generating a huge income how to tap that on how to make that well we first made an analysis of the sporting genres that are performing well and also to see what genres perhaps have slots where we could target gaming we found out similarities of course because most titles are released on both platforms interestingly though you can see a little bit of a different kind of wave in the statistics but that's natural due to the fact that on Apple you can select a couple of categories per game while on Google you're restricted just one a single category so you have to be really really careful about that you choose the correct category for your title because if the expectations for your game are wrong it can make or break

your game so basically you have to be really really sure about the identity of the title so going to the sentiments if we see about the top performing top performing applications on App Store we did the research on the US market and we started seeing what are basically the reviews about they are often just very very brief comments about like ok this was good about the game or news flow just like okay I enjoyed it but nothing like a really really strong sentiment so major the larger of the reviews are just like low information flow level information but then again there are some that are really really highly

sentiment rich on those oftentimes are actually the most important share of the clients actually for the developer because those people are so wasted that on the mobile they actually are putting the effort to write a really really long review of the game discussing what are their feelings about it and our work is actually to do help we have a tool set that actually checks it out immediately rules it out and we can just see which is good with his bad on those I picked up the tip top 10 negative and positive sentiments and it's probably not surprising that the negative ones that are most of the time appearing are related to disappointment with something that is not like according to our expectations I later on touch upon that the psychological you is really

strong people do not like to see unfairness in a game so sheeting is bad glitches on crashes that take away the enjoyment from the game that is you know but the positive thing is that I know it is that there's quite low portion of reviews that are negative so we will say to go it to the top-grossing you have to have a top-notch product because there's really really low sure but you also see that there's not much of people who actually go on Lord if the game is good they oftentimes do not go there and write a detailed review about how what things are expected a very good about it it's really really rare so to understand better the market we started to dig deeper and we started to see about the ratings itself and we found out that okay if we take all the reviews combined five-star reviews at they're still 76%

more likely on a top grossing game but interestingly the medium regulate is only 15 verse it's quite brief a sentence or two and interestingly though there are also those really really really long reviews that actually spend over a thousand words think about that you write on a mobile device thousand word review that must be that you have some Western interests are you you probably are a hardcore gamer who has played spent money there must have been maybe something some patch might have broken it those kind of things you want to see immediately and lots of data mining companies are doing this but what why we are doing this is that we notice that sentiment analyzes on games is something so context sensitive that we need a pair to a system that actually can be customized a lot and can be

improved together as a industry so we have been considering about releasing our work as an open source effort as well but later on I can discuss that with people who are interested in contribution but let's go to the ratings we took that median review length as a divider between short and long reviews and there we put those to be on the same scale so you can see that interestingly though the share of five-star reviews is much larger on short reviews but it's also so that other reviews are much less likely to appear their portion is very very small so what we get there is that often times negative reviews hold more content value in themselves while long

reviews and are often times negative ten in nature but they also what we found out is that oftentimes people express both sides they express what is good and what is not so you cannot only see the rating number you actually have to do the sentiment analysis on seeing that but there are some common pitfalls the same very same pitfalls that appear on the game design process when you're designing a game you have some bias in the team but you usually might be taking psychology in the claim play thinking about the confirmation bias which means that people actually always focus on what supports their preconceptions and this is extremely dangerous also on the data side because if we are doing data analysis we should evade bias at all cost and this is why we have to go for

artificial intelligence machine learning tools that can help us but still without forgetting the human component because if we go completely automatic then there's really really big probability of having systematic error as well and then there might be biased because the programmers did not see everything so clearly early on so they might have made some flaws in the there's also expectation bias so this means that we only know this today that that is in line with our expectations so basically there might be signals there might be outliers like I said about those really really long reviews what if we only pay attention to those then it's also a risk we have to see the whole picture and that's as well a really really important thing that we see about the availability heuristic however the heuristic is what I consider one of the most important things in the

game this time because it's actually the human nature and tendency for overestimating the likelihood of events that are more recent in memory what it means it means the probability that if let's say something happens in our near nearly like for example all the time with the repetition in the gameplay people see that that's happening with really high probability but it might be later on not appearing at that tyro completely but since it already was learned that that kind of a pattern of cured ones in the game that it could happen again this is used a lot in the gambling games for example so people have a feeling all the time like ah they're actually making more money and winning and winning although sometimes overall the house always wins for example like for example in roulette so in this canary oh if you take this to

data sciences as well hoo-wee often times at least what we found out on the commercial solutions we take the industry standard metrics we focus we do really important decisions based on those and oftentimes they are good as a guiding value but we also have to think about the context and also see sometimes that could there be more information available our reason for developing own in-house tool was actually that because the nowadays online is full of data there's a reddit forums the different review platforms there's so much things that oftentimes when you're in designing a game your team might be discussing some things that has been have been discussed in other teams already maybe there's already games published and maybe there's already customers who have

already expressed their ideas about those very same topics and we wanted to bridge that gap so this has been like very very interesting case for us since we've been doing those games and we know it's like in some cases that we actually managed to found out making some some very same mistakes that some other teams have made and having read it we've been able to be more efficient but seeing like computer doing our non bias analysis I mean this is like the normal portion of my presentation but what I want actually to stress now is now the first question that I presented to you guys that was about do positive negative reviews have the same way this is just inspire discussion now because I want to take some questions and especially if you or your team have been using some

solutions that will be brilliant to hear about use experiences as well and if you want to contribute and start building a really really high level thematic context-sensitive semantic analysis for game industry get in touch with me and stay in touch with me anyways because I always like to hear best practices from industry and trade some disk ideas what we've been doing so thank you very much and if you have some questions I would love to take them now so are there questions I almost thought that you put your hands up yes just a second you know I just have a small question

most of the death act like on those top things they buy and sale the likes and reviews by botnets putting that like effect in your data and analytics as well yes for that reason we had about the median length as well and also about review patterns like basically outliers you can do recognize used reviews and if you train the machine to rule them out you can actually take subsets of data so that is a very very good question and that's actually a topic and a really difficult challenge because everyone can do different kind someone could spend the reviews for example with longer how would you say well much longer revellings one interesting effect also is that the reason why there are so many short reviews is also probably most likely to do most of the top crossing games applying also the after a while of

the player playing the game you usually are presented a prompt about reviewing the game on giving a 5-star rating so that's that is the very reason why there's so many five-star ratings as well but it's really hard to say because there's so many things that actually can have been the cause so yeah that's one one big challenge and and for that reason what I was saying here was that actually this is not to be used only on the review data but on any kind of a text data so we actually do it's a tool for taking away the human bias that we naturally am psychologically have instead use computers algorithms to our benefit to do things that humans would spend hours and hours going through days months we can get in couple of seconds millions of reviews analyzed and

basically if we as a joint effort managed to get it so that we can even introduce context to that we'll be interesting to see if we are able to analyze and do clustering of teams for example and there are many many commercial solutions like I said but so far they've been too strict on yet they are kind of like forgetting few important issues and that's why I see the only possibility being an open source solution that will be as a joint effort done to serve every single company together but thank you for the question thank you I could good challenge [Music]

[Music]