How to Use Machine Learning, Live Telemetry Analysis, and Computer Vision to Manage Communities

[Music] thanks everyone for showing up to the session with the longest title it says I had to take the ax found sighs down a tad to make it fit and extended anyway we didn't did not think through when we came up with that title in the first place so my name is Alex I am the co-founder and former CEO of Showdown the infamous here because we got acquired last year so everything I'm going to show you today you can't buy anymore at least I won't be selling it to you but on the upside we learned a ton in the process and this talk is really about getting you guys to kinda figure out how the Rules Committee major is evolving as games are evolving and how a cup of really cool tricks can help your turn a quick question first who

here is a community manager okay who got hooked by the AI and machine learning thing for nerds yes me too okay excellent so for you guys I try to not go too deep into the tech we can have discussion afterwards because three about how we can make it simple and useful for the purpose of creating communities which is kind of why why we're here right now so this is BIA hi I'm BM the former head of marketing at showdown and yeah dive into it all right and yeah Alex will be talking about the techy stuff and I'll be talking about anything that doesn't involve code so let's get started where are your players interacting with one another today we're not here to tell you that communities are beneficial to products or games we already know that but we're here to talk about is where these communities are shifting to and where it's important to begin allocating resources to and what we're talking

about specifically is in game communities there's a clear trend between long-running successful games and games that have in-game communities so player engagement critical today it's as important as the gameplay itself and how do you improve that organically by enabling in-game interactions and game interactions up shown to keep players in the game and make them more likely to spend taking a basic in-game social features such as chats that chats have been shown to double retention this is because players are staying in the game to get their social fix and not only that but they can also request help from players in the game without ever leaving the game there's also been there was research published in 2014 that showed that players that had invested time and in online communities were more likely to stay through the concept of social capital and interdependence so people that invest more heavily in your communities are going to be more likely a stay and throughout this talk

will be using clash Royale as an example because they've just crushed it in terms of building in-game communities and they have every social feature in the book you know unfortunately for most of us most studios can't really deliver this kind of experience so what's realistic if you look at basic social interactions like creating forums creating chats those end up being quite personal so creating a connection with the player is real one-on-one relationship to chat that bi mentioned earlier basically it's about you guys community managers talking with players and getting personal with them so how many of these players can one community manager know and interact with maybe a thousand 10,000 if you're super good in social skills 300 for me so and how do you even choose whom to spend time with because if you have like a million player tenure and player even 100,000

care how do you choose which one to focus on and make sure that you keep in the game is the player that's spending the most the most important for you I think we have proof later on in this talk that this is not the case so what I'm getting at here is that scaling in game interaction is hard because interaction is inherently one-to-one and well I can I know be are we've build shapes I don't know any of you guys we may become friends or not it will take time so this thing there's a trick though because you guys through your games are sitting on a ton of data literally so this is typically simple screenshots from the Google Play game games player analytics it's one of the X analytics tool that you may or may not be using you probably have one if you have a game you are probing getting those events stored somewhere and if you look at it

that's super nice it's actually a drill-down of some of the the events in the game and well if as a human being you can actually take any conclusions from that you my hero because I can't you need technology to do that it's like turn of data very hard to find the patterns through that data likely they stack for that the way we build showdown as we did was really not to create another analytic tool unethical to this none of them we really worked hard with a couple of community manager that we hand-picked of top games trying to figure out what were the objectives that they were trying to achieve and we came up with like literally five of them that we focused on us we created the tool one was creating a community purse reaching up what how is the committee doing is it getting closer together is getting

personal or people actually falling apart they're happy they're helping the fighting what's going on in your community how is it performing preventing churn basically everything you put in place you want it to be to kind of help you even insuring you want people to stay to feel that they belong that they have a space where they have friend that they can go to heaven and be in a cool place you want to put them into paying and you don't want to put them into paying increased conversion by marketing them the new in app to that the friends have them discovered how they can beat a boss easily how they can use that item in a better way and we kind of get that knowledge from the crowd from their peers and not from the marketing then one of the thing that we realized is that you guys are spending way well should you come in to measure get spending way too much time making videos for ads we automated that because

it was boring and we wanted that sense of committee to really be a thing and to wanted to be able to measure it and make sure that this is not a crowd not a group really a community where there is kind of love being created between the players and they are feeling like they belong and that they want to be there so the way we redid approach program was through the videos text is cool but like we discussed just earlier on you have multiple languages there are lots of various attacks that text is complicated but videos are so easy it's the video game in the first place how do you describe what you're doing in the game hey isis jumped here i took this word i slash it through what did wrong what I do wrong country tell me if I don't see how you do it so video is paramount and

we recreated the whole thing about the ability and the ability to identify when something is going on the game capturing that moment creating video sequences and using that to social loops that's pretty much what Bob does here he wins he has a cool video and he get social rewards the same thing happened when he fights he can get help so social loops were really the core of what we build the technology around and and this is based on academic research there's a paper here at the bottom that i really suggest you read if you want to have a good foundation for building your own social loops but this paper basically outlines the components required to build a sense of community and they are reinforcement of needs membership influence and shared emotional connection and here's an example here on the right that i'm going to go into more on the slide so how many of you are familiar with knee rails hook model one good so for most of you you know the nineteen nineties that don't know it's a really

popular product design model especially here in Silicon Valley and what it's designed to do is you know create a habit create habit forming actions at high frequency so too quickly you know go over it a user starts with the trigger it can be either internal or external an example of an internal trigger is boredom external can be of notification an action which should be really easy to complete otherwise people won't because people are lazy then we have the variable reward which is the most important step people like being able to predict things and variability drives them crazy so anticipation is mostly a stress response to what you're about to receive so making sure that your rewards are variable is key here and then finally investment you know the action that you want the user to take that's gonna you know enable him to invest more time or effort into the game and this ideally should load the trigger by setting off a chain of events that enables the delivery of the next trigger so as you

can see social hooks are really a social adaptation of Neriah loop model and we've we have our favorite clash royale here and we have two two examples we have the game hook so you know you you get a notification that it just has been unlocked you open the chest you receive four new cards you don't know what they're going to be and then you you know play a match and then that loads the next trigger because you probably going to try to unlock a new chest so on the social side we have you know you get a notification you have a new clean message so you go and you scroll the clean chat and just quick plug about news feeds here the reason that news feeds are so ubiquitous and products is that they're kind of like a slow machine you know you you go you scroll and you don't know if you're gonna find a post or if you're gonna find something really awesome so you go to the clan shot and you know maybe you'll find an interesting new tactic and thank the person for you know sharing that tactic you might donate a card and then you go to play a match and that really just kind of sets off the loop

so the way we built the system and this is pretty much I think very representative of what you guys we need or are using probably if you were doing that kind of things you first need the telemetry you need access to the data data is paramount is what you build everything else so you need to be able to access not just the events but sometimes real-time data and that well that's what we re built we were catching the events that you already log through your game as a service platform we were also hooking on variables in the game engine and looking at them in real time so that's that's that's really like the cornerstone of everything no data can't know if the user is doing well can't understand anything else then we had the video production so that one you can get away from if you can you should get away from if you can because in our case with building a platform to integrate into games into third-party games so we

couldn't assume that Hugh guys and that game developer had cooked into his game the ability to do replays in a smart way and by smart way I mean just replaying actions class relativity does replay you when you watch a TV wild game it's not a video they are playing the game over through the events and just putting the cards on the deck at the same speed at the same pace comes with a few caveats was definitely much lighter and easier to implement so if you can avoid doing that it's cool if your game is highly physics based super complicated maybe it's not always possible then the social is cool when you have users you understand them you have content you can push and you can connect to the users we currently we need a place to do that so we built chats for groups all kind of system to allow interactions between the users and the ability to publish and promote the content to those guys and finally the one most important thing is for the community manager to actually

understand what's going on who are those guys what are the various interactions that are happening inside the game and creating hooks and creating automation so that you can see hair feel like all those guys that are doing the same action are having the same problem maybe if I put a reward circuit or a reinforcement cycle here that will help out some of the game the game design problems and we actually build that so it's a it's pretty cool because then you can use social and you can use your community to help out when maybe the game design has a few problems that would take too long to solve you can like make it a quick fix by giving the power to use it so that's pretty cool so a couple of things about telemetry so telemetry really is like in a formula one if you ever look to watch a Formula One competition you always see all those curves going up and down in every

direction it's super cool so basically it's analyzing what's going on in the system in real time and as fast as you can so it's getting a lot of data and by a lot I mean a ton you can be asked verbose as you want does it matter to get all the data well if you don't have something to analyze it does not obviously it's if you are going with if you are alone and you are going to be analyzing all the data but yourself and you don't have the right tools to do it it's pretty overwhelming and it's gonna be very hard to analyze but it's the it's like the the step one to be able to train an artificial intelligence model to learn stuff so one of the things that we found is that in order to train a net regeneration system basically what you need is data and information so that like classified information I know that the user managed to pick this word I know that the user performed well know that the user hit the master I know that

the user missed a turn or missed quest or miss something it's basically just all that data data comes from your QA team as you are building the game as there playing the game they create a ton of data so they actually creating you your your your training set for you and this is really enough just to bootstrap the whole process you need a ton of data and the system really learns along the way it needs to be bootstrap and QA does that just file and what's the estimate result of data of that much data tree about figure whether it's a hot dog or not so if your watch Tekin valley history what what it does it tells you if that thing is there or not so in second very hot dog hot dog in our case is a beat the bust not be the boss lost many lives not lasted before he got to the bastard came to the boss in a good shape he and he always reeves too much

and ends up hitting the the curve on a familiar one game it's always information that you are able to identify through the data to figure out what kind of a player you have in front of you and what was his trajectory through the journey in the game data is rebirth its basic its probabilities it's the engine looks at the data that you give him and what's really cool is that once you've had all those data pre-processed you don't need them anymore basically you you have your model and you can just really in life in life in real time analyze the data of any player and hey ok so he's taking the turn just fine he is changing gears at the right time he is not speeding too much on that line because there is a turn coming up he is breaking at the right about the right the right position and you can analyze all this information and make it very holistic and and this is how the engine actually perceives your game it's just

statistics and looking whether you hit or not - threshold so it's very raw but at the end what you know is that you are able to characterize how you play is behaving and something are really complex but whatever we want you to get away with that you can do it very easy in very simple ways because every game we looked at it in a very generic and holistic way if you start looking at your specific game how your gameplay is made and how everything has been built you can oversimplify a lot of the assumptions that we had to keep very generic and making it much easier for you so another added benefit of using this technology and having in-game communities in your game is you have the potential for enhanced metrics so in the past we've looked at things you know such as referrals growth engagement and then when you have in game communities you can start looking at things like network density and social value as Alex was saying earlier network density is basically measuring how many of your players are ornega one another or are

connected to one another and this is a good measure of a community's health another really cool concept is social value which was coined by our friends at ninja metrics check them out and what this identifies is basically the influence of a particular player across the span of the community so what they found was that your your highest spending players might not be the most important ones because there's their players called social wails that actually influenced a lot of the spend and you know through in-game communities in this application you you can really measure that so that's really cool so to show you how this you know this technology can be applied we worked with true axis last summer to prototype a showdown implementation and this video should be playing you know it's nothing to be King well they make true skate which is a really cool mobile skate simulation game it's kind of like a tech deck like you ride the board with your fingers how many of you guys have played it No yeah yeah it's really fun so no my

favorites it's a it's been in like the top ten for over a year now it's like the number two game on on the sports in the sports genre so they already had a replay system and what they really wanted was to you know oughta make it make it a little better cleaner because the runs were limitless so we they wanted to they wanted us to find the highlights and you know quickly produce highlight reels that then players could share and we also wanted to tack on that that social loop that social component to build those in-game interactions and the thing that we were focused on here was enabling player to receive help automatically from their peers so to allow a video to surface for example if a player couldn't nail a certain trick if he had failed X amount of times to be able to see a replay from a player that had successfully completed the trick and here's an example of a specific social loop again our objective was to enable peer collaboration and here we have a case where again these videos already exist because they

because true skate has a replay system so maybe you've been playing this this level and you're unable to beat the high score challenge and you you know you you quit and then a player you know happens to do a certain combo within the realm of where you've been playing and you get a notification that says a new video has been selected for you or something with better copywriting and the action is to watch the short highlight reel and then your variable Ward is maybe you discover a really unique combo that you hadn't tried before that you're able to use to beat that high-score challenge and the investment then goes to you know directly in the game you can complete the level and load the next trigger depending on what kind of if you have any social graphs or you know other social features implemented in the game you can also give that back to that creator with a good ol follow so the way we integrated that is actually pretty simple I'm going to go through the site because a few ideas that I'm really want to share so first we integrated pretty deep into the engine they had their own game engine so we we had to hook pretty deep into the the rendering the

analytics the events grab a few extra events that were missing that we needed to be able to really analyze properly what's going on and basically grab data what was really fun is when we started doing the edits because we're giving the air power for the users to let us know whether one of the what we thought was interesting was it was interesting or not so we decided hey D looks like you were doing a trick who is attempting it or sukhasana negated we think we did it you did a good job but sometimes the user was just like hey no this is this is is not interesting I don't like that let's remove those on the country hey you tree missed that trick so this is how you create a feedback loop into the model by letting the users can't let you tell you how we perform how well the model really understood what they were doing if everything if they meets missed some pieces and every a I system needs those feedback loops because it's not one thing that I think that you build ones

and hope it will live forever and for every one is different some people are also expectations so we ended up actually tuning the the engine for different people so that some would like we're more experiencing we're expecting like much really epic tricks and electrics not the average tricks the others were really like I'll give me everything I would like to share as much as I want and we're less wanted more quantity and not just the best ones and so it was very interesting to create to understand how every user actually had different expectation in the time of the content that they wanted to see created for them and wanted to share and giving them that ability to give a straight back and in creating handsome oh that really allowed us to tune it for for for some different type of us users finding the publishing is really easy one it was like one cake that goes easy on social in the in the the inside feed in the internal feed of the app and what's

really interesting here is that we realize a few sing along the way what we're doing here it's cool for social but you know QA loves it because they're able to actually understand what's going on the game when something is going wrong why are the users failing what like they can't judge when something is wrong from the telemetry they can't always reproduce but when you gave them videos that super happy it's like double thumbs up and the other guy that loved that is the ad networks because you create a ton of content that can be turned into ads if you want to so we actually have had a few interesting encounters based on that that we did not expect at all so it's really like understanding being able to put together videos and you them as basically the steam pollution loops that's that's a pretty cool idea and does not have to do what since last summer when we stopped doing the a lot of a ton of people have tried have started going into that direction whether it's creating communities who

charge whether it's analyzing better who are the key players whether it's cooking motor so do that literally randomly picked four of them to put some stuff on the slide but there's like a ton of them everybody is going to that direction and what we've seen from working with some of the top publishers and editors in the industry is not only are they on power but that deafening those things from the inception of their game now every game that we've been what we've talked with some pretty big big shots and like they were literally including all this mechanics are the start of the game and that's one of the few takeaways that I will want you guys to go home with a as Community Managers your role is changing it's not only about acquisition it's not only about social it's about building a game with social so it's just as important as the game mechanics it's

just as important as the game design and comes first in the game you players are you number one asset it's about figuring out how to keep them here and how to invest on them so that they create a community and stay together you want them in your game you don't want them on Facebook you don't want them on Twitter a lot Facebook a loft which though you want them in your game your game is your medium keep them there and you already have a ton of data you are going to have a ton of data it's about building steps step by step and building on top of the data to understand your players and be relevant and had them find their place in your game very social loops are really your responsibility as community managers I've seen too many times in games to you that divide where game design works on his side data analytics and another side and there's a

social here that tries to get a few ideas to create rules create tournaments get stuff even and this read that disconnect and what we've seen is that games that really make it are games that manage from the start of the process to put all those things in the same room at the same time and build the game from all those different perspectives and that's like paramount and it's real you roll as social as community manager to be there and to put us that idea that social loops are going to make the game just a game mechanics if you have bad game mechanics it's not going to happen but I provided that you can make proper games you need social loops to make it really happen so a little shameless plug now that we know the animal I'm doing blocks blocks and network for cross game inventory tradings we are needing committee managers please come and see me and and I'm the marketing director of black note which is a new personalized game streaming service think netflix for

games and we just launched in select countries in Europe will be in the US soon so check us out and we managed to have a few more means for questions so please feel free to come oh and before I forget you are not allowed to leave trash on the you see it and stuff and you have to keep the room and and you have to give a super cool rating because they're printing there was a trading card games with the name of the speakers from last year next year so your rating really matters please rate us thank you and questions anyone come on don't be shy I won't bite yes there is mic right here oh he does buy it no I work in analytics so I really want to know how you measured effectiveness of that social loop that you just introduced to your community and to is do you give for example AP experiments or at least different treatments to your community

members because one thing my game experiences a lot is when players experience variable experience or variable treatments they yell out be like hey why are other members receiving different things have you encountered situations like that so in our case it was a bit easier than that the so we had the base model basically we had the baseline that was analyzing what is going on in the game and that's pretty much the same for everyone that one was pretty standard we didn't find any need for variations there what we found a lot of natural variations is how we build the the how we assemble the video how we identify what's interesting what's good and what's bad for user and from one user to another some will want to fail move before they they kind of get in touch with someone some we want to fail less so that's the kind of variability and that you know it in okay is it about social interaction so people is not impacting the gameplay at all it's like you you lost and you end up in that chat

and you have that guy hey what happened why did you lose what do you not take each word before going to the bus like which would well you know the one in the closet two to two rooms back wait what it was a room and that's how you had them so DM we had the variability on the food mostly for the social sharing some people would had different needs for social share we came from reward loops and social loops inside the game not so much the only thing we try to to do is not put too many interaction we really want not to we want to not the history of the gameplay gameplay was had to stay as it was because we're coming on top of games that were already there and we didn't want them to the game designer to start blowing a fuse they were changing everything in many games never going to happen so we had to recreate and wrap the game around with the social layer so from that perspective was pretty it was easier let's say so we did not have

desire to do that many issues I start from creating the content insertion how did you measure the effectiveness of the things that introduced to your game for example that interactive video that's going who's idea basically the metrics that we introduced in them so we created a few metrics in network Nancy that we were measuring we worst emeasure in the social social value of the user we are also measuring a few other stuff that I can come here up right here we were measuring the community purse that was kind of an metric that we put together base number of different how many people are following for each other how many are they connecting the turkeying hub how many people are interacting one with another so there were a bunch of variables that we cannot put together to create a community community purse and that was kind of a cool KPI that we're trying to optimize against the other word pretty standard like the churn audio as well the basic

KPI but when we were creating content the ids was ready to use counting to a few to create to spark interactions so we wanted to up that I'm pushing that content to you guys to you because I believe that if you see that content you will react not reacting I did a bad job and when you tweet remember but it is important to create specific KPIs for each social loop and implement them into your your analytics dashboard anything guys any choice you guys got jokes we don't I think 3:30 we are we are doing thanks so much guys for coming in we appreciate don't forget to rate we want to be on the cat next year and thanks for coming