Trolls: The Cost of Doing Nothing

[Music] my name is chris preeb i'm the founder and ceo of 2hat security today we're going to talk about trolls and the cost of doing nothing and i want to just preface this kind of conversation before we get too far because oftentimes when i talk about that we think what we're talking about is profanity and silly little funny quirks that people say and we've seen a lot of that because we process about four billion messages a day for the games industry and we've seen a tremendous amount of hilarious stuff but what we are is we are an ai company that finds the most toxic disgusting things on the internet like rape and genocide and hate speech and sexual abuse of children and child pornography and the worst things that there is and we also work with the games industry in order to deal with the trolling problem uh to build reputations and find our own users and what we'd like to share today is a bunch of research we've been doing out of the research arm of our company where we've dived deep into the problem

of trolling and tried to understand how can we make it better and how can we be involved in it so it's a bit of a serious conversation today but hopefully we'll have a little bit of fun in light of that so this is me long time ago how many people had a computer like that long time ago yes i figured it we've been in this industry for a long time you guys 20 years for me going at this um having like a great time when interested in someone else this guy here is my brother uh my older brother we used to make games together back when we were quite a bit younger and my brother ended up making a really cool game so he started off in 1999 and he's like okay i'm going to build a game it's going to be for kids now nobody builds games for kids not not in 1999. so in 1999 that was the chat rooms were very very dark and not appropriate for children and so there's two things about building a game for kids one you're going to get sued number two no the kids don't have any money so at that that's that was the mentality at the time so he went out and created his game

and he built it and he was able to achieve 300 million users and if we look back on this game it ran from so the early experiments were in 99 and this month they're finally shutting the game down and they're moving it to a mobile experience which is really cool but how did he go about and do that when some of the lessons learned well one of them was that community was incredibly important because he found out that if you put energy into making it incredibly safe so if we're going to make this for kids it has to be exceptionally safe and when we did that it became a place where kids hanged out built friends built community and their parents were like hey my kids are happy they're they're having fun i'll pay money and turns out the parents have credit cards and if it's safe the lawyers don't come after you as much so that was really really cool and what i found out is that when later on disney came by and they offered my brother 350 million dollars for his his game um i was in charge of doing a bunch of the due diligence and they asked me this question and said what is your churn rate i said what the heck is churn so i i wasn't measuring churn

because we didn't really have a concept of term because people loved the game and they would just stay and stay and stay and stay and i had to learn what it means is how fast are you users quitting because when you have a strongly engaged community you can end up with a product that lasts for well over a decade and you can still have people very sad after they've been playing for 13 years and more but it's not just kids based products that we're talking about it's becoming very very popular today that this issue needs to be dealt with of trolls toxicity and all that kind of garbage we can see and watch what's happening with twitter in the media and they've been slaughtered and they're trying really hard they're doing a bunch of things but they've just been slaughtered in the media about how much of a problem it is and so you have famous people that are being onslaught with constant bombardment of toxic and trolling based comments and that's highly problematic because if you want to create a platform where there's free speech if so much hate speech is allowed then

what ends up happening is nobody wants to share you end up with this situation where everyone just shares pictures of puppy dogs and silly little emojis because if anyone engaged in a real conversation if anyone engaged in their actual liberty of being able to share their true mind and their opinion they'll be destroyed because 95 or more maybe even 99 of users are good but the one percent and the five percent are so toxic that they're making these platforms unaccessible to have true expression and truly reach their potential what they could be and so you have kid situations that's that gets even worse where you take your best users your whales the ones that believe in your platform the one they're building up the ones they're inviting other users and they are the ones that are leaving in mass and so you can see on places like twitter where the most famous uh people are abandoning twitter and they're doing this campaign of like i'm quitting twitter and it's happening so frequent that it's becoming common in this case here it's lily allen was being attacked on twitter because she lost her baby uh

many years ago and she's developed mental illness because of it and because of that people are telling her what's your fault you're the one who killed your baby and they'd make little memes about a frog being stillborn and being it was it was like so devastating to her that she's like i've had enough and so the cost of doing nothing the cost of allowing such toxicity to be pervasive has a direct impact on the financial bottom line of a company and so this is uh twitter's stocks and you can watch kind of how they go down now there's many factors involved in that but one of them that's being drawn out in the news in the media is that they have not been able to take this matter um not actually been able to deal with it because they've had a platform where this free speech is everything and this is your own private room and do whatever you want except for we have to understand that the private room has a glass wall standing on main street because we gave everyone a microphone because they're supposed to tweet to the world and there's a disconnect between those two concepts and now they're actively working and trying to do some amazing steps to transform the situation

and launch some new technology but we hope it's not too little too late in comparison you can compare them up to facebook which is in green many different factors involved in there there isn't just one but one of the factors that's different about them is how they've gone about taking care of moderation and facebook hires thousands of people to go and review content and user reports and take action against it and try to remove hate speech within 48 hours they've been actively engaged in it and it makes a financial difference to the whole thing and today we want to measure we want to take a look at that cost of doing nothing i want to follow up from a talk there's a a great company a great presentation a few years ago riot games to look at what they were doing and they said oh my goodness if we analyze our chat logs honestly we know that users are about 21 of of the conversation is toxic and they said they had to do something because what we've observed is that when conversations hit around the 20 mark it's close to death and so riot

games got really smart and they started thinking about it and saying well what can we do about it and jeffrey lin came on the stage of gdc and gave an amazing breakthrough research of what they've been able to uncover and discover and one of the things that they did in their studies they said a user who experiences toxicity is 320 more likely to quit that motivated them to make a change and they did things like they set up a tribunal system so that the the community could take care of some of those issues they started telling people this is the reason why you were banned or why you had action taken against you and they noticed that there was a higher reform rate at one point they brought the toxicity all the way down to two percent if anyone knows riot games i'd love to connect and figure out some of the techniques and stuff and how it's going now for that and then be able to present that more as we go on but sitting in the audience at the time was my buddy um alex and he was from world series poker and he was listening to it and he was thinking back about the different kind of feedback he's had from his users the comments he's had and all since he started thinking okay when you're

playing poker and you put your money in the kitty if another user can be toxic to you and troll you and make your experience terrible and you're like i've had enough and you just throw the cards down and leave and fold there's a disadvantage and that person by trolling you can make a financial benefit and that becomes problematic because in all of our games we work on the game mechanic we spend very careful time meticulously thinking about every single detail of what items are going to add to the games because that balance is critically important for fun and also for retention and growth and monetization and if we allow one item to be there like a giant sword that you can slash everyone and every time you use the sword you always win that game sucks well we need to put the same effort into community mechanics as we do into game mechanics because community is absolutely essential to the growth of our game and we need to grow up as this whole thing and have more and more of this research which we're seeing coming out and saying how can we design our communities just as effectively as we design our games and so we go into the question about what is toxicity toxicity is a poison so

if you imagine that you can put a poison into an organism like ourselves or human body and if it begins to spread the organism will begin to die and we see that when we launch new games they start off with about one percent toxicity roughly speaking and then they go up and they start getting around five percent and then it starts tipping the scale and then they'll quickly grow up to around 20 and it's a toxin it's a poison that's spreading through the organism and an organism that's toxic begins to die which in the game industry means loss death of users users leave it's no more fun the game eventually shuts down now the troll we have to define troll there's a great discussion on youtube on the gdc channel about this talk and a lot of people were challenging and says we need to make sure you define troll because a lot of people are just throwing this term out there everywhere it's basically a lot of uh the accusation was that we use troll to say people that don't agree with me this person is trolling me um on there and there's a very engaged lively discussion

where one guy was you know tell me what you think you know you worthless piece of whatever you need to you need to explain your position and he went on and on and on about this whole thing and the conclusion was is that a troll it has this intent of evil they have an intent to destroy and it's an ongoing repetitive thing and in the case of this one user people accused him of being a troll and by his own confession he just said no i'm just an so we have to be able to tell the difference between some of the two because that is a slight difference but a troll is the goal is just to be destroy the community where other people are just downright obnoxious no problem we need to get rid of both but we should make a little bit of a distinction so that we treat them slightly different so as we move through we wanted to understand the cost of doing nothing and trolling toxicity those kinds of things so we said okay trolling toxicity where can we start oh let's go start with reddit so we went and looked at the first year

of reddit before they developed a reputation and we did a big study of how does each individual subreddit how does health how is it affected by toxicity so we measured health as things like growth you know organism that's growing so more users more conversation more whatever just more of stuff and toxicity we defined as basically let's take anything bad and throw it in there so people raping each other racism vulgarity just whatever it's bad stuff and we measured the two and tried to find a correlation well the expected result of course is that you'd see that as uh the the bottom line of toxicity goes up the red line which is health goes down well that kind of makes sense that's what we're anticipating what we like to do in a study is find stuff that we didn't anticipate and what we didn't anticipate was the second column which is the game industry um that the toxicity was starting around four percent and it grew all the way up to around six or seven percent during the course of the year and guess what happened to the health the health went up so when toxicity went up in the game sector on on the subreddit

then so did the health and so there's not an absolute direct correlation between toxicity and health you can have a toxic community and you can still grow which is a drastically surprising finding however there's some caveats to that now our understanding for that is notice that it's on the lower level of toxicity so a community has enough resilience to put up with a little bit a community can handle a little bit of toxicity in addition to that a community can um begin to have an expectation that surrounds and it begins to draw people of like-mindedness towards the same thing and other people self-elect to just to go away so the health could have potentially been faster we're not sure but one thing we did observe is that when there's a drastic change in toxicity there's a drastic change in health and it works on the positive as well because world of warcraft on their um individual reddit you could see that when they made a difference to the toxicity level so when that toxicity level starts dropping on the bottom you

can see that eventually the health started going up so when the community began to trust the forum again they began to come back and they began to engage again and began to be a place where they could share and on a completely different nature of topics of amateur it's don't look it up it goes up it goes down and goes up so there's a direct correlation between the two and when whenever there's a drastic change there's a drastic change to health and toxicity is common other people are doing research and finding that things like half of people um playing playing these kinds of games discover that they at one point are being um attacked by this toxicity in addition to that a third of us basically at some point in our life get pissed off and and become that toxic trollish type person under certain conditions so that becomes quite interesting and now i wanted to really dive into it and say okay what does this actually look like in the game industry we love the idea of an a b test so let's separate an a b test i'm going to create two different kinds of games

and illustrate for you their journey of how they're going to go ahead and reach 21 million dollars and how what difference it makes based upon the decisions they make using the data that we have now so these are fake games they're based on real stories and real data um behind it but they are fake for the purpose of this demonstration i want you to consider them basically the same the first one is a futuristic space game called ai war zone the second one is a medieval fantasy game called trials of serathian with an emphasis on the word wrath in sarathian now they're both mlrpgs they're both over 13 and they're predominantly male with a good mix of female players for the sake of the study i want you to kind of think about them to be essentially the same so the beginning of our study they both have a million users it costs them 2.78 cents to spend advertising money to buy a user to come and play the game and that user when they join the game spans about 13 and 51 cents and

miscellaneous little tiny things in order to give them power-ups and whatever else inside the game so in-app purchases which is great i mean we'd love to have that kind of uh markup on our things a lot often oftentimes we're fighting for pennies in here they started off with a 20 churn rate month over month and i'm going to show you a moment how they would drop that to 5 churn rate um and then there's executives walk into the room as they always do like me and they say okay this year you guys are going to reach this objective no matter what you're going to get 21 million dollars of revenue and the team goes and looking at it's okay to hit that number we have to add to our base user base 10 every single month so they attended gdc they went to this talk and they revealed another study that we did this year so we did a study of a first person shooter i can't share the name at this time but it's a first person shooter it's been around for a long time mostly out of turkey and this game here when we did the analysis of it like most people do that to understand churn we looked at how many users were on day one that

continue to day two day two to three and five and blah blah blah blah blah and you measure that and that's really important to know if your game's being successful or not of course this is pretty basic but then we segmented the users in a two different groups and we tried to find different behavior models that indicated that makes a drastic difference in behavior and the one that we identified was that users who engaged in chat are three times more likely to come back on day two so any user that types in one line of chat or more is more likely to come back on the second date we didn't figure out the exact why it might be because they brought friends with them they may have already known people they may have found a place of belonging whatever the reason was more study to be to come they're three times more likely to come back in day two in this story and then it extends that after two weeks they're also three times more likely to come back and so what both of our games decided okay let's make chat a primary focus let's make sure that people somehow engage in chat and they find a place of belonging and they're able to extend um their their thing so here's a a

mathematical form basically means the user's next month so the total number users you can have next month is the total number of users you have this month minus all the users who quit plus all the users you added now that you can imagine it like you're climbing a mountain but the mountain's made of sand and every time you take a step some of the sand falls down and so it's really hard to reach the top of the mountain because the mountains keeps eroding so if you're going to reach your 21 million dollar goal you got to go grab a bag of sand and you have to pay money to dump to buy those users and put them back and replace all the ones that lost so you have to constantly be replacing all your users so they each took a different approach to how they're going to climb this mountain and get to the top of 21 million dollars of revenue the ai war zone said we're going to have a moderation strategy we're going to take care of our users and we're going to spend money on that and trials of serathian just simply said you know what uh whatever they i've heard so many wonderful things i've heard those are our best users but it's fun that's what we want the game i've heard so many different kind of stories but that's the path that they

took this they'll take care of itself so when you run down the numbers and you watch and so the key numbers that we can look at is they began to diverge and so uh trial ai warzone had to spend more money to pay for their moderation and they outsourced like those people that they were using to do all that however they were able to achieve more money in the end so the end of first quarter serathian had to pay more money because they had to go and buy that extra users the sand to keep dumping on so they can replace all the users that they were losing so for those numbers then you can roll it out and you can go down to the end of the year and the ceos are sitting down with their media and they're saying okay so how did we do how did the game perform and at the end of the year because they had to spend so much money replacing their users they ended up spending seven million dollars extra and going through one part i missed on the the math there's some little numbers at the top

so the math that i was using there is srathian using that reddit study said that look our users can tolerate some toxicity so they had about a six percent toxicity level my eyes can focus close enough 6.6 yeah and that they said okay our users can put up with that and anything above that we're going to take care of and then serathian said i don't care and so what they ended up but they had the riot game study that a user experiences toxicity is three times more likely to quit so you can see ten percent versus three thirty percent roughly three hundred twenty percent or three three times anyhow so going forward so what did we learn so uh sarathan ended up doing a study and saying okay we got to change this this is we're just losing money like left right and center we got to make a change so the first thing they did i said okay riot games did something cool let's rip off that idea and we're going to do it but not as well and so they decided that every single toxic chat message needs to be or every human report needs to be reviewed by their best users so imagine the scenario you got your best users who love your

game who tell everyone that they should play this game who are deeply committed playing a long time and giving you all your money and you send every single message that is toxic and the worst of the worst to those users now if 300 if user experiences toxicity is 320 more likely to quit guess what's going to happen if you take your best users and give them 100 of the toxic crap right you're going to wear out and burn out and discourage the best users that are paying you all the money and bringing in all the next user base that was like a really bad idea that was just not cool and the second thing they did is they said okay these users they keep being offended they keep annoying me with all these complaints i'm just going to add a button that you can mute another user so whenever that user says anything it's still said but you just don't hear it so let's let's look through the logic of that so if imagine that 95 of users are good because only you know some of us sometimes make talks of things but not all the time if most of the users are good and a few of the users are causing the problem why

is it that the majority of our users have to go and push a button every single time that they're offended it's not like it's their fault it's not like these toxic users are like making us tons of money they're actually destroying our revenue so why did we it's kind of backwards we should have put the algorithm on the other side in addition to that they just keep creating more accounts and they have to keep muting them and they're still talking about them they just can't hear it so everyone in the room's laughing and they don't know why so how do you know if you're a seration so go into your forums go into reddit go into your app store if you're on the if you're on there and look at the comments and see if people are saying i love the game but i hate the community you know you're pushing towards that level so what did what did ai warzone do so ai warzone they decided they realized that not every community is the same communities are created differently so you have to

figure out what kind of community you're creating so in real life we go and create communities like a playground for kids if we're making a playground for kids it has certain cultural expectations so if i'm hanging out there with my four kids and some 13 year old punk comes along and starts swearing to their friend i'm gonna get i it's perfectly seven return to that 13 year old and say hey this is a mixed audience do you mind and most logical people oh sorry i forgot and then they move on now we can do that with kids based products we can say look you can't swear here that's not appropriate now if we're going to go and create a different kind of community let's say we're going to create a nightclub it's for 18 plus there's certain kind of social expectations that can happen in nightclub they're different from the neighborhood park so each location each community each party that we create has a different goal and we create a community for a reason and we have to figure out what that reason is and design that community around that and it's perfectly acceptable to stick a bouncer at a nightclub because we want most the people to be having fun and a good time and if one drunk idiot comes in and starts disturbing everything bounce them out that's perfectly acceptable and if we're going to build burning man

and we're going to take everyone out into the desert and we're going to have a really good time even there there are certain things that are culturally unacceptable like you can't go and murder people and people and do those other kinds of things like that's just not that's just not cool so there's different cultural expectations that we can be enforced for each area so they realize that each zone has a different kind of threshold so if you can think of a vertical axis you can draw a line somewhere in the sand and say look we will up we'll put up with this and no more and that needs to be age appropriate and game appropriate so if you're targeting middle-aged women and house codes there's certain different expectations than if you're targeting hardcore gamers so that but that's only one vertical axis there's also a horizontal axis because we all know that even if you allow swearing there's a certain point where if someone just swears and swears and swears and swears it's like get like what you're so annoying like contribute to the conversation or like at least contribute to the game for crying out loud so there's a point where we have enough is enough and so they're actually vertical lines

of two dimensions one is the severity of the comment the other one is how frequently it's said and for each age group or each demographic you can draw a line and say look if it continues i'm going to draw a line in the sand and say enough is enough and by doing that you can break your users down into different kinds of groups and so you know if you launch a game the first thing and you put chat there the first thing a bunch of users are going to do is they're going to type in because they want to know uh does this system have a filter what's the what's this game for what what is the nature of this community and most people do that because they're going to test the boundary that's perfectly like that's perfectly normal and if they find out that they can't do that they go oh this game is expected behavior of blah and then most of them back off and some of them are like oh this is a mini game can i get around it and so they they have they keep pressing on and they persevere so you have different kinds of users in different kind of ways but we can then take those users and we can deal with them differently thank you so the way we deal them differently is you take that line in the sand

and anything that's above that line is blatantly obvious those are the things that you just don't say in that kind of context and automation can do that so there's no need to crowdsource that comment and send you send your best users to look at the complete filth that you already know the answer for so anything above your line you just filter block whatever you want to do hell bands shadow bound whatever the next thing that you want to do is the stuff that's just a little bit below the line the stuff that's difficult for computers to find out you want to have your humans help you with that so you add a report mechanism people report it well if a human if a human hates it and computer's not sure about it chances are it's not good so you can auto action that one as well now the only thing you're left with is the stuff that is subjective that stuff requires context the stuff that's innuendo the stuff that's difficult and that's where you crowdsource your humans and that's where a tribunal-like system becomes powerful because your humans feel valuable because they're dealing with a real problem and it's not the obvious and they're making a real contribution to the community and then your trolls the ones in the upper right hand corner are the ones that are defined as they keep

ongoing ongoing ongoing ongoing won't stop desire to destroy your community get rid of them just just expense it away because they're just going to cost you money so i want to talk about how this is put out in real world situations so recently the great folks at twitch they implemented a system like this and they allowed for each channel to set their own resilience level and say look for this the purpose of this community is going to draw the line here and it's been incredibly well received it's amazing to watch the reception of people like yes finally this is so wonderful i wish everyone had this functionality so kudos to twitch an amazing job on that product so the three things that we can learn is that if we take care of toxicity we can make money was a seven million dollar difference in that one scenario that i was outlining in addition it will help your

brand so if you want to go and raise more money if you want to go and get more publicity if you want to have people share about your site and invite their friends into it so you don't have to buy those users really really good we talked about it began with the story of club penguin and how you can keep your users for a long long time if they feel that place of belonging but what's also really cool is when you launch your next game you've got a massive user base that loves your brand and they're going to come with you to the next one so thank you for your time i really really appreciate it so from here we're going to be able to take questions we're going to go out the hall down the side and there's a little like place with a bunch of things we're going to stand with take a whole bunch of questions there we got some free shirts we can hand out i got my team that can also help answer questions if you can't get a hold of me in addition to that um please fill out the survey and they told me a bunch of other stuff i'm supposed to do yeah blah blah blah upload it to the internet watch this share it with your friends on the gdc vault okay thank you for everyone for

your help