Rocket League: Language Ban System Postmortem

a lot of people here want to learn about banning players that's pretty cool I've been asked to remind you please put all your phones and tablets and anything else on silent please my name is Devin Conners I'm the community manager for psyonix tedius we make a game called rocket League and today I'm here to talk about the language band system that we put into rocket league last summer and just a few things we'll be talking about today is just a brief catch up on what rocket League is and how we had to catch up to our own growth over the last two and a half years the inspiration and the goals for this language band system that we built how we put the team together and the process that we used the execution of that the results in the reception from our community and some of the changes in future plans we have in mind for those of you who are unfamiliar

rocket League launched in summer 2015 on ps4 and steam it was a PS plus game on Steam right near launch we followed that up on Xbox one the next year and then Thanksgiving last year on Nintendo switch and coming later this year we're gonna be doing a version for Xbox one X as well so far we have over 43 million players worldwide with over two billion matches played and last but not least our LCS our chief eSports League along with all the other esports activations that we have it's one of the few eSports that's truly family-friendly and appropriate for all ages so with rocket League when the game launched we were expecting about 10,000 concurrent players in the launch window that ended up being a little more than that closer to about a hundred thousand which is great you know business is good but when you grow that quickly you have to catch up to your own growth in many

ways whether it's content features or in my case with the community team and the customer care team catching up with you know the pitfalls of player behavior and online games the more players you have the more matches are being played and just the more natural opportunity there is for the typical kinds of online verbal text harassment stuff like that that you see in online games and to help catch up with this we hired four key positions in a few different departments including the community team the customer care team is Franco here Franco may be I think he's on his way our customer care manager is popping in at some point and also our online services team which is you know half of our engineering squad that focuses on our back-end the matchmaking system getting players into servers the trade system stuff like that we're really inspired us to build this tool was kind of the community community

feelings around how toxic players are being handled in rocket League we've had a band tool for a long time since launch but it was a manual process for us so we just weren't hitting the quantity of players that we needed to and even though we weren't talking about it publicly very much the players of our game were pretty hip to that within the first few months after the game launched and you you know here's a great example on our subreddit talking about how the report system does nothing I think this is something that you'll see in a number of online games bigger small you know just seeing harassment matches immediately leads to the conclusion that there's no sort of disciplinary action being taken against these players so that really inspired us to flesh out this system and build it in a way that we were hitting as many players as we could and really driving the narrative that psionics and rocket League are very much community first and we're doing

everything we can to work with the community to make sure that their opinions are heard so some of the goals again community first that's something that we take very seriously at the studio and with that in mind we really wanted to serve the community moderate the community without completely sterilizing the online play experience rocket league is a very competitive game by nature somewhere two fighting games and other sports games that you might see and with that competitive element you're gonna see some amount of appropriate competitive banter we'll leave it at that for now so really you want to draw a line in the sand of what's appropriate versus what's inappropriate and that's where this goal came in we also wanted to reduce the manual workload on the community and customer care teams and we were really focused on the worst offenders when it was a manual process and we wanted to take that same philosophy and apply it to all of our players using this automation it was important to us but we

wanted to make sure that it fit in with our current Terms of Use in our code of conduct and we also wanted to deploy this without dramatically affecting the production schedule we try to push out six to eight major updates a year and being able to build and execute this system without affecting any of those updates was really important to us so assembling the team it was definitely a multi-discipline effort at the studio the community team handles player behavior in the code of conduct that's myself and Kevin Sweeney our customer care team led by Franco with Derek Harvey our chief data scientist Erik carpenter who did a lot of the heavy lifting on the algorithms side Dan Thomas and met and online services for helping to build out the backend tool that we used to you know execute bans and other game altering punitive stuff and also the additional support from Cory our game director Bobby who's our

technical producer he's kind of the lead producer on rocket League prime as we call it and also additional input from Jeremy our VP of publishing and Phil our in-house counsel we call him legal eagle for some reason I'm not sure why and the process for this language band system is pretty simple once all the tools are in place during a match one player uses inappropriate or harassing language they're reported by one or several other players in the match the chat log after that report is filed the entire chat log from that match is saved in our back-end and then our algorithm scans that chat log and looks for words that are on our language band list if it's triggered we also look at the last seven days of reports against that player to see if any of the if the same word is popping up more than once or of other words are popping up as well if flagged words do you appear depending

on the threshold and I'll get into that on the next slide depending on the threshold and if they meet that threshold for these words they're then escalated into the band system starting at 24 hours 72 hours a week two weeks and so on right now we don't have permanent bands as part of the automatic system we still want a human eye to look at these reports when it comes to an eventual permit have been and we also give players the opportunity to de-escalate so after they're banned if they behave themselves for thirty days afterwards they then deescalate so they're not getting dinged as a strike to six months later if they've behaved themselves for those six months here's here's where it gets interesting we call words grams when we're talking about this system and the thresholds tied to those grams so the language band list started with about 20 words including variants for those words misspellings using leet-speak

stuff like that trying to account for any sort of sidestep the players would use to get around this language band system we also don't treat all these words equally racial slurs gender slurs homophobic slurs they all typically have a threshold of one so if you use it once you get reported you're immediately escalated into this band system or if you using the word like we give you a little more leeway because we understand that you might be playing with your friends you and your friends might talk to each other in a certain way so we want to give you a little bit of leeway that way you're not immediately getting the band hammer every time you might have a slip of the tongue this list is constantly evolving we're always looking for new variants of these words looking at misspellings and stuff like that the original list though was completely data-driven where we took a chunk of reports I think it was over six months looked at these reports pulled out all of the common words that

you would use in game play simple stuff like and if or buts or just common rocket League related phrases and words and we're basically left with the worst of the worst so for us it was having a data-driven like that means that we're not putting too much of a subjective spin on it we're looking at words that the community is actively using and policing it from there and then we take a look at this list every once in a while to make sure it's still on point the execution of this system was also a multi team effort we crowdsource the list it was community customer care and our analytics team we used existing UI to inform players for the band reason and the duration as we make changes to the system the UI team when the client teams will definitely be involved the algorithm that we use is created by our analytics team and then deployed into the game by online services on the internal side its

bigquery scanning these reports and how it hooks into the game is c-sharp just few simple lines of code the online services team they did a lot of heavy lifting here as well they completely built what we call the rocket League back-end tool which is where we do game bands coming soon' chat bands looking at trade history match history virtually anything that's tied to player data that we have need access to the customer care team is also involved at every step because they're effectively the frontline when it comes to talking to players who have been banned no matter how hostile or you know mild-tempered they might be after the fact so customer care is involved to help inform the decisions we were making on the list and also to help anticipate player reaction contact volume and then last but not least the community team did most of the messaging for this so blogs social media posts being active on the subreddit on our discord and making sure that players

knew that this new system was coming how it works and how it's going to be beneficial for the community in the long run and again that really ties in with this community first kind of the cornerstone of our internal mission statement and fun part the results I think speak for themselves taking a look at kind of a 6-month range three months before three months after we implemented this system you see a 33 percent increase in the number of reports filed so after our messaging went out about this language ban system being added a lot of players who assume that the report system did nothing started actively using the report system for report toxic players within the first 90 days we had almost a hundred thousand bans 80% of them were 80 1.6 percent of them were 24-hour bands and then from there you see a pretty significant drop from players who were banned once versus twice versus three times and last but not least 98 percent

of players banned in this window came back to play after their band was up so we saw an incredibly low churn rate with this new system being implemented and here's the same information in a graph form everybody loves graphs I think but right in the middle is where you can see a pretty significant jump kind of the waves that you see at the end those tend to be tied to weekends when most of our players are active and the community reception as I said was strong and it was also very positive once we started banning players with this automated system we had players who were banned by the system showing up on the subreddit not posting in anger but saying I'm sorry for what I said I was having a bad night also I'm glad to see this reporting system is working so you guys can totally believe psionics when they say it's working and stuff like that really does have a ripple effect among all of your especially your hardcore players but any of your players who are

on social on the subreddit that you might control or have moderators on and whatever discord or voice service you might be using the public change in attitude is very important for us and help laid the foundation for any of the other player behavior tools and methods we use in the future this is something that we're always evolving especially with the word and the Gramm lists you know we definitely want to add languages beyond English our game is predominantly english-speaking in the US and Europe and of course our studio so now we're trying to effectively crowd-sourced lists of slurs and curse words and other languages which is always a fun conversation with people on reddit or like oh I speak Dutch let me give you the 50 worst words I could possibly think of in my native tongue fun fact Dutch people really like to tell people to get cancer in online games I don't I don't know if that's like a European thing or just Dutch or what but just Dutch well you know health

is important right so we also adjust the rest Schultz for these words when appropriate an example I like to use is the which typically stands for kill yourself it's something you never want to see in a chat log directed at another player we have it on the threshold of to not because we think that isn't serious enough to be a1 but because it's very close to the word keys and because we have a crates in key system in rocket League we give it a little bit of buffer in case someone was legitimately making a spelling mistake we don't want them to immediately be banned with the seven-day window that we use if someone's using that to really say kill yourself they're going to be seeing it more than once in a given week so by setting it to we give you one mistake and then you get escalated into the banned system as appropriate as here are some of our future plans the first one coming up

this is going to be part of rocket leagues spring update in a couple of weeks it's chat bans are not replacing game bins but they're gonna slot in in front of game bins so this language bin system that we have will be tied into champions first so if you're reported you go through the same process instead of being banned for 24 hours from from online gameplay you'll be banned from chat for 24 hours and you'll go through that same escalation process if you're banned from chat enough times you'll eventually escalate and convert into game bands to really help drive the message home that you need to course-correct how you're talking to other players trade bands should be coming later this year those aren't as chat heavy all the time but because people do try and gain the system when trading with other players there is a chat window in the trade system that we're gonna be looking at as well potentially and it's just an another player to player interaction that we want to start policing in some

way further down the road we're also doing research on third party tools that are out there to see what benefits there would be to converting from our homegrown system into a larger more robust third party system and also much further down the road how machine learning could potentially play a role in player behavior this is something that you know we're treading very lightly with because machine learning is a very powerful tool but I think you want to be careful when you're using it in conjunction with how people talk to each other and that's it for the presentation so I'm happy to take any questions hi-yah it's interesting that the bands are always triggered by a player reporting bad conduct have you contemplated triggering a ban automatically based on just like always

scanning the chat logs we definitely talked about it internally I think the reason we stick with active reports is we understand that especially with the party system in rocket League a lot of people are playing with their friends and if you have a certain way of talking with your friends and there's someone outside of your friend group who's in the server who might be cool with that then they're not going to report you if they if it's something that they take offense to they can file a report and that also that's also where thresholds come in is as I said earlier in the deck we're not trying to completely sterilize this competitive gaming experience but we do want to set up some pretty strict boundaries when it comes to certain things so with scanning all of the reports it's something we've talked about but the system as is seems to be working pretty effectively and really it really just comes down to playing with your friends and the kind of language that you're comfortable with with your friends thank you great talk thank you

okay so kind of a two-part question you use the term chat a lot are you talking about chat as and people are typing in text while they are playing the game with a controller or chatez and people are talking a text so it's just it's purely text chat is everything that you've reported there is yeah there is voice chat in rocket League but it doesn't get used very frequently okay so and chat logs are much easier to record and analyze after the fact especially on Steam you know we have a really big population on Steam and console players can type pretty quickly usually just asking to trade other players but if they're having a bad night or whatever the case may be they'll definitely get their point across for better or worse so what about post talk how do you do like you know bad actors in the community that don't actually get reported but are visible for example people who aren't eSports games and then type bad things you know while the games are going on and they have a hundred thousand people watching them what's that what's your current strategy

on that and how do you deal with this great question I can definitely answer that from a point of view of streams that we control basically any any public community public platform that we have that we have any sort of influence on that also extends to in-game behavior so with the subreddit that we have in our discord server our mods treat players the same way using the same code of conduct as we do in game and one thing that we've we've done this a little bit in the past but we're going to be ramping it up as we've made changes to our code of conduct is behavior outside of the game can affect your ability to play the game online in the future it's something that we were worried about originally with with how the code of conduct was set up but now that we're making it a little more clear in that code of conduct telling players like you're gonna have your streams you're gonna have your Twitter and Facebook accounts but if you're constantly harassing other players and you're mentioning rocket league in these tweets and these posts they're potentially going to be in-game

consequences thank you so thank you for his talk that was awesome I work on RTS games so the breath of toxic experiences that players go through beyond verbal abuse the breath just expands a lot and so I was wondering what sort of toxic behaviors your players report or would like ruggedly to be able to react to such as a fkn griefing if anything and want to get some insight into whether you see in the future any sort of automated verification for those toxic behaviors and whether they happen frequently enough in order to make it do you ever worth it sure that's a great question thank you so right now we have a few different report reasons in the game so during the middle of a match you need to report a player you hit start you go there's report slash meets so in the same window that pops up you can mute any player in the server or you can report that player once you get the

report button you have a few different categories you can choose from including verbal text harassment but you also have unsportsmanlike conduct you also you of cheating and if crate farming isn't in there we're gonna be adding it soon so that list we're working on making it static data so we can update it on the fly so as trends pop up we can add new reasons without forcing a client update unsportsmanlike conduct is probably the biggest reason outside of verbal or text harassment because that covers a number of different things including playing for the other team so if you're in a ranked match your team goes down one nothing we've seen players who just immediately take that as an opportunity like well we're gonna lose this game so I'm just going to be as toxic as I possibly can and start playing for the other team against my teammate so unsportsmanlike conduct is a frequent report those we still have a bit of a manual process for where we do analyze the chat logs to see what kind of context we can get and we also look at

frequency of reports against a player the players being reported two dozen times in the span of a couple of days for unsportsmanlike conduct and then we can get other clues from the chat logs with players saying this guy is ongoing can everybody in the server please report him based on how that conversation goes like a measure of trust a touch to that player based on the reports that are being received or the types of reports not at the moment we what we really use as a as a measuring stick for that is how many times the players been reported and also how many times they've been banned we try to be pretty lenient with permanent bins unless it's like a very severe case because we always want to give players ample opportunity that course correct and then come back to the online community and be just a happy normal member of that community as far as feedback goes that's also something we're gonna be doing in the spring update so along with chat bands you're gonna be getting two kinds of feedback

if you're banned you're gonna be getting feedback with chat lines is exactly what you said that triggered the ban so there's no ifs ands or buts about it on the other side we're also including report feedback and it'll be vague intentionally but if you report somebody who's then banned within a couple of days you'll get a pop a message saying like think thanks to your report we've taken action against another player we're not going to show you that players name or when they were banned because we don't want to game a fight too much but that kind of just simple feedback will also reinforce that the report system definitely does something and bears fruit down the road thank you so much thanks hey so when you announced on reddit the the new report system and band system were you like how explicit were you in like detailing like how it works like you or did you leave intentionally some fuzziness in there to keep people from kind of gaming like oh

well if I just don't say these whatever we kept it pretty big yeah and I know that's probably a topic that's up for debate among many in this room and the reason we kept it vague is for that exact reason of we wanted to avoid players trying to game the system we have an appeals process with an appeals email that our customer care team funnels players to so if you contact the appeals email will tell you exactly what you said to get banned and that's generally what players are looking for and that's why we're including it with the chat ban system because our goal when we really kick when we first kicked this off was to have minimal impact on the production schedule we wanted to launch a system that didn't require any major UI or client work which is another factor for that but really it was to keep players from trying to circumvent the system all the time and because we're giving them the exact reasoning now we're we're kind of balancing it out okay thank you thanks hi I really

enjoyed your talk and I found it really useful because I am like involved with a project very similar to that in my own company my question I have two questions first one is like I was wondering how you deal with the indirect insults for example I want to see your mother tonight something like that it is a clearly insult but the word system wouldn't detect such horrible attack so that's one of the reasons why we're evaluating third party tools that just have a team with more experience in player behavior also have a tool set that's already built out that can address the kind of language that we're already addressing but also take that kind of nuance into account and effectively moderated whether it leads to a ban or just muted chat temporarily or just censoring one line like one and done so that's a big reason why we're looking at third-party tools that just have an expertise in

this area so those third-party tools would be like machine learning based detection systems a machine learning base but also more dis based on linguistics and having kind of a more fleshed out word list and sentence and phrase list so I think machine learning is part of it but it's still grounded in human eyes analyzing this data and making sure everything is working effectively I see you when I when I was talking about machine learning earlier it's our goal is to never flip a switch on and then just leave it be you know even with the system that we have here it it requires a lot of human interaction on a day to day basis making sure that players are being banned or you know policed in the right way see my second question is like you'll gain Rockley is also huge in Asia like for example in South Korea we have a lot of users playing the game and I was wondering if that system also serves the Korean language - not yet but we're working on it that's one of our goals is

to get as many other languages into the system as possible it's tough when you don't have anybody at the studio who speaks that natively so we're we're doing a lot of crowdsourcing with this where we're talking with community members we're talking to members of our third-party customer care team 5ca who they have they have people all over the world who do customer care for us so they can offer insight into those languages the tough part is it's you can't you can't really open a dictionary and look for this stuff because a lot of it is very slang based so you really have to get into the weeds and talk to people who are using this kind of language online inside of a gamer or out just with their friends or in just day to day life and kind of build out a list that way thank you very much Thanks hey quick question when it comes to the your you say you're gonna or I don't know if you're doing this yet it was giving them the exact string that they said or the exact message in email are you concerned that they could reverse

engineer and go that's right I said that to so-and-so and then they're gonna go harass that person I think that's something you always have to take into account and we just need to be ready for it because if that happens and we start talking to players and are getting feedback that it's happening frequently we'll adjust the system as best we can and also just go and find the players or player or players who are you know being this level of toxic and just remove them from the game and have a zero-tolerance policy towards that sort of thing okay it's part of the reason why we don't have the reasons in there now it's something that we've had to put a lot of thought into but for us now it's it's worth it's worth the risk because we want players to know exactly what kind of language is inappropriate at some point you have to make that decision and you can't keep it vague forever yeah and then one more question and that is do you have anything in your system that considers friendships when it comes to a complaint filed against them because I'm in moderation as well

and we'll see it where this one person continually gets banned banned banned they're like man I keep getting banned and they gripe to their friend about it and that's the person who is filing the complaints and continually doing this so do right that's the question do you have any kind of like feedback about that or is there anything in your weight band system that you consider we've gotten a little bit of it and the appeal system definitely helps with that we don't have it built into the automated process I mean we could potentially down the road depending on you know if we made any relevant changes to the party system and that sort of stuff but the way the game is built right now there's no I think the only way we could do that is looking at if the person who reported the player was in a party with that player but you can't rely on that too much because they could be in a party because they were trading with each other or found each other on like rocket league garage or rocket league central like a you know a third party fans site where people try and find

teammates all the time so even if you're in a party together you might be relative strangers and after a minute or two you're like I think this guy is terrible I'm just going to start being as mean as possible to him you know whatever the case may be so but it's definitely food for thought okay thanks thank you howdy hi yeah you said on when you're looking at reports for like unsportsmanlike conduct where I'm assuming it's you know difficult to find real data of what's happened and probably also because of that difficult for them for people to appeal are you doing anything to protect like minority groups women racial minorities aisle LGBT people who are often like targeted with reports of unsportsmanlike conduct when really they just noticed a rainbow flag like on their car yeah it's a great question and I think that's why when we're looking at unsportsmanlike conduct reports we still go to chat and analyze those chat logs

because it's a pretty rare case that somebody's exhibiting like unsportsmanlike conduct behavior but they're not saying anything in chat right and with those kind of context clues that we get especially on the other side where if someone sees a rainbow flag and decides to target that player without no matter how much or how little they know about that player they're also likely to say something about it in chat so it's it's definitely not perfect but with the chat logs we draw a pretty good bead on what's leading to this behavior in these kinds of interactions but it's definitely on our priority list you know we were we're an all-inclusive game aridity for everybody we want everybody part but the real inspiration for this system was to make sure there was zero tolerance especially when it comes to gender slurs homophobic slurs racial slurs and stuff like that we don't want anybody to feel unsafe or targeted while they're playing our game okay thank you Thanks probably the last question right hi as always you mentioned before that the people who are

actually getting banned are getting a report of some kind saying why they got banned but the people who are actually doing the reporting they report someone to get banned they don't get one but they get up like a vague report that thank you for your report and everything so are you trying to encourage people to report more by doing this and if that's the if that's a yes have you considered applying gamification techniques to encourage people to report more I think we're walking the line with the spring update and the champion system that we're gonna be putting into rocket League the report feedback is gonna be part of that and we want to give players feed like feedback on when they've successfully reported somebody but by gamifying it too much our concern is it'll lead to too many reports that are irrelevant where you're just you know potentially especially younger players going into a server just immediately reporting everybody and it

just leads to higher volume which makes it harder to sift through all these reports and address new kinds of language being used that we actually have to police so with this report feedback that we're keeping relatively vague we feel like we're walking the appropriate line of giving feedback without incentivizing it to the point of manipulation nice thank you thanks and also please fill out your surveys after after leaving the room of an S to remind you about that as well and yeah thank you so much for coming everybody [Applause]