Panel: The Toxic Content Dilemma – Is In-game Game Chat a Lost Case or Can it be Saved?

[Music] okay hi everyone my name is Gabriella I'm Anna I'm the director of Helsinki games capital and today we're going to talk about a very topic that it's often overlooked but it's of highly importance so without losing any more time I would like to ask our guests our panelists to introduce themselves starting maybe from from there hello I'm Claire Ponte I'm the director of player support at Lockwood publishing who make avoc in life which is a social game and virtual world for mobile hello and I'm Michael Jensen I'm been the CEO of Mamiya which is a social media app for four kids and

now I'm independent consultant working and advising about implementing AI into businesses hello my name is Marisa Nepali I'm the CEO and co-founder of utopia analytics which is the text analytics company and thus online content moderation using AI my background is in machine learning and computational linguistics I have PhD on that hi and my name is Pascal Tobruk head of customer service at hatch entertainment been working for plenty of years in the gaming industry so for now heading the customer support department at hatch entertainment which is our cloud-based game streaming service all right now you have to all repeat that starting from Pesce ok this is a lot a lot to know but great without I would like to start actually from Pascal and ask you a nice question so me and you actually worked on a pretty successful

game me and Pascal worked on a clash of clans in a day and they both had a chat and clearly it was a it was a big part of the success of those games and I would like to ask you to make some sort of list of pros and cons and why having a chat it's important to have ena in a MMO or that type of game alright that's a very loaded question I would have to say first of all you have to understand that we as humans we are social mammals so we like to talk to each other we like to belong to a group of people that we can identify with and with that obviously communication plays a big part in that understanding that I would also say that it would depend on the type of game that you're having whether you would invest in chat or not if you're obviously you're not having a game that has social features I would first ask why but still it would be it would be

important to have it in a game that has social features not necessarily as needed in a game that does not have any social features the good and the bad the good would be well like I said people like to be part of a community they like to actually talk to each other it would increase your retention if people don't feel like they can talk within a game to each other to their friends to their clan mates or anything like that they would most likely either not play or game for a long time or they would move the conversation on a different platform other than your own and then that would mean that you would lose control on what is actually being said the other good part would actually also be that you as a developer you can tap into this almost free kind of feedback channel like if you analyze the data that comes in through that you can figure out what people are talking about about your your game or your app or anything like that for instance features that they actually

find annoying which is immediately can be labelled as a UX improvement and or you have an actionable thing that comes out of that but also things that they really like like if there was a particular kind of item that they like within your game or particular kind of feature then you can double down on that as a developer the cons this is the loaded part I would have to say well we all know that we as human beings we also need rules otherwise it would be anarchy very much the same thing actually in chats as well like without rules or anything like that people will actually show the worst of themselves at certain times not everyone I'm not judging anyone here in this room but there is a lot of people that actually do hide behind anonymity on the Internet in games and that could actually bring the worst out of people and then you're gonna deal with topics like well just plain toxicity it can be anything from vulgarity to swearing to threats self-harm or sexual predators so

those are things that you actually need to be aware of as a developer and now I hope that I didn't say that you should not implement chat because it is important like the benefits outweighed the negativity thank you very much there was a very very good answer and I think there's a lot that we can take from it but moderating is clearly like that the topic here this Pascal said yes there are lots of good things you have to be aware of what's negative so moderation comes into play into this specific topic and I would actually like to ask Claire so I would like to ask since you have games that are in many languages you have lots of players and and so on and what could you share with us a bit like challenges that you have encountered or for example how do you tackle like the possibility of having toxic behaviors in in the in-game chat and how do you make sure the players are safe yes so I'd like to just start off by reiterating

really that the the vast majority of our players are positive fun creative people and that's definitely reflected in the chat and I would encourage anyone to have have such a feature but of course whenever you've got user-generated content then you are opening yourselves up to a certain level of risk luckily for me Pascal's just actually listed quite a lot of the bad ones so I can sort of skirt that one but so we we approach it in a number of different ways so first of all we have an in-game filter and we filter all of our usernames and any of our user-generated content so that's our first challenge because players do not wish to be filtered and so that's definitely a kind of an ongoing process for us and we use community sift and we work very closely with with a great team there you help make sure that the the tool that we're using is constantly evolving as our communities involving and that it's at the right level for our community we tend to have a slightly older

audience older teens and young adults so it is definitely a balance between blocking out kind of potentially really toxic content but allowing them to have a bit of freedom of speech and have the experience that they want to have the second part for us is giving players tools as much as possible for them to be able to choose what sort of content they have access to so we allow players to mute or block other players and failing that they can report those to us and a team we have a team of human moderators that will then review those and then the final final level for us and the most important for me is that we also proactively moderate for the biggest issues that we're most concerned about so we work with a number of different different companies and use different tools to kind of detect the most problematic behavior and so we monitor monitor chat and they will get prioritized again for our moderation

team to review so for example we we search for players saying that they're underage that they're too young to use our service or we might filter for kind of traditionally just kind of vulgar or harassing comments but we actually proactively moderate for bullying behavior which looks at a lot more in depth into the in that kind of behavior over time okay thank you very much actually moving on to marathon so clearer they listed I mean she made a very long list of so many different areas or aspects where were moderator teams or the companies that are really trying to be proactive and try to tackle this this toxicity in the in chat how can how can an AI or a machine-learning can tackle this and replace if you want or help the human interaction or the human moderation would you like to share

something about yeah there are different kind of AI tools which are different features obviously we have utopia we have created an AI that learns from human decisions how to moderate and we build separate AI models for each customer company according to their own moderation policy and there can't be these all these kind of things what do you explain or something else whatever is important in your company community to be moderated automatically and we are able to do 100% of the content moderate 100% of the content automatically which is quite a lot and it's moving on to Michael actually which is the last one of the round of the first round that's gonna be many more

but you used machine learning before and I really would like to ask you like what was your experience how would you recommend it and to whom maybe mmm-hmm yeah I mean first maybe just go a bit back in time because Romeo is a social media for for kids targeted to 7 to 13 year old and it's very important that safety is is top-notch and it's in several countries different languages and in impact in 2017 we had more than a billion messages being being on the platform and it was simply not possible to moderate all that content in any way and I mean you have a challenge also at night and weekends holidays and and you really need to have 24/7 365 moderation going on and it's really really difficult if you rely on human

moderation and also you own filters and systems so so we started looking for for actually applying AI and and we insert a collaboration will you top analytics in in in 2018 and then we basically started out in the Finnish language because we knew that the Finnish language is one of the most difficult in the world and if we could make it work in in Finnish then we could probably make it work anywhere and so we impact together utopian lyrics to to basically make a model in Finnish and it worked really well and then it was just basically getting it into all the other languages thereafter so what is now happening is that every every content is promoted so if it doesn't you know perform up against the rules that we

have defined or as MoMA has defined then it will not be published and the user will also get feedback saying okay what what you are trying to publish here is it's bad content it's it's bullying or or other stuff or you're trying to to get someone else's password or whatever it is and then there's a learning mechanism going on right there and there and so so the kids or the users they basically get an instant feedback if they're not doing the right thing and then they can basically make a new post and then it go through if it's okay so this learning mechanism is also really really important so yes I can just say that applying a i-22 to services that has a lot of of user-generated content is is basically a must and I would definitely recommend it to anyone I mean I would recommend AI to any kind of

processes where you have a lot of humans doing repetitive tasks then I think they all right for my organization thank you very much and so this sounds obviously freaking and I'm really excited about this this sort of technological help and that we can get while we make our games but moving to Mary salmon actually so again this sounds great on paper make a set very nice words but in practice if I am to from a developer and I want to implement such a tool for example any software of that kind in my game what does it take you know there are changes to be made in my project management schedule the programmers to do the integration and so on Krisha what's the process behind it for integration always when we are talking about technology and is like any kind of

technology it needs the proper product which needs to work in practice and we have focused a lot in the the utopia AI how to make it work in practice and as quickly and easily as possible so we have a process that we get training data from our customers and it takes only two weeks for us to build a production quality AI model which follows that customers own moderation policy and we have also made some decisions and integration how to take the tool into use at customer systems that it is as easy as possible so it should only take from two to three days to do the integration to our API and everything is done as easy as possible because you can do it very easily

it's very difficult to maintain and we have decided to keep a high maintenance in our end if you have doctors who are able to do that and who have tens of years of experience of handling production ai models so we do that part all right thank you for the explanation and Pascal I'm gonna get back to you now so what I see now I've been in the game industry for a while and I see more more Studios building the their games and actually staying away from the chat or you know trying to either not integrate it or just you know use third-party services for it that can do the job for you but based on what you just learned and was just hear hear from from our economy Kyle and so on if you were a game developer today would you give it a

go well I would definitely give it a go I mean I I have actually already been using tools that do moderate in-game chat but as maybe a wider kind of point maybe towards the audience as well I would have to say that there are there are most likely different types of studios out there you have you have the ones that are actually thinking looking at social media right now and that are looking like oh my god did you see what happened with that game and how toxicities seeped from their platform on to social media we wouldn't want to do that so let's not build in chat and okay fair enough decision I wouldn't go that route still based on what I said earlier that I do feel that the communication channels you need to have them in a game that is supposed to be social and everything is pointing out towards more social games happening rather than just single player experiences and I'm not talking about console here obviously but

on the other end you also have have developers that would rather say especially startups I would have to say are people who are just bootstrapping their studio or anything like that usually tools or even developing chat actually it takes resources it takes money it takes additional set of hands because I mean what matters is that it's it's not that easy to just plug something in and it's done it requires well on there and it requires of course people that retrain or train the AI on the other hand you also need a human factor on your and does a developer so you're you're putting extra resources into that but yeah I would definitely go for that because if you're not doing it if you're just thinking like hey let's just add chat and not consider the consequences it's gonna be disastrous like you have companies that decide like hey let's do this very 90 style kind of blacklist whitelist kind of

implementation which most likely no one in the company has responsibility over no one knows when it has last been updated and how do you update languages that you're not even fluent in so AI is in that sense the future it allows for way more scalable solution and well just makes your game so much more social and it also it relieves the pressure from your moderation team or your CX professionals they don't need to worry about needing to read through all the chat logs in order to figure out where someone was cursing or someone actually cried out for help and no one actually managed to be there like AI can definitely help with those kind of things so yes definitely go for solutions like this think they're also interns for that let's not go there just kid but they are moving on to you so your games are in multiple languages so clearly what Pascal just said it multiplies in terms of difficulty of

maintenance you know can you say something about how do you maintain all this yeah definitely say that it's one of our biggest challenges particularly because context is so so important in in moderating online content we localized in 12 different languages on African life and we're still a relatively small player support and moderation team I'm obviously English the majority of our team are English and so it's a yes so it's a it's a it's a really big challenge broadly we take the same approach that as we do in English that we filter in our languages and give players the same support as we would in English and then we have some dedicated support in-house to support our biggest markets so for us in addition to English its Russian and Portuguese so we have a dedicated team to support those languages and outside of that at the

moment the most scalable way that we deal with it is to work with some really great partners who offer multi-language support and so we work with keywords on the player support site and Mod Squad on a moderation side and that enables us to to support lots of languages and scale up as needed Thanks oh yeah that's that's the way we also did it in the past where we worked before so I agree that I would rather have what Marisa and I just described but Michael actually we just had a very nice conversation over a cup of water a very boring cup of water actually it was about yes all these solutions are nice all this technology is nice for for the kids we are parents and do you think we should should should there be a way to educate also grown-ups about this I mean it's inevitable that our kids are gonna use this technology and are gonna be exposed to certain kind

of behavior so I think we should do something about how do you have any idea yeah I mean one thing is its mind boggling I think I think it's UNICEF saying that one third of the internet users across the world actually kids or people below 18 years of age one third and and if you look at it I mean it's not one third of all games and services that are made for kids so many kids are and youngsters are actually using adult services or services are not meant for them and they are quite let alone because maybe the parents are not really they don't really know much about what's going on at school they don't talk about it so maybe they have their friends there that they can discuss with but yeah so I think both the the service providers they need to to step up a bit and be a bit more altitudes are responsible and and acknowledge also that you have users that are quite young the way you need to help and then I

think also it is very important that parents get some education basically about how to talk to your kids and what to talk to them about actually because it is you don't I mean if you lose this traffic analogy I mean you don't just put your kids out and say okay let's go from London to Bedford and you just find out how to do it yourself maybe you start in from going in a neighborhood and then you teach them how to move around and then they get better and then you teach them how to to bike and whatnot but I mean in the digital space that is not really happening so yeah I think many many instances many both parents educators service providers need to step up a bit and I think being teachers responsible is also applying AI and actually taking a stance and in pre moderating your content so so the very small fraction of the users are not polluting you would

say the the nice experience of of everyone else yeah and I absolutely agree with you and actually I think it was a few years ago that there was some sort of there was a talk about community manager or community management being one of the most difficult job in today's tech world because of the mental health challenges you're constantly reading or having to go through or moderate this sort of content and it wears you out so I absolutely agree and actually maybe one last question to Maurice on and this is a more of an open question it's based on what we just discussed what Michael just just mentioned do you think this this type of software are used enough and if not why here there are not many companies that use AI in moderation even though there

is huge amount of online user-generated content online people feel bad governments are creating new laws how to to make the companies to take care of the users and also their employees who do to online moderation so definitely there needs to be something done obviously since we have a tool that could be used by companies on saying it words about that but also my personal opinion is that the bad behavior that happens online should be restricted and since the volumes are huge and the high peak of those comments can come any second that needs to be automated ways to take care of all of that and then we need to take I understand the fact that

there are hundreds and thousands of different languages in the world so that makes also it much more difficult to handle them than what has happened in in the real life word earlier so there is definitely a place for AI to get better and take place alright I think we're close to run out of time so if anyone of you have any question yeah do you need a microphone or some okay is this one yes how do you balance the requirements of gdpr and chat logs in ensuring that you're in compliance with both the government but at the same time ensuring that your users are safe and that you have proof in case the authorities come to find out based on

the suicide case or something like this how do you balance that G GD P R is a law and we have to follow it and everyone has to follow it actually in Finland we have had a bit more strict laws earlier so for us it hasn't been a big change the law makes you to be careful in using the users names and identifiable information but of course in the same way as any any any kind of product you need to take care of your data and handle it carefully so I think that is not a big deal in that you just do all the things what the law asks you to do so we definitely do that as a

foreigner and feeling I can say that they follow the law always any other question all right well then you guys were talking about working with different languages when you're working with AI how do you solve the problem that sometimes an expression that is highly offensive in one language probably in the language where you develop the game but then when you translate your other languages not necessarily is as offensive how the way I work with these cultural differences I can I can maybe step in here because basically building AI models you need a lot of training data so it is actually the company they're having their data that that defines you know the rules and is

making up or giving the the AI you know what is what is okay and what is not so if you do I mean if it if you have if you're making an AI model in German and and another one in in Finnish for example I mean then you can have a language specific or cultural specific decisions yeah thank you no no yes well then let's give a hand to our panelists [Applause] you [Music]