Robocalypse Now: Using Deep Learning to Combat Cheating in Counter-Strike: Global Offensive

Uh hi. So, we have a little bit of business before we get started. I'm supposed to ask you to pretty please mute your cell phones so they don't go off. And also to make sure to fill out your surveys when you're done or when we're done. They're super helpful for GDC to figure out who they should invite back and they're also super useful for me to figure out if you like the way I presented and you like what I presented. So, please take a moment when we're done to do that. All right. That's That's the stuff out of the way. Let's get started. So, my name is John McDonald. I'm here from Valve and today I wanted to come and talk a little bit about how we've been using deep learning to effectively combat cheating in CS:GO. Spoiler alert, it works pretty well. We're pretty happy with it. So, that's a bunch of stuff. We have a whole hour, so we're going to We're going to cover a lot of ground. I'm going to spend a little bit of time, probably fair warning, probably about 15

minutes kind of talking about the state of CS:GO with respect to cheating, other things that we do in this space. And then we'll dive into sort of the pitch that I was making to folks around Valve in 2016 for why we should maybe give this try. We'll dive into results and then we'll talk a little bit about how you train and utilize deep networks. We'll talk about some ongoing and near term work and then we'll finish up. One thing that's probably worth mentioning is that notably absent from this list is deep learning. I'm not actually going to train teach you deep learning in an hour because I don't believe that I can do it, but I will point you at the end towards resources so that if when we're done you feel like, "Oh, that's something that maybe I should try." You know where to get started. And good news, it's actually pretty straightforward to to learn. All right. So, let's talk a little bit about CS:GO itself first. CS:GO, if you're unfamiliar, I think most of you probably are, but it is a succession of Counter-Strike.

It is a competitive multiplayer game. It's fast-paced action. We have sort of bullets that are extremely extremely lethal and when you're dead for a round you're out. The combat engagements in CS:GO are very very short. The average engagement time can be like 1 to 2 seconds. And all of this sort of matters because it impacts our cheating problem. CS:GO has 12.4 million monthly uniques and 3 million uniques check back in every day. So, we're we're pretty happy that our player base is so rapid. Finally, it's the 10th title that shipped on Source engine and we'll see how this plays into our cheating problem a little bit later as well. One of our most popular modes for CS:GO is the competitive mode. It's a 5v5 game mode. First to 16 wins and each round is about 2 minutes and 57 seconds. When you sign in for matchmaking, we we ask you to agree that you could be, you know, signing up for a 90-minute commitment. The average game is actually

more like 40 minutes. 65% of our players check back in with this mode at least once a month and there are 600,000 of these matches every day. It varies depending on time of day and sorry, day of week and you know, summer there's a lot more matches, school's out, etc. but It turns out turns out college kids love to play video games. I don't know if any of y'all knew that. Uh But so let's let's dive in and talk about cheating cuz that's that's kind of why we're here. So, back in 2016 I was sort of figuring out what I was going to work on next. I had finished up some other CS:GO work and and I was trying to figure out what I wanted to do and usually when I'm in that space of Valve I just kind of try and sort of figure out what the community wants cuz I figure anytime you sort of pay down some debt in the community that's a pretty good investment while you're trying to figure out sort of larger problem to work on. And it turns out I couldn't do that because the only thing the community was talking about was cheating.

It was this just deafening conversation and we saw this. And of course like this information is all anecdotal, but we saw it on on Reddit and we saw it on our own CS:GO team feedback alias where players mail us directly. And you know, the feedback alias in particular I had some data on it. It was like 90% of the incoming mail was people saying your game is filled with cheaters and that's, you know, that's really frustrating. But we weren't sure whether or not the problem was uh sort of a perception problem or whether there was actually a cheating problem. Except it turned out that VAC bans, which we'll talk about in a little bit, sort of supported this conversation. Like we had a lot of VAC bans that we were handing out. And we we think that CS:GO has sort of two main components that lead to cheating being such a significant problem. There's a third aspect we'll talk about in a little bit, but one is that the design sort of the design is this very satisfying combat experience

where almost any weapon in the game can be a one-shot kill, but that does also sort of fail to disincentivize cheating. On balance, we think that's a good trade because players really enjoy this, but but it does sort of make us a juicy target for cheating. Similarly, since the engine is so old it's and about 75 to 80% of the code in CS:GO is actually unchanged from Half-Life 2, that means that if you have a cheat that somebody wrote back in say 2005 for Half-Life 2 deathmatch and then they posted it on GitHub a couple of years ago on a lark that I think probably actually works on CS:GO with small amount of of changes, like maybe 5 to 10 minutes of tinkering. Now, the good news is we do actually catch pretty much all of those cheats. Like if it's on GitHub you're going to get banned for using it, but but that just means that there is just this availability of of cheats that we don't see in a lot of other places. There are two main sort of cheats that

are prevalent in CS:GO. There's a third one that we're not really going to talk about today, but it also is just this very minor sort of issue. And the primary two are probably the two that you might be used to if you're familiar with cheating in your games. And that is basically something that helps you aim better and something that gives you information that you're not supposed to have. And you can sort of So, the aim the aim hacks or aimbots tend to make it sort of much much easier for you to communicate headshot that guy because because in CS:GO if I see a target and I want to headshot him, I need to identify him. I need to decide that he is an enemy or she could be. Decide that they're an enemy. I need to move my mouse over. I need to place it ideally on their head, push the mouse and then start dragging it down to sort of deal with recoil. Or I could have a cheat that does all of those things if I just press the X button including not shooting if that person is actually a friend of mine. And so that that communication of intent there is a big

delta between sort of the intent of the action and the actual action. And information assistance hacks are exactly what you would expect. Wall hacks, you know, substituting materials or what have you, but it's just giving you information that basically you shouldn't have. And then there's sort of this scale that we sort of think about hacks on. There's the subtle cheats where somebody is trying to basically pretend that they're a better player than they are and those are actually pretty hard to detect and not what we went after first. And then sort of the scale kind of ramps up in obviousness until you get to what we call rage cheats. And the purpose of a rage cheat is not not subtlety. The purpose of a rage cheat is to make you pissed off that I am cheating and there is nothing you can do about it. And all I want to do for the next 40 minutes is ruin your day. And so we decided that seemed like a pretty good target to go after first. Now, cheating because of the 5v5 nature in

CS:GO, you can sort of think like about the cheating problem as it relates to an individual user. So, if I'm going to play a competitive match, given that I am clean, even a a low base rate of cheating like if 2% of players are sort of cheaters and they're randomly distributed, then two out of 10 matches that I play are going to be dirty. And if the rate is of if the base rate of cheating were 7% like half of the matches that I get into will have at least one cheater. And that is again, assuming that I'm a clean player because there are nine other opportunities for a cheater to land. The other thing that is maybe mildly that's slightly even more irritating about that is that given that you are a clean player, you are more likely to play against a cheater than you are to have a cheater land on your team because there are only four slots for them to land on your team and five slots for them to land on the other team. So, that that that makes it an even more irritating problem for our sort of good actor players.

Now, the good news is we're going to I'm going to spend a minute here sort of in a digression. The good news is that we don't actually just randomly distribute our players. We released the system in 2016 called trust score and this sort of mitigates the damage of cheating a fair amount. And the idea is that we sort of pay attention. We train the model based on how you interact with CS:GO and how you interact with Steam and we use that to try and predict how likely it is that you're going to go on to receive a ban of some sort. And then um we basically use that trust score to match as an additional factor when we're doing matchmaking like in addition to your skill and what map you want to play and you know, sort of other things. And that lets us sort of put known good actors in with other known good actors more often. And so as a result, our known good actors run into cheaters way less often. And this can sort of be more easily visualized. Like imagine this is just a population of players. Don't count them. There are 50 dots here. And so we need to set them into five 5 V 5

matches. And because it is way easier to do with PowerPoint, maybe we set our matches up like this, right? And so now we've created five matches. And let's find out how we did. So you have this sort of concept that we're going to introduce called the rainbow of trust. And we think that blue players are very unlikely to go on to receive bans, but red players are very likely to go on to receive bans. And you might be asking if the red players are likely to go on to receive bans, why don't you just ban them? The answer is because they haven't cheated yet. Like we don't know for sure that they're cheaters. They are just more likely to go on to be cheating. But let's assume in this case that five of the red players are cheating. You can see that in the worst case here, like all five of these matches now have cheaters. And what trust lets us do is basically say, "Hey, when we're trying to build matches, if we assume all of these players are sort of that any subgroup of 10 players here would form a reasonable match, we could

draw our matches like this instead." And now we've created four good matches and the players who were most certain are not going to go on to cheat are very unlikely to run into cheaters. So we have a bunch of data that shows that trust works really well, but I kind of prefer the anecdotal version instead. And that's that we actually launched trust about six months before um before we announced it. And during that time we were gathering feedback and looking for people that sort of like had bugs. Like just general feedback, right? They didn't know that we were running the experiment, but and so what would happen is you'd see a thread pop up on like the Steam community forums where somebody would say, "CSGO is filled with cheaters." And then you'd see somebody else reply and usually quite a few someone else's who would reply and say, "I don't really ever run into cheaters anymore. I don't know what you're talking about." And at this point we'd sort of been public about some of the other anti-cheat work that we were doing. And so the player base assumed that that was why they weren't seeing cheaters as often. But we would go and basically say like, "Hey,

let's go figure out, you know, what what this person's trust score is." And we'd go and we'd look at the OP. And you'd be like, "This dude is tied to 50 accounts and 49 of them have bans for cheating. I don't care that that guy runs into cheaters. He should run into cheaters cuz he's definitely going to cheat again, but maybe he hasn't done it yet." Um So that's pretty much all I kind of wanted to say on trust. I did want to mention though like a thing that's cool about trust is that it doesn't it doesn't really reduce the rate of cheating, but it reduces the blast radius when people decide to cheat. So let's dive in and talk about a couple of the other anti-cheat things that that we do. So the main probably the main thrust of work that we do is just mitigating exploits. We just fix you know, fix bugs when they come in and and that that goes away. Somebody reports, "Hey, here's this problem." We investigate. We figure out what the problem is and we fix it. And so one of the canonical examples for this was this speed hack.

It turned out that players realized some malicious players realized that if they this part's a little bit technical, I'm sorry. But they basically realized that if they lied to the process, they lied to CSGO about how fast their clock was running, they could convince the game client that basically it was running 10 ticks at a time. And so then the game client would faithfully be like, "Hey server, here's 10 movement packets." And that would effectively and then the game server was like you know, for reasons that client authoritative feels awesome, we would we would be the game server would just go, "Yeah, okay, looks good to me." And it would just process all 10 movement packets in sort of a single server tick. And then the result of that is that that player would just sort of appear to just teleport around the map effectively at will. And so the fix was basically as brutally simple as you imagine it would be. It is that the server is like, "I'm you know, modulo some latency issues, I'm only

going to accept one movement packet for each client per tick." And the problem just goes away. Now this is like I said, this is sort of our primary form of mitigating exploits. It's always an option. It's always ongoing work. And the thing is is that as we do that work like bad actors continue to probe and try and find new areas. And this sort of evolutionary behavior on their part results in this treadmill of work. And we don't mind doing the treadmill of work. It's important valuable work, but it is a treadmill that we're sort of signed up for. The other primary form of anti-cheat that we utilize is VAC, Valve Anti-Cheat. And Valve VAC basically is a system of sort of client side detections. So I'm going to pay attention to what's going on in your system and then I'm going to anonymize that and scrub it of anything that might be personally identifiable and then I'm going to hand it up to the server. And now that VAC is only active when you're connected to VAC secured servers. And if you receive a VAC ban, it only prohibits you from playing on VAC

secured servers. But that is about 98% of servers. So you're you're pretty locked down at that point. And then the evidentiary standard for VAC, meaning like when VAC takes an action, we are 100% certain that you had your hand in the cookie jar. Uh with a caveat. And the caveat is that sometimes we make mistakes. But when we make mistakes, we rectify them clearly and immediately. There's a blog post. There's a clarification that this was the thing and it was wrong and we fixed it and we're very sorry. And as a result of this, our community has very high confidence that when VAC takes action that we are punishing cheaters, which is born out. But interestingly, I know that a lot of devs actually struggle with the opposite problem. They have an anti-cheat and their players believe that their anti-cheat really gets cheaters, but they're not actually certain that the anti-cheat isn't also punishing innocent users. We we definitely don't have that problem. We're very happy about that.

Okay, so we don't talk about VAC very often. And the reason that we don't talk about VAC is that talking about it makes it less effective. And in particular, talking about detections just stops them from working. So for example, if I got up on stage today and I said, "Hey guys, here's a detection. What we do is when somebody has a process and their process is scribbling into CSGO's memory, we go identify all of the binary parts of that process and then we go find the executable code that loaded that process from disk and then we MD5 summit and then we send it home and we figure out if that is malicious actor or like OBS or something instead." When I tell you that that's a thing that we do, I just lit that thing on fire. Like it doesn't work anymore. Or at least by the time this hits Twitch and you know, 3 seconds, it definitely won't work anymore. The good news is that we actually have thousands of detections that are sort of very sophistication and we write more every day. But again, every time we do

this, our bad actors evolve and the treadmill continues. All right, great. So let's let's get to why we think deep learning might be an effective component in our sort of anti-cheat strategy. But before we do that, let's talk a little bit about deep learning. So if you're not really familiar with deep learning at all or you're just familiar with the buzzwords, then I'm going to give you the 40,000 foot view. And the basic concept behind deep learning is that deep learning can help you basically just automatically build a function that maps inputs to outputs. So if you're familiar with algebra notation, you're probably familiar with Y equals f of x, right? And we're just saying like Y is the result of applying some f to some input x. And the cool thing about deep learning is deep learning says, "If I have enough x and I have enough y, I can solve for f." And that's pretty powerful. It turns out that then you can take that f that you have just built and you can apply it to sort of never before seen x values and

get actually pretty sensible y's. And if you don't use deep learning, then engineers are writing your f values. That's that treadmill of work that we were talking about. And that lets you make this trade that at Valve we're really excited about. That lets us basically trade data and compute for engineering time. Because at Valve we are always always short on engineers, but we have a lot of data and we have a lot of compute power. So if I were to take this analogy if maybe you don't follow along with the algebra and I were to take this into the real world, maybe you would imagine that if I had a bunch of cubes of meat and I had a bunch of hamburger patties and I had enough of them that and I told deep learning, "Hey, build me build me the thing that will convert these cubes of meat into delicious hamburger patties." It's going to give me a meat grinder. And that's so that is the analogy that we're going to stick with a little bit through the talk and I apologize to vegetarians. I'm really sorry. We we workshopped what else could it be? And it's like this one works really well. So

I'm sorry. Now you might have noticed that I said that we're only building a possible function that maps inputs to outputs. And the reason is basically data starvation problem. Um You could if you had enough x and y, like there is a closed form solution that is sort of the best possible f that converts that x to y. But the thing is is that you would need so much data. So so so much data. Like quintillions of examples. Quintillions and quintillions of examples even for very simple functions. So the reality is that like you're not going to do that. You're going to get just one of the myriad of possible solutions. Uh Now deep learning is actually really good at pattern recognition. It's really good at things that you're already familiar with. Like is there a cat sitting on the sofa? Answer is yes. And also if you have computer, the the computer in your pocket, if you have interacted with Siri or Google

Assistant, deep learning is made use of to transform the words that you say into text that then translates to commands that then is used to sort of do fuzzy lookups and it's used in so many pieces along the way. Now deep learning basically operates in two modes. There's sort of training time where you're like, hey I've got a bunch of cubes of meat and I have hamburger patties and I want to build a meat grinder. And it all and then once you've built that meat grinder, it's like great. Now I want to take these cubes of chicken and I want to turn those into a chicken burger cuz that sounds like a thing that might be good. Deep learning is that works in these two modes that we call like I said training and inferencing. Now I might during the talk accidentally say machine learning because we actually do a fair amount of machine learning at Valve as well. And the simplest way to talk about this for the purposes of my talk, if I use machine learning, you should just assume I mean deep learning. But but if if this sort of red circle represents deep learning, then this

orange circle represents machine learning. Like deep learning fundamentally is just subfield of machine learning. And simultaneously when people are talking about deep learning, what they're really talking about is neural nets. So if you learned those 30 years ago, it turns out they finally work. Spoiler alert, the solution to get them to work was to have a million times as much data and a billion times as much compute and then they work great. All right, so let's get back to our elevator pitch. So we were basically we were looking at like how do we attack this cheating problem? We tossed around a ton of ideas, but I didn't I really didn't want to sign up for the treadmill. Um and so looking at this it was like, well, you know, humans when they watch matches, they can tell that cheating is occurring and I think that's probably because they're detecting a pattern of cheating. And deep learning is really good at detecting patterns. So like shouldn't deep learning be able to detect cheating? And as I mentioned before, aimbots seemed like both the largest the biggest

problem because because they're not subtle, at least a lot of them are not subtle. And they're also this sort of really loud signal, at least it seems like. So it seemed like we should go after those first because if we can't do those, then we're definitely not going to be able to do the things that are more subtle. Now if you recall, I mentioned that deep learning is trying to solve y = f of x for f. The problem was like, well, if only we had a source for x and y. Good news, it turns out we had a source of x and y. For years, CS:GO has had this system called Overwatch, no relation, which is a jury review system. And basically what we have is we have players, they watch cases and we that that those are our x values and then they decide guilt through some process that I don't know. I don't know what their process is. And then they render a verdict. And and they do this in this way where they go through this process. They players report other players and then sort of central coordinator selects cases. By the way, that is a that is a

computer, it is not a person. There isn't like a guy at Valve that is like, I am the coordinator. That would be kind of cool, but it's not a thing. And then the players adjudicate those cases. They other players basically decide is this person guilty. And they do that through a process where they're sort of presented with an evidentiary window. The evidentiary window is eight rounds long and they're asked like, hey watch and the the play time is anonymized so you don't know who the players are and you watch basically a demo of eight rounds and then at the end you're asked, did this person use aim assistance? Only convict them if you believe there is evidence beyond a reasonable doubt. And we ask that for aim assistance, movement assistance, information assistance and other assistance that we're not sure about and also griefing. And if you're guilty, you get a ban. And if you're not guilty, there's no punishment, right? So it works kind of the way that you would expect it to. Overwatch uses naive Bayes to basically look for a guilty threshold.

That's a bunch of mathy stuff I'm not really going to go through, but the point is that according we we have a model and basically the evidentiary standard is that 99 that it needs to be 99.8% likely as decided by your peers that you were cheating. Now the actual rate that people get convicted at is usually five nines. Like usually convictions in Overwatch, the jurors are super super convinced that that person is a cheater. So as I mentioned, the system had existed for kind of a while. So back in 2015, we decided, hey we we noticed this trend. We did some work and as a result, it increased participation in Overwatch. And it's probably worth explaining a little bit about this graph. The gray line at the top is the sort of total participation in Overwatch and the blue line is basically the number of cases that were dismissed. The red line is convictions. And so we noticed this great trend. We're like, hey this is awesome. As more people participate in Overwatch, we get more convictions. So we should probably get more people to do Overwatch. That would be awesome.

So we did. Come on. Come on. Come on PowerPoint, you can do it. Oh, are you serious? Come on. You did it earlier. You you can do it. Oh, this graph, it just shoots up in participation. It's amazing. There's so much participation. It turns out the conviction rate doesn't move at all. Like the number of convictions is stone dead flat. And then that is a trend that we see continuing. On this graph, which is also amazing and I will totally make sure that those are fixed when we post. I don't know why they were broken. I'm very sad. I'm sorry. Okay, so the thing that we found in those graphs that you guys totally loved is that Overwatch really helps us with blatant cheaters, but it doesn't handle subtle cheats as well. And that it turned out that increasing participation, like incentivizing participation and increasing participation didn't help at all. Like it seemed like the problem was basically that we were failing to properly select cases.

Now the bad news is that as players got convicted in Overwatch, they would sort of evolve their behavior. But the good news is that jurors evolve as well. Like they're not fooled by your new, oh well, I'll just turn 75 degrees a frame instead of 78 degrees a frame. Jurors are like, no man, that's still cheaty Uh So the elevator pitch for VACnet was born. And the idea was that basically we would use this Overwatch as a giant pool of labeled data and we would train VACnet from Overwatch. And then we would build cluster of machines that would monitor matches with the model that we built. We would use VACnet then to submit cases and then humans would ultimately decide whether or not the person was a cheater. So back to our sort of Overwatch pipeline, all we have to do is slot in VACnet kind of right here. Okay, so how did we do? Man, I hope my graphs work over here. Oh boy.

Okay, good. I I have a later graph that I see is totally good. So we're just going to skip ahead. Great. So uh Oh yeah, hey hey here's the period. I'm sorry. I know I said I wouldn't move. Here is the period where we had increased participation right here. And you see this conviction rate just doesn't move at all. It was just horrifying. And and it and again, throughout here, like our participation fell and it just didn't impact the conviction rate at all. Sorry. All right. Okay, so um So we deployed VACnet. We deployed VACnet in early 2017. And during this time, participation sort of doesn't really change. Like it maybe is going up a little bit if you squint at it, but most likely what's happened is it's sort of just asymptoted down to the level of participation that we expect. It looks a whole lot like sort of any game launch ever that has sales, right? So here we deployed VACnet and you can see, hey we started getting more convictions. That's great. And we deployed a really simple model. We didn't quite have enough data yet,

but we were like, hey this thing's doing pretty good. The conviction rate overall for VACnet started to go up. And then a little bit later, we had the opportunity to sort of train with more data until finally in December, early December, we hit the crossover point where now like we actually get more convictions than non-convictions in Overwatch. So yeah, the system works great. So how great does it work? So human submitted cases, when a human submits a case to Overwatch, the likelihood that they get a conviction is only 15 to 30%. And that sort of varies on a bunch of factors. Like the factors might include the time of day, the day of the week, the time of the year, is the game on sale, is is it spring break? Spring break? And you know, there's a bunch of things. But but the point is that human convictions are very low. And VACnet convictions are very high. Like VACnet gets a conviction when it submits a case, it convicts 80 to 95% of the time. And it varies pretty much on the same factors with one exception. And

that one exception is that whenever we retrain VACnet, its conviction rate is like nearly 100% for a little while until cheaters adapt and start to react to it. All right, so hopefully I've convinced you this is kind of a cool thing and it works pretty well and I'm good at Excel. We can talk about how we train a learner. And we're only going to spend one slide on this because I I really just wanted here to give you a pointer to Keras, which is like if you're new to deep learning, Keras is great. But the sort of main sort of thrust of training a learner is that you need lots of x and y data and at least for supervised learning, which is the thing that we've been sort of talking about here. You need lots of xy data, you need to fit it and then sort of you just repeatedly ask, "Hey, how are you doing? Hey, how are you doing?" You have some measures that you pay attention to, and you're like, "How are you doing? Are you doing good? Are you doing better? Great." And then when it stops getting better, you're like, "Okay, hold the phone. I I've got my meat grinder." And then like I said, you want to take a look at

Keras. So we're going to jump ahead and talk about how we actually feed this thing. So you recall the metaphor. The metaphor is that we have this meat grinder, and all we need to do is feed it cubes of meat. And the problem is that as a game developer, you don't have cubes of meat. What you have is a cow. And that's not a commentary on your engine. My engine is the cowiest of all cows. But but what you have is not the thing that you need. And the good news is Well, if you think about it, like the bulk of the work of like, "How do I turn this cow into a delicious hamburger?" You're like, "Well, the overwhelming majority of the work is how I turn the cow into cubes of meat." Like that is the work. And it turns And it turns out it's actually like 95% of the effort that you're going to spend. And all of that effort is basically effort that you already know how to do. It is just bog standard engineering work. There's no sort of magic box around it. Only 1% of the effort is actually kind of around like deep learning,

turning you know, in our metaphor, it's turning the crank. And 4% is cooking I don't The metaphor breaks down a little bit here. So we should We should probably just leave it there for a little bit. Maybe we'll come back. Um If you have a game, to sort of be a little more concrete, there are some pieces that you really want to have. And I think the most important is that you want to have a replay format. We call it a demo format. You know, other engines call it other things. But the sort of salient point here is like, you basically want to have a record that you ideally recorded from the server or from a third-party observer. Like you don't want the client to record this thing because they will just lie. And you want to be able to play that thing back later. And then you need, you know, all of the sort of infrastructure around storing and fetching those things and transmitting them. And you're going to want to manage metadata so you can say like which player was in this match so that you can evaluate them. And ideally, and this is the big ask, I know, you need a data labeling service,

ideally. Now, I do actually think that there is a way to attack this problem that doesn't require a data labeling service. But I haven't proven it yet, so maybe I'll see you in 2019 and we can talk about it then. Um And I want to switch gears a little bit now and talk about sort of our data herd. So I mentioned before that we have roughly 600,000 5v5 matches per day. And for each of those 5v5 matches, to evaluate all 10 players, there's roughly 4 minutes of work that we have to do. And so if you do the math, you figure that is 2.4 million CPU minutes of work that we need to do every day, and somehow we need to fit that into 1,440 actual minutes because that's how many there are a day. And you can do this by just using a bunch of processors. We need about 1,700. So that's what we did. We bought a giant farm of machines, and this is actually only used This is not used for training. This is only used

for running the inferencing sides of things. And this is actually only half of the farm. The other half is in a different data center. I This is our actual farm. I took this picture. I went out with the DevOps team when we got them. I was like super excited. I'd never been to the data center before. And it was like candy trip. It was great. And then I was like, "Wow, it's really loud in here, and it's really cold on this side, and it's really hot on that side." But it's it's totally worth going. You should go if you get a chance. Anyways, what we're looking at here is basically these two chassis. And um Each chassis basically has 16 blades. Those are the things that are sort of vertical. And each of those blades has 54 cores and 128 gigs of RAM. So they're they're pretty beefy. That's about 3,500 processors total. And we We We bought basically two x what we needed so that we would have room to expand. And we'll maybe talk about that in a little bit, too. Um Like I said though, this is basically only used for inferencing time. This is like, "I've already built the meat grinder, and now I need to actually

watch all of the matches." And like I said, 95% of the work is basically around turning the cow into meat. For us, 95% of the work is like cracking the demos and then building inputs out of that to feed into the learner. And then it's like like 1% of the work is the part where it's like, "By the way, does this look like a cheat? By the way, does this look like a cheat?" The The part that we do the training, the part where we actually build the meat grinder, that part is just run on commodity 1080 TIs that sit under our desks. Like they're it's totally normal hardware. Okay. So now I want to talk brass tacks. I want to talk about like what goes into the cubes of meat that we're feeding our our model. Deep learning has sort of these specific input constraints. In our case, we've said it's cube of meat. And that means that the meat needs to be sort of uniform and fixed size. And if you're uncomfortable with the meat analogy, and maybe you're familiar with SQL, then you can kind of just think about like this thing just needs

to be fed with a database where all of the columns are non-nullable. And every time you ask run a query, you always make sure you ask for all of the columns, and you ask for the columns in the to sort of like deep learning algorithms need to be fed in that way. Now, if you recall, we decided that we were going to specifically target aim assistance first. That was the first thing that we were going after. And if you think about like when somebody is cheating using an aimbot, they're going to basically use They're going to express that cheatiness like when bullets when they're firing bullets down range. And so we basically did some manual feature work around around that. So here, we're going to sort of have this atom on the right, and we'll fill the atom with stuff. And the first thing that we decided to stuff into the atom was like, "What weapon is the person using? Are they using an op? Are they using

an AK-47? An M4A4?" Cuz no one uses anything else. They're using one of those. And then It's cool. There are other guns. They're great. You should try them. Then then we want to know basically what was the result of of that shot. Was it a headshot? Was it a hit like but somewhere else that wasn't the head? Was it a total miss? And we want to stick that thing into our sort of atom of stuff. And then we want to know like, "Hey, given that I actually did manage to get a hit because I don't have the answer otherwise, how far away was the target if it was a hit?" And we have a sort of special value that we reserve for the cases where we're like, "I don't know. It didn't hit." And the learner is fine with that. And then this one requires kind of a little bit more unpacking. What we What we needed was basically the sort of delta view angle pitch. And so what that means is like for about a half second before the shot and up through every tick until a quarter second after the shot, we basically say like, "What is

your delta pitch on this tick versus the previous tick?" And we just repeatedly do that. And a half second before and quarter second after were sort of chosen purely capriciously. But we take all of those, and then we stuff them into our atom. And if you've been watching the atom, you might have noticed that like everything lands in the same place every time. Like we make sure that all of the data lands in the same place every time because deep learning doesn't really care where it is, but it cares that it's consistent every time. Like that is a necessary property in order for it to learn. So then with the the delta view angle Sorry, I probably should have clarified, and I will do that right now. The view angle is obviously just like the way that you're kind of looking around. And before we were looking only at pitch, which is basically your up and down. So we were like, "Hey, like you looked up 1°. Hey, you looked up 2°." And then we need the same thing for yaw. So we want to know like now right to left, how much did your view angle adjust right to left?

But only as expressed as a delta against the previous tick. Like, "I turned right 1°. I turned right 2°." And we're only doing this around in the windows around when you take a shot. And so we do this again, half second before and a quarter second after, and this was sort of picked purely capriciously. And then we stuff that thing into our atom of stuff. And so we have our X, right? Well, not really. Because if you recall, our training data is not that this particular shot came from a cheaty demo, and this particular shot did not come from a cheaty demo. Our data is labeled where it's like, "Hey, in this eight-round window, something somewhere in here triggered a juror to decide that this behavior was cheaty and and convicted." And it Sorry, convict It convinced a bunch of jurors that that this behavior was cheaty. So we can't just use one atom because this is only the information about one shot. We need like a sequence of them. We need like big sequence of them. And in fact,

actually what we want is 140 of them. Or at least that's what the model uses right now. It's also sort of purely capricious. And uh I will make a slight apology because this is a cube of meat to go with the metaphor we've been doing. But actually, it is just a serial representation of 140 shots. We just take the 140 out of an eight-round window, and we stuff those into the model, and then we're like, "Hey, if you If you were to present this sequence of 140 shots to a juror, what is the likelihood you would get a conviction?" Okay. So I spent a bunch of time early in the talk talking about how we can't talk about anti-cheat because talking about anti-cheat makes it not work anymore. So, did I just spend 40 minutes like lighting this technique on fire? It was like a year of work overall. I mean, we deployed it in like 4 months. We're pretty excited about that, but like did we just light this thing on fire? And the good news is no. No, we didn't. It's great. Let's talk about why we

didn't light it on fire. So, if you recall my original sort of elevator pitch was like, "Hey, we're going to train data from VACnet." Uh sorry, from Overwatch and then we're going to have it monitor the matches and we're going to submit cases and the players are going to determine guilt. And the reason that we didn't light this thing on fire is because we're actually going to do that thing in a loop. We're just going to do it over and over and over and over again. And because we're using Overwatch and we didn't actually replace all player reports, we just supplemented player report behaviors, that means that uh the learner is getting the opportunity to evolve along with human jurors. So, as human jurors identify sort of new uh cheaty behavior, the learner has the opportunity to do the same thing. Uh so, you can think about um this a little bit like imagine that uh you're a cheat developer. There might actually be one in the audience, I'm not sure. Uh but uh you're a cheat developer and you're like, "Aha, all I need to do is uh make my cheat look cheaty 3/4 of a second before uh shot and then look

totally normal through the half second and then uh get the kill and then move on. Now, I think that's extremely improbable in the first place, but it actually won't work anyways for two reasons. And one is that human jurors will just be like, "Yeah, dude, I don't care. That looks cheaty." Like boop, conviction. And then uh the learner will retrain and it'll be like, "I don't know why locking on in this particular way is considered to be cheaty to humans, but in order for me to do the best job I can do, I need to label that as cheaty." So, it will just learn the thing. And the other reason is that even if that didn't work, retrain like we chose half a second before and a quarter second afterwards just totally capriciously. We could just make it 3 seconds before and 2 seconds afterwards and the model would be just as happy. It doesn't care. It would uh effectively uh result in us saying like, "Oh, we need to retrain. Uh so, let me take the 5 seconds to find the command in my command history and press the button to start the retraining and then that'll run for 6 hours and when it's done, uh

uh now it's happy with looking at stuff um for that one." All right. So, I think we're mostly the way through. Uh let's uh talk a little bit about sort of ongoing and near-term work that that that's going on around this. Um we're doing uh so, one of the things is like I mentioned like right now when we retrain, we still sort of just go and look uh at uh like we basically just look at like how is it doing? And then we decide like, "Okay, now seems like a reasonable time. I'm going to go retrain." And I don't think that's actually the right answer. I think the right answer is to just totally get us out of the loop and instead basically just say, "Hey, like on a cron job, this should just once a day uh retrain and then if it's convicting more stuff than the previous version, then uh then we should uh just deploy that one and we should just make that heuristic-based and get totally out of the loop." The other thing that we're kind of working on is basically uh new learners uh and uh

the learner that we built, the features that we built are sort of based around catching aimbots. But um we actually think and part of that was because we didn't have enough data yet that we were willing to sort of go whole hog on this, but we think that a better approach is actually instead to just sort of build a first-person narrative for each of the players in the round and say like, "I don't know what matters in here at all." Uh and I I don't want to do any um input engineering. I just want to basically say like, "Here is a first-person narrative for this player. If you take eight-round windows of this first-person narrative, does that look more like a cheaty first-person narrative or more like a non-cheaty first-person narrative?" And we think that that model um is much more likely to uh be able to attack the other types of cheating that uh that we see. So, like wall hacks and and so on. Um the other thing that we've uh been kind of working on is like generalizing VACnet to other game modes. So, we deployed um to uh we have a 2v2 competitive mode that's pretty popular and we deployed to

that uh few weeks ago and the conviction rate in that mode was like 99% for a while. It was great. Cheaters didn't get the memo that we were doing it and players were super happy and we were just busting cheaters left and right. It felt so good. Uh finally, we're actually looking to generalize VACnet to Steam. Um we think that the model both the model we have and the model that we're moving towards uh should actually with a relatively small number of examples uh should generalize to other initially first-person shooters, but then later uh to just other games in general. And we're definitely looking for partners uh to make that happen. So, if that's the thing that you're interested in, we should talk. All right, let's uh finish up. Um So, basically I I think hopefully I maybe convinced you a little bit of this, but our our sort of opinion at Valve is that uh deep learning is just this sea change technology for evolutionary behavior. We think that it is really actually helping us get

developers off of the treadmill without impacting our customers in any way. Like our customers are seeing fewer cheaters today than they have been uh and the conversation around cheating has died um tremendously compared to where it was uh uh before we started this work. Uh like I think the the volume and I I'm sorry to give a crappy anecdote, but I think the volume based on CS:GO team feedback and uh and sort of eyeballing uh the Steam forums and Reddit is like the volume of the conversation is probably 1% as loud as it was uh before we started this. Now, we're applying deep learning to a bunch of problems at Valve. There's anti-cheat and anti-fraud. Uh Dota hero selection uses deep learning in a ship 2 week and a half ago. There's some other Dota stuff that's more complicated and I don't want to explain Dota, so I'm not going to talk about it, but um that stuff shipped using uh deep learning as well. Uh and and there's a bunch of other work going on as well. So, I mentioned uh before I would uh not try and teach you deep learning, which I

successfully avoided and instead I would point you at some resources. Uh the foundational work I would sort of suggest um this Udacity course is uh really useful if you were a super high-level user and you do not give a crap about how the insides work, the Udacity course is actually great. Like you will come away with an understanding of uh like uh sort of how to build systems that use deep learning, but you will definitely not understand the guts. Uh meanwhile, the Coursera course, the machine learning course um will help you really like actually build the pieces from the ground up. So, if you're kind of more like me where you're like, "Nah, I need to see how the sausage is made at least once." Uh I recommend the Coursera course. Um they're both about they're each about 30 hours of study uh you know, that you can do sort of at your own pace and they're they're both great. I recommend them both. Um additionally, once you've done that and I I point you at the machine learning courses because it turns out everything that you or at least most of the things that you learn around machine learning will be super super helpful at understanding deep learning. But once you've basically done either of those, I

suggest basically uh jumping into either the Coursera course uh that's deep learning specific on Coursera or um the Stanford course uh CS231N uh with Fei-Fei Li. Um the all all of the material is online except for the lectures. Good news, there are bootlegged version of all of the lectures on YouTube. I highly recommend the ones with Andrej Karpathy, but they're all good. Uh and uh so, you can go through the basically the whole course. Uh all right. Uh so, hopefully people have some interesting questions. If you have questions, please walk to the mics. Um I did want to mention that sort of the whole thrust of the talk is like how we can sort of increase value that we deliver to customers and deep learning is an important piece of that, but like the other way is like to just get more awesome people at Valve. So, if you thought this was interesting, uh you should definitely apply at valvesoftware.com/jobs. And uh also while it's Sorry, one more thing is please fill out the surveys. Pretty please. Okay, I'm going to say you were there first, but I'm not positive. Oh, so I'm not vegetarian, but

oh okay, I'm fine I'm fine with that part. Anyway, uh so, there's the there's the project there's the project that her people are working on that with the Unreal Tournament. So, where people would actually use machine learning to train robot to play like a human. So, do you actually see that as a concern in in this problem space? I see. So, the the question is am I concerned that somebody will basically counter my deep learning approach with a deep learning approach uh trying to play like a human? So, uh no, that's a that is a really good uh that is a really good question. Um Maybe. So, there might be uh this is a little deep learning specific. I apologize. Um there's maybe a an attack with GANs that might sort of yield fruit, but I don't know uh for sure. That's definitely something that somebody would have to spend a lot a lot a lot of compute time on, so that translates to them spending a lot a lot a lot of money. Uh and um otherwise, they just don't have the data. Like a slide that I didn't have about here up here, but that's sort of fascinating about deep learning is that uh it turns out that the algorithm you pick in deep learning and machine

learning, it doesn't matter. The thing that matters is more data. Like more data wins. And so, the problem is somebody is going to either need to outspend the amount of data that I have cuz I have lots of data. Uh and also, I could just do the same thing. Um or they're uh they're going to um need more data than I have and they definitely don't have that. So, that's a good question. Thanks. Uh yeah. Um when you showed the um the rainbow slide, you said Uh rainbow trust. Let's get back to that one. that you weren't sure about uh any of the players, but you were more sure that some of them were more likely to be cheaters, right? That's right. So, in order to make a better experience for the blue guys, Yeah. five of the red guys who aren't cheaters, but for some reason you mislabeled them as untrustworthy because they're not cheaters, you just think they're more likely, you're going to ruin their experience like completely, right? Huh. Aren't you worried that you're going to make them turn in order to make the others have a better experience? concern

for trust. Like that's definitely a thing that players are concerned about, but we haven't seen it actually bear fruit. Like when uh that model also tends to be very conservative. So, we're actually despite what I'm showing here, we're actually more likely to accidentally slip somebody who is a cheater into blue than we are to get somebody who's red uh to accidentally qualify somebody who's orange as red. So, so would you recommend using this if you're really sure that they're cheaters? Yeah, you want to be sort of very you want to be um certain that they are very likely to go on to be cheaters. Like most of the people that get caught there have are actually in the situation that I gave the anecdote about where they have many accounts and most of them have been back banned. Okay, thank you. Yep. Hi. Hey, yeah. Um so I'm a big fan of you guys using K humans in the loop to make sure you don't have false positives. Great. Uh but how are you planning on extending this layer for it for like for example, you're asking for partners for this and those partners may not have the level of community participation that that uh you

that Valve does. Yeah, that's a good question. Um we haven't thought that far yet. I I think that uh probably um it's possible that the actual feature would be the whole thing like Overwatch and everything. That's one possible approach. The other would be to um use something like this maybe instead to um basically to feed into something like trust score where you're like, "I'm not taking as nearly as harsh an action. Like I'm not banning you and taking away your stuff. Instead, I'm just like, 'Yeah, I don't quite trust you as much. So, I'm going to stick you with other people who I don't trust quite as much.'" So, I think like soft punishments might be a better uh choice for a smaller partner. That's a good question. Thank you. Hi. Hey there. Are you seeing an increase between the overlap of cheaters that are really bad players and actual legitimate good players? And how do you deal with those scenarios? Um I'm not sure that I understand. Can you rephrase that? So, at some point a very good player is going to look like a cheater. How do you deal with that scenario? I see. Uh it

turns out um so Overwatch uh so the the question is, "How do you tell the difference between somebody who's just really good at the game and a cheater, right?" Uh and um it turns out um that uh the things that Overwatch is good at catching are not like Overwatch doesn't catch subtle cheaters because we ask players like don't convict if you're not sure. And so they're like, "That is plausibly just a player that's really good." And so they just tick the not guilty button. And we we've sort of hand verified that tons and tons of times that like, "Hey, let's spot check convictions and make sure that like people are still not convicting things that they shouldn't be." Um and we haven't found a single case of it. That's not to say there hasn't been like one case in all of Overwatch. Like there probably has, but uh but but we haven't seen it. Yeah. Hi. Are you at all concerned about the possibility now that your secret is out of cheat developers polluting your data pool by putting ringers in your juries? Am I worried about uh so am I worried

now that I've basically shown everything about cheaters uh polluting my data? Um they'll try. I'm not really too worried about it because humans should be able to tell the difference. Like that's kind of the fundamental premise that we're going on. And so um if humans decide that that behavior is cheating, then the learner should decide that it's cheating as well. But it's possible. like it's definitely We'll see. Maybe I'll be back in GDC 2019 and I'll be like, "Don't talk about cheating ever. This was a terrible mistake." Uh but I don't think so. That's a good question. Thanks. Hi. Great talk and great analogy. I really like the uh Yes. Yeah, one for and one against. All right. It reminded me of the burger I had yesterday, the super duper burger. Good. Good. Good. Good. Good. Um I have two questions. First one is about the the actual data shape. Like you showed it the the cube shape data, but I don't think that is actually cube. No, no, it is not a cube. Yeah. So, I was wondering like is it like a the one user is represented as a long vector or a matrix

Yeah, so what it is is it is actually it is totally sort of serial data. It's just serial data doesn't look very good on a slide. Uh but it is basically just like here's all of the information about a shot and then repeated 140 times. Um and if a if a player I didn't really talk about this at the time, but like if a player shoots say 200 times in an eight round window, then we just build a like 60 140 shot windows and we ask to inference all of them. Would any of these result in a conviction? Oh, I see. But but there are yeah, like it's like you can just sort of imagine a timeline where somebody takes a shot every single time they take a shot. And then we're like, "Great, let's look at like that is our the whole timeline." And then let's build sliding windows of 140 and see if any sliding windows will result in a conviction. I see. Uh and my next question is about the deep learning architecture. If you have Oh, I have a slide for this. Uh let's Here I'm just going to Oh, wait. Maybe I can No, I'm sure it won't.

Sorry. Maybe I can skip to the No. No, PowerPoint does the I'm so pissed about those graphs. So, I I actually have the slide for exactly what our architecture is. Um and bam. I'm so glad I kept this one in. Uh so, uh we use uh an RNN I'm sorry if you're not a deep learning person. We use an RNN because we have sequence data. Uh and the only sort of weird thing is like we're actually using a GRU and ReLU activation actually works better most of the time. Uh ReLU occasionally when we're training a model, it'll just be like, "I just can't converge." Um but actually most of the time it does converge and it does significantly better than tanh. Uh so, that is surprising. Uh and otherwise this stuff is all just totally purely capricious. Like we just searched uh we just did like a grid search and we treated all of the architecture like a hyperparameter you know, like a set of hyperparameters at all as well. And uh that is like the thing that currently works the best. Could change tomorrow,

but that's what it is today. Um I was wondering if uh you have also tried like one-dimensional convolutional neural network which I uh it is high on my list of things to try. Yeah. Thank you very much. Yeah, cheers. Thanks. Hi. Uh so, as a person who does a lot of Overwatch cases, it's kind of interesting to see how you're using this data. That way, hold on. Thank you for your service. Yeah, no problem. Okay. Um so, I'm kind of curious about the more subtle cases cuz just speaking purely as a person who does does Overwatch cases is there are a lot of times when you aren't certain. Um and I think the language is actually really good about

really now what do I do?" Because like humans are not convicting that and like I would need to have a like extremely high confidence uh that a person was cheating in a subtle way before I would be willing to take action against them cuz like I'm putting a black mark on their Steam account. I'm taking their stuff away. Like they don't get to play in back secured servers. So, we need to be sort of very, very careful around uh what we do uh punitively in that situation. But it may be that you feed it into trust and then you're just like, Right. I I think it's also interesting uh have you done experiments with with uh

how tick rate affects this? Uh I'm sorry, how what? How tick rate affects Tick rate. Because you have a relatively high tick rate um at 64 which gives you sort of a lot of uh moment-by-moment data. And I don't know if having a higher tick rate or a lower tick rate evaluation of this uh has any impacts. So, um Overwatch cases are actually handed down only at 32 K tick. They're already stepped down. And so we we actually just work on that data. Okay, interesting. Cool. Thank you. Cool. Uh I think I have like time for one or two more questions, probably. Although we're the last people in the room, so maybe we can go a minute or two long, maybe. Okay. Cool. Um kind of uh dragging on a bit. Here's my love Counter-Strike, by the way. I play a lot and everything. Um going on with that, right? You definitely I I think the data part will always win. You guys always have the data and you can always get the data. So, I'm sold on the machine learning part. But when SteamOS came out, I was actually hoping, you know, we've got Twitch going big and we've got people making entire livelihoods on this now.

And it made me wonder why, you know, we have secure boot, we have all these systems now. Couldn't we create, you know, in addition to this, could we not create a a secure system in such that it's like for contestant play, you have to boot these sort of encrypted images that are a whole lot more secure. I mean, this is a whole 'nother conversation, but but it allows you to do the things like, "Hey, did he actually move his mouse physically? Like did I get XY input from that? Hey, did the DLLs exactly match you know, I'm doing checksums and you know So we've thought about this and like I actually that was kind of the approach that I ran down initially and there are sort of a few problems around it that led us to go I think the easiest one is like that that feels super invasive, right? Like as a user like I'm like hey what you need to do is play my game on like my OS and you need to think and like I don't know like and the the problem is

ultimately at the end of the day if the user has access to the their to their system physical access there is nothing I can do to determine for certain that they haven't tampered with it. Like cuz any query that you're like well you just ask them this I'm like so what I do is I hijack that function and I lie. Because I did that I've like I've done that it works great. Like it's it's turtles all the way down. Ah, but it may not be and maybe we should talk later. Oh yeah. Hi, great talk. Um quasi related because you mentioned like Reddit and the forums a lot at any point did you develop something to sort of scour those in terms of watching trends for like No, we haven't been maybe we should. The question is like do we have tools to scour Reddit and like quantify things better and other forums. Um we don't but maybe we should. Like that that feels like a thing that actually and you know, performing sentiment analysis in particular being like hey is there like a new topic that is currently lighting the internet on fire like that might

actually be a pretty valuable thing but it's not a thing that we have right now. Yeah, that would be that would be great though. Hey, two questions. First great talk. Thanks. So firstly how long does it take to train the data? And how much data do you use? So we have about 700,000 labeled pieces of data now and it takes about 6 hours to train. On a 1080 Ti on 1 1080 Ti and we're actually working on parallelizing it and feeding it to a bunch of V100s so we'll bring that down to 10 minutes or so pretty soon. Cool. And the second question have hackers tried to play the overwatch system? And ads and people who just pass everything. Uh no. Well sorry maybe maybe they've tried but it but one of the things that we do in overwatch is we pay attention to how good you are at overwatch like how often you go and agree with the consensus. And

so somebody who is just like not guilty not guilty not guilty not guilty and they do that repeatedly on cases that get convicted or the other way like they try and pollute it the other way they get they get basically to the point where we're like you can keep doing cases but I don't believe you anymore so have a good time. Yeah. Hey cool. It's a good question. Hi. Hey um I think these are probably the last two just so we're Thanks a lot for the talk. Now I really know what's your next game going to be. It's a VR simulator of a meat grinder. Right, meat grinder. Sign me up for that. But The actual I have a question from a different point you know, obviously you're talking about real customers the real players playing and that's why you have created that system. I work for a competitive play so we see lots and lots of games happen when people play for actual money and I think CSGO is the biggest game in this amount in terms of in terms of volume. You know so many tournament organizers running so many tournaments so many games are being played.

And the level of cheating there is slightly different you know, people are using enhancement drugs they can throw matches and I'm just curious is there a way for a deep learning network to try to sort of sift through that thing or we as tournament organizers should look elsewhere. Obviously we are doing it on physically when LANs happen everything is secure there but I'm more curious about online qualifiers. So the the question is basically how do we do this for pros right or how do we do this for as tournament organizers and I think that there probably is an approach there that works but I don't have enough confidence to talk about it right this second but it is definitely a thing that we're thinking a lot about because we think the you know, the integrity of the pro scene is super important like it's it's really really important so it's definitely a thing that we're thinking about but I'm not quite ready to talk about exactly how we think it'll work. Yeah, sorry. Yeah, thank you. It's a great question. Last one. Hey, I'm wondering how many of

your players have to weigh in on to a overwatch case before it is closed one way or another and how many overwatch case do you close every day? Uh so I can't comment on the second one because the team agreed that it probably was not actually that useful to talk about. Um the first question how many players need to weigh in? Um so it kind of it it varies. Uh I'm trying to answer and not be hand wavy but um it kind of depends on how long it takes to converge like cases can be sort of rejected like if you get if you just happen to randomly hand the case out to a bunch of people who it turns out are like they're always right they always go with consensus then maybe it takes as few as seven of them to agree that that a person is you know, this thing is super guilty or this person is definitely not guilty. Um it might take as many as 50 or 100 if it's something that's like a little sort

of less certain or if you know, there are people that are new to overwatch and we're not sure you know, about how good they are at identifying the conviction rates. All right, thank you guys so much for sticking around through the graveyard shift. I really appreciate that. It's great.