Skill, Matchmaking, and Ranking Systems Design

[Music] alright let's get started thanks guys are all for all coming out hope this will be interesting for y'all make sure to sign in to your cell phones and fill out your surveys so I'm Josh Menke I'm going to kind of go through some of the things I've learned and some opinions I've created for myself and my own you guys are free to adopt my own opinions or adapt them to your own or completely reject them and this is based on experience I've had working for blizzard and some of their games at 3:43 and halo and now working on at Activision in general so that's kind of where I'm drying my background from we all have different backgrounds so you can a big part of my talk is like I'm gonna talk about some tools and then you guys would have DAP them to your own games here's the outline I'm gonna start with a little bit of design philosophy influenced by my background give some definitions and then just cover in order

skill matchmaking and ranking there's some design philosophy kind of starts off with study of easy to learn hard to master which is a value of probably a lot of you've heard before which is very similar to this in game design for news for me - a classic stephen r.covey like his second habit if you've read seven Habits of Highly Effective People which is begin with the end in mind so start designing thinking about what are people gonna do when they finish the main part of your game or game or the campaign or the progression system in your game how are you going to get them to keep playing which is very similar to something Rob Pardo said at AGC 2006 which is the way Blizzard was designing their games was that the they first tried to add years of replayability that's the first thing they do and one way to add replayability at least in my mind is to add skill depth you make it so there's still things for players to continue to learn years and years into your game and to get better and that goes kind of dovetails into skill and matchmaking because good skill based matchmaking allows you to have skill depth in your game without making it so

you're hurting players that have not yet gotten into the depth of that and learn all the amazing things and about how to how to play that and ironically less skill depth can actually make it harder to match make it makes your game less predictable and makes it harder to find where the good players are so they're still there they're still dominating but they were getting randomly beaten in cases that are unpredictable and so it's harder to find them another kind of design value of kind of cultivated over the years is is even if the court your core audience of your game is small it's dangerous to ignore them because they have a loud voice and the mass audience even if they don't play the same way the core audience plays tends to listen to the core audience and judge your game by their opinion rather than go into your website and reading with you try to convince them of so it's important to engage your community and your core audience in there another value again talking about audience and this picture definitely exaggeration right but is the point that you should consider your audience and and so like I'll present some tools and some ideas and this is

definitely wrong right it's just a joke but it just stick with you in your head that this presentation contains some tools but you should really think about your own audience like what I'm saying is based on my opinions and my experience but you all have your own games and your audiences and when I work with teens I make sure to really try to understand and talk to their your multiple multiplayer designers and figure out what your audiences are like because different gamers have different passions and so some of my stuff will be completely wrong for you but it's hopefully the way I'm thinking about and presenting it will help you to adapt it to your own audiences so all right so here's some definitions and these are three the overloaded terms probably skill at matchmaking and ranking so I wanted to kind of go through how I define them to me skills a skill system is simply anyway if you're gonna how good your players are that's all it is that's the skill system give you lots of different ways I'll talk about that next is a matchmaking system that's just putting players into matress a lot of that is just really hard engineering with dealing with large or small populations and getting them into the matches it may or may not use a skill

system it may or may not need to know about your ranking system but definitely will influence your skill and ranking systems and I'll go over that so it's important if you do want to have skill systems and ranking systems that you consider that your designers and your engineers that are working on those systems should be communicating with each other and then a ranking system is okay because I have a skill system and maybe I know how good my players are should I actually tell them how terrible they are or do I tell them something else and this might use this skill system and it might not use this skill system but they are three separate systems and your audience is going to tell you how to do that so now I'm going to go over each of these pieces from my point of view here so first a skill system what is it again like I said is any method you want to use to measure player ability and ideally it's predictive ideally this myth that you used a measure skill lets you in the future predict what's going to happen in your matches and why do you care about a school system maybe you're going to use it for matchmaking maybe you're gonna use it for ranking or maybe you just want to understand your players and how skill is spreading out in your game and maybe want to understand how depth how

deep your game is and how much skill depth it has and that's something that being able to measure skill in your game can help you with so let's talk about some some skill systems so there's simple stats you can use for skill in like a shooter just kills per death their score things like a p.m. to kind of measure good some are better than others depending on your game and and they can actually help you figure out how good your players are although just as an aside before I get into the matchmaking section skill can have matchmaking interaction issues so if you find that hey kills per death is a great way to matchmaking my game and also I love I really want to match me I really want to rank on kills Bertha so players can see themselves well it turns out if you match will make really well on kills per death everyone ends up with the same kills per death in the end and so now you can't really rank on it or very similar kills per death so if you really feel in your game that I want to show players their true kills upper death instead you need to matchmake on a representative cross-section of your population to make sure that they're

their true kills per death ratios coming out of that another type of rating system you might have heard of our our star skill system the rating systems and this is where we put a magical number on the player and that's ideally predictive that I can compare two players magical numbers and come out with some guests about how they're going to perform if they play against each other this is where you see values like 1500 or in other systems you may see values that are really simple like minus 2.5 3.2 things like that so if you've heard of skill systems you may have heard of yellow-backed yellow is not a skill system it's a rock band this is sorry this is a pet peeve of mine it's a common misnomer I have heard of a guy though named Arpad elo who was a physicist a chess master developed a great rating system that kind of before 1960 good for its time I think the core model he used is still a pretty good core model though it wasn't his like if you want to get really geeky it's even older from sunlight I'd never

get it wrong it's a cognitive psychologist back in the experimental psychologists back in like the 1920s yeah I think it's inefficient by today's standards and I don't think it addresses all today's needs and all a lot of people shoehorn it into their needs and that usually makes it even more inefficient so what is a good skill system then I think it's one that finds players skills super fast especially if you need it for matchmaking t if you're new your game it is important to have good matchmaking then being able to separate the good players and the bad players fast is important because the third is aren't going to stay on too long if that's the case and I also think it should predict match outcomes correctly so if you give me two players a and B hopefully your skill system will tell you how often player a will be player B and hopefully not only will it tell you how often hopefully it's right if your skill system is saying 75% of the time and you have a play B 700 times then they should win 75 of those times it should give the right probability some people get excited that their skill system gets probably abilities but never go to look and see that they're actually

giving wrong probabilities and this is important because you should know what your skill system is doing and this is what I mean by super super fast so the state of the art that I built recently has this type of comparison this is from actual like real data simulations and this is the difference so the dotted line is this player's actual skill and what I have on the bottom there is how many games these players are playing so this is supposed to show you how fast it takes the different methods to find out your skill so if you just naively grab something like elope you're gonna have a gap of about this type of situation where it's been 100 games and elos doesn't know how good this guy is whereas a state of the art system should get it within a few games that's just to kind of show that contrast not the AAE low and then well the other thing I mentioned is this important to make sure it predicts correctly so how do you do this well you get a set of matches that were not match made because that actually biases it and I won't go into all that math and reasons I'm just telling you get a set of matches that were not match made run your rating

system whatever one you picked after each match and then when you do that that rating system should provide you with the probability that team a was going to be team B or whatever you have so you want to record that probability for each of your matches and then for every little probability you've been them let's say just and that might sound complicated let's say just pick every time I saw 75% chance of winning for the better team plus or minus like some small amount point one or maybe 1% or something every time I saw that I'm gonna look at all the times I said 75% and then count up how many times the better team actually won and that would give you a percentage of that there there you'll get say there were a thousand times I saw I had predicted 75% and out of that 749 times the better team won that means your systems kind of calibrated pretty well and so you do that for all percentages and then you plot those what you predicted against what you actually saw and hopefully you get something like this if it's a good rating system and I'll tell you that this is possible in almost every game

both at this point every game I've worked on which is a pretty broad range of games and genres so I want to know that my predictions are precise that maybe that's just me but I want to know that if I'm saying 80% it is 80% and why well let's look an example of here of when you're wrong okay so in this case this is called it to me from my mind overestimating how important the skill gap you had in your game was my system is saying in this case hey there's an 80% chance that player a is going to win and so I'm not going to match make these to you guys I'm gonna make him wait and find somebody else but turns out it was only a 65% chance so you're making them wait longer than they really need to for this match that at least it that's important in your game and you're going to lose players because you're making them wait too long that's the danger of overestimating and arguably even worse perhaps is under estimating in this case I'm saying oh it's a fifty five percent chance that egg is going to beat B so it's not too uneven let's go ahead look at the match but it turns out it's actually an 80% chance so now you're gonna lose players because you're throwing them into lots

of games that they can't win you won't lose the other player because he's winning a lot but the the one that's getting beaten might not be too happy so that's why it's important to kind of get this right to know how your system is calibrated don't just use the rating system actually understand what it's doing now another nice kind of a site about all this is it's kind of nice if you go ahead and make a histogram of all your player ratings all these numbers you've created it's nice if it fits like a bell curve which is also called a Gaussian or a normal distribution because this tells you what skill looks like across all of your players and if it's normal you can make a little cute shortcuts like I can tell a player's relative rank or what percentile they're in just by knowing their number there's functions you can find in Excel for that I won't go into all that but it's pretty easy I can also do interesting things like make fake players so I know my player distribution looks like so I can make a bunch of fake players by sampling from this kind of histogram and and now that I have fake players and I know that my skill system does a good job of predicting who would win matches I can create fake matches and being able to create a fake players

and a fake matches is extremely useful if your wanted to test ranking systems matchmaking systems or just you know debug your rating system that's it so it's pretty helpful and this is an example of actually some real data of like almost two million players two million matches around and the black line is what the players actually look like and the red line is what a true perfect normal distribution would look like and it's pretty close and again this is convenient it's not a hundred percent necessary but it is definitely convenient it lets you sample lets you do lots of fun stuff and and I'll say from experience again I've worked on a lot of different games and I'm getting lazy these days because it always looks like this it's not exactly this sometimes you have to do a little bit of shifting and scaling but it's so pretty close to this and so I when I look out in the audience now I see a bunch of bell curves I'm sorry and so now you're probably wondering well how do I do this get this amazing that's not that amazing but how do I get these skills off players the method I use remotely usually is like elos method it's just

better so if you're used to what ela did then you have some familiar where the ratings go up when you win they go down when you lose the amount that you go up and down depends on the strength of the opponent that he was playing against I think he was probably too slow don't worry about the mouth on the side it's just a picture but a good place to start reading is a paper called online Bayesian ranking they do kind of a survey I'm different rating methods methods and have some good proposals I wouldn't say read that paper and immediately implement it into your game I would say make sure you or your engineers understand how it works because there are subtleties in there constants to tune that definitely affect that curve I was showing you on how precise your method and how calibrated it is but it's a great place to start reading so now let's go on from skill to matchmaking and like I said matchmaking a lot of it actually really is just really good engineering there's a lot of really hard networking and plumbing problems that need to be solved depending on the size of your audience and it could be big or small audiences both bring their own TV problems and a

lot of that just putting these players together getting them to play and so like I sometimes offend our engineers when my kids came doesn't have matchmaking when they just spent like two months putting in like a really complicated system to handle millions of players or whatever and I'm just meaning skill or something like that so design is still important though in matchmaking because designers help decide how you prioritize putting these matches together is skill the important thing for your game or is it veteran see like players don't like to play against player players that have been playing for years is it there's other social elements you'd rather match back on so that players create relationships and play I won't play your game longer because of those or is latency the main thing that your game is so twitchy or so reliant on having low latency that that trumps everything else in your games it's not not a good experience regardless of skill other things of designing should be thinking about doing is what do you communicate to your players while they're waiting for their match and this is it actually turns out to be a pretty important thing because if you show them absolutely nothing and

they're waiting around they're more likely to think something's broken and quit out and actually hurts your servers and hurts the matchmaking overall usually when you do that that's in more of an engineer anything engineer's could tell you better what the problem of that is but restarting searchers have won over again and so but if you can show them hey it's going to take about this much longer this is how long you're waiting or hey we're searching for players within the skill gap or we're searching for players we found this many we're sampling through them all kinds of interesting things players really like that it just shows them you want to show them something all the time so they feel like something's going on and the other thing designers can help decide is how long should players wait for a given drop in match quality I think that's kind of an important thing for me actually when designing match making is that say skills the important thing if you're gonna give a player a bad skilled based match then they should feel like they waited and they should get used to it and they waited a long sort of a long time and what that that is for your game is different for across every games for shooter it might be like 30 to 40 seconds for like a MOBA it might be 6 minutes or something like that or a few minutes so but once you've players

whatever it is for your game once they feel that while I've waited then it's probably ok to give him a worse match because I know why so ideal match making the designer ideal is probably to put players in the games that are fun and maybe has some kind of like planned experience with varying intensity it's kind of a classic design philosophy that don't just have all super-tight matches and all really already whatever had some hard some easy and had to go up and down then you got this like business idea right like the holy grail would hey can I put a long-term monetary value on every possible match I'm going to make per player and then matchmaking the matches that maximize that value over time and that I don't think anyone knows how to do that yet but that'd be amazing and also to keep the most amount of players in art in your game having fun some realistic proxies we used or things like skill gap and sharing analysis and stuff like that the idea of being like we don't necessarily know what fun is or what it's going to be for this match but we kind of know what it isn't another important thing is the thing that one of the things that really hurts matchmaking

in general is if you have too many buckets you have too many game modes too many ways to segment your population so ideally you find ways to not have too many ways to segment and break up your population that helps improve the matchmaking quality and some suggested suggestions for ways to do this our final for example if you're the type of game has a bunch of game modes players choose from instead of having a list for every mode have like a main this they pick from which has some of your main modes in it that are the most popular or the way you define the way to play your game I then have maybe one or two of those and then have a couple of rotational modes lists that you change every day or every week so players kind of know that that one would they really love and they're really mad you took out it's going to come back and when it comes back they jump in and play but not only them the players are usually wouldn't play it they'll play it because hey it's it's available today I've never played that mode so they kind of have some win-win there and and then you want to get crazy you can go with things like let players just thumbs-up and thumbs-down whatever they like and let the matchmaker do the work and try to

optimize some difficult function over you know getting the player into sort of a match he kind of likes it just as on the side when you're working with your engineers on this I just give some advice that not to design a hard rule system where like it's only this type of game mode or this type of map all the time if saves a little bit of heartache if you let it be a more of a soft value so you can say hey there's a 50% chance capture-the-flag comes up next and then if designers really wanted to be no 100% capture the flag they just set that to 100 it's just one of these ideas that we're not having these hard rules and causing weird exceptions and stuff like that I get that from engineers so I don't that's what they tell me so it must be right another interesting point here is so matchmaking is evolved over the years that you're old-school matchmaking back when you used to play on your PC you fire up some game Mike Return to Castle Wolfenstein or whatever and you would do the matchmaking you go to the server list probably start by pinging pick the top one or maybe look

at some of the settings what they call it an or whatever and so that evolved to kind of more modern online console gaming where instead your game says a matchmaking service I need some games it sends like 50 of them and it automatically sort through that list and tries to find one and then there's the evolution to know I just say hey matchmaking service I need a match and it sends me back one match and that's what I'm talking might here with global optimizer a lot of people some games already uses it so some of you this is like no duh for some of you it's like we've never tried this before so I've been on a few games where we've shifted from the old way to the new way and it actually you seem to make a bigger difference than I expected because it's easy to do a little bit of kind of envelope math and figure out ok it's going to be a subtle improvement but actually in practice it was nicer I will make the comment though that it's kind of hard to directly compare those it's because people don't usually built both of them for again there's not enough time to have both the old way and the new way online and one game at the same time to compare because they're both expensive to build so that's my like I think it's better but I really don't have the whole ton of data to show it so I mention this a little bit before

but if you're doing matchmakers or complicated system so it's pretty good to QA them to test them make sure they're working right and one way to do this is again I have a model that I already know is pretty good at predicting I already know what my player distribution looks like I can predict outcomes so make a bunch of fake matches again and match make so put these players in with their skills match make them and then run the model on them and see if if your matchmaking system is working correctly and you can use that to test matchmaking which is more of a debugging thing but you can also use it to test ranking which means both testing the make and model you picked or comparing ranking systems and we'll talk about them in that in a minute but that's it's a good way to you do a lot of things this way you can do this both design and testing and so this leads kind of into measuring so you've got that a few different parts of measuring when you're doing matchmaking first are you predicting right might discuss that already make sure those probabilities makes sense because otherwise who cares if we're using these values and our

matchmaker if they're wrong so second okay I know these ratings are right but are we actually using them the right way and that was that most of this is mostly for debugging so this is just an example of one we did a very simple way to like say hey is my matchmaker doing what I told it to do so let's say your matrimony cannot scale in this case so when I'm graphing here's a histogram of the skill gap that of every match I'm seeing in my system and how many games I've seen at that skill gap so if your matchmaking correctly and tightly this should be shifted kind of like this is up close to zero the closer to zero the better in theory if you think that supertype match but you kiss them is important a June arbitrary line here and I called this the point of which matchmaking beyond this is a bad match and in this case is about three percent that are bad and this is the contrast with something like this where in this case 16 percent at 16 percent of the matches are bad it looks still little shifted over to the left but that's because if players are normally distributed most of them are near them so just completely random matchmaking

actually isn't terrible but that's 16 percent of the players if you you have a type of game where skill match make use an important they're gonna be pretty loud and pretty mad about it so it's not great you still don't want to necessarily get away with this if you feel it's important for your game so there's an interesting thing I've noticed like working with a little bit alongside sandbox teams sometimes and by sandbox like the guys that balance and design weapons and abilities and things like that I've seen some times where they will create a new suite ability or some new weapon they'll go play test it and then like a few of the awesome players that are in the place it's just dominate everybody else and because they're getting dominated the design of stuff that can say no we can't have this ability because like good players you're just going to destroy all the all the weak players but instead you could actually keep those suite abilities in there if your matchmaking correctly because then the players that master those abilities aren't going to get match-made against players that don't know what to do against them so you can have the players I've had Nationals abilities having a great time using them against each other whereas those that haven't aren't going to get dominated

there's until they get good enough to be to play against those players so that way you don't need to remove stuff that's gonna add skill depth you can keep it in and embrace that instead for your game and yeah so that's that the game's gonna depend on you personally to have the game that you're making a bit this is just an aside that it enables this I'm gonna move on a little bit now to talking about parties cuz this comes into play in a lot of games sorry so usually we can't let you play with friends this is like a common idea because hey if I let you play with your friends you're just going to dominate the people that aren't playing with their friends so got a party of four you're just gonna stop over all the guys that aren't in parties before so then we say well okay the solution to this is fine instead of we just won't let you when you play with your friends play against people that are not playing with your friends but then you get something like this you end up with much longer wait times because there are less people playing in parties and if you're just getting together randomly with a few friends that happen to be online you're going to get destroyed by these players that are usually playing in the as

parties and they have a lot of experience doing it and then so that's not a great experience either so a common solution not a common one of those engines that I worked with in the past on this is instead of tracking skill on each individual player when you're in a party I put a skill in the party so now I know exactly how that skill is going to line up with a group of people who are not in a party and we've done this in games and actually works out really well it ends you end up with really even matchmaking between the two teams it doesn't solve the perception issue so if people in your game really don't like the feeling of having to go up against a party it's not going to fix that for you but in a lot of games it will make the matchmaking even okay so and that's important because it opens your matchmaking pool it means that hey when I really when a really poor team comes in but it's a party of friends I can match make them against a decent group of people who are not in a party and that way I have an even match rather than having to try to make these this party wait longer this party your friends and playing with friends is almost always more fun than not if I don't want to make them wait

these guys that are having a good time together wait longer for a worse match or things like that so you can now you can play with your friends but now you can't play without your friends this is another problem you can run into without that same solution is play with your friends for a night your skill kind of goes up because you've actually played to some people do it play it better when they play with their friends and then when they go to play the next night first it's not as fun because you're not with your friends anymore that's usually a much less fun and then when they play alone their skills been inflated and we're giving them harder mattress but they don't know this all they know is now they jump into matches that are harder and they're having this fun and they say I'll come back when my friends are online and then they kind of lose people that way so again the solution is to track the skill of your friends together so whenever a party jumps in from matchmaking you create a new skill just for the party maybe use somebody smart and kind of bootstrap that skill based on some history you have of those players and other parties and when they play alone and you try to adapt quickly using your skill system and it fixes all of the balance it usually fixes all the

balance problems it doesn't again doesn't fix that perception all right so I'm not gonna go I want to talk too about skill matchmaking now I'll segue into ranking systems so again we know how good these players are what should we tell them and this is going to depend on the game like it depending on the type of game you have how competitive is or what systems it does or doesn't already have and I'm going to talk about three different types of ranking systems for now progression systems hybrid systems and then straight-up skill systems I'll make it aside that you may have a game that you love the ranking system in and you just want to copy and paste it into your game and I would say don't just do that taking inspiration from that other game but I think hard about how it should be adapted to your own game so let's start with progression systems so you've all seen probably a lot of progression systems I'm not an expert in progression systems a lot of possum designers out there out there that are great at making leveling systems and unlock systems usually based purely on time investment better players go through a little faster but mostly time in assistance and

these are often for like recreational play definitely not a measure of skills veteran see I will say that there's something interesting about these depending on how you design a loading screen so loading screen is a screen that comes up while you're waiting for your match to start and you see all the other players if your game only adds a progression system and it's not matchmaking on that progression system some games you see this in that you'll have this very disparate ranks from this progression system in it and players will look at those and wonder why am I being match made I'm like a Rank 2 and there's all these very 35s or reg seventies in this match and how is that an even match and so that's you gotta be careful about what you put on your loading screen and I mean my philosophy is a kind of a purist if you guys would haven't gotten that view of me already is don't put anything on there that doesn't actually go into the matchmaking at least don't call it out clearly some games have such messy loading screens that it's probably okay but but some make it a little too obvious that you're not matchmaking on the thing that you're

showing me so this is as an aside and yeah so you guys have seen these systems and RPGs and a lot of popular shooters but again I'm not an expert on that's about it that's just one example of a ranking system another is hybrid system this is very like I want to get the feel at least partly the feel of that progression system in my game where I'm kind of leveling up for a while but I want to have a skill component and so these these systems get pretty complicated because they're trying to kind of take too two different concepts and put them together they're usually pretty tunable there's a range between ones that are kinda skill-based and ones they're more skill-based and more skill incurred usually losses can move you down and by skill anchored I mean there are some of these systems out here that you can you know start up wanting to move up to 50 or whatever it is that will say okay but we've only get that you get any past rank for unless you behind-the-scenes really are some skill and so in my opinion and some games don't do that they just say don't go as far as you want as long as you can win that just whatever so in my opinion again this

isn't this is the Gospel according to Josh is that these are good for games that have no other progression system in them so you know what we're gonna combine both or it kind of a cost analysis type thing they're good for games where the competitiveness of a game they maybe isn't known you kind of think is gonna be competitive you want something in there though to give them some ranks so you'd go with that and and also maybe for games that everything else in the game is an RPG so this should probably also be an RPG it should also feel like I'm leveling up that kind of makes sense I think it's crush number game for a game that already has an amazing progression system on one side and then you put this other progression system on the other because to me those systems become a little too similar and again as a purist I'm like if I'm going to design a completely new system I like it to feel unique and different than the already existing system otherwise I feel like it's just can become another grind and people players will kind of forget about it some other issues that you'll run into in with hybrid systems is ranks end up being kind of meaningless except when you get near the top because everyone's starting at the bottom and moving up and they're not really separating much until they get near the top and with this type of system you end

up with both pre-game kind of loading screen issues and/or matchmaking issues and I say and/or because sometimes if they pick one or the other so if you know a player's skill for sure in the backend it might be tempted to just match make on that skill despite what your hybrid ranking system is currently showing it's showing him as a rank one but I know he's actually more I could write 20 so I'm actually came against rank 20s cuz that's a fair match so that gives you a good better matchmaking but now you get that loading screen issue where I'm actually making I want against the 20 and that's kind of weird or on the flip side you can say no I'm going to only match make 20 against 20 is ones against ones whatever but then in your early ranks you end up with matchmaking issues because now in the early ranks you have all kinds of and if your matchmaking all kinds of players together it's just you're gonna have machine your matchmaking and also it's just the idea that if you start off in a progression system and at some point you hit a wall that can be in some opinions a little bit confusing for the players so these are just some of the the issues with with hybrid systems so

right some hybrid systems like I said will still have an actual skill on the back end and this is kind of this slides kind of about that loading screen issue where hey I'm showing why am i a 1500 to gets a 2200 that doesn't make sense but on the back end we're both a 3.0 and skill so it does and so the 1500 is wondering about why it made this match it's kind of an artifact and I in in past games that worked on this problem has been kind of solved by just showing the average back end skill rating of both teams and so they see it isn't even match even though the rank isn't even and in some games it works fine it depends on how number Z your game is and how much players like engaging with that kind of thing but it's a little awkward alright so now I moved from hybrids on to just pure skill systems so this system ranks are tied to skill if you want to get a rank you need to actually have that skill there's no progression moving up and down we focus more on your current ability and your placement rather than progression and this is good

for clearly competitive games eSports focus games are a part of your game that's more competitive in ranked I think they're good for games that already have a really a rich progression system and can use something unique and competitive for those for the players that want to engage with that and the other thing I think that I like to do in a skill system is make them transparent because the audience engages with this kind of system really likes transparency and the part of your audience that doesn't doesn't care anyways so make it transparent and don't worry about the parts of the audience that don't and I'll say yes ranking systems are always more stressful that doesn't go away but I don't know if I believe these days that being kind of Kara bearish with your ranking system is going to save you anything your game already as competitive and ranked at that point so it'd be nice if we could actually look at what this looks like item as in player experience these kind of different systems without having to actually ship a game with it first and we can because like I said okay I can create fake players now I can match make them I can choose the outcomes of those matches correctly

because I know you know 75% chance teammate is going to win well flip with unfair coin and if it lands on 75 or whatever then the better team wins then I can update these divisible ranking system because the team just wanted to match and some people just lost I can gather those results and then I can graph it over time so I can see where does your rank go over time by simulating it mmm so so let's say we decide to use a raw skill system like just some pretty raw numbers here you probably pretty this up as a designer but this is what a simulation looks like in general if you had a raw skill system so this is showing different different kind of archetype players that I pulled out where I knew their skill because I created them fake as fake players and then I ran them through the system and this each of these lines did not represent a single player it represents the meeting experience across like hundreds of players that were at that skill overtime and so you'll notice I'm in like a pure system like this we immediately quickly tell the player how good they are separate right away and we know right where the good average and

bad players are and we can match make really effectively because there's no overlap here and so that's why I say if there's 98% good matchmaking that take some arbitrary line for that 98% but that's that's pretty nice now let's contrast that with and in kind of inflationary hybrid system this is system when everybody kind of goes up over time some there is a skill component and what you really want is the better players to move up faster so you if you if you make a system like this you get this kind of fun experience where like a I'm progressing over time I'm raking up this is kind of fun but you'll notice that the the matchmaking quality can went down for like 90 percentage to like 69 percent and this is because the effect I adore to before so if you look at rank 10 look at the type of players that are sitting at right 10 you've got good players that have played maybe 25 games you've got average players that are sitting around 50 and then you've got bad players that are pushing 200 games but they're all at break 10 so if you match make on rank which hey if you're going to make a complicated hybrid system you might as well make it because usually you do this for some more kind of simplicity and and

making it more clear to the players what you're doing you're probably going to want to matchmake on that system so you match make on rank and then now you've got players playing against good players and of course the matchmaking quality goes down so then and also you can see that also makes the the ranks at earlier levels less meaningful because you've got everybody at those earlier levels so you might look at this and say hey there's a bit of information you're though to tighten up this matchmaking because about the very least I could match make a number of games played within a rank and then I get really good matchmaking or if maybe I keep a raw skill on the backend I know how good these players are so let's match make on that raw skill instead so I said sure let's simulate that let's see what happens before you just match make really well and the truth is yes you can get great matchmaking but if you'll notice what's happened now is all three types of players our winning just as often and because of that good bad and average players are all moving through your system at the same speed and that you can't discern that part anymore and this is kind of bad because now you've made this really complicated hybrid

ranking system but your matchmaking is completely undone it this is also what happens anecdotally if your designers go off and make an amazing ranking system and engineers go off in and make an amazing matchmaking system and they never talk to each other because your engineers are going to make sure that get type matches your designer is going to be a nice progression system and then this kind of thing can happen and so to say you kind of remove the point of your you don't have a scope a system anymore if you think this is right for your game don't make a really hairy hybrid system make another leveling system use your XP guy to help you out and progression guy and just tell the players it's a ranking system maybe pattern that might be like tricking them too much they might catch on that it's not really skill based but this is definitely a skill based anymore okay so I've talked a bit about what a progression system is I'm not an expert there although some of you guys probably are so I'm not gonna go much more into that a hybrid system is a mix of those so instead of going more in-depth into those two I'm going to talk about what I think are good attributes of a skill system and I'll leave hybridizing as an

exercise for you based on like what you guys may be no better than me about what a good progression system is and what I think for me personally is a better skill system so again my personal philosophy you guys were most really quick I'm kind of a pierced for better for worse likes when I work with game teams I talk to them a lot about what their audience is well not this is right for them but my if you're gonna go skill go all the way because like the players who like skill based systems they're gonna love it and the ones that don't they're not going to get you that anyways they're gonna ignore it so like you don't want to cater to them as much so go all the way it's nice if your skill system has a link to the pro players if I can look at the skill system and see he's least the pro or kind of top amateur players near the top are the same as the pros that are playing in the real tournaments and things like that live ones then I'm gonna be I'm gonna know as a player what I need to do to get up there hot a good feeling for that it's yes sure it's kind of fictional most people are not of course not gonna get there but at least I can you know kids can tell their mom I can win the million dollars if I buy this game and played enough it's also not about it's about your career so I like steel

systems that are players focused more on their current skill position not their progression so I like things like placement matches so like instead of you starting at one and moving your way up I say you know what play ten matches play five matches and afterwards I'll tell you what's your skill is and you've seen that in the in probably few games I've worked on or games that have like that concept and the interesting thing about this as an aside if you look when we went back in these games that played looked at our data players tend to play the placement matches players that you wouldn't expect to be as engaged in PvP in your game will also play the placement matches because like opening a prison I find out what my skill is and they will tell you that the game they played after they finish their placement matches was amazing this way more tight than the one before even though usually if you did just go system I was probably no different that's a perception thing so like some of these issues are definitely perception driven okay so moving on from placement and position I like games that have broad meaningful ranks for relatability so instead of lots of awesome numbers I like when you have me

so you guys have seen my systems I like things like bronze silver gold platinum things like that and I like that because players can remember them better and also because I'll show you in a second what the skill spread usually looks like and lets me spread out the skill of the players such that the difference in skill between one chunk and another chunk is basically a three-to-one odds if those guys play against each other so a silver player will be two bronze player three out of four times and this is cool I think because those players start to identify and feel what that needs like with that what that feels like a silver player knows just like to play a bronze player and he knows how hard it is to play a gold player because you've designed the system with that in mind because again going back to my skill slides you know how to predict the outcome of these matches based on your skill system so I like those kind of broad chunky gaps but on the schizophrenic side or whatever I like high personal granularity for each player to see movement so when you compare yourself between players I like the chunky ones but I also like to give players enough of a granularity so they

can see themself and moving and that's going to depend on your game some games your players want to see movement every single game others of the every few games or whatever it is so this is where you add things in like tiers within these chunky groups numbers some small sub goals they don't really they're kind of meaningless to compare players within those chunky groups but it's nice for the players to see something okay another thing I like school system is an endless ceiling for the pros so at some point you break out into raw numbers and they can go up forever they don't go up forever because in order to go up forever you'd have to have someone who is infinitely good player you definitely get players that are better but they're never infinitely better and what I mean by that is the very best player in your system is going to beat the next best player some amount of time 3 2 1 4 2 1 whatever it is that defines the skill gap and that player won't be able to push much past that skill gap if your systems calibrated right and again I told you I like transparency so anyway this is kind of what how I designed these people distributions to look are it is is one way I like to do it if your audience is a bell-shaped curve you can

slice it up in this case I sliced it by what's called a standard deviation so if anyone knows about normal distributions that often talked about in terms of the number standard deviations moving that from the middle so the middle here is 0 and each line is 1 plus or minus one standard deviation away from the middle and the nice thing about that is you can usually tune your system such that one standard deviation gives you that 3 on 1 odds that you want so this is something you'll commonly see in games of network done and again it gives you this broad chunky ranks that players relate to and they know what they mean when they play against other players and if you kind of get all these things right and you're really lucky you end up with a system that feels really good because it has integrity players trust the rig's in your system they see where the pros are they see where they are and you get this kind of thing here where like because I'm as I play this game I'm getting genuine feedback about my improvement so the stickiness now is competitive stickiness which is I played me I go off I learned something new about strategies in this game I apply it and I and I actually see my rank go up

because of it and that makes me excited and usually you can go on YouTube and Google and search like type in your favorite game promotion reaction and you'll see movies like this that are exciting and meaning that you kind of did the right thing so that actually is the end of my presentation so we probably have lots of time for questions they do a quick question about comparing fine phase matchmaking versus service base you've mentioned one other word the server sends you a list of games and you kind of pick one the best ones and the other model is a server you just ask for a game and just gives you a better yeah better one can you compare the two you see you can do you know directly tourism because usually titles use one or the other but in your experiences is it war

yeah so in my experience the server based approach ends up being better because it's able to see a lot more matches and again it gets back to your bucketing situation we're giving a player only giving a player only 50 matches to go through means you're kind of starting your population already whereas the kind of global server that looks at all possible matches and makes its decisions has a lot more information to choose a better match for you and that's why my experience has ended up in actually better matches and I would have expected so yeah I like that approach it has its pros and cons depending on what is the important thing in your game to matchmake on so if latency is important for you to metric on ping between two clients you need some pretty fancy ways for the server to be able to quickly estimate how what the ping is going to be between all these players you have but if it's really skill oriented and not so much the other ones then you can you can some really good matchmaking there's a lot of

turn if there's a lot of turn so the server based one you know the players come give eyes yeah so this is that's where the whole it's a lot of matchmaking it's a hard engineering problem I better move to Tony great presentation congratulations so since you've been doing this for so long and you've been solving these problems and probably talking to people who have maybe different goals I'm curious as to if you've ever been privy to any goals or motivation to or I or particular models for not having a 50/50 win-loss goal but there because we kind of all just accept that and I'm just curious if having been in it so long yes I've definitely seen both like really good suggestions and how not to do it how to not always be 50/50 and some very definitely biased

reasons so I've heard the like but stomping nudes is fun approach to that so let's not give 5050 matchmaking all the time which depending on your game I like my singing it may seem really weird but there are some games where that is the right thing well it seems to be and I've also heard though the so the approach where it's still kind of 50/50 but you don't always give a 50/50 match you give like a variety of matches so you give sometimes you have some smart way of saying no for this match I'm going to give him actually a hard match for this next match I'm going to give him an even match for this next match and to give him an easy match and that was motivated somewhat by my past experience where like I got I used to run wolfenstein enemy territory servers back in the day in college or whatever and we got to the point where the matches were super even all the time and actually after a while it gets actually kind of tiring and so I kind of came up the opinion that it's okay to allow a little bit of a balance here in there and have a mix of even hard and easy matches so I don't know if I answered your question but we asked you talk yeah yeah hi I have a question

about using like so the matchmaking system you talked to Arthur I guess it's the rate that the skill system you're talking about but specifically just predicated on whether you win or lose kind of not quite you know that's something similar to that so my question is have you thought much about like using a more sabermetric style approach to to ranking the skill of players because like we have so much analytical data that we get back from the game and I wonder in particular for like because the win-loss approach works really well I think for one view one sort of games but we need like a MOBA where there's sort of different roles and different positions like I guess so just what are your thoughts on sort of going to pick this look like an example is you could have a baseball player really bad team but you can still like sorter analytically say okay this is a really good baseball player yeah you know their team's terrible yeah and there yeah there definitely are models that we use that account for that too like a lot of

that stuff like can't talk about as much because some of it works really well but yes you can take stats out of the game and if you're a data scientist or working with data scientists you can come up with ways of either pushing that win-loss system along faster or or measuring players in a different way yeah what is the size the population that we need to yeah it's gonna actually depend somewhat on the game which is why I talked about simulations so you can using this type system and you can feed fake players into your matchmaking system of varying sizes and see if your skill system stays stable it one old rule of thumb I used to use is you want about like at a given time when you go over to matchmake it's and then this isn't actually just

for the skill system this is more for having pretty good matchmaking and again this is a rule of thumb I've used in the past that's may not even be true but I did notice that when we had at least 40 players 40 people per the number of players you needed in the match looking for a match at any given time you could end up with pretty good matchmaking so if this one would be one that would be 80 because there's two players per match if it's four before then and that would be like 320 or whatever it is so yeah thanks yeah okay so a quick question about decay what are your thoughts about that have you any data so what I mean is you presented stuff that looks like better city systems that through scope that kind of stuff there's also this approach of skill decay with like you for instance and it's helpful and so you're asking about like what do you apply those systems and your experiences shoot one that applied for like the idea is that the longer you don't play that the

more skill you lose right any kind of data to T of the sense because there's a meta game in for instance MOBAs so the assumption is if the guy doesn't play for a month down right yeah so I have had actually worked with the cave-in in the past first he's asked my like systems were for Claire's not around for a while you actually decay is rating or it makes them estimate that his skill has changed since he's been gone it's pretty tricky to move a rating around and most of these systems without really knowing what you're doing because the algorithms are fairly complex and moving a rating manually requires some pretty good understanding of how the algorithm works so to go in a little extra math you detail most of these more state-of-the-art algorithms keep track of not only a player's rating but a what's called a variance or an uncertainty about a player's rating now uncertainty is like okay no he's like somewhere between 1,500 and 2,000 I'm not sure exactly what and over time that goes down so if you move a rating you also need to move that variance and in

the past I have moved the rating without moving the variance and that was bad I've decayed ratings I've made some mistakes decay in ratings I think in hindsight the decays often worked out pretty well it's just needed more polish if I were to go back and do that again I would be more confident given the math I know what I understand a little better now moving the rating and then correctly resetting the various to the proper value or not even moving the rating but increasing the variance which means player comes back loses his first match but he's okay after that so yeah I follow up on that so just interested in the math behind the Bayesian one it's like it seems like the very inception variance is there for is to competence account for that says why did why would you need you to cater eight like the math or ities supposed so if you've done some data analysis that tells you something to read it tells you something exact and you know that when a player's been gone two weeks I observed when he comes back he always drops by this much rating if you know that then sure go ahead and move the rating because that's the correct thing to do you just variance part of the algorithm yeah and you could do both you

could drop his written down and increase the variance but increasing variance is tricky I'm not going to how you do it right now but don't just do it blindly because there's a super quick question when you matchmake on the scope on the skill thing you do it in the chunks right you don't do it on the granular I think you're gonna match somebody in silver you just naturally gets anybody in the silver and you do 2,200 versus 23 so that mean that's up to the designer but I you said you liked the chunky stuff so here's what you do I like chunky for ranking okay but I think when I've done skill-based matchmake I've usually been tightly that's not saying it's the only way to do it it could be more exciting for your game to leave a little bit more variance in there so the real question was there's a bigger question was the hybrid the popularity of a hybrid system so all Andes the games that have been like you know okay we're gonna go through this legwork you know progression based system and then transfer you over when we're sure you're ready so we don't wait a lot in a drop or bounce because they're seeing this number go down and stuff can you talk a little bit more about that because you were saying oh just go skill which is obvious it up to the thing if you're making a use for T kind of game but like you're just worried about yeah so no it says I president talk about law

cuz it's not a game I work on or anything like that but I'm perfectly happy with games that have amazing progression systems on the side and then also have a skill system over here and the game designer should know for their game so if you're sitting in law it takes a while before they let you play in the ranks right that means that the designers have made some smart decision that in our type of game it's good to get them through some not as stressful stuff before I move them over here and that I think is totally cool and awesome yeah thanks yeah hi I was curious especially if you have like critical mass issues in games if you how you sort of weigh the trade-offs between like giving people matches that may not be that great for them to get them in quickly versus you know I'm having them wait a little longer but but get a better match later and how how you balance those factors so you thinking my my only rule of thumb I've used over the years for that is that the wait time should be like players should wait until they feel it whatever that is for your game you would understand that better than me soon as they feel it it's kind

of okay to give them a worse match so with smaller population games players are used to waiting long it may be like a bit longer but I'm at a time and he designed that by usually by having something in your matchmaking system that has a tolerance for a bad match that grows over time that's kind of proportional to like the skill gap that you have so this the allowable skill gap starts at some point maybe small maybe big so if you gave us a little population and I have to start bigger I think grows over something at a time and then you have to have other interesting just edged case things like okay now this guy's waited forever and what is forever and we need to matchmake him immediately so that that's usually the type of thing to work on a lot of that has to be decided per game though it's kind of tricky yeah thanks yeah so for some games that have different roles players might have different skill levels depending on what they end up doing I was wondering if you had any thoughts on how you can match make and like determine their skill based on what role they end up playing so as this is are you so you're asking about so the

matchmaker and are you talking about situations where you clearly choose the role or where the role needs to be inferred somewhat bull-like one thought would be like well like if you're a really good carry but then end up being support because you don't how things work with your team but also just like if there's like hey I'm get it capture the plague and like how much does a loss on yeah so there's a lot of ways to handle that that the very simplest way is you put a separate skill on each one the problem with that is it's gonna take longer to learn each skill you're gonna need more games on each of those roles or in each of those maps or in each of those modes to learn that other more complicated ways or you have something like a base skill and then you have a another skill for then another skill for each of those modes which sounds the same but having that base skill there makes it a lot faster to just learn the offset from that extra skill other ways to do it that may be simpler than that is I have a skill for each of the modes but when I go to play a new mode or a new role I kind of look

at all my other ones sort of take an average raise that variants up and then bootstrap it in so that means I have at least to the same starting position those are the types of things I've done over time that seem more or less effective depending on on the game yeah hi I'm interested in the dynamics of skill over time so so you start off with a population with a given skill distribution but you had this graph where you're like a highly skilled player whose skill was sort of constant but the bad player sort of is creeping up over time so I'm wondering if you see the opposite effect where so if everyone's normally distributed then you know if you're a more skilled person you're more likely to face someone who's less skilled than you and vice versa so do you see over time that the variance of skill sort of narrows or do you feel like the variance of skill sort of stays constant over the lifetime of a game it's if you're talking about like with the whole skill distribution looks like it depends on the game and it depends on

often the number of players that you have on the teams actually in my experience in a lot of situations that variance actually goes up over time because the model is wrong so that 30 secretive statistics is all models are wrong so there are going to be extra sources of variance that you didn't account for that are going to creep in and often cause your variance to increase and sometimes cause you mean to move around so most people that have dealt with these systems realize that they actually have to have some mechanism to shift and scale correctly over time I have also noticed though and games where you have like a whole team and you're using the team average whatever or putting a skill on a team or if you're taking the T a larger team taking team averages those situations tend to actually have decreases in variance so it that still depends on the game so the long short answer is you need to analyze your own game and see what happens and then decide how to deal with it yeah hey so I'd like to ask your opinion on how you think rate resets affect models like this in a hybrid system for example so like in League of

Legends you reset a rank once a year right all the good players are still kidding like a better rank to start at for their you know their journey and then the bad players are given or like the worst players are just given a lower range so they still have a discrepancy there's a difference what for instance like in hearthstone they reset it every month so it causes a lot of if you're a player who's you know legend and you'll get ranked at like I don't know 15 stars or something and a bad player never it's like ranked and an idiot they get reset it like ring 20 but you end up had with a lot of players good players that just don't play for a few days bad players who do and then they get caught up in the middle and it ends up with a lot of discomfort for bad players or games yeah and so again that gets that gets to like knowing your game and those are problems that those are the type of problems we solve basically as we look at those effects and we try to make decisions and and if that's going to do those type of things are going to depend on the ranking system you designed and how it works and type of players you have and how often those players play

how many of them play it's it's it's one of those problems you have to solve like my opinion on that I mean you guys have 40 fundamental some kind of appearance I'm not gonna go too much in the more into detail on that but yeah those are those are tricky problems to solve I think they can be solved but yeah with the hybrid system those are the types of running into in general do you think that you would have just a like a larger difference in reset like so if I say if I was a legend player in hearthstone I got reset terrain 15 maybe if I go lower and I set them to like rank 12 or something where they're unlikely to meet up with newer players or just bad players who like to never even get to rank 12 in the first place you think there's like different ways like that yeah so to make a decision like that thought you'd wonder closely analyzed and just so you know exactly what's gonna happen in simulate and try to make your best guess that hey if we make this change the will that have that positive effect we wanted or does it actually in practice make things worse but yeah I I

don't have a strong opinion right now not because I'm not on the hearthstone team right I can't be fair I'm not going to be unfair to them I would talk gladly to hearthstone we're in the same company but again if that's your job at your own company then then right that's those are the type of decisions this type of talk is going to help you make but I don't have a one-size-fits-all I guess is the answer yeah I how would you deal with players that might intentionally lose so kind of like you know they call it smurfing where you try to play against worse players there's a couple ways to deal with them my favorite way so let you guys in on a secret back in the day I actually worked at a company called eBay do it fraud detection catching people that were selling bad stuff or breaking into your account or whatever my favorite way is to get your data science team to build a sweet smurf of detection algorithm and just banned those guys because you could

try to and then you can get a lot of times you can do it depending on the game but there's some games that works as simple as hey the median length of the median game length for these players is only a minute for the median game like the most legitimate games is 15 minutes so other ways to try to deal with it if you can't do that are things like making so this making sure this the variance on the skill never gets too low which makes it so skilled moves around quite a bit which is kind of annoying if you're trying to really hone in on skill on what a player skill is but it makes itself a player does Smurf when they start playing again they get out of there faster it's it's tricky to try to solve it systematically it's better if you can just detect them thank you cool and think I'm at a time if anyone has more questions all go hang out a little bit at the West Hall overlook at 202 to whatever and if no one shows up I'll take off but and yeah thanks a lot for coming guys

you