Matchmaking for Engagement: Lessons from Halo 5

[Music] hi I'm Josh manky thanks for coming and welcome to my talk on matchmaking for engagements hopefully you'll enjoy this and they'll be engaging for you so let's get into the talk so first let's define what matchmaking is we're talking about the kind of matchmaking where players in multiplayer games want to get in matches to compete with each other so you have a group of players they want to play they submit some information to the matchmaker about themselves that the matchmaker can use to create a good match the matchmaker takes this information about many people and then puts them into matches and creates teams for them to play okay now so what is engagement well the dictionary tells us that engagement means to be greatly interested and so we want our players to be greatly interested in our games and this is accomplished through game play matchmaking is there to guarantee that the gameplay is experienced as intended

by the players so a lot of the time we're just preventing disengagement by making sure that players can play the game play and have a good time with it you

so now on a good question to ask is how is matchmaking then related to engagement so we'll go back to our players they want to get into a match here they are they jumped through matchmaking matchmaking puts them into a game they enjoys your engaging gameplay and then afterwards they jump back into matchmaking because they are engaged and they want to keep playing now our matchmaking comes into this though is what happens if they go into a game but the latency is so high on that server they're playing on that they can't really play the game right that could disengage them or maybe they're just not good enough to enjoy the gameplay they're so bad compared to the other players in the match that they aren't even able to play very much or what happens if they just wait so long that they never actually get to the

gameplay so these are some things that we tend to try to prevent and avoid with matchmaking we want to cause those to not happen but a good question is how important are each of these characteristics latency fairness wait times which ones are more important that's what we're gonna get to next and what I'm gonna want to talk about a bit but first of all a good question we can ask is why do we care what's important can't we just throw it all in to the matchmaker let the matchmaker just sort it out and the answer is not really and here's why the creative situation if we do that called having too many buckets so as a player we want to matchmake on skill say well that's already reducing the number of players that I can match make with let's say my skill bucket is only 15% of the whole population okay but now and then we're like well we also need to be careful with latency so that's another only 10% of players that I can match with because of my latency and then we want to do some fun fancy

cute reputation matchmaking I don't say there's only 5% of the players that my reputation makes me able to match with once we've broken out all these different buckets of players we've reduced that player from being able to match with the whole population to just point oh eight percent of the population suddenly a large and even large population becomes very small if you do that that's why you need to be careful about just letting the matchmaker do it at once because we used halo 5 for the analysis in this presentation I'll quickly review what halo 5 is about so halo 5 is a first-person shooter it's the matches and data we used came from competitive matches it's most of them are team-based it's usually four versus four but it can go up to 12 versus 12 well a lot of our matches are team deathmatch although there are some objective matches as well now because skill and fairness are often some of the more

important characteristics we tend to matchmake on I'll do a quick skill system review this talks not going to go in-depth on skill systems because that would take forever but is worth reviewing them to understand how they work into matchmaking so we have a player but we want to know how good that player is so we put a number on them that number should represent their skill now we use the skill to compare players to other players so given to players we can then figure out the difference in their skill from each other now a good skill system if it's working right the player with the higher skill should win most of the time we also want to take this same concept to the team level and one way the simple way to do that for this talk at least is just to take the average of each team take their average skilled average the numbers on both sides and then take the gap between them and I call this the team skill gap and it helps it tell us which team is going to win most of the time now we care

about that team skill gap because that tells us how fair the match is if the players have the perception that they can win the match or not that is probably important to them but there's another type of fairness or another type of skill gap we care about besides the team skill gap and that focuses on a single player and their experience in the match we want to know what's going to happen when this player comes around the corner and runs into another player on the other team are they going to have some success against that player are they going to feel overwhelmed and to represent this we reduce the team the other team's skill again to an average and we just compare the player on one team to the other team's average to get an average feel for what's gonna happen when the red player encounters a blue player and in this case again that should feel pretty good because the blue player has a skill that is higher than the other team's average and in this talk I'll call that the personal gap the personal skill gap so have two pretty important skill gaps we care about the team's skill gap and the personal skill

gap so now what we want to talk about is how important are these different characteristics we've talked about skill we've talked a little bit about latency wait time and you've probably also heard of people being worried about losing streaks these points here are some of the more commonly used items in matchmaking or at least items that developers tend to think are important so the question is are they really important and that's what this presentation is gonna try to find out so how do you find out if something is important the sort of industry standard these days is to use something called a B testing but the problem is matchmaking actually breaks a B testing I'll go it may be a little into that later but it has an issue with it and so we can't just use off-the-shelf a B testing to test test what's important but we can use something called an observational study in this case we could look in our already existing data and find situations that match what we would like to test we find matches where the ping was really bad and see how players reacted or we find really unfair matches

or really fair matches and see what the players did afterwards so we still have some very powerful tools to use and to be honest a lot of the most important studies that have been done in science had been more observational than more controlled experiments like a B testing so let's start with the personal skill gap again that's where I'm a player I run around the corner I see an enemy and well I have any success against that enemy when I have an encounter with them the idea being if personal skill gap is really important what can we observe in our data to see if it is important well one theory is if it's really important then players will play more matches when the personal skill gap is smaller as long as they have some success and they're able to play the game they'll play more so to further illustrate this we have a player who plays a lot of matches and he's playing a lot of matches because the gap the personal gap from match to match was small so he was happy he kept playing we contrast this with a player who has matches that have large personal gaps

this players more angry he's going to play less matches so we should be able to construct an observation pull some data out and see if we actually notice this in our data so we'll go into our data and we'll see what actually happens and see if players do play more matches when those gaps are lower so we grab a player grab all of our players but let's just for example show what happens with one player we get Bob Bob's played several matches so we know how many he's played here now we know Bob's played seven matches and now we want to see what was the gap in those matches right so we write down what the personal gap was for Bob in all those matches now if we do this for all of our players we can start to build a data set that data set kind of looks like this we want to know for every player what was the average personal gap they experienced across all their matches that month like ideally that would be low and then see how many matches they actually played and see if there's a connection between those so write down Bob we've already said that

he's played seven matches we average those gaps out and we see that the average across those values is 67 so bob has had a gap of 67 and played seven matches they then do this for other players like Alice Alice had a higher skill gap personal one in those matches that she played and she tended to play less matches maybe that's related so we'll keep doing that across everybody who played for the whole month and then we want to bucket those to make it a bit easier to graph so we can't just plot them on a graph it could but it could be kind of confusing we want to find a trend though so instead we take all the players who say had a personal average skill gap of 50 for that whole month and then we average how many matches to all the players with a skill gap between say 50 and a hundred how many matches did they play while they played a hundred so we'll write that down and then we'll do that for all of the other buckets for each bucket we'll keep note of how many matches of player played if they had an average personal skill gap so where again we're connecting what was my personal skill gap experience that the

number of matches I played and so we did that then we graphed it let's see what it looked like so if you look at this graph you're like wow you'll notice that as that average skill gap goes close to zero meaning the perfect sort of matchmaking of perfect fairness for the player the number of games of players playing is going is the highest they'll play more games if we keep that gap low so you might think well we're done we've we've shown that personal skill gap is definitely important the problem is were not done this is bogus and let me explain why so whenever you do an observational study a good sort of trick you should try first is to reverse whatever your hypothesis was whatever your theory was and make sure that that reversed hypothesis can't also be a thing so in this case our original hypothesis was if the average gap is small players will play more matches so if we reverse that we ask if

players play more matches will that make the average skill gap that they play smaller and the answer is yes it will because we are in Halo 5 we are skill matching so on average as you play more matches the average will tend towards 0 if you just get a one-off match here or there it may be extreme you may have someone offs there and you say we may find players who have these random matches out there only played one game but we don't know if they only played one game because the match was bad or if the match just happened to be an outlier and they only just happened to play one game and it's because they didn't play a lot of games didn't average out so we can't use this approach right now and and that brings up a general thing to be careful of you want to make sure that whatever your metric is it's the effect and not the cause so our metric is is an average matches played unfortunately play more matches causes better matchmaking so we can't use that approach unfortunately so what can we do

well let's go to a different approach we can maybe we can't average over a bunch of matches but we can probably just look at one match and see what happens so let's see if a bad match makes them quit out of the match match and because if something is important to a player's matchmaking experience where will suppose that they will actually leave the match of its back and again we call this quitting quitting out of a single match now I know there's more to engagement than the player just leaving one match but still players leaving matches is bad it's ruins the match for everyone else in the match not just a quitter and it probably leads to disengagement later so we'll start there because that takes care of our issue that we had with the correlating the two cause and effects so if personal gap really is important players will quit if it's too large that's what we want to now show they'll quit out of a match now how do we do that so here this is what we're talking about just to review we have our players

they go into the matchmakers they ask for match matchmaker puts them together however that dark blue player his skill is way worse than the red team and so he's not even able to play really every time he just tries to do anything he tries to go grab a weapon off the map and he just gets killed so he's not really actually able to enjoy the gameplay and he quits and we want to look in our data and see if this actually happens so let's let's again see how we would set that up so let's talk about what kind of data we're gonna gather to do this study first we need to know what match it was so we're looking at individual matches now not a player across a whole month so we need every match we need to know the player because we're looking at did a individual player quit out of a match and what caused them to quit well our theory is a high personal skill gap will cause them to quit so be looking at the personal gap for this particular match and then we will look at did they actually quit the match did they leave the match early so

we'll have data kind of like this we'll have a match ID some random number will have a player's name in this case we've got Alice and Alice played a match and the personal skill gap she experienced in that match was 100 and did Alice Alice leave them at Charlie no she didn't so Alice was ok with the gap of 100 in this particular match now we do that for Bob also but say Bob was also in that same match but Bob was much worse than Alice and therefore he did leave them naturally and we continued doing that for all of the players for all of the matches in the time paid were measuring and again we're gonna bucket it just like before we're gonna take all of the players who experienced a gap of up to 50 throughout all their matches for every single match and see whether or not they quit and we'll do that for each of those skill gut bet and grab those skill buckets for personal skill and then we will graph that and see if personal skill does leave lead to quitting and it turns out it does it turns out at least in Halo 5 for us the higher the personal gap the more likely

a player will quit so that is important to us and if you do the same study look in your data and you see a similar graph that tells you that you can reduce quitting by matchmaking on that personal skill gap by reducing that gap you know you can reduce quitting so in fact in this case it reduces the quit rate by over 20 percent which is a pretty big deal another fun thing you can do with this graph is you can pick out using this data what skill gaps are okay and which ones aren't this graph sort of suggests that at first players aren't as sensitive for a few hundred worth of that skill gap and has it increases though then they become more sensitive so you can both use the graph to tell you it is important and you can use the graph to tune your matchmaking so the takeaway here is that personal skill gap is important to matchmaking and if you tune that gap you can reduce quit rates to whatever you think is more important for your players so we've spoken about the personal gap we've seen the personal

skill gap is important but what about the team gap we also spoke about so this is less about a player being able to engage in the match and more about as a player on a team do I feel like my team even has a chance to win this match is that important to our players are they okay playing matches that guess they can run around and they can shoot people they get some kills they can do some objectives but they're not able they know pretty early on in the match that they're just not gonna win this match what will happen right so if the team skill gap is important then again we'd expect something like this players go into the matchmaker it puts them in teams and it turns out that the blue team is worse than the red team so much worse that they can feel pretty early on that they're not going to win and this causes them to disengage and they leave this is actually even worse than the single player quitting out because now we can have several players quitting out of a match and we want to do a similar observational study in our data to see if this happens as well to see if players are going to leave early because

they just can't win I might as well just leave so again let's get our data together we want players and we want to know for this match what was the team skill gap and then did they leave so it's very similar to our last one but instead of looking at the personal gap for every player in every match we're looking at the team gap so we got Alice again she played a match where there was a gap of 100 between her team and the other team and she did not leave where's Bob he played a match where the team gap between him his team and the other team was 250 his team had less of a chance of winning and he did leave and again we're going to bucket this so we can graph it conveniently well look at all cases all matches and lock with alt per player where there was a gap of up to 50 and see how often did players leave in that gap now because it's now we're still looking at this per player per match a match can have four players in it each of those players could have some chance

of leaving and in theory that would be a similar around the same average chance of leaving so as a quick hack it's not the exact right map but it's close we multiplied the chance of a single player leaving by the team size which gives us a rough approximation of the chance of any one player leaving the match and we graph that again to see if team fairness is important and in fact it is and in fact it ends up being even more important like the quit rate with just a hundred rating difference will double on our players in Halo 5 we can have a guaranteed quitter when that rate quit when that skill gap gets up in the 400 range so again this tells us here that as the team skill gap increases the chance of players quitting out is going up very quickly and it's very important to our players so the takeaway here is yes the team gap is important even more important in some ways than the personal gap it will lead to more quits and we can use that graph again to tune

which team graphs team gaps are appropriate for our players so go ahead and do that in your own games find out how important this is so that you can improve their matchmaking now some of you may have actually done similar observational studies already and you may say to me you know what we've already tried that doesn't work skill skill gap in our game at least we know it doesn't matter and my answer to that is maybe because there's something else that we did that we were curious about say we actually switched skill systems recently and our old skill system which is closer to what is the standard today in the industry we still run that in parallel with our new skill system so we could go into our data and run the same observation using our old skill system to see if when players have a high skill gap in our old system do they quit out of matches because if they don't that means that it's not that skilled doesn't matter in our game it's that the skill system doesn't capture what's important

to players in terms of matchmaking so let's see what it looked like so as you can see here the old skill system is flat with respect to quitting as the skill gap increased an old system players didn't care they didn't quit what that means is that skill is important because as the blue line shows players do quit as it increases but it's not being captured or controlled by the older skill system so don't just give up on skill because your data says that it's not important you need to first consider whether your skill system is capturing what's important to your players and so you're asking what what is important what's the difference between these systems so let me discuss what the two systems were previously we used classic true skill in classic true skill which is a solid system I've used it for years in many games it only looks at the winning team though when it decides to evaluate how well the players did which players are better and worse and it needs a lot of matches to do that it can be up to 80 to 120 matches or

more when we're talking about four before true skill - also incredibly values the winning team it's a very important part of how it decides who is good but it also takes into account several other factors that enable it to be much faster and more Kirra than the classic trueskill it looks at how many plug games the players played so far it looks at the party size and maybe even more importantly it's looking at kills and deaths which we owe in Halo 5 are very important to player engagement so we have objects of events that are in our skill system that are also important to how pleasures are able to engage within a match so and again having this data available also makes the system much more quickly figure out how good our players are so that when we're looking at those gaps in our data that are more likely to represent an actual skill gap between our players now just to be clear here this is still just one number per system we're not like using a whole bunch of extra numbers we're not matching we're not creating lots of extra buckets both of these

systems boil down skill to just one number so again the takeaway there is don't just discount skill gap being important to your players think about your skill system first because which one you're using does matter so let's do a recap of the importance of skill gap we've seen that personal skill gap is important in Halo 5 we found that we were able to get a four times drop in quit rates by matchmaking carefully on that personal skill gap also teen skill gap is important fairness is important to our players so both of those both whether or not our players are able to come around a corner and have a chance of a good encounter with an enemy opponent and whether or not the match actually feels like I can win as a player those are important to our players we're able to have the quit rates drop by about six that way now you may ask which one matters more well the the data tends to suggest players are more sensitive to that team balance skill gap and luckily team balancing ends up being a little easier as long as

players aren't in full parties you can shuffle them around and get a good team balance even if your initial matchmaking didn't come out as good so it's a little bit easier but yes players tend to be more sensitive you're more likely to get a quit out of a match with a bad team back team gap in their team skill gap so find out which is important to you go look in your data see which skill gap matters to your players and then use those graphs to tune how you're going to match make them and again remember your choice the skill system does matter don't just discount skill because your skill system isn't detecting any sensitivity think about whether your skill system really is measuring what's important to your players so up to this point we've been using whether a player will leave a match to show us how important these different characteristics are in this case skill so far the question is can we go beyond though just what happens in a match because maybe a player quits and maybe he goes and plays again yes it does kind of ruin the match for everyone else but we still want to know can we go beyond just a player leaving a single match and the answer is yeah we can take

a player we can observe them going into matchmaking they experience the gameplay hopefully they're engaged that personal skill gap is small so they keep playing match after match now in theory if that's not the case they would stop playing for the night they may finish the match they may not even quit out of it but because of their experience in that match they may decide that they don't want to play anymore halo 5 that night and this is also important to us maybe even more important because if players are not quitting not just out of matches but out of your entire game it's kind of ruining your population overall and just making the game worse so we want to go and measure this basically we want to see our players not quitting just the match but do they leave for the night because if say personal skill gap is important wouldn't they wouldn't they stop playing so how do we do that well we set it up very similarly to our other experiments already we give each of our players they've got Alice here we know what her team skill gap was for that

match or we know what her personal gap was for that match and instead of asking did they stop playing did they leave the match early we asked did they stop playing for eight hours what did they take off and not show up again and we look in this case that Alice had a match with a gap of 100 and she kept playing whereas Bob had a gap of 250 and he stopped playing and we want to average this over all of our population our players our matches bucking it again and graph it and see if this skill gap not only causes quits out of matches but does it cause them to quit out of the game and after and it turns out it does cause them to leave the game for the night as this personal skill gap that's what this one is the personal skill gap increases so does the chance that a player will not only quit out of a match but they may leave the game for the night and this can go to be as high as a 40 40 percent chance that a player is going to turn off the game for the night because of a large personal gap which means it's very important to our players to not have

matches where they are so much worse than the other team that they can't really engage so let's look at the takeaway from that the takeaways are kind of the obvious ones right players will disengage if the personal skill gap is too large and we can again use these graphs to tune and pick what type of personal skill gaps our team skill gaps are okay for our players and again the nice thing about this is we can go beyond just looking at one match now I want to switch gears a little bit and talk about a more controversial topic that comes up when once you start digging into this data and that deals around stacked parties the question here being you got what I call a premade stacked party this is a group of four friends who are playing together in the party when they go into matchmaking they go in together and that's fine the issue here is that these aren't just for average players or even for very good players these are like four of your best players they've decided they're friends

they're going to get together and play and what this causes is they will play against another group potentially another premade group a party of friends that are playing together and that party could be very good but they're just nowhere near as good as that stacked party and what happens is that the stacked party will stop that pre-made party I beat them very badly and this leads to if we believe our data already players disengaging and not just disengaging out of the match because and to be honest players do quit less out of matches while they're in parties like this good party here but it cause it will still cause them to disengage out of the game entirely which is even worse so what can you do about that because it's and I'll give you a hint that there's not a lot that's there's not a lot of great solutions at this point to what you do you can ask yourself well should we just not allow those stacked parties because they're gonna be scaring players out of our games and if you do that you end up making some of

your best players so you're making it so they can't play together and that's probably gonna make them sad but if you allow the stacked parties you're being unfair potentially to everyone else they play against which is a much larger group than the small group of stacked players that want to play together and again there's no super great solution here but it will just demonstrate what this can cause so if you allow these stacked parties at least I've observed in our data I've gone and and just watched some specific stacked parties and seen what their effect on the population has been you can actually watch a single party of four players scare away up to twenty players per hour and the problem with that is it the math just doesn't work if you ask yourself is it okay that we allow four players to play at the expense of 20 players per hour leaving your game entirely for the night the answer to that is usually that's probably not a good idea so what do people do about this so in Halo 5 what we do is we just set our matchmaker

to allow fair matches and it won't allow unfair ones so if there's not a good match for that stacked player they'll end up with a very long wait this ends up being better for our population overall but it can be very disappointing to those players who'd like to play together now other games have different I have similar solutions League of Legends they you can also result in long wait times but if you're a solo or a dual player or just two of you you can go into a solo duo queue and play only against all the players or players of two but this doesn't protect your players that are in larger parties that also don't want to get stomped by those and it does shard the population resulting slightly longer matchmaking for everybody overwatch has another interesting solution they basically say no if you are too good you cannot play with your friends they just won't even let you match make they if you're a grandmaster and above in overwatch you're only allowed to queue either alone or with one friend and this also has some relation to what sports leagues do in

order to keep matches more interesting they add hard soft or hard or soft salary caps to promote competition say they they tell teams you can only spend so much on your players in order to prevent these same types of issues so it's not without precedent in general and coming in competing but again there's no great solution here but it is something to think about what do you do with these stacked parties how do you let the match mate how do they affect your population is worth digging into that data okay so we've covered skill probably to death at this point we've looked at it from a lot of angles quits versus leaving the game and so let's get back to our other items we were curious about what about latency does high latency cause players to leave the game or does it cause them to quit for the night and you know the answer and the data that was nope not really turns out for our players they were not we're not very sensitive to ping day we didn't find a lot of quits we didn't seem players that are leaving the game for the night seem to be a pretty not

that strong correlation now this may be because we have some safeguards on ping globally already but it didn't we didn't find a lot of connections there all right so we'll cross that one off for now now what about wait time do players get frustrated lots of long waits and with wait time you can't do a quick thing most players don't tend to wait a while and that caused them and then quit out of a match but you can't see if having long wait after long wait causes them to leave the night early and it turns out that no we did not observe that we our players didn't seem to mine even longer than usual wait times they would keep playing there the amount of time they've instead spent for the night did not seem very correlated with how long their wait times were and what about losing streaks cuz this is something I hear a lot about almost anytime I join a company and we talked about matchmaking someone brings up we got a break losing streaks but you know it turns out we did not see players tend to leave early because of a losing streak they would the only thing that was we found that was correlated with the player that

would scared them away for the night was whether or not the match itself was fair a really unfair match would they would leave the night leave for the night because of that but they were not lucky if they had a string of matches where they lost that didn't tend to correlate with players leaving early for the night for quitting so maybe that our players are just more concerned with how fun the match itself was and the fact that they lost isn't as big of a deal so it turns out at least for a halo 5 what our players really seem to care about is that team skill gap which is do I have a shot at winning this match and the personal skill gap which is I come around the corner and I see an enemy what will my success rate be in that encounter can I can I experience the gameplay the other aspects that are commonly thought of as important for matchmaking sort of fell off for us which kind of shows us that at least in Halo 5 we should focus on those team and personal skill gaps now that doesn't mean we let latency be ridiculously bad or wait times crazy

long but it means that our current sort of guidelines on those other objects are pretty on latency in a wait time are already pretty good and you should be able to discover this as well on your own data so let's review the core approach that you would use to do that so again choose a characteristic something that you think is important or maybe important to your players and you want to find out then choose a metric what are you gonna measure against that characteristic we used quit rates and we also used what players quit for the night so we compared you go ahead on your own and compare whether I'll high paying results in quits or results and players churning out and create those types of data points so for every player for every match look at what that skill gap was did they quit or what was the ping and did they stop playing for the night and pluck it those up and plot those characteristics and then use those plots to tell you if it's important and how important it is and at what points you're ok matchmaking at what cut-offs do you have in skill and and in ping or

whatever it may be and some overall sort of takeaways to think about first what you think is important what everyone's been telling you is important to matchmaking it might not be important it might it be even more important so use some observational studies so look for your data and find out what is important and then tune your matchmaking to reduce quits to reduce people leaving for the night to reduce whatever metric you've chosen is important to you and don't forget again I'll just reiterate this that when you're doing with skill the skill system you use does matter so reevaluate your skill system and make sure that it take it takes account of events in your game that are important to players that gameplay is somewhere in there that's sort of a like maybe a surprise takeaway for some unit that your skill system may not even be capturing that now I talked a little bit about a/b testing because I earlier mentioned that it can't be used and the reason kind of why it can't be used goes into some pretty complicated

reasons that can't cover all of them in one slide but some research that Microsoft Research did showed that the optimal skill gap to matchmake on for example depends on the population itself at the time your matchmaking the problem is an a/b test we'll split that population so what you will learn from that a B test is what is optimal for the way you've split but that split may not be what the merged population is like so you can't really do that you you're learning the wrong optimal settings for the wrong population so what you can do instead and what we're looking forward to trying at some point is what you do is you flip a coin every hour and choose every time you flip that coin whether to go with a or B and then where a is like a is how we're matchmaking right now and B is the new thing we want to try you flip the coin because you need to have some randomness you don't want to have like the time of day be messing up causing correlations in this data so you

just flip that coin over an extended period of time every hour and you flip back and forth between these methods and watch how the players react or in those methods and then check your metrics between those a and B it's the same way you would for any other AP test that way you are measuring on the correct population and that is it thank you for coming to my virtual talk here I hope you enjoyed it if you have any questions you can put them in the comments below you can contact me through Twitter I also can be emailed at Microsoft and take care