Luck and Skill in Games

[Music] hello I'm uh scaff elas um I uh work at a company called three donkeys with uh Richard Garfield uh you've probably heard of him and not me uh and um I worked at Wizards of the Coast for a long time for many many years uh about 10 years ago we started this uh Consulting business and um uh just sort of go around and help people with uh their games um I supposed to ask everyone to silence their cell phones uh thanks and um yeah the one piece of advice the Assistant gave me was to not choke so I will attempt to not do that um uh as recently as 24 hours ago I thought this was a 60-minute talk due primarily to a lack of intelligence and uh so uh good luck as the audience um you're going to have to do your best Calvin Johnson impersonations and catch what I am throwing

uh so luck and skill in games the talks primarily about um how luck and skill interact uh with each other and why do we care we care because uh a lot of times you'll hear complaints there's too much luck in a game there's too much skill in a game and uh it matters a lot to your audience and it matters more and more to your audience as time goes on uh the longer they associate themselves with your game um virtually every definition of game that you'll find uh will state that some amount of indeterminacy is required and those definitions that don't state it uh explicitly uh imply it uh it is a necessary part of what it means to be a game and um what creates uh indeterminacy well Randomness does so we get to luck and skill and we try to um start defining these and we find that first of all they're very difficult to Define uh it's not

colloquially we have uh some understanding of these when we try to nail down very strict definitions it can often turn out uh that they lead to counterintuitive results um and I think that's um exactly how to define it we're not going to be uh too careful with but what uh how they interact with each other is sort of what we'll be talking about um so over Randomness in games this is actually easy to see it's easy to Define it happens a lot um you can have dice cards random number generators and that's clearly going to add a degree of Randomness to your game um some things that may be less clear are uh game theoretic Randomness that comes from say simultaneous decisions um uh other players politics um there's a lot of other ways besides sort of overt Randomness uh and then of course there's physical and mental limitations um your

memory uh your accuracy throwing darts at a dartboard uh that's clearly random um again it's not completely random uh but for all practical purposes it's random within a certain um set of parameters uh your speed your strength these are the sorts of things that uh create Randomness in games uh a simple way to have Randomness in a game is maybe maybe you have a simple choice between two doors door a leads to Victory and door B leads to success uh and or sorry failure um and uh and so it's pretty clear right the ogres behind one door and the treasures behind the other um it's completely random but what if uh you had some information about the doors that maybe you don't even see at first maybe there's a very complex way that you could have figured out uh what was behind uh one door uh and what was behind the other and and this choice

that was random to you if you decide that you can't figure out these equations uh is it really random is it really Randomness in the game or is it a matter of skill so often times uh when you ask people uh to name a game that has uh no luck in it uh chess is the first thing that pops up and uh because there's no overt elements there's no hidden decisions um and so uh it seems it feels like this should be a game where it was a game of a pure skill uh my uh my business partner Richard uh has often said oh maybe one day he'll play Gary Kasparov at chess and um to sort of prove that there's Randomness in chess he'll just play randomly and win uh and it's clear that he can do this uh uh Gary Kasparov has not won every chess game he's ever played and uh as long as there is some set of moves uh that uh will lead to Victory against him

uh Richard could stumble upon it uh unfortunately his chances are better of winning the New York State Lottery uh 15 times in a row so he shouldn't go uh betting a lot of money uh on playing against Gary but the point is that um there is Randomness in this game unless of course Gary has the entire move tree memorized in his head and he has a response regardless of what that random move is it feels for a game like chess that it should be pure skill and sometimes we forget that uh well chess is quantitatively different than Tic-Tac toe it's not qualitatively different and Tic-Tac toe is a game where you can observe people playing and it's very similar to the situation of the concealed doors where you and I may not experience Randomness when we play it but children do and uh it's very interesting there's

um there's been some uh reports done on this of watching children play and you can find that the results are uh mapped very similarly in character to uh to the results of chess players playing against each other where you'll have a set of people where it's virtually purely random some people are better they're able to see a little bit in the decision tree so they're making less mistakes and then some people have the game uh essentially solved um we're probably too far behind that but the past our memory of Tic-Tac toe to remember when it seemed random when it seemed difficult and we had to make a guess when we didn't for sure know what the next best move was um so the point to this is that uh the amount of luck in a game is not a function of the game itself uh it's a function of the game and the player and that's something that's uh very important to understand let's take another pure skill game like chess um it's not the most fun game but

uh it's still a game and it's you've got 30 seconds to compute the 50 million 347,000 digit of p uh does anyone want to play um and you take a guess it's obvious to us it may not be so obvious if we look at a game like chess that it is that there is Randomness in it um but it is obvious that in this game there is Randomness in it uh because this is impossible to compute in the amount of time however again without changing the character of the game we can say uh what's the fourth digit of pi and now when we're playing this game this is inside the scope of the human mind and uh with a little simple memorization now all of a sudden this game that was purely random has no more Randomness associated with it the point about this game is that

it's not just about the game and not just about the players but it actually changes over time even if you sort of understood the complexity of the human brain to a much higher degree than we'll ever understand and even if you understood exactly the calculations required to uh to play the the previous game um if when you play the fourth digit of pi game in 2000 BC it's purely random it's no different than when those people are guessing the 50 millionth to Pie when you play it around 250 BC it all of a sudden becomes an incredibly skill testing game uh aredes around this time starts figuring out within uh his guess would be between three different digits uh and uh and now in 200 ad or 2013 uh there's again no uh Randomness left in the game or and there's uh a very little bit of skill uh but there is some and the interesting thing here is that you can't determine by looking at

the game or even the play the capabilities of the players you can't determine how much Randomness there is in the game it's something that varies over time so we think we have an idea of what these pure luck things me in a game but what does it mean to have skill at a game well the dictionary definitions of skill are the ability to do something well uh another way to say it is it's an intrinsic ability to achieve a differential out come in a game um and note that all of these things with the word well or um trying to achieve one outcome over another uh hinge on the fact that skill is essentially a comparison and so the measurements of skill that we have like an ELO rating system or whatever are always uh comparisons um that's the best way to get at what it means to have skill in a game uh when you ask people uh is this game more skill testing than another game does game a have more skill than

game B how do we measure that is there a simple way that we can do that well um when you ask say students in a class or uh have sit around and have a discussion about it some of the things that pop up are what's your maximum win percentage um there's an there's an intuitive thought for people that uh if a game is very skill testing the best player a pro at that should essentially always beat the worst player at the game it's pretty clear if the best player in the world has a 50% win chance against the worst player in the world let's say the game is cutting for high card um there's no skill in the game so uh and the higher that number goes um in some sense there is more skill in the game uh another common way of defining how much skill there is in the game and this is actually a very interesting and effective one because it leads to um say social structures that

or that might develop around the game or the metag game in the game is um how many different buckets can you put people into where bucket two has a 75% chance of beating the bucket from people one how long is that chain um the longer that is uh I think almost everybody would say there's more skill in that game uh the ELO rating which I think uh probably most people are familiar with it's the chest rating system lots of video games use it uh it's been altered over time um actually can reconstruct if the assumptions that ELO made are true can reconstruct that uh level of uh the chains of skill um and what it does is it sort of assumes a random performance on people's part and uh actually is pretty good at modeling a lot of different games but the complete set of information is actually a much much much more complicated thing uh it's if you really want to know how much skill there is a game you would need to know the true expected win percentage of every single single player versus every other

player over time this is not something you're going to be able to get this is virtually impossible and how the curves of player performance change um is actually very interesting because it is not easy to uh a simple linear parameterization of the level of skill of people um so you have to choose what you're going to measure because you're only going to be able to take a slice and you have to Choose Wisely let's take for example uh this game we'll call Rando chess uh we like to use sample games because sometimes it actually gets to the heart of the matter more than say uh a real game does um so you play normal game of chess and then afterward you roll a die uh and then on a one whoever lost the game they are declared the winner um well Rando chess clearly has more luck than chess does but does it have less skill well when you think about it uh any sort of of these reasonable

definitions of skill that you had beforehand in your intuition are that really actually it doesn't have any less skill and what do I mean by that all of your previous skills are still useful your player ranking of every single player in the world has not changed at all you still have the same world championship you same same world champion you still have all of the chest books you still have all the opening moves you still have every little bit of effort and every little bit of skill that you would want to acquire to win chess that's all still existing in the game of rando chess and the point to that is that the skill versus luck is actually a false dichotomy really you should think of it more as a graph where you have a certain amount of skill in games and a certain amount of luck in games and games can be uh sort of placed anywhere uh on that um go has uh go has uh a lot of

skill as does poker but poker has more luck so what did change in Rando chess the game clearly changed skill by a reasonable definition of it didn't change uh but what did change was as luck went up skill became harder to measure in order to achieve that same ranking um you would actually need more trials than before another way to say that your skills is harder to measure is that you get less payoff in terms of your victories for the same amount of skill um so when you add luck to a game and you make skill harder to measure what's happened is that bad it may very well be bad because um maybe your players expect a very high payoff for the amount of skill that they've acquired at the game they have have after all put a lot of effort into it um

and key to understanding this is key is understanding what the value proposition of your game is for your players uh for instance if you're playing a competitive game and there's some pay offer skill as those players continue to play that game or that genre of games over time and they put more and more time and effort into mastering that game they may very well expect a high higher and higher payoff for their skill and uh and that's how you get things like fighting games becoming more and more skill testing over time if you're a dedicated customer of these sorts of things again for a competitive game where inherently in the setup it's you versus the other person on an emotional level as well as uh sort of a rules level um it's not unlikely that you're going to uh be very upset when your payoff for skill uh gets lowered so should we remove all DET all

indeterminacy from games uh when you do that uh you can run into trouble if you make a your game to skill testing you can play the game who's taller uh and then when you do that it's no longer a game because you've removed the indeterminacy really it's more of a measurement and measurements are fun to play at most once when you look at games on our luck versus skill graph you can see things like a foot race there's really not that much Randomness in a foot race uh people's times do vary but they vary by a relatively small amount and that's why back to the measurement thing how many times is it fun to take a measurement you'll see grade children grade school children will play foot race and they'll play it once or maybe twice and there's not a lot of people in high school or post colle or elderly people playing the game foot

race and there's a strong reason for that and that's because it's not that fun to play Time After Time After Time whereas you can have games that have more skill associated with them and again by more skill you can you can take a sort of a empirical measurement of the amount of training that people do to become worldclass sprinters as compared to world class soccer players uh and see that it's less uh in order to achieve the top of the profession and uh but there's a lot more luck in these sorts of things and you can look at this chart and say oh I kind of understand intuitively that um I may not want to reduce my uh indeterminacy by too much I don't want to skip over this too quickly but there are other benefits of luck and Randomness in games um that we are not talking about uh there's a million benefits to Randomness in games um for instance people are just in thralled by unexpected outcomes uh

specifically though since we're talking about the luck and skill interaction uh we're going to that that's a subject for another talk so when you add luck to a game because you've looked at this chart and uh and you see oh well look golf sitting there uh a lot of people play golf uh more than play foot race um and uh what are some of the benefits of adding luck to a game well one of the benefits is that when you play a game that has luck in it uh we saw this a lot uh with magic uh the Gathering um people would tend to blame their successes on skill and uh they would blame their defeats on luck that's great that's an amazing psychological crutch for your players it makes people much happier playing the game even though uh they're losing um because there is an overt Randomness to it it gives them something to uh very

clearly fall back on uh the problem is over time you sort of the the more and more experienced players are the less and less this illusion will hold up and they understand that it really is their fault but you're the great benefit of this is is that early on in a player's career they may uh get an ego boost when they're first starting to play the game uh another benefit of adding luck to the game is um the amount of people that you can have fun with uh that is to say play an indeterminant game goes up in a game with really low luck uh you can have most games be too hard uh or too easy and they're not fun to play again it's closer to a measurement um it's a very difficult problem to solve today with the internet and computers and matching systems you can find that just right person if you're audience is big enough um and the problem is that even after after you find that exact right

person who you're going to have a fun game against that there you might find the right person in terms of skill but are they the right person are they the friend that you want to play with maybe you want to play with your family you want to sit around in a living room so this ability of matching the internet and computers to solve this problem doesn't help if I want to play with Timmy and my game is so skill testing that uh it's not fun to play then I have to think that maybe I'm playing the wrong game as we mentioned before with the fighting games um payoff for skill is something that players uh especially in games where reward for skill is extremely important say poker um uh over time they may expect to get a higher and higher payoff but because luck can brought in audiences Phil is probably not so upset that uh poker has less payoff for the

skill that he's put into it he would much rather be uh the prince of the set of all poker players uh top 10 or 20 then he would be the Undisputed number one of an audience the size of say go players so the situation is a lot more complex than you can see from something like the ELO system because um a one-dimensional parameterization of the amount of skill that people have doesn't necessarily uh really tell you what's going on um however and you can actually see that there's some sample things here you can see for instance uh there might be an element of luck in the game that's only accessed if you have high skill maybe you when you go really really far into the game you're an amazing amazing player then you're able to access this bonus level once you hit that bonus level there's a random you know 99% of your score gets access to that bonus level what that sort of thing will do is

if you add that to your game you have reduced you've increased the luck between top players playing against each other and you haven't changed it at all for your lower level players and you haven't changed it at all for when your higher level players play against your low level low lower level players same sort of thing can also happen because um some elements of Randomness can only be uh access by playing poorly like maybe a shell and Mario Kart or something like that so how your skill curve of each person and how they interact with each other and how much skill there is and how much Randomness there is in their interactions uh is actually a very comp complicated situation we'll take this example um maybe you have a real-time strategy game and uh you can have randomized prices or Technologies in it um before randomization maybe it's very clear what the best strategy is the best to build a tank and you have three

different players and uh their ratings are you know you have two excellent players the game is primarily based upon say hand eye coordination tactical skill that sort of thing your strategic choices aren't really that important but then you play and the prices have become randomized between all these objects so all of a sudden uh maybe dragons are a viable strategy now um they were overpriced before but now that's the way to go well you've clearly added skill to the game for the player to decide in the beginning what the proper choice is is an area of skill and Mastery um so maybe player a can now beat player B more often that 100 Point rating differential between them might expand the levels in your skill chain might actually go up at the same time uh maybe the best skilled player in the world can only uh pick the right thing 90% % of the time and player

C picks the right thing 10% of the time well if the Strategic choices outweigh all of the hand eye coordination and all of the Tactical skill that you could develop at that game guess what the the win differential the expected win differential between player a and player c will now go down even though it went up with player a versus player B so how do you use all this information since basically what you sort of the more you look at it the more you realize how complicated the situation is well I would start by saying you have to know what your you have to know who your audience is you have to know what your intended audience is uh and what's your key value proposition to them do they expect that it's a highly competitive game do they expect a high reward payoff for their skill is it going to be played around the family uh gner table I mean if it is uh you better make sure that you can have a a highly indeterminant

game among almost any set of people that are there uh is it going to be played over the Internet is it going to be played solo all of these things will tell you how much Randomness you ideally would have in your game in order to get a significantly consistent amount of indeterminacy uh in each instance of that game um does your Revenue model require replays this is a really important consideration again back to the foot race example uh if your Revenue model says I want this person to come back and play again and again and again and again you want to stay as far away from that measurement as possible unless perhaps you have a truly enormous audience so you know that you can always match that person up perfectly against someone who's almost at exactly their skill level it's a very hard thing to do but it is possible another important consideration

is do your IP and marketing match your skill and luck interaction Mario and the dragon from RTS sample game number seven uh are not the sorts of things which appeal to the type of players that are going to want to play a game like go which has an extremely extremely extremely long seal chain very high reward uh ratio for the skill that you achieve at the game and it's extremely competitive and sorry that the talk was so short had to skip over some things um uh Richard and Robert and I uh wrote a book characteristics of games a lot of the stuff's in there if you want to read more about it any [Applause] [Music] [Applause] [Music] questions