Data Informed Decisions with the Champions of League of Legends

[Music] welcome today to champion insights um i know a lot of you guys are coming to this talk like okay riot games is giving a talk they're only going to talk about league of legends and their esports and their bizarre stuff with that that's not the case today we're actually going to be covering more than just league of legends uh we're gonna be covering stuff from a lot of games a lot of the data that i'm gonna be presenting today and the methodologies can be applied to your games or to your studios uh we're also going to be talking about some other stuff we're going to talk probably a little bit about k-pop abstract expression expressionism in american art and we're going to talk about scuba suits but most importantly we're going to be talking about champions and league of legends and characters in games overall so who am i uh i am nathan blausjoyce blau um blaustois is a mix of uh blau and the pokemon blastoise nintendo uh attorneys please save your uh questions till the end

um i studied uh neuroscience and political science at the university of chicago uh jason jones who is one of the co-founders of budgie was always a major influence to me my career in the games industry was started at playstation uh i have to take a brief moment to thank the play playstation user research team and justin tanimura specifically they got me my start in games and i owe them everything so thank you guys so much um and now currently i work at riot games maker of league of legends and i am the gameplay insights lead and we will talk a little bit more about what that is what is insights etc because we're talking about champions today though and we're gonna be talking about characters and champions in game both my favorite champion in league of legends and also my favorite character in all of games is yasuo so yeah wow my gosh he's a yasuo main more on that later uh that's certainly covered in the

talk cool let's talk about insights though so what was i saying when i mentioned that i'm the gameplay insights lead what's insights at riot well it's a mixture of analytics research a splash of data science and a healthy serving of strategic advising in short what we do is we gather and analyze data both quantitative and qualitative from our game from our players and we provide uh insights recommendations strategies uh to our designers our artists and our leaders um today's champion or today's talk is going to focus on champions so this is champion insights we're going to be talking specifically about how we use that data to inform decisions around the roster of over 140 champions now it feels like yesterday that it was not that much but now here we are at 140 in our 10th year um over 140 actually uh but every time i reference champions in this talk i want you guys to know that you

can apply that to characters in games and what i mean by that is that characters are everywhere in games it's not just league of legends and their champions right we have games like overwatch battlerite we have mobile games uh we have stuff like brawl stars we even have mobile moba games don't say that 10 times fast uh like mobile legends with the characters in their game uh we even have you know the new hotness battle royale finally has you know some a great character set with with apex legends and then we even have character games where we take sort of collaboration uh of characters across multiple games and genres with something like super smash bros so in all of these games and in league of legends there's only one thing that is more important than characters and you know sorry because i'm cheesy but it's players it's the people who play and engage with these characters so the importance of today's talk is that hey

we're looking at the most important part of your game characters and we're talking about the most important part of the game holistically which is the players who play it we're trying to understand the relationship between those two and make better decisions so characters matter to our players that's why they engage in so many of these games even even games that aren't necessarily you know these competitive pvp games like overwatch or league of legends we can even think about the importance of characters in mass effect or something like that um what i want to talk about is how these you know just go through a couple examples of some players who latch on to certain characters and and build an affinity for them so i'm gonna go through three some of you guys might know these some of you guys might be totally unaware but these are three of the most popular streamers on twitch tv of league of legends so these people play league of legends for a living and have other people watch them so first one i'm going to go over is is mo okay and mo is this guy he's kind of he's kind of this cool kid you

know he's kind of cocky he kind of likes to show off and style on people and and the champion he's best known for is that one that i referenced earlier it's yasuo okay and yasuo is is you know perfectly embodies this show-off sort of style in fact one of yasuo's quotes is kill me you can try so very very fitting uh next we have pokemane and she's a streamer who's you know on a different end is super bright and cheerful super high energy and positive and her favorite champion is lux and lux in our game is this light mage she's the spirited happy character um she's always being positive and uh you know her quote is if you can smile you can be the light and lastly uh we have my my dear friend tyler1 um and and tyler1 is you know hey this guy is competitive this guy is driven competitive fierce he's all about he's kind of this

alpha male all about dominating his opponents and the character he plays has gravitated towards is draven and draven embodies that in our game perfectly draven is all about dominating his opponents in the early game and it's all about sort of that strength and style and his quote is subtle i don't do subtle so our goal with these data is to understand the relationship between players and characters and it's not just these three high-profile you know characters uh these three high-profile players and the characters they have been drawn to but we're talking about the stories of millions of players across the world playing millions of hours on these characters and that's where the importance of all of this comes from so what kinds of things can we inform with this data well let's go through just a quick list and we're going to return to this list at the end with specific examples but just to get your mind thinking about them as we go through the talk today here's what i want you to think about

we can inform decisions on your current game uh or the future state of your game right you can talk about new opportunities for your character roster being more deliberate and informed with updating characters or updating your game you can talk about how you would create games with the same ip say you're creating a new game with the overwatch characters say you're taking the apex characters and turning them into a moba game you can inform that data for those games with an understanding of your current characters you can also use this data to inform rosters with entirely new ip imagine you knew everything there was to know about the overwatch characters and someone tasked you with making a new game with a similar sort of style as overwatch you could use that data from those overwatch characters to help and inform what you're going to do with this new ip and it's not just games it's creatives too we can talk about

how we feature our characters in art and in comics and in movies we can talk about publishing how can we be more targeted about presenting the more aspirational characters to the right audience in our publishing efforts and even things like physical merchandise to make sure that you know the characters that players want to buy the merch for the most are getting that chance so we'll get to these later and i'll show some specific examples but you guys are like okay nathan you showed us tyler1 you showed us a bunch of other games where's the data okay let's get to the data today we're going to be covering three data sets today um and let's just briefly go over what they are so you can be prepared for them uh the first one is a pretty simple engagement data set it's our play rate over time uh data and what this basically tracks is how much players are playing a champion over the course of time very simple our next data set is similar to this but a little bit more complicated

it's our breadth versus depth measurement and this is similar to our play rate data but has a little bit more nuance and a much more complicated tableau graph much more on that later and then lastly we're going to shift away from engagement data and sort that raw telemetry and we're going to focus on our mass champion survey and this is something that's measuring more sentiment more feels from our players about what they think about our champions and this can cover things beyond just people playing the champions but stuff like visuals voice personality and how our players feel about everything outside of the game for the characters as well to start us off though let's go with that play rate over time and we're actually just going to focus on one champion in particular and go through a sort of series of stories about this champion oh and sorry one final note i do have to note we're going to be covering stuff from active league of legends players

keep that caveat in mind we have other data sets that look at stuff like league of legends players who have stopped playing or potential new players of league of legends for today the data we're going to be looking at is active league of legends players okay let's go on a play rate journey and what i mean by a playwright journey is let's see how the play rate of a champion changes over time specifically over a few months about a year uh and we're going to be talking about a collie um now before i get into that juicy data i am going to pose a question that we will try to answer with this data which is what happens when a ninja joins a k-pop group i'll have a series of these esoteric questions sprinkled throughout we'll see if we can get some answers to them but this is what i meant when i referenced k-pop um now for those of you who are not familiar with league of legends i have a brief explanation when i say champion update so let's quickly cover what that is so league of legends is coming up on 10 years now it's quite a long time and part of that longevity is that we're

constantly evolving and changing the game to do that we have to sometimes look at some of our older characters and find ways to make them fresh again whether that's updating their visuals updating their gameplay or in the case of a cali we did what's called a vgu visual gameplay update the full package totally reimagining this character into something new and fresh kind of like if we took that archetype and we're designing it today so akali was previously a more sort of just kind of generic ninja um she uh yeah was a green ninja i guess there wasn't too much particularly interesting about her and we sort of totally reimagined her into this more sort of street style ninja she got these really cool tattoos and her game play was made much more fresh for the current age of league of legends so what does updating a champion do to its play rate let's start the journey so

this is akali's play rate over the course of before a little bit before her update through her update and all the way to today the x-axis is time you can see sort of in very small text the monthly call-outs and the y-axis is just how much akali is getting played relative to other champions all the other champions are the dimmed out ones in the background we've highlighted a cauli in that nice blue teal to start off our journey i want to draw your eyes to the very far left the very very far left of that uh graph to where there's that flat line and there's a little bump before the giant spike and that is akali's last hurrah so what this is is when we have officially uh started to update akali we've put the new version of akali on our public test realm and all of the players on the live servers are like okay it's real now they're actually the mad lads they're actually going to get rid of akali we need to get our last games in so all

of the current akali players will go and get those last games in maybe people who used to play her will go in player but we get this little last hurrah bump and you know it's emphasized by wolverine here fondly remembering old akali okay but it's league of legends out with the old in with the new let's talk about that update boom that spike on the graph is huge because this is when akali actually drops that's the day that the akali update drops and a bunch of players go and play her and these are not just the old players trying out to see if they like the new one these are players like myself i never had a game on the old akali but i saw the new uh the new gameplay the new abilities i saw the new visuals i thought wait you know i play yasuo i'm kind of a weeb that looks like a champion for me so i go and i start to play on a call as well and then other people are just checking her out just to get a sense of what the kit's going to be like when they play against it so you know again we we sort of we sober

up after that initial launch and we get to this sort of new stable play rate which is really exciting because if you guys look where this line sort of flattens out it's so much higher than previously you know akali has played multiple times more than where she was previously and that really hit uh our goals for the character we thought this was a fantasy that we had not delivered on as well and we wanted more players to engage with her and to engage with her more so this was looking like a success and you know what the success doesn't stop there um what happens when a ninja joins a k-pop group well i'll tell you we get a juicy spike of engagement uh and a lot of players playing a k-pop ninja uh so what i was referencing there is of course uh kda um for those of you guys who are unfamiliar this riot basically was like hey let's make a k-pop group yeah so we took four champions we put them in a k-pop group

we gave them new looks new skins in our game and a lot of players who don't typically play akali would go and flock to this because this really cool new visual look new identity for her and we saw a huge bump in uh her play rate at this time but all good stories have to have you know sort of a downfall the downturn so let's talk about that in this period after kda we suffer an era of pro-play dominance league of legends is also played uh by professionals around the world and two things happen one uh she becomes very very powerful in professional play and it's played all the time that's number one number two she becomes really frustrating for players in normal play so not only is she too powerful but she's also extremely frustrating here we see uh actually g2 it's a professional team in europe this is perks uh one of the pro players who plays a lot of akali

um if he's on your team that's great if he's on the other team that's really unfortunate for you when he's playing that champion so we get to this dark time where we were constantly trimming and nerfing power out of a cali and the only way we could actually get her out of pro play reduce some of that frustration was to hit her with substantial nerfs and nerves are where we remove power or remove mechanics from a champion usually it doesn't hit play rate that hard but in this case i mean we we obliterated the kali we just we just yeah it was it was a massacre it was terrible we uh we took so much power out of her uh it was very rough um i think the best way to sort of show what it would look like is with this classic uh scene from the avengers you know are you finally not getting banned yes what did it cost everything so akali dips into this new low almost as low as her pre-update play rate as a result of these hard nerfs

but two patches later in patch 9.5 and the patches are a major update to the game she gets redemption we look at some ways of bringing power back into a cali and buffing her a little bit and so we are able to do that and players flock back and start to come and play her a bit more and we're seeing her play rate rise this is just as early as a few weeks ago actually and here we see a league of legends player now just a few days ago playing some games on akali uh wait that looks like my match history actually and we're gonna ignore the game right there where i went 5 13 and 12. um you know everyone has bad days etc but overall this is our story of akali this is her play rate journey so just using this sort of simple metric and looking at one champion over time we can tell compelling stories about the players who engaged with her over time the content that was released for her over this time it helps us to

understand this character going forward we know where some of the best and worst states of this characters are and so for champions where we know we want to retain some of this play rate we can always look back at this data to better understand that and we're not finished yet this isn't the end of the story you know the akali train will continue and we'll keep tracking her and all the other 140 some odd champions over time with this but we don't just track engagement with champions through play rate we have a really exciting piece of data to share with you guys today this is our breadth verse depth framework and this is another way of looking at champion engagement and i want i want you guys to remember that akali journey because we're going to come back to it framed in this new way uh and maybe we'll learn some more things from that so breadth verse depth let's get into this and answer the question how does jackson pollock help us understand champion engagement now

some of you guys like wait who is jackson pollock so jackson pollock is an american painter and major figure in abstract expression movement i am not an art critic thank you wikipedia for that note um he's widely recognized for his technique of pouring or splashing liquid paint onto his canvases um so now at this point like okay you told me who jackson pollock is uh nathan why are you talking about jackson pollock well let's look at this interesting jackson pollock piece okay it's a little bit more modern all right a little bit a little bit brighter colors you know some more jagged lines but it's a nice it's a nice pollock piece right wrong incorrect this is actually a david nevadi piece uh david nevadi is not an artist he lacks the grimace he lacks that cigarette in his mouth but he does have a wonderful understanding of analytics and a nerf gun and that has allowed him to create some really interesting data visualizations for us which this is

actually our breadth first depth framework i just took out all the labels and the axes and made it look like a painting so let's actually go through and understand what this graph is so graphing champion engagement and to be clear you can actually graph a lot of things of engagement with this method so let's go through it breadth versus depth breadth is simply how many players uh could be a champion could be selecting a weapon it could be selecting a squad mate or selecting a map on the x-axis we're just looking at how many people are selecting something engaging with something on the y-axis we're looking at depth this is how much those players engaged so after they've made that selection after they fall into the bucket the yes no bucket of has selected or has not how often do they make that choice how deep do they go into that selection finally you might

say well okay if you have breadth and you have depth on x and y axis how did you get all of those lines well let's use an example you know say we had that champion akali you know prior to her update and she's unpopular she's you know she's down here not that many people play her they don't play her that much and then we update her and she becomes a popular champion well we can track that movement we add a line in there and over time with each of those updates we can see where the champion's breadth and depth go over time so let's now explore those four quadrants because i mentioned unpopular and popular let's cover those two as well as niche and broad and what those mean to us so we're gonna be looking at four champions that over the course of six patches stay within those quadrants so again you see those lines moving on those champions but they relatively stay in the same quadrant over time first one is an unpopular champion this is karthus and karthus is like this

this undead lich uh he's a mage um you know one of his most powerful moments is when he dies in game not the most appealing fantasy so he kind of sits down there at unpopular we also have a lowie now alawi is this really cool character who has pretty unique gameplay she summons these giant tentacles that smack opponents it's very unique gameplay but not necessarily the standard league of legends uh experience for most people so she doesn't have that many people who play her but the people who do play her play her a lot she's a niche champion we also have annie and annie is our bread and butter mage this is the character where you're having a bad day and you're like i need something very simple to play and just win the game you go and play annie and that's why she's a very broad appeal champion a lot of players will go and play her but not a lot of them are going to go super deep and play a bunch of her and then finally we have uh one of my

favorite champions he's a hypebeast champion as i call them or you can call them a popular champion but lee sin and what i mean by hypebeast is that this character is all about style and show this is a playmaker champion very hard to execute some of these cool plays but when you do it looks really cool so a lot of players go and play lee sin and they get a taste for those big plays so they start to practice and play him more so he has high depth to make those big play happen now i told you this wasn't just gonna be about league of legends so blizzard devs and overwatch devs in particular uh please forgive me if i butcher this but i want to now take our breath first depth framework and let's try to apply it to some other games let's predict where some characters would be in other games so this is my fake data this does not actually exist but we're going to try and plot where the overwatch characters are now i briefly looked at third party

sites to get a general sense of where they are and i used some intuition from league of legends but let's see how well we do blizzard devs please correct me afterwards along with the nintendo legal team um so brigitte i put down at unpopular i think she suffered a similar issue to a cauli she uh was very big in professional play and they had to nerf and take a lot of power out of her for niche characters we have symmetra and chorebirn and these two characters their gameplay revolves around placing turrets i actually intuited that hey alawi is similar she places these tentacles that beat people up for her similarly chordburn symmetra they create these turrets that shoot people for them a similar sort of idea for why they exist there for broad appeal characters i chose to soldier 76 very much is your bread and butter shooter just like annie as your bread and butter mage broadly appealing sort of character

fantasies and very accessible and approachable gameplay meanwhile mercy i thought was probably even more broadly appealing that's the character where um you know for those of you unfamiliar with overwatch there's this problem where a lot of people tend to pick the uh the damage characters so you get into a game it's like well i'm tracer well i'm also tracer and i'm genji and someone on the team must be like hey it's cool i'll play mercy i'll support i'll give us healing right mercy is that very accessible healing character and for popular of course it's got to be tracer tracer must be up there blizzard i know it i mean she's the box character she's one of the most aspirational heroic characters in the game and on another end there's also genji and i think genji represents that sort of edgelord hypebeast fantasy i want to outplay everyone so i expect him to be very popular apex legends please respawn devs correct me if i get this wrong but we're gonna we're gonna go through a couple for uh apex legends as well a bit more of a

new example okay for unpopular i think it's caustic and the reason for this is i've looked at some other league champions like singed or um or urgot and i just think the fantasy of like i'm kind of like this like old balding dude who like gasses people is not like and i don't think people like wake up like yeah like that's that's me that's my vision right but hey sorry caustic players still cool if you do like him i also think his gameplay suffers from some issues as well on the other end though i think a really cool uh character in terms of niche gameplay is uh his bloodhound and bloodhound has this cool thing where he goes and stalks people uh very interesting gameplay i think it's something that's really appealing to a lot of people um but those who do play it they go really deep on it uh for broadly appealing i mean of course it's got to be bangalore um very similar to soldier 76 right clearly aspirational theme very accessible gameplay that standard

shooter in the shooter game and lifeline again very uh very appealing uh visuals similar to mercy and has this accessible healing kit in her game um go-to character for the people that want to be a healer you guys already know who's gonna be popular for apex legends it's going to be wraith of course the trihards and the edgelords who want to outplay their opponents are probably playing wraith so i want to now come back to league of legends after having done this little example and let's try and understand uh some decisions that we can make off of this breadth-first depth framework so the first thing is looking at the popularity of marksmen and monsters and it's okay if you can't read the specific labels but that cluster over there in the broad section and the popular section are a lot of our marksman champions and in league of legends these champions all kind of play the same uh part of the issue with them is that when one is weak it typically falls out of favor when one

is strong players go and gravitate towards it a bit in fact the only sort of niche characters are the ones on that far left quinn and kindred who aren't even typically played in the same position as other marksman champions so i would argue that hey maybe there's an opportunity for us i would talk to our designer back what's it look like to create that traditional marksman in the bot lane fantasy that has more niche appeal gameplay that people not all the marksman players are going to play but those who do play it will stick with it uh interestingly enough i think one of the goals of that was or one of the goals for jyn was to be that kind of character it ended up he just was really popular and everyone played him but still i think this could be a cool opportunity for us on the other hand our monsters are not very popular now part of this could be something that other games would struggle with as well in terms of how you view the characters in the game you know league of legends

is this isometric view you're you're looking at the characters all the models character models have to be relatively the same size so imagine you have this monster s character aurelian soul this is a cosmic space dragon he creates galaxies he's this incredibly impressive figure and then you see him in game and he's like you know maybe only like a little bit bigger than annie the little fire mage girl right it's this thing where it's like okay we've promised you this larger than life fantasy but it's harder to deliver it in the game and that i think has caused some issues for monsters but i also think it highlights an opportunity for unpopularity of monsters i think the opportunities hey what's a really broadly appealing monster look like in league of legends we haven't quite hit that yet and i don't think uh we have to necessarily uh be concerned or constrained with character models to hit at we can look at other games and see extremely broadly appealing monsters i don't know pikachu is a great example of this actually so i think both of these a more niche

appeal uh marksman and a more broadly appealing monster are opportunities that we can identify using this data and i want to come back now to this jackson pollock and let's finish this story about a collie so let's add in some labels let's kind of clear away the cup the all the clutter in there let's not have all of the champions highlighted and let's look at a more reasonable time frame okay this is a little bit easier on the eyes but let's apply that akali play rate story to this so we can better understand where she sits today so right here was that last hurrah this is right where she was before we updated her and then you see this giant spike uh up uh up into the right this is more players playing her and those playing her are playing her even more and that's where her update drops okay again we we asked the question what happens when a ninja joins a k-pop group

we can see another sort of spike to the right that occurs here after she settles down and this is when kda comes out uh i have with thanos represented that really dark time when she was extremely weak um very sad time you know it cost us everything but now let's go to where she is today and right there looks actually a lot healthier than the play rate graph that i showed you previously she doesn't have nearly as much breadth she doesn't have tons of players playing her but those that are playing her are actually playing her quite a lot uh and this is looking like a good state for akali now are we done with her is our work finished no there's still much more that we can do for this champion and for all of our champions but this is how we use some of that data to track where these champions were and where they're headed all right let's shift now away from play rate and champion engagement and let's talk about player sentiment

we're now going to present some data from our mass champion survey and the two questions i want us to answer we have two this time is why do assassins hate diving suits and why is happy windman on the other team let's answer those first of all i want to give everyone an overview of the mass champions survey there's two quick slides just to get everyone on the same page of what we mean by this mass champion survey so this survey goes out to hundreds of thousands of our players where they answer questions about champions in our games that we can get an understanding of where those champions rank on certain attributes we also collect a lot of demographic info we collect demographic info rel uh relevant to us in the game stuff like what are the classes that these players like the most in my classes i mean things like do they like tanks do they like assassins do they like mages we also ask about their position their preferred place to play in league of

legends do they like to play in the mid lane do they like to play in the jungle etc in this version of the mass champions survey we only collect data from four of our major regions uh those four are north america brazil uh korea and china i am sorry canada i i swear i put in the flag that had the the canadian split in there i really apologize to all the canadians in the audience uh i messed that up north america also includes canada uh and we also asked things about um you know uh other demographic info such as gender splits for male female non-binary etc and another note is that this is our 2018 data 2019 data is coming soon and we're actually going to update some more stuff and include more demographic splits in our future data including uh we'll be expanding our regions to europe and vietnam now there's sort of

two ways that we can ask questions or have historically asked questions with our mass champion surveys one of them is very standard in surveys it's just like questions so it's rate your agreement something like echo is one of my favorite champions and you'll get a list of options strongly disagree to strongly agree echo happens to be one of my favorites so i'd say yeah okay i strongly agree another survey that we do run that unfortunately we will not have time to cover today but potentially at another talk is one that involves max diff also known as best worst scaling and the way this works is it actually gives you more granular understanding of the rankings of of the various characters you're looking at so it asks something like which of these champions is your most and least favorite okay so something interesting occurs here if you ask me uh to rate my agreement echo is one of my favorite champions i'd say strongly agree

and i'd also say the same thing for a collie but if you place this in front of me what do i put i can't strongly agree for both echo and a colleague so i have to make a choice and i choose that akali is actually my favorite and i also write which is my least favorite which is shaco of these four and that tells us a lot more information than simply like stuff for the sake of this presentation we're going to cover the likert data one of the benefits though of doing the leicard survey is that we get to ask a lot of questions uh almost 30 questions i believe are asked in this survey about everything regarding the champions don't necessarily try to read all this we're only going to be focusing on two today but it's everything from vo personality visuals thematic cohesiveness uh interest in purging purchasing physical merchandise for the champions we look at all number of things but we're only going to focus on two of those questions today one is for players to rate if the champion is one of their favorites

and if the champion is fair to play against now that might seem simple but thanks to our good friend david nevadi we have clever ways of making complicated graphs about this so let's go and look at one of those again and what we're going to try and understand is the perfect assassin so the graph i'm going to show we're going to be looking at rating of champions mapped on the x-axis and the delta from the average of all players on the y-axis give me some time we're going to we'll walk through it all together so first of all we are only highlighting assassin players all those blue dots if you remember from that demographic information all those blue dots are players who self-select into saying yeah i am an assassin player that's my favorite class we have only highlighted those blue players those blue assassin players so what we're looking at then is the

rating a higher rating more to the right means that players like that champion more for the delta from the average if it's higher up on the x-axis that means they like it more relative to the average player the average of all of the classes so what we find with this data is a champion like zed who is this shadow ninja this shadow assassin is all the way to the top right and this actually tells us a couple things so not even just looking at assassin players but looking at all of the classes of players broken out zed is actually the character or zed for assassins is the character furthest to the right what that means is that zed players or assassin players like zed more than any other sort of class mains like champions in their class the other thing it tells us is that it might be the case that not that many people like zed except for assassin players

because he's so far up on the graph he's so much more liked relative to the average of players and indeed when we look for the other zed dots on the graph when we look at support players or tank players we find they don't like zed nearly as much as the assassin players that makes sense though considering he's an assassin we also see that the character that assassin players like the least is mordekaiser because he's the furthest to the left he's the lowest on the x-axis on rating however while he might be the just overall least liked by assassins nautilus is actually the character that is liked the least relative to all the other characters and nautilus makes a lot of sense he's this big hulking dude in a scuba suit um answering our question back there uh and all of his abilities say hey sit down and assassin players don't like to sit down they want to move about and do their naruto runs and do all

their fancy combos and stuff but nautilus is a champion in our game that says hey you don't get to do that he has all of these abilities and gameplay that prevent people from doing that so zed makes assassin players very happy and nautilus makes them very upset i'm sorry for putting that image on there it is definitely my favorite one in this talk though um so you might say okay nathan that's pretty simple i understand that the graph was complicated but you didn't need that complicated graph to tell me that okay assassin players like zed and they don't like a big tanky dude like nautilus well the reason why i want to emphasize this is because of how much player preference is different uh differ by something like class and what if i told you guys that sometimes those differences are even more exaggerated not by your gameplay preferences whether you are someone who likes to be a team shot caller or leader or you like to play damage or you like

to play as a healer but sometimes it actually comes down to just where you're playing what country you're in so let's talk about finally getting to him our good friend yasuo and let's talk about yasuo around the world we're going to be looking at mass champion data for yasuo and just as we explain some of those differences of assassin players and what they like we're going to look at how players view yasuo not based on if they're an assassin player or a marksman player but actually just based on what country they're from so let's start with some memes our first meme is a classic one from north america and it's their yasuo versus our yasuo and the idea here is that their yasuo is like and this guy has put in thousands of hours on yasuo and he's you know they're yasuo savant they're they're just gonna completely dominate your team and your team's yasuo is uh playing the flute with their nose um that's the idea is basically that yasuo

is going to just ruin the game for you no matter what now here's another interesting meme that we found uh this is a word cloud uh of yasuo from multiple regions that was translated and there's one big word that kind of jumps out right in the middle that says science and i was very confused by this at first i and i talked to uh members of our right uh uh riot korea office i was like hey like this this was mistranslated i noticed that science all of the all of the markings for science in this word cloud came from korea why why is that that must be miss translating like oh no yasuo is science i said well please explain yasso a sense like yeah it's a scientific fact when yasuo is in the game you will lose it is it is their version i think a more colorful version of the their yasuo vs arya so it's just science when he's on the other team you're gonna lose when he's on your team you are going to lose but my favorite meme of them all comes from china uh and is this it is

and i really hope google translate did not mess this up but it is uh and i hope i also don't butcher as i say it it's quilo fungnan and what this roughly translates to is happy windman uh in china their nickname for yasuo is happy win-man because when you play yasuo you're happy um and and and that makes me happy uh but that's interesting we have these different memes coming up in different in different regions for this champion what does the data say what's our mass champion survey say about yasuo well let's look at and let's look at those two questions i highlighted let's see what they think about yasuo as a favorite champion and whether or not he's fair to play against so remember these rankings are out of about 140 champions let's see where he ranks okay north america he's 70 he's in the middle okay that's pretty that's pretty serviceable not too concerned brazil pretty similar to north america he ranks

at 66. in korea okay much better korea is definitely like hey he's in our top quartile of champions we're gonna put him at 34. we get to china and he's number two so i looked at this and this is this was pretty crazy but i was like okay i can understand this you know because yaso has actually still played a lot in north america and brazil and korea they might just be a little salty about the their team's yasuo okay i understand that let's look at what people think about him being fair to play against so in north america brazil and korea he ranks very very low and fair to play against um you know in north america and brazil there's only a few champions a couple who are less fair in their minds than yasuo and i was like i i expect korea won't be or sorry china won't be as bad as these but but i'm a yasuo player i love playing him and i recognize that he is very frustrating so i'm like okay i you know china you can't

surprise me here and yet they did 24. so like the difference between these are incredible so we're looking at this mass champion day and we're seeing these giant regional differences in how people play with the champion so let's try and investigate let's go through an exercise and see if we can look in to why that might be the case so let's look actually now just at these two regions just at north america hey i included canada in there by the way and let's look at china and let's see what are some of the favorite champions of these regions and see if we can tease out a story here so we'll notice that there's four marksmen in the top for each region so in north america we have gin lucian jinx and zaya and in china we have kaiser vein ezreal and jinx jinx being the only one that's the top in both um good job to august browning the designer on jinx uh well designed uh anyways we do notice

though that there's a little bit more of like the hypebeast style ultra carry champions in china already with uh vayne and kaiser but nothing too crazy this is where things start to get interesting these three champions were khan thresh and brahm are all support champions and they're some of the top top champions in north america the only support to show up in this top nine by the way it's top nine because top ten doesn't make it look as pretty you'll see later um biggest reason why uh the only character that shows up is thresh and uh thresh is that ultimate playmaker support in our game uh again he's the uh you can consider him the yasuo or lee sin of support we see a smattering of assassins show up we have kane and echo in north america we have zed showing up in china and then lastly our final three in china are lee sin yasuo and jarvan

now for me i i tried to explore a couple ideas right here okay maybe there's a thematic component you know in north america we see actually two of the most liked champions are black lucian and echo and in china two of the most liked characters uh are more are more eastern or more asian we have yasuo and lee sin so maybe it's a thematic thing but that didn't quite you know fully jive up i think north america tends to lean more on thematics for what they like and raid as a popular champion whereas china is looking more at sort of the gameplay looking for that carry potential that ultimate fantasy in league of legends so i actually turned to another question what are the least fair champions in each region and this is where things really blew up for me and this is where i was like ah north america i see i see what you're doing so first of all in north america assassins are not liked we have uh for the least fair champions in north america we have leblanc akali rengar and fizz

for many people who play league of legends just hearing those names is going to get them upset uh in china the only assassin champion that shows up is shaco and i want you guys to hold on to this point shaco of all of these is the least played fizz leblanc akali and renga are much more played than shaco we then go into this other group of champions that show up way more in north america again some of these ultra hyper carry champions like yasuo tryndamere and master yi and the only kind of similar character that shows up to those in china izu dear and again that pattern holds of those four champions yasuo tryndamere and master year some of the most played champions in our game udir is a very very niche picked champion not played that much finally we round out the story with north america with two characters and these don't quite fit

the story as well but it's mundo and zoe now mundo is a less popular champion zoe is a more sort of niche pick champion but yeah also north america just really hates uh zoe she's actually the number one most disliked they really have a lot of ire for that champion which is sad she's so cute and charming um we then look at china again though and immediately we see this trend continue of this low popularity and high dislike when we look at champions like ivern heimerdinger mordecaiser old nunu not the new nunu uh and singe and we rounded out with two champions that aren't unpopular but aren't really that popular uh ergot and draven so what i've sort of understood here is coupling the sentiment data from mass champion and also looking at some of our engagement data like play rate and breadth versus depth we can start to paint a picture about maybe what north american players

find frustrating and chinese players find frustrating it looks like in china the champions that are very infrequently played when they come up are extremely frustrating maybe because they don't see them that much whereas in north america it's some of these more high play rate high damage high agency champions that cause frustration more on this talk later i'm actually going to be doing a much deeper dive into the regional differences and gameplay preferences of league of legends but this is even corroborated by some data outside of this talk that shows that north american players are actually much more sensitive to damage in the game than china north america players think that there's too much damage just people dying too fast in league of legends relative to china but lastly i want to go through an example as we wrap up here today of how we can use this stuff to make a decision how can we take this data and make an informed decision and i also want to emphasize what it means to be data informed versus data driven

so let's ask the question who are the avengers of league of legends and no this is not a spoiler that an end game you know we've partnered with marvel and that yasuo and jinx are going to show up in in the end game movie but let's let's try and imagine the league avengers so i could take that play rate or champion engagement date or even mass champion data and look at the most popular champions the ones who everyone gravitates towards and i might get something like this i might get those edge lords right i might get yasuo and lee sin and zed and i would also probably get some of the more conventionally attractive female characters especially the marksman females who are extremely popular like miss fortune and caitlyn and kaiser and then i could round this out with these sort of aspirational heroes of our game your garan your ezreal and your lucian but there's a problem with this we have a lot of overlap

in the appeal of these characters and so even though we're using one one lens of data we're asking hey what are the most popular champions let's put them in the avengers what if we asked what is the roster that's most appealing to the most number of players well with that question it actually changes how we could think about our avengers so let's do this again let's only take one champion from those three broadly appealing categories let's just take yasuo from our edge lords and misfortune from our you know conventionally attractive females and lucian from our sort of heroic aspirational champions and then instead of garen this sort of big uh juggernaut knight guy who's you know it's it's a trope that's done over in games let's put someone new in there let's put a lowie in there she's this strong juggernaut female let's put her in and let's tie in something to some of the interesting lore in our game the desert region of shurima has not been flushed out in our game let's let's consider putting in xerath and

talia to better understand that and let's not just do all of these humans let's let's get a little bit of our of our of our different sort of looking characters and then let's put in thresh and blitzcrank and finally uh as much as i love ezreal because i too have blonde anime hair uh i'm gonna be honest with you guys i just like echo more i think he's a cooler champion so let's put echo in and all of a sudden i think this league of legends avengers roster is looking a lot better and i think it's more appealing to more players it might not be the most popular champions straight up but it is more appealing at a global level and indeed that was the decision that we went with when mass champion informed some of our character choices for our 2018 season start trailer the climb and you can see you know sort of the the visual explorations we did for champions like lucian miss fortune blitzcrank talia yasuo and

echo for this so to wrap everything up there's always opportunities for you guys to be using your data for new characters new characters updated characters in your game if you take your roster your current existing roster you can make new games with them uh this is dream draven a draven simulator game or draven dating simulator game that uh some of our uh designers made during our hackathon uh we have things like comics merchandise publishing that we can inform with this things like ash uh war mother which is the the comic that we've partnered with marble with or marvel with um and then it also helps us understand how our players engage with these characters players who are cosplaying these characters who are putting you know so much time into learning them and to understand the art behind them and lastly i want to come back to one

other thing i said i was a yasuo maine and i think a lot of this talk people thought oh so it's league of legends it's riot games they're talking about yasuo i want to emphasize that we can understand characters in any game and in any genre i am not just a yasuo main i know that that might surprise people but i'm also a tracer main in overwatch and i know i'm going to get to apex legends and again i think people they think it's going to be wraith but you're wrong it's actually mirage it's my favorite character in apex legends uh pry has something to do with orange jumpsuits and orange goggles i don't know but on that note i want to say this to the designers in the audience think about the ways you could use data to better make decisions about characters you're designing characters you're updating how to make new games with a roster of characters to make sure that it's that they're broadly appealing that there's also niche characters

uh and that they're you know globally resonant and for my insights people my data scientists my researchers my analysts work with your designers with your producers with your artists don't just give them the data and say here's what you should do work with them help them to be data informed think more about the questions that you're trying to answer with this data and at the end of the day make it about the players deliver the best experiences to them when you're data informed thank you guys so much and real quick before q a a special thanks to the gameplay insights squad david nevadi ben steffler and uh coleman palm and also special thanks to insight's leadership jason moore the brainchild of or sort of the the originator of the mass champions survey betsy anderson and david pavlos uh q

a time so we only have a few uh minutes for questions but i will be going uh outside to the lounge just out this way for questions afterwards so go ahead uh oh i guess it's just me uh hello there great talk um i want to ask about the the slide where you talked about monster popularity and you were talking about how it seems like monsters aren't actually that popular um would there be any caveat to how these characters are currently viewed in the balance of the game because when around that time 7.6 or 7.16 a lot of those characters weren't really that strong and i would like you know what would that be compared to a patch like the stone plate patch with gath where he was really everywhere great question actually and uh before i answer that i briefly want to say make sure you guys fill out those surveys uh it's important for gdc it's important for me to get feedback on my talk so please please fill out those uh surveys i think check your email for them the question was

and i'll briefly summarize how can we look at um you know monsters being unpopular if during that time the balance of the game was such that maybe monsters and a lot of those champions were just sort of weak well again that's why we do the breath verse depth over time so when we in that particular graph we only looked at a few patches but a lot of times when we're making balance decisions about this or if we're looking at something and thinking hey here's an opportunity to make something more popular we would look at the play rate over time and not just over a single patch but that's a great question yeah because they do fluctuate based on the sort of meta of league of legends what's strong and what's weak great question thanks hi thanks for the great talk um i have a simple question about the other than the user survey do you use the playlog or data to categorize the

play style like i mean obsession versus powerholder something like that for some of our surveys we do collect player data based on like what they were doing in that game we actually have a tool that allows us to ask any number of specific questions based on the instances of a game they just played in or based on their their player account so it depends on the survey sometimes we will have them self-select various aspects because you know for instance they they could have been playing a game in a different position and that might not be their main one so sometimes we have themselves select sometimes we grab it from the game data as well to draw whatever demographics we need for them uh i was wondering so you've got the chart with the like popularity depth and niche do you do you generally try to design champions aiming at specific points of that graph or do you try and just make the best champion that you can and see where they land is that like where does that go in the process great question i mean generally when we're making new

champions with league of legends we're never gonna be like yeah we want an unpopular champion um but we do actually have more intentional design where sometimes we especially when we make shot calls about where we expect a champion to be we'll say something like hey we know this gameplay fantasy or this thematic fantasy is not going to appeal to a ton of players but we want to deliver that experience really really well for the players who do like that and we'll specifically call it champions hey we expect this champion to be more niche appeal or we expect this champion to be a more sort of broad appeal or sometimes where those things really align we're like yeah we know this character is going to be sort of a knockout silas is an example of a character silas for those of you wondering was a recently designed character who had a long requested ability in league of legends he can steal the abilities of other champions he was one where we were very confident he was going to be a popular champion and we designed him for that yes uh yeah thanks for the talk um

so when you're going in for the the breadth first steps you were talking about how when you made changes to a champion how that actually affected where they were where they landed um do you guys do more with also investigating like item changes or you know rune changes or anything like that to see how that also affects champions and where they land and whether it's more less on the champion and more on other aspects of the game uh absolutely um while it wasn't covered in here we have i have had to do many presentations internally and do a lot of work with my team around how the marksman item uh the items for like the marksman champions like lucian ezreal caitlyn the characters with the long range weapons we updated all the items that they could purchase we've actually done it now twice in the last year and yeah we track a lot of stuff regarding how systemic changes to the game can affect an entire class or a position so yeah we do track those things as well thank you okay uh i'm going to be out in

the lounge back that way uh to chat a little bit it's the outlook just over here so i'll be around thanks guys