League of Legends | Product Lessons from 10 Years of Operations (Paul Bellezza Riot Games)

hi everyone i'm going to talk about some lessons i've learned from working on league of legends from 10 years just to give you some context on myself i'm currently a senior manager of production on ballarat that's just a fancy way of saying i'm a senior producer that manages other senior producers i've spent about 10 years on league of legends in producer and product management roles i started at riot in 2006 as an intern back when the company was a startup um so i've been able to witness the journey of riot and league of legends going from kind of a dream and a hope to know the game it is today so i've mostly spent most of my time working on in-game content and things around in-game content so mostly creative production work but later in my career i was able to transition more into monetization and storefront stuff to to understand um more the business and kpi driven um

concepts of product and production so i'm going to tell you all about quickly about league of legends for those in the audience who don't so i'll do a real quick run through of what that gate what the game is and what it's doing and then i'm going to talk about three lessons quickly i have a bonus lesson if we get done in time i doubt it but also leave room for q a but in contrast to some of the other talks mine's mostly going to be more about the in-game experiences and doing product work around that so just to temper expectations cool so what is league of legends league of legends is a free-to-play pc game launched in october of 2009. um it is a moba it is a game where five players join up with five other players they choose from a roster over of 150 characters at this point all with unique identities abilities and roles to play in the game and they battle against another team of five with champions and roles in order to destroy each other's base um it's a game that rewards skill it's a game that records mastery

it is free to play so you don't pay for power you are you have access to play the game and and earn um skills and accolades by your gameplay and your abilities over time but we sell cosmetics and so it is a cosmetic economy via skins and other accoutrements around the game experience league has grown into a very very competitive product and that there are leagues around the world who play league of legends competitively we've built a robust esports platform where basically similar to the world cup that we do every year where we have regional leagues that compete for us whole season and they all compete against each other in a worldwide tournament at the end of the year that tournament is called worlds the league world's finals and that's all kind of fits into the ecosystem of the league of legends experience competitive players have something to look forward to if they want to commit to being the best of the best in playing that game and could find a path to mastery and even pro play where they can get sponsorships and get a salary playing with a team sponsored but i

won't get too much into that i'm going to talk about what it means to service a global game at scale league of legends is available in 22 countries it's played by at peak 100 million active players around the world it's definitely doing very very healthy after 10 years of launch and what that says to everyone is that you have an international player base each with their own different expectations of what gaming should be what cool looks like what cool and game content is and so a lesson for for me was how to build content for this global audience that has a lot of different needs how do you make something that is interesting to a player in vietnam and in texas and in brazil and wherever that's that's a hard challenge and we as an organization had to learn through trial and error and and gaining skills how to do this well i wouldn't say perfectly because it's never perfect it's always progress so i want to ask you all a question i'll pull the chat up does anyone know who

this character is if they play league of legends miss garen garen hey bob holtzmann cheater it's a former writer so yes there's what would some of you if i ask you who's the hero hero of league of legends is this the person that comes to mind to you you can answer in the chat nope no okay good someone said timo that's actually the right answer all right so in general when people look at like and protagonists and antagonists for a product right this is a typical like western hero but if you asked eastern audiences who they thought the cert among surveys who the hero of league of legends is often the answer would be this guy this is ezreal he's a he's a scrappy young upstart who uses this gauntlet to shoot people in the face i'm not gonna get into the creative too much but the point i'm trying to make is different countries have different standards of what cool looks like what beauty looks like what masculine what feminine looks like and so when you're making content you have to keep those things in mind to go this isn't a one-size-fits-all solution

when you're making in-game content uh you might have to tune appropriately and in order to do that you as content creators have to do your research and understand what resonates well with people across the world there's certain things that are meta trends that all people think are cool and there's certain things that are very nation-specific so i'm going to tell you a couple stories in the content creation land from my time on skins team in particular so this is tryndamere he's one of our earliest champions in league of legends he occupies a role called top lane and that is someone who plays and just beats beats people up on one corner of the map for the most of the game and then helps their team out by the end now tryndamere is a popular character he's pretty easy to learn hard to master but very approachable uh for the most part for for most players and it's he's a pretty decent onboarding character so when we make our skins for characters in league we we definitely balance out which characters get skins over time which which characters are popular characters that we should invest in lots of skins for which characters are less popular characters and we should

do skins for in time in order to to engage and satisfy the audience and make sure we're not playing favorites so tryndamere tryndamere has a lot of skins many many years ago we created this skin this skin was called warring kingdoms tryndamere we actually had a very big skin line that was inspired by chinese mythology and and chinese legendary heroes like um lubu and whatnot we every year we do a series of global events um a lot of them are celebrate different regions and aspects around the world so we usually do a big lunar rebel event around the time of chinese new year which is a big hit for most audiences because a lot of people understand what the sentiment is and cool warriors be cool yo so when we made this skin we as western developers thought this guy looks cool it's a different variation of tryndamere he's inspired by really cool legendary armor and and and you know looks heroic he's in the midst of battle he got an arrow inside of his arm but he's still leading the charge what a

badass we thought to ourselves well it turns out despite all of our thinking that that would actually be cool he was actually not that well received in particular by the eastern markets he did okay he didn't do terribly but he didn't he didn't do what we thought he would do and we went back and asked ourselves why and we started doing more in-depth research with different audience cohorts we realized our view of what cool and heroic look like was not the same as some of our eastern regions they said this guy looks ugly this guy looks scary this guy looks mean and that's not what a legendary hero is that's not how we see it so compare that to a different skin in that same line this is zinzal he's a jungler very popular character as well his warring kingdom skin actually performed a lot better despite it not having as many bells and whistles as um tryndamere over here and that's because a lot of the fantasy that was being

coming to life in that content wasn't heroic but this one was and that did cause some other issues just in general with this image that we promoted the character with because league of legends characters don't ride horses in games so there was some player management issues we had to do with that but for the most part just even the posing the lighting the positioning made a difference in in resonating with the with the target audience here and in general it was a big miss for us so we had to really take a step back in content creation go hey are we doing our diligence are we doing our research to understanding what people think of when they think of aspirational characters and whatnot so later on uh we took the lessons from heart and this was the skin i contributed on directly we were making the skin for a character called lee sin he's a very very popular character and league of legends he's been around for a long time he's a very high skill place champion uh what i mean by that is there's a joke among league of legends players and that is if you have a lease

in on your team he's either a god or he's trash like because he's very very difficult to really pull off a lot of moves with but despite that he's very aspirational a lot of people especially esports pro can do amazing feats with him so a lot of players look to him as someone they want to master over time so when the time came for us to do what we call a legendary skin this is a premium skin tier for for league of legends we as a team decided we need to really do our homework and learn the lessons from previous skin creation so when we were looking at his catalog we we decided you know we really wanted to do something a bit different i mean he's a he's a blind martial artist so there's pretty much a limited canvas of what you can do creatively or so we thought and so when we're looking at his skin lines we were looking at which ones were performing well which ones were not and what what would we need to do to elevate this character and their fantasy beyond just martial arts bro and cool outfit and so what we did as a team was came up with a couple pillars to really push push the thematic and

really push the fantasy that we're trying to go for here and we wanted to make sure we could find an aspirational figure that a lot of people from around the world could get behind and so where we came up with with this was with this thematic was we wanted someone grandiose and fantasize we and we saw that in general high fantasy crafted well tends to perform well if executed well the idea of ultimate martial arts and artists and training that's also an idea that that tends to do well globally and we wanted to make sure that you as a player would have an overwhelming presence and and when you're experiencing playing this character you felt like a badass and if you were playing against someone you would want them to be somewhat intimidated by you although you know it's all just looks it's not going to make you a better player if you still stink at the game so anywho as a team we started doing concepts we started exploring based off those thematics and we went back and forth and asked ourselves is this aspirational does this skin make you feel like a god and all the explorations we were doing was really gearing and trying to pull that out using those learnings

uh up from from what we've done from other scans and at the end we came to a good spread of skins that we could have gone a number of direction ends and so i want to point and if you can see my pointer i want to point that these two examples on the bottom this one in the middle is like a blue lee sin and we as a team mostly western developers were like that one is amazing right he he feels powerful the blue feels almost like fire it feels really fun to play maybe we should do that one let's do it and we actually had a visiting qa member uh who was visiting from one of our regional offices and as he was helping us develop the skin he said you know y'all should take another look at the color palette here because for me being from this country i don't see that as powerful i see that as looking like an alien this one over here this golden skin that one feels interesting and powerful and we ask well why and he goes well in our culture these colors these tones are actually one of of um high prestige and high of fantasy and ornateness and and it begets power

and we're like wow so we consulted that office even more and they gave us more insight into that we said you know that's really good we definitely want this to resonate as far as we can with as many countries around the world let's go with this gold color and we double down on it and i'm not saying you need to go to your regional office or do a b testing to the extent to just pick a color but i think you can be informed by understanding what matters a lot in different countries and how tone and and palette can really make a difference in the experience because they grew up differently than you so this was the end result this was god fizzly sin um we did a really fun marketing campaign with our own channels telling that fantasy of just being this like awoken god badass and it became one of our most best performing skins of all time that year and so we're really pleased with the diligence and a lot of that was done by really spending the time understanding what would resonate globally and we take that approach on all of our skins nowadays and and it helps a lot so the

conclusion of that lesson to me that i'll take to heart is that the details matter a product is more than the sum of its parts it also lets the audience feel like you understand who they are as people especially if you're doing something that is specific to a region at one time we made a bunch of luchador skins thinking it would crush it in latin america it didn't because luchadors are very specifically a niche to a couple regions so you know it doesn't always work out the way you want but it's in but getting the details right do matter and ultimately when execution and resident hit hard you're gonna have a higher chance of success cool lesson two this is gonna be more about competitive gameplay integrity and how that can make a big impact on the engagement of your product and just keeping players in general so i'm gonna talk about how how you play league of legends so for those who know and for those who don't know league is actually has several game modes the main game mode that you see played competitively is what we call summoner's rift that's the 5v5 destroy the base battle but we also have

alternative modes that rotate in and out we have a ram and we now have team fight taxes which is its own game in itself but the way league had been played for many years is that there was normal play where it's where you just play to compete and have a good time but still you know show up and then there's ranked and ranked is sacred and what i mean by saying ranked is sacred is that is where the true test of your skill can be measured against other opponents that's where the ladders are formed to see who can qualify to be an esports pro it is a very very competitive and to be frank intimidating cue to play in and what we learned from league of legends over the years from surveying was that a lot of people are intimidated by playing ranked and so we knew people who play ranked tended to stay if they can make that jump into that style of play so we had to find ways to go well how can we get people in here and so we came up with a product called dynamic queue and the thought here was you as a ranked

player could queue up with people of different rank because a lot of things we learned about league at the time was that the game is more fun when you're playing with friends it's less intimidating so we wanted to encourage people to play with their friends in ranked as much as possible so we devised this system to encourage that and allowed you as a player to specify what role that you wanted to play which is your position in the game mode and then cue up with your friends to do the same and then be match made with people of similar skill sets now this in theory should have been awesome because you're you're getting people to plain ranked you're playing with your friend so that's reducing the barrier of entry and like it gets more people into the pool which should you know increase competition well that was not the case at all in fact it blew up so much from our community because we missed a couple key notes and that's the competitive integrity of the perceived competitive integrity of the queue it got so loud that we actually had to have a roundtable discussion with key community members

to discuss what this would mean to the competitive level of sport and just the competitive level of play and i think the net net problems that the players brought up to us which showed in the engagement numbers was it diminishes the value of individual skill most people were able to play solo and duo ranks to climb up their ladder and then play with people of equal skills when you start putting in people of a variety of ranks throwing them in together into competitive you're not sure if you're winning against people because you're actually matched right or because they have someone who's just way better than the rest of your group it became a real challenge and it tanked some of our perception that year so much for that we actually pulled it back and and made adjustments to the product and and we had to address this head on to the community where we told them look we're not trying to make league of legends casual uh we're trying to make sure it's a it's a a game that offers real competition to two parties with the same experience regardless of of their ranking and because our

long-term vision is to be a global sport uh we need to make sure we offer competitive play for for everyone from solo players to people want to play with the team so because that was getting violated we pulled it back and and made adjustments but um the big lesson we learned is that company is just because the data was saying players have more fun and thus engage more if if they play together didn't necessarily come true in this execution it's not to say that's not right but maybe we went a little overboard there so the net net conclusion for us was that rank play uh is important but you have to do so in a way that doesn't um ruin the competitive integrity and balancing those needs are tough so it's something you have to treat with a real deep and nuanced understanding and try to make sure is serving the needs because in the case of a competitive multiplayer game where skill matters that perception trickles down to all the other cues the normal players the alt players because if those people aren't

advocating for the product well then the product's not going to look as as attractive overall so that was a big lesson for us and i'm going to get to my final lesson and i'm so sorry i have billions of them but i tried to make them snappy and entertaining for you all and that community engagement matters and i put with an asterix and i'm going to tell you about how we do community management on league in general we listen to a lot of player-owned channels such as reddit reddit actually is a very loud voice for for us as developers as i took this this morning you know we have over 4.5 million people actively participating on the reddit with 27.6 k people hanging out just at that moment reddit can be very very loud and earlier in our days as a developer we were like oh my god red is blowing up we need to pivot we need to adjust we learned over time that that's not entirely wise to do we learned that because reddit just like any other channel of information is a highly engaged population it doesn't always represent

the entire pool of players especially around the world reddit in particular is a north american and and western european place where people can share information it's not the only one there's big groups in in korea like um in ben and other ones that that also are allowed so so we use these things as kind of almost like wind chimes to let us know if there's something happening so we had to learn the lesson that you can't just respond to everything the community flares up about but if there is a flare-up there's probably something underneath that needs to be pay attention to and i'm going to tell you the story of a flare-up that i was directly involved with and it revolved around the 2015 world championship finals for league of legends that happened in europe so there's a team for those who may or may not know and this was a big this was a big moment in league history this this was one of a record-breaking years of viewership so at this point the sport has been well established it's growing and growing the stakes are pretty high for viewership the stakes are pretty high for the teams involved and the team that won that year was skt

t1 they were from korea other this was actually their second championship in three years in the middle of this team is a player named faker he's basically the michael jordan of league of legends um he's he's innovated competitive play over the years and is known for being a die-hard fierce competitor so when you're seeing faker and his crew win that's really exciting so why am i telling you all that stuff well every year starting around 2011 we would commemorate the winners of the world championship finals with a set of skins that we would then sell and and um take some percentage of the earnings of those skins and share it with the organization so for a long time this was good enough just taking some of the colors of their uniform and styling our champions sort of to look like esports pro was was fine it was status quo and and people were fine with it for many years until they weren't so when we debuted these six skins after

skt had won its second victory we got this reaction really well against their team so the 2015 world championship skins have been revealed do you like [Music] but hopefully they will make it better so that's basically michael jordan taking a look at his new air jordans and going meh you know that was a big deal to us and it wasn't and i think the reason it was a big deal was one we accepted status quo and didn't try to do anything new we were surprised by it right usually they were they were low effort i would say in relatively low effort things to create uh in general they were pretty cool but we did a couple of things that were key wrong one if you looked at these uniforms here is back to details matter if you look at these uniforms that they actually won the championship with they don't match the skins uniform we

were actually referencing their season uniform from the season leading up to the world championship so players are like why why aren't you even putting them in their colors they won the championship with you'll also notice that they're you we had five skins they had seven players including which is usually two subs on the team one sub didn't play much but one sub did play a lot of games um his name was easyhoon and um he played in some pivotal matches and so when the players and the players of skt and the players of league of legends saw that we were missing one of the subs it felt like a disrespectful moment to the player base and and the honor of those of those esports players so we took them back in and we actually collaborated with the esport we actually redid all of their colors and actually sourced the esport pros to to send us videos of what we would want the recall animations to be in game now we didn't have to do this at all you know it's an extra animation

per character that's a bit costly but we felt this is a moment to do something special this is a moment to do something commemorative and also to correct something that i think we just completely whipped on and so we worked with that team they were very excited to do this and see a little bit of themselves put into the game we redebuted the skins we added in that sixth character and redid the art so that is not significant that's like not cheap right it took a couple months to redo this and and we had to retime it to launch but at the end of the day the reception was great the skins performed very well and it was all because we took that extra time to do it in fact we took that as a moment to go well why don't we make this it's obvious that we're there's a need here from the audience and that is to make these a bigger deal than we perceive them to be this is a moment like winning the super bowl and they ask you uh what are you gonna do after the super bowl go to disneyland or get your championship ring this is a moment that could have been more than just a set of scans that we do to profit share with so from then on out we

started the tradition of after a world championship meeting with the esport team and soliciting ideas for what they want the whole skin line to be and so for 2016 skt happened to win again so we worked with them loosely to kind of come up with a more fantastical thematic that they wanted to uh theme the skins with and we actually added in their coach their coach had time this was the third victory so their coach had coached them to three times and he begged to be in the game as a teemo skin and we didn't want to do that because he actually wasn't a player but we commemorated him as a ward but that extra bit mattered to the community it helped move the needle for the product line itself and has now become a big tradition for us so the conclusion is reddit may flare up it's directionally accurate but you don't need to respond to everything but there are moments when you can choose to do so and make it a real win and and we as product developers need to to look into that and go where do we want to spend that effort in this case it was it was a good lesson and i think

we made a good pivot and we take that stuff a lot more seriously this one is on promises okay so in general we learned a lot of lessons of communicating a lot of information to our player base and the cost of that if you tell a player something's going to get fixed they expect it to be fixed in five minutes so if you show a piece of content without the right context you could really get in trouble we've learned that the hard way so we try not to commit to anything unless we know it's gonna happen and we call that what we call the soon tm promise so in any case there was a time where i was overseeing a iteration of a uh personalized offer system that we have in league and i'm going to show you all what that is right now this is this is what we call your shop and how this works is using data we can analyze the play patterns of player and which champions they play the most games uh with and over

time we'll get a log of of what those patterns are and a couple times a year we run this program where we give them offers at random not random but you know a a set of discounts at a dif as different thresholds to give them discounts on skins that they may like and there's a bit of a random element you might end up getting a nice sail on a skin you really really like and that's really joyous for players this is a very looked after event it happens about four to five times a year and it's always a blast now what the thing is as with all and i'll stop sharing now because i can i can just tell the story here the thing with software like that is it's not always perfect so for years getting that the algorithm to analyze all that player data was very manual and we would basically export this giant data sheet and have to manually insert it into the software app and insert it into the software application and there were

there were lots of moments because it was a very manual process for that to be wrong and what i mean by that is if you if you run it too soon you might lose capturing a week's worth of data and a player might be upset that they didn't fully get covered by by what they were playing so we tried to pull the data one or two days before the offer would go out and then insert it into the software and we're doing it in 22 regions so it becomes a real manual effort and um we actually had a error in one of the data sets for a north american and brazil launch of this product and we put the offers out there and one of our data scientists came back to us and had said things are off so what we did is we turned the offers off and told the player base hey these are wrong we're going to put the right ones in and it turns out for an hour to players in brazil and north america said well i like those offers i really would like them so we being a player focused game company and and liking to make good on things like that myself and

a player support manager said if anyone wants to write in and take a screenshot of what those offers were uh we'll honor that commitment and what happened was 1.3 million player support tickets that had to be answered and we got through it we addressed it we got through that load but making that promise the players without consulting and understanding the impact of how that could uh result in our internal teams was a big lesson so while players felt good it definitely caused a lot of internal pain and we had to do a lot of work to make sure that that system was a lot more robust when we launched those offers um to make good on it for players because that's terrible to be running the wrong offers and two just to make it better on your internal teams in general but so the lesson i learned from that is pause before you make a commitment like that and do all the diligence internally despite it being a good thing that you can do for your audience in the end it turned out okay but it definitely was uh painful for that week

so so there you go that's the bonus round in the first generation of mobile gamers came off consoles from titles like super mario warcraft and league of legends now as we enter the era of mobile 2.0 with three and a half billion devices globally gaming his mainstream easy to play wherever you are and something for everyone the categories have proliferated the best gaming companies in the world need deeper insights a framework to capture the market opportunity and now introducing game iq a revolutionary analytics product that delivers a taxonomy of genres sub-genres and comprehensive feature analysis matched with app annie's legendary market data you have the insights to level up your performance

designed exclusively for the rock stars of mobile the next question from the q a are what do you think are the pillars of an effective survey feedback collection system we have a central insights team and and there's a variety of proprietary tools that we use and we use qualtrics and some other stuff that we integrate um on league which is you know built on ancient ancient code from billion years ago we've had to implement a lot of proprietary stuff but we use something called the player feedback tool that targets different cohorts of players after they do certain things that we can kind of hit them with surveys whenever we want it's a little janky but it works we've been able to to implement stuff like that and then we have a good analyst team who can help parse that information to help the pm's uh make adjustments but it took a lot of investment to get there but it is helpful if you have those people on staff to do that right from a riot perspective are there

specific games that you guys look at like those guys are doing things right or um in general i'd say there are some games that we definitely kind of admire i think we've learned a lot from just being fans of blizzard games and how they've done the live ops and services especially wow you know i think we learned a lot from how they do things um but i'd say i'd say to be honest um we've taken some good learnings from some games in korea you know a lot of the nexon games in particular of what felt right to offer players what type of events to offer players and then in time just evolving league as we got more refined with our hires and with our methodologies internal to know like hey what what moves a needle in certain ways or not like you know it took us a while to get to the point of really understanding you know the different competitive cohorts and versus a normal player cohort versus a casual player cohort and as we kind of built those capabilities we're able to

tune accordingly but um but yeah if you had one book to recommend that influenced your career work the most what would it be yeah i'll talk about it so i'd say if i i had to stack rank it's it's uh plain to win what strategy really means and it's a book written by some people at procter gamble and it goes through defining product strategy as outcomes what are the outcomes you're trying to do and what do you and what it goes into is what is what you're not doing is just as important as what you are doing and that can help you crisp work crisp up where you put priority and put your resources and that was really seminal for me and my development okay folks glad you could catch this presentation so good to have you join us this was actually part of a larger digital conference around best practices in live ops so click the link to basically get the agenda for all of the other talks that were part of this conference and links so that you can watch the other talks as part of that conference also if you haven't checked it out yet

check out appiany's game iq product game iq is a great way to get market data on the mobile gaming market and app annie was the sponsor of this conference and finally if you want to be notified of future talks in virtual future virtual conference events please make sure to click subscribe to the game makers youtube channel and click the bell notification so you can be notified of future events coming up we got a lot of cool stuff coming so stay tuned and great to have you with us catch you later