Tales From a Crowdsourced Game Design

[Music] okay so hello everybody by some kind of quirk of logistics I'm next door to one of my colleagues who just got a chair I'm not happy about this I was kind of feeling insecure so if you don't want to try cheer now here we go yes eat that corner all right to some a few kind of housekeeping tasks please turn off all silent your mobile phones I'm gonna be in the wrap-up room after this inevitably I talk far too much you'll find this as there probably won't be time for questions but just come to the stage and we'll walk to a room 2022 afterwards and we can discuss crowdsourcing game design please make sure you do with you convert your kind of euphoric bliss of this talk into five stars or if you for it bliss is a four star thing for you please do that as well so first of all I am David Osborne lead designer at jagex and jagex are makers of this so we

make runescape which is a PC browser MMO it's been around for 15 years which is astonishing go escape I've been there for 10 years and various kind of roles but I'm currently lead designer so first of all want to kind of just reassure you that just because we are a 15 year MMO engaged and long-term player based doesn't mean that this talk will be an outlier or it's as an example is an outlier two of them I won't get you don't guys thing in that don't worry it's not now at this table has been crafted just so it's just as applicable to throw an MMO to a mobile game to a board game and why not local politics as well I've got the delusions of grandeur for this talk the only things so the worst parts of most talk is almost talks is the background just to let you know that the only thing that's worth or needed to know is that we have two branches of our game one is runescape main game has been around for 15 years the other is old-school which is a kind of a snapshot of our game from 2007 this is for those people who story as our heyday or our golden era and so people can play this

game and and I would talk about in a moment about how they poll and the interesting things we do about polling first of all so tells from a crowdsource game design and it was pointed out to me that maybe people might miss miss here that will miss that to be crowdsourcing and getting content I'm not not I'm not Media Molecule so that will be not being the talk this is all about relinquishing your design control and handing it to the players what happens the various pitfalls and various mountains of success who all climb as you do that and I'm going to talk about various ways that we have tried to relinquish design control this is the talk that I really hoped would be around was around three years ago we've been doing this for three years and I really wish this talk was around to kind of inform us okay this is really daunting side this is various examples of things that we have done to get the players involved in game design not gonna talk about all of these as I said this is - 25 minutes we can see those kind of kind of moving down in terms of how much design control we relinquish you are blaze you likely do a number of these yourselves involving player feedback or I really want to talk about

is these three the first one publishing design documents about what happens when you bring the players into your design feedback stage we shall have an policies about what happens when you bring your players into the design decision making stage and runelabs is about what happens when you fully hand over your design and give the design to the players they're able to create their own designs and put into the game okay so as I as I mentioned we have two splinters of the game old school and runescape old school does exhaust their polling this is kind of Constitution the Constitution between the developers and the players then nothing will go into the game no feedback even no substantive content will go into the game without the players say so everything is polled so with us this is 75% is required for a content to go into the game or even polled it whether or not we wanted to reduce that to 70% our players said no and 40% of people said yes this generates a pool of content to go into the game and once that Paul is kind of

depleted then we continue the cycle again creating more content they'll going to the game what this does is creates a defensive stockade around your game it costs the players as protectors the other pole system we use is for a runescape game and this is occasional polling well this this is more of an engagement tool we target those projects that are best suited to polling and having that kind of spotlight thrown on it bringing it the attention that Pollan gives them we don't have our 7/5 majority here it's kind of a tool that we can use to dictate the kind of make up of the content or producing Commission level stuff okay so just to cover that source the appalling cosplayers to protectors occasional polling more of an engaging tool that builds investment in the content so for each of these really just just in case you are interested in trying these with your games or your upcoming game it's going to go over some good practice first and the good practice that I would recommend it is definitely put your pole system in game your pole content it's good it's going to be creating context they'll be changing content you need to

bring it to your players who are exploring and are taking part in that content otherwise you're going to be putting out on social media on your website and only engaging with those through with most socially mobile most socially engaged and plus majority of players are likely in game anyway at least I hope they're playing your game show visible real-time results so this is obviously something that polling doesn't necessarily do much but there is nothing more mobilizing in terms of polling than a visible public it is close the poll is closed and two options are close together but nothing mobilizes the player base more allow a change of mind if players are aware that they can change their mind at any point they're going to talk to each other they're going to try to change these other each other's mind the chatter will increase the conversation will increase and we like to understand what loads a question players will know that we are trying they will assume that we are trying to load a question and we know that reward does that more than anything else if one option promises a reward while another one doesn't they will assume that that second option doesn't come

with a reward they're better off voting for the first always understand what motivates player and that pot that's most often their own self-interest rather brevity is number another good one so if you can summarize something in a few words an elevator pitch that will always win out with something that has to be partnered with a design document understand what brevity does to your poem we've also come to terms with player demographics with our poll system if we are polling a combat update against the quest update we start hitting problems I'll quest group request demographic is among our minority in comparison to combat doesn't have enough people so every time we poll a combat poll will or a combat option will always win over a story option we've learnt not to play player types against each other instead we have something like a combat pole we have a story problem and those do you never meet okay so what floors have we seen from poles god forbid the moment you actually put something to the game that is negative or controversial that was not told the immediate reaction you will receive is this should have been polled

we could have stopped this we were the protector of the game what are you doing you start getting hearing that repeatedly it's like in quite stark but the problem with that is players do not instinctively value or understand everything that goes into your game they do not necessarily value your tutorial or your funnel experience they do not necessarily value thee the increasing price of items need for gold sinks perhaps they don't value your kind of monetization strategies this is something that might be critical in a deal breaker when it comes to putting pole systems into your game and allowing players to have input on your design on your decision making process the way we've dealt with it is education color treated for our players as adults and brought that brought them in so we have we bring in kind of influences we talk to them first we put out dev blogs beforehand before a poll goes live we try to talk about what is important to apply frame it and what is important to player if more people are coming into the game there's more people to play with in their multiplayer content if

you've got a good funnel then their friends are more likely to stick around and play with them if there's monetization then it's likely the game will last longer there's a small details here that poles are also relatively expensive a poll will have to finish so you've got to the best I would say roughly two weeks is good for a poll to finish you've got build that into your process and also it may be that your team will perceive a reduction in creativity by handing decision-making to the players I always argue this because as long as the team is happy with every single option in a poll and they definitely should be then there is no reduction in creativity every option is ago--on and a public benefits for us have been huge instead of that kind of sine wave of discussion when it comes to our content where something is released people are talking but then that talking just kind of breaks off and no longer over the course of the week instead we get that Kahn long periods of chatter they're discussing what each option might they're speculating about what it will be like in the game they're trying to mobilize each other they're trying to

push people to other options there are champions of individual options we've got people talking about democratic process you know people lording first-past-the-post systems it's bizarre and what you see is this increased retention because then they want to stick around to see this content that they've told you want to see what it's going to be like when it's realized we see a spike in kind of active users on launch as players if I guess it's that mentality if you've read a book you want to see the movie you've kind of seen the part you've seen the poll what's it gonna be like what it's come out and we're starting to see some cultural changes players are becoming are seeing themselves as protectors there on the wall kind of making sure there are no wholesale gain devaluing changes and there were their friends they would bring their friends onto that wall to protect the game - and they're worried about ever leaving the wall if they do then the White Walkers are just going to storm in and ruin everything players feel safer and there's a lot of value to be put in that often there is this kind of inherent fear or background

fear that a game could change the car pick reports one to leave them at any point as protectors they know that that will not happen and that kind of makes them feel freer and my greatest benefit from polling is that players were more of aware of each other beforehand was very common to see players saying we don't want this we don't want this update what are you doing I've talked to all my friends none of the players want this what we starting to see is that they were reinforced by their own social cabal they were talking to each other there's lots of like-minded people in most battles and there were just reinforced that hey this is the opinion of the entire game and when they come to the poll system they realize that is not true they are now aware of different player types they're aware of Bartle even which is great and obviously they're kind of brought to the developers and the players closer together we're not in our brute ivory tower making average arbitrary decisions so if you go if you're thinking about bring the players into a decision-making process understand that tension the players will want to vote on everything absolutely everything but they do not value everything and there are things

that will wrap it will benefit the long-term of your game that you'll want to do but your players will say no - so understand how are you going to achieve that it might be through edge it might be through a constitution between you and the player that you're not going to pull everything but whether it comes that benefit for strong engagement and retention and a cultural shift with your player base now the next a kind of way of getting players involved in crowd sourcing your game design is getting them involved in feedback the feedback stage this one kind of came about by necessity we were polling something and then seven eight or nine months on from that they would get their game content that's a huge amount of time I mean I lose the plot of Sons of Anarchy between each episode and that's weekly you can't imagine somebody sticking around for nine months and remaining invested because polls are an investment so we tried various ways of kind of bring them into the process being more transparent so dev Blanc's things like that but live streams but really what we want to do is involve the player in the feedback process and the best way of doing that we found was publishing our design documents we would

release a design document one day over the course of the week each day we will post patch notes for that design document reacting to the feedback they're going to know what we're incorporating and then at the end of the week we would rerelease that design document that design document would feel very different from one we first put out now obviously you know good design doctor processes but many of you had to add designers or developers but when it comes to pushing one out to the players there are a lot things to keep in mind please do to update wisely I'm going to find out very soon the cost of producing a design document for the players is huge its massive so doing it for Nisha updates you get disproportional success you will it will not be worth it so choose those updates that are broad that are universal and you have to be so hugely confident in your update if you're putting out a design document this shabby and your players will begin to feel that you just don't care that you just stopped putting in the effort be really confidently in the update don't treat it treat this design document as a beta this should be the most finished design document you have

ever put out everybody should have signed off on it it would need to be kind of presented in the flow of user experience describe it in the way that the player interacts with your content they will be moving through geographically chronologically through your content tell them the story of growing through your content this is the language of their game experience so you're going to have to completely refactor design document to that that and reasoning there's a lot of things in that design document that will feel strange without the context of your acceptance criteria you'd like can you put those already created your core aims at acceptance criteria knowing put them at the top players will start to understand that your decisions not arbitrary and also understand the level of detail if you're putting in drop rates XP rates monetization details ninety percent of your feedback will fixate on those details it might be the one was overpowered or underpowered or maybe there was an error understand that players will pick this apart and will not give you feedback on the rest of it it might be that you want that maybe you're so confident in the efficacy of

your the rest of your design and really break it video and text we didn't see any cannibalization between the different media it means you get it out there so the flaws it's a huge intensity of work putting it together doing the design process at the published design process was a hundred and twenty hours of designer work and understanding it delays development your your design phase is going to swallow up probably another two weeks of work build that in may be a cost for many of you you work a Johnny maybe you don't even do a design document maybe it's kind of iterative in the crater they of course the design may be your design documents more visual this may be a deal-breaker for you if you don't put it out longer more text oriented design documents it may be that the work to convert is too much and there's a potential negative impact on your current content content understand if this is an ongoing game that if you're putting out something a design for something that improves on something is this thing in the game they may not play that content anymore may be manipulating your economy by putting this information out there understand

the effect on your current game content and do not expect wide-ranging feedback from your player base so players as we've seen from the kind of the legions the forum but that the hundreds and thousands of forum posts and social media feedback that we've received on our publish design documents is that players do not give creative feedback they do not offer alternatives I don't often not very constructive that's fine we're going to that in a minute and they do not have a full deck of lenses they won't understand tutorial ization kind of narrative grounding they won't understand a lot of these things so I understand what the players will give you but designed our benefits there's more participation you think for our elf city design documents which were the first ones we put out one-third of our player base read those design documents fewer than one-third could actually play on launch players were really interested in it they really wanted to get involved with it and one of their feedback is limited to certain things is hugely valuable it comes in the context of I will play this or I will not play this

all that is limited that is exact that's ultimately what you're looking for from your content you want to know if the people who are buying your game and play the game and players have begun to understand or get a better sense of the realistic kind of what happens when you were designing content a developing content font it doesn't just appear in the game they often felt like we just treated our community as a beta that we would just throw content out there and because they only saw nerves or improvements they believed that we just treated the whole of the gamers as a as a beta phase now understand how long it takes to create something they've more belief and faith in us as a designers that's very valuable and there's a huge change to release they this is something we didn't expect release days awesome players were often kind of on other content that didn't have designed documents published there was an expectation gap they kind of they got on the hike train and believed it to be something that it wasn't extrapolated far further and when we

launched there was kind of an ER this isn't quite what I wanted that spin down smaller gap it's reduced orientation they don't need fewer players new tutorial ization orientation they're going straight to the content that you want to get to again that's valuable to your player base its reduce the chance of post launch issues the number of times that we have put an update into our game has been completely derailed by one thing that needs to be nerfed or powered up with a little bit more some fix that has to be made and all the conversation again has fixated on that it directly directs conversation where you want it and in our case and something that's very valuable I would encourage you to try this it's put don't put everything in your design document try putting little Easter eggs into your content the players will search out on launch day and let them know that they're there so they can go find them we found a kind of great great value in that and that meant that not everything was in that design document was still something to find so takeaways expensive and time-consuming only suits a limited range of projects but as a tall it does throw a spotlight on your content really does improve that relationship and does

make you really stay kind of stress-free okay so the next one so what happens when you do a fully crowdsource game design we came up with something bespoke called runelabs and kind of see that the process across that across the bottom there and we were feeling the glow of the design documents the players loved the design documents and we thought why the net natural extension was to bring them into making designs for our game it also meant that we could kind of push push push through and talk to press press might be interesting Tina came that was kind of well designed by our players so we start by players generating an idea and adding that to our database we now this is effectively a database players then upvote those designs that they really like and then we take the top top five top four top three whatever we have greed for that for that month and those are then put in game as a poem people in game and choose the update they want to go in there's a little example of some of the ideas we've got I really don't know what 101

things to do with this thick of bamboo is I probably don't want to know but what what we do is um once that putt that winning option is chosen the player is involved they are kind of they go to effectively spread reviews they sign off on concepts they're kind of involved as much as we can D and as long as their name isn't a my little poke my little pony reference or just the number of digits we tried to put them in game to give that kind of credit to them so we tried this for a year and one third of our game content was done in this way was generated as player designs just quickly go over some good practice the first two we learned through our own mistakes bring runelabs in game so bring your your system for generating or and upvoting player ideas in game we didn't do this and what we found was player ideas are not generated from having a book by their side of their bed for dream ideas or anything like this they're not mala nearing just randomly they their ideas are generated by playing the game they come across

something in the game and they have an idea for it that was largely where all of Claire ideas came from talking a bit more about that in a moment so if the remap system is in game it can do that ensure a fair and dynamic trading act attending algorithmic algorithms obviously in bit you know iTunes in the other systems suffer from this we found that good ideas we're not necessarily rising to the top because I tend to get trending algorithms we're not strong and often setting criteria for the players limiting the space for the idea generated more we found that just saying give us any old idea would give us 50 percent less in terms of ideas fewer ideas then if we said create a quest with flying bears so the floors all actually I might just skip that quickly and so we did come a contingency work to try to figure out what we thought the problems would be for REME labs we sat down and discussed that we tried to plan for them and our worries were that would be where that the ideas would be unworkable they would be massive that we massively overpowered would create power creep the ideas that

was simply an unworkable we thought those were the ideas that would come from the user base we thought that will be culling so many of them the players would kind of be disengaged from the whole process because none of the kind of the top ideas were getting into the game that wasn't the case in fact players were low-balling they really wanted their content in the game so they went smaller than the criteria that we set them they really wanted that content in game we would completely so blown away by that the problems with runelabs were that there was soot there was hugely limited engagement in terms of submitting only one in 100 of our player base ever submitted an idea so he started trying to drill down into why that was the case we the first one the players tattle told us was there a number of obstacles to putting an idea into the game was too little too large they had to create an idea was to come up with an idea write it post it wait for feedback and wait to see the numbers and they got polled number of obstacles were too large they also were kind of running riding naked through the pack the court of public opinion they were worried

about that they're really worried about being kind of humiliated by their friends but once he we drilled down we could find exactly why players were not interested in submitting their own ideas the reason was they were perfectly happy with a contract between developer and consumer he liked being given than content and then playing it they weren't interested in expanding that contract anymore thank you very much they were perfectly happy with being delivered content and that was kind of the core of firebase we also found that an up voting some some options were win by only having 100 votes that meant 1/100 of a percent of our player base were determining determining content for 1/3 of our updates that was Bartley disproportional and we also found it appealed to a specific subset of our game base so the people who were interested in submitting ideas were largely those who are already socially quite mobile though communicating on social media there are two reasons for that it makes sense that

those people who are willing to talk and spend a lot of time outside of the game talking to each other also willing to kind of write a lot of stuff and relapse they also had an inbuilt base of hundreds of fans in terms of their Twitter followers reddit for is and so they were because there were so few up votes there immediately rising up to the top and that created a cult of celebrity whereby the topmost voters all of those people who are the most kind of active on social media and when we start breaking down the ideas there are only very specific ideas that were coming from our player base and it's worth noting this when you if you think about kind of generating player designs fixes or improvements were predominant in all of the of in all of the player ideas they were hitting their heads against walls or ceilings in-game content and wanting to remove those walls and ceilings like I can't defeat this base because there is no good not a good enough sword to defeat it I want that sword everything was a reaction against the ceiling of warm the other one was reacting to good content in game they like certain things they want more of

that our first winning puddle was give me two more dragons there was no more text than the design they wanted two more dragons so we gave them in and that's just a sequel to more content and the other one is fanfiction we found the players had stories they wanted to tell characters they wanted to bring in game there was no game mechanics in that design they simply wanted to have that character in game but there are there were benefits not all doom and gloom we actually understand our players more and better from from doing this we now know what they value or don't value because the vast proportion worth fixes we actually found that plays valid things smaller changes to the game quality of life improvements we didn't see them as valuable there's something big a quest perhaps we actually increased headcount within our ninja team which is the team that makes smaller updates to the game because of runelabs we also had several ideas we thought they liked that we've previously mentioned in other kind of live stream stuff like that they didn't value them they never rose through the top we soon learned to kind of disregard lasers chaff and we know what ideas players now

generate we now can go to them for fixes and improvements and sequels we know that they're a solid producer of these things and great for us is there's no more we could have done that better there was a lot of kind of conversation before runelabs that why you put this update up we could have done that better they now understand a value but the design work that we do so my takeaways from relapse are bringing players into the crowd sourcing of game design is that the core of your player base or potential player base is likely not interested in designing their game and that creates a tension that you may have to understand that small minority will be creating content for a majority that doesn't want it our story players we're actually almost think engaged with runelabs that make sense they read a lot of text codices and journals in game but be happy with writing a lot of text on relapse what are a minority when it came to the polls with rune apps though is suddenly the majority so virtually every update that came out with relapse was story oriented you're creating content and majority may not want but valuable

ideas were generated particularly with fixes and maybe look to harness those ideas maybe I by other means so I lied when I said I wouldn't go into more of the crowdsourcing methods but very quickly I'd like to bring you in to release schedule survey this is where we've ended it up effectively our conclusion try to take all of the best parts of the various different methods we come across this so what it is as we approach our idear generators for ideas design ideas we then pull them together with developer ideas we have a kind of open ideas session within our office and we also pull those with the ideas that generated my kind of data and the needs of the game holistic needs of the game we then create a survey Oscar players to rate each update how and then they get to play a game they're creating their own release schedule for that year so they take their 12 updates form put them in two months and they form their own release schedule then we then reveal the results and this is all other kind of big event what I value about this is that what it kind of games takes from the benefits of each individual and loses some of the negatives from relapse it harnesses off creative players make sure there's a

broad range of ideas because we are filling in the gaps with the design documents we understand the players expectations so what that means is because we're putting all of these updates out there we now know what they think of them we know what they value but we lose that cost of the design document being have such such high overheads we're not designing everything and design putting out there in a player design document with polls you get the increasing judgment player impact so so do we get with a really schedule serving but it's a more malleable contract with the players our contract is you get to help us determine the major updates we're going to keep all of the small fixes to ourselves so takeaways for this talk we want you to understand the crowdsourcing game design from our experience a limited number of players out there are interested in developing ideas on creating ideas and there was a limited shape of those ideas understand we will not get a full broad range of ideas from our players probably done could there be a game that successfully incorporates all of their players and to play a design perhaps I think it built

game was built and promoted around that concept it could well happen but for us we were trying to solve a problem that wasn't there and we're going to be pulling back from runelabs and making a moving on to that written scheduled survey but there is such value in bringing players into your design process there's more conversation as I said that sine wave of chatter to change to a constant we're seeing players talking all the more and we know that when players are talking they're more engaged and if they're more engaged the more retained we actually see kind of a link between dialogue talking in game and being retained and you have a focus release David isn't you have not side wounded by unnecessary unnecessary elements and the cultural shift between a red ship with our player base and us is huge our players now understand their own community they understand their different player types they understand the difficulties of design and I look at where we were before and we did everything here and we're all about that whole hell about relationship with the players it's massively changed you'll know whether your game of your community would benefit from incorporating players into

a design but the cost for us it's been worth it and we have a healthier game for it so thank you very much as I predicted there is no time for questions but please come up to the front and I wouldn't have a chat we'll go over to 2022 to discuss more about crowdsourcing game design thank you very much [Applause]