Implementing Pixar's Brain Trust Model at King and How It All Went Wrong

[Music] all right okay yeah this is us i'm steven jarrett vp game design this is bob woodburn director of design operations and i think we have the longest title which is implementing pixar's brain trust model at king how it all went wrong and how we may have finally got it right so first give you a bit of a background about king you know we're from king these are a bunch of our franchises and brands that you probably know of but we're most famous for candy crush obviously so we got a little bit of a quiz candy crush has been played how many game rounds do you think candy crush has been played you can just shout out a number how many two billion one trillion game rounds actually and time spent in the game how if you added up all the time spent in the game how long do you think it could be ten years over five million years so that is actually the time

that uh humans and apes separated from a common ancestor and then uh and we have data people that obviously come up with this all the time so for us so wall bounces if you added up all the wall bounces for bubble witch you got 384 000 kilometers and that's 24 times from earth to the moon that's how much that was so the reason i'm talking about this is because as we all know making games is hard and making a great game is very very hard and we want to make these really successful great games and so because of that we're looking at other studios processes that we're doing it right and pixar's brain trusts we read creativity inc and pixar's brain trust was the one that we thought okay we could we can make this work right and if you don't know what the brain trust was it's where this uh this group of like high-end directors like john lasseter and a bunch of others andrew stanton and they meet together and every few months to assess the kind of movie that they're making

it's like very open it's like uh you know like they're very honest with each other and i think it started around about toy story and they claimed that it's made them have lots of really great hits so we did a lot of research about it and we thought it sounded you know this sounds really really great you know we have a lot of very experienced passionate people at king we have these amazing values at king like karen kraft creative champions humble and open fast and fluid fun and friendly and we thought we could make this uh make this work so we called ourselves the creative champions and we uh had this logo which was always good and we had a mission and a mission was to help studios and we could be called up at a moment an idea is conceived and uh we would work from the beginning of the game all the way to the end of the game and we had this experience like a team of tech game design ux ui our is some producer and then like the brain trust which is puts smart passionate people in a room together and charge them with

identifying and solving problems and encourage them to be candid we had a different problem with pixar because we had a studios all over the place so we would visit studios regularly and then we would do these workshop and deep dives with the teams and uh with the producers and and the game teams and we would help solve these issues all together in a in a candid way just like and it felt great great we can do this but trouble was brewing one of the biggest problems we had is the word we felt the word champion like creative champions was great because it was like we're going to champion a process right we're going to champion the process champion your teams but a lot of studios a lot of our studios uh english is not their first language and so champion kind of meant winner right and and they were kind of thinking who are these guys right that's calling themselves the champions and they're like these experts and are going to come and help us and this was a typical team member

that we used to come and visit they would like freak out and go the creative champions are coming this is a gate everyone worked like crazy to get the game ready they have the power to shut our game down they are the angels of death we were actually called the angels of death mainly by this guy but other people as well and uh and it was very confusing for the creative champions as well they they didn't even know how to define success right they uh they didn't know how to set goals how do you set a meaningful goal when you're just going to a team and then just giving them feedback and how did we show that we were actually adding value to that team as well and why won't a team listen and that was like a real big problem right because you go there and you give you you're working together with a team and you're discussing problems and you feel that you're in putting this like knowledge to them and stuff and then they just don't listen and they go off and do something else and then we were used uh by like the boss boss peeps to be

sent in to like when a team was in trouble it was always like sending the creative champions they're uh they helped the team release so we kind of went from this a team solving problems here to help to looking for problems and don't disappoint me which was complete opposite and uh it was kind of like this and so we had this expectation that we were support but we were actually a review team and uh and so it wasn't working well until so one year later we thought okay this is terrible this is not going well so we thought okay we'll redo it we'll call it creative champions version two yes and we still called it creative champions and uh we decided to okay we can solve some of these issues right about and we did it from the creative champions point of view and not from the team's point of view so we uh we we had about five or six champions and then we assigned different game teams to that champion we thought okay this is really great

because you know like uh like uh sally who's a champion she can like work with these uh teams and she's part of the team but she's not like totally part of the team so she can like review stuff and see it from different eyes and that kind of stuff so we thought that was pretty cool but the team started getting design feedback from multiple places so a team normally gets feedback from game design and producer then from this creative champion that was given this ad hoc feedback every time they played a build or they were reading documentations and giving like ad-hoc feedback or they were providing workshops and giving guidance or they attended this like scrum thing like every two weeks and give feedback then you would have the studio design director that would also give feedback to the team and then you would have the header studio and they would give feedback through the studio design director and actually through the producer and then you have myself the vp of game designer who would sometimes go through header studios sometimes we should do studio design director and then some fire from assigned creative champion and then you

had the cco that was just like bam just like everywhere so the team was like ah so it's like this broken system right so who does the team listen to so a year was like into this as well and then we completely said okay this is not working again but like we're super fast and fluid at king so as soon as we find that something isn't working and it was the actual creative champions that actually said this is not working we this is like a disaster so we uh decided to completely redo it again so first we did an analysis of what is working so when requests actually come from the team or the studio design directors or the header studio that was really great because they were actually asking for uh information and asking for help so requests for workshops so they might say we need a social workshop or we need a workshop around level design or something like that that worked really great and value was seen with helping teams get to greenlight the champions actually

did help a lot of teams like solve a lot of problems and actually got them through pre-production and to green light and we do these things called fqas which bob talk about later which is where it's called a fun quality analysis where we actually the team come up with uh something for us to review and then we review it and that was seen great as well and like i said we did actually increase uh quality and problematic titles but what was not working is when feedback was not requested by a team when multiple sources of feedback with conflicting guidance and who should the team listen to and when teams were incredibly stressed to get a visit from the champions so the first proposal was completely remove uh the creative champions process right and everyone that was a creative champion just ended up working on different projects stop trying to copy another studios process and actually create something better something that actually fit our values something that actually felt right for king and bob's gonna go into that

so at this point we still the same problem we still need to come up with a way to support games uh support teams and also to review so the first thing we did is decide to unpack the two into two separate things we'll we'll support on one hand and review the other but they won't be tied together but we did find though is even though that we were going to unpack them they were essentially some common threads that went through it first of all the teams need to trust the reviewers which wasn't happening previously the feedback needs to actually be good and the teams need to want the feedback so to take a look at the first bit getting teams to trust him we started with the creative champions which was the exact opposite of being trusted and fully honest um and got rid of that so we need to replace it with something and we decided to look in the wider king community of game designers um and involve everybody that we could in this process so we have a game design leadership team

which is some of the most fantastically talented people i've ever worked with they're all across the globe they're extremely experienced and and we've worked really hard over the last couple of years to make sure they're connected with each other we do this by every three months we make sure that we get everybody in the same place so they can actually spend time talk through issues wear stupid viking helmets and go for beers of course steve helps us out as well there and this group of people and not just them but the wider community of designers at king have a fantastic amount of uh experience um these are the games that we've worked on i think these are some of the most highly performing games that have ever happened in mobile um but if you look a bit deeper this group of people has worked on an amazing amazing uh library of titles some of the things from when i was a kid some of the top performings of the day so what we did is we went designer by designer through the entire company and

catalogued what their experiences were what they were good at were they were strengths with genres they know were they great at balancing or great at narrative and we created this list of who was great at what so this is i mean it's not particularly our process but at the same time what we're doing is looking at our overall operating model and how we make games and one of the decisions was to make a more light but decisive governance in the production process and if you look in the play test age that's where this came in we call it senior creative review and the principle behind that is to make sure that the game would be reviewed by experts when it comes to design rather than just a random association of uh senior management um we'd remove the feedback from any sort of formal gate i don't know if how many of you probably most of you have been through some sort of gate and the amount that people are actually interested in getting feedback when you're getting a decision uh is usually next to nil especially if you get a positive

response i mean when people are told yes they don't actually go well why are you saying yes they just go okay great move on um and we want to make sure that we're leveraging this creative talent from across king so this is kind of a snapshot of what the average ser would look like it's a combination of qualitative data and qualitative feedback about the design and also we give numerical assessments of each aspect of the game so we look at core gameplay we look at envelope social the theme audience fit and then we also discuss the reviewer's overall confidence level and how well they think that game is going to proceed excuse me how well they think that game is actually going to do and achieve the goals they're set up uh so reviewers are giving a week to go through play through the build read documentation uh they fill out a google survey that's how we capture the data um and then ultimately report back to the game and that worked that worked perfectly actually it was super super well received by the team i'm just kidding it was we screwed that up as well

yeah we made a huge amount of mistakes and had to go back to the drawing board and come up with something better uh right from the beginning we noticed that we were getting a lot of reviews saying that you know i don't think this this game is going to achieve its goals and the reviewers weren't actually aware of what those goals were so we started putting in a kickoff meeting to make sure that the reviewers and the team was aligned and i think that was after the second or third one when we did so it was an iterative process we did work to keep making it better and then ultimately what we had to do is go out and talk to all the game teams we held workshops with uh not just designers because one of the problems we made from the beginning is we knew what management wanted and we knew what the designers were capable of reviewing but we didn't really talk to the game designer sorry the game teams in detail to make sure we're doing the right thing so ultimately we made a huge amount of changes what we did is is make sure that that feedback was a lot more clear what we were handing people was about 42

pages of random thoughts so we found a way to actually concisely summarize that and take the the biggest takeaways that a team can go and present that back and we launched that this year as far as the support side which is the the second part of the equation what we did is we created a really really robust menu i guess of different feedback channels that are available for the design teams the fqa is one of the ones that steven mentioned earlier this is is similar to the to the other process in that it's a written documentation it's fairly formal um but there's no evaluation it's there strictly to the support of the team they can ask whatever they want it could be a small issue like is this feature right or it could be a big one like is this game fun and we provide that and put it back to the teams we have another one called play dates which is essentially designed for early stage games that are iterating quickly and they don't really have time to wait a week because by the time that comes in you're basically reviewing a game that doesn't exist anymore so this is we get a load of designers

together they all get on video conference play through the game and the team can ask them in questions in real time we also run workshops both face to face and virtual so the face-to-face ones is when they're a serious issue with the game we get everyone together in the same room um to work through a problem and the virtual one is the same same basic thing we just do it over email so for example teams want suggestions on blockers that they can use in their game we get some of the designers who have strengths in that area across king they put together a few slides and send it through [Music] and we are still iterating in these we're coming up with new ideas all the time in the uh for the future i don't know why i put a robot there having a coffee with a lady i think there's something to do with future and coffee but concept coffees are something that we're working on where people can discuss these early ideas so ultimately whenever we do create one of these new channels we go back to the three kind of core principles that we look for in all of our our our feedback channels which is that you

need to trust the reviewer the it has to be quality feedback and the feedback needs to be wanted as far as actually met that is like the ugliest slide i think i've ever done um as far as actually measuring how this is successful it's a little tricky because we've only been doing this about i guess a year and a half at this point uh so a lot of the games that go through this from the beginning have not actually reached launch yet uh but one of the things we wanted to do was it was to make sure that our pipeline looked a little bit more uh healthy and back in summer 2016 you see look we had a huge amount of games that were in late rounds of production either in play test or prototype or sorry prototyper production or play test and very few that were in prototype or concept stage and when we're actually able to implement this and get people to really think about is this game going to be big can we have a huge huge bar for quality um and by introducing these various channels that people can do if you look a year later it looks a lot healthier we get a lot more bets in

in the early stage a lot more things we're working on we're failing quicker and we're allowing the teams to actually focus on the ideas they think are really good that's pretty much it for my part yeah so just to kind of wrap up then is that we made a lot of mistakes trying to follow another studio's process right and so we originally went from a static group of experts kind of like the pixar brain trust and that's what the creative champions were to this rotating panel of experts throughout all of king's senior creatives and that was like super great because like you sometimes a creative would be going through the process themselves of getting the feedback and other times they would actually be giving feedback and just opening it up to everyone really felt like uh it was trusting our creative uh creative experts throughout the whole of king and then the other thing it was like very very team focused right so it actually came from the teams so we had these support channels that bob mentioned with his workshops his fqas and his play dates

and uh and they run from like concept from prototype to production we're currently looking about how we can apply this to live games as well and then we have this senior create review and we and it's a review system it wasn't being hidden like it was before through the champions it was like it's obvious a review system and the team can actually ask for this review system when they want to ask for it which is another another great thing so it's not forced in them when they feel they're ready then it goes to the scr and you know we've been doing it like this now for about a year but we're going to probably end up involving it again and and because it is kind of flexible but it's uniform right it goes across all our studios so now we feel that our teams have gone from this to this that's what it feels like so we are hiring obviously so if you go to king.com jobs and you want to actually come through this yourself and actually uh create games with us then please go there thank you

there any questions is there any carryover between the sdrs like do you take some of the feedback from a previous scr and see how the game has evolved in direct correlation to what was given in in a previous seo yeah i think that happens quite often um normally most teams won't go through the process once but occasionally a second time um the feedback's usually taken over and then teams will come up with a plan on how they're going to mitigate those risks that are raised um and usually if we go back and we do another one in in six months it's one of the things the reviewers will look at is how well did they execute on their plans to mitigate those risks that were flagged up do you think the brain trust would have worked if you just do it

purely as feedback like here's some feedback do what you want with it i don't know i don't think so it it we tried to do it like that but it was uh i i think part of our problem maybe if we were in like one big studio it might have been able to work like that but i think this whole feeling of traveling to a studio and having these like people travel you know there was kind of really stressful for the for the teams um but but like i was mentioning i think we evolved enough for us to get to this point where rather than having this uh this brain trust of experts we have so much expertise around king so actually bringing them all in together felt right for our studio is anybody watching the watchers do you guys give feedback to the scr panel um that's a very good question

to an extent yeah uh i go through i don't actually write any of the reviews when i always go through them all uh and i've had a lot of conversation with the reviewers saying that your your feedbacks may be a little bit too harsh or a little bit the criticism's a little bit off i mean ideally you want to have different opinions so you can't just say somebody gave a bad review to a good game you want to have that person say that they don't believe in the project um sometimes you get people who are a little bit rushed and they'll write about two lines and things like that but um we don't have like a formal system it's more on a kind of case-by-case basis i think a big change that we did recently though it used to be anonymous right the scr and then we changed it now to to like you can actually you know who the reviewer is now and that was something that evolved as it went uh as it went along hi thanks for for your talk so my question was i maybe i lost a little bit but i i saw that you evaluate the designers on your spreadsheet i saw some marks so how are you doing this how you

evaluate if the guy is good or the girl is good um it wasn't an evaluation of the designers you want to have the different colors and and and yeah it was it was more first of all people would self-assess their say i'm i'm really really strong in rpgs or i'm weak in puzzle games um and they're the design director the studio design director who who's responsible for for the designers in each studio we kind of go through and calibrate that before we put it into the system okay and another question how you you mentioned that it's important to build trust so the team must trust the reviewer do you build that trust somehow is there any process for this yeah um i i only touched on it briefly we don't have a huge amount of time but we we created this um game design leadership group a couple of years ago and and we meet uh virtually every two weeks um and then every three months so once a quarter we all get together face to face and and

this has been something that's been going on we've been pretty religious as far as making sure that these these meetings happen so people get to know each other you know each other's strengths and and and that's kind of like a a first step but through that you actually you talk about games you get to know these people and and uh like i said you you you go out for bears and wear stupid hats and and yeah and the other thing is is because the the reviews are done by people from the game design community um you might have your team's game being reviewed one week and then the next week you're reviewing someone else so you've been through that the situation both both as a designer and as a reviewer so it it gives it familiarity with the process well it really helped to stop going from this central group right to this global group and and take a creativity uh direction from everyone in the different studios yeah okay thank you thanks for the talk guys very useful uh so i had a question if this is successful for you in the future

do you plan to expand it to other disciplines as well and not just the game designers to involve more of the team members yes we do i guess we like art right we're doing the uh that we're currently it was we used to have art actually evolved in the scr as well but now they're going to get their own one uh their own kind of review and uh ux as well ux ui is another one that we're doing too hi uh given the poor reception to the creative champion approach did you have to do anything to kind of like encourage the game teams to reach out to the new group for feedback no i think because we were just very honest right a king's actually it's a very flat company a very honest company so we just said this is not working to to everyone and then we went to all the teams that have actually been uh working with the champions said what would you like to improve what would you like to do that and like bob was saying having this designed leadership group as well where we could all talk openly about

with each other and then then that to filter down through their design teams actually you know it became quite easy to everyone to embrace it great thanks thanks hi this is your name so your experts are kind of like helping the teams to get through the gates and green lights but who are those people the same that are actually approving the green light or are proving the gate or how does that work uh yeah sure i i think the goal of the the reviews are not to help get through a gate they're to evaluate it and that's used as one of the data points that would be in a gate um our our chief creative officer is one of the people who's available who involved in that as well and he's one who would kind of bring the opinion of the senior creatives to that meeting uh but there's all sorts of other stakers as well there's people from business performance there's tech people to say like can we actually build this game um there's someone from human resources who looks at um you know do we have the people to make this game as well so uh

the creative reviews is i i feel a pretty important data point but it's just it's just one that we use okay last question yeah um so you mentioned that you're starting to roll this out to other disciplines are there any significant differences but uh giving that review between different disciplines or is it kind of the same system i think there will be we we're only just starting it off now aaron who's over there who's uh uh head of art so he's gonna be we're gonna be working with him and uh trying to figure out the best process one of the problems we had in the original scr is that we could never we we weren't showing images or anything like that because we didn't want designers to take pictures of images and stuff but obviously art needs to show images throughout it so um we'll work with him on the process and then maybe we'll be here next year talking about cool how it felt thanks cool thank you thanks