Star Wars: The Old Republic - Creating A True Competitor in the AAA MMO Space

[Music] we're gonna start probably on time we have a lot to cover it's kind of sad when you go against like valve and others out there some great talks going on but we hope that you get a lot out of this one we ask that you please make sure your cell phones are on vibrate and don't forget to fill out the forms at the end we'll remind you again before you leave so the topic here as you can see because it's on a big projector is is we're talking about Star Wars The Old Republic title there is another and we're gonna talk a little bit about some of the challenges that we that we went up against some of the things that we did in essentially in our production cycle to make this game actually happen to come out and actually make there be some legitimate alternatives to the other three-letter game that that need not be mentioned in any case my name is Dallas Dickinson this is rich Google I have a clicker here that can make this thing go forward and so I will we're

gonna talk a little bit about ourselves here because that's what one does at a conference like this so I started in in the business as a programmer kind of grew up from that as a programmer to a designer than a producer and worked on console games for the first five years of my game career and I've always been an avid online player to play DQ muds a lot even what's a GM of addy Kumud and one of the things I realized I really love that space connected communities very cool and dealt into the opportunity to work at three do with Meridian 59 and then the opportunity to help before you O launched to actually be the launch producer of UO and get that out and then of course Star Wars Galaxies and then I looked at the next generation of the Star Wars game which is Knights of the road with Republic which is which is what I'm working on right now Luthor and it stars the Old Republic and it was a great opportunity for me to kind of like learn lessons I did on other games and apply it to here and it was a pretty

long path to get to where I did and and now I'm kind of stuck on this genre because I think it's very cool to deal with connected communities and I think is really the future of online gaming not just mmo's but connected communities and so proud to be here so I'm Dallas Dickinson I'm the director of production on the project I'm old but I'm not quite as old as RIT so I've only only been in the online space for about 13 years I actually started in the casual online games Wheel of Fortune jeopardy online and then I'd moved my way along into the Latin of the massively multiplayer game and then as Richard said we've worked on a massive massively multiplayer game most recently so the big question here is you know how do you actually become a genre leader how do you get get to this place within the massively multiplayer space and you know the answer is it is not simple but um these are the principles that we try to apply um the first is that you have to be able to broaden the market I mean something that the world World of Warcraft actually did very effectively it took the massively multiplayer space and made it something that millions and millions of people were interested in playing as opposed to

hundreds of thousands which which is honestly where rich and I came from in our previous games after that you do need to differentiate yourself there had there have been a lot of games that don't and they're the ones that have failed especially in this space you need to build enough content it kind of goes without saying but um in games like this these games that are connected these games that are always on where where people play with their friends they get through content really fast and and we took this as one of the challenges we needed to make sure that that we had not just enough content at release but that we had a plan to be updating that content consistently and keeping people really understanding why their subscription is actually worth something we're not gonna get totally into business model talk here but really the differentiation we're trying to point out here is we are still a game that has the subscription base and we believe very strongly that there is space in the market for a handful of these triple-a titles that are worth worth a subscription to your players but you really need to have content and you really need to have it going very quickly and continue updating at all

times and it's really important to know that since this is a subscription game and they're free-to-play MTX world that they have to value as a premium service so that people who pay the subscription feel like they're getting their money's worth every single month and that's why it's critical for the update phase to make sure people when they play this game wow I'm getting great value for our money so again it's very easy to say these four things here are the big four four principles but questions yeah how do you actually do the first step and I want to assure everybody that this is the easiest step there's nothing difficult about this is to start with the world's most popular IP just do that you know once you do that and I like I like to joke with in fact I wouldn't have said this if our marketing director hadn't actually entered the room I like to joke with with the marketing department that their job was to actually walk out in front of a microphone and say Star Wars Bioware MMO and then walk away because that actually gives a huge leg up and and so that helps when you're trying to get into this premium level you need to have something that strong you do need to

innovate I mentioned you can't you can't just release the exact same thing but you need to be very careful about innovating we're going to talk about the way that we chose to innovate and why we chose to innovate in in that way you need to scale and this is scale not just in terms of the content that we're talking about it's scaling in terms of your team and scaling a team we're gonna show you the numbers in a bit it's pretty daunting and it changes everything about your processes about the way that you've actually structured your management the way you even structure the team's themselves that are working on the game and we didn't do it right the first time in fact we went through many many iterations in order to get to the structure that we currently have and then finally triaging and triage is a word that is it's overused mcpeepers about whether the work that you're doing actually serves really your innovations and serves the core market we made a lot of cuts in fact we have some of our partners from Lucas here in the room and

there are some cuts that we it was painful for us to sort of decide to make but we made it because we wanted to focus the game we wanted to make sure that the game that came out delivered very much on our differentiating factors so yeah it's Star Wars we get that that helps a whole lot but that doesn't guarantee you success again our friends Lucas know that they've made many Star Wars games some of which are very successful some of which are not so successful there's a lot more that has to go into a game especially when you're really trying to become a leader within within your genre in space so we're gonna talk a little bit about innovation and kind of our process our thought process as we chose how to in in the Old Republic in innovation one of the things that we want to do is we want to make sure that you differentiate yourself now we don't want to act like every other clone after World of Warcraft came out because we realized if we did that we would not be able to reach or keep the players who wanted to keep when we initially launched this

game so it was extremely important to say okay what are we doing and what makes our game unique out there that's different than any other experience that people are going to play our game now we didn't want to recreate everything because we are an MMORPG which means we have to have a core foundation of an MMORPG so we want to make sure that when people who played these games get into our game they would feel familiar with the game mechanics right away you know that there's a reason that people don't take a first-person shooter and remove their shooting it's because it's a first-person shooter and so we took that as a tenant you're making an MMORPG we need to keep a lot of that core so one of the things we first do and this is something that Blizzard talks about too and it's also something that was our core philosophy going into our game wave before Blizzard talked about it is to make sure that we go for a car core audience first they are our influencers there are the people that are early adopters in our game they're the people are viral because this is an online game there are

viral connection to the community and it's very important we make sure that we treat and a more target audience carefully in this game so we can get them into the game early establish the economy established the base establish the life I call it the heartbeat of our game and that's going to come from your core audience we went after that first then as time goes on with our updates and everything else you'll see us drift more toward a more casual play style in our game very similar to what Blizzard did this is you need to be careful you can't do it too fast and you can't do it in ways that are gonna alienate that core the core is your market I mean they really are your business and so you need to hold them close while you slowly grow to the outer the outer edge yeah it's extremely important to make sure that we when we talk about things like and anybody has done a subscription like game there is what we call the funnel and the people coming into the game needs to be constant I mean the leverage that you use for that is acquisition retention and react position and you have to make sure that

retention which by the way is cheaper to keep people than acquire people is very important that's why you never want to piss off your core and as you're moving towards that more casual audience you have to be very careful how you prioritize one of the things we did is we looked at what the core is and mmo's traditionally and we wanted to know what could we add to that core baseline so we all know that there's a there's a kind of a story arc that goes through mmo's not a real crew story in the game but a story arc and then there's character progression this combat exploration so this is this is the core feature set that we knew that we had to hit on all three of these and we wanted to take something more and we wanted to look at so what is the audience and this is the Bartles for for anybody who doesn't understand what is the core look for in type of these these types again so we don't alienate them right away and make them feel familiar the socializers you want to make sure we have a really good mechanics in the game the support crafting and healing and other things that are kind of sort of non-combat related we want to make sure that the

killers in our game have good PvP to play we wanted to make sure that the achievers have a really good level based system where you got cool loot and it's all about that and we want to make sure that the explorers in a big world have points of interest and side quests so these things are very important to make sure our holistic core was solid and everything on our occluding are all our game design always look back and say are we hitting this core MMORPG feature set now we decided what do we need to do a layer on top of that to make our product innovative but not do everything in the kitchen sink go so broad that we just fail at making we make a lot of poor systems in the game so what we did is we decided saying what is bar where do very well than any better than anybody else it's about story so what we said was we're going to innovate and put the fourth pillar in our game and put story in our game we're actually going to do it with total voiceover not only the NPC but the character voiceover kind of like what I put what I tell the people is

it's kind of like what's what talkies did the silent film it's gonna make the John different in the way you play our game in fact we found a lot of people who play our game can't go back into blizzard and play Wow because they have such a different experience with our game it's more emotional this is something we wanted to do we want to make sure we revolutionize storytelling we also want to say what is another important thing about Bioware games is our companion system we wanted to make people have the ability to go out in the game and feel like they're not alone all the time now it's important to meet other people we had to make sure that mechanic was there too but we waited men wanted to make sure that companions added flavor at a color also added to the storyline and really made the game feel kind of unique in that way the other way we want to do is combat now combat we want to make sure was different than any other memo before because most of the MMOs had Flintstone type combat and we wanted to make sure our combat was very cinematic very much late lightsabres touching lightsabers

and things like that we also wanted to do something that no other game was done before was actually put in a true range combat system in our game most games are melee with a little bit of fake range but really it's like 40 meters we did 70 plus meters with our game with real true cover system as a part of our range combat system getting really the key to all three of these innovations as we sort of chose them and again we had a whole list of stuff that we might want to do early on the project we chose these three not just because we felt that there were places that Bioware had some skills and could it could bring sort of that Bioware feel to the massive multiplayer space you'll note that all three of these things you associate with Star Wars you associate there was the Star Wars before you know that our game exists tremendous story telling that the sidekicks within the game and this cinematic fast-paced combat and so not only were we choosing differentiating factors kind of from the blue of like these would be cool things the Bioware does it's also actually what hits the core of that IP and the other thing that's not on this list was after we

launched we put in the legacy system which is an air everyone knows the legacy of Star Wars again everything we do the innovative lies we try to make sure it fits the Star Wars theme and our big goal was to make sure that you felt like you could live your dreams in the Star Wars universe that was the key goal that drove every design decision that we had and these were the pillars that we communicated to our team constantly and made sure our design always focused on now we're going to talk a little about this is the area that we want to talk about is how could we scale to the size we needed to do and the amount of content and the clicker clicker comes back so first I just want to give you as an idea of this you may have seen some of these numbers before this is a ridiculously big project this is the amount of dialogue in this game as you can see it's six times dragon age nine times mass effect more than all Bioware games previously made combined it is in fact that we're now in the Guinness Book of World Records for the largest voiceover project ever completed it takes a lot of work to build this stuff

and you need to make sure that you that you actually have really really good systems to do so 20 full planets some of the iconic ones you see they're Tatooine and Hoth 90 unique areas these are all the adventure areas that you go to a derelict spacecraft other places that your story will take you and it's a tremendous amount of just locations to build in the game and they're all and they're all fairly unique in their look this was the end goal and this is actually what we're hitting your art we have now the telemetry that tells us that people play somewhere between 150 and 200 hours in the level up to get to sort of to the max level of their class this is a ridiculous amount of content those of you who've worked on RPGs before know that RPGs are frequently 20 to 30 hours or maybe 50 hours if you're really really pushing it that's one story 50 hours we're doing it eight times at 200 hours so the only way you can do this is to make sure that your pipelines are solid we had again a leg up here the the hardest part the longest pipelines were the ones that Bioware

already knows how to do we have a tremendous writing pipeline a tremendous vo pipeline in fact a great localization pipeline so all of the stuff that that you know that actually kind of bites other other groups in the ass when they're trying to building in that has this type of content we knew this stuff and we knew how to get it done we knew the schedule and the timeline you needed in order to get your writing done early enough for all of that stuff that has to happen on top of it we also we also had a tremendous part out sourcing pipeline which is going to talk a bit more about that that's something that that we actually we sort of brought in I should say which had some great experience I had some great experience at Sony where we had we had done this exploration into how to create large amounts of outsourced art and then our environmental artists they just they also kind of really got into the mindset early on of like how to iterate very quickly and to get these planets built out some places we didn't do so well this isn't a full post-mortem so we're not gonna go in into details but we had some some very bad technical issues our build pipeline was horrible that this was impactful not just to the developers

it was impactful to QA we had difficulty getting builds in front of QA that were in fact solid builds that they knew that they were testing against the true latest greatest code this this killed us and it honestly was one of the biggest problems we're still finally figuring out all of the solutions today same is true of some of the designer tools that we have for our data table management string table management scripting again these are less impactful I should say that the the third ones are the second third and fourth ones realized impact the build pipeline was impactful throughout the process as I say we spent a lot of time trying to refine it and get it better but it wasn't perfect and it actually you know it caused a lot of pain in the studio the goal here is to actually have enough content to obscure the horizon and what I mean by that is we've all played the games specifically massively multiplayer games where you play the first five levels ten levels and they're awesome they're really highly polished you can tell that a lot of time and love and iteration went into making this experience great and then you fall off a cliff I mean you

literally you can feel it as a player obviously you were in the games industry you probably feel it more quickly because you know what's going on but the players are the ones who were who or maybe not quite so locked into how game development works they feel it as well I just stopped getting quests at level 10 I just stopped seeing the level of polish in the areas when I left this zone it started to get a little buggy and glitchy and we knew that especially with the game that is so much about the storyline is so much about having this engaging emotional piece that pulls you through we couldn't have that happen so not only did the first five minutes need to be awesome it actually needs to go all the way through if you don't have a climactic storyline that actually pushes you through we sort of had three act structure and if you the third doesn't hit you with the power of like great cinematics great story we actually have failed and so we had to spend a lot of time making sure that those pipelines work so that we could achieve this goal and how you do that is by scaling a team so we're going to show

you some numbers here that are kind of terrifying uh we haven't said them publicly before but anybody who actually looked at the credits could have figured this out so yeah that's fun 15 minutes which is scary and enough in itself I had the task of help putting that together and let me tell you if you ever do credits it's like it's like really interesting of what to say in the game all the politics comes right out one of the things that we had to do is we realized as we go through and I'm going to talk about each one of these things what we had to basically do to develop the content wheat that we talked about earlier is in order to do to manage this we had to have 30 production people these are producers Dedes which would director of development type people and also pm's project managers to help support the team we had 75 designers we had 80 engineers working on this project we had 40 engineers and platform and management because we had to build a

platform to launch this thing and we had an audio the great thing is we had LucasArts do the audio for us but we had to and they had a big team on top of that by the way he's not included in this we also had a team here just to manage the audio and integrate it into the game that's why it's the smallest of the groups and the art and the art pipelines we had both outsourcing and in sort and and also full-time people we have about 140 artists together working around the world on this project and we had 280 QA people at the peak working around the world both outsourcing in also in in-house outsourcing in-house sorry QA because the one thing we had to do is be able to go through all this content and repeatedly go through this content every build you have to have a large team to do that in QA to make that happen how do you manage a team like this we have where we are very badly at first yes is it is actually the answer yeah it took us quite a while to get to the place where we actually feel like we have a handle on on this truck yeah we grew kind of

quickly and we had the growing pans to show it and we had the scars to show it we realized as we were going through this stuff we said oh we really have to make some changes here because we're not scaling very well and the thing about it is is I tell people we are craftsmen we are not cogs on a machine and we all have personalities that we have to deal with and we all have to learn how to deal with those personalities what what does everyone do they take their best programmer and put him in management right he's gonna be lead program even though he's the best program doesn't necessarily mean that he should be put in lead structure because he may not fail very well he may not do very well because that's not what he has in his DNA to do that so we learned a lot about scaling a sense of what's appropriate roles for people and about hiring so one of the biggest things we did for hiring is we made sure that we hire people we want to make sure they have the skill to do it but the biggest thing after we checked that off is can they fit in our team culture that was more important to us than anything else because everyone does deals with the prima donnas and the ego type people we have them too we

aren't we run 100% good at that process and in fact sometimes you want those people they are highly valuable these are these are the the self-described geniuses are sometimes actually geniuses also but if you have too many of them if that becomes your team culture as opposed to they're the exceptions it's really hard to manage a overall team especially two sometimes you gotta isolate them a little bit put in their own room and make sure they're not dangerous or genius stuff yes anybody but there's definitely personalities we had to deal with and we had to do a lot of shifting around because some people didn't do well in their roles and some people did and actually some people surprised us and did really well in certain roles that we didn't even know they could go up came came up to now what we did in this organization to make this happen is we basically if you look over here we basically have these two pillars which is QA and production over here these are supporting the interior thing here you see here we broke it out into one section as content the other section is tech and what we did is these two

pillars right here we're a matrix organization they cut through the tech and design or content which means content anything audio design anything art is all content and then of course tech goes all the way down into features and programmers we made sure these two organizations supported through out the organization with project managers Dedes and producers and PMS and QA had embeds working with and will talk about how we broke all this out in two strike teams to make sure they supported sorry to make to make sure they supported the core group here we learned that this is actually helping us scale a lot we'll talk about a little bit how the matrix management works and later on in the presentation but then we learned there was something we needed more as we grew because we're such a big team is we actually had this is what I call checks and balances on quality and direction so

quality was QA and direction was the PM's to make sure everyone is stepping in the right direction and then technically we wanted to make sure our that we didn't go down the wrong path by the way we did yeah and this is why we formed this group so we didn't do is instructure today this is not the structure we started out with and in fact the last one to come into this what was this the tech director group because we had a lot of people working on working on really important stuff really cool features they were not being brought back into sort of the reality of how to make this all scale with Ana massively multiplayer engine and so we developed a full group who really were just the auditors um they do the deep dives into making sure that features specifically technical features but also some of the ways that designers work that it could scale and could actually fit with it for engine because for a while it would be where we were kind of running off the rails in that direction and the good group good news about this group is they audited our designers because what designers will do if you allow them and give them the tools to do it they will go off the rails to get it

done and what they realize is when you get something off the rails it doesn't scale very well in an MMO everyone anybody has worked on these know that they're very highly integrated systems and if you mess something up down below it's gonna affect something over here you had no idea about so the auditors actually went into all the systems the big questing system that we had in our game and fixed a lot of designer zealous going to make sure it takes because designers are impatient they want to get it in and they want to make great stuff anyone make you don't give them the tools that they want they'll figure out how to make it with the tools that they have and that's often exactly the wrong thing to do so this was a very good addition I highly recommend anybody doing this in a large team about that make sure you have your checks and balances on your schedule quality and tech so when we started that's not how we did it when we started we kept our discipline we started totally agile so we are we we kicked up agile we had all the managers change in agile development we got Mike

Cohen to come down and we went through the workshops and all the training and everything and we formed scrums and we did the stand-ups and all this little things that you normally do we found out as we were moving along and we also had the disciplines that not work as true strike teams or scrum teams we actually they worked but they didn't work together if you understand what I'm saying they were always with their programmers always with the artists except they would go meet in the morning and do their their stand-ups and they will kind of work together with the teams in different locations but never sat together we found that they started to form towers and we and the cultures around them so we said this is not going to scale very well so what we happen was again that we were spread apart did not scale so what we did is we looked at it very carefully and we said I ain't gonna work so we when we we looked for a new facility because we outgrew the one we were at we actually designed it from the ground up to be pods or strike teams we

actually have areas that are totally mobile we have mobile desks mold chairs filing cabinets are all on wheels they all move around we actually configure we've actually configured our our pods probably once every six weeks or so we reconfigure our pods and basically what we do is we put the everyone together and formed multidisciplinary strike teams and this is organized around a single deliverable or like in the case of if it is a if it's a feature it would be the entire team that does a feature if it's content it would be for instance an entire planet so that would be the pod it is everybody working on the planet including audio including environmental art including effects these people would all sit together now one of the things we learned learn we're scaling is when you scale so large there is a problem with accepting responsibility and ownership over a particular item and you always find the finger pointing going this way with the groups we found is when we put together as a single focused strike team they actually put they felt like had

ownership over control over it and it made it for the large-scale team that we had like 400 people it actually made a single point very solid so it worked very well for us but again we didn't do this until way too late we started doing this a couple of years ago but we didn't really get all the way moved over into this structure for quite a while it took it took a long time and there well there's some some roadblocks you're gonna talk yeah one of the things that took a long time is all the leads were terrified of losing control all the leads that I'm talking about the gameplay lead the systems lead all the Leafs out like we're gonna lose control why are we doing this and so one of the things we did is we said we're going to do a matrix management across so all the leads like the gameplay leads and and the client lead and the server lead would basically say okay you control the resourcing for particular strike teams you will help control the review process but you don't control their daily tasks that's the strike teams responsibility of the leader the strike team to do that but you control everything else so this

matrix organization setup by discipline number one help transfer single knowledge because one of the biggest things you don't want to do is when you break up teams like this you sometimes mentor each other programmers mentor other programmers it's really important to make sure you keep that line art the same way and design the same way you have designer lunches a programmer lunches you have different feedback loops so that you don't lose that core knowledge and you make sure that sharing a crop happens across your team's it's very very important because if you do this too much you'll lose sight of that so what we did was we wanted to make sure that that's the other responsibilities that the leads had to do they can meet with their teams and go over things and do white papers or or post mortems other things like that to help provide a outlet for making sure that ours feel like they're artists they can talk to other artists and things like that it really worked out well and it really I mean the Bisbal was a 180 flip from what we started with the you know when we started the team that the discipline

teams were all co-located and they met once a day with their cross-disciplinary groups and we just did the opposite we've got the cross-disciplinary groups actually co-located and then the discipline teams could meet maybe not once a day maybe but once or twice a week and they would get all of that all of that feedback and knowledge they needed all of that touch but the deliverable was the thing that was the thing you were working on that's what the guys around you in a bit we're all working on together so one of the things we also had to do is scale in terms of we call our RTS which are full-time employees T of T is a temporary employees the term we use basically these are people who work through an agency that we go through and they usually have your contracts normally we hire basically students or we hire some really good people out there who just not who we hire part-time to come in or their full-time employees using our system but we don't we hire them for X amount of time they're not full-time employees and we use out sources so one of the things when we start off in production we realized very quickly is we had it we had the scale but we wanted

to scale in a way that's very smart we wanted him to scale that we could do it quickly and we wanted to be able to scale so that when things are done we can pull off as the transition to price go from production to launch there's different aspects of what people do and they change over time so we came up with a process where we went to schools and we look at the best cohorts coming out of the schools for Art and Design and we hire the top ten or so and we put them on for a year they get their year experience is great for their resume they work hard with us that's new blood and before the cohort ends which is usually right up in here we do a three-month training overlapping period we would train up another set of courts coming in and then the cool thing about is we pick in these areas the best of the best we say this cohort we're gonna pick three designers and they're gonna become full-time so it gives people an opportunity to show their stuff and get selected as full-time so it's a try before you buy

but it's also clear to them and I think introduced sort of in in terms of HR management it's very clear when when you hire these folks that it is it's a contract position and I've worked as a contractor I think many of us have worked as contractors it's a very clear deal and it makes it a whole lot simpler for everyone as well as emotionally better when the end of the term is is up that you know some people got got pulled in not everybody did but it allows us as an organization to grow more slowly with the best of the best absolutely so on top of this we pull out sourcing so we not only had outsourcing for art we had outsourcing for QA outsourcing allows us to scale very quickly because what we did is when we'll talk a little bit more about this later is an outsourcing we had the ability to say you know design we design and I we're a very design driven shop model a Bioware's and so when we iterate things we blown apart worlds over and over and over again to get the right feel to it and add things to the world's adding big set pieces the worlds or take a long time to do if you don't have an

art ready to scale like that so the great thing about it is we use our art outsourcing to scale very quickly I need all this armor set that has to go across eight different character type body types okay fine we'll get this outsourcing group let's say go to 25 immediately or go up to 30 immediately and they will and that's fantastic because it allows us to do fast iteration scale very quickly without the burn by the way one of the things is this does is it slows your burn rate on your project to your cost especially in a big project you have to worry about your burn rate especially the size we are and outsourcing the way we manage that is very simple we run around the world and looked at all the outsource houses that we could find where as we flew we didn't run and we went to each one because it was important to see number one what the work environment was like were the people happy what's the churn rate in that group because be honest with you a lot of them do have a lodge churn rate and you don't want to train people over and over again among the leads especially among sort of them

the management structure if there's a bad big churn you don't want to work with that and basically what's the feel that environment can they scale number two and three is do they have the network infrastructure to support a build every day that we have to send to them our come back to so we can get stuff quickly from our outsourcing houses so one of the things is once we once we picked out our outsourcing which we had one in China and one in the United States and one in Russia Mexico and one in Mexico we also had another one in rhenium and one of the things we did is we looked at the core team and we formed a group of sub leads that we made them hire and train for us that we trained with them we brought them to our place we trained them up and they would be the ones that trained others so we can scale very quickly that way because they know what we had to do and they can train other artists very quickly to scale up this is very important including having an art director there that's trained bot that you pay for that's trained by them as well that all assets go through for

QA and process now another thing we says every two months we'd send a crew up to touch base with them so we go out there talk to them see what their problems are because face to face is much better than VidCon or Communications to email a lot of times they don't tell you the problems actually going up there and watching them work is really when you find out all the issues because it culturally some people don't talk about things like that and you have to see that for yourselves integrate them as part of your team is extremely important don't throw things over the fence it won't work you got to give them t-shirts we used to go with another luggage of just t-shirts and swag for them and posters so they need make them feel like they're part of the team the other thing we did is we had pictures in the wall at every outsourcer and they didn't know this but we raided them we actually had a little gold star silver stars bronze stars and fire and we rated them of how good they were and we this is an every group of artists that when

we built our art team we said you're we're building this art team for 60% of outsourcing of art so get used to it you'll have to manage other artists and we started hiring for that in the first place and these artists had a set of people on the wall they were responsible for for feedback and it really worked well and basically we also had bid cons we made sure that we always provided a timely feedback mechanism pipelines for feedback was very important to make sure that if they continue doing at work we get them corrected I hate it when they when people come to me when we first started out saying well they're not doing what we wanted to I said did you train them no we told him about it then we didn't really follow up on it was all highly supposed to get better it takes effort to make this happen it's not easy and it takes a lot of communication especially when you have time zone issues I actually had to work art and shifts to make this happen now the other thing we found out is different studios have different skill

sets some studios are very good on the organic stuff like the characters and things like and creatures other studios are very good at inorganic like lamp posts chairs buildings and we found out it's very good to do that because go to their strengths and scale with their strengths it's one of the reasons we had so many so many studios that we worked with right and the other thing is don't put your eggs and your all in one basket one of the things I've learned is make sure you have multiple outsourcing houses if you're going to scale like 60% like we did it's very important you don't know what the political situation in fact we had an issue with Romania when they had to be shut down a couple months because some political issues that were going on there these are things you have to be prepared for and also it's leverage to you for your outsourcing if they're not doing your job you said well I'm gonna shift it over here and it makes them more motivated to do your work for you now QA outsourcing was a little different for us when we first started out it was very hard it's for us I'd say it's the

hardest thing we had to do to get right it took a lot of effort we started very heavily on outsourcing external with leads just in internal we found it that didn't work very well at all because what we found is most of the outsourcing houses that we're using just couldn't get with a quality level of gameplay experience that we wanted to get out of QA so we had also a lot of technical hurdles when we spend out builds of like 20 or 30 gigs you know out to the outsources it takes them almost a day to to load it so we went and make sure that they had a technical infrastructure there but still it took up to six to seven hours and some times a day to go out get the build so we lose that time so well you found out very quickly is we use QA for regression testing because you see how big our game was we had 1,600 hours of content we had a test plus all the features that wraparound that they're very good at regression test so we actually used them for that so every time we make a change basically

mlo everything is connected we used them to do that we also they weren't very good at play testing the game so we had to work very hard to get them to up to the level but they still never got up to the level of our internal testers so we grew our internal testers into embeds so we had QA embeds that wrote all the test plans for the particular pods and they were responsible for the outsources to test them as well as our people inside the company we had a core group a team that was just there as a strike team to use whatever we wanted to do whatever publish we had so we found out a good balance for us is more like not 10% int int inside and 90% outsource we found a good balance was probably about 70% or in this case probably 65% inside and then 45 or 35% outside was a good mix for us the other thing we did is talk about how we scale our production side of things all that Dallas talk about that yeah so this is

really it's it's a rule of thumb more than it is anything else I mean you will have variations within within your within your studio and such but this is what we've kind of landed on in terms of if you have more than 15 developers and this this is includes designers programmers in fact and sometimes UI artists all working on on particular one of those scrums one of those little uh those pods you need at least one a production resource for them this number is is what we work with today this is where we started it was ridiculous it was it was just we grew really fast and we didn't grow the infrastructure to handle it fast enough and so we very quickly found out that things were dropping through the cracks I think we bought into a little bit that um if you hire senior enough people they don't need producers yeah that doesn't work and so so this is where we were weaned out like that you know the fifteen to one it is a rule of thumb um the reality is we we've seen your people who can 25 developers and you know that that probably means two or three big

initiatives that are being worked on by by these sub teams junior people people are sort of associate producers that we bring in our our associate PMS we start them at ten and see how much they scale it's not the only thing that that I grade my people on but it's one of the most important ones literally is how much do you scale as a producer how many how many bolts can you juggle at one time in the air we've actually taken on a mix of these words are probably different in the students you work in a DDS within the EI culture for our development directors PMS project managers they're actually kind of in the same chain but usually a little bit lower level and then producers and we use them for very specific things so again this is something we grew into when we started I was the producer and we had three Dedes and it was as I said not good in terms of in terms of numbers scale the producers tend to be the ones who actually own high-level deliverables they might own an entire milestone or as we are in kind of a cycle now where

we're producing content on a fairly regular basis they might own one of those big game updates so they are a combination of you know schedule person making sure that things are on track but they also hold a little bit of the quality vision and they are the ones who are wrangling things at a very high level making sure that that QA is is embedded it has a good test plan etc our Dedes tend to own our pure tech initiatives again the the DD within within EA frequently there are people who even have programmers reporting to them so they have a very technical background they're the ones who would actually run like some of our server initiatives the things that the players don't actually see but it's how your game scales and if your game doesn't finish this sub feature that is entirely opaque to to a paying customer the game will crash and so we would we would give that type of initiative to our Dedes and then finally we used rpms as really the the follow up people the detail trackers and so we have more of the PM's than we have of the other two and that's that's kind of how we cover all of the stuff that we're doing both on the content side and on the feature side what we

found and this is obviously a cliche everybody knows this Communications is really really hard it's even harder when you're at the scale that we're talking about it's still something that we don't do perfectly today and so we have to be really really careful we have to pay attention at all times to do the way in which our our different communication methods are either succeeding or failing we actually communication strike team and part of what they do is they actually watch email lists if you just sort of do a query on all the email lists that have ever been created because someone says oh we have a meeting that happens once a week we should have an email list for everybody who attends that meeting we can talk about all that stuff and these people actually track whether or not those email lists are being used anymore whether they're being actively managed whether they've start people who've started sort of concatenated sort of shrank and Stein email list by this is a list of this list list list list list because what we're trying to do is we're trying to eliminate spam you can't eliminate it entirely but as a producer I get so much email I can't even talk about it I would like that not to be true for the people who are actually individual contributors who have a real

skill set because I don't have one so we do that a lot meetings this is pure agile methodology um but it's something that we really try to push on all meetings that are going on if you have a meeting that does not have an agenda does not have a timekeeper and does not have action items that come down it comes out of it it's a failed meeting and we really push that hard I try to I try to anytime I'm in a meeting if nobody is fulfilling those roles I jump into it to say hey I just want us to clarify what is the purpose of this meeting does anybody actually know or we just kind of sitting here and and you know wool gathering and then finally the wikis this is just a funny thing when you have a team this size all of us are vaguely technically apt people start building wikis because it's a really cool way to have information that can be shared we had at one point in time three different software packages running different wikis so like you know someone's like oh we're gonna use that we're gonna use this one and we're gonna it was ridiculous and and and they would not link and one would just it would cycle those sad blogs or well personally I'm gonna start a blog and they would like

write one blog post and in six years later you look back at it so we had those wiki's and so the same communication team does that as well they and that's brutal I hate that but and this is these are people in my production structure who also have day jobs this is the thing they do for fun because they're that kind of mentality that's what producers do they're like I'm gonna go clean up all the wiki's I'm like please that's awesome yeah please tell them to tell you pused I'm sorry we also have changed our tools several times and we've done so explicitly because we found that at this scale certain tools were great for one purpose they were not great for another um in the last six months or were a little bit more than that of the project we stopped using hand soft again we were we were an agile company before that but we found that we were so much into the the triaging of bugs that that was a significant portion of our work that even though we still had some new work new feature work working in those two different places was sort of schizophrenic for our developers so we just stopped using I mean we didn't stop

using entirely some people who had longer soak projects still going we're still living in hand soft but everybody else significant probably 75% of the studio just started living in dev track and we kind of Frankenstein Deb track to to support new dev as well as bug tracking wasn't a perfect solution but it was something that we had to do at the stage of the project we were in we couldn't rely on the integration of hands soft with our with our bug tracking software it wasn't ideal and I've been told that it's gotten better and so we're kind of reassessing now because we've changed what we're working on but as we final the game we had to do this and it was a big change for people changing processes this is the thing that that probably you know there there are probably little dolls of me in the studio that have just needles right through the eye because I do this all the time I take my team and we look at what our processes are and we say where are we failing where are we where are we running into sort of repeated fails we have bugs that keep creeping back and

even though we know we fix them we're gonna fix that process we're gonna change that process the process has changed a lot at some point in time I'm sort of hoping that that the studio will understand that iterative development of a game also means iterative development of processes and that's actually healthy you need to embrace the change it's not there yet there are a lot of people who are very very resistant and upset whenever we change a process and they you know they write in giant screens against like every time we change a process it ruined you know Jesus killed a baby or whatever it said then they just blow it out of there like it's crazy talk but it is a cultural thing that I'm trying to make sure our studio understands you have to change your processes especially when you're at this scale so we're just gonna talk a little bit about production methodologies because we didn't do the standard thing here no so one of the things we realized too is again we started out a j'l but one of the things as we start again in production we realized that not all things fit one package or style of development production development so

project management basically so what we did is we said okay we knew things like writing in VO work very well waterfall because they're very predictable very much a lockstep process we also knew that we wanted to make sure that we developed a process for our art and our world design that time box resource manage and also look for efficiencies to make sure that we can get things done in the timely manner so that we can get our game done and iterated on because iteration be honest with you was our most important driver we had to find ways and systems that we can go in and check our work to make sure it was viable in an area so we might want to make sure we had enough time to iterate on everything we built so we use lean process for that and we use for software development it worked very well for agile so we kept agile for soft development so our core tech and features were all agile driven and that and so what we learn is there's not one size fits all at the scale we're working

we use multiple different processes we use I said whatever teams work whatever you use work and you can track it right it's fun we did say that we use hand saw we actually used hand saw all the way through a lot of the processes we did good a bug track a dev track for triaging bugs down but to burn down for our most valuable unit in our hands off to tell us how we're doing above the line or below the line when we're talking about tasks that's where hands off came in very goods as a tool your burn downs tell you a lot whether you have to crunch or things are not working very well you're not hitting your deadline but again the point in point here is and I don't think this is even necessarily about this scale it's probably something that I'm gonna take with me no matter what scale of project I work on in the future I used to be very strict about like we're choosing one production methodology and everyone is gonna follow that methodology cuz it's the methodology it's how we're gonna our game work and that didn't work for us here and we started to we started to you sort of pick and choose the proper tool for the proper type of pipeline and I'm probably gonna do that in the future regardless of whether I'm working on another gigantic game like this or or

I'm working on something smaller so one of the things we learned it and this is what Dallas is going to talk about is the triage process and how important it is sometimes it's hard to fix a lightsaber so this is where we didn't end up metaphor and stuff anybody know who Colonel John Boyd is raise your hand that's awesome you probably don't so um he actually wrote the the book on fighter pilot strategy and there literally was not a book and then he wrote it it's what they actually use in Air Force training and he has a lot of cool in there things about ailerons that I don't understand but the core principle or at least the core principle I took away from the book is that he believes that the person who wins is the person who makes the most decisions fastest the quality of your decision is obviously important but that's sort of based on your experience if you were an experienced person you're gonna make slightly better decisions the real differentiator is whether or not you made enough decisions if I made two really good decisions and you made 18 decisions in that timeframe I'm probably dead so true in dogfighting I think it's also true in production especially production at scale the way that this is

Illustrated sort of the methodology that we try to sort of put throughout our studio is this loop you observe you find a problem you orient yourself and try and take in as much data as you possibly can you decide what to do and then you act immediately it kind of goes to that what I was talking about with our production method methodology is changing we iterate all the time and if you're not iterating all the time if you are either letting a decision that you made go for a long time because it must have been the right decision we thought about it a lot or you choose not to make a decision because you need to gather more data you're actually failing you you were allowing your project to go off the rails and when a project is this big and we often talk about how hard it is to be steered it to a course correct in the Titanic or in a you know in a battleship it's really really hard you need to be course correcting all the time but this is what happens these are some of the pitfalls and why people don't want to iterate quickly this is you very much in on the tech side people did this what about the things that we don't know we should R&D this for a

while and that's really sort of that in the observe oriented aside and act this is the orient stage where people say we don't have enough data yet we need to look at look more at this we need to decide to find out more let's do some tests let's do some R&D let's do a little bit of prototyping over here if you do that if you allow your project to get stuck in that in that cycle which is not the cycle of actual action you're gonna you're gonna ace ulta Phi and B you're actually probably gonna go off course because you're not deciding we tried very hard to to get to action even if it's not necessarily the perfect solution start going because we can always correct again and it leads to this which is a thing that you've probably seen in slide shows before and so I threw it in there it's the worst phrase ever I'm a terrible person the other piece and this is this is kind of the flip side of that um that we found is history is very useful again those of you who are project managers you know that this is how you you figure out like a person a person or a team's actual velocity how much work they can get done in time is based on the historical

velocity really really important what where history starts to become a problem is in the post-mortem process I have been in post-mortems and it's certainly even on this project post mortems that lasts a day and you come out of it with a list of 45 action items and guess what nobody ever does them you spent a full day of a lot of people's time kind of analyzing and saying what would have been the perfect perfect way to have done that and know what we should do we should look at this and that is not useful as a rule of thumb again I like post-mortems to be an hour long if it's more than that it's actually two post-mortems you need to figure out how to break it apart into these people should talk about this system that needs post-mortem aching these people need to talk about this system or methodology that he's post more to me you come out of it with more than three or four action items you're not going to do them it's just the list is too long and you won't get them scheduled and other stuff will come up and again this is really this is really the the other end of wool gathering if you're doing this for too long you're not actually observing the

new problem yet you were sitting there between having acted and then finding what the new problem is intuitive to make to make it a changes for so the big deal here and this is true of all of all projects not just projects of this size and scale is that it's hard to ship a game there are all sorts of things that get in your way things that can derail you at this scale it's a longer road so there are many many more of them and and so what we tried again keep our mentality about that iterative process because you want to solve your problems as quickly as you can and you always have to keep your eye on the goal which in this case is roadside assistance for the dude who went off the road you know the the goal is the goal is to make a great game it is to make the best game you possibly can with your resources your time and your budget we are aiming for the triple-a title of the triple-a sized MMO but it's the same methodology here wherever you are you need to identify those problems and iterate as quickly as you possibly can don't let them sit there these are different tiers of our triage process that we're going

to kind of talk through a little bit of so one of the things we did is when we first did agile development no one ever talked about the roadmap they always talked about the what's in front of them or the the backlog no one really pull it all together and so one of the things I had to do is I kind of thought about it you know when you do agile you're so close to the weeds you don't see where you're going sometimes before it's too late so what we did is we developed a release schedule I call it which were every six months we look at our back current backlog and put it together in a release schedule for what we want to do this is number one to get rid of the risk that we talk about in this project do them upfront and early prototype develop things that we can get the unknowns known fast as we can so we did is we in the first release we did our first playable as our first release was six months of work we broke it all down we figured out that after the end of six months we had a playable deliverable call the first playable of our game and what we did is we took vertical slices of different things that are risky and

we put them into a six month cadence what and we in that cadence we basically had seven week milestones now 17 milestones were made up of two sprints of three weeks in duration so hey Len led to a set about next week and a bug-fix I get to seven and the bug fix week is when you pull everything and you do the integration we know integration by the way takes longer so that's why we do two sprints because we integrate at the end of sprint one and we integrate at the end of sprint two and we also when we put it all together for a release we actually bug-fix it and we also did something at the near the end of the project we formed a Death Star the Death Star is where the director of gaint where all the tear top tears you saw in that first organizational chart was sitting in the room together and what we did is we brought everyone to us to explain where they are in the background of our room we had a board that was like 20 feet across and had magnetic strips of different things that were outlining what we needed to do to final our game

and they were tracked daily we're moved across the board daily by the PM's and the producers they're responsible for the different strike teams we had going on remember if you have 15 people on introspecting we up to 40 strike teams running at same time so we had to make sure all that was gathering together and we had this huge punch list there on the walls of different groups that would come into the Death Star and present where they are and we did this every single day before we launched like four or five months out we went into this plan which really helped us everyone was together they came to us and they went to us and explained where we are and we triage in that room that's including bug-fixing - now the six week or seven week plan gave us the resolution that we needed to to do the three weeks that we needed to do the daily triage is that we had all this was running concurrently till we finished the game and really the whole point is that each scale here you have to be cutting yeah you have to be triaging you say hey this is our six month plan looks like we're gonna miss

something we need to actually talk about a major feature or a major piece of content that is not going to make it and if you're disciplined you actually cut it or if you're undisciplined and they're working on an MMO where you can do later which is what we did a lot of you're like put it in the backlog that's post-launch content but you need to at least do that you need to take it off the table and we do that by each of these steps when we're doing our seven week plan or a milestone plan what do we want to do aways twice as much as we actually have time for now can we actually cut this stuff can we really cut it let's really cut it if we can we were pretty brutal about this certainly during the finally yeah it's very important to cut your babies in half early and often and it's very important to also make sure that you don't do nice-to-haves right that's what everyone wants to have we made people crying death store many times because of that and the reason saying that like he enjoyed it yeah I did because I just love them because you see them how upset they get sore so think I gotta have this feature in and we're gonna die no you're not just give me that particular feature and we'll see where we are but the issue is

everyone's passionate about their systems we understand that but a high level you have to triage you have to triage often to make your goals it's a problem with agile development a lot of times I seen is you put everything in the backlog that you can't make in the milestones you don't realize how big your backlog is because you're not looking at it all the time and it's huge your epics by the way can be very huge so we realize especially in the scale in the timeframe that we're working in we really had to manage that closely to the end we actually did daily triage because it really was about bug fixing at that point so we basically said what's our s and B's what's our C's Indies what are we going to put in this particular build for beta testing that we would want to do and we actually did that on a daily basis we actually do something that's interesting we do it to a tiered triage again with a team this big we have actually had to give ownership of not just the work that's being done in a particular sprint but actually the bug backlog on those strike teams we mentioned before so the you know the QA lead in that strike team the product owner of whatever that feature or piece of content is and the unde the the producer or a project manager who's

assigned it's their job to keep that prioritized and actually keep people burning through it we had some teams that were very good at that some teams there were less good at that and it really kind of had to do with with how strong the producer was and whether or not they're their actual product owner or lead had that sort of triage mentality and so their job was to keep that stuff going as much as possible and then raise up to us in the Death Star any place where they needed sort of a higher level call or they're like well we can't get this done in this in this sprint right now but I think you guys really wanted do you and they'd bring that to us and so that would be the next tier of the triage and we would do that also on a daily basis is sort of the second tier that stuff that bubble pops it's really important we didn't have enough time to talk about this what done means done in a lot of different areas means acceptance by a product owner in QA it's really important because now that people think done means something else and it's really done in phases a lot of times things don't fit in a perfect milestone I wish we had time to talk about that but the the key thing here is about cutting and scoping all the time it's important to think about everything is on the table except

for your core you got to look at your core always a value what you're cutting to how it affects your core there are certain core that you have features you have to launch with or you're not competitive if you always have to look about how it affects your core all the time and what's your quarry all the requirements that are affecting your core have to make your decisions in triage and then again cutting the nice-to-haves early and often are key to doing good triage and triage is really what breaks in my opinion the low quality games it what separates them from the low quality games to the high quality games and again what is this all about be very honest with you what we're all been talking about for the past thirty minutes it's about scope quality resources of time and how you balance it out through especially a project that took five years to build how you balance it out to get done and not dabble all the time it's all about trade-offs it really really is you always have to constantly do trade-offs all the way through your development process from the beginning to the end to make sure that you balance

against what your core feature set is making sure that your vision of the game has always stayed through and make sure you're competitive and differentiating at launch and have a quality launch experience and we didn't touch about that's a whole topic all together it's how to make a smooth MMO watch the things we did were all based on this thing about scope trade-offs we had to do resources in time we had to do and all the balancing out that we did it and that's what brought us together so we are running over a little bit but this is the last session we actually have a little bit more time so people want to ask this questions please feel free to come up and answer Thanks thank you [Applause]