Lessons Learned in the Trenches: Developing Games as a Service

I have a lot to talk about, so I'm going to start now. This is a huge amount of information I'm going to pop at you. Um, I basically themed this talk around development versus live. I could give a whole another talk on live, too, which I may do next year. Um, and the reason I do that is because I think a lot of the problems that I see in live are caused by production problems. And so, what I want to cover today is more focused on that. Now, this this is probably one of the hardest presentations I've done because in an hour to talk about everything about games as a service is almost impossible that I've learned in the in the past 20 years doing it, but I picked up some things that I bubbled up based on what I call patterns, mistakes that people continually make. So, the first thing I'm going to do is this is a This is a statement I I said in 1998.

It's a service, not a game, that you're making. And I think it's really important as you're developing the game that it's a service as well. So, the game, I always refer this as the iceberg theory. The game is right up above. And what people don't realize when they get into this is there's a whole bunch of things below that are huge, invisible to the player, but very important because if you don't do any of those things right, you're going to have a bad player experience at launch. So, as you see on this list, there's tons of things that have to be built around your game. That makes it a very complex piece. I would say games as a service is probably the most complex thing any developer can build. This is another theme that I have throughout my presentation that's extremely important. It's not a sprint, it's a marathon. Everything you do in development will

affect you in live some way, somehow. Some are good, some are bad. It's how we handle things like pre-production. You don't want to sprint through pre-production when you're building games as a service, but a lot of people do cuz they want to get into things as fast as possible. It's how you treat production as well, and and I'll talk about that later. The whole thing about marketing and PR, great story about this was with SWTOR. We launched SWTOR and EA's marketing department says, "Are we done?" Uh hope what? No, no, no. We do this as beats. You only did one little part of the beat. We're open now. Now we have to do a whole bunch of beats of how we're going to market and get user acquisition and awareness. And community management is another one. You have to be this in this for the long haul. This is probably one of the biggest things I've seen as a pattern

that people don't realize is when you launch a service, it's live. It's there. You have a lot of people playing that game. And if you don't maintain it, people are going to leave, especially with the competitiveness that's out there today, especially today. One of the things I saw on SWTOR was EA made a decision on SWTOR to pull it down in favor of Anthem without replacing the people they were pulling down on. They made a million dollars a person for every person that was on SWTOR. Made a million dollars. And they were generating over 200 million dollars in revenue, but the fact is their next project, which is a new thing over here, was pulling people away and they weren't backfilling. So now that product is going down a lot. Has been for two two years. That's you every investment you make on that product every dollar you put in you'll make 50 cents easily on it versus something over here that's an unknown brand new IP.

What I would have done as that game, I would have invested in that game, put it on console and easily probably pumped in $300 million a year, which is not insignificant. And that's what I mean by you're in it for the long haul. I look at building online games as a service is like the Apollo program. It's amazing how that thing launched if you think about it. It's amazing how games as a service games launch when you think about all the complexity I showed you earlier under the iceberg. Everything has to be working right. Everything has to be asynchronous. Everything has to apply well to run a service well and to make the revenue that you want. Lots of moving parts basically. My first lesson that I'm going to talk about is short-term versus long-term thinking. This is the one I always preach about and I'm sorry if I sound like a sermon here. This is one I always preach about

because this is the one that most people don't get when they're building games as a service. As I said before, everything you decide in the game as you're building it will affect you somehow in live. It is a fact I've seen things like tech debt, I've seen design debt that are very hard to get back once you go live. We'll talk about that a little later. Systems as you're developing game systems or any systems that you're working on, it has to be simple, scalable, extensible, secure, and stable. Those are the words I use all the time when I look at game systems, when I look at any other thing that supports the games as a service platform, is it simple? When you develop complex game systems that have multiple global variables that crawl over the place and you're not sure what happens, you can fix something in live and break something else.

You can fix a bug in live and break multiple bugs in live. Secure is another one. Whatever you do, you can't trust the client. Whatever you do has to be validated by the server. A lot of times I see in teams and they're developing games as a service, they become short-sighted and that's why I talk about short-term decisions because they want to get things done. They're under a tight schedule, a tight deadline, and they make decisions and I say, "Wait a minute. You make that decision, you're going to have this effect when you go live." And it's real. And sometimes they don't see that. And that's why I call it being short-sighted. When I talk about that, asset pipelines is a great one. You have to develop a cadence at live, a delivering content at a certain pace, as well as features. You better make sure your asset pipelines are lean and mean. You don't want characters to take 5 months to develop. You want characters to take far less time to develop. Don't over complex

things. Make sure you have solid asset pipelines for one, if you're doing free-to-play game, to make sure you can stock your store up appropriately. Two, if you're making a premium plus game, what I call premium plus, I'm talking about you charge retail and then microtransactions afterwards, is you got to make sure that you always have content available after you launch. It's extremely important. Look at the cadence on Fortnite. That's a great example of good pipelines working. Technology choices matter. Everything you choose and select in development will affect you in live, including the engine you use and the technology you have behind the server and the platform. Be very careful of those choices to refer back to what I said earlier, scale bone is a very important and stable. The getting the core loops and the meta game right for a session based game, okay, for and a virtual world, the elder game, you want to call it that. These things are really important to take your

time and get them right. Don't move past them. Monetization is another one. Think about what type of product you're building, what type of audience you're building on, what platform you're building on because console and PC are different than mobile. Good case study is SWTOR. In SWTOR we made we're going to make a BioWare game online. If you know how much content is in a BioWare game, that should scare the hell out of you. It did me when I first was told about this. We decided that we wanted we had a choice between two engines at that time. One Hero engine and the Big World engine. Well, we chose the HeroBlade engine. That was a very, very bad choice. That caused lots of issues. I actually wanted the other engine cuz it was more like UO and UO is something I was used to and I knew it could scale. I was very concerned about

this choice and it came true. What we had to do on this list, the tech debt that we gathered for that was so huge that I literally had to stop work on it get a whole bunch of TDs to look at where we are and figure out how we get this thing finished and to make an online game work. And what that did is it took away programmers from the system side of the game. We could develop content, but not the systems. So, the problem is you need a good balance of content and game systems so you you have emergent game behavior and the stickiness that's important. As well as technology, especially since that was an MMO, that you can do character transfer, you can do collapsing of servers if you need to without the players knowing about it. That stuff didn't exist at launch because of the massive tech debt that we had. And it slows your ability to update, which is critical for you for making sustaining players. And one of the

things I've learned by that but that decision, it cost us big time in live. So, be careful what you choose. Finding the right cadence, this is another one as you're developing games as a service because of the amount of content you have to deliver and the and at launch, you really are have to be careful about driving the team. Continuous acceleration is not a good thing for games as a service cuz one of the things as versus a a single player product is it's opening night. You're just starting when you launch this thing. You can't take a break as we said. Excuse me. It will cause burnout. I have done this a couple of times and I learned my big lesson about burnout. You can't take time off the ship. People are tired. They're not making good

decisions. They basically will leave actually. What I've learned is when and how to accelerate. And what I did is I divided my schedule up into releases and pretty much these are gates we call it today, but that this is not remember this is 2011 2009. I developed some gates. Everyone knows what the gates are. They're pretty much standard today. Your first playable, your vertical slices. I cut actually do more than one vertical slice in games as a service. And at those points, I release. But, one of the things if I ask people to put in the extra effort when it's necessary, I have clear goals for everyone on the team. They know what they need to build. I establish clear guidelines, and it's voluntary. Because if you don't, again, you run the risk of burning out. You have to be very, very careful about how you do it.

One of the other things is how to release. One of the things I've learned on that is after about halfway through a development, you should do a game jam. Game jam is 2 weeks. Do anything you want. Anybody can do it. Anybody can get together with anybody else in the team. You have 2 weeks to put something together that can be used in the game. You can get more done in that 2 weeks than you have in in months. It also re-energizes the team midway through. Highly recommend you do that. Um I also start hiring DevOps and live ops well before we launch go into beta testing, because you want to buffer the team that's currently working there. You have to buffer them through test. And I usually pull rotate people out of DevOps for a reason, because I want them to be able to feel the pain that DevOps has during testing if they need to. So,

I rotate them around. With UO and SWIG, one of the things I've learned is you could see the massive amount of crunching that I did. UO was in horrible shape when I came on board. It had to have It had to be crunched. And because of that, I had no choice. But, it was awful. It was 9 months of crunch, and then 9 months afterwards. We had a team of 64. I think we went down to 12, and then went down to eight uh people for running that kind of franchise. That's how burned out people were. We had no live ops, dev ops, or anything like that then. We all every everybody on the team had to bear the burden of work working 24/7. Swig, because we're under a massive time constraints, we basically had to crunch the team as well. And I learned my lesson a lot from that. I will never ever do that again.

It It's wrong. It It and I learned how to set tempos and schedules for the releases. It's just something to try to avoid. One of the things I did with Swotor, which is a good thing, I didn't crunch the team like that in Swotor at all. And it was a huge team, 550 people. And what I did was I actually had better planning. That's why I started out with the start. I had a a release road map for the product. Every 6 months, I went bottoms up on the schedule. I defined what the MVP was. I triaged all the releases to that MVP. I added producers. I broke the team up, since it was 550 people, into 32 autonomous teams that worked on different parts of it. I established a technical review board, and I built dev ops live ops early. All that created the way of actually pacing the team where I did not have to do any long-term crunch at all. There was very little at all putting in effort to get the game

out, which I was very proud of. So, it does work. One of the biggest things that I that I see is not enough pre-pro. I don't think anybody would love to see their house foundation rushed. I know I wouldn't. That's what you do when you push through pre-pro. I've seen so many times people going too wide, too fast. Uh that is something that's very common. Don't do that. Take the time that you need to get your core loops, in particular your core loop, your moment-to-moment gameplay, your inner loops working really well before you move on to the meta, or before you move on to anything else in the game. Case study was Battlecry. Battlecry was a brawler game that I worked on with Bethesda. We were trying to do an online brawler game. No one's ever done that before,

fighting game. And lots of risk. But one of the things we did is we moved through pre-production way too fast. We had blockouts on our levels, we had uh what I call crash test dummies for our characters. We only We only put three classes, we had total of five. We should have put all five classes in and worked out that moment-to-moment before we ordered it up and made it look pretty. And what happened was we lost the core loop just wasn't there when we needed it, and it just haunted us. Excuse me. Sorry about Content treadmill. This is another one. It takes a lot of effort to build these games. These games take a long time to make and to build. And the content side is the reason. One of the problems I had with SWTOR at

the start is we wanted to build a BioWare game, and we wanted to have voice over on all the NPCs. So we had 200 We wanted to build 200 hours of content per class. There were 18 cla- eight classes, so you know that's 1,600 hours of content. We wanted 185 It would work out to be about of dialogue. 4,000 cinematics. The first thing I thought was, "How are we going to get this done?" And what I did, I talked about this earlier, is I broke the team down. I It broke as we were scaling. We scaled from 80 to 200. It didn't From 80 to 150, it broke. I had to redo the way we're we're planning, the way the whole organization worked. It just fell apart. We went from 150 to 250, it fell apart again. I stopped for 2 weeks and said, "No one else is doing anything. We're going to stop right now and figure out how to proceed in a way that we can be

successful and scale." So, what I found out is I broke it down to small autonomous teams that work together. They're interdisciplinary. I broke down my tech because it was 100 programmers. I broke down my tech into seven TDs that watched different aspects of the game in in real time. And they had responsibilities. I had people sit who were had dependencies on each other, autonomous groups that had dependencies on them, sit close to each other. I had weekly scrum of scrums, and I did dailies. So, when I did dailies, the thing I said is it takes 5 minutes to write, 1 minute to read. That's really important cuz I don't want a a thesis. I just want to know what people are doing. Now, why did I do dailies? Dailies were awesome because it allowed the producers to see what people are doing inside all the 550 people in the

studio. It allows us to look at, "Well, wait a minute. Why is he working on this? He's supposed to be working on that." Because there's so many things going on at once. It gives you that micro level detail that's important. So, the other thing I did was I hired contractors, a lot of them. I had 150 contractors on that project. I found I found a code at P partners. I had people in uh St. Petersburg working on it. I had people in Asia working on it. I had people up in uh Virginia working on it. And of course EA, one of the great things about working at EA, they do have a fire hose. Uh the people hose. Um and I had the people hose access. I don't like the people hose access. I I learned that that's really hard to scale very quickly. Um and that's what actually broke the last time. I had the people hose turned on full blast. I said, "Whoa, stop. We're breaking." Um and I outsourced large pieces of audio

and art. That worked out really well. I had a core team of artists that did reviews that also did our the more complicated inter- iterative art, but I also had a ton of outsourcing done on characters, weapons, and all the customization. So, that's how we got that done. Now, what I've learned through doing this so many times is you really have to start generating your content for launch if you want things at D1, D5, D20, D30, D90, or D60, you really need to start that before you ship. That's really hard, by the way. That's not an easy task because you're on fire before ship anyway. And so, but you have to ring-fence some small teams, not big ones, small teams to get the starting point for doing your small updates to your uh monthly updates to your quarterly updates. And then you got to leave the

dev team alone once they ship. They should start planning what the big update's going to be. That work That works well. That goes into what I another topic is your heartbeat. When I talk about your heartbeat, I'm talking about uh how quickly you can respond to fixing something, how quickly you can update and move content through your to the users to the players. It's critical. What affects your heartbeat? Your patch size. I've seen some things lately that blows me away. When I see a D1 patch bigger than the game, I go, "What the hell is going on?" You have to look as building developing games as a service, you have to get disciplined on your patch sizes. Because people don't have the time to

wait for an hour or so to get a patch down. Think about that. You lose people. The more friction you put in front of the people the funnel, the more they may just say, "Hey, I'm going to go play Fortnite. I'm going to go play Division." You know, there's tons of things out there to play. Complexity of your art assets. How quickly you can get new weapons, new effects, new animations, new character stuff, new characters. System design. How quickly you can get things updated in the game and out there is critical. Your pipelines. And I'm not just talking about art, I'm talking about design as well. Branching is another one. Branching is critical. Um there are a lot of friction when some people use certain engines for branching. I'm very experienced with that one. Um you have to think about how you're going to branch when you're live

because if you're on multiple platforms, you have to have multiple SKUs, and you have to have multiple branches. You also have to have as I said, continuous stream of content being developed by three or four different groups as you're working through live. And you have to keep your main line, which is your main thing that you are the players have. You never want to touch that because if you ever mess that up, you have to re-patch the game. That you never want to do. Trust me. And your build system. How quickly you can deploy and build something. I'll talk about one thing that people don't think about a lot until they're up against it. It's hot fixes. These are emergency fixes that you have to do in live. This takes development time before you launch live. A partial build should take no more than 45 minutes to build. A full build should not take more than 4 hours. By the way,

a lot of people I've seen do this, our build takes 32 hours. Uh no. Okay, we got it down to 24. No. Uh we got it down to eight. No. You need to be 4 hours. Because you don't know what you're going to have to deal with on executables when you're in live. A deploy should take less than 6 hours. So, you better have a good QA process on top of that. 85% of the fixes in order to make this happen are on the server, not the client. And they don't revolve changing the executables. Sorry. How is that done? The data is stored on the server. Design systems are data driven. This is one mistake I've seen over and over and over again about people not deve- the tools

for development in design where they have an advancement tree and by the way should not be done in Excel should be done in something that can scale especially a store when you're dealing with MTA microtransactions you better make sure you don't put your all your items in an Excel spreadsheet cuz you will not be able to Excel won't be able to handle it after a certain point in time. Um automated QA Uh how many people have S tests or computer programmers who do QA? See those are lifesavers and necessary because you you know what if you make a hot fix can you check it in time? You better have some automated tools to help you white glove it to see if it works. A scripting language is one way to help uh when you're developing game systems to be able to make fixes on the fly without recompiling the servers.

It should be seamless to the user basically what a hot fix should be. What does a cadence look like when you're in live? You should do weekly patches. Whether you're fixing bugs store updates small additions that I call breadcrumbs to your next kind of beat they're important. Monthly patches weapons abilities level and you whatever you decide. These types of things you look at Fortnite they're getting it. Quarterly seasonal seasons are a great thing. New levels new characters new mouths anything you can think of that fits kind of a nice big kind of not a huge update but a nice little tick on the register to get users to come back. Quarterlies large content features should take no longer than 6 to 8 months sometimes you can wait a little longer to a year but they if you wait longer they should be really really big. The next topic I'm going to talk about is transitioning into lot.

Into testing. You've done this throughout the development process. Your focus test, what I call tissue testing, is basically getting in people that you can just throw away easily to test the game. They're awesome thing to do. And internal play testing. I believe that everyone that's working on the game should play test the game, and I actually have mandatory play test to make that happen. Because if you don't, people won't. People need to play what they make. I don't care if you're an artist, I don't care if you're a designer, I don't care if you're a programmer, you need to sit down and play the game. Now, when you start going into test, you have three more loops coming into play. You have community, the players, you have the customer support loop near the end, and you have the KPIs. When you're building a live services group, this is when you start. When you start going be right before you start, when you're ready to go into test, you start building all these elements

out. Not all of them need to be fully fleshed, but you need to start building these these these services out. Now, what I usually do is I use dev ops and live ops to buffer the dev team while before we launch. Because what's going to happen is you're going to get a lot of feedback when you start testing, external testing, that you didn't get when you internal test. Internal test is not going to get you the same thing that external testers have. Don't make that mistake that think you can do an internal testing group that's a QA group. Now, if you have a company that has over a million employees or more or you can get a million players to play, then yes, that's a great sample set, and yes, you should do it that way internally. If you don't, which most peo- most people don't, you're going to get different feedback than if you were even with internal I'd say with player with a company that big, you could having going out is even better because it's random more randomized and it's not as biased. But,

you're going to find things come up and you're going to have to balance and triage all the things you're finding out while you're uh testing the game to what you're building in test. And that's the that's the hard part. So, I have devops and liveops buffer the team enough so they can do their work to finish the game. Now, things have changed once you get into test. Everything has changed on how you release the game, how you make updates. You're branching now off a main line. You're triaging every release. You're writing patch notes. Community management is a whole uh totally involved in this now. And so, this is the phase in closed beta where you're starting to act like a live team. And all these things are done, need to be done, and this is where you get the team acting like a live team.

Transitioning a team from development to live is hard. Every time I've done this, a lot of people haven't been there and they don't understand it until they are. And so, this is the time to start training them, putting training wheels on them before you launch. If you look at development process, it's now a loop. You're sitting there, you're going through pre-production on your stuff. And by the way, pre-production just doesn't happen in the beginning. On a games as a service, it happens all the way through development. You're constantly doing pre-pro on things. Features to content. Good example, you build a new level, you block it out first, you play test it, then you start putting in your your uh uh uh your level zero to level three to level four type of assets. Um all going to final. And so this is this feedback loop I was talking about. Now that you're starting it. Now you're starting to act like a live development

process. This is maybe basic to people, but I I I feel like I need to somewhat go under this a little bit, go over this a little bit. I'll try to get through this quickly. Closed beta I use closed beta. It's basically means that you have a key and you're invited to it. It's used to validate what you have. You can do a lot of focus testing for your R UI and usability. That's awesome. But for the way the game feels and plays, I think this is really where you're going to get the validation. Cuz I I have done a lot of focus tests that been somewhat biased. The data that I got there versus the data I got when I went live was surprisingly different. And so this is a time where you validate your assumptions cuz that's what they are. And you drive to figure out what works, what doesn't work. And this validation is very important to

figure out if you have a fun engaging game. That's what closed beta is about. Is it fun? Is it engaging? What I usually do is I start out with 5K. Get to 10. 20 and then 100. That's kind of like a good sample size in my opinion for a closed beta session. It's it's it's a good enough. If you can get higher, that's great. In fact, I recommend it. One of the hardest things, by the way, to scale when you're especially if you're doing your first games as a service is how many players you have on the server. It's really hard to break that 100. It It is It is a very painful. Once you break the 100, getting to the next thousand is hard. Getting to 5,000 is hard. Once you start breaking above 50 to 100, things are a lot easier. What happens I warn people about is when you start hitting 200 PCU is when things

start popping again. Closed beta is all about balancing and tuning the content as well. It's about the virtual currency, especially if you're doing and most people who do games of the service have some kind of revenue model. Most of it's either premium plus or free to play. And so, you want to make sure your virtual economy is working well before you start charging people real money. So, that's important. Player wipes are possible during this phase, and I want to warn you about that. Every time you do a player wipe, you're going to have a problem with retention. So, be careful with that one. Telemetry is very important, by the way. Designers always ask for a list that's 18 pages long, and I laugh at it. And I said, "Cut it down. Pick your top." Uh cuz you can all you get really bogged

down in data if you're not careful and not have the the capability of storing it all. It's not 24/7. I usually don't run these things 24/7. They should last 6 to 8 months. Anybody that does it less than that is asking for serious problems at launch. Especially for an online persistent world game, even a session-based game. It's Online games are complex. They need the time to work through the gates of how many PCU, of how your technology will scale, and needs the gates of how your design and game systems will scale. You need those gates. And closed beta is where you practice and go through that. Open beta is not doesn't have to be that long. Open beta is really try before you buy. It is you better make sure that game is extremely polished. Really what you're doing in closed beta open beta is seeing how far you can scale beyond on your PCU limits. Is really push it hard because you want to make sure if that thing is going to

break, it's going to break then. And how your patching and distribution is working. How all your monetization is working. You actually turn on the storefront for the first time for the players. You do that earlier later in closed beta but really out there now. There's a tendency to have to refund by the way and test. You should make that very clear to your player base. Try not to but there is always that tendency of rebalancing. And I usually start out with limited sessions and go near the end to full-blown open. You need to practice that full-blown 24 hours to see where your memory leaks are, your CPU utilization and everything else at scale. So it's important to get to that point. The reason I do durations at the beginning is so I can get mass amount of people in at once because if you have an open, you will see your curve start driving down your PCU curve. And so what I want to do is close it, open it up for a weekend and get that mass rush in.

This is another thing I want to talk about. There's a lots of ways of measuring KPIs in a game. But they're extremely important. Game design for instance, you have a lot of telemetry. These are the things that you're looking at into with the telemetry on game design. They're just as important as your business KPIs. By the way. And they will develop KPIs of their own for design. What they do is they set the goalpost for all your phases of test. So, what you should do before you go into closed beta is when should we go into closed beta? We have to meet these key indicators. When we go from closed beta open beta, we make sure that we have these key indicators. Sorry. Let me get this right.

These are the key indicators that I look at. Ratio to invites to logins, you can read the list here. The D1 is really important when you're in closed beta. How many people come back at D7 to play your game that went in from D1? How many people after 15 days, and I try to keep it up at least 15 days on closed beta at once to see how I'm tracking there. I occasionally have a test server, which I set up, and I'm open to all the time, but I have exclusive invites into that test server. So, that I have a a subsample of people playing, so I can get my D30 going on. And that test server, by the way, is something I don't see many people do, but I highly recommend that goes out in live, too. It's where you put your latest stuff on. Yes, the user has to patch a new client to play it, but it's

so worth it. The you don't have a lot of testers testing there, but the amount of testers you have in the feedback loops you get are amazing. It's well worth it. And if anybody understands what K factor is, K factor is the viralness. This is really a key indicator as I'm moving towards open beta. The first question I ask is how many people would recommend a friend to play this game. And if I get a high percentage of people saying yes, that's a good sign. A lot of times people say if you don't get over 90% everything else is irrelevant to say that I would give recommend to a friend. In Swtor I did this and 80% 85% of the people said I'd recommend it to a friend. That was a good indicator for me. Key factors another one is how many keys I give out to friends cuz I always put friends keys in there. How many friends are playing that actually got the keys and logging in. That's a good sign too to track.

Predictive analytics, I can't talk about this enough. I had it in Swtor for the first time and this is in 2011. I had forecasting models built. I had trend analysis being done. I actually had some people from USC come in to my data that I was collecting in live and do machine learning for me and they would predict for example, in Swtor we had farming problems. They wrote a machine algorithm that tracked how farmers were doing things and learned from that and would temporary ban all the farmers. With when we put that in, 80% of the farm 85% of the farmers went away. 85%. If today this should be common. Anything you should do, try to use machine learning to do it. Another one they did was predictive churn. They could predict through player patterns when people would churn out. That was awesome. It would initiate a CRM

customer relations management system, would put out an email to go tag that person and say, "Hey, come back or what's wrong?" with a survey. All automated. Another one is if you want to do things for like bots. Bots are big. Bots can ruin an experience on a on a a session based uh first person shooter type of game. If you can do machine learning to automatically temporary ban those people and have CSB called, that is well worth doing. Anything that affects the player in any bad way for bad behavior, you should automate today because you can. I didn't have that ability on UO and SWIG. I started it with SWTOR and I see where it's going now. And all the talks being done about definitely look into that. It's well worth it. One of the things I I talk about and then I'm going to blast through this is what you need to have for live. Look at the list here. You need to have the ability to record

bugs in the game. A lot of people forget about that. No, you need to have the ability to automatically build a system where you have a slash bug or something like that. They automatically get routed to a specific place that QA can look at and validate. You better run be able to have surveys in the game that you can pop up after after an a mission, after an encounter to see if they liked it or not. Do that. All the community management tools, you define your KPIs. This is all should be done before you open up for closed beta. It's the same thing for your survey to your website. Your virtual store needs to be operational. An escalation plan for community and for live. What do I mean by that? If something happens, you need to have a plan to deal with it. Open beta. MTX storefront should be live and operational. Everything should be working like it does and this is in closed beta that you're launching like you're launching everything in the cloud, all your

platform connections, everything should be fully functional by open beta. And that is in closed beta by the way as well. CRM tools, CS tools should be all online. Scenarios for community management is real important. I did scenarios with legal to say, "What if this happens, what should we say?" Let me tell you how valuable that is. You as a community management team could be caught cuz when something bad happens on a service, it goes to all the media. If you have a a a reactionary plan in place that you can immediately comment on without legal review or executive review, that is a very good thing to have because if you don't and they take a long time for someone to respond to that, it just inflames the community. Again, community management. I I I'll just say one thing about community management, you're in an emotional relationship with your community. It is important to not treat them like

PR piece, a PR person ta- talking to them. Talk to them honestly, openly. Talk to them not like you're doing a PR beat type of thing. That's where a lot of single-player products companies get involved in the community management is they do it like PR beats. No, you have to be constantly involved with the community. Think about Twitch and streaming and all the things we have available social media. All should be utilized for community management. I'll leave with this one slide. I said this probably a hundred times in my development career to a lot of different execs. You only get one opening night. If you have a good launch, here's some hypothetical curves, you're doing really well. If you have a bad launch, that potential, that gap where that curve is and where you're you see where it peaks out, that gap, you missed the moon shot. You're never going to get that back.

Can you recover? Yes, there are lots of instances of games out there who have have done very well in recovering from a launch that the game didn't meet expectations. However, you'll never meet that bar, the moonshot type of revenue potential that you could have. That is something everyone in this room really needs to think hard about. Because that will either damage your franchise that you're working on or your new IP. And it also could, if it's bad enough, if your core loops weren't done well, and you had to do major surgery on the game, and you had a sentiment running bad for 90 days out, you could start damaging your cohorts, your zero cohorts, your first people in the game. They're your stickiest people. You could damage, you could go right into that curve and all those people, and you'll never get those people back. And you risk shutting down. One of the things is SWIG.

SWIG had a horrible launch. The It melted down. It had lots of issues with the the world feeling not filling up with content because we only had Well, our tools came online 8 months before launch. So, we did something very risky in the development of a platform, but that platform was amazingly powerful. That's a whole another discussion I could talk about later. But, there was risk there. And we didn't do enough stress testing. We lost it. We had 5 million people on our site uniques before launch. We lost easily half of first round of cohorts, easily that week. We never got it back. We stabilized around 350. That could have been a million player game if it was done right. And had another 6 12 months, we really did need another 12 months of development on that game. We did everything from beginning to end on that

project in 2 years and 8 months, which was amazing on new tech and everything else that we did. But, we needed another 12 months to ship. So, that that decision for a long-term bet wasn't the best. But, there were a lot of contractual things that again, let's don't look in a vacuum. There's a lot of contractual things and business cases that don't allow you to do that, right? That's the reality of some of that them some of that I realized. But, think about what does that mean? So, if you're know going into it there going to be problems, realize what it does to your revenue. Face it. Fallout 76 is a great example. You can't test a game of that scale and magnitude in less than 2 months. It can't be done. You cannot rely on your external testing group to get a game like that and all the bugs. What that did is dinged that franchise.

I don't think it devastatingly hurt it, but it dinged it. And they started, as you saw that initial curve of the bad launch, they went down very fast after launch. They'll never recover where they could have been. They never got They'll never get the moon shot. And one of the key decisions was is this too many bugs. They were trying to fix too many bugs at once to try to make up the time, which was bad because then if you fix too many bugs you can't check all those bugs thoroughly and you create new bugs. It's important to realize where you are and how you proceed. You can that you can turn things around, as I said. The expectations on that game was it was going to be a Fallout game. Everyone knows NPCs are in Fallout game. That is how people play Fallout. They love it. And in console world, this is a fact, uh known from watching different games

is almost over 50% of the players in console play single player. Which was surprising to me. And because of that, if you knew that kind of quantity and that kind of people were single that you need to have single player content that represents that particular IP. And so those are the things the lessons learned on that. I think it's really important cuz it made the feel made the world feel empty. So everything you do affects you in development, affects you in live one way or another. Decisions you make. Now I open for questions. So it'd be a good time to pause. I know there's a lot of stuff here and I wish I had more time cuz I had I had over probably 150 slides I could talk about and I just had to call it down. There's lots of things I could talk about. Anybody have any questions they want to ask? I can't see you really well. Um

Thank you. I I will say please fill out the surveys on there. It's really important that I get feedback even for me to help me improve. Um and provide me with that kind of feedback would be very good. Anybody want to step up to the microphone and ask questions? This is a great Q&A time. I highly recommend you ask them. Yes. Uh hi. I had a uh I had a question on your uh your analysis on The Old Republic, Star Wars: The Old Republic, about the decision you made between the HeroBlade engine and the Big World engine. Yeah. I mean, I obviously you did uh you know, pre pre studies on what the advantages and disadvantages of those particular engines were, but uh how would you have done those pre how would you do those pre studies differently now seeing the uh consequences that resulted from that? Uh I would have gone with BigWorld cuz it's more like what I've used to with UO. Uh it was a seamless server engine and had a scripting language above it.

There was an issue with BigWorld with with source control on the seamless server boundaries, but I think we could have pressured to to work through that. That was one of the decisions that's one of the decision points of that. Uh but I really think that choosing the HeroBlade engine just was not going to scale and was not going to It was a great prototype tool. Beware of that. Don't pick an engine because you can prototype quickly. Pick an engine that will scale. Okay, if you can pick an engine that if you're on the consoles, if you're going to be AAA, that can do AAA develop uh development and including look uh and be able to run at the frame rate that you're going to need to do. If you're a shooter, if you need to be 60 frames or at least no less than 30, you need to pick something that can do that. If you're doing something that's both glass as an input, controller, and keyboard, then you need to pick something that can do all that, too, and scale with you. Yes. Uh hi. I'm over here. My name's Blake Mizzi. I'm from League of Geeks. We run our

games as a service across PC and consoles. My question for you is how you evaluate your content through your monthly's and your quarterly's. Um cuz you're generating that content on the fly very quickly, how do you lock it down, test it internally for its effectiveness before you launch it to the public? So, that's a lot of what we do is we think what we like to the public. We use uh we see what players are doing. So, there's a lot of telemetry in design to see what players are doing. So, if you look at that telemetry thing, you see the players' path in the world, you see what players do, uh and what abilities they use the most, what features they like the most, what customization they want the most. And so, what you want to do is you want to tune uh your content toward what you're seeing uh in your telemetry and design. What players are migrating to? Um one of the things I use telemetry for, too, is it it helps validate some of your design assumptions. But, one of the things I think is very important that I think I want to tell people is how something feels may not necessarily equate in telemetry

or KPIs to what you want to do, right? It's sometimes the way it feels may over it may overweight your decision for that versus true data points, right? I use data as validation points, but to me it's how it feels is more important. In a lot of ways in game design. If I can ask a quick follow-up, do you beta test your quarterlies and your rolling content? All right, so the way we do that is we have uh QA embeds that write test plans for those. We run uh bot testing on that to make sure it's it works and everything. And then we go out and we put it on the the test server I was talking about and see how players like it. Right? Uh we do uh some focus testing on that, but like I said, uh tissue testing is what I call that more, friends and family and things like that. But, a lot of times putting it onto a test server out there and letting the players see how it does

is the best way, in my opinion, to get the feedback loops you need. And also talking to community, too. Asking them about certain things and getting a community involved in some of the design stuff isn't a bad idea. Excellent. Thank you very much. Yeah. Hi Rexton, thank you for today. Can I ask the question around your uh technical review board? Um, I'm just keen to understand how wide was that board, who was on it? Um, what were you looking from that board? Yeah, so the technical review board had one person was the TD for game game systems. Another one was TD for platform. Another one was TD for um the client. Another one was TD for the server. Another one was TD for the database side of things. Um another one was um let's see that TD for the analytics side of things, right? All the all the things we had to track, how do we track it, making sure everything was tagged appropriately, getting into the database, the ETLs are done properly. Um everyone had a division of area

responsibility. And that worked out really well. And Wednesday they met every week. Right. And they started triaging together. Uh that helped with the large team a lot, actually. Instead of making one person owner, no. We had seven. It was awesome. And and dividing it up like that was great. Cuz everyone has different problem sets that they're dealing with in that team. And talking amongst each other, they work as a unit to help solve it. That's great. Uh thank you very much. Appreciate it. Yeah. Hi. I'm a community developer uh on For Honor at Ubisoft Montreal. So, I've lived the live life. Uh so, this is super interesting. Thank you very much. Uh my question is do you have any insights into doing good patch notes? Cuz that always seems to be a nightmare. Yes. So, one of the key thing requirements, and thank you for saying that, that's why I love Q&A more than anything else, cuz I kind of forget about certain things. Uh your Anybody that's doing coding, you

better have your comments right and formatted in a way so that I can pull out changes that I need uh for what's changed uh in code as well as design as well. So, Confluence is a great place to start doing that. Uh we use Confluence. Uh we have Jira. We have Shotgun. Uh we use GitLab as our If anybody doesn't know what GitLab is, it's a great It's a great um version control tool, but boy does it have a lot of power behind it. Uh as And we're trying to get rid of P4, cuz I hate it. Um Hm. Perforce. Um cuz it's antiquated. I hate to say that in a don't a lot of ways, but it is antiquated. Um, and the new tools out today, it is important to make sure that people tag their changes is basically what I'm trying to tell you, so you can easily pull it out. And it and uh it is a hard that's a hard thing. By the way, any big community manager understands building patch notes is hard cuz you're wrangling cats.

And and sometimes you miss things. Yes. Yes. Uh you spoke about uh outsourcing to USC as the machine learning some predictive data modeling. Yes. And you talked about how you recommend that going forward for games as a service. And I'm wondering if that's something that you see coming on to the team where it's valuable enough that you would have be data science on the team and you're iterating those models going forward. Absolutely. Absolutely. And there's there's great um they're great Python is actually a R plug-in for Python or light plug-in for Python actually that does all the statistical analysis and everything in Python cuz I could find Python programmers a lot easier than I can find uh R programmers. Uh but yeah, anything machine learning AI wise, yes. To to do and I use to do the predictive behavior in the and be able to ban for bad behavior as well as do CRM initiatives, absolutely think that a data scientist on a team is part of the crew now. And should be by the way.

Okay. Thank you. But I I don't think people know how to look for that type of person. I really don't even know how to write the JD for that type of person actually. Uh and and and understanding how to hire someone like that because they'll look at a JD, well you have no gaming experience. No, that's not what you're trying to see. You're trying to see what the what their root of what their experience is that you can apply to gaming. Right? I I had to write a JD for that and it was really hard to me to get that approved. I agree. It's at the tip of the spear right now, but I'm I'm excited to hear that. Thank you. They don't see the value yet, but it's just like for example, I'll tell you with Swotor, when I left, they they owed some money on that, and they decided it wasn't worth doing all the way through. We had it all prototyped and working on CRM stuff, and they said it wasn't worth it. And I said, "How could you say that? That that amount of money they're asking is trinkets to what it will do, right?" But again, that was 2011. Um that that was done. I think today I I

probably not get that anymore. Great. Thank you. Yeah. Hi, Linda Payne from Bungie. Um just to follow on, I wanted to know if um in moving to games as a service, if there are new roles that need to have prominence, and that need to have and how that power shift might happen, and also is there a different ratio of types of roles? Yeah, so QA to think QA is on the outside on a games as a service group, that's insane. QA needs to be integrated, part of the leadership of that team, not a lot of I've seen this happen, I'm not kidding you, where they're not in the leadership when I and I say, "What are you doing?" QA is the person, the group that says, "Are you ready to go or not?" They're the ones that push the publish in the release management structure to the service. They're the ones that should say, "No, it's not ready." But that that's a lot to get to that's a lot to absorb to a company that's used to making

not games as a service products. So, QA is one of them. The data a scientist is another one, data analyst, right? The person that does the ETLs and everything else that you hook in all your telemetry and gathers that data and interprets that data for you and develops your dashboards for you. That person is extremely valuable and should be on the leadership of of a games as a service team. Having in beds on your QA team to doing different things that from platform to development or all important people and I would say the one thing that's missing too on your leadership some people need to put community management on their leadership board right and it they the community the lead community guy should be on the leadership board and have a voice in saying our sentiment is this it's negative now it's turned neutral now it's positive these are all indicators that you need to know and measure. So yeah I would say first and foremost

is QA and how you treat QA the second one is data analyst the person that actually puts the details in and tells you what you can or can't track are very important. And which roles are reduced in importance I mean that's really kind of elephant in the room a bit when you transition Um so I would say the live producer has a stronger voice than the lead producer. Mhm. Uh in development um at that point when you're when you're live cuz they're right up to that right they have everything in responsibility. Uh I try not to separate live when I say live development team is part of the live process they're not separated out. They're everyone that's working on that game is live. A lot of times people try to make this little wall between it it's in No no no it's all live we're all live we and I rotate people in and out of those other teams as I was telling you it's

real important to do that. As far as hierarchy goes I think it's important to establish the right hierarchy in the beginning as you move forward and all you're doing is hiring people at certain times to take those positions that are not filled because you don't need them right now. It's the way I I would respond to that question. All right thank you. Thank you. Good. I'm sorry I can't see you really well with these lights. I apologize. so. Okay, thank you. Okay. Uh uh my name is Wei from NetEase. So, you talked about the hot fix. Uh I'm wondering, do you have any specific process uh between the hot fix and the the branches? Because in real life, sometimes you have a uh dev ops team is working on the content that you are going to release uh in the future. Uh at the same time, you have your live ops team is working on hot fix on online, right? That's correct. So, so do you have anything to share about the So, the live fix is going into main. Right? It'll be going to test. It'll be put into test. Mhm.

And then when it's ready to go, it goes into main onto the service. Uh so so mean the hot fix? The hot fix, correct. It is a QA branch, a test branch. Okay. That goes into that gets tested, white gloved, then it goes into main lines. So, do you consider you you fix the hot fix on the on the live branch and go back to a main? Because sometimes the hot fix is fixing the very urgent issue. So, you want to be on the for the The live The live, by the way, the live branch is the main line. Oh, okay. That is There's no difference between two when you're live. That was why I telling you the big transition when you go into test for the first time is your main line is your live. It It That's it. That's when you really kind of That's when the main line becomes real to the team. Okay. Okay, thank you. Yeah. And sorry. That's okay. So, uh hi. I'm a designer from Vicarious Visions, and I'm curious on your thoughts on burnout. Because as a studio, we try really hard to keep that

from happening, but it does occasionally. And I was curious if you had thoughts of like if you see people burning out or someone burning out, do you have any thoughts on how to bring them back from that? Or do you generally find it's like they're out, they're burned, they're done? No, so what I do is I have my leads do one-on-ones with everyone at a at a week cadence. And it you can see that. I walk the floor personally myself and look for it as well cuz I kind of know what to look for. And I try to train my leadership team to know what to look for when burnout occurs. Cuz some people like to work no matter some people have these hours that are crazy. There could be life issues that cause that whatever they don't want to go home or whatever. And I I literally have to force them out the door when I see that. Right? Stop. Don't do that. You're not effective. And and and a lot of studies have shown not being effective once you reach a certain point anyway. So why you do that? I I'll tell you what causes crunch. The number one thing that causes crunch in my opinion is top-down schedules.

If I had to say anything it's top-down schedules. Mhm. Not bottoms-up. And that's why I try to do the discipline of every 6 months in a 3-year project to do bottoms-up roadmaps to triage against to figure out where my risks are. Okay. Thank you. Yeah. Oh, thank you. Sorry. Hi, thanks for the presentation. My name is Elliot from Square Enix. Um when you said uh Swiggy at launch the back-end melted, was it due to uh poor capacity planning or elasticity issue and how did you guys recover from that? Uh it was poor uh poor planning. Uh basically they were confident because of EQ that they could launch without any problems. Didn't realize I told you there were 500 5 million uniques hitting that website, right? That's if that's not an indicator a KPI to look out for or or a spot to look for, you better be careful. And so that uh

came as a shock. And in the end it melted down the whole Apache distribution system that we use for load balancing and things like that. It just it collapsed. And it took about a day and a half to get fixed, which is a denial of service for a lot of people. It was just uh it was very hard to watch from a development point of view cuz that's platform, but it it gave me a a big lessons learned of what to do when testing before launch. I literally have done this on a platform that we're launching on. I literally disconnect the login servers to see if they roll over properly. I've actually put bugs in the game to see how hot the hot fixes work and didn't tell anybody. I actually did that. And I want to see how quickly cuz I knew when it was going in, they'd find it, fix it, and get it out. Good. Is it severe bugs where you had to redo recompile. I did a where you only had to re do it uh not a full compile uh not a full build, a partial build. I did things like that as well.

And I found out when they had to do a partial a full build, it took 14 hours. Mhm. I didn't know that. They said it wasn't. And I and I I don't believe them. I trust but verify. And I actually had engineers do that for me. I highly recommend you do that. Trust but verify. Thanks. Just quick follow up if uh if if uh the environment had been more elastic, perhaps, would that have helped you recover easier from Yeah. Well, man, this is 2003, okay? Everything was starting to get virtualized by that time, right? So, they didn't have a lot of things we have today that makes things a lot easier. God, I the I mean, when I look at platforms and we're going, right? I mean, I look at C++ we used to use or Java. I use Golang now, right? I use GitLab. I use Kubernetes and all the other things that exist today to help you push out push out distribute and have a service layer that's agnostic to everything that can attach to it. It's awesome. And virtualizing everything you can do today just makes things so much easier

uh than when I started. Thank you. Yeah. Hi Rich. Uh I'm my my name is Dominic. I'm the QA manager at Square Enix. So, the presentation was awesome and I love hearing the questions from fellow Square Enixers. One of the things I wanted to touch on was just the the security point that you had there. Yes. What was the general split for you between the infrastructure level security testing and the application security testing and the timing of that? Yeah, so basically one of the problems with game programmers, especially doing game systems, is security. They don't realize that they have to validate certain things. And so one of the hardest lessons to teach games as a service when people haven't done it before is to learn how to do that server validation. It does It has to do with movement. It has to do with collision. It has to do with if they have a system that is dealing with advancement that there's checks to make sure people are not out of bounds and all sorts of things like that. And

so I had at Square Enix or I hired a security officer. And the security officer did two things for me. He hired a team. I wish I This is one of my slides I didn't get to. It's like one of those things I threw out. But I had a security team with a security guy I hired. He did both external security and internal game security. So, for example, he locked down all the all the USB ports. Right? In the company. He made sure that everything had dual off. Right? We went to the service, the cloud, everything had to be dual off, right? He was responsible with his team cuz there was facilities and then there was IT and then there was the game. Because if you're going to get hacked it's going to come from outside into your IT infrastructure, not into your game. I guarantee it. And so that helped us out a lot. And then he had to make sure the game programmers he had a person responsible

to work with them to make sure things are tidied up with assets to to do the testing for them. But again, you'll miss it. I mean, every time I go out into live, there's something that happens again, "Did you not put the no, I forgot it." Because you have hundreds of systems that are interrelated, especially in RPG. Um cuz one of the things I'll tell you game system-wise and on games as a service, it's so much of this. Interconnected. I've had things that I fixed on and we checked and when it went live through some kind of scale issue that we never discovered until you got scale, something happened over here that had nothing to do with we had here, but there was a global variable that was called upon and we had no idea hit over here that caused a massive problem. And it took a long time to trace back because it happened at scale. And you can't test even all the bots in the world, no offense, cannot test things at scale. And this is what I talked about things popping. You are going to have

that, especially if you start hitting into the millions of PCU. Anybody else? Thank you. Thank you. So, I can't see. All right, thank you very much.