From Box Products to Live Service: How 'Destiny 2' Transformed Bungie

[Music] thank you yeah so let's uh let's talk about Destiny so I'm I'm Justin Truman I'm the general manager for Destiny 2. I've got about 20 years of Game Dev experience on consoles and PC and the last 12 years I've been working on Destiny at Bungie and on Destiny I've worked on a bunch of different roles engineering design production business now and so my goal in this talk is to talk about Bungie's Evolution as a studio and a really cross-disciplined way it's also worth saying while I'm the only one up here speaking right now every learning that we're going to cover today was the collective contribution of a ton of like really talented and insightful Folks at Bungie so the only

part that I'll take sole credit for is any mistakes or dumb statements that I'm about to make and the evolution that I want to talk about for the next hour is our journey from our box product Roots as a company to the live service team that we have today and this journey really catalyzed in some some kind of painful ways with the launch of Destiny 2. so way back in 2017 we launched destiny 2. and at the time Destiny 1 was still a thriving live service but we had committed with our publisher to release a full sequel to that live service that we had just started nurturing three years prior and so so for us at Bungie we knew that if we were going to have to SQL we wanted to do it right we wanted to make something that we and the entire Destiny Community were excited about because we liked a lot about the

original Destiny one but we knew that it had a reputation as a somewhat flawed game like it was really fun but it was also pretty nerdy and hardcore and so we wanted to turn that on its head and make a much more widely appealing version of Destiny so we built a better campaign with a much better narrative we turn we tuned to the rewards to be faster and more generous with more instant gratification our primary razor during development was that we wanted to put a two on the box you can see how it kind of subtly influenced our marketing treatment leading up to Launch and we did it we launched two great reviews Destiny 2's Metacritic was more than 10 points above its predecessor and our players and our fans told us that we've made something more appealing more polished and more rewarding and that was also reflected by our

in-game data because Destiny 2's player engagement was both broader with more players and deeper with more hours than it had ever been in Destiny 1. for about six weeks and then issues started to emerge with the first signs being our most Avid hardcore Community running out of content this then rapidly spread to broader more casual players and eventually these problems spilled all the way out of our community and started creating a bad reputation for our brand in a major way even amongst Gamers who didn't play Destiny and that threatened the total ecosystem of the game as we first frustrated and then shed our player base to show this visually so this was our weekly active users for Destiny 2 in the first few months and for comparison this is what Destiny

1 looked like so so far so good but then this started happening and this moment right here in February of 2018 was [ __ ] terrifying so this graph shows the trend of our weekly active users and at the rate that we were shedding players we did the math I remember being in the meeting we did the math and if this continued for just five more weeks we would run out of players the entire population would be gone and we were seriously like looking at potentially being one month away from having to just close up shop on Destiny 2 altogether so that was in February of 2018. now fast forward four years and we're on track in 2022 to have our most successful year ever in the history of the destiny franchise we've had our

highest critical and Community reception in the history of our series and we're on track to have the strongest business year we've ever had and that's that's five years after launching Destiny 2. that's eight years after the launch of the franchise and that runs like totally counter to the AAA Peak and Decay model that I've experienced with every other game I've worked on in my career so what I want to talk about for the rest of this talk is what had to change about Bungie to go from here to hear and this moment when we were literally five weeks away from shuttering Destiny 2 as a product it was it was like exciting it was intense it was terrifying and I'm going to talk about it some today but more than any one moment I want to talk about the long multi-year process

that this moment catalyzed that required us to rethink so many of our development practices and our studio culture from the ground up over multiple years and it's core this is this is about the transition from a box product mindset of developing games to a live service mindset and so what do I mean when I say a service mindset and I'm not talking about like the product definition here like I think we all know what a live service game looks like on the product side but what is the mentality the culture the practices that let you win in this environment and so for the rest of this hour I want to try to Define what we had to learn to get to where we are today and each of these transitions that I'm going to be talking about they were really hard for us to First identify and recognize and then even once we understood them they were still hard to navigate and transform to as an

organization and so I think that they're each pretty Universal in application so I'm hoping that we can help anyone in the audience who's undergoing similar Transformations on your team at least to get past that first step of identifying and recognizing these and then the actual transformation is still still pretty tough but but actually I want to go back I want to go back a step earlier because before we can Define the end point of what a service mindset is we have to define the starting point so what do we mean at Bungie when we say the Box product mindset for developing games and I would say for us at Bungie like Halo was our canonical example of AAA box product because when we built Halo we found success on every axis we had critical Acclaim we had broad popular appeal cultural relevance High sales sustained player engagement

Halo was a winner by every possible definition of success but unfortunately that meant that we didn't have to think too deeply about which of those successes really mattered by contrast Destiny 1 was the first game that Bungie made in well over a decade that didn't receive Universal critical acclaim and this was like a really big blow to our studio culture at the time because in many ways it felt like failure and what we didn't realize when we launched Destiny one was that Destiny was already becoming a success in the ways that mattered engagement was the true measure of success for a franchise like ours but we didn't we didn't know it yet and a lot of our efforts to make a better Destiny 2 were because we had a wrong internal definition of what success was because you can you can fix your review scores you can fix your sentiment you can even fix your sales as long as you

have engagement as long as you have a community of players who care about and are passionate about your game even when that passion is being expressed is like anger or frustration anger we eventually learn like anger is not the opposite of loving a game anger is like two degrees off from loving a game and they both come from Passion from people who are highly engaged with what you're making and the opposite of loving a game the thing that could kill you is apathy and that's so much more scary than anger but I'm I'm getting distracted so so from way back in the Halo days all the way through Destiny 2 launch our process was pretty straightforward we would hire the best talent in the industry we would give them a lot of freedom to use their creative guts and their creative process to make whatever they considered to be the best game

possible we'd release that product with a big bang and the largest marketing push we could manage and we trust that success by every definition would follow from that and this was all summed up at the time by our studio motto we make games we want to play and this is in the wall in front of our studio and we believe that when you trust your gut on making something awesome making something that you can't wait to play that all of the different forms of success that you might want critical Acclaim sales engaged players those all just flow naturally and intuitively from that and this picture is old and grainy because it's no it's an old picture actually found on my cell phone because we took down that motto a little while after D2 launch as we evolved our view of what success really meant and our new studio motto this is a more recent picture is we create worlds that Inspire friendship

and this new studio purpose has a bunch of benefits over the old one like including the fact that we make games we want to play it sounds a little too much like it's just like gatekeeping for bald middle-aged white guys like me but besides adopting a more Global and inclusive view this is really about a shift away from that box product mindset into the service mindset and again I'm not talking about the product design I'm not talking about business models or like free to play or micro transactions I'm talking about something much more fundamental to how we think about success like if you think about the best AAA box product games out there like the last of us is the tsushimas the Eldon Rings there's plenty more that I can throw up on here Horizon like core to the Box product mindset in these games is the goal of exceeding your expectations in every way across every axis they blow your mind with their scale with their

polish with their quality at their best they often almost feel like perfect games but we all know that these products don't start perfect like one of my favorite old quotes from a Pixar interview that I love is all of our movies suck at first which like it sounds weird it's almost it's almost incomprehensible when as as consumers all we get to see is this near perfect run of artistic products coming out of Pixar and now Disney but we're all familiar with what the creative process is actually like behind the scenes that nothing starts great nothing starts perfect making anything great is always about iterating through failure after failure making it just a little bit better each time until you finally have something incredible and this idea of taking a game taking any creative idea and doing all of that

iteration in secret at your studio so that the only thing you release the only thing your audience sees is near perfect that is what we're talking about at Bungie when we say the Box product mindset because you iterate and iterate guide guided mostly by your talent and your gut until you have something amazing then you release it with this giant marketing explosion and hope that it blows all of the doors off and and to be clear like I'm not knocking this process it's amazing I have tons of respects for every one of these Studios every one of these games like this this idea of seeming like you never miss is why I originally applied to work at Bungie and so I'm not trying to criticize it I'm just setting it up to contrast it with a different approach that we've now had to adopt for Destiny so so if that's the boxed product approach then what is the service approach and mindset by contrast and more than anything else being a

service is about being fast for a live service more than anything more important even than quality a live service is about being fast like back in 2010 you could you could have a long lasting high retention game just by releasing something that was super high quality really well balanced and unchanged that players kept coming back to for years but but to stay afloat as a service today you need to be constantly feeding your players you need to be responding to their concerns you need to be making the game better and so if box products are in the business of over delivery live service is the business of constant delivery and to provide that constant delivery to players you need to be fast you need to be as fast as your organization can possibly be

and so a Bungie one of the key lessons that we codified here to shift our mindset is that velocity is more important than position like your position is your game's quality right now how how good is it what issues does it have how much how much content does it have to engage your players and these are all important questions but so much more important for us now is how fast can you change that position if your players need more content how fast can you funnel it into your game if your game has an unbalanced meta how long does it stay on balanced and if your community has a valid complaint how quickly can you respond to that complaint and this is why in this important but like unintuitive way I'd rather have fast than perfect because being perfect even even just

being great is your current position but fast is your velocity and a fast game will become great but a great slow game will very quickly fall behind so like let's tease apart a tangible example for us at Bungie this shift in our mindset to focus on velocity has meant that for our future games that we're building we mentally shift a lot of our Focus away from the launch of the product because like all of our industry experience and training all that box product training has us traditionally laser focused from like day one on having the best possible game on the day it launches that's what you're losing sleepover is like what's gonna happen when we finally get to launch day but if you're trying to make a live service we found you should actually kind of de-emphasize that launch in your mind like a much more important question is

what is your roadmap and release tools going to look like for the first year after your game launches because you can launch with a game that blows people's minds and is super high polish or you can launch with a game that has lots of promise but a bunch of like janky issues on day one and the sifter that's going to determine which of those two games is relevant 12 24 months later is their velocity how many updates came out how much better was the game getting each month because that is what builds and retains fans in 2022 and if you're optimizing for that then the position that you're in on the day you release becomes less and less relevant because it's just one beat among many on your way towards a perfect game and the reality is like if you try to launch a new live service game in the 2020s it's kind of like that

Pixar quote like a simple fact is your game is probably gonna suck at launch and that's and that's okay like launch is not the end of your journey launch is the beginning like no new game iterated on in secret for however many years can show up and immediately beat the existing live services that have had the advantage of being able to iterate directly with their Community for five or ten years you don't you don't get to win that battle on day one but if you're focused on your velocity not your position at launch then the better challenge becomes how fast can you improve your game once you are in that same direct dialogue with your community how quickly can you change your existing roadmap if you find something in the live game can you deploy weekly patches daily patches if you have a major exploit in your game can you identify the issue then triage it test it and deploy a

hotfix in 24 hours because if you're focused on those kinds of challenges building up that velocity before you launch instead of just focusing on your position at launch then your game will get to perfect with shocking speed and you'll be making a game that is perfectly tailored to your community and built to last so a bunch of this like the big cultural hurdle for for our team and and our organization one of the main things that I'm going to keep returning to is about redefining success like what does quality mean to you and to your team and I don't just mean in terms of like business goals or kpis or something like I mean I mean in your bones like in your heart why do you get up and go to work making video games every day and for us at Bungie for a lot of us like it used to mean that feeling of of

seeming perfect of being impeccable being uncriticizable that is all part of that box product mindset but now we're instead focused on being fast and the Really the really tough part here though is that to do that you have to get comfortable with public failure like you have to get comfortable with all those failures and those iterations that Journey from your game sucking to it being great more of that happening out in the open instead of in secret at your studio and this trade-off is worth making like it gives you it gives you speed it lets you feedback and iterate and and get better faster it makes the final product even better but you have to be willing to take those lumps which are a lot if if your ego like like ours if if your definition of

success as an artist is tied up in that Mystique of only letting your failure happen in secret to give one more example in this space like broader than just developing a game like I believe this shift to an ongoing service oriented iterative conversation is happening in every sector like even with people's personal Brands this is why you're seeing so many more like public apologies from from corporations from celebrities whomever it's more and more of that failure just like we had on Destiny happening out in the open and in success it means by embracing this service mindset you're getting to a better position even faster through that feedback through that critique and that extra transparency that's honestly like I think not optional anymore to any highly visible brand all right so that was our our first big

learning that velocity is more important than position for the next part I want to dive in deeper into what it means to really redefine success like I was saying about the Halo days when you're when you're really successful it's really easy to not bother with a precise definition here and I'd wager that for most people in this room you probably define success as some combination of two things whatever your chosen version of quality is like that could be steam user reviews or Game of the Year Awards or Metacritic I don't know do people still use Metacritic I'm I'm old um but but whatever that definition of quality is mixed with some definition of reach like that could be financial success it could be unit sales or it could just be how like culturally impactful or big your your game is with

your chosen audience but in order to adopt a service mindset we've had to get a lot better at defining multiple axes of success and doing that very precisely so let me start with our first tangible example this is when we really started learning to break apart and be more precise with our definitions so remember this terrifying moment I was talking about in February of 2018. at the time living through it felt like a stressful real-life version of FTL if you've ever played that game like if your engine room is on fire and your O2 is draining out through a hull breach and your Shields are down which of those is most important to fix first because if you get it wrong your ship's gonna explode and the Mantra that we came up with

during that difficult year was First Trust then retention then Revenue and we realized that our first priority was player Trust because we'd assumed when we built Destiny 2 the players who ran out of content would just happily leave and come back for the next expansion but what they were actually doing once they ran out was they were sticking around in all of our online communities with these megaphones shouting to everyone to avoid this game because it sucked and this was especially bad with streamers and influencers because we had literally hurt their livelihood by making streaming Destiny a less viable profession so they were loudly and like understandably shouting their complaints to every viewer and so we knew that we needed to solve these trust issues first so we focused a

lot of our early work on trying to repair our reputation with our players and our community and our influencers we we bumped to the top of our priority queue several early changes to the game not because our designers believed that these were the most important root causes to our engagement issues but because they were vocal pain points in the community and addressing them quickly would clearly indicate that we were listening and we were responding to our player community and we were just trying to do our best and this this trust battle took us almost a full year if I'm being honest and that's also when we did stuff like we removed all of the microtransactions from some of our releases in that year taking multi-million dollar losses in the process because the risk of losing more trust was so much worse at the time than the known loss in Revenue and the reason that we were able to eat

that Revenue loss was because we knew that Revenue wasn't the goal yet trust was Revenue can come later and then once we had player trust and sentiment under control we then focused on retention and this was the real focus of our first major expansion after Destiny 2 launched forsaken forsaken's product focus more than any other specific goal for that release was to fix the end game for Destiny 2. to fix our retention and to ensure that there were plenty of long-term Pursuits and content in Destiny 2 for our most hardcore players and this Focus allowed the product to succeed not just succeed in being a great expansion which which it was but being the expansion that Destiny 2 needed at that moment in time as a

service and it worked so this was that terrifying moment in February of 2018. but by the time forsaken launched one year in our brand trust was back on track our retention had recovered from that precipitous drop after D2 launch and only after we had solved for trust and solved for retention did we then actually start paying attention to the business again that was when we started experimenting with Revenue Innovation that we knew would ultimately be player wins but were complex and we knew that they needed to be built off of a foundation of trust so that was when we started charting our course to get out of loot boxes or um and and move our sort of direct purchase that's also when we started to add a free-to-play entry point into our game like these were all designed to further sustain the long-term Destiny Hobby and

I think they've made our game better for all players but to go back to that FTL analogy like it would be suicide to be messing with our Shields when our O2 systems were down and I truly believe that each of these different releases along this road would not have succeeded if we didn't have clear laser focus each time on what our specific goals and definition of success were and importantly these goals they change from release to release so so that was a really simple example three definitions of success layering them on one at a time as we've iterated since then it gets it gets more and more nuanced because we keep trying to build more and more precise definitions of success so for a more recent example here's one of our typical uh kpi report cards I

pulled this straight from one of our internal presentations not not too long ago kpi means key performance indicator it's quantifiable measurements of performance that we measure against a defined Target we said ahead of time and we call it a report card because while every release might have their own unique metrics that we want to measure so like for example in the witch Queen we release the new glaive weapon so we have a metric for like how many people are using the glaive weapon those are those are unique per release this report card is the Baseline grades that every release is going to get no matter what and it gives us a Common Language to Define and discuss success every time so let me talk through what our kpi report card looks like and what what each of these different measurements mean to us so first up we have that trust in Bungie trust in our game and we measure it in two ways one is vocal sentiment so what are

people saying about us online on social media on Twitch and vocal sentiment is super important because it's that's your reputation right like that is what people say you are but it's also often distorted by a small vocal minority there's a few people who talk a lot and a lot of people who have an opinion and don't talk very much so to get beyond that vocal minority we also do our own weekly surveys of player happiness using a rate last week metric so we randomly reach out to I think a few thousand folks every week and we ask them to rate the last week of Destiny on a one to five point scale and we've tried some different systems here like we've done more subjective measurements of quality we've done net promoter scores but we've found that that rate last week hits that sweet spot for us of Rapid precise data that we can

quantify about player happiness if you release an amazing live event you can know the very next week if your players are happy about it or if you do something terrible and you break something you can know immediately if your players are unhappy about it and so between the two of these you can get a good week over week trending of both what players think of your game and what they're saying they think of your game and and both of those are interesting and different data points about player Trust so next up we have the the lifeblood of our game our engagement and our primary metric for this is our average Dau so average daily active users how big is our community that is willing to show up and play each day and you can measure your active users daily weekly or monthly there's interesting Nuance there and different learnings that you can pull from each of those different types of measurements but for our report cards we like Dau

because it's not very latent you can get new Dau scores every single day and while diu is ultimately one of the best indicators of Health for us as a franchise is kind of like our blood pressure like it's super important but it's not very actionable on its own to try to impact Dau we have to go one layer deeper so do you use ultimately a function of really just how many players are coming into your game each week and how many players are leaving it you're funnel in funnel out and so we measure this through new players so brand new accounts that are created on a given week our weekly return rate so these are players that played last week and have returned this week to play again and then win back so these are players who did not play last week but played at some point in the past it could have been two weeks ago it could have been five years ago but for whatever lapse

they've they've returned to us and you put these factors together and then you can see pretty easily your funnel in and funnel out of players and importantly each of these kpis are going to be pushed differently like the type of content that typically causes your engaged players to stick around that raises your return rate is going to be really different from the type of content that's going to cause new players to show up and this is where where getting higher granularity and how you're defining success for a given feature a piece of content or a release gets really important so so finally beyond our blood pressure we've got the revenue that we're trying to accomplish made up both in unit sales and in micro transactions and importantly here like while we're tracking Revenue daily with these report cards we aren't actually optimizing for Revenue because like it would be easy to

make decisions but temporarily spiked out Revenue but were very bad for the long-term health of our game and that's why we focus a lot more on our trust and our retention scores than just on our Revenue scores all right so so we're gathering this data we're bundling up in these report cards but the data only matters if you're doing something with it and so the remaining step for us is about validating our hypotheses and really unpacking and learning from this data each release so we have three main ways that we use this data that each have different levels of latency they range from very rapid and timely use of the data to very thoroughly analyzed but delayed foreign so the lowest latency access is all these kpis have dashboards that

are live updated and they're visible to our whole studio every day we have them up in the kitchens in our offices and if you're if you're remote you can log in and you can see them directly in your browser and this was like a meaningful cultural shift because we're talking about live updating and displaying all of our core metrics including our business metrics about stuff like Revenue to the whole team but what this means is that at a glance anyone in our studio who's trying to make a decision about something they're building can see how we're trending against our kpis for the year for the week or for the given release and that data is updated every single day we then do what we call liveops reviews of this data every two weeks so we dissect the performance of the last two weeks of our game with the team and importantly the audience here for these live Ops reviews is not just like our PMS or Analytics we try to get all of our feature leads

like all of our design directors our art leads anyone who's making creative decisions about the product to prioritize attending these meetings every two weeks because in these meetings besides going over our report cards we do deep dives into specific timely topics in the game so like as a recent example we just launched uh the witch Queen uh what has it been like a month some about a month ago and we launched our first legendary campaign with the witch Queen so this was a hard mode that you could you could opt into and play instead of the normal mode and so two weeks after that launch in our live Ops review we did a deep dive on how that legendary mode feature performed and how many players engaged with it how many of them completed it we could have that conversation with the whole team just a couple weeks after launch and and what this means is that we can try different creative experiments like that and validate them three four times

across different releases and our creatives then get to develop a much more intuitive understanding of what kinds of creative approaches are going to yield the results that they want and finally we have these large two-hour lecture style Recaps that we do at the end of every season where we recap the all-up performance of every aspect of a season or an expansion release several months after it's gone live and this is where we get the most in-depth analysis but it's also where we have the most latency because we spend several months building these presentations and consolidating our analysis but we then present these in our theater and we record them and put them all up on our streaming portal for any employee to watch so this picture was from one of our pre-covered sessions that's why we're not wearing masks but they're basically our college level

courses on learning the destiny business and they've become a great entry point for any employee to start understanding how their work is going to impact our players at scale and so what this all drives to what we're trying to accomplish with this nuanced definition of success is to start doing data informed design and importantly like it's not data driven we are not putting any guard rails on our creative process or our team's Vision we regularly try radical new experiments that Veer far away from our existing data that Veer away from our standard formula like a recent example we in last December we released a 30th Anniversary pack and that was a very different product from anything we had built in the last several years that was driven like the definition of that product was driven by the creative Passion of the team and how they wanted

to celebrate 30 years of Bungie more really than than any any anything else but even when we take those big swings when we're Guided by that Creative Vision those creators then get to be scientists what we do is we we try something new but ahead of time we predict what we think is going to happen we then take those predictions and quantify them and then finally after we release we validate whether those results matched our predictions and almost every release through this process we get at least one just like shocking and surprising result back that totally violates our expectations and our hypotheses of our game and I love when this happens because that's when I know that we're really learning like that's when I know that we're honing our craft we're discovering something that's true about Destiny or something that's true about our community that didn't fit any of our

shared conventional wisdom up until that moment and as a result we're becoming better designers becoming better artists and creators Okay so we've talked about velocity we've talked about data the third major way that I want to talk about redefining success is in the product itself and the way we talk about this mindset shift internally is you're building a train station not a train so if you think about that that box product mindset as building the most impressive fast luxurious new train that the world has ever seen you're pouring over the details of the trim in every cabin the friction on the wheels everything to make that train the most amazing thing the world has ever seen and you probably have some sort of like intense push at the end of the project it could be a targeted crunch or it could just be like extra Focus where all

anyone is thinking about is how to get those last few steps to release to a gold Master to a final perfectly Polished experience but as a live service what we're actually building is a train station and the moment that that train is Departed you have another one arriving and in fact the track probably needs to be cleared for that next train before the previous one has even Departed so this um to put it visually this was what our release schedule looked like a development schedule during Destiny 1 and Destiny 2 development so we released Destiny one we built a couple expansions one after another for Destiny one and then in parallel we were building Destiny 2 and eventually released that but otherwise it's a fairly straight shot we were building trains in this model

but this is what our release schedule looks like today and so at any given moment we have five to six releases in various stages of development and we're not a significantly larger team than we were when we were shipping Destiny 2. like this hasn't been a dramatic scale up in team size but it has been a dramatic scale up in schedule complexity year over year with largely the same team and what comes with that increased complexity is yet another redefinition of what success looks like because success for a great train station means predictability it means scheduling rigor it means no one should be doing any heroics like there's this great I don't know maybe maybe it's apocryphal Miyamoto quote that a delayed game is eventually good but a bad game is bad forever

but we found that once you're in a train station mindset you actually need to have almost the completely opposite approach because a delayed train doesn't just hurt that train it potentially causes Cascade failure and Chaos in the entire station whereas the reality is today a bad game can be fixed and like I hope you all are hearing the point that I'm trying to make here like I'm not trying to dunk on Miyamoto I'm not trying to say the quality isn't important but like that earlier point about velocity it's that you don't get the quality on a service through perfectionism or through heroics and overextending yourself in any one given moment that is train thinking it's not station thinking instead you get to Quality one train at a time like our seasons in Destiny right now are the best they've ever been I think

that's a safe statement that our our community would agree with so are our expansions right now the best they've ever been and the reality is our Seasons got there a fair bit quicker than our expansions did despite our Seasons being under tighter schedule constraints with less room to like iterate and polish and so do you know why they got there faster because right now we're on season 16. we've been able to iterate so many times as train after train passes through that station to get to where we are today but that also means that to keep the station running you need to let trains go out the door that carry risk on them that you might not be comfortable with or that have things that you know could be just like a little bit better if you just backed up the train station just for a couple weeks and did a heroic savings throw and we've had to learn painfully not to do that

because that only makes things worse so a tangible example that we found here is with crunch because now a truism that we have at Bungie is that on a live service you're either never crunching or you're always crunching because if you put in that crunch level effort even when it's just individuals choosing it for themselves from their own passion you overextend yourself you burn down your stamina bar but unlike those big box products that we used to build where you can roll from that release with a low stamina bar into some early pre-production and concept phase where you get to recharge in a trained station you are rolling from that release straight into another high pressure release in the middle of production you don't get a chance to

recover from any over extensions if you spent an extra week closing that last release you've just lost a week on your next release and you're already behind and so in basically every case we've seen over extension create cascading debt where you're transferring that debt from One credit card to the next getting further and further behind and and we see that with with delayed schedule delays we see that with crunch we see that with any kind of pressure just building and getting worse and worse each release and that is what we mean by by that road leading on a service to always crunching and the only way that we've been able to combat that is with rigor it's with dismantling the whole idea of heroics that like you know that leave it all on the field final push to close a project that we've all felt in the past and we have to kind of get rid of that

mindset and instead just be disciplined be constrained operate in a way that you know can be repeated sustainably and consistently every release that is the train station mindset and and don't get me wrong I'm not saying that we've perfected this I'm talking about the mindset here the recognition that if you allow even a little bit of overextension to creep onto your teams or under your project it's going to Cascade and it's going to build release after release we've seen it happen over and over and so the reality on the team is this is a constantly ongoing battle we don't catch every case on every team but with every new release we are trying to constantly proactively identify where our teams are overextending themselves whether this could be through their own passion or it could be through feature over scoping but regardless we we try to actively help each team member to learn this

lesson so they can ultimately get onto that never crunching path because you don't want to be on the other path and so here's the last Almost painful realization for us about train stations so remember back when I said that box products are in the business of over delivery over delivery is actually dangerous for a train station with every release that you put out there you're setting a pattern for your community and for your players and so if we release an expansion with two raids let's say next year our community is going to be upset if we go back to only one raid we've hurt the station by making a rogue decision on just one of the trains to over deliver but at the same time and this is why this lesson is is painful to us is because so much of quality and like Delight in our craft comes from over delivery right like it comes from giving

folks stuff above and beyond what they expected and we still want to do that we still want to surprise and delight and sometimes even overwhelm our players but we need to be mindful of the patterns that we're creating the implied future commitments that we've set for ourselves each time and this is tough like this is honestly one of the ones that we're still grappling the hardest with because it's hard to tell a team a team that has extra cycles and they have energy and they want to do something amazing and they know how to do it and it totally would be amazing and awesome for the game sometimes we have to tell them like we shouldn't ship this because it's an over delivery that is going to set us up for failure on future trains and to paint it in a more positive more like proactive light the lesson that we're still learning is that everything we deliver needs to be a blueprint

we need to understand how the blueprint will be repeated and whether or not we can do that repeating sustainably and if we can't repeat the blueprint or if we don't want to we need to make sure that our delivery of that feature or that content sets up player expectations to match ours that the blueprint is not going to be repeated so like a good example of this recently was our legendary difficulty for our campaign that was a Cornerstone feature of the witch Queen and it's been really well received like I think it really made that that expansion better better than than our previous ones and it also was a meaningful step up in the amount of work that we had to do building a campaign because we had two difficulty modes that were finally balanced for every single mission and we knew that going in so we designed it as a blueprint and we weren't just worried about whether we could ship the witch Queen we were worried about can we

ship this blueprint and repeat it in every future expansion which fortunately now like we we want to because it's a proven success so yeah returning all the way back to this moment in 2018. at the time it felt like it honestly felt like it could be the end for us like for for D2 all together but what it really was was the inflection point that was the beginning of us really understanding and embracing a live service mindset this was the inciting incident that led us to some big multi-year shifts in our mindset this is what helped us solidify a bunch of these learnings that I've tried to walk through with y'all like how your velocity is so much more important than your current position that to achieve that velocity we have to get really comfortable with failing out in the open in front of our players

that when you're in a live service crisis prioritizing trust then retention then Revenue could help you get out of it and that you need all of your designs and all of your creative decisions to be data informed and nuanced and quantifiable ways and that we're building a train station not a train which means that you need to Stamp Out Crunch and any other form of heroics that you're doing that overextends your team and to be aware of over delivery that can cause future problems for your station and ultimately all of these mindset shifts what they really boil down to is redefining what success means what quality means for your entire team and culture and if you do that I think that any organization can make the transition from a storied history of great AAA box

products into the future of live service so yeah I hope that you've gotten some value out of the lessons that we had to learn in this space because we we paid for every one of these lessons with a lot of scrapes a lot of bruises a lot of blood and a lot of soul searching and so I'm hoping that by sharing you can get to that same destination with far fewer injuries and if you like what you heard like we'd love to have you on this Learning Journey with us building the future of live services at Bungie so we're hiring right now for almost every type of role imaginable we have development publishing from entry level to director level and we would love to speak with you So yeah thank you everybody uh [Applause] I think we've got about five minutes for

Q a if anyone asks questions uh thanks for the talk that was really revealing appreciate it uh I I appreciate any Candor that you can offer for this specific question from uh the beginning having a specific corporate partner to becoming fully independent and moving into having another big partner how did these learnings all collide with that in terms of what uh Bayan you had originally maybe and then from there what changed in terms of this train station metaphor as you became more independent and how that relates to moving into a brand new partnership thank you yeah I mean I think I think most of the manifestations of these learnings are not they're not at the green light phase they're not at the Milestone phase they're in the like the day-to-day decisions that everyone on your team is making culturally and so so we found that like if we can make that transition internally in our organization then we feel really

confident that we can make something great and we convince we can convince anyone that that it's gonna be amazing whether that's our players or our partners or parents what's up man yeah it's a great talk I'm a little curious about your overall sense and the shift of the industry towards live Ops I feel like a lot of games are slowly becoming the live Ops kind of model of things and until you get to a live box you just kind of keep going for it until you get there so I'm just curious right you're well known for a lot of different games and expansions and sequels but now we're in a live Ops mode do you see that as like a future for the next decade or so yeah I mean I think I think if you're trying to make a game that builds a community that sticks around then I think it's worth thinking of it as a live service like no matter what type of game you're making you could be making a a rogue like like 8-bit indie game but if you're planning to release that Indie access like Early Access and iterate and build a community I think you can still think of that as a live service it's got the same lessons about

your velocity mattering about needing to be in a constant dialogue with your players and being willing to fail publicly to learn faster like I think all of those those still apply if what you're trying to do is build and nurture a community over time maybe just as a quick follow-up based on your kpi do you have a point where like live Ops this is where we would start to ramp down have you thought about that far in the future uh not anytime soon we are very happy with Destiny where Destiny is and it is it is getting bigger each year so well thank you yeah what's up man yeah thanks Justin for the talk really inspiring I have a question about the leadership and you know discipline these directors and this new approach of the train station uh how you convince them in the company that it is a good way and how you recognize that they changed their mindset um so can you can you help me understand the question a little better what do you mean yeah because for sure they have all

the leads the directors you know they had this the previous box mindset yeah yeah and then the conversion the wrote the new one yeah so like uh let me give you an example we we've found that there are things in the game that we could release that might uh very briefly create uh some negative sentiment but then that quickly rebounds those players like because sometimes players just don't like change and then they they understand the difference but they might be really great for the overall health of the game it causes people to show up more to play more to ultimately long term be happy and we need to provide all of that data to our team because if you don't provide all that data they're going to see the sentiment side like anyone can go on the internet and can see what your players are saying but if you don't know what like the the silent majority are thinking if you don't know what's happening with engagement you're probably making the wrong decisions and so we try to make all of our leaders like see that full picture of what success is so that they can balance those different axes

okay maybe the second one because you said about the stamina stamina and the creativity so everybody are boosted with those yep how do you know have a control that they are all work in the same direction the same goals released by Release Train by train yeah that's hard um it's hard because I mean whatever it's hard on any creative project to get everybody working the same direction it gets easier the more similar your blueprint is to something you've done before like like there's a lot of things that we release in our Seasons that we know even though we're going to be creative on them we know the the rough size of the box and the shape and what it's supposed to look like um and so we tend to focus when we build our creative pitches on the differences like what are the ways in which we're going to innovate and be different from our previous Blueprints and then really focus our efforts on like aligning the team around those differences okay thanks yeah thank you hey uh where the time so I'll be really quick I'm a long time Destiny player and

I love it so thanks again for an amazing game um my question was really about um some of those like data-driven insights about the player community itself could you share any examples of the things that you said were surprising that you've learned maybe from the 30th Anniversary pack which was like way out there very different kind of experience here's what I'll give you it's not a uh I'm not gonna give you numbers but uh the number of players who played the legendary campaign was shockingly higher than we um like and and even completed it like like we expected it to be a narrow minority of players who would play it um but it was yeah it had a very different uptake in our community than what we expected that's great yeah no wonder oh I think I think let's do one more question and then and then we're done um okay thank you for the great talk love Destiny thank you um so when you're focusing on Dau and

um weekly Au yep uh do you break down your audience kind of like by motivation it's okay for this part of the audience to just like they care about the story they're going to show up for a very short time per week to play through this story because how do you avoid the Trap of focusing so much on that you might burn out your players um because they feel like they always need to be engaged they can't play anything else yeah and that's kind of that distinction between engagement and uh sentiment and why we have to track them both because you can create toxic patterns where you motivate people to log in but they get less and less happy about it and so when that happens we see that show up in sentiment first and then inevitably it's going to show up in your engagement too because eventually someone's going to be like I'm in an unhealthy relationship I need to stop this but we usually catch that first on on the sentiment side okay even and there's no way to track it if they're being silent about it if they just silently well this the the sentiment on the silent side with like Ray last week were able to track even the players who aren't complaining we

can see their interest waning over time okay thanks yeah so thank you everybody [Applause] [Music]