User Research on Destiny

[Music] so the story of user research on Destiny is really a microcosm of the story of the development of Destiny as a whole and so just you can't really talk about Destiny without talking about Halo you really can't talk about user research on Destiny without also talking about user research on Halo so for 10 years mie made this game called Halo and we that's therefore that's what we did our research on and we got pretty good at it we made five games in that series and we pretty much thought we had this job down That We Knew by the time we got to the fifth one we knew exactly how to test a Halo campaign we knew how to test uh competitive multiplayer we thought we were pretty good at this uh unfortunately it turns out that move the move to Destiny sort of up the difficulty quite a bit that there were a lot of built-in assumptions into how we tested Halo that into the Halo game that made user research easier for us um simple things like the fact that

each game mode was a distinct experience that you could test the story mode by itself as an isolated experience and not have people mixing and matching in between activities uh the fact that all the missions were always played in the same order turns out to have been a great luxury that we didn't know we had at the time uh suddenly we had to make every Mission not only fun but also the right difficulty level no matter when people got to it and what state they were in when they got to it um the missions were more or less independent experiences so for example it's very common in game development that like mission five isn't working this week and therefore that's fine we have people play up to Mission four we read we have them read a little paragraph of what happened in the story in Mission 5 and then we have them proceed on skip the mission six and let them proceed and that works pretty realistically in Destiny suddenly if you have to skip mission five you haven't gotten any of the experience points from that mission your character is now underleveled for Mission Six you don't have any of the gear or the talent that

you needed to beat that vision and suddenly the entire rest of your testing experience is out of whack because that one piece wasn't there and Destiny was this very integrated game where all the pieces had to function in order to get good data um a couple of our issues were also sort of metaphysical uh on the Halo games certainly by the time we got to the final ones we very much knew what kind of game we were making we knew that that what the modes were going to be we generally knew what the general mechanics were going to be a lot of the the big sort of hard decisions about game design were already made uh in Destiny we were discovering a lot of that and a lot of that was changing over the course of development and finally we knew what kind of players we wanted to play the game I mean the kind of people we wanted to play Halo Reach were pretty much the same kind of people who' played Halo 3 that that was a well understood audience that we really had had been in touch with for years they played our games we had data on exactly how they played our games and we had this sort of very good understanding suddenly with Destiny we had RP PG elements okay should we be

recruiting RPG players are we aiming for that are we aiming for MMO players are we aiming for people who play both are these action adventure Gamers there was a whole bunch of just fundamental assumptions about our audience that were suddenly called into question at the same time Destiny was just huge uh so we had four planets all of which were bigger than any of the Halo missions we'd ever done four enemy races each as complex as the flood of the Covenant three classes hundreds of weapons story missions strike missions and finally we had Patrol and raid which were completely new experiences that really had no analog to anything we'd ever made or tested before so this is pretty daunting um and at the same time we were also uh this is our first game we were making outside of Microsoft so in previous on previous titles we'd had the benefit of Microsoft's lovely user research infrastructure which is very elaborate and very well uh and very well planned out and so we had to do this all ourselves and we were free of that infastructure so we had to build our own

tools at the same time we were actually testing and building the game so building the lab is actually the easy part this is a picture of our lab um it's pretty straightforward pretty standard compared to most of the industry um but the hard part actually is putting people in the lab is actually recruiting the participants that and I'm going to geek out a little bit here please bear with me um so I know that talking about recruiting participants is not the sexiest Topic in the world but it really is okay I I wanted to Enthusiast um but it really is what defines the limits of your re user research program because if your recruiting methods don't get you casual players you can't run that kind of test if your recruiting methods take a few weeks to get your participants suddenly you have to be testing on a very well-planned long-term time scale uh if and you can't do snap uh snap tests at short notice so I'm going to talk a little bit about how we went about recruiting participants for Destiny play test uh

bear with me this actually is important um so first off we created a signup form on our website and let people volunteer we've had over 350,000 people sign up to be play testers and they all give us demographic data their age gender where they live a bunch of their attitudes towards games and they tell us a lot about and it tell us a lot about themselves and we connect all that to their Halo gameplay data and now to their Destiny Gameplay data so we know who they were in their actual play as well as who they said they were and this led us when we wanted to run a study we could then pull exactly the profile participant we needed we say okay we want people who've beaten the campaign on legendary we we want people who played less than an hour of multiplayer people who played more than 100 hours of multiplayer we could use the survey data to say exactly where they were all their demographics and their game play data to know how they played our games and we pull that sample of people and then send them an online

survey to schedule them so this is unusual most recruiting agencies call up all their participants on the phone and Screen them that way which tends to be a very Manpower intensive process and take a lot of time and ends up costing something on the order of $200 a participant to recruit a participant this way and because we could send it a survey that would screen them and uh schedule them and Slot them into their time slots we could basically schedule a study with very little work maybe a couple of hours of work total by the researcher and on very short notice so anywhere from a week to 3 days uh we did a couple of studies on sort of two or one day notice um we've never done a same day study where we kicked off the scheduling in the morning and had people show up in the in the evening uh the system certainly would support that but I don't want my designers thinking they can drop things on me in the last minute don't tell them I trust you guys uh and one thing I'll one other geek note I'll mention on this not all survey tools handle quotas the same way so not all survey tools are necessarily

suitable for this kind of scheduling setup so there's some fine technical details and how they handle them that make them more or less suitable um so we had all these systems working and they had them all connected together so we could have this one system that would let us pull participants that we wanted schedule them screen them out and have them show up in the lab and we had really good success with this uh we scheduled something like 1,400 participants over the course of four years for this game uh saving us on the order of $400,000 by doing all the scheduling in-house using an automated system now you may be right to be skeptical that our participants were probably a little biased that these were people who signed up on a Bungie website to test a bunge game in a Bungie building so yes they were fans um but we dealt with that bias in a couple of different ways that first off a lot of the questions we were asking them weren't the kind of questions that would be affected by bias so if we asked them do you like Destiny better than Call of Duty

yes their answers would be biased but if you ask them do you like mission five better than Mission Six there's no reason to believe that their bias would affect One mission more than another so it sort of canceled out we also did some comparison studies where we actually did use outside recruiting agencies and found that the internally recruited participants gave us essentially the same data um particularly uh directional data that if the internal people said a mission was too hard the external people said it was too hard and so on and their rankings of what were their favorite missions least favorite missions were all identical we also did some poststratification work to look at how our sample differed from the real population so on along the xaxis here are different player types and the y- axis is how common they were in our sample and the general population and as you can see like type one players were very common our sample those are the online engaged players but because we knew how common they were in the general population we could wait their answers and give them uh the proper uh the

proper impact on our ratings so the first thing we did with all this wonderful data was to segment the reach the Halo Reach population player population and say how many types of Halo players there are and there's a bunch of different ways to do this and the way we ended up settling on was doing it by Play Habits that we're saying if you have a population of players are there subtypes of similar players that play the game in the same way and there's bunch of different algorithms and we can segment them by things like uh demographics their ages where they live all that kind of things but what we ended up settling on and using mostly is the how they played the game what their What proportion of time they spent in the different game modes in Halo so some people spend lots of time one mode some people spend more balanced but we could use that to basically give each player a behavioral fingerprint and then group players by having who had similar fingerprints to say there are a c a small set of specific subtypes of

players and one thing I'll say about this process so in case you decide to do this yourself um I'm not going to talk about the technical algorithm so much but I found the process of segmenting the player base to be almost more interesting than the final answers themselves I mean it's very satisfying to say there are five types of players and here are the mathematically the best five clusters of players but when you're uh doing the analysis you're trying a lot of different types you you're saying what happens if I factor in age or take out age what happens if I split up story mode from just one bucket into two buckets o sorry into two bu into bucket five difficulty level uh and the the answers you get change as you change the question and how those answers change are almost more interesting than the final answer themselves and tell you a lot about the texture of your player base in a way that just the there are only five players answer doesn't tell

you so one model might say okay there are three types of players there's red blue and green Players let's let's say we CH we add in a different a new factor into the clustering algorithm to chew on and suddenly that green population splits into two subpopulations and now there the algorithm says the best fit is having four po types of players the really interesting thing here isn't are there three kinds of players or four it's why did that happen why did that factor make that difference and why did those players group together or split apart and all those kinds of uh little texture moments to your data I found really uh important and interesting and I'll stop teasing you and talk about the actual type we found uh so we found there were depending on how you counted either 5 10 or three types of players so there were three overarching categories that there were campaigners there were people who played Halo primarily for the campaign then played multiplayer for a little bit and dribbled off there were omnivores who who played in a very balanced way

spreading their time across multiple game modes and there were Specialists there were people who loved one game mode and just played it for like 90% of their player play time um some examples might be Capture the Flag Specialists or people who played only custom games with friends and each individual subtype of specialist was very rare only a percentage or two of the population but taken together as a group they made up a substantial minority and so as we were analyzing this data and thinking about it and incorporating into our plan for Destiny uh we really the idea of the omnivore is really struck home the idea that there were players who were playing in this balanced way spreading their time across multiple game modes and there it had always been a idea for destiny that there would be a variety of activities that there would be one of the essential design print pillars was an activity for every mood that we would have intense competitive activities and very Comm activities

story missions that you were guaranteed to win if you spent a long enough time and raids that were intended to be deliberately very hard but once we had the idea of the omnivore we started thinking that actually we really want to encourage people to play not just the right activity for their mood but also to play a variety of activities and to not specialize too much and this was connected in our heads to the idea of the buffet effect from psychology the idea that if you have one flavor of food like a giant bowl of like white rice you'll feel full and you'll get sick of it sooner than if you have five flavors or or 10 flavors out in front of you and so that's why when you go to a buffet you always end up eating too much because it takes longer to get bored of having so many different flavors and so many different uh tastes and Toyo in terms of food it feels a little negative and has some uh bad associations but if you think about it in terms of things like CrossFit where you go to the gym and there's a workout of the day up on the board and it's randomly determined and you won't get bored it's not like you're okay I'm going to the gym and I'm going to swim

laps and I know I'm going to swim laps until I my eyes bleed there's something different every time and you never get sort of used to the routine and it makes it for a more sort of robust uh athlete and other games have done this sort of thing before they've deliberately set out to encourage omnivores so if you play Hearthstone which I do too much um they have their daily Quest where they tell well they say okay we're going to give you extra gold if you play this class you don't normally play and so maybe you never play a priest deck but you have a daily quest for it and okay yeah I'll go look it up I'll go look look up a priest deck and give it a shot and that way if your other deck that you normally play stops working well you had that priest deck You' played yesterday me maybe I'll play that instead that having that being a broader player a more robust a more sort of omnivorous player actually makes you a more robust player if you only play one game mode in some ways you're very fragile if that game mode stops working for you if it stops having rewards for you if it stops uh being possible for

you it's like if you only play cooperatively with your friend online and your friend stops playing you probably stop playing too but if that cooperative play is just one segment of a wider array of activities you engage in you'll have you'll probably be more likely to keep playing even after one segment of that stops working for you and therefore if the decision was really made to slope the floor towards helping all players play like omn and please note my phrasing on this we are not requiring players to be on the fors we are not forcing them to be on the boards we're just always making a little bit easier to play that way than to play as a a focused specialist player and we're going to give little nudes to you to go play as a to play in a more balanced way so for example even if you're playing a warlock in Destiny we'll drop gear for other classes to just to give you a little nudge towards trying another character so I was like oh I don't normally play a Titan but hey I got this really cool helmet yeah I'll give it a shot and we'll just encourage

you to be playing in a more balanced way um the probably the extreme example of this is the Exotic Bounties where we uh for example The Thorn Bounty where we require players to both do strikes and PVP to get this really good weapon um I'm not the only person to blame for the thorn Bounty if you've had it and hate it uh but certainly this was one of the components that sort of led to that idea so we did a lot of res research on Destiny and we knew we wanted to start really early so this chart shows uh the research studies we did per month on a number of different games uh starting from launch at the far right side of the graph and going backwards in time to the left and so we really wanted to start early on Destiny and we knew we wanted to get a lot we had these big design open design questions and we knew we wanted to get the game in the lab early and do a lot of early impact unfortunately there's some very real uh concerns about okay you're testing the game four years

before it comes out a leak is much worse than it's going to be later on because we really didn't want the first information out there about Destiny to be coming out of our Labs so we upped our security quite a bit uh this is Jerry our head of security um this picture was actually voted one of the eight manliest pictures on the internet uh and I fully agree but we upped our security quite a bit so we our players go through metal detectors to make sure they're not bring any recording devices into the lab we make them lock up their cell phones uh they are never left alone with the game and uh we do uh our testing platforms are very highly locked down to make sure that no one could take a screenshot and email it to themselves or anything and our security procs are probably Overkill in a strictly sort of practical sense but they let the studio trust us to put the game in the lab probably something like 6 months earlier than they would have otherwise so we got six months more impact on the game than

we would have we hadn't gone to such elaborate links on security and one of thing first things we found as we were testing the game is that there was a split between testing the game as a whole and testing it in specific Parts um and this is connected in my mind to the idea of the Heisenberg Principle not the Breaking Bad Heisenberg the the real Heisenberg the idea that the more precisely you know the persistance of of a particle the less you know about its velocity because measuring one interferes with the other and we really found if we we couldn't test both the game as a holistic unit and any specific Mission or part of the game at the same time because if we wanted to test the game as a whole we said okay just sit down go ahead and play well we couldn't guarantee that enough people would play any part of the game to give us real statistical power to say anything about it and even if they did if if half of them went straight to that mission by just skipping straight there going to be underleveled and half of them go off and

do something else for 3 hours and then come back they're overleveled how do you combine that information to get a real uh mathematical picture of how hard that mission is or how fun that mission is when players are coming out from such a variety of uh angles so we ended up basically testing every piece of content multiple times and pretty much in parallel the idea that we would test it very specifically and under controlled conditions so we had the statistical power to make strong claims about it about its difficulty and so on and also test it as part of a larger holistic free form test where people were allowed to progress naturally and so to give you an example of a free form test uh the kind of results we got so this is uh it's each column here is a mission in our story C mode and these are the Earth missions Moon Mars Venus Mars the black lines represent the ideal uh level for the characters going to that mission that that's the level the enemies are so and the blue dots represent the level the

players were when they got to that mission in this particular play test and so as you can see people start off on track missions 1 two three pretty good then they're underleveled from Mission 4 five start to catch up on moon they're underleveled on Venus and then by the time they get to Mars they're very underleveled and in particular that particular uh level jump between Venus and Mars was a deliberate design decision and the idea was that people would get there they'd look at the Mars missions and see they were higher level than they were and they'd say okay well I can either go straight into those missions and have them be a little challenging or I can go do something else for a while and come back I'll go do a strike I'll go do some PVP and I'll come back and I'll be the appropriate level and it as it turns out players didn't even see this as a choice they just plowed straight from one story mission to the other uh the way people can and just said oh I guess the game gets really hard now and it almost sort of the way people ended up consuming the story missions almost reminds me of the way people watch uh television series on Netflix

they like the new series of House of Cards opened I guess this weekend right and they drop all the episodes at once and people just sit down watch it all in the weekend and just hate themselves afterwards and that's pretty much what seems to happen with story missions is people that even when we say there's other stuff to do go go try something else and come back it'll be still be here they still want to just plow through Story Mission after Story Mission and so we had to actually sort of correct our assumption and make it Mars a little more accessible than it actually ended up being it was in this test so this is an example of our free form testing uh the holistic testing and then for example of uh precision testing this is a difficulty for each of the story missions in Destiny so these are again these are Earth moov Venus and Mars and each column is a mission and the length of the bar is how hard that mission was so if the bar is going up it's too hard if the bar is going down it's too easy so and we did this kind of testing repeatedly over the course of development because things would change all the time even if nothing had changed

in the mission itself uh there would be small changes to combatants there would be changes to other mechanics if suddenly ammunition was dropping less frequently the mission could get more difficult even if the mission designers hadn't changed anything so we did a lot of repeated testing to make sure that things were staying twed up all the way along and by and large we were pretty successful in hammering down the difficulty spikes although there were still a few in the in the released game uh this is uh the the Chamber Of Night mission is one that stands out in my mind this is Tel Thor The Unborn he's a cutie um and so this is the final mission on the moon and this is actually this mission is actually example of a design pattern that we used a couple of different times that ended up being very difficult for a lot of players where we basically lock the player in a round room there's limited cover and there's multiple waves of enemies you have to beat with finally a boss tell Thor at the end and if you die at any point in that process uh you have to start over from the from the the beginning of the encounter and do all the waves again

there's no save points in between waves and this really ended up being a hard skill check because there wasn't really a whole lot of options for players to uh handle this encounter differently you couldn't because you're locked in the room with the enemies you really couldn't hang back and snipe you couldn't sort of kill a few and then run away and come back it was you're in there with them and you're either good enough to survive or not and this wasn't a huge blocker I mean we didn't lose a majority of players or anything but we lost a few percentage players of points of players in the campaign when they just weren't good enough at that moment to come to beat that encounter and didn't even come back after they were higher level so that's all I'm going to talk about is those two types of testing the holistic and reductionist testing and but and I'm I'm going to talk for the rest of this talk I'm going to talk about some special topics and some particularly interesting research findings about Destiny but please make no mistake mistake this kind of testing

that I'm showing here is was the majority of what we did on Destiny was very straightforward user research where it's can people progress to the game overall and how what does a progression curve look like and how does each and is each piece of content working the way we think it should be working it's not again not particularly sexy but was the vast majority of what we did uh so to go into some special topics so Destiny is a class Bas game and we really had an intent that players would be able to recognize that what class was coming at them in PvP instantly that if you're if something comes around the corner at you if it's a warlock you have to react very differently than if it's Titan so we really wanted players to be able to instantly recognize what class there was a lot of work that went into that on class Silhouettes class color schemes and shapes and of course once those things are actually those assets are actually built they're very expensive to change so we wanted to test them early back when they're just concept art and

they're very easy to change so these are some examples of Destiny concept art we put in front of players and so if you look at concept A there essentially everybody said yes that's a hunter they they looked that they said yep Earth Tones triangular silouette yep that's a hunter uh but character b a lot of players misidentified as a warlock something about the shape that was intended as a hunter concept but something about the color scheme and the clean outlines tended to Signal Warlock to players uh likewise uh character C here players correctly ident everybody correctly identified as a Titan but uh character D A lot of people misidentified as a hunter because again because the Earth Tone the color scheme and something about the uh the Ragged cloth in the outline made people misidentify what class so we were able to give a lot of feedback early to help people to help the artists sort of tune their designs before they actually built anything um another topic we did research on was class regret so like all

RPGs we ask players upfront to choose what class they want to play and it's really stupid because we ask people to make that class choice at that moment at the moment where they know the least about the game and we make them ask them to make a choice that's then going to affect the rest of their play for the entire experience and they're pretty much we're setting them up for failure by asking to do it that early but that's when you have to make the choice so early on we asked we gave them class descriptions that were very lore based that were that very that talked about the theme of the class and the feel of the class and the fantasy of the class but those ended up a lot of players ended up regretting what class they were playing by the time they were done because that lore didn't tell them a lot about how the class would actually play in combat so we ended up moving to much more gameplay based descriptions where we were talking about the play the class is super super abilities that really defined how they would play in battle and players were much happier with this and we so and by the end of testing we

had something like 85% of players saying they were very satisfied with their class by the end of development by the end of their play session so another uh topic we did research on was uh player recognition so Destiny is what we call a share World shooter and it's a game where you run into other players no matter what you're doing there really is no offline single player experience even if you play a single player Story Mission we're going to randomly matchmake you with people and you'll just come around to corn a rock and there'll be other players and that's awesome and that's part of sort of the magic of Destiny and I'm actually going to play you a quick clip from one of our earliest play tests where that system was in action uh there's no sound in this clip so don't panic so these are two players playing through in our lab uh Player 11 there is a little tired from his adventures and he's going to have a seat on the Rock in cosmodrome and look out over the Vista oh yes no

yes what okay let me try that again okay so Player 11 is going to have a seat and take a little rest and he and player six is just going to happen to run into him these players have not are not in a party together they're just happen to be playing the same Mission at the same time e

and they run off to have adventures together so this is a great moment they can't talk we have no we don't turn on voice chat by default they they're not match made they don't know each other we just put them together and they have this great sort of human moment of contact and as it turned out and I I think pretty much every employee of Bungie can tell you the first moment when that system came online and we were testing an internal build and you just happen to run to another employee it's like hey cool you started following them around the environment because like it works it's working um but players were missing that moment in some of our testing because they didn't realize that what they were seeing was another player they couldn't tell the difference between a player in an AI and this is actually a video showing example of that so the what you're going to hear the moderator asking him as he plays is are the players you're looking at human or computer controlled computer controlled or controlled by people uh computer

controlled because this Slayer just threw a grenade that I don't know yeah hold on just give me a little bit and this and let me make clear I'm not making fun of this guy he is not stupid he's failing because we failed him we didn't show him we didn't give him the information he needed to know that those were real people he was looking at and therefore he didn't get that sort of magical moment of hey it's a real person I'm really interacting in a living world because he couldn't recognize that those were people so uh we ended up improving the name plates to make the players look stand out a little bit from the background so on the left is what you saw in that video on the right is what we finally ended up with to make them much more distinct from Ai and to make them clear that these were that these were real players um let's see so we could guarantee they had that sort of Magic Moment another topic looked at was directional damage indicators so these

were sort of this is sort of a this was almost a religious debate on the design team of what how directional damage indicators could should work and so in both these images the player is actually being shot from the same location he's being shot from above and behind and the top image is what we call an absolute directional damage indicator where we're we're showing a little arrow pointing behind and we're showing a flash on the top of the screen the bottom image was what we called a relative uh directional damage indicator that would sort of lead you towards the person who was shooting at you the idea was that you would turn towards that red Flash and turn towards the arrow and we' adjusted on the fly to aim you towards the per your opponent and Destiny had a much faster time to kill than than our previous titles so we really wanted people to be good at reading the directional damage and responding to it so again this is one of those cases where the system is expensive to build and we didn't want to build it and then decide we're wrong so we built mock these mockups and asked players to click on a little picture of a character and

tell us where he was being shot from so for the top image here's what here with the heat map of where players clicked they clicked above the character and behind him but for the bottom image pretty much everybody misread that and said no no I'm being shot from the right so of course we finally ended up going with the uh absolute damage indicators in the top and this was almost a a service we provided to the design team in just breaking the religious debate that sort of like okay everyone can agree here here's data we can all move forward now destiny had a really complex UI so this was the first game where Bungie made extensive use of ey tracking on our on our games uh and mostly we primarily used it in the moment to tell what player was looking at during usability studies and so I want to show you an example of that uh so uh I'll warn you in advance so this character this player is about to accidentally delete a bunch of his gear while he thinks he's equipping it if you've played Destiny this is extremely painful to watch I I warned

you yeah that's much better do that what's going on with that um oh just I see some power which might be helpful all right he just did it you can see the uh materials he's getting from dismantling armor popping up in the upper right hand corner that's handy look at the green arrows look at that you already worked those in there that was pretty impressive quick turnaround all right I'm just going to try to load up okay so and as you can see from the eye tracking his eyes don't even flick to that upper right hand corner they don't even see the materials pop up because there that channel is just out of his field of view and you can you can tell he is reading the tool tip that just saying pressing button X dismantles the gear clearly isn't enough to prevent him

from doing it um and so we actually ended up adding a safety mechanism that you don't just have to Tap a button to dismantle your gear in dusy you have to press and hold the button and the more valuable the piece of gear the longer you have to hold it to dismantle it so if you have an exotic and you want to dismantle it it actually takes a long time to F to fully destroy it and so just a personal anote on this like the first time I got in a legendary and internal build I did this I accidentally dismantled it because we didn't have any safeguards and I was so angry I was ready to go over and punch somebody on the UI team and at the same time though it struck me that this meant the game was working that that little pixel gun that IID just gotten I was so happy to have it and I was so invested in that virtual gun that I was so angry that I wanted to go over and hit somebody and so I had this great moment of oh we're screwing up but oh that worked so we changed a lot so among the many changes from Halo to Destiny one of

the most m one of the minor ones was that we couldn't pause the game because Destiny is a shared rule game and we expect players to it doesn't make sense to pause in multiplayer like that and as it turns out we used pausing for a whole bunch of basic logistical things in our user research testing from simple things like being able to pause and go to the bathroom or telling everyone hey pause your game we have we have lunch for you outside and so suddenly we couldn't do that because the game doesn't pause um but one of the things we also used pausing for was giving feedback while people were playing that as they were playing along in a campaign mission we'd pop up a little dialogue saying how are you doing and they could say I'm bored I'm fine I'm frustrated I'm lost whatever as they were playing and then we'd pause the game behind that menu as they as it came up um and of course in Destiny if we'd done that players would die because the Game Wouldn't pause behind the menu so we had to find a way to get feedback while they were still playing yeah that's oh yep youread yep there you go uh so we added a contextual

feedback system so you basically press the touchpad and a face button and the data mining system would log a moment of saying I'm frustrated here and we'd log it with all appropriate sort of XYZ coordinates we'd log who you're playing and where you're playing all that kind of stuff so we can analyze it later and uh and most importantly uh connected to our video so we could plot out maps of sort of where people said they were frustrated where this thing they were lost they were and importantly where they were having awesome moments cuz I mean we think of user research a lot of times as finding the problems in the game but the best moments in user research are when you find the awesome moments in the game and get to tell design team we need more of this this is working do that statue is awesome how do we how do we make sure everyone sees it and so here's an example of an awesome moment of you okay you catch your your teammate being heroic right in front of you you get yes okay how do I how do we give people more chances to see that kind of thing

uh we also imported our data into our world editor so uh the environment artists and designers in Destiny work on a tool in called Grog which is our World Builder uh and a lot of times when we and we were making this incredibly Rich data but play our designers had to come to us to get see the data they had to come use sort of strange visualization tools that are really only well understood by user researchers so we want wanted to be a little more uh accessible so we so we decided we were going to bring the data to them and we imported into their World Builder tool so they could look at it in the same tool they use every day the one that they use to make changes to the game and for me this actually represented sort of how I believe fuser research should work is that we brought the data to them we made it accessible to them and it's our responsibility to put it in a format they can read in a place where they can easily access we also streamed our play test from the

lab out into the studio oh yes yes there we go yes okay we also streamed our play test out into the studio so not so everyone who didn't come down to the lab we certainly wouldn't want them all to come down to the lab at the same time uh could watch from their desks and this is the first game where we included face cams in all the play test video we recorded and I think that was really helpful and really in humanizing the data because it's one thing just to watch a player die respawn and go and charge back into battle it's another thing to watch him die see him bang his head on the desk respawn and go back into battle and just that moment of seeing what the human reaction was to what's going on in the game I think really helps even if it's hard to quantify mathematically um one of the challenges we had one of the other challenges we had I'm sorry to be skipping around here so much but uh a lot to get through uh one of the challenges we had in Destiny was test how we were going to test the

endg game so Destiny had this raid experience that it launched the first people to beat it took I believe over 10 hours to beat it even the best teams in the world and the problem is we had this endgame content that was designed for experts in a game that didn't exist yet and there were no Destiny experts to test the content so we end up breaking one of the Cardinal rules of user research and we brought participants back multiple times it I mean pretty much which pretty much doesn't happen very often and most of the time we bring in participants we get their initial reactions to the game and we thank them we we give them their gratuiti and let them go but in this case we were bringing back people who would say come in for Story play test one weekend we'd bring them back a few weeks later to do a late game strike play test because we needed to have that expertise that they built up in the early experience we also tested our raid by bringing in importing an entire shooter Clan uh this is the uh the Gunslingers Clan who graciously used their vacation days to fly out to the studio and play in and play in our lab uh they were

really good sports playing a very hard and many places broken raid experience early on uh they didn't beat Aon they got them down to about 30% I believe which was pretty impressive for the first weekend and gave a lot of really valuable raid feedback so now the game shipped we're collecting a lot of data on how people actually play Destiny out in the wild and by and large uh that data has been incredibly uh rewarding and and very encouraging so this is actually a graph of overall hour spent in Destiny uh by game mode and people are generally playing that sort a balanced way we want them to and actually this this mix has been pretty consistent since launch uh about the same proportion of time spend various activities however there have been occasions where players have wandered off the path that we thought they were going to take um probably the most famous being the loot cave so you can sort of see in this picture uh actually just to check how

many people in this room shot into the loot cave yeah that's what I figured okay so I'm I'll do a quick explanation for those of you who weren't there uh where this player is aiming there's a little tiny Point there's a little tiny cave and enemies would spawn out of that cave at a steady at a steady clip every few seconds somebody would come out of the cave and the players would shoot them they'd die and they' respawn and run out of the cave and they were very weak enemies that didn't drop very good loot but because players could stand at this great distance as you see in this picture and just ping into that cave for hours once they finally did run over there even though there was a very low drop rate for rot there would be this lovely horde of treasure lying in the bottom of the cave and players literally did this for hours they and the funny thing is we knew about this before launch we knew that that this was a potentially uh exploitable activity but we didn't care because we said well the the actual drop rate per minute spent is not any different than anything else so you

actually will get less loot doing this per hour than just playing the game um but the players weren't actually doing the math that way they were doing the math of they weren't doing the loot per hour they were looking at loot per effort and this was so low effort that it felt like infinite free loot to them um and so and the interesting thing is this is actually pretty controversial in the player base it uh itself that this was actually the the the time of the loot cave was the highest peak for player reporting in Destiny of players reporting each other for cheating is a lot of players were reporting each other were people who didn't want to exploit the loot cave were reporting the people who were in the loot cave who were shooting the loot cave and the people who and some people were even going so far as to go into loot cave themselves and sit on the spawn point which would would prevent the enemies from spawning and so the people who wanted to shoot into the loot cave were then reporting those people for being uh obstructing their chosen play

style so this wasn't just a case where s of Bungie hated it and fixed it there were a lot of players who didn't like this as well but we did patch the loot cave there actually is a memorial in the cave if you haven't been back there's a memorial that's worth visiting and looking out um and since and since then we've also found a number of other loot caves in the time so this is a top down map one of our destinations and those red spots over there on the right are some minor loot caves that we discovered through algorithmically uh before they got on YouTube or or Reddit or whatever uh and we were able to patch them before they became popular and so we're constantly do this with pretty much throughout the game of looking for moments where players are beating things very quickly taking too long taking too short and looking for places where the experience isn't what we expected so we can then investigate and find out what's going on one of the great things about the modern age is that everything is on YouTube and twitch so if you can find some strange play style in the data somebody out there has put it online for you to

watch that's a wonderful resource for us researchers um so a couple of uh lessons I learned over the course of this and I'm I'm going to say we here I'm not speaking for Bungie in this case in this slide I'm speaking for myself and my research team uh or okay mostly for myself I won't I won't get uh my research in trouble uh so I absolutely underestimated our blessings on Halo that there were we we didn't realize how good we had of the time uh as I spoke earlier we underestimate how rabid our fans would be so one of the first questions I got was when I started a Bungie was so how much are people going to play the game and we they wanted to say okay we want to drop a certain amount of loot per per week so how many hours per week are people going to play so we can adjust drop rates to give people about the right experience and I based it on Halo and underestimated by about 50% uh people our hands on Destiny are just way more enthusiastic than I think

anybody anticipated they would be uh we really didn't correctly manage the transition to the Elder game so Destiny has an initial an early game where you're leveling up through experience points you're leveling up your character then you get to level 20 and you start leveling up through your gear and a lot of people miss that transition that that's not intuitive for a lot of shooter players uh and when I say a lot of people I mean something like 4% of players got stuck at 20 they just never quite made that jump to the Elder game progression system uh we did some we we knew that there might be a problem there and we did some work on it but we never fully smoothed it out before release uh and I think probably the biggest grw up uh I thought players would hate power in PvP that every play test we did on PVP in Destiny in the lab players hated the concept of level in PvP that the idea that they would get killed by someone who was higher level than them them and they would get really angry that they were in this unfair

match and they wouldn't even get angry at the other player they would get angry at us they would say screw you Bungie you put me in a game with that guy your matchmaking system failed me and put me in a situation where I couldn't win and they got really upset with us even Even in our play test where the power didn't matter and again at all for stats but just we had the numbers over your head players got angry about power and PVP it was just a concept that players hated in the lab so of course in our first Iron Banner we started out with a very minor minor effect of power in PvP saying okay we're going to let it matter a little bit and see what happens and thinking players would hate it and actually the overwhelming response was this this should be worse this I mean a level 30 should totally destroy me in PvP and this was just startling to us and there's some there was something funny happened in that transition from the character was the throwaway character they were playing in the lab and the real character they and gear

they'd earned out in the real world that once it was their real gear they that they leveled their character up to 30 personally and they' sweated blood to get earn that those guns they wanted those guns to matter so we've in subsequent versions we've uh increased the effect of power in the iron banner and people seem to be happy with it I think they're masochists but they they seem to be happy with it uh so overall in Des spite of uh the errors we made the game's doing okay um currently we're at 17 million registered players uh playing a little over a billion hours total in the first six months since the games come out uh they've killed 200 billion aliens amongst them and enough other Guardians in PvP to depopulate the Earth um uh the average total play time per player is over 77 hours uh that that includes trial players by the way if you look at just uh paying players it's

actually I think something over 100 hours um so which is way more than we expected uh the average player who logs on today will play for an average of three hours um our estimates pre-launch were something like 30 to 90 minutes a day uh 3 hours was not even on the scale and that's now whatever 175 days after launch uh and the average player will complete 200 has completed 230 activities so far so our us research effort certainly wasn't perfect uh but it seems to have contributed to something that our players really love thank you all very much and do you have any questions for [Applause] me if you have any questions uh please come up to the mics hello yes thank you for the talk of course for everybody I was wondering if you can shed some insight into how you're assessing certain gameplay problems you're seeing when you have

incomplete UI cuz you're in production but you know the UI is actually critical to people understanding the gameplay problem that people are experiencing and how often you take something to a certain level of user testing so people can understand what you're trying to test often you're waiting until the UI is complete but that happens at the end and what the strategies are um yes it's hard I mean sorry uh yes it's very difficult to test early uh one of the great sort of luxuries that we all had in Halo that I actually didn't mention was the fact that the UI didn't change much from game to game so for like a lot of the Halo 3 development we could just leave the Halo 2 UI in place it because pretty much most of the mechanics were going to be the same uh for Destiny we had to build a lot of that UI before we could actually test the game so a lot of our early testings were things that didn't need UI and where they needed to be UI we would do paper prototypes or mockups or things like that but yes it's it's really hard

to do a game like this when when the UI is broken thanks yeah sorry I forgot to I forgot to repeat the question yes or oh sorry let me go to the other side yes big fan thank you for the game uh how do you guys went from the lab to actual uh Crossing data from players on the live game how how's how that proc happen so the question was how we went from collecting data in the lab to collecting data in the live game and a lot of it is the same so we actually collect so we have very rich data mining in the lab and we actually collect that exact same set of data mining for a very small slice of players out in the real world and we collect a much less Rich uh selection of data for every player in the game so pretty much anything you see in your bunget profile we have access to so we can look at gear we can look at activities uh there's a very rich data set we use a number of different Technologies uh including Hadoop and high Bas systems too yeah can you please

ask sir to stop selling the No Man's Land please he he he doesn't answer to me all right uh hi hi I wanted to know a little bit more about how you handle the the long game you mentioned that you brought players back uh from the from the earlier play test but how really how often did you do that because uh one of my games like uh I base a lot of my play test on your work on Halo right so uh but a problem that I had I had anticipated is how players relationships with that same content changed after they went to 10 hours 20 30 60 90 hours so how did you guys learn about that um a lot of guessing um so we had there was only so long we could have people play in the lab and we always had this dream that we would establish uh fixed panels of players of like okay we're going to say these 100 players are the Saturday afternoon panel they can come in every Saturday afternoon for months and they can get really deep into the content and play the content and the game just was never quite uh and it just never actually made

sense to do it because by the time that they had gotten months into the content the developers were months along in production and therefore any feedback they had on that months old build was just worthless and so we never quite settled down into being able to do that I really wish we could have and so we are exploring some different ways to handle that in the future yes thank you oh I was just wondering have you formed any polies of using like twits data as like a post relase user research or is it more sorry I can't even hear you yeah sorry I had a question uh if you have formed any strategies uh about using twits and YouTube as like after release user research okay so so the question is how we're using uh Twitter and YouTube in post relas user research and mostly we're using it as evidence that once we find interesting patterns in the data uh we're we're then going in finding evidence of them on YouTube and finding

uh videos to support that position and we're also going vice versa we'll see somebody post a weird video on YouTube and then we'll look in the data and say okay how how many people actually can do that uh okay uh over here having played for 700 hours I feel this is my destiny Anonymous bit and what what testing did you do into the RNG side uh that seems to get people so sort of innovated about kind of drop rates and stuff were you surprised by how upset they they got about not being able to find a particular thing they were looking for uh upset no I mean I certainly have a friend in uh wow who ran the same dungeon something like 300 times and never got his hat to drop I mean I'm famili we we were familiar with the problem uh I think the thing that most surprised us was how focused people were on the RNG aspect I mean the basic intent was always that most of the loot would come from Fairly uh regular sources through things like Vanguard marks and then you'd buy the the armor off the vendor and that we expected that to be sort of the normal progression and

the random would be the sort of very cool cream on top but some but that random was so just became so fixated in players Minds even though it's a fairly small portion of the actual loot they get it it's just it dominates the imagination in a way I don't think we anticipate it finally dropped so we're good okay cool thank you um thank you for the presentation this is an RNG question as well uh I've been playing Destiny since September 9th and like 4 hours every day at least so my question was you spoke about reward and encouraging the players to play other characters like for example I'm I'm a warlock and I get a sa 14 so I want to try my hunter now so but I feel the game relies on RNG a lot cuz you spoke about vanguard armor and all that but the end game content is still the yah hor why do I not have a yah hor yet it's I've been playing since a long time and all my friends get it I sha I SW bar and still people get and I don't get it I mean it so the question is why he doesn't have a yah horn yet and and the answer is somebody has to get that unlucky I mean it's it is a random

number generator and somebody has to get that lucky but the thing that always strikes me is how many stories come out of that and how much how much dialogue comes out of that and how many stories people tell about how they get their gun in the end and that difficulty and the challenge in getting it uh creates War Stories and creates a great sort of community feeling about the game so it it does hurt when you personally have haven't gotten that the gun you want um I don't think I've ever gotten a random exotic weapon drop uh personally but I've bought all mine out of pity from Zur but it's it's one of those things that we expect everybody to eventually get an exotic we don't expect everybody to eventually get all Exotics but do you guys think you'll be working on this front at any point in time or you just going to leave it at RNG at all points in upcoming expansions upcoming weapons or whatever I I think the so I'm not a designer so I don't actually control this I suspect that RNG will always be a part of Destiny I think we will tune

exactly how much of a part it is okay to M as our sort of understanding of the game improves over time but I think there will always be a certain RNG Factor yeah I guess thanks yeah okay hey how's it going awesome presentation uh I have a question uh so I work at a technology company and one of the things that we track pretty heavily is user sentiment and how the new features that we implement change how people perceive the company that's creating the product uh which is kind of interesting um so and it often leads to really uh interesting unexpected things either greatly improving or uh negatively affecting user sentiment for the underlying company so what are in your experience some things that you've seen uh when you're testing that either made people say like oh wow I really love Bungie for doing this or uh surprising things that you saw people hate Bungie for I mean I think the number one so the question was what

things in our testing we found made people like Bungie like or hate Bungie in particular as opposed to the game um I think the number one thing that actually stands out in my mind is matchmaking is they don't see matchmaking and as uh you heard earlier RNG as being a product of the game they see it more as us being evil and messing with them is the is the number one thing uh things that make them like us I'm not sure the the game itself makes them like us I think the community our community managers and that those those efforts make them like the company and the brand but I don't think the game itself lends them to like Bungie over here hi uh thank you for the talk by the way um I'm really fascinated with the uh contextual feedback that you were talking about and uh how it was integrated into the World Builder uh can you talk a little bit more about what that would look like for say a level designer or an artist and um now that the game is live obviously we don't have this feedback functionality on our controllers but uh are you still

collecting similar types of data and putting that into the World Builder uh with with the live aspect of the game so the question is uh are you still collecting similar types data and putting in the live version and uh and what what would that look like what would that look like for a mobile game so even for a mobile game what what what does that look like for the actual people working inside the editor like work a level designer or an artist what would that uh what would they be seeing I saw I saw the the image that you showed yes it pretty much it looks like this to them okay so I mean this is this is the the a screenshot from the world editor itself uh and those little uh red dots there are players getting killed the or sorry the blue dots are players getting killed the red dots are what's killing them and so you can see there's a turret up there in the middle of the screen that's getting a lot of kills in that case this actually also I'll toss in uh helped our QA Department quite a bit because we found cases where like people were being shot through geometry this way oh they're turning off the lights on me uh

so I think that is our last question I will hang around outside if anybody has any more questions thank you