The Importance of Everything: Analytics of Map Design

[Music] yay so by way of prequel I had a lot of questions about what in the hell that title meant uh I'm not actually talking about a specific game or specific thing today as much as a methodology so people were kind of confused and so I wanted to start before I get into the me of the talk with actually a little bit of a background as to why I chose the title I did and really there were three reasons uh the first was the literal meeting uh information stats analytics all the things right like big data is everywhere now so much of what you do every day is being monitored and recorded and analyzing that data has become a major worldwide business and as a designer you need to be able to understand what it all means in order to use that data effectively in your maps and in your games it's all relevant and it's all important and the second reason

is that it's a play on words with another talk I'm doing tomorrow called the importance of nothing one is everything one is nothing haha very funny and I mention that not to boost attendance and get everybody to go obviously you're all invited but because in my talks with the GDC board when I was planning this talk somehow that's the only talk in the entire main conference that is specifically about level design and I've got to wonder how in the world that happens every one of you could have have and should have submitted a talk I saw a quote yesterday that said that everybody you meet knows something that you don't and you have to understand that you are not a button monkey you do impossible amazing things every single day and even a simple topic like how to tie your shoes from this Ted Talk can be important and inspiring ideas no matter how small are worth sharing and that brings me to the third reason behind uh the title the talk and

then said every time I come up here and I I do something like this my heart is racing I'm pushing myself to try something new and to do something push myself to do better and I would strongly encourage you all to push yourself to new limits to start a conversation and to share your ideas you may not realize it but everything you do every day is important so onto the actual Talk itself information is now everywhere if you work for a large company or a publisher somebody's going to come to you at some point with charts and numbers and you're going to have to be able to have a conversation with them you have to be able to understand them and talk about them intelligently in order to put them to good use as a student or an indie you're not exempt from this all this information is also out on uh industry websites you're going to get feedback from your play tests you're going to see reports online it's all there and again you need to be able to understand it in order to talk about it and to be able to

implement it effectively so my goal today is not to talk about math I'm not talking about formulas or statistical analysis I want to talk about actual methodology my goal is to show you that you can use data to improve the way you build from the ground up and the process of analytics is really really similar to the process of level design when you work on a project you need to ask yourself honestly and objectively what you're building why you're building it and who you're building it for and those sound like really simple questions but if you can honestly and objectively answer them the answers may surprise you so I recently came back from a trip to Disney World and they have this new Pro uh program called The Magic Band which is this little plastic bracelet they give you with an RFID chip in it and it has your name your nickname you get to pick a color and you can decorate it and do all this stuff with it and then they mail it to you once you've ordered it

and it comes in this cool box and it's like this family event it's really fun and exciting to open it and Disney goes out of their way to really make this important to you because you wear it during your entire trip when we got off the plane we put on our wristbands and then we used this band to get a a shuttle to the hotel at the hotel we waved the band and it registered us we went to our hotel room you touch the band the door the door opens it was our ticket into the park they've even gotten rid of their fastpasses so that now you actually register online for what rides you want to go on and then you show up and you walk right on there's no waiting it really is kind of magic it's cool right and then after my trip when I got home they had sent me a photo Journal that said everything I did while I was in the park and all these cool amazing things that I did with my family and this is when it dawned on me I never asked them to take pictures of me I didn't even know that they were taking pictures of me if they had these

pictures then they knew everything they knew when I arrived in my hotel room when I arrived at the park how long I was there they know what food I ordered where I ordered it when I ordered it what rides I went on how long I was in line and even the exact car that I was in can you imagine the power this gives them to customize and experience can you imagine the possibilities of what they could do with this kind of information and wouldn't you like to Magic Band one of your games how much would your work improve if you could better predict what your players were going to do or what they were going to enjoy pardon me so not surprisingly this is exactly what a lot of companies are doing back in 2009 Google's Chief Economist it's an awesome title

said that uh the next the sexiest job in the next 10 years was going to be statisticians and sure enough uh analytics has become the driving force behind mobile freeo play and all these different game systems and all of the effects of that are bleeding over into every aspect of game design and these techniques are often considered to be dirty because they force you into these compulsion Loops they're trying to get you to spend money but they don't have to be that way use ly and generously these techniques can actually be beneficial our industry has grown and we've changed and we've gone beyond the point where just your gut feeling that isn't enough to predict whether or not your game is going to be successful anymore and it's not just about money it's about making something that will impress Inspire and really impact people's lives and to me this is the heart of level design you create the part of the

game that faces out to the player the part that gives them that touch point to interact with whatever crazy ideas and Concepts your team has come come up with everything you do is important so I started to dabble in analytics and I kind of stumbled in blindly and I thought that if I had enough information if I could just get all these numbers and everything in front of me I could make sense of them but I actually ended up overwhelmed and it was all way way more difficult than it appeared so little bit of audience participation how many of you have ever heard of the company traod wow I'm actually impressed we have someone so traod data was a small company from Seattle Washington formed in 1972 to analyze traffic patterns founding members Paul Allen and Bill Gates would go on to combine their passion for microprocessors and software to grow their part parip into a slightly

larger company that I'm sure you've heard of today is anyone familiar with the home brew Computer Club probably a few more yeah so this was Gordon French invited some his friends over to his house in 1975 they just decided to geek out and talk about computers for a bit but the glub the club grew attracted people from all over Silicon Valley until one day uh two of its members where Steve wnc and Steve Jobs put their heads together and decided to build something needless to say their creation the Apple computer impressed the club and a few other billion people around the world and I'll be impressed if anyone knows who this is no hands awesome small boy 1982 in pic Maryland filled his parents basement with a bunch of computers and set out to make something he tried and tried and tried 10 years later he released his first shareware game called zzy this is Tim Sweeney now a young man he

he didn't want people to know that he was just working out of his parents basement so he came up with two really big words and crom together to make his company epic Mega games as the company grew had a few successes along the way and since we have been fortunate enough to grow into an international business so why am I talking about all this how could you have looked at these teams and determined beforehand if they would have been a success if you could have put a Magic Band on their lives what patterns could you have found and recreated to ensure your own success if only it were that simple we'd all be doing it there is no formula that can predict success analytics aren't magic bands they simply don't work that way it's not like these teams just sprouted out of Zeus's head emerging as gods they each had their share of both success and failure along the way what set them apart is they were driven by

their passion to learn from those mistakes and try again it's a great lesson something we should all do in in our lives for sure it's also a big part of iterative design which we talked about a lot today and coincidentally is the very foundation for analytics you make a hypothesis you gather data and then you test and through an understanding of the failures of that hypothesis you form a new one you retest and ideally improve your outcome every time you cycle the goal of analytics is what's called improving your probability space over time if it were simply simple as looking at a bunch of numbers and reading answers we wouldn't be having this conversation so here is a great example this chart shows a list of the top 25 games of 2011 it's like 18 months ago um the games of the blue popularity were all very successful games they were they were good they were profitable the games in with green and their popular were meah hits they were profitable on a

scale 10 to 100 times that of the blue games so the question then is what separates the blue from the green how do you get your game to cross over from a blue game to a green game the simple assumption is that the green games are better games and if that's true their uh review scores should reflect that right not really it's kind of all over the board it's not to say that quality isn't important these were all pretty well reviewed games they were all fun games but beyond a base threshold of acceptable quality score becomes a somewhat meaningless statistic and remember I said it's not always just about money you need to create a meaningful experience for your players so rather than review score look at how much time people spent playing these games that's a little bit stronger of a correlation so while Baseline quality is obviously

important uh a more significant factor of success might be how to keep your players engaged and again this doesn't give you any answers it just leads to the next question was how do you accomplish that and to be perfectly clear I don't expect you to take anything away from the results of this particular study I'm trying simply trying to show you that my initial assumptions would I would actually say are pretty common assumptions were Incorrect and as I drilled down deeper I didn't necessarily find any answers only more questions I had to cycle and improve my probability space over time assumptions just weren't enough but just having mountains of data isn't useful you need to actually do something with it right so really we're looking at a three-step process here and each step is a little bit more complicated than it seems the first step is gathering and recording data this is the information is everywhere thing that I talked about before four the second step is

organizing and reading that data and that's metrics that's how you visualize your data you you create measurements or comparisons of that data that you've already collected and then finally you analyze and this is where you come into the picture as a designer you need to understand how to interpret all this information and then combine it with your instincts and your design insights data metrics analytics it might seem like a knowing seman Antics but it's really important to understand each of the terms and their differences in order to apply them correctly and data is really important because it's kind of the building block for all of these different pieces and as I mentioned data is just information which is great as long as you really understand that that's all that it is so let's start with this study it's a poll to show who plays video games simple question but the results of this

were actually pretty surprising to me because it skewed a lot older than I thought it would with um the largest group being college aged and it was more female Gamers than male now everyone knows that gamers are like geeky teenage boys right except that this data says otherwise so take one last look at that chart and then let's look at the four most popular games in the world today can you guess which games they are based just off demographics this one is possibly probably is the most popular game in the world right now note that the bulk of the players are women over 35 years old anyone have a guess as to what it might be you're all On Target all right number two massively male audience bulk of whom is around 18 years old we're probably looking at a shooter game any thoughts

and note note bulk of the users are 18 years old and older this is not necessarily a kids game all right so this one massively male audience uh slightly older than Minecraft any thoughts on what this one might be Call of Duty heard League of Legends no this is World of Warcraft and again not as female as most people would generally assume all right last one before we move on massively male audience scream shooter what is it nothing nobody has a guess I heard one person say Call of Duty so I'll give it away it is not it is League of Legends none of these games are Shooters the most popular games in the world are Shooters played by teenage boys right not at all and then take another look not a single one of those matches up with the general data of who plays games if you had used that data to inform

whatever it was you were building you would have been way off base and here's where I melt your brain even a little bit more not a single one of those charts is accurate they're all wrong because it's never that simple right all of this data is self-reported which skews it and it all comes from Facebook which biases the source and I did this to show you that data by itself is meaningless it's just information it has to be processed and it has to be interpreted and it all comes down to how you decide to look at it so data is objective and it's factual but its tangible connection to the real world is exactly what makes it interesting and easy to misinterpret every map that you create is just a collection of data points but how many times have you heard about driving The Narrative of your game or using it as a means of environmental storytelling intentionally or otherwise every map tells a story this is a map of a

neighborhood uh in Raleigh North Carolina where I live uh it's called boan Heights from a cartographer named Dennis wood and he used uses data sets to tell stories he says that Maps exhibit and promote a political orientation they're about something and they have an agenda as an example here's another one of his maps overlaid on top of the original and rather than houses this this one shows the distribution of power lines in that same neighborhood okay and now here's a third one that shows the trees uh in his neighborhood and did that switch it did Why didn't it switch here there we go all right so individually each of these Maps is just aggregated data but viewed together they start to establish a narrative it

talks about urban sprawl expansion and encroachment again maybe that's how you might interpret that narrative and probably that's what he wants you to build as a narrative but the reality is you can't say for sure because data is just information there are no answers there wood hasn't written anything he hasn't made any statements but he's used raw data to make a social commentary so the lesson to remember here is that everything that you do in your map even the very Act of you being the one who creates it tells a story and especially that's especially true when it's viewed in context now some companies like Google and valve rely very heavily on data to drive their decision-making process at their recent steam developers days they discussed how like many modern companies they monitor everything that goes through the system everything that goes across steam but valve distinguishes himself by making every bit of that data

available to each and every one of their employees and so anybody who wants to start a conversation or make an argument is actually encouraged to reference that data in support of their point because an informed decision carries much more weight than a straight opinion when an a decision is made using data there's less chance for a random outcome but remember that data is only information out of context so data should inform your decision ision making process it's the middle of the equation but since it's only information data alone should never make a decision for you and this is where things get tricky because you see data being used in the context of a social science and in the social sciences there are no hard and correct answers so whenever you see data being presented in any visual format any chart any graph anything like that understand that you're seeing it through someone else's filter often with their bias this isn't something that people do maliciously or even intentionally it's

just human nature data can always tell you a story but only if you try to interpret what that story is so like many things in life you have to start by asking questions and this is where metrics come into play and they can be absolutely useful but you have to embrace and understand that they two are limited technically speaking metrics is distance in metric space it's just a measurement all these charts and graphs can be very pretty and very interesting to look at but they don't suggest change they compare data or measure Aggregates of data across different points but it's just information for you to interpret you need to provide that context you need to use your design skills and in order to get to the right uh answers you need to ask the right questions now the level design team uses at Epic uses metrics in various ways this is one

example from Gears of War judgment that shows how the games uh a couple different levels how their frame rate changed over time and it's cool to see you can see the line Trends upward which is exactly what you want right but once you have this kind of information you really need to start questioning it does this data indicate that the map is getting better put the map aside for a moment did someone change something in the rendering pipeline has pathfind in been fixed have they fixed that quirky bug with the particle system if you answer yes to any one of those questions then your game and your levels frame rate is going to get up even if the level itself itself is still just a huge unpolished mess these T charts track change but they don't say what caused that change in order to figure out what was going on here we had to dig and find uh more answers now here's another one that shows uh tracks Deltas in population in

our online servers and the different colors represent different game types and you can see here that there is a spike where people just suddenly stopped playing execution which was the number one game type and then started playing a bunch of other stuff so what happened well you can infer that that change was due to a server reboot and you might even look and the the timestamp on that might uh coincide with that change in population but it's not enough you have to question even deeper because there's so many other factors that could influence a change like this what day of the week are we talking about was it a weekend or were people going off to school was it a holiday was there some major news event in the world that pulled people away from gaming for a little bit who knows at the end of the day I have no idea what caused that spike it just happened it was this a random chart I just pulled off of our servers I can't explain it because this chart by itself isn't enough information in all of these cases metrics are your

starting point if you want to dive deeper you need to account for those social science aspects of design get into the mind of your players and understand the reasons behind their actions so a few years ago I did a talk about a map called trenches is anybody there for that yay a couple people uh heat map showed that uh players were only using one side of the map and the other side was being completely ignored and we figured this was some artifact of uh player preference or player behavior and so we just kind of push things around and and didn't really deal with it and then during the beta this map was a complete and utter failure people stopped playing the map almost entirely and this was entirely our fault we forgot that heat maps are just measurements at no point did they imply why people were in those specific locations they simply measured how many people were there and where they were

and since it did so poorly we decided to look at that map again after the beta and this time we went in objectively and sure enough we discovered that we had trapped players in their spawns this was not player preference when we fix that problem the real problem the map went on to become one of our most successful Maps ever now this is exactly why you need to ask questions that explain your metrics you need to stay honest and objective sometimes you fall in love with a design because it looks cool or because I mean quite frankly you built it right so it has to be good when in reality it may not be that great of a design and when that happens you run the risk of only seeing the data that supports your argument this is one of the reasons that iteration and communication is so important you need to establish your hypothesis test measure and repeat question everything you even need to question

your questions and what I mean by that is ask yourself why you want to know to begin with are you expecting a specific answer when you ask a question if you are then you have to be especially careful not to bias that question by Leading the results to your expected answer and this is where we failed in trenches we used heat maps to justify our design decisions rather than looking at it as just information leading to a solution and then once you do have answers it's pretty safe to start questioning those as well is that answer an actionable item remember your curiosity here isn't what you're important that's not why you're doing this your goal should always be meaningful change and since metrics don't provide you with answers but uh what does make them special is that they help you identify the important questions and questions are what for you to be objective to think outside the box and

look at your work from a unique perspective and the why is the most important thing you can ask why was the frame rate low in those charts why did the population numbers fall why were players on only one side of the map and the most important why of all and I can't stress this enough is to ask yourself why do you want to know the answer so in the trenches map we failed to ask why we were looking for answers that justified our assumptions without even really understanding what the hell it was we were looking for but we did learn a valuable lesson and when a similar problem popped up in a different map we handle it quite differently this is Academy which was from a later gears DLC and it had this round area in the middle of the map that was elevated with very tight and controllable entrances and lots of cover leading up to those entrances and of course of course heat map showed an incredible accumulation of

movement and death all in this one spot while the rest of the map was being ignored sound familiar so we wanted to figure out why we had some ideas but rather than just assuming we knew what the problem was we approached it objectively we looked at how players were dying not just where where they were in cover and why they were there to begin with and what we were able to determine was that teams would charge into this area to get the boom shot which is arguably the most powerful weapon in any gears game and then they would just Camp there and as they Camp there more and more people would run up there and since they had the most powerful weapon in the game they would kill them over and over and over so why would you ever leave right you had the power spot in the map so to fix it we moved the boom shot to the far opposite end of the map and then we continued to test and we saw a dramatic shift in player movement away from this one area the new results fell much more

in line with what we expected and hoped to see we had we were able to say hey this was a fixed problem and it was high fives all around until people started dying in the spot where we moved the boom shot too all of this data was telling us was that everywhere we put the boom shot people were going to die and this is exactly where our previous lesson came into play question everything ask why why why were players dying in this spot remember once you have your hypothesis you need to test it find the failures and then formulate a new hypothesis based on the current results just because the answer to the previous why was people were died because of the boom shot that didn't mean it was the same answer to this question right now so we set all that boom shot data aside and we started from scratch and here's a screenshot of that same area from a slightly different camera angle

and it turns out that people were abusing there over the camera shoulder and had a line of sight Advantage right by that Arrow so we didn't know at first why that happened so we went back and looked and all of our previous testing had been done when the map was an undecorated shell and if you watch right by where the arrow is you'll see that the walls in that version were significantly higher so when the map was decorated and those walls were inadvertently lowered it opened up a line of sight that didn't previously exist so we simply just filled that space back in more we tested sure enough everything fell into line heat Maps started to show a nice distribution of deaths all across the map and everything was golden uh we knew based off of what we had in trenches that we had to look at these as two separate problems there was a correlation to the boom shot moving but there was no causation you have to resist the urge to be satisfied with the

status quo and always remember to question everything all right so data is information metrics or measurements comparisons of that information analytics and this is the meat of the thing are used to interpret that information and analytics and metrics are very often confused but they have some very very distinct differences uh metrics is math-based it's it's it's hard science versus analytics which and this is a little bit controversial to some people is a social science that doesn't necessarily imply a real true answer it's something that you have to interpret or infer into the future versus metrics which is a measurement of past data really analytics has two functions first you use data analysis to gain insight and the goal there is to separate fact from opinion everyone has opinions and so it can be impossible to ever decide when you have two conflicting opinions which one is right

and opinions lead to argument and that can create unresolvable conflict so in order to present yourself uh prevent yourself from just aimlessly spinning on and on forever you need to look at the facts and see how they relate to the team's common goal it makes sense if you don't have a Finish Line you're never going to get there if you don't have a starting point and a Finish Line then there's nothing to measure so before you start anything outline your goals and Define your success State the purpose of analytics isn't just to impress people with pretty charts and and mountains of numbers um you need to separate facts from opinions and then use those facts to help you understand what's going on in your maps and this is the second major function of analytics using those facts to guide decision making if your goals are outlined before you start then they can be in ed to inform your process rather

than the result this makes you proactive rather than reactionary go back to the beginning of my talk and what were the three things I wanted to talk about what ask what you're building why you're building it and who you're building it for those answers Define your goals and if you stick to them you can tell when change in your map is actually going to be something meaningful rather than just decorative outlining your goals helps keep you focused there's less chance you'll be distracted by misleading information you have to be especially careful of really good presentations because visualization can be very important um but remember those maps of the Bo and Heights neighborhood the visualization didn't necessarily explain anything at all analytics is all about patterns and our brains love patterns we have this kind of inherent need to explain patterns so you see a pie chart and your brain veers off to thoughts Pac-Man and you end up Chasing

Ghosts charts can mislead you to believe that false information is real like seeing deaths due to boomshot in Academy when really there is no deaths due to boomshot in academy or think about those player charts that I showed you in the very beginning of the talk and if how if you had used them to inform your design process you would have been so far off base this is why some people call data visualization a halfway house it's that m point it's the stopping point from data to decision and remember my steam slide there it is right in the middle it is an answers it helps analytics help guide you toward a decision all right and this is exactly where you come into the picture designer take all those pretty charts and all that information and fold it into your knowledge and your design sensibilities by combining your instincts with data you go beyond functional Solutions into something that can become emotionally resonant that enables your players to

make those connections with your game with your level and with your work this is how you ask or how you answer what why and who and this is a very specific skill set that distinguishes a designer on the team from everybody else who contributes to design by offering their opinions everyone has opinions and ideas but not everyone understands the importance of how how and when to actually make changes and how to make that change meaningful does anyone use Spotify or Pandora well see most of you know what they are they use algorithms to predict what music you might like to listen to based off your basic listening patterns and for the most part they are pretty accurate but to make a playlist that's really good one that I really care about still requires human input into the process you have to go through through and thumb up and thumb down those songs

to get Beyond just a list of computer generated suggestions data only gets them halfway there it still requires that human input to get from predictive to meaningful now uh Netflix has a similar system on television and and movies but they've taken it one step further again data can't tell them how to make a TV show but they think it can tell them what they should be making when they created House of Cards they weren't just guessing at what people wanted they were analyzing patterns in their database so what do those patterns reveal the Perry Mason effect what would you guess is the most popular Show streaming across Netflix Perry Mason who are the most popular actors on everything viewed on Netflix the actors from Perry Mason okay let's get abstract if you look at just the themes of the shows across all the shows that are being streamed on Netflix

what comes up as the top list of themes those viewed in Perry Mason all the data that they had suggested that they should be remaking Perry Mason so why did they make house of cards as an individual data point Perry Mason is meaningless Netflix calls this a ghost in the machine it's a byproduct of complexity in slightly more technical terms as you increase the number of variables that you're measuring you increase the odds of a false positive result in any system there's always the chance that some of your results will come back as being statistically significant even if they're totally fake or wrong someone at some point is always going to camp at that that boom shot location in Academy that's just human nature and anyone could suggest a change to make them move but it's the designer's job to question why they were camping there why did Perry Mason show

up because that was a playlist generated by algorithm with no one to thumb it up or down I can look at any Call of Duty map or any popular map from any game deconstruct the technical aspects of it go through everything in there and then recreate it but is my new version going to be a success is it going to be emotionally resonant or is it just going to be a copycat if you're looking at a bunch of data then probably so is so is everyone else if you use that data they're going to as well and you're probably going to end up with really similar results you're going to make a copycat you're going to go and you're going to make Perry Mason and that remember that key component is the human touch and that's Netflix a secret one that they guard very closely what human influence do they put on their data how is it interpreted how did they get from Perry Mason to house of cards I don't know they won't say one can assume that they followed the logical associations of uh character

archetypes and themes and things like that but the specifics of that process how they combine their data with their instincts is part of the secret sauce that makes them special and this is the same design Instinct that makes you special too you can use data to inform your process but it don't let it lead you blindly into a decision or you'll just end up making another copycat you'll make the same level as everyone else over and over so throughout the history of the gears franchise we struggled with balancing our shotgun every time single time we looked at the stats The Nasher was at the top sometimes responsible for as much as 90% of all kills in a given month that's broken so we tried to fix it and every time we tried to fix it we messed it up our player base kept yelling at us they kept getting upset saying we just didn't understand but we continue to watch the data and we shared everything with them we made these big

presentations like this one that tried to explain the changes we were making and we buried them and ourselves under mountains and mountains of data we did range we did kill counts we did damage values we analyzed the pellet distribution patterns we did everything but we never added that human touch we never stopped to ask why people were using the shotgun in the first place we had no end goal no idea of what a fixed shotgun would actually look like and as it turns out The Nasher wasn't broken there was this huge passion for how it felt to move through the map with the Nasher at your side firing the gun was fun the effects and the sounds were satisfying connecting with it was totally empowering using The Nasher was a hugely successful and rewarding experience from a design perspective it was achieving every goal that we were striving for it was meaningful it was all the other weapons that were completely deficient and it wasn't until

the third game in the series that I think we really started to realize this and truly appreciate what was going on this was our ghost in the machine we had all the information in the world about The Nasher and tons of really pretty charts and graphs to look at but we were attached to them and they prevented us from seeing the real problems because we had all these cool looking charts and there was so much information there that we couldn't get through it we saw a problem in weapon balance but we had no idea the solution that we were looking for there was no defined state of success so in Gears Award judgment we were finally able to address this in what I think was a meaningful way and with all of our weapons balanced equally uh we toyed with the idea of giving everyone one single base weapon to start with the Lancer rifle and nothing else all of the other weapons including the national were scattered throughout the map as pickups and because each weapon was now

equally worthwhile they were all worth searching for and this gave the game a little bit more of an old school shooter feel which is great but at the end of the day we wanted to give our players more choice we wanted them to have some customization and personalization so we decided to move to kind of a more modern feeling uh Loadout model knowing our player's affinity for The Nasher it was being used as a lure in that original model so that we had scat shotguns scattered all over the map and people would go to all four corners and you can see a nice even distribution of play after we changed the loadout system our players immediately fell back into patterns of camping that plagued all the previous iterations of Gears now our heat map showed people moving and dying in these tight collected groups and we started to question if we had really addressed those problems in

weapon balance why was this happening well in contrast to our previous games judgment Maps were all asymmetrical rather than a single individual front that pulled players to a common ground every point in the map had to be a defensible front and since players were now starting with their favorite weapon The Nasher and they could use it effectively everywhere why would you ever leave not just spawn wait for somebody to run up to you and shoot them in the face you didn't have to go anywhere this is exactly what they did they all started camping this was of course a fixable problem and it led to some overhaul in a few of our Maps this is uh the overhead from a map called Gondola if you're at all familiar with the series and we had to add some extra access points to the top of the map and open up this kind of Highly defensible side area because those two areas were really easy to Camp of course this also meant that we had to rearrange our weapon loadouts add some extra Escape paths down from the bottom

uh rework some approaches remove a couple walls to accommodate other changes open up lines of sights on and on it was this endless Cascade of issues that we were completely unprepared for and this was only one map just like data without context change without defined purpose is meaningless it can only lead to churn wasting time and resources we rebalanced all of the other weapons in the game to accommodate the shotgun and that changing and then changing to the loadout model was just one more Small Change in the system but we were never able to address all the other problems that it created down the road and the game always felt imbalanced and off as a result of that now there's no question that our goal was absolutely to improve our players experience but how do you define what that actually is is there's no end there's no measurable result there there

was no tangible goal and no understanding of the complexity of all the other systems that were involved in this process we didn't address what we were building we didn't address why we were making the change and we didn't address who we were making those changes for we changed just for the sake of change and we ended up churning for months and remember no problem is ever like this is ever really 100% fixed when we moved that boomshot in Academy we didn't solve the problem we created a new hypothesis that players would would move away from that area and we found that the new data supported our new hypothesis statistically seeking we saw that players were on average moving away from that area instead of camping it theoretically speaking we could continue to test forever get more information every time you do it right you refine your hypothesis you improve your probability space it's not like that data is ever going to get old or be thrown away it just piles up but the

reality is that at some point you do need to stop you have a schedule you have a budget and you have all these other things that need prioritization in complicated systems like a multiplayer game there's always the chance that the results of your next play test are going to reveal a new ghost in the machine so at some point you need to stop you need to accept that you know the best that you can know right now and move on in order to avoid churn in order to keep from Chasing Ghosts you need to outline your goals outline your goals and define success from the start all right cool stories bro what do I do with all this information uh good design is very often about simplification streamlining and clarification I look at analytics as a mindset that helps me focus on those principles and use them to inform the decisions that I make every single day

continuing to ask why strengthens the impact and validity of every decision that we make it helps focus on what you're building and how you should approach it and this helps us to Define our goals and once change is made it can be Quantified and measured this separates opinions from information and leads to meaningful change in your maps and and remember there are no guarantees one of the most famous sayings in Hollywood comes from a screenwriter William Goldman who says nobody knows anything not one person in the entire Motion Picture field knows for certainty what's going to work every time out it's a guess this is why people passed on Raiders of the Lost Arc and Star Wars and Harry Potter and the Twilight series all predicting their inevitable failure failures just like those goofy kids that started Microsoft and apple and epic games but reality is games do fail every

day it's bound to happen Temple Run was this huge success right out of the gates but you have to remember that they were an anomaly it's more telling that roio had 56 other games before Angry Birds and that Candy Crush the number one game in the world was more than 1050th in King's roster so it's really easy to look back and say yeah this is what we did wrer that's what they did right but it's even harder to look forward and predict this is what we should be doing because there are no guarantees but analytics can improve your chances for Success they help you both understand your players and the content that those players are consuming the content you are building for your players the better you understand your own maps The Better You can predict the way people will interact with them and the better you'll be able to innovate and come up with new and interesting and meaningful surprises the cycle of

innovation and experimentation is how you Magic Band your Maps experimentation is the foundation for analytics in the same way that iteration is the foundation for Design This is Six Sigma it's a uh analytical model that was designed for the manufacturing industry it's this weird process that requires like training and certification and I don't have any of it but when I saw this it struck me how similar the methodology is to level design start at the top right and look at the Define State what are you building why are you building it who are you building it for does it support a game system or the narrative what are your goals what are you trying to achieve outline your goals and Define your success and then go on to measure with a clear outlining of your goals you can Define your starting point if you know where you are now you can measure where you're going to be later this is metrics

maybe you're introducing a new weapon and you want to track success maybe there's some important visualization or a visual element like a tower and you want to use heat maps to track where and when people are getting there just remember that once you have this data it still needs to be interpreted and analyzed data is information it is not answers so then you analyze cover your bases don't just accept that first answer that pops up maybe players are dying frequently around a powerful weapon but maybe there's other factors like map geometry separate correlation and causation one of the biggest lessons I've learned as a designer is that if I need to repeatedly defend a design decision with my team with my players it's probably a bad decision and that's especially true if the data that I'm or the information that I'm using to support that decision is pure statistics there's always going to people who don't immediately get a

change that you make or a design decision that you make but if they don't get it quickly through testing and iteration you need to step back and ask why what part of your design isn't being communicated question everything analytics don't give you answers but they do improve your chances for Success you are the one who adds that human element this is where we get to improve this is your moment to shine this is your role as an LD anyone can look at those data and read the numbers but it's the designer's job to know how and when to incorporate that data to make a change that results in meaningful impact and then finally control understand that every change you make has consequences systems are complicated change one system and it affects another always ask what's the bigger impact of this small change what's the goal of that change nothing nothing is simple every change matters and you're going to mess it up

but that's okay that's part of the process be honest about it and learn from your mistakes when you fail at something go back redefine retest this is exactly why this process is a continuous loop you identify a question you develop a hypothesis you test you measure and you repeat this is iteration every time the loop repeats your chances of success get a little better you improve your probability space so much of science is about patterns and as I've mentioned analytics is about discovering those patterns but there's patterns in non-science areas as well look at Pop Culture few years ago we had shooter after shooter coming to the market and everyone was fighting this endless stream of World War II battles kill Nazis kill Nazis kill Nazis until Modern Warfare showed up and it changed the context to something that was suddenly culturally relevant they mixed in some RPG elements and and did some other things but that setting made sense

to the current pop culture patterns they broke that copycat mold by innovating they shifted the Paradigm of the entire industry and how do people react well they started making Modern Warfare Modern Warfare Modern Warfare it's early but I feel pretty safe in saying that the same guys are going to break that pattern again with Titanfall they understand the important of innovation a the human element of setting the bar and achieving goals they know what they're building they know why they're building it and they really know who they're building it for they understand the importance of everything and this is your challenge there's one thing I want you to take away from today is that analytics can guide you toward a true understanding of your game and your players it's not about the math it's about asking the right questions and knowing when and how to mix in your own personal instincts and insights create meaningful change for

your players understand not just who they are but how and why they play not to milk them for money but to help you create an experience that they deem is worth the time uh and investment of their attention their experience is in your hands and everything that you do is important thank you exting