The Game Outcomes Project: How Teamwork, Leadership and Culture Drive Results

[Music] okay looks like it's time for me to start just a reminder if you could shut off your cell phones and remember to fill out the evaluation forms after we're over can everybody hear me in the back good good thank you all right my name is Paul tozer I'm here to talk about the game outcomes project and what it taught us about the science of creating effective teams and how teamwork leadership and culture drive outcomes okay my surface pro 4 has hung apparently oh there we go got something ok quick introduction I I'm the owner and general manager of mothership entertainment in Austin small studio built a building an unannounced science fiction strategy game I also earned an

MSC in technology management from University of Pennsylvania and the Wharton School of Business and the only reason that's relevant is because the Wharton School of Business inspired that this the game outcomes project not a consultant I'm not trying to sell you anything I'm a 22 year industry veteran motivated by intellectual curiosity I'd like to take the opportunity to introduce the game outcomes project team we are an independent industry academic partnership in the upper left dr. Karen Bureau of MacEwan University was our an enormous help with statistics also julianne a pillar of the Wharton School of Business she is a PhD student she helped us a lot with our question design our industry members included erik byron david wegg bright Jing hua yang also known as Z Ben Weber lucien parsons and n dark tang who translated all of our articles into chinese i'd also like to

thank those of you in the audience who participated in the study it would not have been possible without you a couple quick notes i'm going to be doing the starting with the survey design and notes on how we designed this they gonna talk about non cultural factors gonna talk a little bit about crunch gonna talk about culture gonna very briefly discuss non-correlated factors and then draw some conclusions from that so in the fall of 2014 our team designed a survey we decided to ask game developers a bunch of questions about their most recently finished game project this included a hundred and sixteen questions about culture followed by four questions about the project's outcome we thought okay there have to be at least some factors in this list about teen culture that are going to correlate with success or failure and if we can get a couple hundred responses and we can correlate them and see which of the responses are positively and negatively correlated with different kinds of outcomes we can learn a lot from that so this was a first-of-its-kind correlational study that correlated a development teams internal culture with

their actual results we received 273 responses from game projects that had not had been cancelled nor abandoned during development and based on that we did a lot of data analysis and wrote up a series of five articles on Gamasutra and for those who have you who have read the Gamasutra articles i want to assure you the lot of the data that you're going to see today is different in its form if not the conclusions from what was in the five gamasutra articles our hypothesis was that different teams have different cultures and we also hypothesized that chance always plays a role in game development but we felt certain that outcomes have to be strongly influenced by their team culture in the aggregate and we certainly found that the first one is undeniably true and we believe strongly the second one is true as well and we'll talk about why we feel that way a little later on so we wanted to know is this true and if so which cultural factors most strongly influence the odds of success and what can we learn from that

we had an optional question at the end asking game developers of what project they were actually referring to I'm not going to read through this whole list only fewer than 5% of respondents actually told us which game they were referring to but based on that we know that our 270 responses included at the very least these games survey design few disclaimers before we start this study was not an academic paper so it was not peer reviewed it's an ex post facto analysis in other words it's based on a survey asking people about projects they recently completed in an ideal world of course we'd have a time machine and we could do a little a B test and go back in time find the same development team and say okay same people same time working on the exact same game project now I want you to make a slightly different decision during development I want you to use scrum instead of agile or waterfall instead of scrum or I want you decide not to crunch instead of crunching how does that affect your outcome the reality is of course we have no way of doing that there is no way to

get exactly the same two team the same team to do the same thing twice because they will have learned the first time going through so in the real world there's no way to run that experiment all right so it's cut the subject to cognitive biases I might look back on a project with a disastrous ending and be bitter about that and that might affect my survey response finally correlation isn't causation our study gave us a little ton of correlations we could look at but we can't technically prove that any of these correlations are causal factors by themselves and I'll discuss correlation and causation a little bit later in the talk a few quick notes about our survey design and the steps we took to minimize bias we randomized the order of the cultural questions and Survey Monkey to remove any influence that a single ordering might have had on the way the questions were answered we had a lot of redundant questions where we'd asked the same question twice with redundant tones we would ask did the team do X in addition to asking did the team avoid doing X and it was very interesting

because we found there was an almost perfect inverse correlation between those answers so the fact of the matter is those questions were kind of redundant we could have cut down on the number of questions just by choosing one or the other but it was a nice reinforcement that our questions were being answered honestly and accurately finally we asked our 4 questions about outcomes at the very end of the study because we didn't want to bias anyone about what we're actually looking for when asking the culture questions let's talk about what we mean by outcomes obviously there's no single definition of what makes a good outcome different game projects have different goals and different definitions of success if you're working on a triple-a project you probably care more than anything about else about your return on investment if you're working on an indie project maybe you care more about critical acclaim if it's your first time making a video game maybe you just care about getting it done maybe you're trying to make a political statement with your game maybe you're trying to kill fascists like woody Gregory's guitar or something like

that and that's fine so in that case maybe you care more about your team's internal satisfaction with meeting your project goals so our team came up with a list of four factors that we felt most developers would agree generally describe positive outcomes our feeling was that most game developers would say these are the factors that generally describe success so we asked the following four questions I'm not going to read all these in the interest of time so we asked things like to the best of your knowledge what was the game's financial return on investment or ROI in other words what kind of profit or loss did the company developing the game take as a result of publication and we'd asked this on a seven-point scale and all four of these questions were on a six or seven point scale so we were able to take those four questions convert them into normalized values between zero and one then multiply each of those by twenty well I'll get to that a little later so we felt it made sense to combine all four of those into an aggregate outcome score our view was that a truly successful project was one

that had a good ROI was completed on time received high critical acclaim and that the team felt achieved its goals we tried a bunch of different ways of building that aggregate outcome score we tried one based on probability theory multiplying the outcome scores together we tried one based on weighted sums and tuning the optimal weights in the end we found that the thing that worked best and correlated the best with everything else in the study was simply adding those four scores together so we've got four values between zero and one just add them together and then that number between zero and four correlated better than anything else we came up with so we stuck with that and then we multiplied that value by 25 to get a number between 0 and 100 so it's like a letter grade so we have a bunch of cultural factors as represented by over a hundred and ten survey questions and we can correlate any individual culture question or group of questions with any one of those outcomes and the aggregate outcome score but what does that correlation mean for

example maybe we find that teams that used more outsourced labor instead of internal labor influences the aggregate outcome in some way maybe it makes it higher or lower what can we read into that well because the outcomes happened after the game shipped there is really no way the outcomes could have caused the cultural factors so if there's any causal relationship at all it has to be the cultural factors causing the outcomes now sure it's a survey and it's possible somebody's memory of the team culture while taking the survey was influenced by the outcome that's possible and maybe a biased the survey a little bit but assuming bias didn't sway the survey too much either there is no causal relationship or it's the cultural factors caused or some other factors behind them causing the outcome a quick note on the statistics used in our analysis of the survey results not going to go into this in too much detail most you're probably familiar with the concept of correlation just to be sure here's a quick refresher the correlation between two variables is the degree to which those variables show a tendency to

vary together so you see the little boxes the one on the far upper right there's no correlation X and y have nothing to do with each other if I know the value of x I cannot say anything about the value of y in the upper left-hand corner that's a perfect in almost perfect inverse correlation as X gets bigger Y gets smaller and so I can say a lot about the value of y if I know the value of X similarly in the lower right corner we have a perfect positive correlation x and y both get bigger or smaller together so if I know the value of Y I can perfectly state the value of if I know the value of x I can perfectly state the value of y we used a Spearman correlation with the outcome score for most questions I'm not going to go into details on that look it up on Google if you're interested I'm sure Google can explain it much better than I can and the reason for it we used Spearman instead of the more popular and commonly used Pearson correlation was because many of our questions were based on a Likert scale for questions that had two two mutually exclusive answers we use a wilcoxon rank-sum test to detect in differences for questions with more

than two mutually sigh mutually exclusive answers we use the kruskal-wallis one-way analysis of variance but again if you're interested google it look it up in the articles there's a lot more detail there that I can't get into so let's assume we have some cultural factor like teams that drink coffee obviously we did not actually ask about coffee drinking in the study but let's roll with it so we find the answers to that question on our survey correlate with higher values on the outcome scale for project delays higher meaning better the graph might look something like this the gray dots form a rising diagonal line and there's a strong positive correlation between coffee drinking and teams that ship to the games on time what claims can we make about this if we have a correlation like this can we say that drinking more coffee improves your odds of shipping a game on time well no not exactly but we can say that one of three things must be happening first there might be no causal relationship depending on the statistical p-value of the correlation

we can rule that out as highly unlikely in some cases and we talked about p-values and the articles I don't really have time to get into it here again strongly recommend googling that if you're interested briefly the p-value is a measure of how likely the results are to have occurred by chance the second possibility is that maybe a really does cause B and maybe drinking more coffee really does help you ship on time the third possibility is maybe there's some other hidden causal factor involved for example maybe we've just hired people hired people who work harder and harder working people happen to drink more coffee that's totally a possibility but it does let us pretty much rule out the opposite if someone says hey drinking coffee actually hurts your chances of shipping a game on time we can show them this chart and say okay if that were the case it should have shown up in this chart we should see an inverse correlation but we see a positive correlation instead and you can say okay for any correlations where we find the statistical p-values are low enough to pretty much rule out coincidence we can say that there could

have been some other hidden input C which accounts for the results some hidden causal factor like just hiring better people but we asked over a hundred and ten questions over a very broad range of cultural factors and it's very hard to imagine that there really is very much we left out of it so we can't a hundred percent rule out the idea that maybe more successful teams just hire better people and the correlations of all the culture questions were just coincidence and there's some magical thing those better people are doing that isn't accounted for in our survey questions that's totally a possibility we can't rule that out and you're welcome to believe there are other factors involved but what I would ask you to think about is this if you find there are things that other teams are doing that are strongly correlated with success or failure it might not hurt to learn about them and see how that applies to your own teams maybe there's some other effector involved and maybe not but if there are simple changes you can make that might impact your outcomes why not make them I'm going to start off the discussion of the results with what I'll call non cultural factors this is a

graph of the average team size plotted against the aggregate outcome the horizontal axis is the team sized based on a logarithmic scale with larger teams toward the right the vertical axis is the outcome with better outcomes at the top we've also colored the dots based on outcome so Green is better red is bad so as you can see the trend line goes upward toward the right indicating the very large team seemed to have slightly better outcomes although the correlation here is not statistically significant the total project duration had a negative correlation and it was statistically significant you'll notice the black trend line here goes downward so and the horizontal axis is the total project duration but that's not really surprising projects that are in trouble are naturally going to take longer and experience more project delays so nobody should really be surprised by this we asked five yes-or-no questions about financial incentives if you're spending money or at least offering money to try to motivate your team it's interesting

to ask whether it really makes a difference and four out of five of those we found there was absolutely no significant correlation from left to right we're looking at team based incentives incentives based on royalties incentives based on Metacritic review scores and other royalties and in each case the no answer is on the left and the yes answer is on the right and we used a wilcoxon rank-sum test to detect significance and results here but one place where financial incentives did make a difference was with individually tailored financial incentives in other words pay for performance plans and things like that the mean outcomes for teams that did use individually tailored incentives was sixty three point two versus 56.5 for teams that did not again we use the wilcoxon rank-sum test for statistical significance and we found a p-value of 0.7 teen for this we also found that this effect seemed to drop off for large teams that is teams of size greater than 50 and we're not sure why this would be the case however it does suggest that if

you're going to offer financial incentives to your team it's probably a better idea to offer them to specific individuals to specific tasks within a limited time frame along the lines of a paper performance plan rather than offering royalties or anything like that we asked a question about what production methodology a team used whether it was waterfall agile agile using scrum or other we also had an option for don't know and we found that there was essentially no statistically significant difference waterfall seems slightly lower in this image but it's not statistically significant so this is a very surprising result advocates for agile and scrum in particular seem to often treat it as a Holy Grail and we spend a ton of time talking about methodologies at GDC but our results really make you scratch your head if a Gillen's from really are such holy grails why don't we see a difference in outcomes here's another way of looking at the same data each of the colored dots is a single team and our survey the orange teams used waterfall the green

teams used agile the red dots used agile with scrum the purple dots use other and the blue dots answered I don't know so do you see the pattern here no because there is no pattern right obviously trick questions so you'll notice that all of the lines the orange line the purple line the red line green line are almost coincidence and that is waterfall scrum agile and other and they are almost identical in terms of their outcomes you'll you'll notice that the blue line which is the people who answered I don't know what our team's production methodology is which clearly indicates that there were serious production issues on that team that plummets pretty hard team size of five the horizontal axis here is the size of the team so I want to be clear that I'm not saying that production is pointless I'm not saying that it's important only that it probably has the effectiveness of production probably has a lot more to do with how you do your production how you fit it to your team rather than whether you're using agile waterfall or anything else we also looked at team experience

levels we asked it question early in the survey about the average level of experience on the team in terms of the number of years and then measured statistical significance using a kruskal-wallis test as you can see there's a real progression here especially as you move from four to five years which is the middle column to six or seven years which is the fourth column outcomes start to get a lot higher notice how in particular four teams with eight or more years of experience not only is the average much higher but none of the outcome aggregate outcome scores are below 45 that lower right-hand corner is almost completely empty teams with an average of eight or more years of experience clearly don't fail nearly as much right that doesn't necessarily prove that the experience caused the better outcomes it could simply mean that to a large extent more experienced developers are attracted to teams that have historically been more successful and their experience makes them more likely to be hired by those teams but either way experience clearly does matter we asked a question about what technology solution the team used to

build its game from top to bottom the rows are top row new engineer technology solution second row internal were proprietary engineer technology such as EA frostbite the third row is external or licensed to engineer technology such as unity or unreal or Crytek the fourth row was a technology or engine from previous version of the same game or a similar game in the series on the bottom row was other and what we found was that none of these really had any statistically significant correlation with the project outcome except for the fourth column which is the engine for a sequel based on the previous games engine and when you think about it that shouldn't be even a little bit surprising because if you are making a sequel to a game obviously first of all the first game was successful you know what the game is you know what your market is you know how to identify your market you already have a technology base that you use to build the engine the first time you probably have the same team that worked with the engine and they already know how to use it for this type

of game so there are a whole lot of advantages in that particular case so it's really no surprising that those teams showed much higher aggregate and statistically significantly higher aggregate outcomes let's talk about crunch there's an attitude among many developers that crunch is a necessary part of any good game development project there's a quote from an interview interview with a certain veteran game developer that sums up this attitude crunch will always exist in studios that strive for quality and a large percentage of our industry shares this view and I want to point out that the implicit assumption here is that extraordinary results can only come from extraordinary effort and that extraordinary effort necessarily implies extraordinary overtime now I think most of us are aware of the data of the harmful effects extended overtime on individuals there's tons of data clearly illustrating that the abusive overtime not only leads to lower total

productivity not only drives a lot of talented people out of our industry but it leads to a higher risk of relationship failure mental illness alcohol abuse depression and a host of other problems I think many of us are also probably aware of the recent findings that pulling all nighters causes permanent brain damage and outside the game industry it's generally accepted that overtime abuse is a bad idea and the practice of crunch is relatively rare but there's an implicit assumption here first of all the crunch actually improves the quality of the game but how do we know that's true I'm sure I know a lot of you were probably looking at me like I'm crazy and thinking hey wait a minute on my last game we crunched 80 hours a week we never would have finished it if we hadn't crunched and yeah maybe that's true and sure there have been quite a few undeniably great games made with crunch but I also know a lady who chain-smoked and lived to 95 right can we say that because she was chain-smoking the chain-smoking caused her enhanced longevity or should we look at the

broader data and say okay putting it in perspective that ladies extreme outlier it actually takes seven years off your life on average and how can we know for sure that that lady wouldn't have lived to 112 if she hadn't been a chain-smoker and similarly if you ship a great game and you crunched to make it do you really want to give crunch all the credit for that can you really say our game was great because of the crunch how can you be sure it wouldn't have been even better if you hadn't you don't have a time machine any more than I do you can't go back in time and rerun that experiment to see how it would have turned out if you'd made slightly different decisions so I want to ask is there really a solid link in the aggregate between more crunch and better outcomes or are we just assuming the crunch works without any real aggregate data to back it up we asked a total of five questions relating to crunch in our survey two of those questions related to the overall amount of crunch the team's engaged in the

three bars in this graph show all of our 273 samples grouped into three categories the bar on the Left shows the 25% of teams that engaged in the least amount of crunch the bar on the right shows the 25% of teams that engaged in the greatest amount of crunch and the bar in the middle is the central 50% of teams that did a medium amount of crunch and the vertical axis is aggregate outcome score with the top of the scale as before representing teams with the most strongly positive outcomes so that makes you wonder if crunch is a necessary precondition for quality how do you explain the green dots on the top left we're looking at a whole lot of successful teams there and they achieve their success with the least amount of crunch in fact the group at the least crunch had the greatest number of project with positive outcomes of all three groups I'm sure the three date of the data points inside that box all strived for quality and yet they didn't crunch very much if at all so how did they manage to achieve success and

similarly take a look at the lower right we see a whole lot of red dots toward the bottom showing teams that experienced very poor outcomes despite doing more crunch than anybody else in fact most of the teams that experienced poor outcomes use the greatest amount of crunch let's dive in a little deeper among the teams that did the most crunch on the right column there were 71 data points with an average outcome score of 48 the middle 50% of boxes in this category that is the dots inside the gray boxes in the vertical box and whisker plot had outcome score is between 34 and 63 now let's look at the bar on the Left these are the 25% of teams that did the least amount of crunch here we have 64 data points with an average outcome score of 62 the middle 50% of those samples again the ones inside the gray boxes had outcome scores between 51 and 74 so you may be looking at that and saying okay well that's nice but what about when crunch is voluntary I mean Hey look at me I'm doing overtime

because I'm passionate about the product not because I have a boss holding a gun to my head right so now let's look at solely voluntary crunch we had two questions on our survey asking whether any crunch done on the project was voluntary or mandatory regardless of how much crunch actually occurred the left column is the mandatory crunch group the right column is the voluntary crunch group and just as before the middle bar is the middle 50% of teams here we see a very strong and statistically significant correlation the teams that reported the crunch was mandatory were far more likely to report low outcomes and the teams that reported that any crunch was voluntary were far more likely to report positive outcomes so in the 25% most mandatory crunch group 60 data points in that bar the average outcome score was 49 50 percent of them were between 34 and 63 in the group of the 25% most voluntary there was 71 data

points with an average outcome score of 67 50% of those were between 54 and 80 so here's a graph little table showing all the data sorted by how much crunch they did and whether that crunch was voluntary or mandatory so you can see the horizontal axis left side is voluntary right side as mandatory top row is the most crunch the bottom row is the least amount of crunch now look at the upper right corner the teams that that were in the 25% quartile of saying the crunch was the most mandatory and that also did the most amount of crunch had a median outcome score of 39 point 84 in the lower left you notice the little green box with a value of 70 point 62 this is the teams that were simultaneously reporting that they were in the most voluntary quartile and the least amount of crunch quartile now let's say you're coming into the industry for the first time you don't

have any preconceived notions about crunch at all maybe you don't even know what crunch even is right which of these groups would you want to be in now you may be looking at this and saying wait hold on a second Paul this is all BS right you're picking up a spurious correlation all you're really picking up is whether teams are experiencing problems right because everyone knows the teams that are experiencing problems are more likely to try to use crunch to solve them and that's true I've certainly know teams that were just dedicated to crunch no matter what because they thought it worked but yeah there are definitely a lot of teams that say hey we're now having problems let's crunch and that's how we will try and dig ourselves out of this hole so we're going to refer to this as the crunch Salvage hypothesis which is crunch is more likely to occur on projects that are already in trouble and these results are just picking up that underlying correlation so here's what we did to try to tease out whether that was actually what was going on we built a predictive model based on a

linear regression for all of the cultural factors in our entire study right and we found that there were 30 of them that we could use put them into a linear regression and build a linear regression that had a correlation of 0.82 with the aggregate outcome score that is an incredibly high correlation and the p-value here was incredibly close to zero so we could predict with a very high degree of accuracy what a team to aggregate outcome was going to be based on those 30 cultural factors then what we did was we took that same linear regression and we removed all the questions related to crunch this roof lowered the correlation from 0.8 to to 0.8 one so you'll notice here in the upper left side culture teamwork and production factors excluding crunch that was what what went into the crunch free model so based on that we were able to look at every team and say okay what would the crunch free

model predict the aggregate outcome score would be and then we could ask how much did they crunch and what is the difference between their actual outcome and the model that are letting a regression reported in other words how much lift did crunch give them beyond what this very powerful linear regression model reported and here's what we got here's the same chart with the same categories instead of showing the actual outcome score we're just showing the lift that these teams seem to experience from added crunch and again note that this is a zero to a hundred scale for outcome scores right so a value of three or four is tiny it's almost insignificant here you'll notice in the upper right the teams that were in the most mandatory quartile and the most crunch quartile had scores that were 3.91 at outcome points lower than the crunch free model would have predicted and in the lower left corner

the teams that were both in the most voluntary quartile and the least amount of crunch quartile had aggregate outcome scores three points 17 points higher than you would have predicted so this suggests that maybe crunch actually has a minimal effect on the outcome of your project another way of putting it is your studio studios culture probably already dictates most of the outcome and crunching really isn't going to change that I want to be clear that we didn't set out with any preconceived notions we're trying to prove anything we just followed where the data led us with an open mind but the data spoke very strongly on this now you might be saying hey my team crunch and our game turned out great and that's true some projects do a lot of crunch and they turn out just fine but I have to ask do we crunch because it works or do we do it because we believe it works a single data point is meaningless on its own remember the 95 year old chain-smoker there's no way to tell unless you do a thorough statistical analysis on a large

data set and all of us in this room have different opinions on crunch all of us have different experiences and if we go by who's an expert or who's got the most experience we'll still just end up with a roomful of arguing because any position you can take on an issue like this I can find an equally qualified person on the other end of the spectrum to argue the other side there are a couple potential weaknesses with our findings on crunch one is that it doesn't precisely define what crunch is right we were not able to nail down here's how many hours a week constitute crunch right it doesn't nail down the point of diminishing returns for overtime in this next version of the study later this year we're going to try to quantify that but my challenge to you if you disagree is to prove me wrong but prove me wrong with data nothing else is going to convince me because ok let's talk about culture and this is where we saw the really strong correlations in our study so in order to be able to

measure culture and analyze it systematically and quantitatively we need some sort of conceptual model for it and we throw around a lot of words in the industry like team work and culture but those are all very big and wishy-washy words we need to nail down exactly what we mean by good teamwork or bad teamwork or how one Studios culture differs from another Studios culture on a very precise level and this is important you can't debug computer code unless you understand the programming language it's written in and similarly you can't debug your team unless you really understand the language of good teams just to give you an example you know of a conceptual model for effective teams I know some managers who try to use Freudian psychoanalysis as a management tool and that is not the right tool for the job if you're playing armchair psychiatrist with your employees you're not going to fix anything you're not going to understand

your employees and you're really just going to tick them off and you're probably going to make a dysfunction team even worse so it's very important to have the right conceptual model of effective teams back in 29 2009 through 2012 I got an MSC degree in technology management from University of Pennsylvania in the Wharton School and as part of that I studied organizational behavior in design under Adam Graham the top-rated professor at the Wharton School and in that class they taught a scientifically validated model for team effectiveness you can use to analyze and diagnose any team in any industry and I found that really fascinating because I'd never realized there was such a thing before or that there was such a thing as a model of team effectiveness or they could be scientifically validated and it put all of my own experiences in the game industry into an incredible perspective and I started to look back on my own past experiences and see how so many of the teams that I had worked with had different cultures that correlated dramatically with the results they actually achieved the Wharton

effectiveness model is based on the book leading teams setting the stage for great performances by J Richard Hackman so we decided to use Heckman's model as one of the foundations of our survey here's Heckman's model in a nutshell number one real team this means there have to be clear team boundaries who's in the team who isn't team members have to have clear authority over their own tasks they have to be able to build those tests themselves not have them handed to them from on high the team composition has to be paid based on diverse skills there has to be stable membership if people are constantly coming and coming and going it's not a real team number two there has to be a compelling Direction a clear motivating goal that guides and motivates the team's efforts number three enabling structure tasks roles and responsibilities have to be clearly specified and designed for individual members and there has to be a clear definition of who is and who isn't on the team number four supportive context this means there should be incentives

encouraging that desired behaviors and discouraging the unwanted ones there should be tools and affordances to get the job done and team members need to have a level of psychological safety that they can raise the red flag call out when the ship is about to hit an iceberg and know that they're going to be listened to and not politically punished for that finally they have to have access to expert coaching this means motivators consultants and educators outside the boundaries of the team who can help them raise the game in addition to that we also looked at the model is described in the book the five dysfunctions of a team by Patrick Lencioni this book describes a very different model and it describes how things typically go wrong with the team's culture but rather than explaining it this way I'm going to use Yoda to help me out it starts with a lack of trust between team members which leads to a fear of conflict because nobody trusts anybody else and nobody feels like they can speak out safely this leads to a lack of commitment because nobody feels engaged with the project the lack of commitment leads to

an avoidance of accountability this leads to an inattention to results and a disconnect between the team's perception of itself and the results they're actually getting and that leads to games that don't live up to their potential and that leads to hate hate leads to anger anger leads to the dark side you know how it all goes so what's nice about these two models is there essentially orthogonal to each other for the most part the Wharton School model that is the Hackman model described external factors structural factors context how to set up the right environment for a team to really excel whereas the five dysfunctions model the Lencioni model describes the internal dynamics of a team and how those internal relationships can go awry and what to do to fix them when that happens I'm going to refer to these as the Hackman model on the left and the Lencioni model on the right so here's what we found and yes there's a lot of numbers on here but before you freak out I promise I'm not actually getting it show the numbers this way I'm going these are all in the articles on Gamasutra if you want to read them there I'm going to give you a very different

view of the data really quickly on the left side we have five of the questions that we asked with this particular aspect of the Hackman model along the top we show the correlations with project delays ROI Metacritic internal goals all the various outcome factors as well as the aggregate outcome score right but numbers are boring and we rather than showing a whole bunch of correlation values what I'd like to do is show you the actual responses from the top teams and the bottom teams on this slide we have three of those questions from that section I'm sorry two of those questions from that section I've taken just the top 25% of teens from our survey in the bottom 25% of teams inter as ranked by their aggregate outcome score so the blue team's represent the blue area represents the top teams the red represents the bottom teams we've omitted the middle 50% of teams completely for these slides that I'm going to show you and across the horizontal axis we had the level of agreement or disagreement with each

question where agree completely is all the way on the right to neutral in the middle to disagree completely in the lower left and the vertical axis represents the total count of responses in that category for either group okay so look at what happens the question is the team composition didn't change during the course of development other than growing when needed among the top teams in blue 16 people responded agree completely and 17 people said strongly agreed among the bottom 25% of teams only six people said agree completely to this question and ten said strongly agree so the curve is shifted way out to the right and you'll see this with almost all these questions as we go forward where the top teams agree much more strongly with positive statements on the left the organizational structure and membership of the team and we're clear from the outset seventeen people

from top teams said agree completely well only four people from the bottom 25% of teams said agree completely and on that same chart 11 or more people from the bottom teams said either disagree completely or strongly disagree that the organizational structure and membership were clear while among the top teams exactly five people gave each of those responses on the right side that team composition didn't change during the course of development other than growing when needed so second aspect of the Hackman model compelling direction does the team have a clear motivating goal that helps focus and guide their efforts on the left side the development plan for the game was clear and well communicated to the team on the side the vision for the final version of the game was clear and well communicated to the team you can see the blue team's agree much more strongly the third factor enabling structure our tasks roles and responsibilities clearly specified and designed for individual

members and properly match to their individual skills on the left side team members tasks were well defined and clearly specified on the right side team members responsibilities and job roles would carefully matched with their particular skills and abilities you can see the blue teams in both cases agree much more strongly with the red team's support of context does the team have the psychological safety they need to speak openly about problems and did they have faith that if they speak up you'll take appropriate action and they won't be politically punished for that do they have incentives encouraging the desired behaviors and disincentivizing the undesirable behaviors and do they have the tools and affordances they need to get their game to get their jobs done so on the left side we have the question mistakes were treated as learning opportunities or a chance to improve not a nail in your coffin on the right side we asked people personnel issues within

the team teamwork problems and HR issues were dealt with professionally and appropriately expert coaching does the team have access to experts outside the team's boundaries including motivators to help push them forward consultants to help guide them and educators to help improve their skills on the left side you see the team explicitly attempted to develop team members skills or otherwise assist them in their skill development outside their normal day-to-day work you see the blue team's agree much more strongly with this on the right side we received some form of coaching or guidance to enhance our effort or improve our effectiveness as team members now this is a little interesting because we can see in this case both the red team's and the blue teams generally disagree with this so this is an industry-wide pattern we don't do a lot of training we don't do a lot of coaching it tends to be pretty sink-or-swim but still in this case there is a lot more agreement from the blue team's than the red team's we also asked a bunch of questions

related to Lencioni five dysfunctions model we've got very similar correlations here to the strong correlations we got with the Hackman model we asked around 20 questions in this category but in the interest of time I'm only going to show you three of those the first one it was safe to take a risk on this team and stick your neck out to say something that needed to be said in the middle most team members bought into the decisions that were made now on the right side we've got a very interesting outlier because here we've got a question that was asked in a negative tone we're not asking did the team do something good we're asking about something bad that happened on the team and as you would expect here we see the red and blue have flipped the blue teams are on the Left indicating much stronger disagreement with a statement the red teams are on the right so the question is some team members put their ego or their own need for recognition above the collective goals of the game projects the blue team's the top 25% of teams strongly disagreed with this in

the aggregate and the red team's strongly agreed overall we also used a third book which is 12 the elements of great managing this book is based on a team effectiveness model built by Gallup which is based on thousands of interviews with teams around the world so like the Hackman model used by the Wharton School this is a scientifically validated model for team effectiveness now for the for the most part this book is the least useful of the three and I'm not going to spend anything more than this one slide on it because most of its factors are already covered by the Hackman model in the Lencioni model but there are five factors I wanted to call out that are unique to this book a connection with the company's values had a correlation of the point 39 that's pretty strong correlation receiving regular powerful insightful feedback having peers who are committed to doing quality work my opinion matters and recognition and praise we also asked over two dozen questions regarding other factors specifically

related to game development having a shared vision for the game had a correlation of 0.5 that's an incredibly strong correlation for anything in the social sciences and that is the single strongest correlation of any of the questions that we asked in our entire study resolving disagreements about the game design also very strong point 45 justifying and communicating design changes Point 45 celebrating novel ideas even if they don't achieve their intended result point 38 a respectful relationship between the management and the team point 36 so here I want to quickly discuss factors that were not correlated with outcomes and as far as we can tell these factors had no global correlations we've already discussed at the beginning team size on any absolute scale production methodology whether team used agile scrum Waterfall anything else and financial incentives other than individually tailored financial incentives none of those had any global correlations with outcomes we had a

question a couple questions about cross-functional vs. / disciplined teams here we saw no global correlation we asked a question about game genre and unfortunately I like to think that they're hidden correlations here but we gave a provided about 25 different genre categories and once you take 273 answers and split them up into 25 different game genre categories shooter tower defense city builder all these other things your sample sizes are now too small to really compare them to each other with any statistical significance having a friend on the team was not correlated sharing team members with other projects having a preference for face-to-face communication versus email or reliance on temp workers or contractors or the use of outsourced labor so it's interesting toward the end that external labor had no global correlation or the use of contractors or temp workers and what that probably just implies is that the real key here is not whether you use external Talent it's what external talent you work with

and how you build and manage that relationship and how carefully and diligently you work with an outsource team okay let's talk about some of the conclusions we can draw from this study and here's the part where I get up on my soapbox a little bit if I'm not there already number one let's start demanding evidence so the next time somebody tells you the waterfall is the devil and you have to use agile or scrum and your game is totally doomed if you don't use their particular production methodology ask them for evidence ask them to prove that it works our industry is full of untested assumptions and many practices persist because we don't question them and similarly if someone tells you the only way to make your game truly great is to work 80 hour weeks your game is going to be total crap if you all do that ask for evidence for that to ask for numbers ask for data conclusion

number two the real opportunity for improving our odds of better outcomes are mostly cultural back in 2012 bethesda game studios executive producer todd howard gave a keynote speech at dice and he showed this slide and he said the following and i'm going to quote him verbatim here because i love this quote this is one of the big rules that we have which is the plan that you have is not as important as your culture so you see a lot of game makers will say so here's the big schedule here's everything we're going to do you know if they're really trying they're going to run into problems and those problems are solved by the culture you have on your team I have my own way of putting it which is this I believe every game is a reflection of the team that made it if you want to make a better game you need to build a better team and a better culture on your team in other words your

game comes from your team and your team is a product of your culture so if you want to make a better game build a better culture so the real opportunities for improving our odds of better outcomes are mostly cultural leadership teamwork and cultural factors accounted for more than 85 percent of the difference between the best teams and the rest and before you resort to crunch have you tried debugging your culture have you tried debugging your development process and have you tried making sure the first 40 or 50 hours every week that you're actually working are actually being used efficiently why scale something that's broken number three there is an established science to building effective teams let's start using it our findings mirror those of management science exactly be aware of the limitations of your own experience and instincts recognize that our

industry is not a unique and special snowflake we are still humans working in organizations and management research is equally relevant to us number four if you want to get the most out of your culture you have to design it it's not going to happen by itself write or update your mission and value statement it matters it really does hire people who will embrace your values it's really not enough to say hey my philosophy is to hire great people and get out of the way I've heard a lot of people in the game industry say that I just hired great people and get out of the way and if you really have that attitude if that's really how you go into leadership and game development you're missing about 95 percent of your job because you can hire the best people in the world they're still going to run into a million problems over the course of development and if you are just letting those problems fester you're gonna run into big problems and your job as a leader is to proactively manage risk designing your culture intentionally

means it's your job to incentivize the behaviors that align with your values and disincentivize those that do not actively shape your team's internal discourse to promote healthy feedback and creative conflict in the creative process and minimize politics and interpersonal conflict and keep the millions of small problems and values misalignment from turning into big ones and that's the end of my talk are gamasutra articles are at that link if you're interested in reading through those some more if you have questions or feedback the game outcomes project has a twitter account which is game outcomes on Twitter those are the three books used in our study and again I definitely recommend the two on the Left Hackman and Lencioni we're going to do a new survey toward the end of this year and our plan is to keep evolving this every year or two and refining it and finding better and better questions we're also doing another project on leader interviews one of our team Eric Byron is

interviewing various leaders within the game industry so if you're interested in participating in that there is Eric's email and I strongly recommend emailing him and finally if you want to contact me there's my contact info at the bottom and we've got some time for questions if anybody has any yes serious if you check to see if other factors could account to explain crunch time in a linear regression yeah whether so the question was whether we check to see if other factors could explain crunch we looked at cross correlations of the quantity of crunch and the outcomes with pretty much every other factor that we could think of including you know project duration team size and there's a note in our article it does actually if you it's the fourth article in the series if you go to gamasutra there are

actually pretty strong correlations with especially things like lack of organization and a lack of respect for team members right it actually has very the quantity of crunch you people are far more likely to experience crunch on a project if there is a not a respectful relationship between the management and the team which is exactly what you would expect so next question sure how exactly did you define the different methodologies on the survey or did you just ask people if they were doing agile without any kind of framework to that yeah essentially we if you go to the Gamasutra articles that has a link to our methodology page and you can download the actual questions directly from that but what we did was we asked you know what production methodology a team was generally using along with a description of what that meant so are you using waterfall and here's a brief description of waterfall as a staged planning process versus a brief description of agile and so on so next question hi great talk enjoyed it had a question about the years of experience he used and went

yeah it was kind of like almost two three four up to eight plus yep and my you started off by mentioning you've been in the industry for over twenty years so I thought that eight plus I'd be interested in knowing how that continues beyond right what about ten and fifteen and I really wish I had more data there and certainly in the future we're going to refine those categories we'll probably ask you know we'll probably branch them into like average of ten fifteen or more and so on and we may also add a question asking the actual number of years and I again I'm sorry that we didn't do that but I in the next version of the study will do that for sure next question excellent talk Congrats in future surveys will you be able to ask prerelease so that a positive outcome doesn't influence their recollections right the question was in the future will we be able to ask teams about their culture pre-release so that the outcome does not bias them we would definitely like to do that we just have to make sure that we actually are able to match

people up properly and it's going to require the people and essentially fill out the survey twice or at least two different versions of the same survey the culture and then later on ask the outcome questions so I think that would be a great way to help reduce some of the possible bias yes for experience was that team experience or experiences individuals yes so they the question was was how we asked about the team experience and what we asked was the average years of experience for people on the team so we asked people to tell us do people on your team have an average of one to three years of experience four to six you know seven to eight and so on so there was nothing in there about their their length of time together working together right there was nothing in there about their length of time working together as a team and that's actually a very powerful factor because teams that have just been put together for the first time you know certainly all the management research they talked about at Wharton School was that that is it has a very very powerful effect so we should ask that in the future and we'll include that on the next version of the study yes so anyone

who flies you should a graph that essentially shows how you know fellow game developers helped you know one another outside of you know outside of work do you think a mentorship mentality of like Shigeru Miyamoto's like while he's you know developing the game you think that's also beneficial to a healthy game developing culture I'm sorry the question was I couldn't quite make out some of it you were asking about mentorship was that correct yes do you think it's yeah I think mentorship is absolutely you know extremely important and being able to learn on the job grow your skills continuing continually and it's something that I think in the game industry even the top teams tend to take an attitude of people should be taking that on themselves rather than studios and companies actually building development plans intentionally they're developers for their employees all right and I think that would help thank you very much yeah yes hi I was wondering if you could

expound a little bit more on communication you had a little point about how email versus face-to-face didn't have a high correlation to outcome I was wondering if you studied I may have missed this but like virtual teams or like grouping by discipline versus grouping by team stuff like that in the office yeah well so the question was about communication and what question we asked about that that was pretty much all we had as far as you know overall how the format in which the communication occurred we did not ask any questions about virtual teams but we did ask about let me go back here and oh wait actually it's closer to the end oh come on now sorry all right let's go to the end here yeah we did ask two separate questions about here we go reliance on temp workers and contractors in the use of outsourced labor we did not ask about virtual teams but we do want to get more into this in the future and find out are there actual correlations so thanks yes

next one did you notice any correlation between like methodologies and durations specifically as opposed to the aggregate yes so was there any correlation between methodologies and things like team size and duration and yes there were one of the things was we found that larger teams are somewhat more likely to use waterfall and smaller teams were somewhat more likely to use agile which kind of makes sense because if you're building some massive game that's like the next game and the Call of Duty franchise and you've got you know 200 developers on your team you've got a huge burn rate and you really really need to plan things out very very carefully you can't afford to be agile and say hey we've suddenly come up with this great idea for our game design let's just pivot this entire 200 person team it's a lot harder to fit it so you'll notice this one here the orange line is waterfall it sort of goes up toward the right indicating that larger teams are more likely a little more likely to use waterfall and the smaller

teams are more likely to use agile and scrum because they are small enough and versatile and agile enough to use you know those methodologies we also occasionally post do additional analysis of our findings find correlations between some of the different factors and not just with outcomes and if you follow us that game outcomes on Twitter we occasionally post them there and so we've also got some historical stuff on that if you if you scroll through our Twitter feed you'll you should find them so next question yeah did you ask about did you ask about either individual or team level of experience with any of the individual like did you ask about how experienced is your team with agile versus waterfall right yes are you in this in great great question so the question was did we ask how experienced is the team with agile or scrum or how long has the team worked together what's the team's overall level of experience we did not ask that my instincts tell me and I think what I learned about management science would tell me that that both of those would be huge factors so we will try to include those on the next version of our talk if at all

possible our survey if at all possible yes hi great talk is awesome and I apologize if this is a question that I believe you addressed in the talk Minh I didn't quite catch um in terms of how the outcomes were viewed I got that a lot of it was provided in terms of the team's perception of outcome um was there any information on actual outcome so money made or right by the game etc yes so that's a good point the outcome score is a lot of it was based on the team's perception of outcome and the question of how do we know the team wasn't just saying oh we did great when actually they did terrible or vice versa right so one of the things we did and this isn't necessarily a hundred percent scientific here but we had an optional question at the end you know asking what project people actually worked on and so based on that we were able to analyze you know we had the outcome question of what was the game's Metacritic score what was the overall critic critical reception then people would answer oh my Metacritic was between 70 and 75

between 76 and 80 between 81 85 and so on answered based on those categories so we were actually able to cross correlate those and say hey this you know this person worked on shadows of Mordor King Kong Country Returns whatever game and they answered that this was their critical reception let's actually look up the Metacritic score and even though there were about 20 of them we found that they were all answered honestly every single one of them the category that they had filled in for the Metacritic score was what the actual Metacritic score for that game was and again it's too bad that we only knew what 5% of the games were and that we can't tell you who answered what right because that's confidential but that was a nice reinforcement that people did seem to be answering these questions honestly thank you thank you yes our survey methodology for the questions that you pulled from Lund Sione and Hackman were those questions did they haven't established chromebox awful level for them or like an alpha level like that there were they were reliable

and valid questions or were they just questions from the book oh oh yeah so how did we design messages from this planet Lencioni mark yeah yeah essentially it was it was the whole game outcomes project team going through these models and just trying to find the questions that we thought correlated most closely with what the Hackman and Lencioni model represented okay so alright yeah there's no I don't think there are any official like surveys of the Hackman and once having models I didn't think so that's why I want to I think there is one for the Gallup model but I don't know I don't have access to the actual question saying so yep last question yep by the way if anybody I think we're running out of time but if anybody has any more questions ping us anytime at game outcomes on Twitter let me put up that link again whoops thank you thank you