The Bane of All Development Postmortems: Communication is Hard So Give Up Now

Thanks for being here. Um, I appreciate you all. I appreciate you wanting to develop and and uh improve your skills as producers. And I will I will do my best to uh pay some justice to this topic. Uh, communication's a huge huge huge topic and um it's it's important to me. It should be important to all of us. So, I was happy to to come and be able to participate, but I don't know if anybody could really do it full justice. So, uh we'll give we'll give it our best shot. Um, just a couple points of order here. Uh, one, please make sure that your phones are on vibrate. I'm sure you've already done that. Uh, two, uh we we take the reviews uh all the speaker reviews very seriously. So, um please fill that out at at the end of the talk um or you could take a guess what it'll be and do it now. Uh, either way, just please make sure that you that you turn those in. Um, and uh and thanks very much

uh to uh to you guys for being here. So, producer boot camp I boy, I remember um I used to come to producer talks here at GDC uh actually when it was down uh uh about uh what 40 miles south of here. Um, y- years ago when we would all get together in the Fairmont Ballroom and uh and be kind of naughty. It was a much smaller industry then and we had uh we had smaller teams and communication was a problem then and uh it's still a problem now, but sometimes we get it right and when we do great things happen. I'm head of production and executive producer at Gearbox Software. My name is Aaron Thibault. I also teach from time to time. I'm teaching this semester a capstone class at Baylor. I've been in education for a while, so I love uh talking about things that I get to learn more about. So, first on communication for game development is hard, right? We

we know or we're learning that game development is really really really hard. One of the hardest things about it is communication. Um Now, I like to learn I read books, I find material on communication so I can improve my skills in communication with the teams that I work with and the people that I interact with. I do that all the time and hopefully you're doing that all the time also. Um one of the books that has held up for a long time for me and he's written more material than this um on on things having to do with trust and and truths uh is this one, Seven Habits of Highly Effective People. Um in it, one of those habits that he points out is communication. Seek first to understand, then be understood. And he makes the point that our whole lives we're spending time speaking and writing and in our classes in school, we learn how to communicate outward. We're broadcasting. We learn a lot about broadcasting messages, things that we care about cuz we assume a lot of people care about what we care about.

So, we want to tell them all about it, but we don't spend a lot of time listening. And most of what we do as producers or what we should be doing as producers is listening and observing and gaining situational awareness. Really cluing in to what's happening with our projects and our teams and if you're on the development side with the publisher and vice versa if you're a publisher. We don't spend nearly enough time really synthesizing the information that's coming to us. So, we really have to develop our skills in listening. And it's a practice. So, since this is sort of an overview of communication, I'll just co- cover a couple of basic topics, things that you've probably heard many times in school, but it doesn't hurt to refresh our memory. Here's some things that are barriers to communication just generally, just broadly. These are These are broad principles, things that can impede good communication between people. So, physical barriers. Uh nowadays, it's not uncommon at all to work with

teams who are on the other side of the world. Um very often, we work in buildings where we may not see other people who are in the building with us for days, weeks, months. I was talking to one of our producers last night who said he got to meet one of our animators for the first time. He said, "How long have you been here?" He assumed it was maybe a month or two. "2 years." So, it's funny how long time can go by for that we don't interact with others who are actually working on the same project with us or another projects near us. Choice of the wrong medium. We do this all the time. What's one of the What's one of those mediums that we use all the time where we probably should have chosen some other medium? Hm? Email, for sure. That's an That's like low-hanging fruit, right? All the time we use email for things that we shouldn't be using email for. Semantic barriers are language. We use a word that somebody else takes an

exception to that that sort of stops their the flow of communication, their listening. Might stop you from talking. Cultural barriers. Some of these are pretty obvious. I won't go too deep into these. Just it's good to be aware of these things. Psychological barriers is one of the biggest ones, actually, because you can only ever control yourself, the communication that you're providing, and the listening that you're doing. You can never control anyone else, right? So, psychological and attitudinal barriers, while they might frustrate us, they might be things that we're aware we have to pay attention to. It's one of those things that makes communication hard because we can't force the other person to really hear our message. All we can do is a better job on our end of conveying it and being aware if they're getting it or not. And varying perceptions of reality, which are caused by levels of understanding and comprehension. So, everybody has a different view, a different filter that they use to see the world, and to see projects. And some

people might think game development's easy, or should be easy, and some people think it's hard, and it should be hard if you want to do a good job. You know, there's a a lot of different ideas of what game development is and what it's not. Nobody shares one People don't share one idea of that. So, it's important to know where the other person's coming from. And it's really easy for us just to assume it's the other person. We have to get through that. And it comes up all the time. And at Gearbox, with the people that we work with, our partners inside the company, we all you always have to kind of just pause for a second and think about is this is this quality issue something that this person, you know, did they do this on purpose? Was it a timing issue? Uh, am I saying something where they already have built up a wall to it, and I wasn't aware of that? What's going on there? And you must encounter these things all the time, I would guess, if you're making games.

So, the description of the talk is about communication being the bane of all postmortems. Well, I'm I'm really sad that Game Developer magazine's not still published in the way it used to be, although we still see great articles on Gamasutra. Um, but one of the articles that as producers we always enjoyed looking for were these postmortems. And the reason I say communication's the bane of all postmortems is cuz you just waited for it. You waited for that team, the person who's writing that article, that postmortem, to say communication was one of the top problems. Every time. And if they didn't, you called them a liar. Especially cuz you were probably having those problems. Well, in this particular issue, which was um at the beginning of '09, I think. Um Brandon Sheffield

uh wrote up just a a quick recap of some things that he'd run into and or he'd come across in postmortems. And it kind of sets the tone, I think, just that over and over again we come across these same issues. So, Penny Arcade. They simply failed to employ best practices, which are things that you agree in communication amongst each other are things that you should be doing so that you don't have to have more formal rules. They're they're guidelines for you to follow. And you set these in agreement with the team because these are the things that are going to keep your project from going off the rails. You know, they they knew better. They simply failed to employ best practices. How often does it happen? I know it happens pretty often. Um happens at Gearbox. I know from developer friends I talked to, it happens throughout the industry. We just forget. Um Harmonix.

On Guitar Hero. Well, they created an early schedule. They had a communication plan for how they would go about making the game. And then what happened? As things went along, they forgot to keep up-to-date with specification of what they were doing. So, if if you're aware of the PMBOK or the uh Project Management Institute's guide on how to do project management, uh PMI tends to avoid talking about Agile development. At least I don't see PMI talking a lot about Agile. They they use other language to talk about more traditional approaches Um to project management which still fit within an an agile structure. Um, one of those they talk about a rolling wave. And that's a common theme. And in communication as a producer, you have to be ready to catch the project as it moves along. The idea of this sort of rolling wave in development to stay on top of the wave and to continue to bring

I ideas and specifications back into the project so that they can be tasked properly and be broken down. If you're running an agile project, then you're breaking down epic stories into smaller stories. Those The smaller that you get in the story, the better spec, the better can be estimated. They just sort of forgot to do that. Max Hoberman, great developer, uh, developed multiplayer for Halo, uh, went off and made his own studio, Certain Affinity, was used to open pit environments, open communication, had a passion project, Age of Booty, was an original IP. Even in that environment, they forgot that their teams had to communicate with each other. They didn't have a formal tasking plan for how they would ensure that dependencies worked out between those teams. It ended up costing them a lot of time and money. And I won't I won't go on and on on on each of these for now.

What Brandon says, a lot of these issues have to do with scheduling and communication which in turn has to do with the quality of producers. And I'm really heartened to see you all here because I know you want to be quality producers. Um, not everyone can be here right now. It just shows your your interest though in in furthering your careers. And these have to be things that we remind ourselves of all the time to be quality producers. But we just run into these. Thanks, Brandon. Uh, in postmortems all the time. So, what are the kinds of things that we can do to make sure that we're learning from these mistakes? And how do we use communication to reinforce with the team that we have learned something and that we're going to get better. And that it won't happen again. Well, sprint retrospectives, how many of you here are employing an Agile methodology in your studio in your development environment? Many of you. How many of you are not?

Specifically not. Okay. So, for those of you who one or two, but for the most part for those of you who who didn't raise your hand, either you're not sure or you're looking to get into development. Um, you've heard maybe a lot about Agile and you've heard about other methodologies. Um, sprint retrospectives are a great way to make sure that you are actually absorbing these best practices and that you are culturally building up um, a a good way of talking to each other on your teams so that you avoid uh, wasting effort and having angry developers around you. And in those sprint retrospectives, you have to talk about things that you need to start doing, things that you need to stop doing, and things that you that are going pretty good, continue, observe, good so far. That's a really useful team discussion. Now, there's also a thing called an AAR. This comes from military training. We use it Some people call it a mid-mortem. It's very really similar to a mid-mortem. Okay. Um,

difference here though is where a sprint retrospective tends to be, at least in my experience, I'm sure many of you probably have different experiences, but my experience usually is a big discussion. And if you as a producer there, you maybe sort of guiding people through it if you've collected topics, you're guiding your way through those topics. Um, an AAR tends to be more of a coaching moment for you as a producer or someone who's senior in your organization to ensure that in fact learning really This is your your moment to say uh have you really got it. The person who's on your team when you're talking to them and you're you can be in a group environment. You can be in you can break up your your teams into smaller groups to really focus that conversation. But that after action review that's about performance in the team context. How was your performance as a developer? It's a tool for you to use. How was your performance as a developer? How was that with what we were trying to accomplish on the team? It's not coaching like performance

review coaching. It's not the kind of thing So one of the problems is we don't always coach people at the right moments. We wait until they're way off the deep end and they've sunk morale of everybody around them. And we waited until that moment to to coach them in good behaviors. Don't wait. Do this all the time. I drop into offices around the studio all the time to see how things are going, but also I've heard something about the possibility of a problem and I just want to go see what's up. Or I've heard something where someone's performance is starting to lag and I ask that person's lead to go follow up. Just spend more time with them. If you guys are familiar with the Hawthorne effect um there's a social science experiment years ago where um in uh time series studies they wanted to see if there was a way to have more productivity in in factories and they set up a a test in a factory and they had uh survey set up where

um the uh the scientists would walk around and ask questions. It was a survey of people who were working on the floor. They wanted to see if lighting conditions would change productivity uh in that in that factory. Well, productivity increased. And at first they believed that correlation was directly to these new conditions in the factory. And later they realized that it probably had a lot to do with in fact they discovered it did have a lot to do with just that interaction of people of those scientists walking around asking questions on the floor. The Hawthorne effect has to do with changing the outcome of an experiment simply by holding it and getting involved in the subjects of of that experiment. Um So we know that interaction with with people who are around us is going to improve their performance cuz they know that you care. What you have to do is then turn that into data and turn that into specification turn that into things that you're actually tracking to know the health of your project.

Maintaining best practices is hard. This is part of why communication is hard. Maintenance is a hard thing. You got to roll up your sleeves you have to go do it. Right? It's hard so a lot of times it just doesn't get done. But if you want to have a great postmortem and a great AAR if you want to be aware of what's happening in your project you have to develop situational awareness and keep it. You can't let it go cuz your project and your team changes all the time dynamics change your project's changing what's the status of it? Don't just go asking people for status updates you have to know and if you don't know you if you don't know you're not doing your job. Frequency has to be high and you have to be vigilant about it. Eons ago I was working on a project and the producer on that project realized that there weren't enough touch points with the team and wanted to improve process and wanted to do things differently from how this particular studio I was working at was doing things at the time.

So that person wanted to institute midmortems very reasonable. Told the leads that now we were going to have updates regularly and these were going to be built around essentially sprints. We didn't call them sprints. We knew that it was time to crunch a little bit. And we didn't want to use the word crunch. And we came up with a different word. I'm saying we generously. But a different word was used. And and that word was surge. So we're going to surge ahead. This This period of time we're going to surge ahead and then at the end of this surge we're going to get together and talk about how it went. And then we're going to improve. So at the end of the surge people have been crunching a bit. They had some goals that they'd started with. They got together. The whole team The leads had been tasked with gathering up information from their team members

about how things went. They got together. Big room like this. And the leads came up one by one. And they wrote on a board what their top items were. And they all agreed on what the number one thing was that they could fix for the next time. Well, guess what? The first one number one thing communication. Big letters on the board. Cool. We'll fix it. It went away. Another surge. Took goals from that first one. Went into the next one. Had that next AAR. Same thing. Communication. Big letters. Number one thing. At this point I was starting to worry a little bit. Just that you know I don't know how the team's thinking about this, but you know. Let it go. See how See how that works out. Um At that point I wasn't in the same role that I'm in now.

Although I did speak with that producer a bit about some concerns that I had. Third time. What do you suppose happened? Well, here's what. As the lead started coming up, that producer said, "But this time you can't say communication." So, the leads looked around and said, "We're good. Everything's great. Let's go back." That project shut down 2 weeks later. Communication's hard, and if you don't do something about it, you're going to fail. And this is how we felt. So, you can't ignore it. Here's a good best practice. By the way, I tend to like to bounce around a bit. So, for those of you who were in the talk I did last year,

thank you for coming back this year. Uh and for those who weren't, I'm just letting you know ahead of time, I like to move around a bit. So. Saying I don't know is a best practice. I love that. This is how I'm looking for when I look for new producers to work with. And I listen for how much experience people have when they say I don't know, cuz I I say I don't know all the time. You can't possibly This industry's moving so fast, and the things that we do are so complex. You have to say I don't know sometimes. Um I don't I don't know why anyone would try to fool us by saying that they do know, or by indicating they know more than they do. But this is also how how you gather up information to to improve and to schedule and to work with your team. Um so, look for that. You should value those simple words. Especially so we can avoid a death march. Do you guys recognize this graph? It's pretty well known. Okay. Well, uh this is considered to be at least Tufte, who whose opinion I I trust,

um says this is probably the best infographic ever made. Um by Charles Minard. Um, and this is an infographic about uh Napoleon's army's march on uh into Russia, into Moscow. And what you see is the line that starts out quite broad on top and gets progressively narrower to the right as it approaches Moscow is the number of troops still in Napoleon's campaign. And then that black line on the way back is the continued death march home. And the graphics that you see here um, are about distance. They're about temperature. You can see temperature charted in the bottom. Um, they're about uh, latitude and longitude. Um, and they tell a pretty stark story. Well, if Napoleon knew in advance

that this is what the outcome would be, would he have still engaged his troops? I don't know. If he thought he'd won, maybe. Hard to say. Um, th- this cost him his his future um, in France um, to to become the great emperor that he was looking to be. Um, and as an infographic, it's very interesting. I want to make the point that as producers, we use graphs all the time to communicate things. We use Gantt charts and burn downs and build fancy things in Excel to show the relationship. You know, we try to break things down to make them as simple as we can be. Um, I think we probably still haven't gotten quite as elegant uh, as what we see here, but I'd encourage you to think about breaking away from uh, a a a rigid practice of how you display information, especially quantitative

information to the teams that you work with. They're looking for something that distills the essence of where the project is. That's all we really care about at any time. Is where is the project, where's it going to be, especially at the executive level, is this project going to be worth more than it cost by whatever number the P&L says it should be. Um you can convey that in ways that show the attitude of your project and the quality of the game that you're making. So just think about ways that you can visualize that data. Uh that will really come across to the uh audience you have in a way that that conveys the heart of what makes your game special. Discipline, organization, and maturity are not things that we always found in the game industry. But I think we're doing a lot better job of that now broadly as um game developers have gotten really experienced and mature over the years. And we should value the

wisdom of those that have been around, but discipline is hard. Organization can be hard, especially for it to not feel overly bureaucratic in an environment that's looking for creativity. And maturity, you know, I I I don't know much to say about maturity other than we value it. Um In this post-mortem, um this this was a good game that came out um last year, Lords of the Fallen. Um this team looks back and thinks, we're not really sure how we made that game um given the problems we had going into it. Um And they make the point that something they were really looking for when they realized they needed it was a person. They're looking for a person to come in and ask the tough questions and make the production decisions that needed to be made because they as a team had very difficult time pulling things out of the game,

cutting things out. And this really caught my eye. I don't know what happens on a project to get to the point where the publisher is now coming in uh heavily in creative. They talk about the publisher at this point is uh actually doing level design, is doing map design, and mission design. Um something's happened in that in crazy, chaotic environment um where those tough questions aren't being asked, where the publisher now is having a creative role certainly beyond whatever they thought they were signing up to in the beginning, and not enough communication's been happening. So, somebody needed to come in and create some structure. And every studio environment, you need structure to give people um the trust that things are going uh the way they should to be making a a good game. So, I say, basically, they'd not thought about establishing those guidelines for communication. And once they got the right person in

the environment, that stopped crazy communication happening between the publisher and the developer and employing channel people. Now, uh every good um organization, it's clear who has responsibility and authority for what, right? So, you should know who in your team is talking with the person on on the other side if you're working on that developer-publisher relationship. If you're self-funded, cool. At least then, though, you should be understanding more about who the customer is, uh where you're going to market, uh if you're having that sort of direct interaction with um with with customers. Uh and in that sense, I mean, customers who are who are buying the game. Um you have to really understand the structure, though, and if you don't, go make sure that you that you do. Start asking hard questions about who's responsible for which parts of communication with finance, with publishing, with marketing.

But information's not going to move by itself. You have to do things. The problem that I've seen time and time again is simply people not getting out of their chairs. So you can lead by example to get up, walk around as I said, I do that all the time. Just by you getting up and walking around you can show you don't have to be stuck in your chair all day. But also some people need some prodding. You want to do that if the information that they have is vital to others. For example, there may be a rendering programmer, a graphics programmer, who's working on things that are really important to the technical artists and artists and and you know, level designers and your lighting artists on the team. And for some reason that communication has hasn't happened in a way where they know what to expect at their next big milestone. Something has to happen and you have to be the one to do it to cause that communication to happen, for the specifications to be written down. But there are a lot of kinds of you know, we different people are different.

Um and they get into this rut of not sharing information. So you have to get it from people into a format that's usable. And it's hard. But reminding people what they're doing is also really helps. That encourages them to get up and walk. It encourages the kind of specificity that you need to move forward cuz they get passionate, they feel good about what they're doing and then you give them a structure in which to work. And you can funnel that passion into something. Those who are checked out are not actually going to provide you the kind of data that you need, the kind of specificity for others to understand what they should be doing. Or how to use this tool that's being developed or this tech that's being developed. Or when does level art come in to what your mappers are doing? Those who feel good about what they're doing are going to be more vocal in a positive way. You guys play Shadows of Mordor? It's a good game.

Well, in their postmortem, it's funny the thing everybody talks about with that game is the Nemesis system. And they talk about how they didn't do a very good job of communicating the promise of it internally or externally for a long time. And it created them some issues as far as spending extra cycles on things that they could have cut out much sooner than they did. And they needed somebody in that team, and you as a producer can do this, to evangelize a system like that, the thing that's going to make your game special, whatever it is, to promote that all around. When I was at Origin, we would do that. We would make posters of things, and I saw other teams inside of EA do it to really promote what they were doing inside of the the system to really gain buzz in favor for for their projects. And that's a good thing to let people know what is special about what you're

doing. And then to connect the people who were working on that special thing with others. It fuels the passion for making it. This is a postmortem from 1999. It's one of the ones that I used a lot when I was teaching at the Guild Hall still. Um So, Ken Birdwell here talks about the Cabal process, how's that how that evolved on the first Half-Life. Um and what it evolved in response to. And now we call you know, we this is pretty common now, strike teams, tiger teams, some people just call them Cabals. We just We like to group people around um features, um things where you can have efficient communication. But there's a couple interesting points here that I still like to look to. So, one of them is, during that process, they realized that even though level designers knew about some feature that was made in this In this case, he's talking about a beam tech that, um, coded worked on.

Nobody was using it. They didn't know what it was for. Hadn't been communicated to them what was special about it. All those points that were just made, they weren't doing that. And their solution was, well, the beam tech's made. It'd be cool if it was used. So, the solution is tell them they need to integrate it, train them on how to do it, and then follow through to make sure they really do it. Now, smart and creative people don't usually like to be told they have to do a thing unless they think, especially if you're a level designer in this kind of environment. You want to know, I mean, you basically own that level, right? As that old D. So, you want to be cool enough where you want to use it, not not that you're told you have to. But sometimes as producers, we have to ride that line. So, you have to make sure you know and that everybody else knows who the customer and the provider of everything that's being done are. And in your tasks, if you use Jira, let's say, or Hansoft, or whatever

you're using, make sure that it's really clear who's responsible, who who had the vision for a thing, who should the implementer go to for more information. And then, if you're in a system like ours, a lot of times what we do is we essentially hand off ownership to a person who says they really get the intent. But for questions, they know how to follow up. Um, in the past, on Borderlands 1, for example, we saw this. Some feature requests would go in. It would sit in our task management system for a while. Wouldn't get a lot of love. Eventually, it get picked up and then it would implemented, and essentially a vacuum. Until we realized, in a playable build, maybe a month later, what had gone wrong, and seen if we could catch up to fix it. Well, that's not a best practice, so hope that you're doing something better than that. Um and one of the things that we do now is we always make sure we know who the customer is for a feature. Now, in this case, I mean customer on the team. Who's the person on the team that needs

a thing? And out of that, also, we developed we we built our our tools programming team. Um who always know who their customers are. Um around around the studio. We see We don't see this as often anymore. But man, it's always frustrating. You You must have been in a meeting at some point where you realized the thing that everybody's arguing about is something nobody really knows about. And then you start making calls, hopefully, or you do something, or you postpone that meeting. But if it's a decision that has to be made right then, you shouldn't probably have gotten to that point. Um or you should be more careful about how your meetings are structured. So that prior discussion happens before you get to that point. So think about actually a um a plan for your meetings that stages them in a way that the right information gets to the point where that eventual

decision-making meeting happens. Or allow decisions to be made in those meetings by the people who are actually doing the work. You have to decide, but you don't want to get to this point. We still see that. But not as often, like I said. So, again, communication's hard. Some of the harder things that we have to do in production are these three. Hiring. Hiring is one of the most difficult and important things that we can do and get it right, and it costs us so much when we get it wrong. And it costs us so much just to do it. Um Right now we're going through thousands of resumes in in the studio, um where I am, at Gearbox, and um it's a huge process. Um and all of you know this, too. If you've hired anybody, you know how unless it was your buddy, you know, which happened, then at least you have trust for your buddy. But, um it's it's it's hard when you get it wrong.

Assessing, assessing a situation when you have two different points of view is enormously difficult. Um there are tactics that you can employ um when talking to different with pe- people with different points of view, and sometimes one of those people isn't wrong. Sometimes they're both right. Or they could both be wrong. But, you have to be able to assess that for yourself. And if you don't have the technical or domain expertise, and it's on a technical issue, and you're trying to solve it, either you're making a mistake, or you just need to bring others into that discussion to help you. Um or you're just really comfortable kind of playing it from the hip. And maybe you'll get it right. Maybe it doesn't matter. Maybe you just need to make a decision so you can move on. Which is sometimes the best answer. Sometimes you just need to make a decision so that you can move forward. But, hopefully you're being diligent about that. Because if you make a technical decision on a topic you don't really know about, but maybe you think you do, and you're kind of fudging it,

the people that you're talking to know you don't know it. So, watch out. And resolving disputes. Similar topic, but in this one, a little more aggressive, right? Now, on on the topic of hiring, um have you heard of behavioral descriptive questioning before? Few A few of us. Couple. It's something that's really good for you to understand. It's It's really simple. There's no magic to it. It's just asking people questions that gets to the specifics of what they actually did. New hiring managers or people who are involved in hiring, um sometimes just make assumptions. Maybe not so much that the person is, you know, always has actually done the things that they're talking about, but more maybe they're maybe the person that you're talking to that that candidate um really means well. Maybe they did things like what they're saying. Um so, we'll kind of give him a pass.

And I don't do this really anymore. Um especially after some training in specific methods of questioning that get at what that person really did. And I highly recommend that you follow up and learn more about um how behavioral descriptive questions are structured and more so how to follow up once you get an answer. Um you've probably been asked questions like these or that told me about a time just like you see here. Give me an example of those kinds of things. But you again, if you don't have the domain expertise in an area where you need that domain expertise, you need to take another step back uh and and bring in a bit of help. Um Behavioral descriptives really help with this. Now, since this is a communication talk, I I think I'd better make sure I at least cover this. Um you need to know what questions are

legal and what questions are not legal. Um make sure you don't ask illegal questions. It's going to be really bad for you and your organization if you do. And in line with that, you should have a formal hiring process. It's going to help the candidate to know that you have a formal hiring process. It'll make you look more professional. If you're at a smaller studio now or if you're in a larger organization, you should go around and ask questions of other hiring managers of what their approaches have been. If you're not sharing that kind of information already, you should start, especially because you're you're all working on the same projects, or you might be at some point. You want to know that the people that they hired under them are going to fit into the team structure and be as good as the people that you hire. Um so, hopefully you've kind of congealed all of you in your leadership structures, or you will be in terms of how you actually hire people. At Gearbox, I I like our hiring process quite a lot, and in fact, we've done that where our code team built a formal hiring process

that the other teams, once they saw it, um loved it so much they began bringing in. And I can't get into the specifics of all what that is too much other than to say um it's it's kind of industry standard now, and it goes back to behavioral descriptive um questioning. Uh we really want to know um both what you did in the past experience, and we're going to throw things at you that give us some clue of how you will perform in the future based on what you say about what you did in the past, not based on guesses. Do you not ask about personal identifiers, affiliations, religion, disability? I'm just going to say it, right? Uh and get professionally HR training if you can. You should. Uh from that notion of questioning and the way that questions are structured, I'll say this. This is a um kind of my strategy, I guess, for how I

find out things when I'm talking to people. And also, it's a it's a tool that I use to motivate developers to focus in on what's important and to and for them to convey back to me they are in fact actually focused, or they've actually done the diligence of the thing that they need to do to do. May be difficult to to read so I'll I'll read some of these. Um Do we know how performance is impacted by these changes? That could be about a lot of different things, right? Um But in asking the question, you find out very quickly is cross-team collaboration discussion happening or not? And have we done the instrumentation that we need to do? And are we aware of what the deadlines are, what the build requirements are of the release that is coming up for evaluation? Is somebody about to do something that's going to tank this build that's going to be evaluated for an important milestone without even realizing that that change they made is going to do that. Well, you need to know.

Do we know how, you know, for obviously for say animation or something that maybe's being built in movers we use Unreal? Um do we know how it's going to be rigged? Have we considered the impact of co-op? That's a big one. That was huge for us uh on Borderlands 1 as we decided co-op was going to be an important thing. We were all we continue to be surprised actually by some of the things that impact co-op in ways that we hadn't considered. Um is there another way to accomplish the same thing? How will we know if it's good? These are all focused on the on the actual issue in development. Not on the person. All right, you want to be really careful especially cuz you're now being seen as someone in a position of authority, you never want to be seen as a person who's attacking the developers. They're just doing their job. Right? Um Fortunately, I haven't had too many managers who've made me feel like I was um the focus of broader development uh

issues. But this does happen. I know and I know it still happens. You guys remember Team Bondi? Um so, in summary on this, um questions are powerful. Uh you have to practice listening, practice getting conversations moving. You have to do that. You have to practice it. You can use humor as a tool. You can use different things as tools, not sarcasm, but humor. Well, I use sarcasm a lot, but only with people who I'm friends with. All right. You have to really know who who the people are that you're talking to. Um stay on topic because you'll be seen as more professional. And then you can move on from whatever that topic is from there. But make sure you get to the heart of what the thing was that you came in to talk to people about or that you're in the meeting about. And if you can end the meeting early, do it. Um speak with intent. Know your audience. You have to know what their filters are.

December 2008. This was another kind of postmortem recap. So, in this one, on Final Fantasy XII, I thought this was pretty interesting. Um here, uh Toku talks about Tokusohn talks about um what what happened with the team given the freedom to make changes at at various stages of development. So, they they they're expressing that they had some freedom to make changes at some stages of development. Um but that freedom led to miscommunication, confusion, disorder. I thought this was sort of an odd thing that freedom would lead to that. Um unless in the freedom that they had, they didn't communicate any of those changes that were being made. They didn't They failed to set the expectation. That's very dangerous. We have to be constantly setting expectations with people around us, don't we?

Um and then it said right here, "How work was to be distributed was also often ambiguous, which contributed to the problem. So, that really speaks to the problem of there not being a producer. And of course, I'm you know, I'm basing this on what I've read in this postmortem. That's best we can do sometimes. At least when we're reading these, we're making guesses, right? So, that being ambiguous means that there wasn't a producer making sure that as things were being defined as goals for the project, those turned into specifications. They didn't stay on top of that rolling wave as they moved through in development. Things just happened. And um the I mean, it's an older article. Things aren't exactly the same. You know, cultural things happen, be aware of them. I think that's a good summary. Recently in a Gamasutra actually has just wrapped up. I think they're done with the series now is the games outcomes project. Has anybody been tracking this? It's a great project. I love this project. Um

Paul Tzanetopoulos who I respect a lot is a great AI programmer and designer. Um talks here about um communication as one of the few things they discovered had real correlation uh in the questions that were used in the in the um the structure of this study um used a few um uh approaches um theories on why why things fail with teams, why things succeed. And here they realized that uh this one uh team members would often work for weeks at a time without receiving feedback from project leads or managers. Led directly to those projects which were assessed as being some of the failure projects or lower in the success scale. Um And we we've seen that. Um people who go for a long time without feedback

first of all just like we talked before with the Hawthorne effect, right? Like if you're not getting interaction and feedback, you don't have the context for what you're doing, you don't know where you stand in in the work that you're doing, you start to feel a little bad maybe about it or you don't feel as good you you're just not as connected. And we hear that all the time around every studio that people don't always feel as connected as they used to with smaller teams. And that will impact the quality of their their work. And also, now you're starting to fail expectations. You can build up an expectation gap with those people who are the product owners, the EPs, whoever's going to be ultimately deciding if the quality of what's been built is good enough or not and release releasable or not. Um you've built up debt. Also, since this is a topic of communication, um thought I should make sure to point out cuz we were talking about humor and

sarcasm a second ago. You have the responsibility of making sure that your workplace is a safe environment and that feels good for people. You have that responsibility. And you have the authority to make it so. I don't care who in your environment is creating issues with co-workers, you have a responsibility to do something about it. So, first, please no sarcasm that puts down co-workers. That's not acceptable. Okay? Um no condescension, no backhanded task comments. You really have to watch out for this one. Um historically in some parts of the industry, there was always a divide. Now, not so much now, especially not in our studio, I don't think, but there was a divide where um those who were considering themselves developers uh saw there being sort of a gulf

between themselves and those who who were in QA. And now we have a lot of great producers and executives and people who came up through QA. And we really appreciate the close collaboration uh between all the departments. And now I just see QA is another one of those development departments. Um, but occasionally you see really shitty comments from people in development to people who are reporting bugs. You have to watch for that. Don't let that happen. It will make your product worse. Um, condescension as well. Not that long ago, it was called to my attention that um someone who is in a call it a technical creative discipline um was making comments in tasks about a tool that they were developing. Which by the way, it's a tool. It implies that there are customers on the creative side who are going to be using that tool. They were sort of calling out some of the comments about why people thought that tool or that appro the approach that person was using wasn't the best.

And then I saw this kind of back and forth very thinly veiled passive-aggressive back and forth happen um that we had to put the kibosh on. Um, and we and we did. And then it's funny, once we talked to the people involved, you know, whether or not they really knew what they were doing, they kind of expressed they didn't. They were just kind of trying to get their point across. And you as a producer have to be the adult there. Um, no name-calling. And at the same time though, if you're hearing those kind of things, "So-and-so is stupid." Now that's different from this thing that we're doing is stupid, right? Uh, don't tolerate low performance cuz if you let people stay around your environment who are low performers, that's going to tank the performance and the morale of everybody else, right? So if you have a low performer, you have to get rid of them. Don't let them just sit there in the environment get kicked around a lot. They have to go. Uh you can't have sacred cows there. Um and you have to get rid of bad

processes. So, if somebody says this process is stupid, well, talk it through. It's possible the process is stupid. Um but it it at the same time, unless you've really understood where that person's coming from, you can't assess. Just same as what we were saying before. If you don't have the technical expertise and domain, you can't really know if you don't get more help. So, going back to communication's hard. I was talking to a project producer a few years ago, uh who had a kind of a monumental set of tasks on top of them. They they had a really hard job to do. Um b- based on the nature of what they were trying to ship. Uh and then they had more and more things heaped on top of them. Um and so, that person kind of delegated a bit, a very small amount, um some production responsibility of some of those other things. And that producer came to me

and just completely befuddled. Totally befuddled. Said this, "They had the same information as me. How could they possibly have come to a different conclusion? Something's wrong with that person. You should fire them." Af- after sitting down um with the person in question and this producer, um realized all the decisions that had been made were great decisions for these things that had been delegated. They were actually great decisions. They they made the product better. But, there had been a failure to communicate and set expectations as changes happened. Kind of like staying on top of that rolling wave. And so, because the end result of those decisions was so different from what the producer thought was going to happen in the beginning, all they could see what the result was and immediately see it as all bad. And it took a long time actually. And in this case, I mean every hour is expensive. It took a couple of days to work it out and see that actually it was okay.

Um but that communication goes both ways. It's It's why it's important for you to communicate up well to your leads and set those expectations and do the same with the with the team that you're working with, your peers, collaborators and and in your teams. You have to be constantly setting those expectations or people will see a thing that happens later and react to it very differently than if they understood the expectation up front, right? Communication is hard. Um some cats are going to sneak by you. It's impossible to get them all. Um you could be patient. You could learn about people more. You can find out that you can do things to bring them to you. Even the most belligerent people up front um getting to know you, understanding not just your intent cuz everybody can have good intent. But if they see that you're trying hard and that you're learning and you're learning about what they do and that you appreciate and respect them and

you don't allow yourself to fall into bad habits. If you If you're really setting the standard, people will flock to you. They'll talk about you like well God, I was going to say the catch me out but can't can't help it. Free association. Um they they will they will always want to work with you, okay? Let them come to you. Perfect communication, guys, is not possible. It's why in the description for the talk I said, "It's hard, might as well give up now." Okay? Um and I'm giving it some hyperbole, right? It's It isn't possible to have perfect communication because again, you only control what you do. All right. So, if you want perfect If you're always going to be frustrated because things you say are going to be mistaken, all you can do is improve your side of it. Um you have to practice your communication skills, the way that you break down

tasks, what you learn, the domains of the people around you, the wave of what's happening in technology around you. This industry's changing so fast. You have to stay on top of it. And I want to see you all as you succeed or not, whatever's happening on your projects. Keep learning. Help us as a community learn. Share what you've learned. And write new postmortems. And that's it. Thank you. Thank you. So, I think we got a little bit of Q&A time if anybody wants to ask any questions or else uh I know we have a great panel coming up pretty soon. Yeah. Anybody have any questions? Okay. Uh so, you were mentioning before about visualizations. Yes. Um and I was

wondering if you could give give examples of uh some of the visualizations that you've used very effectively in the past. Thank you. Um So, some some visualizations I think I've used effectively. Some of them have been things that um were in fact kind of um time series uh between basically showing dependencies between departments. I don't have them to show you here. Maybe I could follow up with a few that I'd show, but I can describe a couple. Um One is I was working on a project that was uh new IP. It was seeking to innovate and it had an uncertain budget. It was a large budget, but it was uncertain. And between ourselves and the publisher, we wanted to understand um the uh kind of the difference between our perceived value of what we'd built and um uh and and the true value of where it

was. Um so I I built a graph that showed the value of other projects um that had just shipped, that had some of those innovations. And um the entropy of uh ba- basically in people's minds how fresh would those ideas still be when they came together as one. And how long would it take us actually to finish depending on how many of those innovations we finished. Um in this case, we were doing some very heavy physics stuff. Um we were doing some some brand new um animation tech. Um we were doing destruction. This is all long before uh this generation of platforms launched. And it was um going to be a a last-gen project. So, basically sketching all those things out and also showing what the gap was to when we thought the next-gen platforms would be. Basically all the costs involved and um what the market expectation would be and what it would cost to get us there versus

basically cutting those things and shipping what still would have been a good game um without all those innovations. And doing it in a way that didn't take those people who'd been working on them um uh and not pay respect to the effort that they'd put in and their feelings about it. Um so, in that showing not just the work that they'd done, say a couple programmers and a designer or animation tech and um and a few animators and a programmer, but all the other people in that dependency chain who also would have to work on it. So, that if we decided to cut that feature, that all the people who'd been working on it really understood what the larger risk was and cost was as opposed to just what they'd done. And also, how we could um Well, that's basically what the the graph focused on. From there, though, it was used in communication to talk about how we could recycle those things for future projects. Um anytime you're going to deliver some you know, something that could be bad news, I want to see if we can refocus that to um show how something will be

better in the future. Um A few other a few other things uh would be um I had a great uh AP on a project who helped me um with if you guys have tried to use Jira and Project together, you know how much of a bear that is. But, um helped me with a near real-time um flow of data between Jira and Project, so that we could visualize the impact of tasks that were being created um on a regular basis. Once we had to sequence those out into our project plan, um and we could visualize at a much more micro level than we typically would in Project. Actually, in that Gantt view, which can be helpful because it gives you um uh it gives you the brackets within which things are going to happen. It It can It can be used as a helpful tool because it's a bit of information overload. It's like there's all this stuff suddenly this work that's suddenly been been created. Could we please just finish the work that we already committed to you because we know that those bars are really going to grow for

the polish of every one of those things. It was just a It was a handy tool. Um Those are a couple. Thank you very much. Sure. Any other questions? No time. Okay. Thanks, guys.