Better Development Through Science: How Aliens, Odysseus, and Toyota Can Help Improve Production

how's everybody doing you guys just gotta survive like one more round of GDC hangovers and you're home free I'm definitely looking forward to that housekeeping you guys gonna get the emails you know how that works so just pay attention for that please if you got cell phone silence um if you don't if you're not using your laptop to take notes please close it up and and let's get to it first of all I just like to say it's a huge honor be presenting here GDC 2017 it's a huge honor that all of you came to see my presentation I know there's a lot of good stuff going on I know there's a lot of important meetings so I take it as a personal honor I'd specifically like to thank Dave Reynard from the GDC Board for his feedback on this presentation too mom's working on it this presentation is about improving efficiency eliminating waste to get to the meat of the presentation I got to give you guys a bit of a crash course in what is known as operational science and that means there's gonna be some math so

if your math phobic like I am don't worry about it don't freak out see some formulas some maths and equations and we're gonna be moving a little fast to start so just hang in there at the end of the presentation I'm gonna give you a link to download the annotated slides so don't worry about keeping up with every note or you know getting every equation exactly right if you can follow the basic concepts you're good and then you can download the slides today and review them at your leisure fair enough all right let's get to it so I want to start off with a story because they say to always start these presentations with stories and this story takes place 350,000 years ago when our genetic ancestors the homonyms made a discovery fire and over the hundreds and thousands of years that discovery through iteration and repetition became something else it became a process and really all human activities fall on this sort of spectrum of on one extreme you have things that are processed you have things like boiling water there's no mystery there you apply energy to water and eventually it will boil at the other

end of the spectrum you have discovery you have the Large Hadron Collider the cutting edge of physics the unknown the unknowable but there's a few things to recognize about the spectrum first of all there are very few extremes most activities are process and discovery there are very few things that are pure process and even fewer things that are pure 100 percent discovery for example cooking lots of processor on cooking boiling water basic combinations of ingredients but there's also a lot of room for experimentation and discovery second thing to recognize about this spectrum is that all activities start towards the discovery end and move towards process over time so the first time you made your own pasta sauce probably a lot of discovery a lot of learning by the thousandth time a little more wrote a little more routine and the final thing to recognize about the spectrum is that our business is discovery we spend a lot of time on that end of the spectrum and to underscore that I would like to point out that the famous phrase find the fun has the word fine [ __ ] in it and as

the bard would say there's the rub because discovery by its very nature brings with it the unknown and the unknown brings with it risk or as operation scientists like to call it variance and that variance is the source of so much of our pain as we try to manage projects over long term over months and years it makes predictions hard it makes management hard so what we do do we just abandon discovery and just make the same game over and over again to make it more routine or it's not that's not fun but we also don't have to be at the mercy of variants if we can acknowledge two effects number one variants come pounds over sequential activities so if every one of these activities takes between one and five days the total time to get through all five is between five and twenty five days the second thing that recognizes that nothing we do is pure discovery even the most avant-garde experimental in the weeds gameplay design is still underscored by some elements of process the process by which

we design features by which we code them by which we compiled them into builds by which we upload them by which we QA them and if and to underscore that point I would point out that the famous phrase creative process has the word processing it and if that we're true we wouldn't need all this [ __ ] so in as much as the activities we do our process our known our understandable our our you know understood by us we should seek to make those processes efficient predictable or consistent and whenever possible automated both so we can minimize our overall level of variance and so we can maximize our ability to absorb variance and that's what I'm gonna be talking about today so a little roadmap for the talk this talk is gonna start micro and I go macro you're the first people to finally laugh at that [ __ ] joke alright so start with a bio of me this about micro as it

gets they're gonna move to some fundamentals of operations science and process flows they're gonna move to a little more practical example something called a capacity chart and the second half of the presentation is the real meat of it when we get into what's called lean production and how you can apply it to games then we'll have some closing thoughts and if there's time some question and answers so Who am I I started the industry about 10 years ago as a humble production intern at Wild load games in Chicago Illinois work my way up to a senior producer studio lead position on games like guilty party Avengers initiative and helping our sisters studio out avalanche with Disney infinity and over that time I really developed a firm belief that the human cost of making games in terms of layoffs and crunch and studio closures and uncertainty and stress related health issues was a little more than I could bear and I decided the best way for me to be able to positively impact that would be to learn as much about business and management as I could so in addition to working full-time and raising kids I went to night school to get my MBA from Northwestern University I'm currently working as a consultant nobley with my friends at rag tag studio while they work to finish raise the dead I'm also

trying to get my own game development company agency principle off the ground and I write regularly on my blog breaking the wheel so if you enjoy this presentation there's a nine part series called game planning with science you should really check out shameless plug another thing about me my favorite movie of all time is aliens yes this is cliche I know I know it's cliche but it's the truth you sent me to a desert island I'm taking that one movie if that ceiling will be again and I'm guessing games industry I'm guessing all of you've probably seen aliens so you may recall towards the end Ripley has to go to the hive to rescue Newt the atmosphere processors because exploding is she has to take a detour and she wind up face-to-face with the alien queen and the queen has this gross slimy bulbous sac attached to her that she uses to lay facehugger eggs on the floor after years much of a [ __ ] nerd as I am you may have wondered at some point how many eggs does she have in that thing so let's find out let's imagine that on average the Queen lays seven facehugger eggs a day on the high floor let's also imagine that it takes an average of five days for any single egg digesting the only way the Queen's can

sustain a seven egg per day throughput with a five day per egg turnaround time is if there are seven eggs in each day of gestation therefore the Queen has five times seven equals 35 eggs in the sack or it's put in another way her average egg inventory AI is equal to her average egg throughput bar multiplied by the time to produce a single facehugger egg T or more simply I equals RT an operation science this is known as littles law it is the most fundamental equation of operation science and it works for all three permutations I equals RT R equals AI over T DT equals AI over R which is a verbose way of saying if you know two of those values you can calculate the third so I'm gonna belabor this point for just a second because if you can hang with littles law of the rest of this presentation is gonna be a lot easier to follow so for any ongoing process if it's assembling cars or manufacturing cans of soup or perhaps creating video game assets the amount of things currently being processed to the

inventory is equal to the rate at which things come out of the process the throughput multiplied by the time to create a single team that's a single thing that the flow time now how do you go about calculating those values up so inventory equals throughput times flow time you're going to see this equation contextually throughout the presentation so it's it's gonna be our companion today so how do you determine those values let's imagine that inside the Queen's egg sac is a network of activities all these activities are necessary to produce a single facehugger egg they're all required now how do you make sense of this thing if brittish's it's not all clean it's not everything lines up well it's actually not that hard first of all let's recognize that one single path through the pipeline is longer than any other this is known as the critical path I'm sure you've heard that term in different contexts this is the operation science definition it is the longest single path from input to output the activities are known as critical activities and the cumulative flow time of those critical activities dictates your flow time for one batch you cannot get a batch done in less time than it takes to run through every

critical activity and that flow time is denoted as T let's also recognize that one single activity is longer than any other this is the bottleneck again I'm sure you've heard that term in various context this is the operation science definition it is the single longest activity irrespective of critical path it can be on the critical path that doesn't have to be the throughput of that bottleneck dictates the throughput of your process in other words you cannot get things out of the process at a rate faster than the bottleneck pull out and that throughput is denoted as R so to summarize your critical path determines your flow time your bottleneck determines your throughput now let's move to perhaps a more relevant example here's a hypothetical flowchart forgetting the character in the game that covers anything you might want to do concepts are casting vo mocap everything now this looks like SpaghettiOs right it looks really complicated but operation science makes it a lot easier to figure it out first of all despite all the mess we have a very clean critical path it's 29 days long so that means our average expected

time to complete a character from scratch is 29 days so there's our flow time and here is our bottleneck it takes nine days so we should expect to get a character done out of this pipeline once every nine days or 1/9 of a character per day on average there's our throughput so with very little effort we have just estimated the average time to complete a sketch a character from scratch the expected time before the first character is in-game and the average rate at which subsequent characters will be in the game or a visual form something like this now there is a caveat there always is if you add up all your critical activities like this just in sequence to get your flow time what you actually have is what's known as your theoretical flow time it's your flow time if there is zero downtime so the the unit may be if it's a character model never has to wait for anything it just moves through in one continuous flow in reality that's not going to be the case if you want to get at your actual flow time you need to account for that wait time theoretical flow time plus wait time you

your actual flow time so example those cues handoffs bathroom breaks anything that would stop an asset moving through the pipeline now if you can calculate estimate your theoretical flow time and estimate your actual flow time you can calculate something called a flow time efficiency this is the ratio between the two and what it is is an estimate of how much time your assets been waiting around not being processed not generating value so if you have a flow time efficient so you have 78% your assets spent about 22% of their time waiting around for somebody to work on them your flow time efficiency is always going to be less than one if it's ever greater than one year three article flow time is wrong it means you high balled it you thought you think your activities take way longer than they take so you need to go back and recalculate those that being said the closer you get to one the more efficient your process is so what if you don't know the average time to complete any activity or the entire sequence well littles law is - that comes to the rescue if you know about the average rate things come out of that sequence and you know how many things are currently somewhere in the sequence being processed you can back into your actual flow time your flow time so it's going

to be the inventory divided by the throughput and that would be your actual flow time not your theoretical the same thing applies to individual activities or chains of activities littles law applies to any level of granularity so if you're trying to figure out what an activity how long an activity really takes figure out how much how many units come out of it at what rate figure out how many things are currently being processed and you can back into your actual flow time again so now we've established that fundamental bit we can move on to something a little more practical so you know your throughput and your flow time you can use something called the capacity chart to plan resource assignments so you use a simplified version of that same pipeline I just cut off some of the extraneous stuff for just clarity now what you do is you go into Excel you create a very simple seven column chart at the end of the slides I'll give you a link or you can download what I've made for you but it's very easy to make so let's cut some column chart excel in the first column you list every activity that's necessary to complete a single unit so for this character model we need concept art high poly low poly pass rigging a few different sets of animation and a keyway passed the second column you list the

average time to process one single unit so concepts are passed six three days high poly model takes five days QA takes five minutes etc and the third column you want to figure out the throughput for one individual unit so it according to if we're talking about one unit so inventory equals 1 that means throughput and flow time are just inverse of each other and if you didn't quite follow that don't worry just trust me that this column is just the inverse of the second one so for instance our average throughput from concept art is about a third of a character per day average throughput of facial animations is about half a character per day etc fourth comm you list how many people are actually working on the activity so we got one concept artist three high poly model it's for combat animators etc fifth comm you multiply throughput times number of people that gives you your combined throughput across each team so across all our high poly models they can do about two-thirds of a character per day all their facial animators can do half a character per day that sort of thing in the fifth sorry the sixth column you calculate the process throughput you

find the lowest throughput of any of the activities that's your overall process throughput we think back to the alien queen example it's the bottleneck it dictates how fast we move and the final calm you put in when you calculate what's called your utilization you take the process throughput and divide it by each respective individual throughput what this tells you is how much of that groups bandwidth this activity is going to take so this character modeling path is going to consume about two-thirds of our concept artist time it's going to consume about a fifth of our rigours time and it's going to consume 100% of the cinematic animators time they are the bottleneck they determine what speed this flowchart moves so if they're moving full time they're gonna be a hundred percent you used up by this one pipelines can take all their time to keep it moving so some takeaways here the number of resources you have working on activity impacts your throughput 100% utilization means you're a bottleneck when the way your staff at pipeline can shift where the bottleneck is between activities and shifting that bottleneck will change the throughput but not the flow time so don't fall for the one month baby trap you can't throw a five

and you can't throw a five modelers at the same high poly model and get it done in one-fifth the time don't fall for that so now what happens if we start adding some resources so here's our chart cinematic animators are the bottleneck and I'm going to add one cinematic animator now two things have happened one our throughput has gone up to a third of a character per day so we're going a little faster but now we have two bottlenecks we have two resources that are a hundred percent utilized by this process flow that doesn't mean we're worse off because we have to violent acts when we had we're going faster but it means we can't improve throughput anymore with a single head so if we just add one more cinematic animator throughput still capped at 1/3 of character per day the concept artist is still a 100% utilised he's still the bottleneck and now what we did is basically we gave the cinematic animators a little spare capacity that gave him a little wiggle room that's all we accomplished with that if we had another cinematic animator throughput still cap to the third character per day concept art still the bottleneck and all that extra head on the payroll did was create more spare capacity which is probably not the

most efficient use of your resources on the other hand if we add another concept artist now our throughput goes up to about 2/5 of a character per day cinematic animators in concept art are no longer the bottleneck in QA is now the bottleneck so you can see how you get this little you get a little bit of a shell game going on when you add people the bottleneck jumps around you got to chase it so the big takeaway here is that process flows exhibit a weakest link phenomenon you only move as fast as the pipe as the group of the lowest throughput adding resources will shift the bottleneck between activities so when you're using the capacity chart it's very important you add resources one at a time so you can catch exactly when and where the bottleneck moves if we added all three cinematic animators at once we would have seen that the bottom that gets shifted to the concept artist but we wouldn't have known exactly when so who would have known which which person we hired is actually just creating spare capacity and adding resources to non Banach activities won't help your overall throughput it just creates spare capacity so probably not the most effective use of money and there's always going to be a bottleneck this is important to recognize some

activity or activities will always have the lowest throughput so your goal is not to eliminate bottlenecks that's impossible there's always gonna be one your goal instead is to secure a throughput that lets you meet your schedule so this chart made some simplifying assumptions it takes a very clean view of the universe your view of the universe at night not be as clean some of these things might not be true if that if that's so that's ok you just need to use some slightly different math I covered in the appendix of the slides you can review it it's basic division it's not very hard but it's it's solvable you just need to adjust your calculations a little bit so on the subject of spare capacity and it's important to provide some guidance from an operation science perspective so it's a high pressure and there attempting to ramp everybody up to 100% utilization from operation science perspective this is actually counterproductive because remember 100% utilization means somebody has a bottleneck you're through your Banach is the deciding factor of your throughput the bottom of determines how fast you move so your goal as an Operations Manager is to make sure the bottleneck

is always moving and never waiting on anybody else wrapping everybody up to 100% utilization might make you feel good as a manager because if you can be a little busy but the reality is you're just creating more bottlenecks and those artificial bottlenecks might not be able to support the actual bottleneck so in our example if you're trying up the rigger with a bunch of busy work just to make sure he's always working and earning his pay the cinematic animators run out of work they turn to the rigger and say hey we did another model he says I can't because John has to be doing this other ship which means you have created a scenario where the actual bottleneck is being held up by the artificial bottleneck you created by being a taskmaster which means you're failing your primary goal of keeping the bottleneck moving by all means take targets of opportunity but don't keep people busy just for the sake of keeping them busy from an operation science perspective the best use of spare capacity is to alleviate the bottleneck so if you got people to spare capacity who can do bottleneck activities that's the best place to apply their talents all right we just covered in about 18 minutes which took me about three to four weeks of grad school so if your brains a little stingy tilly

understandable we're gonna take a little pat cleanser could take a little break some story time there's a story I've loved there since I was ten when I learned it it's the parable of Scylla and Charybdis so book 12 of the Odyssey Odysseus and his men spent ten years fighting the war at Troy they're tired they're trying to make it back to Ithaca they spent years trying to get back every month the crew has died they got one ship left one crew every else is dead and they have to go through the strait of messina which we monitor Sicily in Italy now I don't know if you knew this but at that period in history the Strait of Messina had two awful sea serpents living there but it's true story on one side you had Scylla who was a six headed serpent and each head would pull a man out of a boat and he boat that went past on the other side you had Charybdis who create whirlpools and maelstroms and she might not get you but if she did the entire boat went down everybody died and the strait is narrow enough you can't go between them you have to pick so Odysseus has to choose definitely six men or possibly lose all of them now from a managerial perspective this is a classic crisis is to test values Odysseus has to pick the needs of the many over the needs of the few

go ask ela or shared privations all for one one for all go with Charybdis from an investment standpoint his options are stick with certainty to minimize costs there's no risk miss Keela you know exactly what's gonna happen or you accept some risk to maximize upside you take some risk on Charybdis and you try to get everybody through so it's a it's a basic advancement decision now in the actual story odysseus decided to go ask ela but it perhaps a question well management decision he didn't tell anybody what was gonna happen so boats going bass and you know six heads come out and then six poor bastards look I'm going hey what's that alright so you know I'll let you decide if that was the right thing to do all right breaks over so moving beyond art littles law applies to any sequential process flow and that includes features with a very obvious caveat and there's that discovery again feature design is more varied than art asset generation because the scope differences bugs uncertainty and human error now variance is an analysis of

risk it can be understood and mathematically and it adds complexity or forecast you can't just deal with an average expected outcome speed times time you have to account for a range of potential outcomes and it increases your cost both your expected what you need to be able to pay and keep in reserve and what you actually do end up paying operation science is largely concerned with minimizing the presence and impact of variance and one of the main ways to do that is by eliminating waste so then the major question becomes who has eliminated waste better than anybody else on the planet for my money you need to go back to the heady days of 1948 Japan is still recovering from the Second World War and a little company called Toyota finds itself competing with companies from the world's newest economic superpower Merkel how can it possibly succeed necessity is the mother of invention Toyota invented the Toyota Production system and they focused maniacally on eliminating waste waste of

inventory defects meetings even movement if you're somebody who puts doors on cars they don't once you have to move too far physically to get grabbed and mounted the net effect is that Toyota is now the world's largest car company and established a reputation for high quality and TPS Toyota Production system came to me no more generically as lean production now the moral the story here is that what I'm about to talk about is not some squishy academic concept oh that's nice Justin what else did you learn in grad school no this is not Toyota made a lot of money and established reputation product quality wall could be with companies that have far greater access to capital and resources now lean is a Scylla approach to management you are spending some time now in order to possibly lose avoid losing a lot of time to waste later it's like you're going with the six deaths to avoid possibly losing everybody you're trying to minimize your outcome variance and maximize control like I said Sela there's no risk with scale you know exactly what's gonna happen six people no more no less you know exactly what's going on there so when

you're thinking about lean don't just focus on what it costs don't just think about the six men think about it saves the other people who get to go home so let's talk about how an ounce of prevention is worth a pound of cure there are a lot of elements to lean but there five I want to cover in particular today Bokke are you okay combin judoka muda and heijunka so first up okay okay it literally translates as mistake proofing originally was bakka okay or idiot proofing and Toyota I guess decided to tone that [ __ ] down a little bit it means design a product so they can only be used the correct way so your HDMI cable will only go on your ps4 in the proper orientation your ski boots will only clamp in a ski so the toes facing forwards in a production context it means designing parts so they can only be assembled in the correct way so eliminate sources of human error during assembly now what we do is clearly more complicated than assembling cars to spec but that doesn't mean you can't embrace the philosophy of poky okay and the best

way I've seen in my professional experience is with the humble user story from scrum user stories are a form of poka-yoke a they are an attempt to establish intent who wants something what do they want and why do they want it now again in my professional experience a good user story has three key elements first of all the story itself as some sort of user ID performs sort of activity to engage some sort of desired outcome as a player I jump so that I can traverse an environment as an animator I have animation blending tools so I can provide a smooth combat experience as an engineer I have continuous integration so I can have an efficient build process the second element are acceptance criteria you want to get if you're the person requesting a feature you want to get people clear goals as to what they're actually expected to deliver give them a clear target and finally technical requirements this feature needs to interact with these classes and deal with these objects needs to occupy no more than this memory footprint it needs to accept this input and provide this output try to think ahead to what what a feature is going to need to be technically sound yes this takes time

but again don't just ask what it costs think about what it might save if you're routinely experiencing miss executed features missed acceptance criteria a little pokey okay might be exactly what you mean additionally if you have a creative leader who has sort of impulse synaptically driven design decisions who creates a lot of churn and feature creep adding a little bit of administrative cost can curtail that it's the same thing when they say if you go to a website and one click will drop your your your visitor throughput by about 50% the same kinda thing just adding a little bit of administrative cost can kind of curtail that that feature churn because somebody has to ask is it worth me writing this thing to actually request this feature that I just thought up while I was taking a whiz next up is Kanban you with experience in agile and scrub you might have encountered Kanban literally translate says card or sign I've also heard billboard it's what's known as a pull based production system what that means is when I'm when I'm if I'm the downstream guy I find the guy who puts doors on the car when I'm out

of doors I put a card in the card I send it to the person who assembles the doors he assembles as much doors puzzle back in the cart sent to back my way and that card corresponds to a certain number of parts I can't just say give me seven doors give me three doors we can only communicate numbers and request via cards the team members it means I pull work and I'm ready for it it's it's a pull based system instead of a push based system like a can of soup that's constantly going down the line you're not constantly pushing work down towards me I'm not getting car doors piled all around me that I have to rush and keep up and the amount of inventory in the system is entirely controlled by how many cards in circulation we don't request things other than with cards over time managers remove cards from the system in order to maintain the absolute level of inventory that's currently being processed why is that helpful two reasons let's think back to littles law the holy throughput constant assuming they can get cars out of the out of the line at the same rate if they have less inventory in the system that means their overall flow time is faster because there's less huge the the the parts don't spend as much time queuing up waiting for the next

process to start which means your actual flow time is getting closer to the theoretical flow time which means your efficiency is much higher the second reason is that excess inventory masks production issues it's like a protective layer of fat there's always stuff going around there's always something to work on so it's hard to see where issues are propping up and inefficiencies exist now if you pull excess inventory out those efficiencies become far easier to spot because you look around for the person who doesn't have anything to do because they're waiting on somebody else that's the first place you want to look to figure out where the production imbalances now we don't have cars to yank typically in game development but we do have work in process limits hand soft JIRA a lot of those tools will allow you to set limits on Kanban flows to how much work anybody can have assigned so if you want to take the Kanban approach start loose and then dial in the restraints over time next up is jidoka this translates to autonomy ssin or more literally automation with the human touch so at Toyota they have sensors that are

on the line that automatically detect production issues and defects and will alert the workers of them that if necessary they'll stop the entire production process to get them resolved now closest analog I've seen our automated testing scripts you you you have testing scripts that are written up you just run them in the end the script itself checks for any known issues issues you've experienced other predictable issues and tries to iron them out so you have to consume a lot of developer and QA time finding them other good examples I've seen bots definitely a game automatically so when wild load was making Avengers initiative we had a robot we called Auto Hulk who just played the game for us all day beat up robots 24/7 and he found so many issues for us that we didn't have to find through manual regression another good example is automated crash reports game crashes and somebody gets an email of a kite that includes any valuable data how much what was on the top of the stack screen shut up was going on at the time anything that can help me diagnose an issue that'll save you a lot of time and investigations and continuous integration is another great idea of of streamlining processes through automation now yes these things take

time to develop and establish but they say far more time with the length of the project and the earlier you get them in place the more time they're gonna be able to save next up move out this literally translates this waste and toyota classifies waste into seven categories defects overproduction inventories extra processing motion transportation and waiting now defects tend to be where you experience most of our pain as far as waste I'd put meetings up there pretty high but we'll stick with defects they come down to generally missed acceptance criteria or bugs no experiencing zero defects is not a practicable goal we are gonna make mistakes we are human so instead our goal should be to find it fix defects when it is least expensive which is as soon as possible ideally what about myself ideally before those bugs even make it into your repository before they make it into get github or perforce or subversion barring that before that contaminated code gets merged or at the very least before other code gets built on top of it I saw a great presentation

by a Bioware QA manager named Barbara clinic on Wednesday and she said something very similar to this so you know if it's good for Bioware's QA should be good enough for anybody so a lean approach to QA would have a few steps the first step would be buddy testing you bring something else over and have them check your codes locally in you on your machine before you request it gets merged in the codebase so you can catch missed acceptance criteria other sources of error the second step will be automated tests you run those jidoka style tests of changes so you can find technical defects before you actually request that the code gets merged into the code base so you can find those technical defects without consuming a lot of dev hours third step will be peer review have other people reviewer changes in the repository before the merge actually gets approved so you can catch missed technical requirements other sources of error before they content can contaminate a build and the final step will be actual manual QA and regression you have dedicated QA members test

the changes but once they're in the bill before they go off to two leads for review so you can catch defects before other couple gets built on top of it then you avoid that house of cards effect we've all seen it where you have a defect in the code base and you build other things around it that depend on that defect being broken in just the way it's broken into Sue's you fix it you gotta fix everything else you want to avoid that you also want to avoid wasting lead and director time your leads their time tends to be the most valuable because it's the most in demand so a lean process it behooves you to not send them anything unless you can have the highest possible level of confidence that you're not giving them defects now is this onerous yeah maybe but I've seen it in action and I've seen people use a very disciplined approach to queuing and they have remarkably clean code bases they have remarkably stable code bases I've seen them go from six-month projects two weeks to final QA and then it's live with with a lot of stability and no crazy crashes when it launched so there is an upside to having discipline

in your QA process now some of you might think a four-step process when in the context of something called lean and say a lean approach like this doesn't sound lean I mean that's more steps because you hear the word lean you think of something like that so slender not a skin on when I hear the word lean I think of this guy muscular technically disciplined and oh so handsome so when you think lean think Jean Claude Van Damme all right heijunka this means leveling literally now major misconception about operations is that batching is a best practice because it avoids switching costs the problem with batching is that it leaves a lot of in progress inventory and thus increases your flow time and thus decreases your efficiency so for example you got two products red and blue you're trying to minimize switching costs so you build up a bunch of reds and then you process them all at once meanwhile you build up a bunch of blues and you process them all lunch you're trying to minimize some of the switching cost yet the problem is then you have this perpetual level of inventory that's

always in your system inventory in the in the in the system is a liability it represents money that's tied up in in a in a production process that can't develop any sort of return on investment until it's been completed and the longer it lingers in the system the longer that flow time worse that opportunity cost is so give you an example let's say your cup your company's cost of capital is 10% so for the non finance nerds that means that your company can on average expect to make about 10% return on every dollar invest your leave because of inefficiencies you have 10 million dollars worth of bloat in your system that don't need to be there but are because you're just inefficient that means there's an annual opportunity cost of a million dollars that your company is absorbing if you can liberate that 10 million you could make a million dollars a year on it but you can't because you're inefficient now you're a good thing about small batches they make it easier to find and fix recurring defects it's way better to fix something if impacts 5 units than if it impacts a hundred now we typically don't deal in

physical I'm sorry the goal of operation science is to avoid smooth it's not to avoid switching cost it's to minimize them ideally you want a switching cost so low that you can cost effectively have batch sizes of 1 which is what Toyota got 2 they have car batch sizes of one car so an ideal inventory buildup diagram would look like this absolute minimum level of inventory goes in and it gets out right away now we don't typically deal in physical inventory but there is an analog if you change your definition of a feature being complete from in the codebase and functional to q8 and ready to ship because every incomplete feature represents a liability of its own much like incomplete products it's Deb hours you've sunk into your game that you can actually sell yet unless it's defect free so you can't capitalize on that work until you've run it through a QA process so our typical kind of protocol especially with box copy Triple A games build a diagram is you go through production you build up a bunch of features over time and you accrue defects it's some multiple of that then

you get to your alpha beta cert phase you start closing defects which lets you close down stories and every two lives to the office and nobody sleeps because they're doing with that house of cards and the builds constantly exploding they finally hit a wall and you go out ship it you know we'll do a day one patch you've been there we've all been there you want to ensure that features are defect free when they're added to the codebase move from the kind of prototypical build build build build build build build build build fix you know production alpha beta cert to a build fix bill fix build fix six build fixed bill fix cadence right now that sort of classical I'll cue at the end model where you're essentially batching an entire games worth of features which from an operation science perspective is insanity you want to make Q Bay patches as small as possible not the whole game smaller ideally QA paths for every submission and then you want to focus on making the cost per QA batch as low as possible both in terms of money and time now some you might balk at the notion of slowing down development to allow for a parallel QA cycle fair enough except I'm not

actually advocating that you slow down I'm advocating that you consolidate the work instead of doing 80 percent of work on a feature that development now and the 20% of the QA in the future when you get tells me to assert I'm saying doing that hundred percent of the work in one sequence defrag your production process the same way you would defrag a hard drive now here's the way I like to think about this in business you have what's called the time value of money it's one of the most fundamental concepts of business a dollar today is more valuable than that same dollar in the future for two reasons risk and opportunity costs risk because you don't know if you're going to get that dollar in the future an opportunity cost because you could have done something with it in the meantime this is why you pay interest you're compensating your bank or your credit card company for the risk and opportunity costs they're absorbing by providing you with money I think we have a direct analog in software development that I like to call the time value of fixes I think the fix today is more valuable than that same fix in the future for the same two reasons

risk because it might not be the same fix in the future it might be a much larger fix in the future because it deals with other systems an opportunity cost because if it is a larger fix in the future and requires more time to resolve that's time you could have spent somewhere else making your game better and I'm not alone in thinking in terms of financial analogues for software development because we have the existing term of technical debt if you make the shitty happy hacky quit code to get something in the build fast and then you leave it in your in your codebase you're essentially running up a debt on your production credit card and the longer that that lies in your credit card the bigger the payout is going to be to finally cleared up because you're gonna have a larger and larger refactor the longer you let it linger so a diagram of production in its entirety we're doing feature specs you want to use pokey okay you want to carefully define specs to eliminate sources of human error before they can happen in development you want to utilize Kanban limit work in progress to increase efficiency and make it easier to spot production issues when you're actually doing testing you want to do a judoka you want to deal with

automated tests and Mura you want to focus on eliminating waste through disciplined QA processes and finally when you're compiling you want to observe high Jukka you want to minimize the level of unfinished features and avoid batching you want to keep continuous integration going and keep applying QA to make sure that things are ready to ship not to get started with lean start small build up over time don't try to don't be the person who shows up the gym on New Year's Day and says hey I'm gonna I'm gonna stop smoking and I'm gonna cut down stop drinking sodas entirely I'm gonna eat eat nothing but raw vegan food find one pain point your biggest pain point try try to absolve on that and then move to the next one gather objective data don't just say like Oh things are better try to find real real real metrics I'm sure you track metrics find them track them objectively apply the scientific method observation hypothesis test results conclusion repeat rinse and repeat and then I tested eight loss aversion so

natural human psychology is that when there's a change we focus on what it costs us before we think about what it gains losses always a little larger the game for games for us so and my experience as a manager driving any change I always run headfirst in a loss aversion now loss aversion it's kind of like an oil fire oil fire to get it to go out you set off a bigger explosion next to it to choke it out lots of bears it's kind of the same way if somebody's experiencing loss aversion the way to get them over that is to put a bigger source of loss aversion Dexter you want to express the cost of inaction so yes this change will cost you but if we don't put this change it's gonna cost you a lot more will management psychology trick alright some closing thoughts we don't work in factories games are not widgets studios are not factories discovery uncertainty variance experimentation they come with the territory and that's fine that's natural but we also don't work in performance art we have obligations to be responsible with the money that's provided to us in the form of backers investors publishers

and we also would need to take account for the fact that we have our team's livelihood in our hands we have their careers in their hands we have their family's well-being in our hands and we need to be responsible about that and acknowledge that so I'm not proposing that we eliminate or reduce discovery it's clearly vital to what we do but I think we need to take every opportunity to eliminate waste not to curtail experimentation but to facilitate more of it than to make it more productive finally nothing I've talked about today is really all that complicated it was a lot of materials so I know it's a lot to go through but any one instance of it is actually not that complicated it's really not so getting started on any of the stuff is not that hard the hard part is maintaining discipline in their use and not cutting corners when it's expedient and discipline is absolutely key to seeking results so if you take away nothing else in this presentation I'm going to take this one last thought which is that discipline equals freedom

this is a saying of a man named Jacko willing if any of you are fans of the Tim Ferriss show or the Joe Rogan podcast you may have encountered this character here's a consultant and an author more interesting Lee he's an ex-navy seal and a black belt in jujitsu so I want you to imagine your head what a guy like - who'd say some [ __ ] like this and have this kind of resume like what he looks like imagine him probably looks like that like I got that's his LinkedIn photo that's where I got it that's how severe this [ __ ] is he can't even smile he could just frown less right discipline equals freedom is his motto if you have the discipline to eat well and get regular exercise like this gentleman right here you will have the freedom afforded to you by good health to enjoy your life if you're a discipline about studying throughout the length of the quarter like I get in grad school you have the freedom to sleep the night before the final because you won't have to cream and if you have the discipline to follow processes that reduce and eliminate waste you will have the freedom to spend your time in generating more value for gamers instead

of fighting fires and with that I will turn it over for questions but first here's my contact info if you enjoy the presentation a lot more material my blog if you scan that QR code that'll give you my my personal contact info there's also I stacked a bunch of business cards right by the camera back there and also uh post-it notes for my blog just you know little tchotchkes so please avail yourself of that and I'll take questions please I actually really appreciated this talk I do have a question so in essence you this applies for process very well okay so you if it's a well understood problem you can produce and manufacture the game but what about the all that time leading up to the point where you actually not the requirements aren't clear so in essence yeah you want to minimize the amount of waste and [ __ ] you have to throw out right and make you're gonna end up making the wrong things throughout how do you have a good research and experimental development

process beforehand sure so there are two broad categories of activities value-adding things that make the product whatever it is in our case games better for the end-user and non value-adding things that don't like meetings rebooting pcs yeah so even within your your prototyping phase you're gonna have some recurring events that you're gonna deal with all that for anything that's recurring that you that you deal with multiple times in a routine fashion can it be automated eliminated outsourced or streamlined you know and you can't contain the chaos of the prototype but you can find those other things to streamline and you can start building in start establishing things like unit tests are a lot easier to do if you establish them from beginning and build them up over time so trying to make those investments you can reliably make now to facilitate that experimentation and then also if you're when you're doing prototype code it's fine if you want to make the hacky code to get to testing feature faster just make sure you don't put it in the real database put it in like a shitty hacky prototype database understand so that it's your question personally like says

one of the other things like so are you familiar with the Bethesda a level design process I am actually not alright so in essence what they do is they plan all the levels then they lay out all the levels and they put all the gameplay data into all the levels and test it then they put up make sure the final content or close to final content goes into all levels then they do a lighting and was it lighting and I think effects pass and then polished forever and one of the main reasons why I think this seems to work is in essence that you are put starting off with content that is cheap to make cheap to throw away and as you get further and further further down the line you understand what the hell you're making and what actually the final game is going to be far better yep and so you're in essence you should continue working on the things you're actually going to have in your game yeah all right so which doesn't quite align with

this flow process where in essence you're talking about getting as many characters out as efficiently as possible in a final you know way and so I mean I understand the whole you want a QA stuff and make sure it works but it's also useful to be able to figure out what actually you're going to have to QA yeah that's that's totally fair I don't want to make too many comments on that because all of my understanding is there what you just told us um so one of my favorite quotes I think was Emerson said a foolish consistency is the hobgoblin of little minds so ya know no one process this takes it kind of a clearer view of the universe and Bethesda's view of the universe of the reality it might be deeming for that that's really fine and if that is the process that's most effective yes do it I understand so does this cover all situations now does it cover all cultures all value systems no Maya made for us to say like here are the tools that exist young here's a way you can think of the universe in terms of eliminating waste adapt mixing for me

it's the same thing with scrum like pure scrum was like pure socialism it sounds like a good idea until you actually try it you get add apps come to your culture so it's the same to me it's the same thing really what would is most important is finding ways to eliminate waste so you can facilitate that discovery like that discovery is totally valid yeah what are the thing what but even a company's grades Bethesda I'm sure has pain points that they could resolve so where can you make investments to resolve that to facilitate even more of that does that is that fair yeah no it's totally fair I was just understand how do you like how do you switch over as you go from discovery into production sure I mean that what how do you switch over I mean what if you know what you're trying to make it's a lot easier to say okay now we know the characters are we know what it takes to make them and now we can run through a process I mean that's we're having clearly delineated pre-production prototyping into prod in a production into post-production kind of plays in it to me it's like good production is each phase should have its intention and for me production is the intention is okay we've proven the vertical slice we've proven what the

thrust is we've proven how to build these things and what they should look like so now how can we make them in a predictable as predictable way as possible so be know to be predictable assumes that you have that information which is what the process you're talking about and inquire and like the the way I think is a one thing that I think is very valuable as they also have minimize cost of change so in essence yeah that that's switching minimize switching costs totally good totally good thank you I'm curious if you've seen the work you've seen lean and Kanban work for the artists workflow in a team that's doing more batched like scrum yeah so any hybrid any advice for I've I've never found scrum works well for artists because just like because the flow chart the critical path has never never lines up cleanly with scrum breaks so when I've when I was doing scrum I was doing

scrum for the feature designed the developers and the engineers and then Kanban for the artist in parallel because it just my kind of my problem scrums I've gotten more and more experiences that scrum kind of artificially makes these batches and so I've I've kind of moved away from scum into pure Kanban because it's just I want a small batch as possible so I find that works better for artists but also in my experience I think it works better for engineers - thank you thank you please Hey so for your example and when you're talking about like a sequence of work that has to happen dependent upon one another use the art pipeline cuz then you could really know you know okay the you know modeling takes this amount of time and say texturing it takes this amount of time how do you break it up for like programming tasks which can have high variance you know we get into this group all engineering tasks all together don't even take a short time long there you go about you know breaking this up or do you just say all a generic engineering tasks sticks you know on X

based on you know your model so what I don't like to try to anticipate the time it takes to code something because that just it's too unpredictable what I like to do is focus on the time it takes to do all the supporting things that go to the coding like you know writing the feature request property so it's you make your life as easy as possible make the QA process as understandable as possible make sure like continuous build integration as fast as possible you have a reliable source database like try to lay all those cards out to make sure your life is as easy as it can be what helps for actually dealing with code the code development self is a I think it's better to have smaller as small feature as you can I just give them on the smallest possible increments so you're not like bogged down with one thing for a while that's my experience and then the other one is I don't I'm less concerned with how long a feature takes which is hard to track and not so useful that's a that's a measurement of precision I'm not talking to the precision I'm concerned with accuracy so I'm much more interested in how much how not less in how long something takes but more how much we can get done in a

certain time so like story points I'm really careful if a if a if a feature takes you eight hours or 12 hours since who are to predict what I am interested in is how many points do you clear up per week so instead of time per issue I'm worried about issue for time does that make sense yep so it makes sense that's where the variance comes in like there's only so much we can control the variance of what you experience so I want to eliminate the variance around that so that you have the easiest time possible that's my generally my approach to working with engineers especially great thank you anybody else any questions go once go twice all right like I said cards in the back you help you slip the post-it notes please reach out to me I love talking shop anytime and thank you very much you guys me great you you