Defense Against the Dark Arts: Budgets and Budget Reporting

[Music] hi welcome my name is Paul Vosper and today I'd like to talk to you about budgets and budget reports in 1998 I went to work for a small game studio in Chicago that made games exclusively for the Macintosh I worked on myth 2 as an artist an animator I created content within strict guidelines all were art work was done on sprite plates so you had a limited number of plates and each plate had a limited number of images we didn't have a schedule in ship games only when they were done at that point I didn't know about budgets or budget reporting the limits of what I could do are strictly guided by an engineer the size of our maps as well as the size and number of the sprites were well understood limitations two years later

we've been bought by Microsoft and found ourselves shipping halo on a strict timeline that had more to do with a new Xbox console than it did with the game's development it was a new engine our first would to move away from sprites to create a new IP on a new platform as our certification date approached we realized we're having trouble fitting into the memory constraints of the Xbox console we optimized code we cleaned up content and we rebuilt the Warthog 37 times we did all the logical optimizations which we could think of without sacrificing quality but nothing was really working as time ran out we realized that we needed to take drastic measures so we cut the resolution of every texture in the environment by half at this point it wasn't really a choice it wasn't even discussed with the artists we kicked off another build and checked the memory usage we were still

not in budget so we did the only thing we could we cut the resolution of the textures in half again what we shipped was only a shadow of what we had intended and certainly not what all the had been looking at day in and day out in the weeks before chef's most artists never saw the changes until they were playing the final release filled in 2004 we released halo 2 we continued to work in the halo engine improving the physics graphics and gameplay but also starting to include budget reporting in our toolset content creators became more of a part of the budget process and began to influence the creation of better tools in 2007 we transitioned the engine to the xbox 360 console with halo 3 we continued to refine the game and the gameplay but also honing the tools budget reporting improved as well becoming a fully featured shift system

we shipped ODST and reach as we became an independent studio again and started on our next big project in 2014 we released destiny a brand new engine but also two new platforms while there were significant challenges and plenty of unpleasant choices to be made none of them were a real crisis and none had a negative impact on the quality of the final game we established realistic budgets in the early stages of pre-production and continually updated them as the project evolved into the game you see today before content memory peeked we had reporting tools in place and it already been working with content creators on how to best use them to optimize content this is an insanely complex problem destiny is a shared world experience with multiple activity lines that can overlap in different ways at different points in the game these new game

elements were a real challenge to track and keep in budget but our reporting system started strongly and evolved through the entire development of the game for destiny I became more involved with budgets and budget reporting than I ever had before I found that I really liked it because it provided a fascinating look into the game engine as well as how the game comes together as a project my role was classic ta territory of analysis in evangelism I fought fires I dug through reports I had endless conversations with both content creators and engineers about their intentions I evangelize best practices in functioned as a human UI for our in development budget reporting system more than anything I had to understand our system and explain it to others again and again I've made plenty

of mistakes and had to continually challenge what I thought I knew about the system understanding any system means learning it from the inside out I still spend a lot of time in Excel poring over data to learn as much as possible about our system and how it works but no matter how much I how well I know the system new problems are constantly appearing that require careful analysis to understand why assets cost what they do where they come from or why they changed ta certainly need to understand the system from the DCCC perspective so you understand where the data goes once you leave photoshop or Maya but you also need to look at it from the engines perspective as well these are the hard and often unpleasant facts about budgets in the bottom line any modern game engine contains many complex systems that all have to work

together these systems are written by different people at different times each following their own motivations and constraints I try to make sense of them to paint a larger picture that can be presented to others what you're seeing up on the screen is a conceptual overview of destiny's content system from DCC source on the outer ring to the runtime game engine at the center I've drawn it many times at meetings and this whiteboard became a slide that was included in one of the lunchtime brown-bag talks I gave about the content system in our theater this puts me squarely between these two characters part Oracle in part accountant this is the territory the TAS thrive understanding a complex pipeline and toolset explaining that to artists and designers chasing down bugs helping to create better tools in trying to make the whole process as painless as possible what I'd like to share today is some of the lessons I have learned while working

on destiny I'd like to talk about ta roles and budgets and budget reports and how we can help optimize assets and build better processes but before we start I need to explain a bit about what content means inside of the destiny engine in the destiny engine the system in the world art get loaded first so all of the environment is in and able to run next is the player including the customisable gear and weapons then all of the designer placed assets get loaded combatants loot chests objects etc you can see that most assets in the game from the bottom of this pyramid to the top can use animations and effects so these they get loaded they get brought in at many levels assets that are higher in this pyramid can share content with assets below them but not vice versa so combatants can make use of world art textures but world art can't make use of

anyone else's data our spaces in the game are broken up into bubbles with transitions between those bubbles when we talk about a loading set we're talking about two bubbles and all the transitions that touch them this means that whenever you're looking at bubble B you have to think about both bubble a and bubble C as well this presents significant challenges to how spaces are laid out but also on how we manage the budgets for these as well in this talk I'd like to discuss some of the forces that work against keeping assets on track these are natural phenomenon and never really go away but by understanding what they are and why they happen you can manage their effect on your project as a TA I'd like you to keep in mind four things you need to understand the

motivations of both content creators and engineers you need to understand the cycles within your project both big and small you need to understand the technology of your tools and your engine and you should never stop learning and you should always share what you've learned with others this might seem self-evident but it's important to acknowledge right from the start content creators want to create content content creators want to create content this includes artists musicians animators and designers we hire creative passionate people who want to make capital a art their strongest motivation is to create assets not to manage them creating something of wonder and beauty is a joy but managing the costs for it is a pain in the ass no one gets in the art books by shaving 4k off of the texture or by reusing someone else's

geometry to save cost these are the folks that will pour their souls into creating magical beautiful things yet these are also the folks that will suffer the most from bad decisions and poorly understood data while they are working on something and it's the most important thing in the world but after they hand it off it must be stacked up against all other assets to be scrutinized criticized and possibly cut Tonk content creators usually learn by bitter experience that there are dark forces working against them but at the heart of every artist is that nagging feeling that they just want to create content most people think of assets is having a value that is measured in kilobytes but really the value of any asset has three distinct axes and as a memory cost it also has visual impact and it has the impact on gameplay this makes the value of any piece of content difficult to define and

very hard to measure the first axis is the easiest bytes our bytes after all even when sharing textures or geometry between assets memories should be simple to report and easy to understand this is the hard data that is the start of any budget discussion this is the axis of value that fits into a spreadsheet this is at the bottom line it's black and white and it all adds up but there are other important axes as well and they evade easy measure visually impact is a huge part of deciding what resources you should devote to an asset how large is the model how big or a massive is it supposed to look how close will the player get to it a character's face will have a different impact than a tree and a tree along a player's path will have a very different impact that one at the top of the hill on a horizon size scale

proximity and readability will all affect how much resources you'll want to devote to that asset but its impact on gameplay that is actually the most critical axis and is easily the most difficult to define or measure what part does the asset play in engaging the player what information does this asset need to convey is it fun does it advance the story does it help establish goals or explain problems to the player the radar array you see in the horizon in the upper left of this screenshot is one of the most important assets in the scene it's an important landmark that serves as a destination for several missions so it's both guideposts and goal it has a unique silhouette that can be read from anywhere in the space it's visual impact is small but it's a gameplay impact is enormous all of the things we

just discussed memory visual impact impact on gameplay are going to change over the course of the project this means that the value for any piece of content will change as well when you start a project everything has equal value anything is possible and you have far more good ideas then you'll ever have resources to finish as the project matures some assets rise in prominence while others sink a minor character is discovered to be unique and compelling so becomes a star a major character flops and is cut that hard edge steampunk theme everybody so admired at the outset is discarded in favor of ethereal futuristic sci-fi Holograms the story you set out isn't to tell is never the story you wind up telling which means that assets you use to tell it are going to change in value the rate

that groups add content to the game will change over time and the rate of change will vary from group to group environments must be masked out in engine before other work can happen world artists are working on pallets and explorations as well but they must build the physical mass of the game world before anyone else can play in it world art will be first in in first out sandbox that is characters vehicles and weapons follow closely behind and deliver one of each of their assets design starts on paper and only towards the end can they make full use of the assets that world are in sandbox have created they are adding to the assets to the game all along but the rate they're adding assets Peaks only at the very end of the project so it's not until design does their work that you see the quantity of

content increase to its full extent a character artist builds a single character but it's the designer who puts 50 of them in a scene each asset has its own cycle of cost growth as well a door starts off as a gray box then becomes a wood door then becomes a wood door with iron hinges and then a complex locking mechanism gets added to the to the complement gameplay this means that the memory cost of the added content will change over time combined with the rate of content gets added this creates a constantly rising slope whose summit can be hard to predict the resources required to remove assets are going to change over time as well every asset will have dependencies which have a huge effect on its creation as well as any editing or removal later on you can't animate a character that hasn't been rigged and changing the rig

on an character that has already been animated can have a staggering cost what might be an easy change early on consume have a huge downstream cost that require extraordinary resources to resolve this can lead to having to make pragmatic decisions about cutting assets not because it makes sense for aesthetic or gameplay reasons but because the pragmatic action is your only option based on available resources it's easy to measure how much content is actually loaded but what's loaded and why it's much more interesting for games balancing the what's loaded equation is actually much more important than the inevitable bottom line balancing within a single asset is an understandable exercise as you can easily see how all the parts work within a single unit comparing the size of one texture against another is easy to see and balance but what happens

when you have to balance one asset against another or against many others given a huge stack of assets how do you evaluate why one should cost more than the other or which one should be cut what happens when a cool new monster that design wants to add means that world art has to compromise on the visibility and play playable area in a level-4 destiny the disciplines that had the most control over their assets were also the most successful at keeping them in budget I want to share two very different stories the first is world art and the second is effects you can see world art up here at the of the content pyramid with effects off on the right hand side world art did a fantastic job on engaging with and owning their budget process they controlled the balance for all of their own assets so even though they were working with a complex and hard to

understand system they were able to get everything in budget and even handle outside requests for change every action they took reflected on their own bottom line so they were quickly able to see cause and effect and mastered their own domain effects had a much harder time they delivered best assets that are used by other groups often beyond the scope of their original intentions so effects would be in budget on Monday but over budget on Tuesday without them having changed anything they had a hard time seeing the causes for the changes in their daily usages they still did an awesome job in optimizing their content while delivering world-class effects but we never found a simple understandable way to balance their budget but it was designed that had the hardest time of any discipline not only were they at the apex of a complex system but design itself is made up of many subgroups each

with their own goals and motivations each of these groups affects the other and yet they all share the same budget communication and negotiation is difficult as each of the groups is only putting forward the best efforts for their area and the ones that they feel most passionately about finally finding the balance here is a problem that transcends design and becomes a project level problem in many artistic disciplines the rules for proportion and design have been well understood and documented for centuries this is simply not true for games there is no formula for a fun and rewarding experience during the production of a game you'll often hear people say this is going to be cool or this is going to be fun but you won't know if that's really true until it's in-game and in the hands of players Bungie uses a play test-driven development process which means that we try a lot of ideas by playing them over and

over again many ideas will get tossed out or get changed until they are unrecognizable this is great for the game but it means that you are trying to balance not only the assets themselves but also their intention and execution against the actual player experience there is no formula to follow it's a heated discussion that will go on all during production and even after you ship the game engineers can't create effective budget tools if content creators can't or aren't willing to use them the tools have to be built to enable content creators to balance the content budgets but if they never really use the tools then no one notices the problems or thinks about the ways they might be improved this stalemate leads to an accumulation of bugs that don't get fixed or improvements that never get implemented after we ship destiny we ask content creators to list out what they

thought were the 10 most important tools I mean excuse me we asked the content creators to list out what they thought were the 10 most important tools and improvements the engineers could work on next more than a dozen groups took part but not one discipline listed budget reports in their requests as you might guess content creators don't ever include building better budget reports anywhere in their job description so think about this cycle world art has measurable assets first so they start using the budget tools earlier and make requests for changes and improvements while the project is still early in the production cycle but design doesn't have any measurable assets until late in the project but by then tools support is scarce and everyone is against hard deadlines by the time a new tool or improvement might be finished the project will have already shipped so the task gets pushed off until the next

project so it's no surprise that tech debt piles up faster for the disciplines that finish later this accumulation of technical debt creates a problem that is difficult to overcome it's often a vicious cycle that repeats itself from project to project there's an unspoken pact that between content creators and engineers content creators must trust the engineers and the engineers must trust the content creators engineers are delivering game engine in the tools to create and analyze content they define what the game can and cannot do including many of the constraints that will face content creators they require that content creators be knowledgeable about and monitor their own content content creators are delivering content textures and geometry arranged and engaging and entertaining ways they define the look and feel of the game and look to engineering to facilitate that they

require that engineers are knowledgeable and forthcoming about their systems and how it affects that content in a perfect world this is a wonderful symbiosis but that is rarely what happens in games development as systems and the fabric of the game itself changed rapidly Trust is a fragile thing that must be constantly refreshed and renewed Trust is something that is hard to create and easy to break it's all too easy for trust to erode for and for all sides to retreat into their own corners and no matter how much foresight and planning you have you're inevitably going to stumble upon a poorly understood or completely overlooked constraint that affect the project's development and the goodwill of everyone involved all of the ground rules for asset allocation and destiny were laid down in 2009 five years before the game actually shipped while this initial set of rules held amazingly true adjustments had to be made as in the

game itself grew and changed and there were things that simply broke we always imagined that content creators would only have to deal with the horizontal axis on this budget chart that is they would never need to know the distinction between assets used by the GPU and assets used by the CPU towards the end of destiny we started seeing problems with tags memory on v1 consoles Xbox 360 in the ps3 this was a constraint that the content creators had never been aware made aware of and this is where I learned the definition of the turn non fumble because memory is non fumble which means that you can't use CPU memory for the GPU and vice versa the GPU will pay a penalty if it tries to pull data from memory that is set up for the CPU and the CPU will pay a penalty if it tries to pull data from memory that is set up

for the GPU so the memory that you use for textures and geometry has to be felt be defined separately from the memory you use for the remainder and you can't move the line between them after it's set that's what non fungle means for last generation consoles ps3 and xbox360 and this is about a 60/40 split where 60% of the memory is used for graphics and 40% for the remainder for current gen the ps4 and the Xbox one this is about 8020 with tags being much less of an issue and leaving lots more room for graphics so we ran out of memory for assets being sent to the CPU and couldn't borrow any memory that had been slated for the GPU we wound up having to cut assets not because of the graphics or their overall cost but simply because of the cost of the CPU bound data we didn't find this until towards the end of the project I

know though the changes were thankfully small the concessions were very painful to make and no matter how big or small your project is you have finite resources to make your game so the game of project development becomes how you can manage resources in the most efficient way possible so you can best deliver on your intentions this means you have to manage people you have to manage time and you have to manage assets in their cost there is nothing more frustrating than delivering an asset or experience only to find out it simply doesn't fit and must be either reduced in cost or cut entirely ultimately the game of making games is one of resource management using it's important to keep in mind that you are fighting forces that are constantly mutating and are indestructible yet there is much a part of making games

as the gameplay or the assets themselves as a TA you'll need to learn as much as possible about what drives the technology as well as what drives the people that write the code and the people that are making the content you'll need to help educate both content creators and engineers about what has happened on the project and what's going to happen next you'll need to answer questions many of which you will not have the answer to you'll need to evangelize best practices and often remind people of what they should be doing and what part they should play in making a better game while the dark arts will never be tamed there are tools and processes that will help you understand them by better tracking their behavior while we are talking about tools and processes there

are four things to keep in mind to understand any problem you'll need to understand the data but raw data is not enough any decision is going to require context different disciplines need different ways of looking at the data and always learn for your mistakes and tell others what you have learned when you load your game the engine has to assemble a staggering amount of data into a comprehensible experience of sight and sound but that same data has to be visualized in other ways as well so that can be managed according to platform and project restrictions budget reports are what allow you to make sense of that data to dive in and organize it by location by discipline by type and by asset understanding the data allows you to understand if you have a problem and

if you do where that problem is and how best to solve it when talking about budgets you'll need to view data in its raw form not just from the view of the DCC apps like Photoshop or Maya but in the way the game engine itself sees the data going back to this slide a minute you'll see that the content creators spend most of their time working in the outer ring that is the DCC apps like Max Maya and Photoshop or in grognak our world editor think of the outer ring as the intentions of the content creators but it's only at the center that you find the real truth that is inside the runtime engine remember that no matter what you thought was going to happen as you were offering content in Maya once the game ships the only thing that really matters is what's happening at runtime this is a huge distinction for content creators who spend the vast majority of their time

the outer ring of intentions so look at this data we have tools for analyzing budget costs from DCC apps through to runtime as well as many steps along the way these tools range from carefully crafted user experiences to gritty dumps of raw data in this diagram you'll see the intentions of the content creators on the left with a runtime reality on the right in this ecosystem for understanding data you're going to notice some distinct areas on the Left we have preventative tools which guide the creation of better content before it ever gets into the game engine on the right we have forensic tools for measuring the data the engine actually sees for any content creator understanding data can be hard but by

but when building models the geometry data reported inside Max and Maya is actually misleading and so artists might believe there's something wrong with the system how a DCC app perceives Bert count is vastly different than how our real-time game engine does UV Islands smoothing groups shaders vertex colors skin weighting and even game markup can lead to doubling of verts and rendering real-time rendering pipeline which adds to the cost of geometry this can be confusing for content creators who are looking at the vert camp as reported by the DCC app and assuming that's what the count should be remember that this is the Outer Ring the artists intentions and not the runtime reality so we maintain vertex estimation tools in both Max and Maya while still an approximation and is close enough to the final in engine results to eliminate the

need for the round trip through the importer to measure an assets runtime costs you can see in this example that Max is reporting these meshes as having about 3000 verts where in-game this is closer to 5,000 the tools show the artists which categories are the least efficient in this case the boundaries of smoothing having these tools as close as possible to where the content decisions are being made is crucial for connecting cause and effect while not perfect these estimates make geometry optimizations much more efficient and are also a valuable tool for helping content creators understand the difference between DCC and runtime costs in destiny textures have their compression and mid level automatically applied we also drop one MIT for the v1 platforms the ps3 and the Xbox 360

artists will often want to override these settings when creating specialized textures or when optimizing textures in order to balance a budget so we have budget feedback tools alongside the settings inside of the texture tag editor you can see across the top here that the source resolution is listed as well as the xbox360 cost in the compression method we allow v1 and v2 resolution to be edited independently from the source so the original intention of the texture can always be maintained common parameters are at the top but more detail can be exposed when needed this allows editing of the mid chain and the compression method for the in-game data you'll need different methods to display the data for different users and there are two main roles for budget reporting but first is for auditing an overview of conditions and trends and the second is for taking action forensic tools to locate and

resolve problems the first is to inspect the data from a high level to get an overall sense of current or future problem areas these users are often project or discipline leads who want to create or maintain a visual model of the assets and how they're tracking against budgets they need to cover large aspects of the game and easily identify trouble spots the second requires a much more detailed view these users are often content creators who will be making direct changes to the content they need to be able to dig much further into which assets are present how they got there and why they cost what they do both groups need to see this data in context so they can see how making changes to one thing will affect others or how an asset compares to the others around it it's important to note that all budget reports are really a contract with both curators and creators it's critical that they be true and

accurate and that they don't change over the course of a project for both curators and creators you need a quick method to identify any problem areas and to put them in context the web report provides information on every content build we do a quick glance at the color coded page will show which areas are having issues this information is accessible to anyone in the studio content creators project leads producers all can use this page for an overview of how the destination and activities are doing the contents of the page can also be emailed we have several distribution lists that get these reports emailed to them daily the web page also allows you to track builds over time this allows you to look at trends as well as defined when changes were introduced each point on this graph represents a Content build hovering over the data provides basic information about the build as well as

serving as a link to that build page when digging into why changes occur between builds you need to show the difference between the two one before and one after all the reports we're going to talk about support diffing so you can track down these changes here I'm comparing the activity report for two bills you can see items that were added or in green those removed or in red and you can see changes in blue with the before-and-after values listed I have two red items selected so they show up in orange and the total for those selected item shows at the bottom being able to pick out in total items quickly and easily is important when trying to track down problems all the budget reporting tools exists in a network of database servers with web front-ends and local tools you can run on the data for further analysis so you

can see the game build server at the center here in green wind builds are created they also generate budget data that is held on a different server both of these have web front-ends to make searching and getting info easier all the budget data can be loaded and processed on a user's local machine once processed an even number of custom reports can be run for environment for animation design etc so this is the starting point for the local budget reports this tool allows users to browse through a list of current builds from the server and load the budget data for any destination or a platform once loaded you can run reports specifically tailored to each discipline or simply view the raw data so let's start with a common case for environment a lead scans the overview report and email and sees that the overall data usages for environments is at 4 in your budget

using the link in the email they open the report to do a bit of digging and find the textures or a bit over where they should be they check the surrounding areas to establish context and make sure there is no systemic problem the Leben casts an artist's familiar with the space to cut down the textures along with some recommendations the artist checks the figures from the top down as well to make sure that they understand the problem in context remember the complexity of this problem when we talk about a loading set we're talking about two bubbles and all of the transitions that touch them so if a texture is in both bubble B and bubble C then we're moving it from B doesn't actually gain anything every decision must be judged with in the loading set and then balanced against all the surrounding bubbles as well so knowing which textures are in which bubbles becomes a critical piece of information

here's a full-screen view the environment report up at the top you can see that I'm looking at the moon in the first story activity and I filtered it down to show only the loading set of accelerator Hellmouth world art controls a wide variety of assets from graphics resources like textures and geometry to pathfinding terrain and geometry world art controls a wide variety of assets from graphics resources like textures and geometry to pathfinding terrain and visibility data everything here is broken down into logical units according to how world art needs to deal with them in the middle section of the environment report you can see that each of the categories are broken down into their own tabs textures geometry AO LOD textures LOD geometry and game data each tab has a unique custom view of the data according to the needs of the content creators inside of the textures tab here

you can see that all of the environment textures in this loading set are broken down further according to logical units so you can see environment textures listed separately from sky object and speed tree textures you can also see that not everything breaks down neatly so there's often a left overs category there's a drop-down at the bottom of the section for filtering the detail view and here's the detailed view of the environment textures when listing out the textures it's important to put them into context it's important to list not only their size but also where the texture is used and how many times it's used a texture that is only used once or twice is much more likely to be reduced or cut rather than one that is used 98 times you'll see on the right side that the bubbles that the texture is used in are listed most textures shown here are used in both bubbles but that fourth

texture which is only used in one after you've answered the question of where and how many times the texture has been used the next question that needs to be answered is what objects actually use this texture hovering over the usages will pop up a list of all assets that use that texture note that not only the assets listed but also where to find them in the world building tool being able to connect the reports to other tools is a critical step in taking action from this point you can take several actions that lead away from the budget reports themselves from this point you can dig further into the data open the texture directly in the editor or simply copy out a list of the objects that use the texture to paste them into an email all of the reports support exporting data to excel for further analysis this way we don't

have to duplicate all of the spreadsheet functions within the budget reports themselves you'll see here that there are far more data elements listed than what we saw in the previous texture report this is a complete list of the raw data the reality that's running under the UI many of these elements get filtered out because they are of limited use for content creators but can be critical when analyzing a problem to allow design more flexibility with the combatant races the graphics for each of those races had to fit into a specific budget this meant the designers could be confident that any two races would fit into any of their mission spaces this way designers don't have to worry about the cost for any particular character or vehicle or weapon all they have to worry about is that they never have more than two races for loading set so all of the fallen are the same cost as all of the Cabal we limited the budget reporting

for sandbox art to graphics resources only this allowed those content creators to focus 100% on the graphics and not have to worry about the cost of abilities or AI this also allowed the art group to focus on the visuals for each character by giving them easily understandable boundaries they had complete control over their own domain and could allocate resources as they saw fit to do this we needed to assemble assets into logical units other than loading sets so we created budget sets which allows an arbitrary collection of assets to be imported into a single package budget sets can include anything what are most often used as a collection of assets and all of their dependents each race is made up of multiple characters vehicles and weapons they are considered as a single unit for budgets but each of these assets can also be reported on individually importing any asset also

imports all of the dependent of abilities AI animation and effects so the costs for these are always available even if they are not being reported so here's the sandbox report view of the Cabal with the gladiator highlighted for inspection you can see the cost of the races a whole broken down in the upper left column with the selected gladiator on the right I've included the cost of animation and effects here but these are usually hidden by default like the environment report the sandbox report is broken down into subcategories the sandbox artists are mainly concerned with the graphics resources of textures geometry and impostors but all of the assets cost can be listed this is the geometry view for a single asset all the sandbox texture report is similar to what the environment artists use they require a more detailed look into the geometry the assets themselves are

generally more complex and usually include multiple source files so the report includes information on triangles verts and LEDs but also breaks down the mesh into all of its parts to report on the cost of damaged sections and variants this is another example of a custom report defined by content creators to make it easier for them to pinpoint problems and identify where to fix them one of the elements that defines destiny is gear our system for providing a near infinite assortment of player outfits weapons and ships each piece of gear is made up of a number of slots each slot can have hundreds or thousands of options artists combine multiple sub components called bits to create the options for each slot the specific overall configuration is called an arrangement so the budget reporting has to list the bits to fill

the slots configured by arrangement to show the overall cost of the gear while the overall budget for any piece of gear within a class is the same a shotgun has the same overall cost as a sniper rifle the cost for the slots are very different for example the costs offer a scope on a sniper rifle is very different than the sights on a shotgun this presented some interesting challenges for budget reports here's a view of a single weapon at the very bottom of the screen you can see the gear arrangement that's selected in this case the auto rifle you start the game with above this the main part of the slide here you can see all of the bits listed along with the slot that they belong to as you can see even a single weapon can be quite complex it's critical that the data be presented in such a way that the user can quickly locate problems and determine what action should be taken to fix it here's

a view of a different kind of gear this is the personal gear for the player helmets shoulder pads capes and etc player gear doesn't have any preset arrangements so everything is all a cart within a specific class or gender all of these reports are based on the same data types but they are presented in a way that makes the most sense to the group using them the presentation is partially driven by process used to create the assets but also by the preference of the content creators that have to make use of the data each of the fly 5 slots here has its own individual budget these budgets are individually set by class gender and slot to report on this we needed another custom view that would support not only the customer arrangements but also their associated budgets the design report presents some interesting challenges as we talked

about earlier design is actually five unique but interdependent groups they exist at the apex of the content pyramid and are constantly juggling assets to find the perfect balance between costs and gameplay for the main activities on any map the two design groups are mission design or story an ambien ambien includes patrols public events bounties loop and other events not directly related to the missions so in addition to the complexity of our loading sets design also has to track multiple activities running in parallel designers can plate asset can place assets directly or they can spawn them through squads or behaviors because assets are brought in from so many sources it's important to be able to identify which of the activities brought it brought the assets into the loading set designers are also pulling together assets from

many different sources they might place a squad of menacing AI to fight or a handy pile of crates to cover for cover or even an enormous spectacle that culminates a battle so designers need to not only so designers need to not only know the overall cost of the asset but how much of that cost is shared with other assets in the scene so when reporting an assets cost we also need a report on how much would be saved if we were to remove the asset the common questions that design needs to answer are what changed between these two builds how much memory would I save if I removed asset X and how much memory how much would memory increase if I added asset Y you learned for your mistakes by taking the time to stop and reflect on what things went wrong and what things went right after we ship destiny

every discipline took the time to write up what went right on the project and what went wrong there was a lot of discussion about we could do better next time but in doing so we uncovered a lot about our processes so even if they never change our understanding has improved tremendously these discussions turned into valuable documentation for setting standards and identifying trouble spots moving forward so we've talked a bit about the dark arts and some of the spells and potions you can use to defend your project against them as a TA you'll need to understand the technology not just from the side of the users intentions but from how the game engine receives the data you'll need to help educate both content creators and engineers about the budget tools and how to use them you'll need to answer questions lots of questions many of these questions will require you to do serious detective work in order to answer you'll need to constantly

evangelize best practices and often remind people of what they should be doing and what part they should play in making a better game remember that humans only absorb and retain a portion of any explanation they hear or see don't be surprised if you have to explain something to the same person several times over and don't expect a group to absorb information uniformly or rationally expect communication breakdowns and misconceptions and isolated pockets of ignorance and last remember that great games are made by ordinary people just like you thanks for listening [Applause] so I think we're gonna do questions now anybody has any questions I covered everything that is awesome are you all

are just so tired by this point in the day you're just like you're ready for the coffee break sweet oh one question some of the kind of things you discovered with other forms of analysis besides the stuff you you showed here one of the things that we often need to do is we'll need to take the entirety of a loading set and dump it out into Excel and then do a pivot table inside of Excel so we can determine the GPU resources versus the CPU resources and so a lot of that kind of the breakdown I showed on one of the earlier slides was done by just that by taking the raw data dumping it into Excel and then pulling it apart into the different types of resources that that were present in the loading set reports other things that

you can only there are they're usually not directly related to content memory the main thing I was just focusing on was measuring sort of the package data that we have the data that's on disk but there's a whole nother set of things that we do for the actual runtime costs and things like the memory that's required to update the game at runtime but that's actually a different set of tools but we do have those there's a whole set of stuff for doing that sorry that's not a terribly satisfactory answer to your question that's a different talk I think yeah thank you all right yes yeah great talk thank you I was just curious from a team organization point you've got a pretty complex tool system there just for the budgeting and I was wondering how many personnel maintained that it's I'm that's your this complicated answer

yeah just for the kind of the user facing inner side of it just the interface it's one person oh we're actually shifting a lot of this stuff over now and kind of rewriting a bunch of it that one person that's writing the tools that the users see is making use of a staggering amount of data that comes in from a variety of sources so whenever we write an import process and there's probably 15 different engineers that are writing import processes each of those import processes is writing out data that gets stored on all of the assets so if you ask an asset and say hey well how many verts do you have in this asset that data was actually written out during the import process so there was a one of them are 15 content engineers where's actually the person that actually I should write the data so

we have one person that's just doing the reporting process but then we have another person who is actually writing the import process so it's very distributed it is it is and then we have individual engineers that are working on all the web services you know storing the data and then also the web front-ends is and yet another person that's up there cool well thank you very much for that overview sure anyone else out there I'm getting a yellow card held up I think it's time for a coffee break thanks very much you