A Cost-Benefit Analysis: Is That New Tech Worth It?

[Music] hello everyone the world's a bit crazy out there today but happy bringing you this talk virtually so welcome to my virtual should you see 2020 talk a cost-benefit analysis is that new tech worth it my name is Ben Laidlaw I consider myself a technical director by trade I'm multiple art degrees including a master in fine arts I worked at the gamut of 3d from commercials to features to Triple A games to XR virtual production including now in software development on multiple continents and a wide variety of Studios I'll start this talk off by asking what makes GDC great what are we missing out on at this great event to me personally it's connecting with all the new friends all my buddies from across the industry from all the different studios I've worked at for a lot of people it's trying to get the next leg up in your

career well I'm willing to bet for most people especially the people that describe themselves as technical artists that you go to GDC discover and learn about what your competition is doing how you can improve upon your own pipeline somehow the tech out there is really amazing in others you know you could just kind of shrug your shoulder at what we really need to be truth about is all technology it's not created equal back at your studio you want to be able to implement that really good viable text the quality tech some of you may be lucky and you can just dive into coding and creating our glory for the rest of us who have managers investors you need to sell it to them first so what will convince the investor to come running to you with their money often it's the case they are not specialized in what you do they work in a different business their language is not I trust you go make art their language is why your answer should

be a succinct three letter apply CBA investor should hopefully come running to you at that point with their money or they may actually rain on your parade and say there's no money for that trust me you'd rather have the smart investor no matter how much it might suck if they say no so the CBA is short for cost-benefit analysis wearing you can appropriately analyze the human factor of any investment which is another method of showing the return on investment the ROI for the investor of what you're about to do - the intangibles they're our ROI is a common financial and business term but it doesn't get complicated with this fangled angle art technology that we're working on and this human life factor this is in the simplest terms how much your investor is going to actually profit off the technology that you're about to implement what both of these equate to is this tech actually worth it

is it worth the investment is it worth the time is it worth the human resources it doesn't matter if the whole industry is singing its praise that you're going to see if the hearts and souls of your company that the artist will love you and dance in the aisle well all very heartwarming and maybe if you had a tad bit touch of the force it could be enough what the CB allows you to do is put actual numbers behind that tech you might wonder why CBA's are effective and analyzing art from a historical point of view and even to today CBA's have been crafted and applied to large public work projects with large societal costs and benefits that are more difficult to quantify than hard technology cost if we were still in San Francisco and not virtually in San Francisco you'd be able to walk around the parks around us and one of the cool things is the San Francisco parks provided an economic

benefit approaching 1 million dollars it's a very kind of a cool inside of a CBA it was just apply that's really relevant to where we were all might have been had this week happened not virtually however we're really not so we're not putting an arbitrary value on our art has a real cost for instance worker time alone is a simple value we're not sticking a banana to a wall and duct taping it and calling that art so in the thirty minutes of this talk will it's a two hundred virtual people in attendance at an average of a hundred K animal cost is fifty dollars an hour which makes us a five thousand dollars worth of people sitting in our virtual audience sometimes what we were hoping for is just get rid of the more expensive cost of idle time so now that I have burned four dollars of your time let's go on to the guts and glories of the CBA so a quick primer this is the outline of the talk as we dive in I'll refer back to

this from segment two second as we progress quick thing to note is depending on the scope CBA may take a day or more of work to modify or create this document especially if you're small India or small project or you're just doing a small little like tech tech edition for extremely large initiatives with tens of millions dollars on the line large groups of people task teams IT and equipment this could take quite a while to implement even a month or more to do the proper due diligence but that one month of time versus potentially the millions of dollars on the line certainly worth doing that little bit of research even before you even put your finger to the keyboard to work on your CPA you need to know a CBA is a comparison of alternatives this is where the CBA gets its greatest strength room the same as any scientific comparison finding relevant comparisons is really the key though luckily the first two are always the easiest I generally start with what I

call the buzz worthy new tech this is a tech you see at GDC and catches the most buzz you'll see it on the latest feeds level 80 all the forums slack posts discord channels your buddies will be talking to about it at work I classify this actually as my second alternatives the first ala terms might be the most scariest the first is what if you do nothing and keep the existing workflow that you have at your studio at any good scientific analysis you need your baseline and the alternative method you cannot define value for one without the other more importantly for us when pitching why you should implement the new tech you need to define why the other is actually so bad to make a rock-solid CBA Harbor you should consider a few other alternatives this is what gives the CBA its strength and validity whether it's two three or four alternatives doesn't really matter compare it but it only matters to the type of tech that you are comparing between if you're choosing say DC C's is more important to list all the relevant ones than just the top two this is

considered doing your due diligence and research the best thing is to consider are the realistic possibilities or even something everyone talks about but you personally may know it's not related this should help smooth out some confusions to other stakeholders I do have to say a little secret though the scariest thing of all these alternatives and the analysis is that they have the power to prove what is actually good or bad whether you intended to or not sometimes there's a real reason why you're still working with that outdated engine in a pipeline the bleeding edge does not have a solid foundation as something that is a few few other Studios have worked out for a few more years and a few more projects as you walk her as you walk around virtually and you talk to other GDC talk so you see other talks across the globe see if anyone else is talking about the same technology today odds are if multiple studios are talking about it this this year and not just one this tech is maturing the studio is cutting the edge may not really be telling you the whole story I've been on the

bleeding edge and I have scars to prove it now that you have your list of alternatives when you're feeling out your CBA I generally work on the buzz where the options first I fill up my do nothing as I lament about the first then I cross-checked myself with my alternative options this keeps me excited in focus CBA can be a document boring as most things but important document nonetheless I often fill these alternatives out of order with a template lay it out because as you research is you know easier to look up the answers where a versus B then do each question in sequence in reality the priority doesn't matter the biggest key is to treat all options as equal so now that we're ready to work on our CBA for each alternative you need to do a short introduction and then you calculate the cost and benefits finally you compare your alternatives with the analysis defining the best method so when talking to your investor you need to prepare a series of elevator pitches

to explain what each option is beyond just a buzz worthy title an elevator pitch is if you had one minute in an elevator with your investor and you had to sell them on something say it to me quick say it to me fast say the most viable method forward you don't have any more than one minute to talk about it first describe how the method function works trust me no one will understand this more than you you're becoming the paid export but keep it simple layman's terms remember your investors your producers your managers are not gonna actually understand all these jargon that all the industry jargon stay what the goals are the unique elements of this method what makes this alternative special from others how does it resolve the current problem what are you really trying to fix make sure it's the cause not the causality state the expected performance and outcomes do durations and calculation get hard real calculations whether it's in microseconds or milliseconds what are

you saving are you saving minutes hours days is it an aggregate or just a means get reference or feedback who else is using this method did you contact them what were their comments or concerns has anyone else use this in production or called on it do not make this a twenty page product summary it is awesome to have references like that to this and other documents or web links but remember keep this cba simple and reference other documents no one has ever read all of Game of Thrones for this main reason for those that Indies are small studios you might not have to worry about these later complex introduction parts as much but for larger studios you may want to take these more seriously they may be outside your domain depending on your level so make sure you talk with the correct stakeholders we never work in a vacuum there are upstream and downstream dependencies so identify who cares about this work and if you change stuff who and how it will affect them identify who or view the changes think

of this as code flow or sign-off you don't check in your code out Friday on the way out the door this may already be a standard document at the studio you can just link to it acquisition strategy can you just buy the tech or the plug-in right in what is the process to get this approved where is the money coming from this could be a standard studio process you can share a link with I just say confluence is amazing for this does it need a few tweaks to get it integrated who eating the engine for instance is amazing product but it's cost a whole bunch more to put it into a custom engine than it does to use in a public engine how will the project be managed who is the leader the project owner there could be multiple owners for different parts of this tech set yourself some high level milestones you don't need a breakout jury yet is this a five-year project or a five-week project this may show that for instance the lower cost trade-offs against a longer strict schedule now we're ready to find

the actual cost Lucinda tech by looking at several types of cost drug costs are the easiest to calculate the amount of time you work on it is the gold standard your physical hours are money towards the studio photogrametry for instance has a lot of hardware and software cost for setting up the rigs and processing the massive amount of data before you actually get anything worthwhile out of it acquired assets can be turbosquid models or API frameworks or patents you mean it's a license indirect costs are not for the faint of heart often they are calculated and lump sum adjusted values by a studio the number of 100,000 cost annually I tossed out earlier as one of these it's different for each business however 100k was assigned cost at the rnh bankruptcy for the average wage of an employee from intern to senior plus the cost of their benefits workstation in the kitchen in the kitchen such as sort of machines and stuff like that I lived through that I went through the bankruptcy documents

that was our value that was about seven eight years ago now so cost has definitely gone up for the general part of the industry with inflation Microsoft and Google may actually calculate that your cost 250k so a quarter million dollars on average for sitting at your desk with their nice workstation will on ice provide a food or subsidized food for you and the nice benefit package that you have another good way to practice this is to figure out do you need a full-time or part-time producer do you need studio based IT for this project how many people do you pull into the big fancy meeting and burn their time how often do you go to the coffee machine and just drink all that free coffee in tangible costs cut at the heart of the matter does your tech take three to five minutes to iterate on a 128 by 128 tile and a 10-kilometer world causing your artists to press a button and watch YouTube and Netflix cuz it takes 3 to 5 minutes does your tech stink so bad the artists hate working with it because

they can draw on Photoshop in five minutes it takes a day or two to do it in the engine just the ability to make quality art stink so bad that you can only produce art like it's 2010 not to 20/20 god forbid it's 1990s tile tech did you make such a bad decision as to why there are no female characters in the game because you made a male specific character causing your studio received blowback and needing to spend hundreds of thousands of dollars even millions of dollars on marketing and internal team meetings because of a male rig perhaps your pipeline is so bad you need to work stupid crunch in order to ship then the rest of the industry knows your crappy studio to work for rising costs and hiring to replace all the people you just burned out cost maintenance is the long-term outlook on this tech especially that do-nothing approach what does it cost to add additional features can third parties integrate into your API do you even have an API will your pipeline slowly rot or do you need a babysitter to maintain the tech into infinity is this one year old

five year old decade old tech are you just patching an engine in a way past its prime are you guarding the US military code base that is used on 50-year old engine code 60-year old code and you're having those nukes just sitting in the launch silo on outdated technology I'm five and a half inch floppy disc benefits is really the human humanizing factor that separates a CBA from an roi art and quality of life are very subjective and human factors we can't really do find them directly but you can appropriately define them by looking at these external factors a small studio some tech changes can have huge increases the bigger the studio a lot of people don't have the same oversized impact as a small indie some of the direct benefits is did you go procedural and were you able to create more tone content and sell that content did you change how microtransactions work to get higher click-through just changing a couple

clicks a couple UI settings so that you click it goes through the purchase and they don't let the prick five six seven clicks to actually purchase something maybe your engine change so you can hit more platforms or you change your render to hit Universal and HDR content indirect tech are indirect benefits in tech part or sometimes the most measurable for us this is often text that can be can create more content did you increase worker productivity ten times over so they can create more assets each week did you decrease the amount of work so much you don't need that extra artist or the outsource company to produce the work did you take that three minutes per tile cook down two seconds eliminating hours of you time time due to time maybe even hundreds of thousands of hours each month or a year for an artist of all the intangible benefits that might exist my most sincere one is the thank you to me that is priceless this is when you actually do some type of code change you

actually make something and somebody actually comes up to you and says thank you you just solved a pain in my butt that has been slowing me down for day and somebody actually comes up and just thanks you more less ephemeral ones is say you fixed an annoying bug and your customers are more happy improving your customer loyalty is a really intangible benefit but a house buy back into your game for later and longer products your tech change maybe say your tech changed the playing field now you took over a larger market share or you reduce crunch and you know you can pop off games faster now make everybody happy while doing it analysis is the last part of the CBA this is where through simple graphs do all your background work that you can compare apples to apples with all your questions answers you can take those findings and add them into your spreadsheet this is the same task is produced you're scoping a sprint but with the time being converted to money in the end make sure you compare alternatives across the relevant time period work done in the past does not

count beyond the start this could be something as simple as making sure the cost of a plugin are all in the same common currency and not spread or out throughout all the different languages and finances across the whole world time and discounting or for longer-term projects with significant costs this applies a rate to the cost to normalize them to our present value in short it's a check on inflation say an HDR pipeline now as opposed to a you rpe pipeline but the tech won't be used until the next console you may have increased the number of consoles you sell in the future but you need to compare that cost to the number console you sell now for instance the cost of a gigabyte of ram now is significantly less than ten years ago so you're adjusting for that change over time calculating risk is another big studio long term project outline with all the big choices there's always that level uncertainty in the task even with small studios and Indies this is risk is is there it's very valid so say your

lead programmers leaves and that work that and he was working on that they were working on that buzz worthy tech and it will take a few months to replace them or you have a dark horse programmer in the wings and you don't skip a beat that the lead programmer just walked away it could have cost you several thousands tens hundreds of thousands of lost work the longer and the bigger the problem the higher this chance is sometimes it's better to take a less risky option with lower returns than a higher one maybe don't rely on that one lead programmer use two or three they might add to your costs but you eliminate the because you diverted over multiple people you can even go to the point where in you can do a sensitivity analysis risk adjusting multiple factors my favorite personally is if you're going real deep into calculating risk you can actually apply the Monte Carlos method to get a great distribution of risk fall-off getting a nice little mean curve to see how much it actually kind

of calculates out that's pretty much flying a render algorithm to risk analysis you do opt also in your CB a watch out for fake news once a CB a becomes a series of numbers it can be very easy to swing the values to a predetermined opinion I would encourage the stakeholders to find all the biases upfront and in writing in the CBA for instance in one case I was told on a project our team lost a company 7.5 million over the course of project you can tell this number was BS because first they did not want to show you the analysis it was just glossed over that oh yeah we lost seven a half million and if someone never hides analysis you should be very cautious right away also if we only cost the company a certain amount then that means we had no benefit so zero net benefit it's kind of a interesting argument that somehow we contributed nothing while doing maybe couple III presentations or defining a whole new branch of technology there are

legitimate reasons to not show CB just anyone for instance if they actually report workers wages but in most video this is an aggregate value as I mentioned before for instance one company this company quoted 250 K a quarter million dollars per year for any staff because it included all employee benefits cost your desk incident so cost like HR and cafeteria subsidies so if you do a simple calculation of a team of say six people for five years working full time that number 250 K adds up to 7.5 million also if you consider there was no benefit then would be a total loss however if you consider that maybe they never had six people working on it at one time some people only worked on it for say a year or a few months that was pretty easy to call on that figure simplest way to evaluate the total benefit versus cost as a simple ratio I recommend using a percentage based pie chart as a visual guide benefits over costs if the value is higher than one it's good the highest value is often

better when the comparisons are similar but if money or or if money is no option to the investor so make sure to look at the total investment needed and my suno not so fake example you can tell the green is good lewis cost to hire the benefit the better it is but alternative to that buzzworthy tech is a sixth for so it's 1.5 but your investor needs to have six million dollars to actually pay for that if they say they only have four billion dollars then alternative three is actually a better option the benefit cost ratio has identified that the buzzword tech as the best alternative however cost of opportunity identifies what you're missing out on since this contextualizes what you're missing out on by choosing one option perhaps you have another product down the road that this tech might not be good for protects the staff is best used

elsewhere on another project on another tech plan per se say you have two choices with a GP GP you purchase plan one buys them now at brand new costs while you develop the tech the other waits a year so you develop the tech with cheaper older models and then wait for the price to drop on those things was the work you could have done worth the price of that drop in the meantime or could you have used those GPUs straight away on something else in the meantime you can look at this as a final self check that you can roll back into your decision tree a comparison chart of alternative is kind of the simplest analysis extract of a CBA this is a summary interview that is presented to start the discussion of any CBA ranking in visual form it is a 1 to 4 rank of Rundlett x rank of each alternative and the reason for ranking this may not necessarily right up with a benefit cost ratio because we checked against the opportunity costs and found that the

outside-the-box tech in this case it's not actually necessary for this brand of product so even though it actually has a higher ranking it's actually less meaningful but if we develop another product it could go up so this means our actually our do-nothing is still more advantage because we're not having to develop a whole new tech branch just to get this one implemented but say we wanted to go down that whole new tech branch on the next project it might actually get bumped up quite a bit then based on your analysis you now have to make the best choice you need all parties to agree in writing that what that choice is if they don't agree on certain points write it out everything should be in writing a CBA can be very fragile also so it's built on a lot of assumptions as you've not actually done the work yet so if it comes if info comes in that changes the factors then update it make it a living document say a company buys a plugin that you were gonna use and gives it away for free this is really good thing to do it might actually move certain

text up the thing or said they buy a plugin that you were going to use and it costs ten times more and it's not really worth it anymore or so maybe they say they buy the plug-in and they shut it down like stinger for a brick engine and you can no longer use this plugin so all that investment that you were leading for is now just thrown out the window in order to get this stakeholder buy-in there are three main ways to do it and through present it first is to get local buy-in from all the stakeholders do legwork talk to each person in person or virtually beforehand second SCBA talkin that represents your data the third is a presentation of this data because very few will actually read the real CBA so the CBA document itself is your research this is the actual proof to show keep it simple it should be shown to as many people as that can be for some larger corporations due to dollar figures placed on people this may be kept above a certain pay grade there's a lot to be

said about truth and transparency but there's also a lot of said about keeping the secret sauce in-house why one company is actually more profitable than the other however the CBA document itself will honestly not be read by many how many people actually read the docs in the end the slide deck version is actually how you present this research the most people at one time do this after the legwork of each stakeholder do not surprise them in a meeting when you actually present to present it in Reverse of your researching it so to present your discovery first and science this would be normally the hypothesis you presented first but we're gonna throw out the hypothesis for the sake of presentation format people only care about the fact why at the end of the day not how you got there make sure as always keep it simple with lots of lovely graphics if you're working on visual technology show it for anything important like product specs include a reference at a PDF version never show

any lines of code that's the death most presentations that's the end of my virtual GDC talk I'd like to thank you for watching there any questions feel free to reach out to me with email or slack there is no survey but I would love to hear your comments everything helps going into the future thank you for your time and have a lovely year