PM Planning & Prioritization in Game Development (Philosophy Friday #7)

philosophy friday welcome everybody we're here we're back took a week off i was in in l.a moving and shaking doing deals partying it up hanging with dope people you guys know how it is anyway we have not really [Laughter] kuwait is still a thing the wrong the wrong people to be talking about that anyway if i don't people you mean my my five-year-old son then yes yeah of course of course awesome making deals making deals so for the audience we have a special guest joining us today to talk about product management a topic we actually get a ton of requests for and the topic for today is going to focus on approaches for planning and how to do prioritization for development

and so joining us our special guest is adam telfer who is senior director of product and head of central product at games welcome adam and before we jump in it'd be great if you can say a few words in terms of who you are in your background um yeah i've been around in mobile free to play since um before that it was a console dev for ea and before that i was actually indie it was kind of a weird thing i was in indie for 10 years making flash games for sites like new grounds and that's kind of where i got my start um and i've always been in like straddling between being like a systems designer and a pm um because most of the time like i'd be joining a pretty small indie team and i'd be the only one that wanted to do that work or was capable of doing the work um everybody else wanted to do the balance or do the say like focus on graphics and gameplay i was always focusing more on balancing um but yeah and then

eventually i transitioned to game lead at wuga for about four and a half years where i actually learned how to do free to play properly um and then was a consultant for two years while i was kind of building up startup in berlin and then now i'm actually at wb where i'm touching product strategy across the portfolio so coming actually full circle back to pc console mobile so it's good um but yeah for the last five years i've been doing also deconstructor fun and ranting on the podcast so that's always nice and just for usual corporate stuff yes i am employed by wb but my opinions today are my own uh not warner brothers it's all good all right cool so maybe jumping right in i thought we could start by talking about the different kinds of planning and i know adam you're probably looking at stuff not only from a product or studio basis but from a portfolio perspective but maybe we could start with the studio side and then just talk about again what are if you were to categorize high

level what the different kinds of planning are that you would typically see for a game or a studio what would that look like and adam if you want to jump in with some of your initial thoughts sure so i'm assuming we're talking mainly like planning is to broad topic you touched on in terms of like portfolio planning as a larger publisher versus a studio um trying to plan out all the different games that it's working on versus at a game level um so what i probably do is is just to not talk your ears off for the next two hours um really like planning breaks down into different stages of development whether you're in that early say concepting phase prototyping your game um doing reproduction production and then live so i think the most interesting and philosophical um stage that i think we can talk about today is really focusing on concept stage uh since that's really where it sets the foundations for your entire pre-launch production and where i think a lot of pms fall down

so really just to also then step back and confirm what a pm does and is expected to do pre-launch versus say a producer or a game designer my perspective is that a pm is say the owner of the product strategy so producers game designers are focused on executing on a product strategy ensuring that the game that they're making is hard hitting say a target opportunity but a product manager's role is then to ensure that what they are aiming at is actually viable and gives the highest chance of success like i think that's my general philosophy about the the distinguishing nature between those and what product strategy really is um and of course every studio is different different roles blend with that but generally that's what product strategy is so the best pms during pre-launch especially during that concept phase are ones that can continually think holistically about the market and understand the deep reasons for why a game would ever succeed and react to inside say internal from internal

feedback sources or externally so things like consultants things like market data um things like playing games and and translating those learnings right and ensure that the target that the team is aimed at is then viable and then communicating that vision or that target effectively to the entire team um so with that metaphor i would say product strategy is the critical part to understand really planning in the early stages and making sure that this is well researched and it's understood by the entire team but also flexible and that over time it gets clearer and more precise um so then planning all starts with the product strategy or this thesis um and really then breaking down from there so for example if we took a thesis like for everdel by supercell you could assume that the thesis would be something along the lines of this the the way for um say invest in express games like heyday to reach a new tier of success is to build around truly social almost

4x like endgame that the theme fantasy of you know running your farm um building up your little village will retain the downloads interest and cpi but the focus on say guilds and training and cooperative goals will drive retention and monetization higher than the genre that could have been the thesis that say supercell came up with so taking that thesis then the pm's job is to start breaking that out into what must be right for this to be true and i think that's really what sets the foundation of then the entire production and what validation the pm has to be doing through all the stages of development and the interesting thing is right like validation does not necessarily just come in prototyping and production it also comes validation through competitive research say through the onset of the project bringing on external consultants and challenging your vision as well as you know production so prototype in hand or also

just saying that certain risks or certain assumptions apart that part of that thesis just cannot be validated until the game launches and the team of course accepts that risk that's generally how i assume of planning during that concept phase is build up that thesis well researched communicated to the team and then broken apart um so that then all future stages start working towards that got it and it seems like i i think i agree with you adam and the way that you kind of characterizes characterize different stages of planning but i probably take a little bit of a simpler approach in terms of the early phases what you're calling concept prototyping and pre-production i kind of roll up into like a single stage of pre-production where it would be about to your point like trying to prove out a thesis or the way that i kind of approach it is more around key risks so for this idea that we have what are the key risks is it is it new gameplay is it

the adoption of a monetization mechanic by an audience is it you know the a new kind of art style and and so for me when i think about the planning part in terms of the pre-production it it would be around trying to figure out how to characterize and then try to mitigate a lot of those things that happen a lot of a lot of the risks that that we identify early on in in the game project and then i think that the other kind of complicating factor as far as planning and concept is that it also to speak about a theme that brett and i have talked a lot about in the past in terms of situational context is that you know what kind of game are we developing right so for example i would differentiate games based upon like the world we come from which is more free to play mobile that are more systems oriented and where and if you're let's say or if you're doing a game that's based upon like you're fast following an

existing game then that's a little bit more science less less risk in terms of like the gameplay sort of finding the fun whereas if you're trying to create new gameplay then the planning for that in and the way that you would try to characterize that to me is a lot different from if it's more systems more you know fast follow versus something new but um you know i think the other tricky part of planning just has to do with you know we've spoken about peter thiel before in the past but sometimes the biggest and the best ideas are the ones that are what peter theo calls a secret meaning everyone else thinks this is a dumb idea but through someone's insight or some artist's intuition we know that or that person knows that if we were to launch this kind of product in the market that it would actually be big and so then that would you know that kind of intuition and

insight would generally contradict the natural feelings of an mba who's like well show me the proof in in the data from the uh from the app annie data that shows me that this thing is going to be a hit or not so anyway those are some of the things that i think about but um i would i would say that like i i really feel that the key risk-based approach is in my opinion the best way to go and i would also recommend that a lot of folks in our audience check out there was a gamma suture post many years ago on a research project called the games outcomes uh games outcomes project where they looked at a lot of different characteristics of different games and then they looked at the outcomes was it successful was it not and the one of the high level takeaways was that you know studios should be oriented more around the key risks what are the biggest risks and then trying to mitigate those first and not

not like waiting until later but i know brett do you have any any other thoughts before we move on yeah well i'm just glad we're 11 minutes in and you already craft on mbas so pick that off the box jk work as a quasi mba myself i appreciate it uh i only got i i went to dharma's business school but i left after my first year to join zynga anyways uh i a couple things i mean well one i uh adam i had a question for you because i when you were talking about stuff one thing you said was to support your hypothesis well one i love the hypothesis this i i think that's great to align the product and other teams right and i think that's what a lot of game companies struggle with is the product team will think of this really cool product and then they'll go off and they'll say oh marketing team like can make this work and without having a strategy it's very hard to align those teams so that strategy really helps to

say okay our strategy is this and then because of that strategy we have a thesis one question i had you and one thing i wanted to talk about a little bit is you said then you go get data to support your thesis and one cool framework i love to use is analysis of competing hypotheses and i don't think we've talked about this here but the fbi created it to basically figure out how to solve very difficult questions or trying to answer difficult questions and one thing that they learned is there's a lot of bias to go and get data to support your opinion it's like you have a you have a bias to seeing data that supports it so what they would do is build a matrix of different hypotheses and then try to basically disprove them and then weaken the hypotheses and then the strongest one would then surface and i think that in addition to your process you may want to implement something where it's trying to disprove the hypotheses before you go into the what i would probably call the road mapping stage right or the planning stage where

you're going to go build something because i i look at it as product strategies i agree there then there's sort of the what you would probably call the hypothesis building stage hypothesis testing stage and then there's the road mapping and then there's the individual feature execution um stage so that's a great point yeah yeah absolutely we're if you start from a point of this is our thesis let's go prove it uh with data and especially when say competitive intelligence data is very correlative right nothing is causal you can look at a spike of data and say that's the thing i want it to be rather than it being um say actually caused by that and coming up with the wrong takeaways um absolutely but i still think um like coming back to uh jk your approach and my approach it sounds like we're roughly similar where you have it that you have a idea of of what you're trying to do right that drives your design pillars in your game um then you

have key risks or a thesis that's broken down into what must go right um and you're kind of saying well you need to make sure that there's room for um the peter thiel secrets right of let's say the royal match team um xpeak they they clearly had a ton of insights on the match 3 marketplace and said you know what [ __ ] it we're going to build this thing anyways we're going to go after the biggest players in the space right and we think we know based on core gameplay innovation right uh based on polish um and i'm assuming they had things in their back pocket from what they saw at peak to to support that theory right it wasn't just something pulled out of an ass it was no we we've we've run these things we've built enough of these games to say these things are important right uh which of course supports that when you're looking at startups that you're you're picking people that have worked on successful games but also that part of that key thesis needs to include that and i think it's

always interesting looking at like different teams and how they look at theses and yeah you'll always have it that um that thing if you break it out you start to say okay we're going to go after this player um where one person say let's say you're going after like world of warcraft right there's a whole thing of everyone assuming world war craft is number one no one will ever beat world of warcraft and then you have another team coming at them and saying look we actually understand very deeply why world of warcraft works we're going to go after them and try to beat them um a somebody who's at a surface level in that genre will probably dismiss the whole thing right we'll just say yeah no there's no [ __ ] way you can win in mmo you're going up against blizzard and then there's that team that says actually we have a deeper understanding and there will always be a conflict with surface level understanding and deeper understanding people yeah i agree and maybe a couple other points in terms of the concept phase brett to your point on like different

tools you could use so one of the things that we we also think about when it comes to key risks in that concept phase is to use a logic tree so and you know for for any of the mbas or x mckinsey folks out there that they'll be familiar with the use of logic trees but like so there there are some good things from from getting an mba but like yeah i think it's a useful exercise to just characterize you know some of the risks and then try to try to hone in on what the specific things that you could try to approve or disprove or test to see if it works or not and the one other thing that i would actually advise folks to think about in the concept phase has to do with kind of a theory that um brad i think we might have touched upon it before is but it has to do with a theory that i have around games as a manifestation of human latent desires so i believe that the reason why people play games is

basically to engage in a certain emotion or or because of some desire that they have right and so like maybe our our you know maybe there's a desire back from our ancient ancestry to dominate conquer land to like you do certain things and so that gets manifested when you know you're playing like king of avalon or game of war or like if there you know if you have a lot going on in your life and you want to show progress and it's hard to see any progress in your life but in royal match you're now at level 500 and you can see that progress and like like lights and and like positive reinforcement um happen then i i think that like the other way to think about some of these games would be to try to understand is there a latent human desire that you're tapping into and if you do if you're doing that how can you potentially magnify that desire that you're tapping into or

satisfying for your user base but just just another another way that we kind of have a diff have any like an additional lens on how to evaluate and think about games in the concept phase interesting i mean in terms of the innovation i just want one thing i wanted to add to this the way that i would think about it is that i would break the game up into different components and then i would apply a risk a risk number or risk valuation to the different components of it and then i would basically based on that risk and that i would look at the upside right i mean some of the cases that we're talking about if if they believe if i believe that i'm building a game and it's i'm going after a certain audience and i have a hypothesis that they this audience secretly desires to conquer right they've been doing match three and now they want to conquer whatever they want to conquer castles that is my sort of thesis and when i lay

out the different components of the game i want to put my risk component and my innovation component in that thesis statement and i think sometimes what i see when i work with game teams is they just want to innovate to innovate so they'll be putting innovation in the new user experience they'll be putting innovation in the what the live ops system you know innovation over here and it doesn't align with their thesis statement and so all those innovations are taking on a lot of risk and they may have hit the thesis statement but because they took all this risk in these other areas those are bogging them down where they don't even have let's say retention in the first three days to get even players to this really awesome core meta that they've created that does fit the thesis and i think that going through your game and outlining different components of it and being very upfront with this is our thesis and this is where we're going to innovate on and this is also where we're going to spend the majority of our time and focus and testing and design and iteration and these other things we aren't going to

is a great way to do it and i and i know what you're thinking jk because at the top level i think all of the when you think about those how much risk you want to take into those different components that's based on the strategy of the company right if you're at a big public company they're not going to want to take risk in all those components because the market's going to get really upset at them and they're going to have a if it doesn't hit so they're going to want to go okay well one out of eight of those we're going to take that risk if you're of startup like you are jk you might want to be taking six out of the eight or six out of ten and go we're going to innovate because but if you do the key component of it is if you take that risk if it hits it has to have a multiple on the results and that's also something i see where people take a risk on something where they take a risk on let's say the push notifications or something it's like dude if the push notifications hit it's not going to change your game you're going to get like a five percent maybe lift on engagement so why are you taking the risk there and so you also

have to look at it in the context of what's the value it's adding there so that's something i i don't think people do enough is like explicitly labeling something this is where we're going to innovate and this is where we're not and that's totally okay too you know like that you don't have to be innovative in everything to make an innovative game absolutely and i think um the thing that i always see um with designers breaking down their thesis like you pointed like push notifications not big enough i see a lot with designers pointing towards onboarding and ui ux as the thing they're going to focus on which is not for live services right like if we look at the graph of retention for live services if you're focusing all on those d1 all you're doing is converting more dabblers and certain instead of actually trying to retain a core audience so yeah that thesis has to be powerful and it can't be aligned with with surface level improvements and i think in terms of mapping it to

key risks the best way to do it is actually to start doing play tests like using play test cloud and these types of things with focus tests around that innovation because then you actually see real player reactions to these things rather than or i've seen things fall down where you do very focused or very biased market research um or you do say biased competitive analysis right like if you're doing play tests and you're you're testing against your competitive um your your incumbent's audience right and they're actually saying oh i really like this part of this game this feels different i want to try this more those are all signals that your key risks are aligned in the right place okay and then maybe just kind of so we're talking about the early early phase and and kind of pre-production concept so for this phase adam in terms of like before you decide to like green light a project or go in terms of planning or the types of analysis that you like to see

what would you like how much planning how much analysis do you think is appropriate for this stage to make that decision about whether you kind of proceed or not um so keep in mind that like i can only speak from myself as a person yes um not not wb right like wv has their own opinions on these things but from my perspective the ideal for those concept green lights is that um everybody is like like you have that concept you have that thesis and you do a opportunity sizing based on that right you look at that you say we're going to x percentage above this game it's very top down very complex and i think that's absolutely fine but then you still get a sense of this is the opportunity size that we're getting to and if we do these things right if these key risks are solved this is the potential up there and then you can even break that down and associate each one of those parts that we talked about that breakdown by risks here you can say

this is if we get this right this actually builds us towards a defensible position because we have this competitive advantage as a company we believe that we take on this amount of risk we get through this it becomes very difficult for somebody to follow us did that that's when you say okay this concept is aligned with a very strong product strategy that's the ideal absolutely right brett do you have any thoughts on on that i just want to double down on that because i couldn't agree more i think people don't want to look at the top down look at things and it can be very eye opening when they just say okay this game the top grossing game makes 10 million a year and your goal is 50 right and you're building a game similar to this game so what you're telling me is you're going to do 5x this game is that what you're saying and it's amazing how impactful just those kind of numbers can be and i think some people kind of avoid it because they want to stay in the land of like design imagination which is the fun

fun place and it's like okay just now you have your hypothesis now tell me the the raw numbers i know it's not a fun part of the process but we need to do it and that's that's where i do think the pm kind of plays a serious role of just making sure that you're looking at those brass tax numbers and and really believe hey yeah we're going to make 5x what this game is or we're going to take on five different games that make a billion dollars a year and we're going to get 20 right and we believe in that yeah yeah and i think that's the difference between when you have a pm during that concept phase that's really really structured in how they think about that phase versus a very creative team only designed and then a pm joins in late um or or say the pm just isn't structured enough in their thoughts that you then get to a point where you're like okay a pm looks at the game in mid production says okay this is our winning hypothesis right there this right this is a thesis for for how we're going to win in the space right and then starts breaking up

and starts realizing okay we're either not aligned to a large enough opportunity right or the the risks here have not been proven out and then it becomes thinking in bets right of do we really think the outcome here is going to be positive it's a very difficult position for pms to get into yeah and i kind of agree that the let's say the specific revenue revenue forecast or things of that nature in my opinion aren't are not necessary at this phase but i actually like to do it in this not not like a detailed model but more in a back of the envelope hand wavy sense where you know you build essentially as a sensitivity analysis around the ins potential install volume relative to the you know amount of monetization that you can generate on a per player basis and then you you kind of then try to characterize for this game to achieve various levels of success what would need to be true what would need to be true from installs

in terms of you know paid and organic and what would need to be true in terms of monetization and generally i like to um to create something that i call a monetization contribution table that would show the different forms of monetization you have in the game and how much potentially you could make in terms of of money from players in that game and again it's not the point of this exercise wouldn't be to be accurate it would be to understand what needs to be true what are the key assumptions that we're basing on for this game to perform at various levels of scale and i would agree that this isn't like a practice like you can have a creative come in and just say and just have a strong belief that i know this product is going to be successful it's a big problem it will work out but also for me i i like the practice of having this more disciplined

and proactive level of thinking where you're really trying to understand what has to be true and then you can then you know you that it'll help inform some of the planning and and how you can potentially think about the game as you go into soft launch and as you as you launch as well can you go into any more detail on that i'd love to know exactly especially super early on so if you could take um the shooter space or or something yeah you can say here's here's a breakdown that we see happening in a game in the space and here's how that maps to um like insights for the design team yeah so for example i think that one of the things you could do is let's say you are you so you mentioned shooter if you're if you're making a shooter game what i would do is to try and without knowing the specific design let's say but if you're making a shooter game that's a battle royale i mean you've got all the data that you

could use app annie to collect data on the organic volume they'll even give you a sense of how much is paid versus organic so you have a notion of how well different battle royale games do in the market today and then i think what you would want to do is to based upon your specific vision of the game you would then try to understand well how well could it do in comparison to those games what would have to be true and then also on the monetization side you can you know appian a sensor tower have revenue per data information but if you're let's say you're changing the monetization model then you could say well actually our monetization is more similar to another game a ballpool or some other game then you can see how those games monetize and then you can either blend or you can modify your monetization based upon the rp curve of let's say a shooter versus the other type of monetization that you plan

on using and so then once you have a installs model and again when i say model this is you know very hand wavy simple spreadsheet and then once you have a monetization model in terms of like what your potential rp curve would look like then that could then you can basically run sensitivity against it to to suggest what the potential revenue outcomes could be for the game does that make sense yeah yeah but when you go for funding and you put this in front of vcs do they just love this they're like oh my god these guys have it like down to a science they know exactly how much money they're gonna make well i would say this i have a bunch of friends who have raised a lot more money than i have and they don't do this so i i definitely don't think this is necessary but i would say that you know for me coming from the free-to-play space and i would say in general free-to-play guys are a little bit more analytically

driven than let's say if i were coming from you know like if i were coming from active blizzard or riot i doubt you'd see this kind of analysis and it would be more based upon the content and and the kind of play that they're going for right but i think for for like if you're more systems oriented if you're coming from free to play i would expect a little bit more analysis personally just because that's kind of the world that we could so if you say i'm going after battle royale so i'm going after call of duty mobile or pubg mobile they have their rpr curves obviously very low so like a dollar two dollar kind of one year two year rpis and you're saying i'm gonna go after it with a new monetization model are you just grafting the new rpi curve onto those installs and then multiplying the two of them or well it would depend on i could right adjust one of each so so it depends like what what are you

what are you doing uh if if let's say you have a monetization model that's more similar to a different game then i would probably try and take that rp curve and then adapt it and then generally speaking i would discount for risk in terms of the execution of that model but that's what i would you know that again that gives you a rough sense of what you could do and you know that what assumptions are baked into that so let's if you were using let's say you were doing gotcha collection well you know what the gotcha collection rp curve looks like now applied to shooters you'd have to think about how that would fit you know are you having the full collection depth are you having a smaller collection depth and then you would discount appropriately both in terms of time duration monetization potential but you could in theory you know come up with a reasonable estimation of what your rp curve could

look like based upon gotcha collection as an example and then you know and and then based upon your assumptions in terms of installs based upon the fact that you're coming in late what the organics would potentially look like what you could do from a paid perspective and then that will give you a rough idea a you know rough sensitivity model for what you could potentially do under various scenarios of installs various scenarios of monetization yeah interesting um because i definitely find when we're doing these types of top-down analyses um sometimes it's easy to graft downloads onto things like shift downloads from another game yeah and say we could recreate that same downloader yeah but i find that type of analysis very very flawed because downloads are typically a function of ua which is a function of rpi right like if if you have a a real advantage on the ltv side of the equation you can drive a lot more downloads than you

would otherwise yeah which is why you typically start to look at the market go ahead yeah yeah that that should go into your contemplation in terms of assumptions for your model and so one of the things that you can i mean it's not perfect but if any actually using their data you can kind of get a rough idea of organics relative to paid installs and so if it's if it's heavily organic you don't have that issue if it's heavily paid you would have that issue interesting yep very interesting all right and by the way on mba stuff um funny story so i don't have my mba i'm a computer science guy um so i've started getting into business roles obviously being a pm so i bought a book which is basically like the color picture book of getting an mba okay it's like the most bear basics mba thing read that never had a problem with anybody coming up to me about business ever since i read my picture book mba so

what what what is the book what is the book recommendation i think it's just cool it's it's a hundred thousand dollars a year some of these yeah visualization it's like a 20 visual engineering yeah well fourth for the audience rather than spending your hundreds of thousands on an mba you can buy that book um yeah actually i'm i'm generally a little negative data tables i was going to call out i know i know you're a data table guy when you're mentioning sensitivity analysis i just want as an mba a half mba here data tables are very powerful i don't think they're used enough in mobile gaming but they're really great for helping moving decisions along and i think that's what a next a good excel model can help do and that's what i used to use it a lot as a product manager is when i would get in discussions with the team and i would put you know i present what i would i was thinking would happen someone might debate with me and say well i don't believe we're gonna get as many of those downloads or i don't believe our

retention is gonna be that great and then if you have built a data table with different ranges of inputs you can show them like okay great well let's look at the output of the model based on your per your thoughts of the future your perspective of what the retention is going to be okay we're still positive there or oh wow we're really negative there that is a serious concern and i think that's what really great pms do is they don't they they sort of are like the psychiatrist of the team and they help everyone get to the decision that's the right decision using tools that they've created like a spec a strategy excel model a data table they and those aren't things that they just sat and created they've you know taken the input but then they use those tools to say okay let me let me explore your concern and let's figure out a way to either address it or alleviate it from being concerned anymore and sensitivity analysis and data tables are very powerful tools in doing that right yeah i would just say the the one danger

from a lot of the mbas who i've worked with in the past is that stop start at the start with the bucket throwing jk so i'm going to throw a little more [ __ ] now i have an mba too i have an mba but just to throw a little bit of extra shade here is that like sometimes like hard knocks jk like models are easily manipulated right and so like you know i i mean there are times when i've looked at models and like okay guys you guys are really trying to green light this game right it's so obvious in the model where they cuz you know like i i just think that models can be easily manipulated so i would say like the um the tricky part and so i completely agree with you brett like for a disciplined pm who's not being influenced by political forces model data models all that is always better to have and to be able to have an

honest discussion around a model is great but there are there is always that danger of politicizing and weaponizing some of these models which i i hope people in our audience take to heart and don't do because i've seen it way too often all right so but that's the the dark side of planning anyway all right so i i thought we could shift over now into production and so when you think about production guys any thoughts here in terms of you know what in terms you can uh you can build a house and you can kill somebody with a hammer i mean it's like kind of here goes giant right sorry absolutely

yeah now we can really talk about some stuff all right production planning any thoughts in terms of you know how you build a a road map milestones the the kind of planning that um you guys generally would like to see for let's say now that kind of moving from pre-production concept into like the production planning any thoughts on planning there well i was just gonna reiterate that if you're building a road map i do think that it's valuable to have an you know estimated cost estimated impact and a risk factor to that i think as you look at your portfolio and this is more for live games than it is for pre-production or in the sort of early stages when you look at your portfolio of a road map you want to think of yourself as sort of like a financial manager of a portfolio financial portfolio and on one end you want to have your vc style risks bets that are going to be these large multiples if they hit but a high

likelihood to to fail and then on the other end you're going to want to have your safer bets that are going to have the modest lift in revenue if they hit but more of like a 90 confidence that that they're gonna be successful and that makeup of how risky you want that portfolio and how secure you want it should be directly related to the strategy of the company right again if you're a public company and you want to hit a quarterly goal of a one percent increase in revenue then you want a safer portfolio if you're a smaller company and you want to double your company's revenue you're going to want to have a riskier portfolio and then generally the portfolio should just represent that strategy at the company at a high level um so in terms of like real production planning um what types of questions were what specifically are you trying to ask about that well i mean i think we can hit on both points so like you know if you want to if you want to speak to uh some of the thoughts on portfolio

analysis that that'd be fine but also in terms of like like single game production plan as far as i don't know if if you are or if you were to look at the production plan for a game what do you think would be best practice in terms of how you would do that planning in terms of the road map different milestones things of that nature any thoughts on that yeah um i'm not a producer and actually like i've gotten into multiple situations where like people assume that i'm a producer absolutely not my skill set i'm terrible at being a producer or project manager um so from my like from my outside perspective typically as a pm i'm dealing with much more like milestone delivery and saying these are our goals this is what the entire team needs to be focused on over this next milestone okay which is really typically a a function of what we talked about before breaking down the theses figuring

what our key risks are that we're trying to prototype or build against within that and those include risks of production and content delivery right is like especially for live services even during pre-production phases you're already talking about okay this is the live cadence that we want to be hitting with our with our beat strategy with our live plan strategy can we actually feasibly build a pipeline to produce that amount of content that quickly and um that that can go all the way down to okay so if we're building a match three game can we produce levels that quickly right what it was what are all the tooling that we need to actually do to build that um within a mechanic or a level structure can we remix that content very very effectively and these can be milestone goals that the the goal or the team looks at as goals and is measured against each time we we hit those dates right yeah and maybe like to your earlier point about different types of roles in terms of producers and pms

maybe this is one area of difference from my own experiences that you know at my company we actually do have we're more pm oriented and so like pms do interface directly with engineering and are responsible for p l are responsible for the road map and milestones for our game but maybe one i think one of the the biggest differences i would say and maybe i should ask this of you adam since you are also working on the hd side is that in my experience a lot of like systems oriented game teams or game teams that come from the mobile or free-to-play side are different than some of these guys that are let's say x blizzard x riot and with games that are more gameplay-oriented in the sense that from a planning per perspective for production like i would say the majority of game teams

usually have a full production plan meaning pre-pro alpha beta soft hard something along those lines but for like if you're coming from one of these companies like blizzard like riot on the hd side then you might actually stop at let's you have some some version of pre-production whatever that is whether it's concept or whatever and you'll have something more like vertical slice and you stop there in terms of planning and the rationale behind that is that you know you're trying to quote unquote find the fun right you're you're kind of trying to make something new and the amount of time it'll take is unknown and the and what will actually come out of that is unknown so that you don't do your your full production planning until you finish that vertical slice aspect of production but adam do you have thoughts on this and by the way i actually belie

i personally one i don't have the the reputational um credibility to just stop at the vertical slice like some other you know folks do but i also like a full production plan for other reasons which we can go into but what are your thoughts on that adam um so i can't really speak this to working with blizzard riot folks um if we're just throwing all of them under the bus during this video no no there's much by the way by the way very nice people by the way that could i have never worked in a company that i have a blank check for production they they love to work for one of them so let me be clear that i'm not throwing i'm not throwing shade on them i actually think that to some degree that's a superior approach right but i guess what i'm asking is more the question from a planning perspective how do you think about that um so typically during say the pre-production phase going into production the whole one of the key purposes of pre-production is to be able to fully understand

our production so that when we get to there we're not getting into a situation that that production keeps getting extended right the ideal is that pre-production is building up a very very detailed view of exactly all the content that's needed but of course the purpose of reproduction is also to front load as much of the risk as you can so that by the time you're exiting pre-production you are now ready to hit the ground running into production and you have a very very clear road map of how long it will take you to build it so the vertical slice point um typically will fall around that pre-production and production range but i've found that there's the ideal which is the end of prototyping you reach a vertical slice it fully proves all of your risks and then you start on building up all the systems necessary to build up the content as fast as possible but those two things can't run say next to each other like like step wise

they have to be run in parallel otherwise you get into these huge long projects right um where you almost have to like sometimes like reset technology uh when that that step changes and that's not really healthy um versus i think what's better is you build up the sandbox of tools during that pre-production phase in order to enable you to um correct your key risks and to to validate your key risks and then by the point of the end of predict pre-production you have that vertical slice you say this is the you know that this proves our key risk this this is a fun section of the game and here's the visibility that we have into the full production so that you have both um because i think before that point you are making guesses about your production online and i think it's still valid because you need to have a gut check if i'm making gentian impact right now i'm pretty sure i'm going to use the comp of it's probably going to cost 100 million in dev costs for us to get there if not more because we don't have hong kong

impact to you know leverage and build upon um so yeah i'm going to use top down comps up until the point that i can actually look at my real production look at the tools that we built and then do proper estimation does that answer your question yeah i guess so i mean like so for example like the way that i think about it is that it's just two different schools of thought or two two different schools with i guess advantages and disadvantage against each right so you know brett and i talked about before the story of um plants versus zombies and again you know very open-ended these two guys just started iterating they just kept iterating iterating iterating and then they had a cool product and then you can look at successes in the market whether it's hearthstone or whether it's valorant and those guys did an incredible job of making incredible products over time and you know i don't know the the specific detail but coming from blizzard coming from right my assumption is that's how they

probably oriented the design of production and the advantage there is that because you know you kind of don't know how long it will take to like and some models may fail in terms of a gameplay concept that you're working on so i i can see the advantage to that but i would say that the other way to do it which is again just based upon my own experience is you know no nobody's gonna write me a blank check so so we do have to come from more of a perspective of having a full production plan but i would say the one thing i like about having a full production plan even though there may be some you know what i call like non-linear aspects to like some parts of the game which we can't fully predict is that i think it creates pressure meaning when i think about some of the key leaders who i have a lot of respect for guys like elon musk guys like steve jobs one of the things that they have done is

they set very aggressive targets for their team but there's there's a date there's a time there's like a specific outcome expected and i think that by having that by having the full production plan it creates a pressure and that pressure i think forces key decisions to be made it forces it forces and stimulates innovation and there's a quote and i'll read it to you here but there's an old movie called the third man from orson welles and this this quote reads as follows in italy for 30 years under the borgias they had warfare terror murder and bloodshed but they produced michelangelo leonardo da vinci and the renaissance in switzerland they had brotherly love they had 500 years of democracy and peace and what did that produce the cuckoo clock so my own philosophy on this and the

reason why i orient towards having the full plan for production is really because you know i do believe in conflict pressure lack of resources forcing decisions forcing innovation making people do things they're not comfortable with pushing them to then create a better outcome i don't know just a dictator too yeah okay no work life balance right like just get it done by this date otherwise joe's gonna come yeah i'm going to go ahead and disagree on on a lot of that first of all on the first point in terms of whether or not you want to do what you're sort of talking about with is the iterative process versus the sort of pre-plan set plan i mean i i think what you guys are kind of alluding to is the cost of

change right like with the plants and zombies example it was two guys right so the cost to change the whole game is what minimal right you have two guys that scrap it delete go start again right if you have a team like working on hearthstone and you decide you want to create plants versus zombies versus a ccg or something that's a massive cost of change right so the pre-planning is more important with a larger cost of change that's not to say that you can't change it because if the change is relatively massive in value versus the cost of the change you're going to want to do it regardless how big or whatever the team is but that iterative approach is is much easier when you have smaller teams more control over decision making and less cost whereas if you have a massive teams it's the opposite so that's my that's my first point in terms of the second point the pressure thing i'm gonna go completely against that i am not a deadlines guy at

all we have no deadlines in my company and i couldn't stand them in gaming as well and the reason is is because they are so arbitrary and inaccurate and then people so much is held to these completely arbitrary very inaccurate estimations on how long something should take to be made right you have no idea how long your game is going to take to make or how long this bold beat's going to take or what's going to happen and then you go okay this needs to be done in 90 days and then all of a sudden isn't done in 90 days and you're yelling at your team because it isn't done when who knows if it should have been a year or 30 days or two days or five years there's really it's so inaccurate and so in my company it's there's an understanding and this is how i would run a gaming team too there's understanding that you're going to push as hard as you can to get it done in a certain time period and if we want to hit a quality bar that's really high we're going to be unsure of when we're going to land now i think you can do a range right like you're not going to just go

i think your point is you don't want to just have them build stuff for years and years and years but i think you want to bake in this like inaccuracy of your estimation valuability yeah and be like okay we're going to go about a month give her you know give or take two weeks and try to get to that point and then reassess and i don't know about i feel like that quote about switzerland again that's that's throwing shade that's throwing switzerland on the bus jk so that's number three on the podcast because there's a lot of awesome stuff besides the cuckoo clock i'm not like have no associates in switzerland but i know there's a lot of great stuff that came out of that country other than the cuckoo clock and i don't know if the fact that there was a dictatorship in italy that that's causation of great artists and by the way let me click i could just be like just random okay the point is is i i disagree with that completely i don't run my company and i and i err on the side of running it much more but it's just because it's so inaccurate and then you're then it's

just so inaccurate to be able to predict these things it's i mean everyone's been there why is this done today well you know it who knows it's well yeah because you told me to change the whole spec no no no but yeah yeah you just told me to do a week's worth of work in a day that's why okay jerk you know what i mean like that's why dictator jk okay so let me look at the cuckoo clock here i'm building the 16th chapel all right if i if i can how long did that take all right so you can't speak no you say so first of all like if you look at tesla tesla is notorious for missing their deadlines right i think that so it's not like you can't you can't say hey you know uh everyone's going to get fired for missing this deadline i think that's not the point the point isn't to yell and scream at people but the point is to like have a have a target a target that is

aggressive that is set for the team to try to figure out and try to achieve now i'm not saying when you miss the time i don't even know how do you even know if it's aggressive how do you even know it's aggressive it could be a total slot i mean i hated that you know when i was at companies too they'd be like well we're gonna make one percent this quarter how do you know that's aggressive well i think it's all of a sudden you have ten percent yeah so i think it's important to have a goal like so so one of the other thing that that elon musk states is like he has this famous quote where he says if you give yourself three months to clean your house it'll take you three months to clean your house but if you give yourself three hours to clean your house then the house will be clean in three hours and so uh i think the point is that is not so much uh not my house dude no way absolutely not [Laughter] three hours lean of those two centuries and they'll have my house be clean in three hours but that doesn't like three months

sure but i i think i i i i i understand you know what you know what you have elon musk can shove it okay i'm gonna go ahead he's not here he's not here okay so i don't really care about elon musk's opinion i care about your opinion about what you do okay i honestly don't believe half these guys who get on podcasts and make up this stuff i work 80 hours a week i work you know eight hours a day it's like dude come on man like you're dating these bottles and stuff you're not working i mean i would be careful let's not let's let's talk about jk what does jk believe what is jk i believe it's important to have reasonable but aggressive targets now i think it's important to have something that a team is orienting around and it's a clearly defined target that everybody that everybody understands right so and i do think that pressure does help create innovation it forces decisions and all that stuff is good there are you know it

in in the venture startup world there are numerous examples of companies that actually died not because of lack of resource but because of too much resource so from that perspective i that's the philosophy that i adhere to and again you know i mean we don't all have to agree here but that for me i believe that's that's the better approach for for me um i think like i agree with you in in in part jk yeah and i think if i put it in the context of live in the ideal situation in live and and brett like your company has a cadence of releasing of reports right but you do have deadlines even though you don't want them of your clients are expecting a report every x weeks right and they have to deliver those and those have to be up to quality and i think that's typically when the best motivation happens is when you have a deadline when you say this is the cadence we are updating the game every two weeks right we want to have the best quality

for players as possible and you motivate your team around this is the time you get to judge the quality right and there's goals and there's kpis and i i i think consistently that's been the most motivational situation so try to replicate that during your production as well right is you have those set cadences of releases and there's goals and key risks that you're trying to to go against um but of course like when you get to that point and you realize okay the game's not going to work i would not be so draconian to say let's just push this thing out because i really hate those productions of somebody arbitrarily set a date two years ago right and we're looking at the game and saying this is not tracking against our key risks there has to be some flexibility to say okay we need to change course we need to adjust this in order to hit those dates yeah and to the point brett you made are you doing it you bring it up just one sec one sec so i just one point i wanted to talk to adam's point there all right

go for it i mean he got up agree he you brought up a great point which is a clarification on what i was saying which is if it's something that you have knowledge of how long it takes to make right like our reports go out every month so we have very good knowledge of the production process and the time it takes then you can be jk level of intensity on the deadlines if it's something like making a new game that you've only done once or five times in the last six years and you decide to put a deadline on it like you do something that you know you know very well for example our our expansion products have no deadlines but our reports have very specific deadlines right so there's different systems for different type of products that you're developing is my point so i think it's a good clarification yeah but one advantage even for something that you're not sure about in terms of setting a deadline is actually the retrospective against why you missed right and so like let's say you set a deadline for two months from

now then after that two months i think it's it's actually a good exercise to then have a retrospective with the team that said okay we thought this we could make this in two months what were the assumptions going in that we that we felt we could make it and what happened why did we not make it and so having that specific deadline and then trying to think more critically about what went into that deadline i think is a good practice that then builds better understanding more depth of understanding of what what you're actually doing and areas that you could potentially improve upon or not and then we'll also improve future predictions as you do a similar exercise in the future all right so we hit early we hit production still all right let let's go into live ops something brett i know you you have a lot of go on this dude yeah how long are we going to go on this

by the way like a whole hour already we hit the hour tomorrow what time is it there we're we're deep we're philosophers we're full we're philosophers brad well 5 30. it's 5 30 on friday for adam here i think we should be like i got a beer over here it's fine holy water here okay we got the beer pass all right for live ops um talking about planning both in terms of whether it's like the development plan or content cadence or anything else you know obviously zynga is known for planning based upon beats things like that but can you guys talk about what you think our best practices in terms of planning for development for content in liveops and adam is our guest again to you yeah so um i think we'll hand wave over saying like okay we've built up a live plan based on competitive intelligence said okay this

is when our big beats come this is when our quality of life improvements come this is the type of content that we're releasing etc um so typically when i'm thinking about um uh live um uh sorry a live production planning um so you have a set of content that you are say on the hook for you're saying okay we need to produce for say a match to the game we need to produce so many levels in order to feed our player engagement or to keep that up we need a certain quality cadence of that so this 20 levels needs to make sure that has x number of say level modifiers or level goals to make sure there's enough novelty introduced in each one of these content packs we have events that we're producing all of this kind of fits into a bucket of i would say like baseline content requirements for that team yeah and of course the quality level of that

goes up and down and and the team should always be retrospecting on on on making sure that that is as focused as possible in the highest value and creating the biggest uplift in each one of those content drops but then there's a whole other section of your team which is more flexible which is then saying okay so what are we doing on top of that what types of features do we need to develop what is our big beat strategy for some new big thing we're launching um what types of new events should we be delivering to our event library to be feeding into our live operations what new technology should we be adding to our say offer system all of this stuff has to be prioritized against each other so all of this feeds into some version of a revenue model so we all love excel in this conversation so that's good but eventually a team from pre-launch has a revenue model that they've built up which is fairly basic they go live they start predicting against that revenue model they might adjust it based on the play patterns that they see from real segments and

players and adjust that and then they get better and better and better at predicting what happens right they everybody makes a bet and says this will increase this by this much this kpi bid this much and then they see what happens and they get better and better predicting and of course you give that team you know at least a year of making bad bets because they're going gonna make that bets i think that always what happens is something like a soft launch right everyone's like oh we're adding this killer new feature day seven is gonna double we're gonna be top top 10 next year right and it doesn't happen right and then they go back and swing and they swing and they swing but then they get better and better at kpi so you have that revenue model you have a better understanding of your segments that then feeds into your overall expected outcome mapping and you can prioritize based on that so here's your revenue impact here's a crude estimation of your costs that maps out and you say where does this fit that's the basics okay does that answer your question yeah

and mr zynga poker what's the what's the best practice for live ops for you uh well i mean i i think i'm just gonna go the short answer on this but i i think that operations in gaming is a huge opportunity that i don't think is given enough attention i think it has you can see games be successful because they have better operations and we see that with our data through for example slot machine output right like they just have the ability to produce more slot machines in a shorter amount of time now there's a quality variable to that but it's i think it's just something overlooked and gaming is as you've mentioned jk very can be very complicated system a lot of batch design and i think that if i was building a game which i i i sometimes debate if i want to i would definitely spend a lot more time if i was in a large company i would i would hire someone who's specifically

dedicated to operations and not somebody who is just a producer but someone who is actually a mba level operations engineer to look at how to improve the operations that's that's kind of what i'll stop at okay yeah maybe so yeah because there's some very basic things that people don't don't adopt that are that are incredibly valuable in terms of operations i've read some beginner operations books adam some uh some more advanced ones but there's things in operations that people don't do like having not a full utilized system it's a perfect example right whenever i was at game teams and with game teams god forbid you have anybody who's not working right oh we have a dev over here it's not working can we fill in with someone and that is not a an optimized operation system an optimized operation system has buffer in their system just like a highway has empty spaces on it that's the best most productive driving system right and that's just a

good example if you have someone on board who understands these concept of operations you'll you'll benefit a lot for them particularly with larger live game teams right yeah i mean i agree with both of you guys yeah and to your point brett that the shift in focus especially live just orients more around operational efficiency and and that that sort of thing the only other nuance i would add is that i kind of separate one like hard launch into kind of two phases the first is like the initial hard launch phase and then more of the steady state live cadence part and the the difference for me is that in that let's say the initial and it could be anywhere from let's say two to six months depending on the game and depending on the nuances of the game kind of what we're trying to do so so my philosophy on this is that what you're trying to do as you shift into hard is in soft you have a number of

assumptions and models around different things and you're trying to actually prove out unitary economics on your player in soft and so that and that's what you know at least i do and then i really orient around what does that rpu curve looks like look like in soft and what can it look like at scale and you're building different kinds of models around your game but when you get into hard launch now this shift there's a shift from unitary economics to project economics and then it's for me it's all about how do i make this game break even and profitable so then all you you have different models around the economy around monetization around your player behavior around your content model in the content cadence and then around marketing scale and so that initial period would be okay how does the game scale how you know will this game break even and then how far can we push it it is my economy depth model that i haven't haven't soft as i scale up

players does that actually prove out you know the the monetization um assumptions i made in terms of conversion in terms of the number of free versus minnows dolphins whales whatever or however you do it i i don't use that model but that could be one uh the content cadence and model all that stuff does can you prove that out to and are those models accurate or not and then once you get to your breakeven profitability status and you've kind of confirmed those models then i would shift into this more okay all right guys now we can not necessarily relax but now we can move into a much more steady state operation and rhythm around you know a beats-based model um having a fairly regular cadence around expected outcomes every month or whenever and then just kind of driving the model from exactly how you guys had described but um i don't know well i i would slightly disagree with

that i would say that the best model is to actually if you're in a successful live game to incorporate both i would think of it more of like the military where you would have a navy seals team that was going to be investing in higher beta ideas i think that you do not want to just rely on a what i think you call like an operations model where you're just you know plugging out one percent 5.5 features you want a team that does that and i think that's not to throw any shade but i think that's the marine corps uh in in terms of the military i don't know the military that way well but um you know they're much more of very operationalized and i think that's what you would do with your live ops team or your content cadence team but you'd still want to have like a team that's going to be building the really new cool content or the really new cool live ops or whatever and they're going to be taking those high beta bats so i i think it would be a hybrid approach and each team would have different type of structures based on what you're trying

to achieve with them yeah and i would disagree with that and so for example let's say you're taking a typical zynga structure where let's say you've got you know your regular content cadence and you might have a a single yearly bold beat how do you guys think about coming up with that or planning for that or the difference if you would call that a second team brett uh how they would think about planning for that well i would adopt something similar to what they talked about with supercell right the team would have a lot more autonomy they would have a lot more ownership over it they wouldn't have to deal with the bureaucracy that was sort of applied to the other arms of the business they would be off on their own and they would be allowed to build something sort of like whatever they want because i think once you you integrate them into sort of the other system you water it down you make it be too much too collaborative you you built a camel instead of a horse right and so i i think that's what i would do all right adam what are you thinking and

just to be honest go ahead yeah it has to be on a separate track it has to be its own pod but of course as we talked about it before in terms of planning that almost becomes production right so now again there's still time limits there there's still you need to prove out key risks in that old beat by x date that could involve launching smaller features within the game itself and tacking into that release cadence to then see those impacts or launch a survey or something like this to be able to start backing up your assumptions going into it so i see that as a mini game production um as a live content beat right to then provide your risks get that thing out all right and just to kind of yeah and i think one one one big one comment on that though i think one big mistake with that type of team that teams make is that they build the team and they go okay make this huge yeah you know innovative thing right take a year and when you look at innovative companies and innovative teams and whatever i mean they generally

have built a rapport a skill set a practice together before they build that thing that's super awesome and i think that you're sort of setting teams up for failure if you just form them and then you set them off i think what i would do is i would form the team and then i would have them build a series of small innovations and get their sort of bearing and prove some of the capabilities get out some of the kinks figure out create that chemistry and then once they had done that i would expand the investment right i would go from a month investment to two months investments to three months investments of four and too often i see them break off and go okay here have a six month investment like that's a huge risk for a company it's a huge risk for the individuals sometimes they can give a false negative in a way where they're like oh that didn't work you know it's like well did it not work because of this or because it just sort of they didn't have their chemistry yet you know so i i would also sort of like create this plan for the team to to then eventually work their way up to make a big bold beef

all right and just to wrap up the conversation here any other ideas in terms of how you would build in more innovation higher risk ideas into into a game even while live i think the best thing building on brett's previous point is events you have a structure where you can be continually delivering novelty to your game in a time limited manner so if you mess it up it's only an event it's time limited so you use that as your track of okay this team forms they build that event they see what happens and then they they build on that event over time that's the great thing about live services and live operations is you can build out these pods they can work in parallel and you have event structures where you can test these things out and then build up towards something bigger and bolder i think that's the smartest and safest way to do it but of course still sometimes it's important

to do those big beats on a bold idea and i think that's still absolutely fine i'd rather be doing big bold beats in a live service than green lighting a whole new game um based on that i think it's just the risk level is significantly lower right all right guys i think one of the guys well i just well i think one of the keys on that and i just want to sorry i'm getting a lot of thoughts here but i think about this stuff all the time is you have to create a system that you are going to move forward on an idea that you as the boss or manager does not think is going to work because contrarian high innovative ideas are generally ideas that not a lot of people are going to agree with and we've talked about this before so going and creating this bold beat and then going yeah we all think this is a great idea you've already set yourself up for not probably having an innovative idea so you have to figure out how internally you're gonna green light something

as a boss as a manager that you're like i don't believe this is gonna work but adam who's the lead designer is super passionate about it so we're gonna we're gonna allow him to de-risk it and test his theory and he's gonna allow to prove it to everyone else that it's an awesome idea and that's really hard to do in larger organizations much easier in smaller organizations so you have to somehow figure out a way to overcome that but ultimately the decision is not being made by you know everybody on the team everybody can have a different opinion but ultimately the gm has to be able to make a make a call of this lead designers passionate about it they've brought together they've proved out you know portions of their key risks enough that i'm gonna be like yep there's your budget and is the gm the right person to make that call ideally yeah i think so i mean because gm can back an idea that they don't believe in right as long as if it's proved out all right you know another topic for another job

yeah you got that last that last one in there dude number four all gm nba switzerland blizzard riot yeah i i believe more in aries responsibility but anyway all right guys thanks for tuning in this far very long discussion here today thank you adam for joining us please join us again and i know a lot of you guys have specific uh requests for different pm topics please keep those coming we will absolutely hit on some of those at some point in the future so until next week catch y'all later goodbye from philosophy friday bye [Music]

[Music] you