Angus Lovitt And Hong Nguyen On The Fundamentals Of An Effective Forecasting Framework

[Music] thank you hello world I am your host Peggy Ann saltz and my awesome co-host Brian baglow well he isn't here with me today because he is prepping for Scottish games week which is kicking off October 30th and you're going to hear a lot more about that going forward so he isn't here and I could say well I feel a little bummed but actually no I'm feeling great because I'm excited to break new ground we love to do that here on pg.biz and it reminds me a little bit of what Brian Balfour said a while back at reforge he said all benchmarks are bogus well maybe we can apply that here because maybe forecasting is BS well maybe it is maybe it isn't that's the point we're going to be talking about it because forecasting

is of course all about the data that you need to make decisions about your game so to help you make the right choices and above all data informed decisions we have two very different perspectives from two very different ends of the spectrum so we're going to welcome today to pg.biz Angus Lovett he's ex-vp of marketing at King now co-founder at ramp a company on The Cutting Edge of forecasting and we have Hong Newen CFO at Space Games lohong a company on a mission to make the best mobile games so first of all great to have you you're United by love of doing data right and you're also United by Australian Roots so I am going to strap myself in I think for what's going to be kind of edgy and unfiltered discussion indeed how's it going all right look at looking forward to it yeah and I love a cricket so um I think we've been expats for quite some time so our accents may not

be as broad a little bit more more mellow I think Angus you are a champion of fundamental change in-house Studios approach forecasting let's start off with that how do you define forecasting and why do you think it's ripe for a rethink okay forecast like you said in your intro it's really about you know predicting the performance of the game or an entire studio in order to make uh Better Business decisions right but fundamentally it's a process of alignment you know it's alignment of resources to the size of the opportunity you know whether that's people or time or marketing money or what have you and it's about alignment of people in teams you know internal and external expectation setting getting teams to own the overall Revenue targets and aligning on the kpis that are going to get a step you know as a marketer by trade I'm pretty passionate about this topic but it mostly stems from the fact that you

know I've worked for a bunch of different gaming companies over the years and you know uh and like in most companies bad decisions are made at the top and brawls downhill right you know what frustrates me is a lot of these bad decisions could have been prevented by leveraging them Absolute mountain of data that we sit on as game developers or people working in the game development industry and and so not nearly enough care and effort typically is put into the actual forecasting process at the forecast and the forecasting process itself given how large the consequences are for getting it wrong right so the majority of times when I see see Target's not getting here where budget's missing right isn't necessarily about poor execution right is it it's usually down to poor forecasting you know it is it's green lighting a game you shouldn't or kill not killing a game fast enough

you know and they're the things that can really get you into into trouble right and if we're honest with ourselves you know if we're just down a homework better with the data we already have at the time we probably should have known better and that's what really has frustrated me I guess and what I said on a mission to try and do an Expose and um you know that's why I I found a ramp which is effectively a startup that does granular code was based forecasting we work with game studios adventure and private equity and yeah effectively we help do the homework to help with this better decision making you talk about what the outcomes can be Angus but we're talking about this now is there something about where we are now in the gaming industry that makes this more important more valuable you know as the gaming in industry matures I think you know I think finally by and large knows what good looks like when it comes to forecasting and forecasting process

but the problem is is they lack the tools what happens as a result is that manifests in in some pretty bad practices and I kind of jokingly put these bad practices in into buckets like we talk about the eyeballer method where people are like okay we're drawing a linear line from one point to another it's like okay we made 100 bucks yesterday we're going to make 110 wire because you know no reason that's just a trend line we're drawing through points or the boss man method where the CEO or some other one comes in and says okay we need to be hitting this Target because that's why asbestos are begging for us to hit so you must make it and of course there's no Rhyme or Reason you missed Target and then people get demoralized right or the roll-up method right this is the one where Finance teams completely absolve themselves with the responsibility and they're like to the games teams in this in this case it's like okay well you know this game's teams you know and they might underestimate uh you know if they want

to hit their bonuses or they might overestimate if they're bullish and they want to get a game green and what you do is you take all these crappy forecasts you can roll them all up into one big shitty forecast we call that the the roll up um I can swear on here right yeah yeah and yeah and so basically that they're the bad practices I see you know instead of what a good starting point is a bottom up um you know forecasting process based on robust data science so Hong Angus obviously long time veteran in games companies you currently CFO at space safe games and he's talking about the way forecasting is done been done traditionally uh by Finance people people like you um what are your thoughts I think a couple of things I think for me to add I think firstly around top down and the you know the big boss man and all the founders you know like perspective

they've got a lot in their plate and the last thing they want to be doing is getting the weeds of forecasting but it's completely understandable that they do just because forecasting is so fundamental to your Runway and strategic planning the decisions you make through your company um I think the point here is for Founders to trust their Finance teams that they can get good forecasting out of them but then it's kind of not rocket science really like uh you know it's Angus at the point you know it's really based on good forecasting is based on good assumptions which means Bottoms Up approaches is kind of required because what that means is you you kind of have a granular model that you can iterate very quickly on and it can take into account a whole lot of different variables so it's kind of it's very very obvious and yet sometimes it's kind of this First Fundamental step is kind of missed just on the point around why forecasting is right for a rethink now is making a hit game is extremely hard and it's not a precise science so the

second thing is it's extremely data driven so you kind of get this two points kind of crossing over and yet there is no innovation in Financial forecasting because you know in some other Industries you kind of don't need to but in this one you kind of have to it's so data driven so like for me you're coming into gaming sort of you know five years ago uh that's the one thing I kind of uh see as a gap I I would say um and and to be honest it kind of might be okay that there's been no innovation beyond the spreadsheet for the last 20 years but in gaming I think you know it's at the point where you kind of have to to differentiate yourself as a finance team that's a great point you know games have moved on we have different genres different sub-genres here we just had data AI recently on the show talking about these new opportunities bubbling up under the radar because there's a lot going on and there's a lot of innovation but that's not happening and forecasting Angus why

is that why is it not happening broadcasting I think it's it's like you know to say in this industry we're we're game developers right we we make games it's not our job to you know push the boundaries on on forecasting and usually you know people like myself in marketing or maybe Hong and finance you know we're the last in line for additional resources right like we're not going to get a team of developers to be working on our specific uh problems and that's kind of understandable you got priorities right you know a number of priority number one is to make a hit game that that people love um and so I totally get that you know and it's not just a gaming industry thing I mean this is a worldwide thing you know 99 of people are still stuck in Excel right or spreadsheets to do this forecasting work where you know quite clearly we all have the data sitting to do a much better job to make you know this real-time connected models which would vastly improve on what we do

um but they're just not happening right now you know Hong you've been doing this for 20 years you know across accounting Finance both senior positions General management roles you've seen this happening across media entertainment before coming to games now Angus here he is talking about how forecasting is done what can happen when it's done poorly or badly and that's what does happen um this is what also happens when you have Junior people who don't have a clue you know they're just sort of inputting data what's your experience what have you seen about the inputs because the inputs when they are not right then the outcomes are also just as bad yeah absolutely Peggy uh the stories I can tell you for another day I worked for a lot of startups in Silicon Valley in New York you know San Fran you name it La um the question kind of takes me back to uh a time when I was running forecasting at Myspace so I'm not sure if anyone's

old enough but it was when my special was the number one social network uh just don't ask me about how it ended I was running Regional teams doing forecasting to your point uh you know Peggy I've worked across lots of different Industries across a lot of tech uh the reality is you know Tangers point the stakes extremely high um you know it's not about one person one bad fpna person getting fired it's literally about teams and companies going bust and being sold because of bad forecasting and I've had some personal experience there as well like the consequences of getting it wrong are quite significant and sadly like a lot of companies don't really kind of really Place forecasting as a really important fundamental thing that they should be doing um and you know I've faced that harsh reality he had the coal face companies going boss so I think the the real the real challenge I think in gaming as well um and finances it's complicated so you're asking potentially someone who

doesn't have the the skills or experience to kind of look at uh look at the game development cycle project that into Revenue in the coming years I understand the businesses costs so fundamentally that they understand having a really accurate read on Burn rate which impacts on your Runway you're taking all those variables into accounts with Revenue being the the most complex thing you know of course it's going to go wrong you know this is not about a formulator this is about making sure that you've got the correct you know Runway because the entire company's future is kind of based based on it so it's really the fundamentals of running a business fundamental to key strategic decision making and I think it's easy to just take a lens of I'm just preparing this report and that's that's not why we're there we're there to fundamentally provide accurate reporting so that all the stakeholders can make the most informed decisions they possibly can about their studio and to angus's point earlier it's

not this is not about sort of processes or structure or anything like this is not about understanding your vision it's just in my mind I Define it as providing that value that the value that's really going to drive the studio forward or really going to make sure that we we pick up the early warning signs if things aren't going well I'm curious as well you know you're here for a reason is there a history at the at King and elsewhere it was like you said oh well you know this is horrible have to fix it this is what I'm going to do yeah so I gotta be careful you know but we too openly I won't have any clients left here's a cool story about how I think I screwed the entire gaming industry for for years um and this is one about me so I can tell it okay so the story as I tell it is is that um you know King Digital entertainment um you know makers of Candy Crush steaming towards uh an IPO all right I'm

the guy that's responsible for you know uh or primarily you know together with the finance team um to to help do that that for forecasting uh you know and making sure that we've you know we've got a forecast that we've communicated to you know our potential investors and and that that we're uh you know gonna hit those targets right um in the preceding few quarters we've been like percentage Point accuracy in in this so it's like okay we know what we're doing right I mean the story with King is that we predicted that it was going to be like a two billion dollar company um like at the beginning of the year and it ended up being like 1.95 like you know in terms of gross bookings right so so super accurate and we were pretty proud of of this process so when we went towards the this this this I IPO the first quarter out um you know we we missed out on our first earnings call we missed our um you know our Target's pretty

significantly the reason that that happened wasn't one of hey you know the game's going poorly the games teams have been doing poorly what had actually happened is that we started expanding into other countries with lower monetization fundamentally our our mix of users was changing on Candy Crush and that meant that that we didn't need the targets because we had a model which which basically looked at people you know it took into account that we were declining but not as severely as uh uh you know in terms of monetization and the consequences of that you know that won't what the billion of of a market cap right for what forecasting issue it meant after that that people were you know and this is a fundamentally new industry at the time right like free to play gaming oh it's a flash in the pan it'll never last well actually Candy Crush what is it 10 15 years later still generating the money that that it does there's no reason to be that nervous but that act of

not being on top of it not being you know not doing what you said it would created you know not the confidence of investors you know not the valuation for of companies for quite a long time um and so that's that's one example uh of you know at a high level of of getting it wrong right and that was you know something I got wrong right and so partly I guess the you know the ramp was like okay I'm never going to let that happen again what you're saying here is basically and Hong has said it as well right formula is not the problem process is not really the problem it's more about a framework having a single source of Truth having something you can rely on how do you approach that at ramp how do you make what a bad forecasting process is into a good forecasting outcome probably easier for me to describe what what I think good looks like right the first of all as we've said before it's bottom up it's

lever driven and in this case in gaming it should be cohort driven right there's no excuses we're one of the few Industries to have you know perfect visibility uh you know across all our customers so we know when they joined we know what their behavior is over time uh and you know co-work-based forecasting is you know clearly the most predictive way of doing this and to be fair a lot of companies you know make an attempt at this even if it's you know a poor one those levers they need to be uh levers that you use to optimize the business if they need to be the language that the games teams speak and everybody speaks within the business otherwise when things go wrong with your forecasts or your model or your financial models you don't really understand why right so you should be able to say at the end of the quarter or whatever when you're reviewing hey you know we missed because oh our our towers

down or we didn't have the amount of new installs that that we thought we would the model should be decomposable to levers which the games teams and management understand because if you don't do that how we're going to improve right so that's the first thing bottom up lever driven and those need to be relatively granular right so you should be able to decompose it into different customer segments that provide value right so the next thing is is you've always got to start with your best guess right we're trying to make an accurate forecast when that's first and foremost right what what is not great is when you start building in either aspiration or conservatism uh in that first pass early in into the process because at some doubt some place someone probably will but everyone kind of needs to agree on hey this is how we're tracking and this is where we're at in a

very bottom-up you know sort of way otherwise you know you don't have a basis for agreement on which which to build build on and when you do build in if you want to do there's nothing wrong with it right we're going to build a stretch Target to motivate your games team right this is where we want to be this yeah it's fine but make those assumptions explicit so when you miss right it wasn't just how you've done a bad job right it's like you missed the stretch Target but at least you know you hit the bass on we haven't done worse so maybe you just reach too high that's all but you know why yeah you know what um it's important that that the input assumptions are are owned by the games teams you know they're agreed upon they're owned by the games the games teams um I think it's better if it originates from Finance because more a neutral Arbiter probably with the skills probably just one step removed from actually working on on on the project itself so they're not you know looking

through it with rose-colored glasses they get to be constantly important to be refreshing you know because otherwise it's just a bit of paper that sits in a in someone's desk for a quarter right this is the budget we should be okay hard on it and and we should be refreshing it as possible so we can track our progress against it and and we should review it so that when we hit the quota we understand why we have missed and so we can improve as a company but we should also improve the model itself the finance team should be kpi on getting better at this because it's one of their most important jobs right the last point is okay we've got a new game right um where you don't have a long operating history you know we also are in a unique position where the the they're a great Market comps you know which you can anchor your your forecasts

in but you know you need to do it in a way which is reasonable you know like hey you know oh our game's gonna have the best retention the best I've ever installs you you need to assume that you'll win but you know you can't assume that you're the next Royal match or you know by by default and because otherwise you know that's how you know mistakes get made so so yeah it's kind of what a good for forecasting process to me looks like let's talk about how you do that in reality how you do that at space shaped games I mean to start off exposing the truth and the assumptions the biases how do you sort of rain all that in yeah so Angus has done a brilliant job of articulating why it's so important to have a single source of Truth we as a finance team we want to be innovative but we're a cost-based no one wants their Finance team spending any money like that's a reality so it's kind of hard to build a business case again like can we invest in tools on the team like that's where the last protocol in a gaming studio so um I think the first

bias is like Founders in management actually supporting this initiative and understanding that Finance needs certain tools that we're really going to drive things drive this forward we want to get good we want to get better like that that's always there that's fundamental um so I think here at spaceship the one thing that really helped me and my team is uh I call it like the third generation of games which on our CEO talks about third generation against where we just got more data driven so suddenly just with this lens it became easier for us to contemplate hey how about an automated forecasting tool so I think that's the first one Angus also talked about the inherent bias you're like I'm a finance guy so I'm naturally conservative um CEOs are naturally optimistic like people want to believe what that number may or may not be in their own mind and you get game team split wear conservative uh game teams and game leads want to like sandbag and

then aggressive game leads and game teams want to push it out there and they're really bullish um so I do like the point Angus made around sort of Finance being this objective Arbiter and I think that's that's kind of key I genuinely believe Finance should lead um at the same time there's this big collaboration amongst all the different teams because um as I said you've got management you've got UA teams you've got analytics you've got Finance you've got lots of people who are looking at data and forecasting and forming opinions with all their own different perspectives goals and objectives when you kind of mix this all in if there's no a lack of a single source of Truth when this all Cascades down in the team there's like huge confusion around what the actual number is and you know like um we experience and you know firsthand implementing this tool where you know three or four different teams having a different perspective on game margin is is a you know a fundamental fail and so

uh the one way we we overcame that is kind of quite simple is we realize the problem we accepted hey we need one single sort of Truth historically um that was contained by the data and analytics team but we thought well why not invest in an automated forecasting tool because it does all the heavy lifting for you it's one single sort of Truth and what happened is when we made that decision because this is based on mathematical modeling and there's a lot of nerds in a game Studio everyone color went okay yeah like math is good so um so I think on that premise once it was shared amongst the teams what we saw was the Cascade of you know when I talked about every team has their own goals and outcomes they just revise reshaped it around okay we have a single source of Truth and we can work around this and so I you know surprisingly it was kind of adopted quite quickly and it worked kind of quite well um and the one thing for me the True Value add uh is the fact that

the finance teams the analytics teams because we have smart people who can create their own models they were now unlocked to actually just do their job so look for the insights look at what the data is telling us and then inform the rest of the company to make sure we can make good commercial decisions based on all this information for us it kind of works because as I said we're really data driven we want to put it Central in all of our game development and for us having this tool ultimately has kind of helped us in the long run it's about accepting this as a team but there's also a benefit or something here to discuss Angus around yes a single source of truth but automation what does automation bring here what does automation allow first and foremost it's time saving but in terms of the benefit to the actual forecasting process itself right you know if you've got the person who is responsible for doing this forecasting and budgeting process they're spending

less time building updating and qaing modeling you know I know firsthand I was up you know this is my life you know I was in spreadsheet hell until 3am in the morning doing a presentation whether it's for potential investors you know or trying to secure funding or or so on and so forth the amount sheer amount of time that goes into doing that what that comes at the expense of is actually the most important bit which is actually questioning the assumptions which go into the model at the top of the segment on said your model is only as good as the things that go into it right um and it's absolutely true right so we need you know the process of questioning that sense checking it aligning it with the team getting them to own it that's the most important bit that's the bit where the the mess-ups happen right because usually it's like okay that's a model we're done right let's move on to the on to the next task

um so that's the benefit the quality of the forecasting improves because you're spending more time questioning the inputs uh I I that's that's what it's all about so we've discussed that you need a forecasting framework you need to understand the input assumptions you need to build from the bottom up and automation does help but hey you don't get there overnight Hong because you know it's trial and error what was that before you get to that that that Eureka moment that aha moment that says you know what this has to change yeah so uh Frameworks all are well and good uh but I think there has to be the lens of what good looks like and as a team do you really want to innovate in this space uh so if I had say I had like a Eureka moment it kind of I'll wind back the clock I want to work to a company called Newton uh it was an edtech company they did personalized learning so the concept was that when you're teaching a subject to a

kid based on their learning profile what's the next bit of content that will help them learn the previous one now that was completely powered by AI machine learning and predictive models so I think at a very early stage I believe Newton was ahead of its time I lonely the importance of mathematically modeling and how it can really enhance your your product solution so I think for me that was really the the aha moment um I think in its evolution in in working in gaming was our Founders started speaking to Angus and then suddenly there was this opportunity to kind of implement an automated forecasting tool with a recently globally launched game Chrome Valley customs and yeah it was it's it was a really uh collaborative effort and in terms of sort of what gave us most confidence and and what the best sort of outcomes were I think from this experience was we we validated both models so firstly our

models using the same assumptions kind of gave very similar output so that gave us more confidence as a team in what we did but it also gave us confidence that the tool was kind of pretty accurate um where we're sort of it goes beyond the framework um is the average Finance professional can't do regression analysis we can't project how like this forecast beyond the first 12 months generally with relatively you know relative accuracy so what the software is able to do is Project your business over a two three year period and suddenly as a company you can make performed decisions over a longer window so that's I think the I would say the second quite valuable sort of outcome and then the third one I kind of mentioned it already I call it human curation I used to work in music so I always think of curation um but basically it's what is the value

of a finance team it's it's not to as I said to prepare reports it's actually to create value so I always think about a finance team trying to find Value through insights through the data through you know big decisions from the studio so um and by having that automated system it just builds a lot of confidence now if I knew just to be objective as well it really requires good assumptions so the model in isolation doesn't do anything the technology in isolation doesn't do anything it requires really good input because it's mathematical modeling so it just creates an extra degree of confidence and certainty and accuracy around the around the models as well as save a lot of time as I said for this human creation for us to uh gain confidence in the model and then you know finally be able to accurately sort of forecast the business into the future Angus human curation at the same time Ai

and automation how does that come together in in ramp and how are you maybe building on that AI is having a profound impact you know AIML you know on on the core of the business of what we do you know in gaming content creation I'm sure that you've had podcasts about about this I I saw a I saw a a demo the other day completely no code of someone writing a game saying okay now move this now create a spaceship now make it move faster now increase the rate of fire absolutely mind-like just with with voice um um and I think that will extend to you know other areas more business related functions um of the business as well but I think what's fundamental is that you need to um you need to to to fully understand why the machines are making the

decisions that they're making otherwise you're not going to trust them it can't be a black box and that's why say ramp you know it's just an algorithmic framework right we use ml to have a first pass at some of the assumptions we round the edges in certain areas but at the end of the day inputs and their outputs should be very easily understood you know in terms of we pull this lever we get this out out the other end and as long as humans are working in the business that will always need to need to be be the case so for example you know at Ram we do do things like use ml to uh to simulate seasonality right to to examine the past data set and to say okay well we're stronger in the winter and then now in the summer or the thing is you understand that because it's visible to you there right and you get to eyeball that as a human and say that looks right to me right um so so look it it's going to have you know a profound impact on

some some of these things you know going going forward um there's no no doubt about that um the other thing about AI ml is like you know if you completely let the machines do the decisions then where's the sense of ownership right for teams it's an important process because people need to feel ownership that they're giving a been given a Target that is realistic achievable um and they're they're going to have the potential to smash it out of the park foreign this show is all about how to do your job better how to make an amazing game how to Market it and you have a say so if you have a story or know someone we need to shine a light on then we would love to hear from you we want to hear from you we want to reflect the reality of the mobile games market and all its wonderful complexity and strangeness so if you have any suggestions for us if you have any feedback for us you can always get in touch you can email us at

podcast pocketgamer.biz you can find us on Twitter at PG Biz and you can reach out to us through the pocketgamer.biz website if you're interested in listening to all of our podcasts you can find them at pocketgamer.biz forward slash podcast and we would love to hear your thoughts on future shows and we've got you covered on all the major platforms so subscribe to the audio podcast as Brian said look for us on YouTube if you want to read it hey you can do that too because we have a companion post for you as well on thepocketgamer.biz website tune in again for the next edition of the puppygamer.biz podcast and we look forward to speaking to you in the near future until then I'm Brian baglo I'm Peggy Ann saltz and that's a wrap until next week foreign