Building for Scale: How not to get Caught out at Global Launch, Teemu Haila, Metaplay.

[Music] [Applause] [Music] rolling right on into the next session um which is titled building for scale how not to get called out at Global launch and I'd like to invite to the stage Tim muila from meta play I know it's late in the evening but let's talk about scaling games and why that's hard couple of words first about myself and now who am I why am I here um let me actually roll back for a few you there

we go so first of all Tam nice to meet you uh co-founder Chief product officer at metap playay uh but I've been in the industry for about 15 years I think at this point uh mostly as a game lead game developer later founder all that some of the games that I've worked on that I can publicly talk about uh but most importantly mobile free to play all the way um I do a bunch of stuff my free time on games communities got a lifetime award from my GDA a couple years back but most importantly for this talk uh uh metap plan we are a technology company we do in-house back ends for top tier uh top Crossing games and that's where I get today's learnings that I want to share um so you and me let's found a game studio right now and being clever people as we are we probably think about time to Market a lot right we want to validate our whole business that's kind of the big point and kind of if we need to understand the context as well if you can justify today's business you need a lot of data right Discovery is and I'm

going to be crude absolute right at the moment you can't trust the stores you need to do your own marketing um and since ads are an auction place we need to be able to be uh paying more than our fellow competitors on the side and unfortunately they have established businesses with live games with very Deep Pockets so we need a lot of confidence to really enter that game and that's our dilemma right so our founding team needs to be amazingly good at solving this problem and that historically has meant let's find that hit game that's worth investing in and let's build a small team that's super focused on solving that problem so number one we're going to be focused on you know do the core game play we want to get to soft launch to get to data so we need analytics like let's build a thing that focuses on those three and basically do it as as ASAP as possible Right skip all the fluff and just get this out get the data and see if it's

worth investing in and this is good a good plan like this is how the past 10 years of companies have been built but there's a problem with it uh that's not all that obvious so when you build a team like this and my past companies are a very good example you have this highly performing unit that has just amazing productivity all the way to let's call it Global launch that point where you need to scale your company and you you know that experience when it just kind of fils out and not just fizel out but like your productivity just absolutely through the floor down into the basement and what that feels like is that the team that was shipping features left and right suddenly takes instead of weeks it takes months to do iterations on the game or like it just takes forever to get anything done even though everybody's kind of doing something all the time because you're kind of responding to situations rather than working in the situation and and like that moment is really curious to me why does it happen

and there's roughly four things like four major things that are kind of affecting here this here where number one the number of players goes up duh you start getting Millions tens of millions of players in that's complexity right there and at the same time um you need to keep developing the game to get to feature parity to get to the metrics that you need to be able to be paying for those ads so A Game's complexity explodes while at the same time to meet these things you're hiring like mad to be able to handle all of this so your operational complexity goes up and at the same time suddenly you're a revenue generating business so you have real customers spending real money and you have responsibilities out of that right all of these four things we need to be able to handle uh if any any any one of them fails then we can't scale our operation like we're physically unable to do it now each one of these is quite a big topic that I'd love to spend the

whole session on but I'm just going to like pick out the first two and let's go a little bit deeper and what we might be able to what does it look like and what might we be able to do about this so obviously I have a technology background so this the first point is kind of easiest to me like for example when you end up making this um I'm not going the Prototype but this first game that you push out for data when you hyperfocus on just doing that you can cut a lot of corners and it's a good idea to do so right like for example that is an actual high level graph of what um a backend system might look like for a simple game and it's it's brilliant it works until you pour a lot of players on it because then um like not all technology is built to handle scale for example when you have a lot of people coming in then the database is usually the first one to go um and incidentally they are also the most difficult technology to scale like in in human history it's a very difficult thing thing to do so we are

still smart we will be able to handle this situation given enough time but then our whole Tech team is focused on solving the scalability problem instead of for example building more features to the game so it takes our Focus away which is the main main problem here at the same time as remember that game complexity shooting up um the actual problem is that pre-launch again we have some things we need and everything else is kind of nice to have but the moment we start uh hitting that scale uh suddenly if we look at the middle column of a hit game it has things like localization over thear updates liveops tools a cheat proof economy these are not necessary for the first kind of retention numbers but they're absolutely critical today to make something that is able to sustain a player's interest for years so if you want to be competitive on those ads right then you probably need all of this stuff just to get your

LTV up so suddenly that simple thing that we did in the beginning is no longer it it's you can't do all of this and that's even before we look at the top Crossing box which is like well ahead in the future so now if we if we at this point we're going to stop and think about it we look at the feature list and then we get out of our napkins and start doodling right okay again taking the technology angle what do we need to build that is able to roughly fulfill this uh design spec okay it's not too bad and I I'll raise through it quickly if you're interested take a picture but it goes something like this we have lots of players coming in and there's millions and millions of them so we need load balancers and then those load balancers uh take players toward the game logic nodes that run the cheat proof economy on the servers and single player logic is very different to multiplayer logic so we need multiplayer nodes and somebody mentioned over their update so we need a CDN to deliver our

content and all of these systems need to talk to services like in a purchase validation and notifications so okay let's build that we need to persist it so databases and short them so that they can scale from the beginning and all of this is kind of a game server now uh let's call it a cluster and the moment I mention a cluster that means we need tools to manage the cluster itself we need tools to manage the managing of the cluster because this is how it works automation pipelines and so on and now we need to know what's happening inside the cluster that's called monitoring and we need to know what's happening with the game that's running in our cluster that's called logging and we need to know how the player experience is within the game that's called game analytics and did somebody mention live Ops so let's start building some admin dashboards and so on that's where I run out of slide space and all of that is absolutely doable we're smart we know

how this gets built and there's more to it there's always more to it backup security automation all that but the problem is that we're in the firefighting mode with a system that's very different objectively there's like there's no overlap between these two slides right so the transition from this sort of crisis mode into um a mode where we have what we need to build what we need that is a very very difficult thing to do and that's why operating Games Live is so so so slow and so the one message if anything my favorite quote in the industry is that hope is not a plan it never was like it doesn't work you can't just figure it out later that thinking about this right now is about a 100 times cheaper in real numbers than you know reacting to it and taking money months and months and months later so let's

right now us think about this a little bit there's roughly three things you could do immediately when you go back home um we could take a couple of hard discussions and maybe double check ourselves and maybe save a lot of time um number one consider now but Implement later you don't have to start coding as of this second we only need to think about that spec of what we need to do and then make sure that whatever we're building is able to accommodate the full spec once once that becomes relevant so we can later on adopt complex things and specifically what we've learned uh from our customers is that the most highest uh Roi you know the time well spent aspect of this is in your over-the-air update pipelines as a developer to me this was kind of surprising it's not obvious to me at all but basically what that means is that if you early on put in the time to figure out how you're

going to update your game live then that actually gives you huge amount of momentum as you're racing towards Global launch and then immediately afterwards that you get to tune your economy easily um and that's very a a game specific thing there's no magic bullet to that but that's like the if you are going to focus on one thing maybe make it that um and then the third thing is that generally speaking as we're starting to build this thing whatever Services you're using or how you're building it let's try to make it in a way that we can change our mind later right modular design and all that that if like I'll use a simple analogy but like let's make sure that you can switch your database later on let's not hardcode that in but like make it in a way that we can swap uh technical blocks in and out because you will need to be able to do that later uh so let's like make it easier for ourselves and and because all of this is quite handwavy I'm going to pick one example and go deep on it and I'll just actually random pick my favorite which

is the over theair data pipelines because that's something I happen to be working with quite a lot and this gets technical but bear with me so typically when you make a game you do something like I don't know I'm a programmer let's just hardcode everything into the game press play test it that's the fastest thing um and nobody does this at scale it's a very bad idea at scale of course because Distributing builds is slow uh it's not a design tool right only programmers know how to program that's not a thing you can give to a designer um and especially in a bigger team there's just no migration out of this like ask anybody who's had to implement localization to a game that hasn't been designed for localization like that that is a dark place in in a programmer's heart that migration and okay so obviously you want to do something else most people end up doing something like hey let's take something like Json files which is like

a text file Chuck those on a web server download it when the app starts and Tada you've got over thee updates which is good it works um and you know upsides it's technology is here since the '90s it's pretty good it's very easy to do but then there's subtle downsides that still make it completely unworkable that are maybe less obvious for example text based files very slow to parse on Old mobile devices to the point where in in real life we've seen games crash on the loading screen because it takes too long and the process has to be terminated to open a text file uh when it gets too big and these Json files and so on they're still difficult for non-technical people so like what was the point again to make it easier for designers to work with this stuff uh Mission failed and also it doesn't really answer any of the truly hard questions about this stuff which is versioning release management AB testing roll backs partial updates of the data

and so on and that would be okay but it's not like we live and die based off of our ability to grow the game and if we can't then like as somebody from supercell said why did you put in the work without putting in the work like Details Matter let's figure this out so okay what might uh sort of complete solution look like well for example at metap play we ended up doing something where um you essentially have a designer tool let's call it spreadsheets where you define your economy and then since you can't trust humans we have a step that basically optim downloads optimizes and validates all of the designs uh before it gets released then we archive and do like sort of release Management on the game server itself so that you can kind of fall back to previous working versions of the configs and so on and then we deliver automatic updates and keep them in sync on the server and the client and all that so that's sort of a system very good it's sort of what

you need at the end of the day high performance and we can like binary optimize the assets and and all that but the downside is that what what I just said kind of sounds like work I don't like work we don't have time for that right because we still operate on the same early uh problem space where we don't have any time we need to get to Market quickly I don't want to start setting up servers and build pipelines and so on like damn what the hell so if we think back to the T three live hacks the first one was think about it now implement it later so how about with this in mind we just do something where we stub out the game config upload download process uh and hardcode in a default so that means that we're back into the change code press play and just like Run the game as we wanted in the beginning but now when we eventually absolutely need to introduce the complexity we're now prepared to slot it

in instead of refactoring the whole system from the ground up right and now we have the flexibility do do it when we need to do it if we need to do it um without having to spend months and months in the beginning to do a setup so do now it's a future proof thing that doesn't take a lot of time and the point of this very concrete you know let's talk about the details uh example is just this like one idea that doing stuff in a future proof way doesn't mean that you have to take forever doing it right taking smart choices doesn't take a lot of time we just need to think about it for a while and of course then all of this it can be a little bit like negative and doom and gloom but I I really want to convey the idea that Victory is possible it is possible to make games and structure your team in a way that when you need to start scaling whether it's

on the ad side or like the app side of it uh we can kind of have the right Tools in place I have the right kind of team in place and the only thing we need to do is plan for it it doesn't happen by accident hope is not a plan when it comes to scaling games so in a very practical sense again those are my three takeaways that have nothing to do with technology or game genre you can do it hopefully whatever you're working on but of course if this is something that you're interested in good news we've made our technology free on the website to try source code and everything go download it uh we spent 5 years working on all of that stuff go and steal it uh and give me feedback I'm interested we also have a mixer on Wednesday you're welcome to come by and have a beer on us um and come talk to me and let me know what you're what you're working on I'm interested in that so thank you we might have a you know time for a question or two but otherwise

enjoy GDC and thanks for chopping stopping [Music] [Applause] by