Scalable servers for social city builders on a budget

[Music] [Music] all right so welcome everybody so I will be talking about our experiences building a social city-builder game and especially doing so on a fairly fairly small budget and on a team so my name is Patrick I'm one of the co-founders the CTO and as it happens to pretty much the whole back-end team of Ministry of games so trade town this is basically our latest game it just went live on Monday

into some technical testing so if you happen to be in the Philippines or in Finland on Android you can find in the App Store under the under the beta section just to tell a little bit about the history of this game how it came to be basically all started from this question of luck what if somebody made this really old classic from the 90s called the settlers into a mobile game and today and what that means a little bit in practice settlers was always a city builder game and connect crafting based resource management game so we tried to try to stay true to that but then at the same time we had already been a little bit burnt by a previous project where we I think you know where it may be a little bit too much and it wasn't that great of a success so we also wanted to get a fairly kinda proven core but as it happens there's a bunch of games like hey and since they build it that have a really nice engaging inner loop which then allowed us to kind of take a lot of

inspiration from those games make a unique mix of our own but then on top of that let us innovate more on the outer loops and especially so on they on this awesome sort of things so there we actually took a lot of cues from games like game of war so these MMO strategies where people really like gonna go into kills and cooperate into guilds coordinated with each other and and to achieve common goals so this is pretty much the kind of project wisdom that we set out to do and and I think what we cannot deliver done fairly well as well if you look at this from the package point of view there's a couple of these really like and a big issue items that we needed to tackle one is that there's this single persistent world so any action that you're doing the game basically changes the kind of state of this whole whole world and essentially every single axis also potentially observable by somebody else and we wanted all of that stuff to happen in real time so no matter if you

start producing stuff or if you hang out in the guild and chat with your friends we wanted everything to kinda also happen instantly and these put a lot of pressure on on how you can actually manufacture or how you need to build the backend services and then looking at from the connect game feature point of view we have a lot of social features in the game so it's both in frequency so a lot of stuff happen all the time in social so player the player interactions but also we have a huge variety of difference also accents so we have multiple types of helping you can visit other people there's chat there's all kinds of interesting stuff like guild puffs and and so forth so we're really trying to push the push the envelope on the social features and maybe as they kind of crown jewel of the game is this guild vs. guild competition and these weekly events that we do called expeditions it's in each one basically last week and we really wanted to capture this like on a combat style way

of playing from game of war but in in a peaceful setting so that it's more points raised very much inspired by our kind of classical board games as opposed to combat and then last but not least is displayer two-player trading so anybody in the game can put items on sale into the global economy and and everybody else can basically see them very similar the heydays or or since it is trading except that it again it's it's happening in real time so instead of the state being refreshed every minute or so we actually pushed in real time every couple of seconds new sales all the players who happen to be online [Applause] and then some more more technical requirements and so in order for a mobile game to be successful you need to be able to support a lot of players tens of thousands concurrent at the minimum basically and obviously in every engineer's bad dreams you can't imagine the case where you have the five or so million concurrence that the top book

games on the market have also you really don't want to be down the servers to be down it's a really big bummer for the players so the stuff needs to pick on the resilient to failure and you cannot really let the budget also run out of hand so you have to be a bit careful how many computers you you rent and how much database accesses you're doing and then this is kind of particular to us but we also wanted to make this with just one amount team so sounds sounds fairly ambitious so how would you actually go about doing that turns out the plan is not that super complicated so what we did is to identify the pieces that we really needed to build ourselves and stuff that you cannot really buy from the market and then basically just buy everything else as a service or otherwise and it turns out that there really wasn't at least at the time in the middle of her for doing such a such a kind of

complicated back-end so that that was something we needed to build ourselves but everything else they basically bought from from somebody else or buy as a service so this includes obviously being in the cloud in the first place you get all your load balancers computer rentals and that sort of stuff but also there's a huge amount of kind of in services that you can buy like logging and analytics and metrics and and whatnot and maybe as one kind of interesting case that has happened fairly recently is that you can boy basically boy data pages that are infinitely scalable and start at very reasonable prices so both Amazon and Google have have their own Latino sequel database that you can just access and these are typically one of the more kind of tricky wants to maintain and in our case we really did not want to be spending a lot of time on kind of operating the the machinery but rather spend as much time developing the game itself so adding new features and making the game better and

then they're gonna preach these two worlds you still have to run your game server that you made somewhere so there's no arrays new tools called containers and an orchestration of them that we are we are heavily using nowadays so those really helped us deploy the servers into into the into the cloud are reasonably and efficiently so I'm not gonna go too deep into the actual game server itself I gave a talk about the more detail talk about that in this year's elixir conference so if you're interested to look it up or or you can come talk to me I really love talking about this stuff at the end of the world but just to highlight a couple of key features so everything in the server's is basically stateful so we keep all the out state all the player States all the guild states everything that we need to run the game in the memory all the time so we basically use database as just this kind of backing store where we you know

if something is inactive for a long enough time it's just kind of serialized down there and then bring it back when we need it again and this allows us to do a very complicated operations because those those objects can be smart and they can communicate with other which extends so forth another interesting thing is that all the clients while while you're playing the game they stay connected all the time this basically allows us to do real-time stuff so that you know when something happens on the server that the client needs to know about we can immediately send em a message are supposed to waiting for maybe the next next poll cycle to happen and this really gives us the ability to do stuff in in like the order of a couple of hundred milliseconds from from when you do send a chat message for me to meet your associate in game and also the deservers they're fully distributed so we can run them on one machine or we can run them on 20 machines to the outside world it pretty much looks like that there's just a single box in there doing this stuff so

we don't do any arbitrary bucketing of you know these players are stay on these servers and those players stay on that server and they cannot really talk not not what we do we everybody is going in the same pocket and every card party can connect to everybody else one of Barry's why why we were even able to do this with such a small team I think is because of this new programming language called elixir it's a its roots go back all the way to do like the telecom industry in the 70s there's this language called Erlang or Ericsson language I guess short for that they used to build basically these telecoms boxes for and they really were valuing like high availability and and so forth an elixir basically compiles down into the same virtual machine so you get all the power of that like on a decades-old battle-tested infrastructure but obviously as a downside of kind of being invented in the 70s the Erlang language

itself is fairly archaic so elixir kind of modern noises all of that and makes makes using it and onboarding into it very easy just some of the highlights of the language so it's a functional language meaning that you can get a lot done in a very short amount of time it does a very good job at distribution and clustering meaning again that you can scale and do all these real-time social stuff without spending a huge amount of time building your own own communication layers and so forth it is fault-tolerant so this may be the most interesting thing about the whole whole Erlang ecosystem and maybe unique thing about it I won't go into too much detail about it but it really allows us to in in practice kind of keep our servers up even when there's trouble happening most often to the particular failures they're isolated so that if you know it some players watching course state there's some kind of problem maybe that you know player is not able to play the game for a while and then until somebody fixes

that game but almost never it propagates to do elsewhere it is the virtual machine is fully or natively actor based meaning that every actor or players dad or so forth has their own own heap meaning that there are no dislike long stop the world garbage collects and pauses that tend to happen with many other languages like I don't know Java for instance so we don't need to worry about that at all and then it can do stateful stuff very easily so we can keep the player States in memory it's very very natural for that system and that allows us to do not go in the database almost ever so we only periodically basically say if the player states there just to make sure that if things crash that we don't lose a lot of state or or maybe if somebody has done something like an in-app purchase we see realized that in the database so that we don't lose it and there's been like some I think

dislike ecosystem is being rediscovered in the past years so for instance there's really no success stories like what's up being done lately I think they used Erlang there's this court using elixir working with very small team even gaming space I think machines owned is using Erlang for four day km of war architecture and so how do we actually get this stuff to then run on the cloud this used to be like a very really painful process at least for me I'm really like a development type okay I like writing code but I'm not really good at maintaining machines in the cloud so I I never liked that part but nowadays there's luckily in office I think a pretty good solution for that namely containers and an orchestration of containers so we actually run everything all the game servers in in Google's cloud on their hosted kubernetes and it really kind of turns the thing kind of upside down instead of us renting a

machine on that from them and then maybe installing some software we we just tell Google call that ok run you know seven copies of this piece of software in a cluster somewhere and then it kind of figures out by itself that ok seven copies means that there needs to be three machines and then it will it will actually deploy those three machines and install the software there whatever requirements or otherwise needed so it also puts them there like all the log forwarding and and so forth and that's the kind of repercussion of this you also get lock basically scaling for free so you just tell the system that ok I need one more so let's go up to eight and then up to nine and it will automatically figure out again like when you need new machines it will it will also provision them for you other benefits that we really got compared to at least our old way of doing things is that the deployments themselves are really fast nowadays so five minutes from from a comet that I can actually

see the change live and usually the update downtime is somewhere in the order of 10 or 15 seconds that the service is down for four people also the docker system like because it basically is very self-contained when you build a talker container out of your server it has all the dependencies that it needs from the system so there's no more maintaining of these virtual machines that you know making sure that everything has just the same set of and same versions of applicators and dependencies installed but instead they all come come in in a one pack so that it's easy to make sure that everything is kind of running the same thing and also because of the very how it operates most of the failures that usually occur in a system like this it can actually sell of hail so if your note goes down it still knows that I'm supposed to be running eight of these servers it figures out oh but I don't have enough machines so it will provision another

machine automatically install your servers there and just keep keep running them obviously you need to handle those those kind of are situations that decree yourself but but it really helps out a lot so just to summarize identify what you need to build yourself do it and then buy everything possible from from somewhere else it always is cheaper to Boyd and under reinvent the wheel and try to focus as much of your time on on developing the application and minimized its on your stand on operating a database or or doing other craft work like that that really doesn't add value to your players [Applause] thank you that's everything I'm old school I remember when I was back in like 98 and we were basically none of this stuff existed and we were we're actually using laptop motherboards in

order to make what was probably the first rack system when I was a British Telecom to run great games and when you think about the journey we've come from since not that long ago if you think about it you know yeah it's not that long ago where we are now we're like you say you've got these server farms which you can sit there and just spin up on demand that you can define and self-heal all of this stuff is incredibly important and if you're not thinking about the server can state constraint in your game you can't deliver services without I mean it would you agree you can't deliver service experiences without using that kind of technology yeah it is it is super tricky in the sense that obviously you you have to do the before hunt otherwise if you really get those players you know you're gonna be beating yourself over it that you know oh we can't scale and then you basically lose the whole case but also otherwise like if you don't get those players then you spend a whole lot of time just like I know architecting and implementing it so

it's it's yeah I think they're layering cuz yeah she'd the Langley showed is very similar to what we had in the old wire play days and they all kind of dial-up modem days it's not that dissimilar in terms of kind of core concept but the actual kind of components inside that are so much profound ly different in terms of their stability and they structured I might say the kinds of languages you use can affect that too that's why I find that I'm fascinated with we kind of having the same conversation that we had when we talked about platforms that long ago and yet it's becoming even more you know important I mean in some of the projects I'm using I'm using tech in like game sparks or chili connect or play fab you know off the shelf tools because I'm basically designing projects which are about what can I do within these constraints with these low budgets but I love the fact you were able to that she say that Hana will use what we can but are these things we can't do without making it ourselves how did you make

that cool of what what to make and what not to make so basically we actually had like a previous projection that we had already built some of this technology for so we're gonna had it starting place there and obviously it did affect someone more when we were deciding on this project what which should be built and so there were some things that and ideas that you know would have been very very difficult to kind of map onto that existing architecture and we're gonna have to discard those for that reason and neither they have been doable with kind of the self tooling I mean obviously if you can use of the soft rolling you know you really should go for it because building this stuff is it still like man years of effort oh it's non-trivial as I mean I'm the coder but I've spent so much of my career working with server coders as well as you know game developers and it is like bashing your head against a brick wall sometimes most of the time it's you know often in my ex about yourself it's a small doubt it's like this small difference between this will run smoothly if we build it

ourselves versus this would be easier to implement if we change the way we work and it's so difficult to make I find it so difficult sometimes to make that call and it generally comes down to who's the guy making it it is very true and like that's I guess that's why making games is in in part so very difficult that it's not just the server and the design but it's also the Orton that I saw in and you know everything under design the graphics cause and graphics code and the client code and everything gotta needs to be coordinated some more that you try to figure out things that are smart to do with you know with the team that you have and with the skills that everybody in the team has and so for then it's it's really really difficult navigating all of that not not just on the server side but obviously also on there thank you very much thank you very all right thank just lastly I really want to block my colleague or creative director we'll be giving a talk in force a conference later this week about the kind of more

about the design and the product stuff off of trade town so if that interests you do do check out his talk I'm sure it'll be interesting great thank you watch all right thank you [Music] [Music]