Scaling to 10 Concurrent Users: Online Infrastructure as an Indie

[Music] hello I'm drew Eric and welcome to GGC 2020 in my living room we're gonna be talking about building online services for independent games today and first I'd like to start with just talking about what that is why you might need that and whether or not you should keep watching this video or stream so why would a independent games need online services there are in fact I should a lot of great options out there right now for dependent games to start working with play 5 in game Sparks and lots of things that offer a ton of functionality but at the end of the day there tend to be things that are unique to your game and things that you might want to offer yourself or things that you want to take control of so you can do them differently one of the common ones is people want to have their own at matchmaking algorithms are things like that any rate I've had a lot of experience throughout my career working on online services or from the

ground up in independent games with the release of project called the maestro's which I'll talk about today and will kind of follow that games journey from a student project to a hobby project and some of the things that we've learned along the way because I interestingly had a wonderful gift of hindsight as I was going through that process and working in triple-a studios working at Riot Games building the same kind of online services but at a much larger scale and I took a lot of those ideas and a lot of thinking from triple-a and I applied it to my impending am sometimes successfully sometimes not successfully I want to give you some of the things that I learned of that process so if you go to implement some of these things yourself you can do better than I did so yeah this is just a little bit of my timeline I was started working on micros in pre-production in 2012 I started working at Riot Games in 2013-2014 timeframe and spent about five

years there I shipped maestro's about two years ago we're and a half ago now and onto steam and early access and today I run my own work for hire and consulting shop doing online services for independent games so you know companies that need back in services as a platform for their games me and my team come in and build those things so I've done to see this on a number of different companies add a variety of skills and see how it's done so under the project we're talking about the maestro's was a hybrid realtime strategy game and MOBA and we had these great aspirations for it I mentioned briefly that I was actually a student project this was sort of my senior project my capstone in college at University of Southern California and we had a huge aspirations we've been off way more than we could chew we learned a tremendous amount so we were going to build the 3vs3 online multiplayer game we're gonna run with dedicated servers and it was going to be built in Unreal 3

which was the Unreal developer's kit because at the time Unreal 4 was not announced yet I think maybe nine months into our development or something like that after we've gotten a huge amount of done they announced Unreal 3 so the next thing was sort of the composition of the team the size of the team we were pretty large for an independent game we were all working part-time for hobbyists and stuff but there were 15 engineers on the project but only really two people at a time at any time we're working on the platform itself so pretty small group there and I think one or two people was probably what you'll have working on your back and services any given time especially when we were committing you know 10 to 20 hours a week and eventually after we graduated and turn it into a hobby project the team got a lot smaller sort of a unique circumstance something to be aware of us and talking but Steve got a lot smaller we still had one to two people working on the platform at a given time though so afford this game online multiplayer

we had a lot of features we needed to support we knew that we would need a login and a way for logging in and storing accounts we knew that we would need lobbies and folks chat with each other and we'd have to manage that a gated servers because we had kind of bought into that as a better user experience and we needed a lot of other things it turns out there's a lot that needs to be provided when you're building what amounts to a gain in the service anything that's going to be online there's quite a lot of features kind of under the fold or under covers when you go to working on this let's rewind to the timeframe we were at and when I had a fantastic haircut this is my business partner Sean working that project with me and still works with me today both of us had you know much better hair at the time but back in 2013 more than just funny pictures the circumstances from building services were really different the first bullet will not surprise you unity was still really hot five years ago five seven years ago gosh seven

years ago unreal wasn't out yet clay fab and game sparks were I believe wolf present but not in a big way and their feature sets were much much smaller I wasn't even aware of them at the time steam greenlight wasn't available so you couldn't really use steams systems if you weren't approved to publish on the platform and eto which is another distribution platform wasn't around in its current state it wasn't it didn't have a lot of downloading capabilities and patching capabilities than it has now great platform for that amount in the services side nodejs was getting really hot Google and Microsoft had started to make their plays into the cloud space and we there was no server list there was no kubernetes so we're still dealing with some pretty basic primitives in the infrastructure space like virtual machines you know basic offerings from ec2 things like that until that brought us to you know we're looking at that space and this is also

we're looking at in terms of what languages we were trying to decide on there were two or three of us in the room saying okay how are we gonna build this platform these are some of the things we evaluated PHP and Java are kind of old standbys and Java specifically is used by quite a few Triple A studios and enterprise software organizations for building these kinds of systems lots of great tools in place very verbose rails is extremely fast it was really really popular around that time for building web applications really quickly and we follow of course very much in the realm of web applications nodejs was new kid on the block and it was squarely targeted at this space but I had a lot of things that got us really excited so we ended up choosing node and the reasons are the reason that we chose node were that a we were really excited about it it was sort of a primary one where students we're

still trying to learn new things and this looked really exciting it seemed to solve a lot of things that we were unfamiliar with at the time and it had actually some really good frameworks even in its early days so question is is node in India approved choice would I recommend this to other people having gone and built services again and worked with other independent studios the answer is actually a resounding yes I really love working with node when I'm on small teams when I have the type of constraints and time and time concerns that you have as an independent studio I've recommended it to multiple people since then and they definitely I actually just got a text the other day from someone who had recommend this to a sort of a one-man programming shop and he was like 24 hours after I told him to go go check out node to solve some of your problems with the lobby system he 24 hours looked that later he came back he was like that was awesome thank you so much for that I think this is gonna you know get me all the way to Washington they end up launching shortly thereafter and

it was really some of the things we're excited about some of things we didn't think about so we were quite excited about Express and does turn out to be a really great really easy to work with web framework with lots and lots of sort of plugins and middleware that you could use the biggest thing that I didn't know at the time that I know now is that no sort of solves is a strong word but it definitely targets the hard problems in online services to home so if you're building a platform that consists of matchmakers and account systems and all these different things multi-threading and asynchronous i/o you know on the network or against the disk or to a database those are the hard hard problems and node acts as a single threaded system it's event based so you're you feel like you're writing more single threaded code which was really really important for us to just make progress quickly and not have to solve really strange edge cases obviously the Devils

in the details if you try and go deep on certain things but we found this to be really really useful and again we were just excited about it so we've worked harder motivation cannot be underplayed from you're working on independent games and anyone who's done it for any length of time knows that's a really important thing to consider as well so know definitely India proved there were some obstacles of course it was very very nascent at the time I generally don't recommend choosing a pre 1.0 software for independent games development because you're gonna be on the forefront of fixing other people's bugs and that's generally not your primary concern in fact building a game is your primary concern and the platform needs to facilitate that all of your online services are just meant to facilitate that so if you can remove hard problems from that space I highly recommend it especially as an independent tool may be a personal taste thing but as any code base grows I find that static typing is extremely helpful

eventually we introduce typescript to know jess and that helped us the click base and maintain it over time so that's like the big decision and I know a lot of people have a lot of spend a lot of calories on this and ask that question first there's actually a number of great choices and pretty much everything I put up there is good for that but once we have that done we were super stoked and we're like let's get to work and so we had our HP framework with Express and our language with node and we just started hacking first thing we needed to do was login as most systems do using the password so we said ok we've got an extra HP framework let's make any call and in fact it's going to send a password up so let's be secure let's make an HTTP call which is super easy to do with Express so we set that up I made a call with the username and password in there awesome cruising step 2 we need to host a game

and so let's make another HTTP call we'll post up the you the name of the game that we want to create I'm making south east game awesome and people are gonna join in it I'm gonna you know add people to it or have you and then I need to change what map I have so I'm gonna came to the map Terra Oh huh how can I do that with an HTTP call I can send up the map terra but then how are you all of my buddies in the lobby get notified about it it's actually not a trivial question in HP because HP is unidirectional you send up a request and you get a response back that makes it a little bit more complex for how you would send an update or a notification that sort of out-of-band that no one had requested anything for or at least that the particular user didn't request something for and so posting gets an HP are not enough

again it's unidirectional one thing you could do is you can start pulling for updates and say every five seconds I'm going to ask the server if anything changed in the lobby so let me go o-obviously this is kind of inefficient and you don't get your updates for maybe up to five seconds after something happened and overall not an ideal pattern it absolutely can work and I've seen it scale but I would not recommend it I don't think it represents the domain super well for what you're trying to do the other thing that's a little bit more subtle and and over time I've experienced this you don't have this idea of a session so you don't really know if someone is present because you're making HTTP calls and you open a connection make a request response and close a connection in between making two requests you actually don't know if a player is still present so in order to keep them out of the lobby you don't have sort of an event you again need to rely on some sort of pulling them so some sort of heartbeat to note that they're alive there's no native session you're sort of

implementing a session on top of HTTP so okay again knowing when to kick somebody out knowing when they've left are harder to know someone you know turn off the power on their computer you actually don't know it - your timeout happens and that time on something you idiot let yourself so you need another protocol or you need to you know work with it should be polling and implement sessions on top of it and there's a lot of options out there especially today so you could go to polling route you could use TCP which is a native reliable socket connection it's been around since the inception of the internet you could use UDP like the rest of your game is implemented in there's no session but it's known to be really performant etc WebSockets are really cool and this is sort of a newfangled thing we can upgrade an HP connection to be an open socket that you can have bi-directional communication over and G RPC is similar to that except for it operates on newer - and this is a

part of buffers and and remote procedure really any of these consultants wrong we were using HTTP and so we definitely thought about HB polling but in order to get that out-of-band stuff we actually decided to use TCP it was just easier to implement in unreal 3 at that time and of course we're still using HP so we're basically making calls up to the service and HP and receiving messages back over TCP primarily so is that an indie approved path we certainly brimful to do it and ship with it but I wouldn't recommend it it sort of poorly represents the domain that we were in it poorly addresses the problem so if that's not a good way to do it what is this is such an important question because the process that we went through I've actually seen multiple studios sort of repeat where they do the obvious thing and they start making HP calls but then they realize later that they really wanted a session and they really need to

notify people out-of-band so having a socket open really important again how should we do it then what should we choose I think you need to look at our criteria we know that we want a bi-directional communication over ideally a single socket connection because we shouldn't need more than that we want something that's going to easy to set up it's gonna have you know native implementations and a lot of our languages and it's going to have support for a session the UDP for instance not as good of a option because it doesn't have native sessions and we don't really want to worry about the wheel on that time is of the essence as an independent so I would actually recommend any of these three I think there are perfectly great options to do this TCP sockets being sort of the most low-level perhaps you should consider gear up you see your web sockets first excuse me but any of those three things meets those criteria and I would recommend them

so once we've decided on our protocol for communicating and and over what pipe were communicating the question of what to put we put in our messages and especially what are we put in messages when we're in a lobby so we're just talking about the create game flow and then I update a map what should that look like what should I send the server in what should I receive back there tend to be a couple of schools of thought on this you can put in just the changes so if I say update the map Tutera it could tell all of the different players that are there it's like there's a new map it's Tara it also send on the entire state of the lobby and be like this this is all the players that are in it the map is Tara and then they could just diff sort of look at the Delta between the two payloads so or you could do a hybrid of the two but let's look at only the changes so I've set up a message and this is the message that's coming back

down to all the players it just has an action it's a somebody updated the map the new map is terrible and so on your client you flip the image of the map and you know change the name that seems appealing it's it's certainly a lot less data going back and forth and if you've got a reliable connection it seems like this should work really well the opposite would be to send the entire state at once you make a request up to change the map and it tells everyone the entire state of the lobby all at once and they're just looking for the Delta so obviously more data in there you have this thing where you need to look at what changed from one payload to the next so the question is which what should we put in each I actually sort of counter intuitively and especially for independent developers recommend sending down the entire state of the lobby and this go a bunch of different things it could be a poppy it could be there's a whole

bunch of different things but anything that's not going to grow exponentially in terms of his size it's gonna have a pretty fixed state for its size I actually recommend sending down the whole state in platformer land don't do this in your game for sure but in platformer sending down just the state they're sending on the entire state is great it removes a lot of edge cases and if you have unreliable connections missing a message means that you need this whole protocol to implement to send the entire state anyways so it turns out that implementing just sending the deltas is actually implementing that plus it's been implementing and sending the entire state and again time is of the essence so the difference in size it's not going to be measurable under bottom line yeah this is the approved thing it's sending the entire state cool we implemented some basic features and then we kind of took a step back and said all right what should our architecture overall be for this we've

been writing code in a node.js server but how should we implement all of these features that we're gonna need and you saw that large list well there are many different schools of thought on this right now or different approaches the old-school way is known as the monolith and you just have ones or a large code base I've seen these get up to a million or a couple million lines of code and all of your functionality lives in one code base you tend to need to have multiple of them if you want to scale or you're deploying multiple of the same piece of code alternatively microservices got really popular around the time that we're working on this and that is the mechanism by where you have maybe a dozen different projects and each project is tackling a really specific feature set so you've got one project for a player account and you've got another project for matchmaking some of the properties of this that people like or that it can scale because you're most

intensely used project can be separated from all the other one so your performance in your lobbies could be really good if all of the CPU is really just being used on your matchmaker and then their server lists can you use lambdas or some other mechanism for executing code remotely without having to manage all the operations of servers is really really appealing I can only tell you what we've worked with and sort of evolution that I saw this is where we started with the maestro's there's it looks like there's a lot here and platforms tend to kind of grow large in and the number of different pieces of technology that they're using and pulling so I'm gonna line it talking to our mega plat where we started that one nodejs project that nodejs project serves up a webpage that we can use for administrating and making changes it talks to our database it talks to a cache we did have a separate project for our chat server because we realized that also needed me

on the TCP connection prior to learning that we would need our main name to be on TCP connection and the game servers themselves had another little note just executable to manage starting and stopping that you decayed or the unreal process on them that had to live on the same boxes so we had three projects at this point and our client taught me that a couple other pieces of technology that were deployed as sort of supporting systems and we looked at this and we said okay what are the problems with architecture let's do a little review well we know we don't have enough redundancy there's one server that kind of underpins the whole thing that does the lobby's and the logging in a whole bunch of different things so there's not a lot of redundancy if we lose that box if we were on a show at the time if Microsoft Azure decides to turn off that box our game is down that's not good our game server management was kind of buggy and so we wanted to change that

around the deployment system we were using was copy paste which was not very robust and we tend to have errors in that sometimes and it probably won't scale not the biggest concern as an independent game but you would love to know that given you were having scale there was some way that you could just throw money at the problem to scale in this case that would have just been increasing the size of that one box but even more specifically with no chance we would actually just need to be increasing the performance of a single core of the CPU or single thread performance so it doesn't scale super well and that's the thing that we're thinking about our solution because we're so infatuated with micro services and I was learning all about them in triple-a land was to apply a micro services philosophy to the platform that we had and what we ended up with was our play that's and Clara counselor Lobby has turned into three separate services

three separate projects there were deployed behind load balancers so we have multiple of each and that gave us that redundancy and yeah I mean we had this whole newfangled deployment in microservices we didn't get all the way through so our chat system was not multi serviced it was not turned into micro service but we implemented this whole thing and so the question is was this solution or not again hindsight being 20/20 I think that we did encounter a lot of unknown effort underneath the covers of deciding we're going to get micro services ecosystem it actually ended up costing me months of tooling especially in the deployment side we're copying folders felt bad with a single service but worked it was just impossible when I needed to deploy I think we ended up with seven micro

services on probably you know seven minor services across multiple different boxes you're talking about maybe 14 to 22 poise not ideal and then all of our projects had to look identical in order to be deployed the same way operated the same way we needed a lot of tooling to monitor sharing code was a lot less easy when it was actually shrinks or libraries or copy-pasting files across and much more difficult than just reusing a class for instance and it made the code a lot more complicated in a lot of ways because where we used to have a method call between two classes just calling function now we might have to do call over HP a really specific use case for this was when we separated player accounts and player stats it felt like a really great idea at a time and I think for large-scale games it's really important to do that for an independent game you

incur a specific cost so creating an account means that you're gonna have to create an a a row in your database or a document in MongoDB which we were using for the account and email etc and then you're going to have to tell player stat service as well to create statistics for this player saying that they've played zero games they're at level zero with zero experience etc and now you have this multi-hop asynchronous situation where the player calls up to the server and that's where it come to be created player accounts calls from its database and says making a record then player accounts calls to the player stat service who says go make some new stats and then the stat service let's go back and take a look at this then the stat service calls to his database and says make a row for their stats and then that returns and then that returns and then that attorneys so we've just introduced something like five to six hops just to create a new account the problem comes that give a failure in any one of those

links you have to walk back each one of those creations one at a time or else you'll end up in sort of a corrupt state with half of a player created so lots of code complications and where does the data live and who owns it lots of interesting questions they're hard problems to solve that we didn't know going into it so which of these many different paths are India approved I certainly felt like I put a lot of effort into that but the system we ended up with was really a big ecosystem of work in which we solved some of the issues so the answer is I'm not sure and in fact for your particular case I would recommend you go and read some literature on this listen to some smart people with me Martin Fowler wrote an entire thing about micro services premium which this article talks about the cost of implementing micro services and that sort of things that we

experienced naively sort of charging into this space and Paul Hammond says has an interesting article on cookie cutter scaling which addresses this notion of well what if we were to build a monolith and replicate it so that we have you know three of these monoliths so that we can scale horizontally by adding more boxes what are the costs associated with that how does that compare to micro services or even a single server things like that definitely be mindful of your time and know what you're getting into so having talked about all of this I think that there are some really key takeaways here one is that as you'll probably learn or maybe you already have your most limited resources your time it feels infinite but for every day that you're working on your game especially if you're not doing it if you're doing it as your primary job you know you've got to pay you rent you've got to pay for your food and time

is also important in terms of market timing so things that take a long time are things you should second-guess as an independent developer and it's hard to be mindful in a space that you're not super aware of but hopefully this is highlighted some things that can be black hole for your time motivation is is sort of equally important because a lot of times the amount of time that you spend is really determined not just by how much time you have but also how you are by the project and by the ideas so choosing no dress for us was a great motivator and a lot of to learn new things and get excited we made mistakes along the way but I would say the train office in India is that you get a little bit more your players hopefully are a little bit more relaxed about your bugs and are more just excited about the ideas that you're excited about constant you trying to get across the other one is sort of to be where the hype train there are definitely dragons in the hype train the hype train for us have anything micro-services I think and specifically about being aware of the

hype train is as an independent developer you probably have some fantastic game mechanics that you want to be on the bleeding edge of but you can only be on the bleeding edge of so many things you probably don't want to be the on the bleeding edge of online services as an independent game because it's gonna cost you a lot of time it's not going to give you the value that it might give a triple-a studio or a publisher where they're getting cost savings across tens of thousands of players or millions of players and multiple games and things like that so your your goal should be to release your game and so beware the hype train because there are unlock and so on unsolved problems excuse me unsolved problems in new technologies and it's probably not your best use of your time to be solving those problems unless that's your Jim cool but thank you for your time