Race Kings port-mortem

[Music] cool I thanks guys Salah name is Sean I'm a developer and support engineer for gang sparks some of you might have heard about games parks where a back end is a service platform I'm going to talk a little bit about that later but I'm attacked I so this is sort of a light Tech Talk sort of a retrospective of a feature that we added last year for real-time multiplayer networking it came about by a collaboration with hutch games who are one of our customers at the time and it sort of just a journey on the decisions that we made and why we made them and what problems are with developing real-time multiplayer networks for mobile games so I am a tech guy but the sales and marketing people for game sparks would be really annoyed at me if I didn't do a little bit of a sales pitch and so to explain to you very briefly and what game spark is the

best we do sort of anything that you can think that would go on to the cloud part of your game services we cover that so moving things like social authentication and the social integration so for authentication would be your google play Apple Twitter Facebook that kind of thing but will also do third-party integrations for payments like the app store and the Play Store and also things like Amazon this would also include enough purchases and receipt validation and virtual goods so the social is a core part of the platform so we also do things that you'd expect from a social game services like leaderboards chats achievements and push notifications we have a big suite of analytics at the moment the analytics is kind of like a google analytics thing so it's key events tau retention arc tau that kind of thing but we are unveiling a new more flexible suite of analytics tools which is based on feedback we've had from our customers to basically do whatever weird

analytics means they have and reporting and exporting and so on the developer side of things we have a couple cool features like cloud data and downloadable content on the blog we also have this concept of cloud code which allows you to create custom events that some JavaScript allows you to create custom requests that come from the client to the server you send whatever data you want to customize it however you want and then you run some JavaScript and that JavaScript could be reading and writing to the DB just running some maps on whatever you need or passing out to another web service to get something and then forming the request and sending it back to the client and so what I'm talking about today is the multiplayer aspect that we have so the company's been running for about four years now we've got 170 million players ten thousand registered developers 600 live games seven pies and games and

development and a tarmac last year sore are big sort of a server capacity thing that happened last year is a one of our games hits over half a million concurrent users at one time and that was in the space of about two or three days after launch and so we we have games published by some some big studios like big season publishers like telltale bandai namco Square Enix rovio Amazon gang Studios Capcom and Ubisoft and we have a bunch of other games on there as well and so how the real time services started off was coach games had already made a game with us in 2015 call MMX racing and they used basically the core part of our services which is virtual goes leaderboards and social integration that's what most of our customers use from gang sparks or they mix it match parts of that they approached us asked us to develop a real-time multiplayer network and they weren't the first to approach us but we had a good working relations with a relationship with MMX

and it was a good opportunity for us to not just develop these services and then release them and hope that they worked we were able to get a company that we knew we could trust to work with they were happy to work with us to develop these services to their needs and iterate it through it so that we can develop as tested with this game and then say now we have great sis and let's release it to the rest of our coast and so what the next part of this talk is about is just sort of what what them what challenges there are four for developing real time multiple network services for mobile games so the example that hutch needed was really simple it's a head-to-head racing game and they really only need three things to be sent which is the position of the players in the track the speed of each player's the drift angle and then they needed the ability to intercept packets to provide cheat detection and that was that was pretty much all they needed the biggest problem with developing real-time networks from mobile games is the fact

that there are huge bandwidth limitations you don't have the same speed that you have with a broadband network and the coverage is very unreliable that's a 3g coverage map for I think Vodafone and it looks like this is a pretty expensive amount of coverage right but the problem is is that the mobile providers do not provide you with when they say to provide you with 3g and that only means that they're providing me 3g when when when they want to so there's no limitations for them to decide we need more bandwidth somewhere else and now you have 2g the other problem is that mobile games are played when people are traveling and you're never guaranteed that you're traveling in a 3g network area for the duration of your journey and that causes huge problems that are not the same as network in networks that are used for multiplayer games consoles RPC if you have a craft connection when you start off you're going to have a crap connection halfway to your game that's not the same for mobile so the biggest

problem is the latency issues caused by these unreliable networks and this is not something that Gangstar can solve so what we focused on was what do we have control over and how can we get the fast Network possible and so I don't want to get too much into this or a technical background of network design but something that's always interesting when you're talking by making a network and making those decisions is what transport protocols you use you use TCP or UDP and these are so basically TCP is though is is the internet paradigm for for how you send packets I sent data across the network and how teachers he works is you the sender computer sends a message to recipients saying I want to send this data this is the amount of data and this is the order is going to come in us and then the recipient computer says the message back to the sender saying I'm okay send me the data so this is really good because the all the packets all the

messages are our sequence so they arrive even if they arrived out of order they can be rejoined in the right order and the the recipient computers happy because the recipient knows what you're sending it can correct for errors or if packets are lost in the network because send them over the Internet and the internet is not always reliable if a packet is lost and the recipient computer is waiting for it and never comes it can send a message back to the descender and say I think you were supposed to send me this please send it back so in contrast the UDP aud has no money of this so UDP is effectively I want to send this message to this IP address and this Porsche go and then it doesn't care if you get it so the obvious thing for most people if they would encounter these networking paradigms is you use TCP because why would you ever use UDP and the actuality is that nobody really uses TCAP for game networks and the big a big reason for

this is because of the extra stuff the TCP has when you're transmitting the data it's just bigger the the packets that you're sending are bigger so UDP is a lot smaller to you about sixty percent smaller and PCP also does not give you full control over when you send us whereas when you want to send a unity practice it sent so depends on your Bruder and stuff but it's generally a sense and so most networks use UDP the problem with UDP is that what you needed UDP the hutch needed a network for a racing game and because you EP is not sequence there will be this problem if we develop a network just with UDP which is you send a packet now and the fact that you send now gets a little bit delayed and the new packet arrives out of order and all of a sudden you're getting position update from a position that's old and that's a like a very bad thing to design if you're making a racing game so our solution was pretty simple was

just to make an ordered UDP protocol which is the advantages of having TCP some of some of it it's a packet arrives out of order if it's too old or too late it's dropped you don't you ignore it but we have the flexibility of having a much larger packet and it's less data and the less less data means more speed and because we're availing this to all of our other customers we just decided to put all of it in so you want to send a packet we don't care how you send it we we give you the option of UDP TCP or order GDP so the next thing to consider is the network distribution and this is also huge because coach had many million million players on account remember how many million pairs but many million players on MMX racing and we could expect the same for whatever game they're going to make and it's going to be a global network so how do we deal with having game servers let's say situated in Europe players in the US and

the UK and now we have to send packets in that Network and it's obviously going to take a hell of a lot longer for the packets to go from the US player to the server than the UK player and the big problem with that is even though the UK player is closer it degrades the gameplay experience for both players so both players have to be in sync in order for the game to run properly what the cafe in sync because there's so much lag from the US player that it's going to degrade the play gameplay experience there are ways around this but it's easier just to basically build a distributed Network so what we did is we change the way our matchmaking work to start these games so instead of entering a match based on one level which would be like your skill or your level or something like that and you answer your match when we're doing real-time matches we always check the region your interest and then we find a server that we can distribute so we can set up clusters real-time clusters wherever we need them and women you game starts we can spin

spin up a new instance of that game and have all the people from that region match to that server and so I'm not going to talk too much about the peer-to-peer but the next thing we have to consider is are we going to use a peer-to-peer networking or server authority of networking and for peer-to-peer networking you're sending packets from one player to them to another player and those players are sending them back this usually means that one player has to be authoritative which is that one player has to kind of keep a track of the state of the game this has a bunch of problems like what if that player gets disconnected you have to do some very complicated stuff to get the mac server to take up the mantle of being the host and transfer all that data and you also have huge problems with bandwidth the more players you get on a peer-to-peer network the more bandwidth you take up and there is another disadvantage or one advantage over peer-to-peer or we compared a comparative the server authority of

models is that if you have two people sitting on the train playing it's right beside each other and it's peer-to-peer they don't have a very long distance to go whereas if you have a server in the middle ye the players packets have to go from the first player to the server and then to the second player and then back so there is disadvantages and advantages to peer-to-peer but in our case server authority was a lot better and the main reason is better is because one of which is requirements was that they needed to be able to intercept packets to check for cheating if we have a server running all of the network then we can do that so the server basically you send a packet player one once and send the packet to player 2 they Center the server the server decides is it a violet packet and then it sends it on and the disadvantages of course are what I just said but also because you've got to maintain the server and you've got to scale the server and you've got to run up instances of your game as they're needed but gang sparks already deals with this all the time the gang sparks

is it's a back-end as a service platform so we have DevOps people who deal with this all the time so by having this distributor network and the server sort of the model we don't need to have we have local clusters and then when we want to spin up an instance of the game we could do that on the fly and have it scale whatever way we want playa these are the advantages the load balancing being abscess Berkeley or horizontally as we need them and we have control of it we can add our own network monitoring tools and we can build a distributor network but we can also do because again sparks the game starts portal which is your way of managing all your game sparks features and it has its own ID in the JavaScript ideas runs the cloud code so we're also able to do is to set up hooks that allow packets that come in from the client to be associated with callbacks in your cloud codes and now you can have a server authority of game so your server can literally run the entire game

all of the data and the game state stuff can go on the server rather than having to have one client running what we can also do because the real time clusters and the gang sparks game clusters are different but because we build both of those parts we can add hook in our real-time scripts that allow you to communicate from your real-time game to your database to your social authentications use your push notifications use your leader boards of features and do whatever you want ok so that's basically the model of our server and the restrictions are always going to be latency so what we thought about a best stage was what else do we need and if you look at other network multiple networks of providers like let's say for example photon or you met usually they're going to have clocks thinking clocks thinking is other than making a server i'm making a network that works

cluck thinking is probably the most important thing for games and you need to make sure that if you have your server and you have a latency of 30 milliseconds to player 1 on a latency over 100 milliseconds to player 2 they have to be getting the game updates at the same time there are sort of game design decisions that you can use to overcome these legs but if you have a good clock thinker of you good sink on your network the server can take care of that and it can delay packets and you can get your game designers to kind of fill in that delay with different kind of game game design techniques the second thing is predictive movements if you want to have your server authority of all the movements and all of the players in your game predictive movements is incredibly important and it's also a basic part of having like a complete server authority of architecture you don't want clients to say the clients that the players say I'm moving here and the service side server pretty much says well I'll be the judge of that and then it decides what

what position updates descent everyone else the other thing that's really important for people is a network analytics like your latency your bandwidth analytics packet analytics and what kind of reports and charts you need for that so what we decided on was we had a network that was as lightweight as possible was incredibly flexible with network distribution and allows you to hook into where our cloud code and gave you an IDE that allowed you to run a run a bunch of other stuff basically whatever you wanted and if we put all this other stuff in we'd be taking a bandwidth and limiting developers options we'd be introducing things into the network that developers didn't necessarily need and then eating up all of their bandwidth so we decided on was sort of it sounds like we just didn't want to do the work but what we decided I was not to do that we just basically allowed them to do whatever they wanted with the scripting engine cook into the database and whatever the functions they

want and the tools are there and the tutorials and dr. there for them to build at the server the way they want so if you want to send a packet for game sparks there's pretty much just one inch and then whatever else you want this is all optional there's an opcode which is your tag for what this packet does what a when they hit the server you could use your delivery intent which is unreliable UDP unreliable sequence or reliable tcp and then we have this 40 data which is a json wrapper so you can put incredibly complicated status that's in there and then you very much pick your recipients and if you leave this as it goes to the server only if you put a number in here it's a one of your it's a player 1 player 2 player 3 and so this is how we kept the network as fast as possible that we allowed the developers who wanted to use this network to do it themselves they we didn't want to mess with them we didn't want to have get into argument saying you're using the wrong type of clock thinking or using the inadequate prediction model for your

predictive movement so I was going to talk about Hutch's new game but they're going through the some rebranding at the moment so another company that user that adopted our real time services very early on was fun punch games and their game strikers edge it's a releasing inspire than 17 it's a multiplayer action game for scheming and ps4 and they we worked with them at the start because they're quite early adopters of your key stuff and but interestingly enough they use our real time services but they built their own matchmaking services and we do offer expenses matchmaking services and instead didn't use any of our leader boards they use our databases and they use some of our social authentication but other than that they sort of adopted the parts that they wanted so that's sort of the the main the may take away from the talk was how we how we minimized what we needed

to build and but then to the bare essentials to give developers what they needed would that restricting them or taking bandwidth away from them for reasons that they didn't need and allowing them to just use gang sparks in whatever way they want and pick and choose what's how they're going to work and that's it I don't think being insulted left for questions at this stage but yeah i'll be up there is any ways about pockets and over the people sorry case is it tells something about pockets over tcp because you know we are developing again and we are like we were using dark forest and we are like losing pockets and it seems like you shouldn't be I knew you shouldn't worry about this because tcp should know that your do you depart go ahead but if I we're getting kind of cutting for time so I'm upstairs on the second floor do you want to just

come up and talk to me afterwards yeah okay I believe it [Music] you