F2P Tech Talks: Server Orchestration

today we're going to be talking about server back-end services for games and in particular server orchestration services so what is it who are the external service providers in the space key considerations things that you need to know both in terms of from a development developer's perspective as well as a product manager an executive perspective and to speak on these topics we have three speakers first elmer bulthaus cto from game i who provides external server orchestration second we have chris cobb president and cto of pragma platform pragma are building custom backend server architecture for massively multiplayer games and finally we have avinash pandey cto from free to play shooter game studio leela games welcome guys hi yeah thank you jk so i thought we could start by just talking about what is server orchestration in in terms of like you know how should we be

thinking about it what is the scope of what it is and maybe elmer we could start with you since you guys are specifically focused in this area could you kind of talk to us about what you're doing yeah uh well uh we think when we talk about server orchestration we talk about um yeah basically to make it very simple starting up a server uh in a specific location that's it for us in the very and um that is the easy way to explain it but behind that there's a lot of technology and complexity one of the things is you need the distribution of your game in this case a game binary and also you need the placement of the server done correctly maintenance is also a very important thing in our case we are of course a server orchestrator but we take away all of the the burden of managing servers actually so we have an api you call it and that's it you have to do nothing more got it and then avinash i know you have

been looking at a number of server orchestration companies as well but would you characterize would you agree in terms of that characterization of server orchestration or how do you think about it from a game developer's perspective so i mean uh al almer talked about a very i would say point-to-point definition of what server orchestration is and uh as game developers what you uh so it's a different angle is there is an abstraction layer on top of server orchestration so as a game developer what you would be doing ideally is uh and when we talk about server orchestration i'm going to talk about dedicated servers so like you know this is a very generic term which can be used for even hosting your bank subscription so i would i would talk about like dedicated game game servers so you would be building your game you would be like writing your uh game play logic then you would have your net code which you will maybe test out in your editor whether you are using unity or

unreal on your own setup once you have that ready now you have to host that game so what would you do you you would have your server binary uh optimized for hosting uh which means that you won't be running graphics because you know on server you don't have to run graphics so once you have that binary you need a platform where you can upload that binary and then do some sort of matchmaking so uh depending on your own text stack either you have your own matchmaking service or the orchestration uh provider they have they provide matchmaking services or you have a backend provider who has matchmaking services so depending on these these are some of the criterias which you would look at while choosing a solution so you upload your binary to server organization do matchmaking that's that's i think the first step for a game developer from like you know building the gameplay locally then you know having some sort of matchmaking and then having binary running on server would you consider orchestration uh a

matchmaking part of the orchestration uh i have seen some providers they provide that but i would still keep them separate so like uh yeah it's more like the glue right right if you provide the solution that's good enough if if uh there is other solution but i would still treat them as two separate solutions yeah and and chris i know you did a lot of work in terms of server orchestration when you were at riot but and i believe like in terms of the current solution with pragma you guys have a broader set of services that you guys are working on but could you talk about in terms of what you guys are doing at pragma and in terms of how you think about server orchestration if you if you want to add any additional context yeah so you know i agree with everything that was just shared and i think at the 10 000 foot view as a you know if you're a multiplayer game developer your goal is to get players into a game loop which is inviting their friends configuring a load out which is you know what player character or configuration they're playing that match making loop most that focus

gets spent on connecting players together before you get to the match allocation step however if you're doing geo-located server targeting which is you're trying to find players near each other so that they can all share the same game server you know and be close by so that they all have low latency that's when you have to start coordinating between those matchmaking layers and the the server orchestration so you know our intent um as a service provider is basically to do that kind of game loop and that back end platform and then use partners that have that kind of game server orchestration expertise of driving okay we have these players we need to get them into a game we need that game server spun up hopefully close by to all those players um get those credentials down to the players so that they can make that connection to the game server and get through a match and then collect all the data at the end uh have the game server ship that back to the platform and kind of go through that cycle uh as a player so yeah right so chris just to understand uh

more specifically so for you guys a pragma you're kind of acting more as a glue and do you have an a different server orchestration provider that you provide as part of an overall package is that what you're saying or yeah so our intention there's you know some great providers out there who are specialized in you know data center placement and optimization uh you know doing that kind of cloud bursting of you know if you exceed the amount of capacity that you've leased being able to you know um expand so our intention is to cover all of the platform and back-end services um but partnering with other providers for the game server orchestration so really we're just looking at you know a solution like a game eye that is something that we would uh partner with and do that we have exactly the same strategy but uh we partner we partner with like we integrate with various matchmakers because in the end that's where the integration is going to happen and you know it makes you know matchmaking is a specialized thing and so is hosting so we specialize on the hosting and we integrate with our matchmaker friends yeah yeah

that's right so yeah for example at riot games i spent five years working on our matchmaking solutions there and so that's kind of the the two halves of that exchange you know and surrounding all that is that game that outer game loop of getting players together with their friends you also provide a matchmaker yes that's right yeah you have a potential yeah why not you want to integrate as many matchmakers as possible and i think you know for layman's language uh uh uh and then several orchestration like you have all these ip and ports and uh how do clients find which ip and port to connect and that's where like that's the match making measurements find the right ip and port to connect which is coming from several orchestration that's like a very simple explanation of uh this entire thing yeah yeah and so maybe avinash and chris maybe i can ask you guys so you know so obviously in terms of building the

stuff internally like like avinash you're not thinking about doing that chris you want to partner for that so what are some of the challenges like what what is what what are some of the things that elmer is working on that's so tough so that we you wouldn't do this in-house oh i mean building game itself is so challenging you know so i would i would try to not take up any more challenges of figuring it out and the other thing is like even if i had a team where you know everyone has the capability of building it uh with any any software like the first version is is not the you know final version you will see so many problems even if you have this orchestration layer done once your game goes live then the real thing starts like you know server will go down there are so many layers inside several orchestrations which uh if you are building it for your game you don't want your game to fail and then you will know hey this thing is not working let's fix it and then you know uh game will be up so

you want to avoid those failures for your game so you know it doesn't even if you're building it i would say it's a very risky thing to try at the same time so either you build it try it out with other games and then you know that the primary game which you are building then once you feel that your orchestration there is very stable that's when you should do it but it's a big challenge there are lots of complexities but it's not only a technical challenge it's also getting servers as simple as that because you need a lot of servers all over the world and you want to burst into cloud because you know it's very easy to to do that but you don't want to host all the time on cloud it's very expensive it's also um not the best it doesn't give the best performance so you always want to host on bare metal so if you want if you have one partner for the entire world for bare metal then he's going to be great in one part of the world but in another part of the world he's not going to be great and maybe he's going to be great in january but not in february so you need all those alternatives also to uh to switch over

if you want to have a globally uh a global game uh yeah that's just just a lot of work man yeah yeah to kind of speak to that do you agree with that yeah yeah absolutely um i think that you know in the battle days your only option was to basically you know hire a team and find data centers around the world and you're leasing metal and you know game servers are one of your highest operational costs as a studio running a live service game and so a lot of this challenge is about you know cost optimization so if you're leasing metal or buying metal um you know you're paying for those servers that's racked 24 hours a day but you're going to have peak times where you have a ton of players playing and then you're going to have the off-peak hours where everyone's asleep and so you know having a partner to manage that inventory for you and basically paying only for what you're using can be a tremendous optimization um whereas back in the day we had to just buy our own you know server racks and you just are eating up

that whole cost all the time even if you know most of your capacity isn't uh available and you know if if you hit that home run and you have a ton of players and a ton of interest if you can't support that you know all those players concurrently you end up having these giant login queues and players just unable to play so that's another reason to have a service provider who can actually help you scale is that if you have this you know more players show up than you estimated you're not just stuck instead you can do that kind of cloud bursting and the reason you don't want to be on the cloud 24 hours a day again is because of cost it's so expensive to lease that space perpetually but being able to dynamically um you know burst into there uh it can be really really helpful yeah and also you don't want any vendor lock-in from from clouds yeah that's right is great yeah and then if you are running a dedicated server game then you want to run it in multiple regions as well so it's not that you have a centralized server where everybody connects to like you know you

want your asian class connectivization server players and you just connect to your server so all that complexity is also there well another thing one of our clients is doing is that he's basically experimenting with uh exotic locations so his game is running fine in the u.s and in europe but you know that's not a challenge so now he's going to try and see how the thing works in bahrain it's very easy for him to do that because it's for him it's it's it's no change to to his to his code there's no change to his setup um it's just a parameter that he changes and maybe he does some marketing in that area to see if it if it if it if it works there so you get a lot of flexibility uh you can experiment you can also experiment with different kinds of matchmakers because the integration is already in if it's becoming more like a sales pitch now but um um if you want a different matchmaker you don't really have to code for it you just integrate it and it's not a lot of work so i think also in the development phase you could also use it for testing and stuff like that it can also be of real value so one of

the things that we're focusing on now is because one of the problems that we have is that if we if we find a studio that is building a game uh it's gonna take like i don't know a year half year two years before the game finally goes live um and it a lot of can happen in two years right uh so we're now looking at providing tools to uh to aid um uh studios in the development phase so we can get them as a client earlier and then the only thing we have to do is keep them as a client for two years but that's not that's what we're you know if we're good that's not that's hard to do i guess yeah so maybe we can now talk about like in terms of the let's say the basis of the competition or how studios or chris when you're looking for a partner for your kind of bigger portfolio services but when you guys are looking at server orchestration providers what are the key things like whether it's specific metrics or capabilities that you guys are looking for i mean you know cost geographical coverage in terms of the servers

you guys have mentioned but are those the key things or what are the other things that you'd be looking for when you try to pick a specific vendor for server orchestration yeah i mean i think you just kind of touched on or at least called out a lot of the topics that you want to think through as you're evaluating this and you know i think elmer makes a great point that in development and early development it's mostly just you building and play testing with your team but if your goal is to ship a live service game i think it's really important for students to start practicing that entire loop um because early on you might just be hosting a peer-to-peer you know game server off of a developer's box but that's really not helping teach yourself and your studio how to behave in that live service flow of going through a matchmaker having accounts logged in connecting with players and so i think um you know the first challenge early on is just getting an entire integration all the way through and having the real game server dedicated server that's being built deployed hosted and driven even in early

development is really important to start getting that experience in those cycles as you look to the the solutions for for launching at scale you know there's a huge amount of live service metrics and telemetry and data and so having a provider that can help you get insights into how much cpu and memory are we consuming keeping track of that throughout your development because when your game is early in your gray boxing you know um the server logic and you know the network simulations are going to just be lighter weight and as those features start layering on it's going to get heavier and heavier and so i think looking for tools to kind of get insight into how your servers are performing is really valuable the geolocation and having a footprint that is global is really powerful um because again that's something really difficult to like call up you know data centers around the world and you're not really going to have an easy time of that cloud hosting makes that easier to achieve in terms of targeting but then you get into real cost questions and considerations and the vendor lock-in stuff that elmer talks about which is um you know if your only option is one

of the big cloud providers you know they don't really have much incentive to work with you and if you're a smaller studio they're not necessarily going to give you the kind of bulk discounts that you might receive if you're a giant you know four night size title um so i think that having some flexibility on being able to target metal and use the cloud as needed and then having that independence from the cloud providers so that there is some there's either a negotiation or if you're partnering with um a game servers you know hosting solution they've got bulk discounting because they're hosting you know multiple uh customers so they're you know hitting a much larger footprint so that's my kind of quick takes yeah and avinash i know you're kind of early in the process of trying to select a sort of orchestration vendor but what what's going on through your mind it's kind of similar or how are you going to pick someone yeah i mean right uh so chris has already talked about some of the important things but for example if you're just starting the development how

and you have like four vendors to talk to like how do you choose because you know uh it takes time so my approach is uh there's this standard checklist which you should definitely check so for example they should have multi-region support they should have like bare metal and you know cloud whenever needed so all these things are standard things and i guess most of the providers have these services now as if you are like a nd studio or a small studio cost is going to be one of the primary factors for you and running dedicated server is is costly it's it's not a relay based game you will be running your game binary on the server so what you want to make sure is uh find a way to kind of benchmark all these providers so you know like as uh game i also provides this developer testing period and that's really helpful because testing in local environment with even in your simulation sandbox that's not the real use case like when you make a server binary host it on

the thing go through matchmaking that's that's the kind of thing you should definitely do from the start of the development and then try to simulate as close as possible so for example if you're building a battle royale with hundred players uh try to simulate that because you know now it's not only about the server it's also about you have to run those 100 players on the server so you know server is aware of all these players it's not about the client because you know client will be doing some area of interest or some type of interest management but server is always aware of everything server knows where you are server is transmitting data about everyone and uh if you are doing like prediction all these things so server is doing that calculation so now try to simulate using ai whatever then you know talk to the vendor like i think these guys are really helpful in terms of helping you out to optimize your binary and come up with real numbers like hey if i run 100 player binary this is the kind of machine i would need like these these many cores i would need this is the memory consumption this is the bandwidth consumption and try to get those numbers initially

that will not only help you to you know decide a vendor it will also help you to estimate your cost what if like it doesn't make sense to make that game because you know your server cost is so high that you know you you have not figured out a multi monetization model so i would i would definitely try to simulate real world scenario try to get the real numbers for your game and you know keep doing that over the period of time as your game evolves uh so yeah i know you're still early in your evaluation process but who are the other vendors out there and well have you found any major differences between the different vendors or have you looked at game ideas so i mean so we are already talking to game i uh so there are others like multiplayer game lift juice which recently i got i think acquired by improbable in terms of services i think most of uh these platforms they have core services there so the standard checklist which i was talking about all of them have it so i think it would come down to uh

first of all your personal relationship with the team because you know if you are early in development as much support as you can get because you know it's very hard to predict what you are going to have after one year so you know how the deployment system is if you are building it like you know uh you don't want to go through like five hours process of just deploying binary on the organization service and how supportive the team is and then like if you if you can find those numbers which makes sense for you then and if you have this thing where you can compare those numbers because i think for for the platform it's very hard to give you a standard pricing model because you know every game is different or what if uh if you come come up with csu based pricing one cco for my game might differ a lot from the cc of uh for example you were hosting a chess game or something like that so so you know it's it's uh it's something which you have to figure out with the party yeah you have to figure it out together you're basically starting a relationship it's one of the biggest no it's it's you

know it's i think it's wise to to to let it handle let it handle but get it handled by an external company but it is if if that external company uh makes a mistake it could be disastrous for your game so it's i think it's it's a very uh responsible task at least but also you should really uh pick a client to pick a partner that is um that you know that is friendly with you one of the most important i think you know if you're not nice to each other you cannot really work together you have to work together it's like you say you also need we call it handholding so we help our partners get together get integrated and also some advice on what what's best to do in terms of like optimizing a build process is also something that is usually an afterthought but can bring you very good things you know i know csgo is i think nowadays it's 29 gigabytes which is a lot for a game to distribute among the world so that is an example of an unoptimized uh

build i think right so and albert are there anything when you're talking to potential customers is there anything else that they're asking for or have these guys pretty much covered like the main things that your your customers are ask you about or inquire um matchmaking is a very uh is what they want they don't want to integrate or you know they they integrate the matchmaker and that is enough work already and then you know you're configuring it and all of that but if you if they've done that then you know there's no reason for them to integrate us as well because we can easily integrate with the matchmaker so that's what we're focusing on at the moment and the good news is that they're going to integrate that matchmaker anyway so they might as well do it with us because the the the if they integrated the matchmaker then integrating us is like uh two hours work or something like that less than a day um so they can try us

that is something that we are really um hoping for we we want to um show our product and let people decide if they want it if they don't want it that's also fine then we made a mistake maybe um so um those integrations are something that is uh that yeah especially the matchmakers what people ask for also ddos protection is hot for the bigger studios especially and i think those are the most uh ask things at the moment that i know of yeah i think in terms of the studios we've been working with there have been a lot of active questions about how do we do the napkin math to estimate our costs and how do we measure you know what the bill is going to be and planning for a release is really an interesting problem because one you can have a bottleneck at any area in those those platform services so it's very common for example the account service to get knocked over potentially if you've got hundreds of thousands or millions of players showing up and so i think as teams prepare for release it's one of those where

you know you're hoping for a good outcome but obviously if you pay up front for all of that game server hosting uh that can be really expensive prospect and you just don't know where you're going to land and you know there's some pretty famous stories like with you know pokemon go's release they said well we estimated something like 10x higher than the highest uh you know player base that we had seen in their previous product and then the actual player base was you know another 50 times larger than that or something crazy you know and so you have these kind of exponential outcomes which can be a challenge so i think um tools for benchmarking and estimating um you know that that server performance and then ultimately trying to understand cost i think is a really uh important thing for studios and it's pretty difficult to get data because traditionally a lot of studios have had to do this themselves and now as these server hosting providers there's just so many variables at play it just gets really hard to anchor to because you know again the efficiency of your network code or the efficiency of your memory footprint can be a challenge so i think that benchmarking tools is definitely

something that um we've noticed a lot of students asking us kind of either support or just advice on how to approach that yeah but is it even possible i sometimes wonder right yeah so we saw that you know like pokemon go is a good example which is a game i love by the way but um um it is you know they never predicted that it was going to be that big pokemon go is like something that goes into history books i think at least that's where it should be i think and uh but it's it was so big nobody could have predicted that was going to happen and at the same time it could also go the other way around like we also launched the game um and the players were they just didn't come so it can go two ways it's just almost impossible to predict what your game is going to do i think so i think the only thing you can do is prepare for something that you don't know and you know if you have a lot of flexibility that that's that's i think that's the solution for

that but um um you can only uh you know maybe give give a wide range of what's going to happen but um there's nothing as unpredictable as launching a game i guess right but can we talk about that a little bit more so like in terms of the downside you predict too high and you reserve a bunch of extra resources for that you probably burn cash so you waste some money but on the other side if you under predict like let's say this pokemon go example so what is the impact of of that in terms of like you know how do you recover from that is it easy to recover from do you is it does it take days or does it take hours or uh can you guys speak to that yeah i mean my quick response is that you know when there's the most excitement about your title you know you're gonna have this giant influx of players and if people ultimately cannot play your game that's about you know the worst impossible review you could uh could ask for so i think that um honestly that might be the strongest

case for partnering with a you know an orchestration provider because having the ability to dynamically kind of scale up and they're very high-profile cases you know i won't name any names but there are big studios who have struggled on launch day to kind of keep up with demand and it's really hard to measure whether players are going to come back or not but attrition rates tend to be high in service based games as a general rule so if your servers fall over on launch day you know that's just it does some pretty meaningful harm and i think so it's better to be prepared um and so again part of that is understanding your unit economics of how much does it cost us in cpu cycles or memory that doesn't tell you how many players show up at least it can tell you okay but we get this many players per server hour that kind of thing lets you do some some napkin math and at that point then it is about partnering with a uh orchestration provider so that you can do some stress testing ahead of time or you can do some prediction or you can do some reservations where you know if you know something hits um

other alternatives are um you know there are peer-to-peer right server modes where players can play with one another without paying those server hosting costs the challenge with that though is if you're a live service game and you have a game economy um you know you can't trust a game server that's running on a player's machine so at that point it becomes the equivalent of you know client-side inputs so if you have a game economy of selling content you have to be really careful that on the one hand you can save a lot of server hosting costs on the other hand your entire platform model and game economy has to not trust the game server results which means every player will show up looking as though they beat every enemy on the game if you you can't roll you know rewards on a game server if you do a peer-to-peer solution which means you have to have that platform that other piece that is kind of the area that pragma is pursuing to be able to run that logic and so that it remains platform authoritative so

there are some trade-offs you can look at as a studio of saying hey we can do peer-to-peer and on the one hand that can be great on the other hand if you're trying to build a live service economy um you have to be super careful that you don't just destroy your game economics because everyone can just pretend as though they got the you know the rare content you know on their first try and and that would kind of ruin your ability to monetize your game got it i think that's a good example sorry uh i was just about to say that there is one good example of a game that kind of fell over which was ea's simpsons mobile game so they launched servers kind of blew up and then they shut down the game for six months and it didn't come back online for another six months oh wow elmer you're you're saying go ahead no one thing that helps us at least is that we we have a long period of uh of alpha and vita launches of the game or actually mostly alpha games so this is usually game launch where the game is not finished sometimes

there's an alpha launch and it doesn't really go through because the game is too buggy but this also gives a good feel on how people respond to it so it doesn't you know you cannot really it doesn't really it doesn't predict what's going to happen but it gives you a little bit of a feel on how the game is going to perform and it's going to be launched so that is also i think that is for us that is a really nice tool to figure out what the game is going to do and also it's a good way to to work together on building the game because in alpha all those things that are not great come above and we can fix it or tune it yeah so that is i think that is a very useful tool for us at least the alpha releases yeah yeah and i would say like through these conversations we can clearly see that if you are going for a dedicated server solution the it's a more much more invo evolved process so like you know if you have find a provider who you really like you know they're really approachable and you you can reach out to them with any

problems and you can come with solution that's one of the strong strong criterias for choosing a partner because it's much more involved it's not just a peer-to-peer or relay based where you just host it and you forget about it because you know you you will need to talk to these guys with your clients and partners that's also the fun part of it i think but yeah it's also important for the business to make the thing go well if you if there's a problem you should be able to talk to each other like like well you know like peers and solve it together yeah that is very important you know when something goes wrong with the game it's very stressful and uh that is where your relationship is tested i think yeah yeah the next thing i thought we could talk about is just more around lessons learned or things to watch out for so chris i know you had a lot of experience with the ryan pop cap and elmer with a lot of your current customers today you know are are there big mistakes that you're seeing or any any lessons that people should pay attention to listens

well the first one i'll say is uh don't host your game servers on windows licenses oh yeah so so right games famously was on a windows server for many many years and it is an unbelievable number of millions of dollars in just uh windows license costs so you know configuring your build to to run a linux server is really important uh early on in that process um i i said it earlier before but i would reiterate it i think it's really important for a studio to practice being a live service studio before you've launched and what that means is having a real game loop and a real flow through an account service into you know a lobby where you can do invites into a matchmaker and then getting allocated out to your servers and you know it takes a lot of practice to get used to that because it's a lot different than maybe a traditional single player game in which you build a game binary hit play and then it's all about the feel and the game is fun and the narrative and those things and those are incredible

challenges to produce you know a compelling game experience but they're just a different set of technical uh problems when you start getting into live service multiplayer games and so i would just encourage studios to get those cycles start practicing early those early early access and you know early alphas and and closed and open betas can be tremendous there are different studio styles some studios want an ugly product out early so that they can really co-author the game with their customers i think there's huge benefits to that from a technical perspective and a feedback loop but at the same time sometimes you know studios also want just that level of polish and care and bring something to life that is a little bit more complete but there are challenges with that which is you might not know you've got serious issues until later in the game and so it's definitely a trade-off but those are great tools and you can do that you know there's a continuum there um some studios want to release the title and have it up 24 hours a day for years in early access others might do those you know uh closed or open kind of alpha weekends and beta

weekends and those can be tremendous too it'll see how many players are going to show up does does the technology we have hold up so that can be great any guy if i would give one advice but it's not only for for for gaming but for technology as a whole is don't don't throw a release party so i know i know i don't know and and the thing with gaming is that gaming every every a launch of a game is always kind of a release party and uh it's all it's a very um stressful and um even if it goes well it's very stressful it's it's it's a lot of uh pressure on your uh technology but not only technology also on on every aspect of uh of of your game basically you know will the will the players enjoy it it's one very so if you if you if you go from zero to 100 that's that's not a good idea with any form of technology or uh with gaming and it kind of matches up a little bit uh with what what with what you've been

saying chris um if you just find a way that's that's it's not easy to do but it's i think it's a good thing to do to grow slowly into what you want to achieve then um you know um you're gonna get there or maybe you're gonna find out halfway that it wasn't really a good thing that you were thinking at the beginning so you can throw it all away or change it to do whatever you like uh but going from um no from zero to one and and especially you're giving a party is not a not a good idea but it's so tempting i think um my previous uh company was a little more corporate we had released parties all the time and it was horrible because we were really enthusiastic about this new feature that you were rolling back within a week later because another last thing you have for dick sorry for technical reasons but you know if you're talking about we are all humans and we are changes are difficult for us so if you would introduce a new feature

um you know sometimes people just don't want it or they don't get it or it's too big of a change for them take baby steps yeah right and avanash do you have any lessons from like just the various service providers that you've been looking at or from your evaluation no i would say like uh in multiplayer games you can make it as complicated as complex as you want but is that required for your test users so for example if uh like we are building a shooter game so do we want to do like this complex hit detection on the server side you know do we want to like roll back and all these simulations everything is it required for your test users maybe not like we would just do maybe authoritative movement on server side very lightweight and uh just have an idea like how to do those things in future but maybe you know you can do milestones like hey in soft launch maybe we'll just do this much you know uh in in global launch we would do the rest

of the part so i think being out there is more important than getting everything right at the first go and the other things i could recommend uh everyone is that uh so you will see lots of these shooter games especially on mobile you will see they they are using bots and most of the time or some of the time like what you could do if you are playing only against bots you don't have to connect to server at all like you can just run those bots on the device because you know at any point of time you're not seeing all 20 30 or 100 players you are just seeing bars around you so you know if you can come up with some behaviors which you because initial players are just learning your game they are not looking for that complicated user behavior they are figuring out your control or something like that so even couple of matches and many games do that and you will see like games do that till a very long time like till 10 15 matches you are actually playing with bots and maybe just on your own device so those are some techniques which you can use yeah that's i think that's a great piece of advice and um it actually kind of leads to an even

bigger broader theme which is to really think about your game design as it relates to your live service products so you know i was talking before where you can save money by doing peer-to-peer but suddenly your game server is no longer trustworthy well one thing that you can think about is well maybe there are some game modes that are less extreme and those are more for the either casual players or early players and so then you might you know optimize by making it a you know client only you know solution and you can just structure your rewards and how your game progression and systems work such that you know the rewards that are given for those you know bot only games are you know allow player to progress early but it may not get them into the later stages where you would want that you know server authoritative trusted solutions so you can get really creative in terms of how you structure the game systems and how players progress through and it's become a very common tool on mobile is doing exactly what avinash is saying which is just play against some bots and ai you know matches so that the player can get familiar

and then you're not loading up on server costs in that early early do you think it should but one one thing how do you say that one thing that is in what you both say is that um you should look at it from more like a holistic point of view hosting a game is not a separate thing from the game if you pick it up together so uh for instance you could also maybe you can host in a cheaper location and have more latency and this will result in a game for some games this is fine you know maybe there's a different variant of the game or maybe the players that are not that doesn't have a high skill they can be put on those uh latencies since this servers maybe so you can reduce costs if you pick it up as a whole and not a separate things you can do all of these things and that is something that is i think is that that is very important the game is the unit the hosting is not a separate thing as is the gameplay it's all together it's all you know if you pick it up at all it gets better i think yeah yeah i absolutely agree with that um that reminds me of another kind of

oh so this this is actually getting a little bit more onto the technical side of things but uh when you think about trying to explain your live service pipeline so you have platform services which are your accounts and your matchmakers and your player inventory and your data processing and you got your game servers which is the netcode and you know striving for simplicity you know oftentimes people will mix the words simple and easy but it is easy to make a complex mess it is very hard to keep things simple so i definitely um advocate strongly that your your engineering team should be able to draw you know the pipeline of how the game gets built and deployed and shipped and released exactly and that should be something understandable so you know i know for example containers have become a huge area of topic and exploration and they can be a great tool but there's a certain kind of inclination for folks to just start layering on top of each other all of these different abstractions

and suddenly the answer of the question of how do we build it ship it and play it becomes like 19 different you know boxes and arrows and graphs and you go up and down and sideways and this i think it can really overburden and overwhelm a team and so really working hard to keep that tool chain simple and really be clear like at the end of the day we have to build a game binary we might have some platform services we need to get those onto a server somewhere and host them and and just really striving to keep that understandable and the number of times i've seen you know an 18-step sequence of the 10 different layers of virtualization and docker and kubernetes and again i'm not i'm not attacking those technologies i'm just saying that if you kind of approach them naively as though they are the solution to your problems you might find yourself with a bigger mess than maybe you started with um and so just being really thoughtful about each piece yeah and then also if there is going to be complexity at some point but what we do is we isolate the

complexity so then it's simple again and some some isolated parts are complex but on their own they are actually not that complex because they are isolated from the outside world they have an input and output and what's happening between those two things may be complex but that just means you have to try really hard to read through it and everything like that so in that sense complexity only will take time for the rest um that i think for us that is one way to make complex things also simple because yeah i i totally agree with what you say um making things simple is one of the hardest things and also very misunderstood things i as with technology i sometimes have the feeling that if you make something that nobody understands it gives you some kind of a status or something like that if you make somebody if i make something that my mother would understand then i'm probably not that good as a developer or i'm probably not that smart because you know my mom gets it but that is actually the hardest thing

to do to make your mom understand it and uh so that's yeah so it's it's a goal on its own yeah a mentor of mine shared that like if if you can explain an extremely complex or sophisticated topic to you know a lame a layperson you know that really demonstrates actual mastery which is kind of opposite of you know people wanting to use complex sounding words and phrases to confuse people and make themselves feel more sophisticated perhaps but sometimes i also feel that it you have to be in order to do it you have to be you have to be self-confident as a person as well because you're kind of going naked a little bit if you're trying to make something simple that you spent maybe you spend a half a year on building it or designing it or whatever then you're gonna explain it and then you say okay it's this thing and then people say this is that's what you spent or all of you why why didn't i i never saw you in the weekend anymore is that is this what you did and then you know

so you have to be um it's much easier it's much easier to come up with some very complex thing and everybody will say wow this is so big did you do that so yeah i would say like sometimes your game does require complexity but it might not require that complexity to start with so you know start this simple but have a plan to support that in future so you know it doesn't mean like you don't have to design your system in more robust or more secure way but uh have a plan for that so you know you should be aware of the shortcomings of your game and one of the advantages of running a dedicated server or a hosted uh game is that uh it's security like it's in your control like you know you don't have to send one more game update to fix something on the server side but you can do that while the game is live so you know have that awareness that those points where you know that this system can be hacked but if it gets sagged you should be prepared for it so you know it's more of an awareness thing and guys in terms of like the

future whether it's the future of server orchestration or game hosting more generally could you guys talk about what are the upcoming trends whether it's from a technology perspective or to the point about are we going to see more simplification or things of that nature could you guys speak to that maybe starting with you elmer yeah i think it's going to be future of of game i think the future of the game of gaming in general is is not gaming anymore actually um i think the future of gaming is things like going to concerts on your computer maybe in vr i don't know about that because you're probably going to do it with your friends as well so going to concerts meeting people um but in a you know in in a real it will feel as if you are with the person it's already happening a little bit but like for instance you can go to uh before that you can go see a movie with your friends and you can all wear your well i have a meow soul suit which i like to wear in fortnite and you go there and you do your emotes and that's how you watch a movie together you can also go to a concert there together and i

um this is i think this is what what the future is going to look like for gaming and of course you can also game but at some point those roles are going to overflow into each other and we're going to have something that is similar to maybe the matrix or a ready player one or you know all those movies it's gonna go into that direction and and and i see that every day actually i think the reason why it's not there already is because the technology is not there it is currently not possible to make an infinitely big world in a way that is easy to develop easy to host of course you need specialized hosting capabilities for that and um but it's that is something that that that a lot of companies including gamer is is working on to making a world that can be infinitely big and as big as the real world and then you can have a concert and when you're done with the concert you go to another venue and you go play a port of battle royale all in the

same it's not a game anymore it's more like a do you remember second life it's like that only done well and second life was a very weird thing i think um i know so people it's a really weird stuff there but um it is you know actually there's a bank in in holland here where i live and where i am and the bank had a had a shop in second life and that is um in that that was like 10 years ago i know maybe 15 years ago and at that time it was crazy but in the future this is something that people will actually make use of and will appreciate i think and that is so uh gaming as it is now is gonna be more it's gonna be really big i think yeah i see it with my children already okay avinash or chris future of game hosting or dis gaming any thoughts uh so just to talk about game hosting i

would say so i look at it in two ways one is the evolution of platform itself so uh so right now you see all the all orchestration platforms are getting really matured uh we don't have to you can actually you it's very less there are very less chances that you can go wrong with any of these solutions as long as you have lots of money so the the main optimization part comes like you know you want to have cost optimized solution but i see all these platforms evolving to an estate where they will provide more holistic solution so you know maybe they will have all these services which games need you know server orchestration matchmaking maybe they will start providing netcode as well or maybe they will have their own game engine so like unity i think they are already doing it they have multiplayer and after some time what they would plan is that just your game just at a button click it's a hosted game or something like that you know so more seamless integration for the developers

so that could be a near future i'm not talking about like uh it's it's not very complicated but it's already happening and the other thing is which alma already talked about is the scale of the world so right now we have this limitation of number of players you can support on a server so you know there are solutions where you can cluster them you know you can create a layer where you know you can support large number of players but i think that is something which has to happen you know you have these talks about like all these meta hours and everything so i think that's one future where i see these platforms becoming really really important because you know more players are coming online they they will be sharing same universe with with others so uh that's other thing where i can see lots of development happening in future yeah i mean i think that just from the hosting side i think we're going to see costs getting you know more optimized i think the cloud providers are taking an interest here and i think having a dedicated provider that's partnering with you know hosting services is just going to kind of you know reduce cost for students make it more achievable um just at a high level you know the

idea of a small studio being able to support a big you know ambitious live service game i think is becoming more achievable over time so it's really exciting to see that and kind of champion that from the metaverse perspective you know it's definitely an interesting topic and i i like to keep tabs on you know various leaders in the game space talking about this um you know i'm less excited about the prospect of having 10 million concurrent players because at my home you know i might have a handful of people at a park there might be 20 people for me it's less about the need for everyone to be simulated in one spot and more about that seamlessness with which i have some kind of online identity i can connect with my friends and if you look at something like roblox you know especially in recent years as they've kind of hit their stride and growth is that you know kids have a single identity but they can go to unique experiences that they can share across a group of friends and i think that something

in around that i do think that there is a ethical and moral conversation that's happening right now and it's happening in social media you know across the tech giants and i think that there is a lot of work to be done there because there are a lot of difficult problems to address in terms of how people interact with one another what that means and you know i spent a lot of years at riot on a team called player behavior whose mission was to address toxicity and improve sportsmanship and i think that that's a crucial area that we need to invest deeply in which is how do you as a parent how do i give a device to my child and just know that they can spend time with their friends and they have this digital place that they hang out with agree with elmer it doesn't have to be games it can just be a social experience but how do we give people tools to keep themselves safe and protected to engage with other people in the way they want to engage um how do you put you know the kind of mechanisms for people to uh establish good social norms and have the same kind of behaviors we have in real life with one another instead of some of the kind of aberrant

behaviors we see yeah but also your identity how do you it's you know one of the things that because this if this is going to happen and it's going to happen in some form i think and like i said it's already happening a little bit um but it brings a whole set of new problems as well and one of the most interesting things i always think is how do you what is your identity if you are if your avatar is really um is you it feels as if if you if if your avatar makes friends if your avatar has a job if we um if he has an insurance why not all of that stuff is going to happen at some point you can have that then your identity is basically you know my identity is basically a property of the dutch government but now if i would be in fortnite my identity is owned by epic this is a completely different entity with completely different norms or how do you say different

incentives than than that our government has and i think that is a very interesting question that is at some point gonna gonna pop up because if half of your life is spent on the internet and i think this is something that is gonna happen to my children because actually half a life is already on the internet if i'm not around um that's all they want to do um so um then um yeah that is something that that is going to bring up a lot of discussion like you say you know you need a whole different way of managing a culture basically it's very interesting it's also a bit scary all right guys well those are all the questions that i have i know you you were kind of talking about the life of a cto prior to test recording do you have any discussion topics if you want to talk about uh anything around those audiences but for me i've got no more questions three of us were all cto uh ish or even cto and and we we all uh work at a startup right and we all

founded it i guess um yeah i'm we i'm with with uh with it's me and two two co-founders and um uh so that's that's a funny thing right we're in the same same boat basically um so one of the interesting things that i always want to i want to talk about with when i meet another c cto is are you still developing and are you ever going to stop developing how do you see that in in in the future so uh chris yeah chris i guess for myself yeah it's been the slow steady you know descent into people management um but i do think that there's a tremendous amount of power in trying to you know build a team and set direction and obviously uh with covet in last year we've had to figure out how to do remote which you know we've always had some remote but there was always a blend and so trying to explore how we do that and the fact that you know video calls only work half the time even today so you know i think that there's a lot of challenges there as a technologist i've found myself more

and more attracted to the ultimate experience that the user is going to going to have whether that's a game developer managing their live service game or that's the player using that matchmaking service and and those multiplayer networked you know experiences so yeah for me i've i code less than i used to i still love it and so i try to but uh i do find a lot of it gets into like how do we build a team how do we get everyone head to the right direction how do you empower people to have their own ideas and champion that while at the same time keeping the product cohesive and make sure that you know you can have 10 great patterns but each if each one of them is different in your technology then it can still feel very disjoint at the same time it can be tough if one person is just dictating every answer you know your team might not feel as empowered so finding that right balance of okay we're all headed the same way we want to create this great product that people love to use it's cohesive you know i learned the pattern in one place and it works in another um while still taking in all those ideas from the team so yeah for me it's for better or worse you know i love working on a team so that part's

tremendous but there is also that stress of well i don't get to put my headphones on and just code as much yeah yeah i think all these uh things are very true for for anyone who is uh in this role and one thing which i personally realize is i still enjoy coding and hands-on as much as time i get i i directly jump on it but the other thing which i think we have to realize is always looking at progress in everything so you know at the end of the day i know i step back and i see where everything is going so like whether you know sometimes it's easy to get lost in work because you know everybody is busy everybody is doing work but are we moving forward like are we because you can spend a lot of time doing one thing for a very long time but you know as as as as someone who is leading the team you have to make sure that you are also moving forward and that's where this concept of simple versus complex comes in picture which we were talking about like you know making it simple like you know you will

you have to find a team where people are like really really smart but you know sometimes we will find balance of people where they want to get the best solution in first shot but maybe you know you have to kind of lead your team in a way that hey this is good we'll do it but you know let's let's do it this right now set the right milestone so that you are moving you know towards your destination so that's something which i find really interesting because you know if i were in that position coding it i would i might have done the same you know i want to get this thing right now yeah if you should do it but like you say it's you we have to move forward in life and uh the good news is that i uh you know the programming is is now a hobby again so that is good i used to you used to do that on a saturday night and um build something that nobody wanted except me and that's what a hobby is basically so i got my hobby back but like you say it's more and more people think nowadays and it's also um

except being a leader setting and setting an example or you know if something goes wrong on the technology side but also on a lot of other areas in my company then people are looking at me and if i don't have the answer then everybody's going to be panicking and getting very stressful so you always have to have that answer and um i kind of like that it's a lot of work definitely yeah but i love my work okay all right guys just to wrap up do you if you have any final message in terms of like uh you know uh customers you're looking for hires or and how people can get in touch with you guys maybe we could wrap it up in that way maybe starting with you avinash yeah so uh you know uh go uh uh read about lila games we started recently and uh we are hiring so you know we are looking for the best

talent in india and right now we are also kind of uh opening up like anywhere we find people who want to make next stop crossing game like we we are the company find us yeah we're also hiring like now if you if you uh we're not hiring anybody we only hire people who want to work absolutely want to work for gay my so not for any company and we're looking for everything basically developers but also people who know servers very well also people who are good at support and uh but you know we have to uh for us the most important thing is that there's a cultural fit basically this means that you're humble and nice you know er there's no place for egos at our company at least and um yeah so if you want to get in touch with me about that or if you want to have help imagine you if you're if you're a young guy or a girl or something like that you need some mentorship i'm already also i love to do that um if you're

serious if you want to contact me then um try linkedin and yeah i think that's the best place to or send me an email at elmer at gaymay.com and hopefully somebody will read it um i'm not an email person um yeah okay so chris yeah so i mean you know kind of the mission with pragma is really oriented towards creating that vibrant healthy online communities and we're doing that by building uh basically a large-scale back-end platform to power live service games so we're partnering with game studios who are trying to create something ambitious and uh you know reach a lot of players so uh you can find us at pragma.gg and that'll have links to all our social accounts so yeah it's been a great uh time talking about do you have a matchmaker right that's cool i'm gonna look into it we'll have to talk after yeah no sure we're gonna check it out and then we're gonna we're gonna in the next wave of the beginning of a deal right here

all right guys chris thank you so much for your time and insights and until next time everybody that's it all right [Applause] [Music] [Music] hey you