The Business of Cloud and Game Servers

[Music] welcome to the novic gaming podcast where it is our mission to explore the business and future of video games we bring together the industry's brightest Builders investors and thinkers to keep a pulse on current events dissect emerging Trends and games share Lessons Learned and have a great time this podcast is also part of novic growing ecosystem which ranges from free and premium research to Consulting and AD advisory services for more information visit [Music] www.nic.edu and Esports tournament organizers grids data infrastructure and Distribution Solutions help game rights holders unlock new revenue streams while empowering their communities and professional players no matter what you are building grid can unlock your game's

data potential through its off-the-shelf grid SDK or fully customizable Solutions Plus if you're a developer or entrepreneur with a live data powered project idea make sure to apply for grid open access to start building with supported official game data today learn more and sign up at grid. or check out the link in the show notes and with that let's jump into the episode what's up everyone hello hello welcome to the novic gaming podcast I'm I'm your host Alex deay and this of course is interview and insights today our focus is on the role of cloud and server orchestration in the distribution of the modern video game the menu of options Studio have and the financial implications of those choices I.E the business model around Cloud the topic is absolutely massive for any server hosted game think Minecraft or online multiplayer game think League of Legends or CF thieves cloud services and server orchestration make or break the experience does not matter how good the

game is unless a playable experience is delivered into the hands of Gamers we're going to open with clarification around what this episode is not about why we should care from an operative level about your Cloud stack and precisely what is a cloud stack an overview of the options that Studios have an overview of the providers and discuss the different business models present in the cloud business and how your Cloud strategy should change over the course of a gamees life cycle we've got a ton to get into today so we're just going to jump right in and introduce Our Guest who are experts in this field first up my homie Sid dulip founder and CEO of haora a company that's powering server orchestration and the cloud backend of many upand cominging VCB Studios and supporting launches like frost Giants recent stormgate beta welcome to the Pod Sid Alex thanks for having me on awesome and next I have stepen Hartland who has a long technical career in server management event orchestration and all things gaming technical infrastructure Stephen now at rocket science previously

spent 20 plus years at multiplay and has a background in Enterprise hosting at Electronic Arts welcome welcome thank you Alex uh yes we worked with Electronic Arts not actually worked at Electronics art just a little clarification excellent correction thank you um and so awesome well I'm super GL super glad to have you guys both on um and before we start talking about anything real today um you know our first our first attempt to do is to get some of our terminology straight so cloud gaming though not as ubiquitous as buzzwords like metaverse and AI means a lot of different things depending on who you talk to in context and today as our title indicates we're here to talk about the business of cloud in gaming not cloud gaming cloud gaming being things like Nvidia GeForce or Google studia rest in peace um and so Stephen for our audience I would love for you to kick us off um do you mind describing what cloud gaming is as to make it distinctive from what we're actually going to talk about today yeah so cloud gaming as you kind

of uh articulated there there is a a lot of misnomer around it you know what we're talking about here today is how do you get your infrastructure up and running how do you scale those virtual AstroTurf environments up to ensure that your your Gamers your your customers at the end of the day have that space to go and play um and that takes a number of different forms it can have um the core service is in infrastructure side of things so think Matchmaker think identity the ability to just log on the ability to search for matches or it could be that virtual AstroTurf that big scale piece where you're playing in in an apex Legends match or you're playing in League of Legends that individual instance where you are logged on and you are collaborating with others in the that space uh and attach to a specific

uh piece of compute got it okay well that's really helpful gives us some grounding that we're not talking about cloud gaming we're talking about cloud ingaming and with that and now that we have that kind of cleared away um I would love to actually pass it to you guys for some intros um Sid how about you kick us off tell us a little bit about yourself and tell us about um haora sounds good um so I I guess like you know come from a very technical background I studied computer science in college was always really interested in the systems side of things even College um and then I actually uh went into uh paler out of school and paler was making a transition away from selling on-prem licenses to Cloud hosted SAS um and I was kind of asked to get my hands dirty in that transition um it was a super exciting period for me personally because I learned a lot about how infrastructure at scale

should be run so everything from you know what's the best way for the other product teams that we're using our platform to have like seamless cicd um how do we bring observability back to those teams so when something's going out wrong in production they can see all of that um but also like you know when you're spending like upwards of $100 million do a year on cloud you know you got to really optimize that because it really affects the bottom line of the business itself um and you know 's more things around like security compliance and you know there there it was great to kind of get my hands dirty across like all of those dimensions and then um I was at datab bricks for a couple years after where we were working on multicloud uh so datab bricks is a platform was available on gcp and Azure before I joined but they just um signed a a deal with Google as well and my team was basically tasked with the responsibility of like hey can we build a cloud abstraction layer so internal

product teams don't have to rebuild the same thing three times over but rather they're building against this platform that you guys are building and then you guys are responsible for making that platform work across all three clouds right um and then from there uh you know I actually got into gaming because my co-founder was was the one like you know we we've been friends since college but he got into programming in the first place because he wanted to build games and then kind of got really deep into building multiplayer games um and he identified this this Gap in in infrastructure uh when it comes to like building games at scale and being able to provide it to a global audience what he observed was you know a lot of game studios have excellent Talent um when it comes to finding the fun which makes sense because that's what studios are meant to do but when it comes to all of the supporting set of

services behind that it's you know with game launches going not as well it's kind of apparent sometimes but we wanted to kind of bring a platform to the gaming industry that made you know launches seamless developers focused more on finding the fun and then ultimately leading to better player experience so today haora works with a lot of really exciting up incoming Studios um can't name all of them yet but we're excited for some launches coming up and once the game is out we' be happy to talk about that but as a product uh we we have servers in 12 different regions and we know take the customer the studio built games and then we have it deployed everywhere and then when you have players in a certain region of the world you hit our API and we start that instance for you awes that's how in a nutshell awesome nice very nice um Stephen what about you so yeah um I've actually got a little bit similar background to Sid started at

University uh computer science degree with management um uh originally came out of University and did a stint um in the content management space so I worked with a a bunch of people um uh about scaling infrastructure scaling dat databases backend Services uh back in that day it was all Oracle databases and things like that but always was a massive gameer love like uh love playing games Unreal Tournament 99 was my original game that really got me into the kind of the competitive scene and and attending lands and uh off the back of that attending lands I I bumed into a company called multiplay in the UK and they were one of the biggest land providers still a fairly small company um at that particular point in time um and to cut a a little bit of a long story short I built some technology that allowed that team to run their land

infrastructure um both from the competitive side but also game server scaling right so had the ability to scale up servers on demand for Counter Strike for Doom for Quake for unreal tournament and and that that uh infrastructure solution that we ran the events on um became really popular at the events but and the customers were asking well why don't you do that online so we spun that off we created a solution that was very much a b2c solution back in the day and that gained again more traction in the industry we got noticed by people like um Electronic Arts um and supported them with commercial instances as well as the BTC side of things so we built a uh that business up continued to build build that business up that was acquired a couple of times over once by game digital um and then a little bit uh after that by unity I worked with unity for a uh six or so years and continuing

to build that out and that multiplay product was one of the first game server scalers game server orchestrators on the market um and we've had some great success with that technology so I essentially I was the the the CTO of that business for quite a few years and our big kind of kickoff success was Titan 4 two um that we uh built that out across multiple different Cloud providers and then Smash Hit successes with Apex Legends um uh after I left Unity I joined the team over at rocket science where we focus on um all of the kind of the backend Service Solutions um so from Cloud Server gaining but more on a contractual basis coming into big Publishers big developers as well as some smaller developers uh people like Gang Beast we work with them um ensuring that they've got that backin knowledge that backend experience to create those

seamless experiences online all right well from those intros um I am super lucky and privileg should be surrounded by people who have such deep technical expertise in this field and so um I'm really excited because we're going to get really comfortable with a lot of things that are pretty complicated um and so we did a little bit of definition bingo with what cloud gaming is Stephen thank you for that and we're going to keep playing this game and Stephen I'm going to lean on you here again um there are three terms that I want to get um some clarification on because I think it will help contextualize for our audience some of the things that we talk about later on and these are three things utility computing grid Computing and Edge Computing um Stephen do you mind kind of describing what each of those three things are um for our audience so that we can get those terminology straight and so that we have context for how we use them later on throughout the episode sure so grid Computing really is about bringing disparate infrastructure together a bunch of computing resources that are spread over different

geographical locations typically um to achieve a single task so you're combining multiple compute units all to achieve that single specific thing it's not multiple disparate tasks it's a single specific task in comparison um your utility computing is more akin to what cloud is is typically understood as today so that's provisioning specific pieces or specific blocks of computing result sources uh to specific clients to achieve an OnDemand basis requirement uh and finally that kind of edge Computing component is about bringing that geographically close to the user so something that is highly latency sensitive um is something that uh really benefits from Edge Computing there are definitely challenges when it comes to Edge Computing and how well you can kind of use it because you know you don't typically have as much of it about

because it is at the edge it's close to the users you don't have massive data centers distributed everywhere um interesting okay so repeating what I heard back um grid Computing is kind of like socialist servers um you know lots of combining different resources to accomplish a single goal um utility computing is more what we're talking about when it comes to cloud game Cloud the business of cloud in games it's like Public Utilities like water you get charged on a specific usage and then Edge Computing is physically locating those servers near the users um which is I think very counterintuitive because I would think the edge is far away but maybe the edge is close um and that is four four games and so all right so that's really helpful thank you for giving us that rundown on those three terms from I would love to hear from both you guys you know which of these three is you know it's obviously a sensible that utility computing is the most applicable for for games but what

do we think about Edge Computing um as it relates towards games and whether or not that's actually being orchestrated today and if it's something that's actually being done um Sid maybe uh any thoughts here sure yeah I think Edge Computing is the one that people think of as typically very helpful for games and in practice it is helpful up to a certain level but taking to his extreme version it's actually not the most useful of things so when you think about latency in games right like staying under a 50 millisecond profile latency to the server in back is very important but the difference between like 15 versus like 3 milliseconds is actually not as large because there's just like fundamental like bottlenecks in other parts of the stack um including just like rendering a frame in the user's machine itself so when you think about Edge Computing there's there's a few downsides taken to its extreme like

imagine like you know every neighborhood had a a like Edge node that you could schedule um matches into the problem then becomes well you're going to have a lot of like idle capacity all over the world that you'll never truly be able to fill so there's a cost problem that you'll run into and then there's a secondary problem which is player liquidity right so if you're trying to do fancy matchmaking where you're trying to like you know pull people together based on like language they speak or rank or you know match modes the set of players that you have available starts to shrink in um a pool and by the time you end up you know picking like five 10 you know 64 players you you need to kind of optimize for all all of those players and not just the one player who happens to have a node in their neighborhood essentially so that's why a lot of games have kind of gone to the hey we're going to pick like 10 to 15 regions globally and have large singular pools of compute in each

region but you know everyone within that region connects to that same pool and then you know it's between like 10 to 50 milliseconds for those players but you know it's it's in the realm of very reasonable wild balancing cost and other things like matchmaking that's pretty interesting and yeah and actually an excellent segue because this has a lot to do with the player experience and why we care about the cloud stack in the first place um and I think this would be a great time to jump into you know why do we care you know you're talking about oh the difference being 10 to 15 milliseconds between each player but um you know why as a developer you know beyond that should I care about my cloud stack um you know and this is what's going to inform the menu of options and the implications of each options you know whether or not you wanted to choose an a true Edge compute or this geographically server uh Lo co-located setup that you just described um you know what does a cloud provider actually do for me and why do I care as a game Studio about what I how I select how should I think

about that Sid yeah absolutely so I'd say like the number one thing at the end of the day is like player experience right like if players are not having a good time then the game is going to to suffer so even when you think about your Cloud stack the first thing that you should be thinking about is like okay well how does this affect player performance um and there's a few different dimensions within player performance right so the first we've talked a bunch about is latency uh but beyond that too you have concerns like availability so like you know suddenly you have 10x the players like can't do you have enough computer available in that region to handle that and how quickly can you add new capacity right and then the last piece is reliability as well on that front like you if your players are hopping in and you know their servers keep disconnecting every like you know 10 minutes or you know they have a high like dropped packet rate it's also a poor experience right so performance uh player experience as a whole is like one bucket then second I

would say like almost equally is important but just slightly less important is is cost because like you know you can you can have the most cost-efficient thing but if it leads to a poor player experience that's still not like that that's not good but like you know cost does matter at the end of the day and so if you like deploy purely on the cloud there's like a set of cost considerations if you deploy purely on bare metal there's another if you don't even use dedicated servers you know there's another cost consideration so that's something we'll probably get into a lot later on as well so I'll leave cost there and then lastly you know reasons like 3 to 10 there's there's like a whole bunch of them you know the top ones that come to mind are like developer velocity right like how quickly can you like ship updates to your game um like if you're doing like Seasons like do you feel comfortable like launching on a on a fast Cadence and feel comfortable that when you're doing this like you have enough capacity and all the other problems that come along with it then you have concerns like security like you know are you patching the underlying machines fast

enough like are you getting like are your firewall rules set up correctly are you do you have your deos protection set up correctly um and there's a whole bunch of things around security as well but I'll kind of pause there you know it concerns like you know five through 10 are probably like way lower tier than the first four yeah well I mean but again all of a lot of it boils down to one the player experience for security it's also a player experience question right um and then a developer experience which is how much money am I going to be making um for the services that I'm providing hopefully I'm providing a great game and a great experience but you know is my is my margin going to be so diminished by providing this great experience that I as a developer willon be able to continue my vocation um and that's actually some really helpful grounding around security latency reliability um availability as you said said um and Stephen you know I know that you have experienced working on some pretty big titles um at like the hyperscaler level like you said in Titanfall and Apex Legends um and I actually recently read an article in

preparing for this episode about apex's multi-tiered Cloud infrastructure um and Fry one wondering the article is aptly named um uptime Legends and is linked in the show notes or rather I'll link it in the show notes um and so Stephen could you share a little bit about their strategy and how that contributed to the Roaring success of the Apex Legends launch yeah so as Sid articulated really well there consistency and player experience are utter key right um uh but not only that the ability to to scale to the demand again uh as mentioned um and one of the problems there is people will will hear the word cloud and you know they'll assume cloud is infinite right you can just keep requesting more and more and more Cloud that's actually not the case right you know when you're at that higher tiers at that larger scales you actually need to be collaborating with your Cloud providers to ensure that

there is going to be enough capacity so one of the ways that you know you can you can uh reduce the risk essentially to you and to your players is to ensure that you've got this kind of tiered solution um and that was what we built uh over at multiplayer which was what we called it was a hybrid solution so we had the ability to select regions as said uh articulated earlier but where did that capacity for those regions come from um and our initial piece you know talking back to the cost element was coming from bare metal capacity so that's like layer one tier one it's worth of capacity um but that takes a long time to provision it's on a monthly Cadence in terms of renewal so you don't want to buy too much of it you don't know when you launch a game are you going to be wildly successful mediumly successful or underperform so there's a

there's a risk in terms of the cost of your business there um and and the next year is your primary cloud provider now that might be uh from a cost perspective you might have a deal with one of the major Cloud providers you might have commitments with one of the major Cloud providers where you get uh you you you have to say that you're got to use a specific amount of compute it's going to cost x uh but you have to guarantee that you're going to lose it and you're going to if you don't use it then you lose it you still have to pay for it uh and the Third Kind of tier really is what is your fallback so in the in the situation where your cloud provider doesn't your primary cloud provider doesn't have enough capacity what are you going to do at that particular point in time so that's kind of your third tier um but there's also an additional reason for having that kind of third tier and having multiple Cloud providers and being able to be kind of cloud diagnos and that's in the case of failes um we

had uh a situation at one of the games conss um where there is a a major outage on one of the cloud providers over in the US and the Apex Legends uh solution that we put in place for that team stayed online because we were able to transition uh players away from the affected compute and onto a totally different Cloud providers compute um bypassing the the AG in its entirety um pretty much seamlessly across the board now there is a a cost implication for that right because you're spinning up an entire new set of C compute in a different um location but in terms of the impact your players you're minimizing that impact to your players so again the Sid kind of mentioned there there's there's a balancing act that the the developer and uh is doing between experience and cost H that's a really yeah that's really

interesting and the way that I'm thinking about that is it sounds like again business but like there's the there's the debt payment waterfalls um where you have your like senior secur debt and then you have your mezanine and everything else it kind of sounds like Cloud setup similarly where you might have a stack of hybrid Solutions where you have your dedicated if you achieve escape velocity beyond that you bounce into the cloud Zone and then you you know go up and up in tears um and in summary kind of what I'm reacting to is that it just seems like you know for Apex Legends their strategy was to be partnered with I think almost three Cloud providers they were partnering with gcp Azure and I think also Amazon um and so the idea being here is that just like not one provider Works um you need to have backups to the backup in the case of failures um and to keep that you know insulated um so thank you for sharing a little bit about that experience um and you know again Apex is obviously one of the biggest multiplayer games and you know a lot of why it's been so successful is because the launch was successful players were able to

access the experience and continue to play um every day since since it's launched one one of the big challenges there just to jump in for a second was you know how do we test that how do we get it to that state to the where we knew it was going to be successful so the that that multiplayer architecture that we built out that supported that game launch was tested to millions of concurrence in very short amounts of time um I'm sure there's a video knocking around on the inter aware of us of showing the figures so you know it was up into the uh multiple Millions within 15 minutes worth of capacity so that and that really allowed us to to give that seamless experience to the users one of the things that a developer always kind of wants um is a boring launch from the infrastructure yeah definitely a boring launch sounds sounds really good um and so yeah I guess like let's talk about how we can best find find our way to a boring launch um in talking about the

cloud stack and the overview of options and we all know the phrase build or bu and and I would assume there's a lot of gray though in between here um we'll get to this later on but one of the thing my favorite things about cloud infro is that there's not only options to build or buy um but they're also the answer to whether or not to build or buy is dependent on the stage of development um and Stephen since you've been in the engineering driver seat at rocket science um and in many of your roles in the past do you think you could give me an rview of the options of the modern day technical architecture for each and maybe give me one to two pros and cons for each of the four setups which I Loosely understand are bare metal dedicated peer-to-peer and hybrid and we talked a little bit about some of them in the in the Apex Legend example but yeah so bare metal um it kind of is what it says on the tin it's it's typically compute which is rented out on a month-by-month basis the benefit of that the real big benefit of that is it typically comes with a large chunk of

free essentially bandwidth right games are bandwidth hungry right they they are not like a website where you know you can ship tens of thousands uh of page impressions in a really small amount of bandwidth um you are typically shipping around um hundreds of kilobytes per second per player so when you've got large instance servers you are shipping around a lot of capacity for a large amount of time time so when you've got that those bare metal instances um and they come with multiple terabytes worth of capacity bandwidth to be able to use for free that is where that kind of big benefit is the the the con of bare metal is how are you going to automate it how are you going to orchestrate it how are you going to set it up um and that is where you need a solution like multiplay or like haerer to ensure that you've got that automation that can take care of

the heavy lifting of that when it comes to Cloud um scalability is it's Jam right at the end of the day um it it has the ability to to burst and create lots and lots of capacity you can make the wrong decision in terms of uh specific instance size and adjust it really quickly you know typically spin it down spin it up within minutes if not seconds depending on on what you're doing there so flexibility is the ultimate thing there but the challenge comes from the opposite side of what bare metal is and that's cost of bandwidth right so you're playing per per per bite per kilobyte per megabyte per gigabyte's worth of bandwidth um and when it comes in comparison to what you can buy though that capacity kind wholesale you're actually paying quite a steep premium for um the

flexibility um uh but that flexibility is not just flexibility also comes with a great level of surface right that sits behind it so you got to get really kind of good reliability you going to have um to a certain extent dos solutions kind of built in so you are kind of weighing the pros and cons there and the final one that you mentioned there go on Sid jump in no to add a little bit of color to the the cost uh to the bandwidth cost angle Cloud providers have almost like a 97 to 99% margin on the ESS bandwidth charges that they have so it cost them almost nothing but it's more of a strategic play for them to like lock the workloads into their clout platform essentially so that's kind of why bare metal providers are able to like offer that at a significant discount because underline cost is basically nothing yeah so that that that's you know where the if if you can balance it correctly you can get the bit

of theit Bo of Best of Both ws and that's where the hybrid solution kind of comes in um for the multiply solution which is now run by unity um we were looking at a 7030 split to get the optimal climate cost r with so that's 70% of the capacity would be in bare metal and 30% that burst capacity um would be uh hosted in cloud and the one that you mentioned there is kind of peer-to peer now that is a bit of a an edge case piece because when we were talking earlier about you know what is the player experience one of those players is going to get a really great experience because they are the host right um but as we all kind of know um the internet connections that we all sit on at home can be particularly unreliable so you you kind of pay the penalty uh by reducing the costs of your

overall hosting because it's free you know you are not paying for a server instance but your player's experience might not be ideal um and that comes in in a number of different forms the first one being latency you will get a host what's known as a host Advantage um for the player that is hosting that server instance and everybody else is connecting in um uh but also from that is you know if it disconnects everybody's go to to fall down unless you've got host migration built in and then you're adding complexity so when we mentioned you know what is the developer experience what is the how do how how much effort is it to build these Solutions it starts off simple but adds in complexity when you have to think about things like host migration and the one kind of final um piece that sits between all of those um

is when you've got something where you need the security or the confidence that somebody can't change the game experience so if you've got a competition server as an example a competition server where you you want to ensure that there's not an aim blot uh running you know uh and I say ensure that's a really hard thing to do but you uh with a dedicated server you have uh additional control over what you will let do so you the server can say no to a player traveling at a million miles an hour across the bench and appearing over there uh because it can do validation against it um Sid anything else to add on on those kind of points no I I think you hit the nail on the head I would just kind of frame it as like maybe like two different axes though right there one is like the net code axis which is like peer-to-peer or relay or

deterministic lock step or server authoritative and then the other dimension is if you need servers then you know the spectrum of like bare metal all the way through like you know pure pure Cloud I there's some overlap there between the two but really they're kind of two spectrums and the the decision that you take on the net code kind of impacts some of the options that you have available on the hosting side absolutely see I see yeah and maybe that's actually that's actually very helpful framing there because I think what's also you know coming from the gaming perspective what types of games use peer-to-peer relay servers versus like you just talked about the and obviously I would believe that in a competitive game the competition server you need the competition server because you don't want someone to be able to play Jin and have infinite infinite ultimate right like in in League or something like that and so Sid maybe you can I'll lean to you on this but can you help distinguish which games with types

of games and genres use each of those options most frequently and maybe also throw in there a little bit of a curveball on talking about when you know games like Minecraft that you know where players host a server um where does that come from where do they get it um and are they player servers in the sense that you were talking about Stephen where the host the machine is the host sorry the host machine is the host or you know their cont acting with a with another cloud or server provider yeah for sure so peer-to-peer was extremely popular in the 90s um just because like like hosting servers was just ridiculously expensive back then um so if you look at like you know the original Halo game right Halo CE like you would spin it up you would you would host a match and like players would connect um to your server so you know the the definition of per to peer kind of evolves over time but ultimately what it means is there's no like you know

dedicated server somewhere hosted by the game developer right and so within peer-to-peer you can kind of you can have like relay servers that um kind of pass the packets forward things like steam relay or Photon Fusion are great examples of that um or you can have player hosted servers like Minecraft is a great example of this where I can go run Minecraft anywhere in the world like could be on like you know my laptop it could be I could pay a service like nitr uh to to run my player hosted server um but at the end of the day everyone's connected to like a server somewhere um and then you have the you know the more like dedicated like hey I'm the studio and I'm going to be responsible for the the network experience and the player and the computer experience as well and so even within that there's like a spectrum right you take a game Starcraft 2 for example right Starcraft 2 uses

what's called deterministic lock step which basically is a set of actions that the player is taking will get relayed to a central server and that server will filter out actions that it thinks are like cheating and then Reit the actions that it thinks are saying back to all the players right so it's closer to the peer-to-peer but like a little further along in like the anti-che and like you know some more um like credibility of the actions being taken essentially authenticity of the actions validation isn't this it exactly it's validation yeah um and then you know you have your pure like dedicated server options and most modern super successful multiplayer titles all do that um including you know Halo today to you know League of Legends to fortnite um and in the in that model what's actually happening behind the scenes is every player is sending their actions to the server um 60 times a second and then

the server basically compiles all those actions does the physics simulation and then REITs whatever relevant state to a specific player back to that player right so just because I tell the server I've moved from one end of the map to the other end server is like I don't believe like I don't like there's no action you can take that would make that possible so most games that are either competitive or or you know even a lot of like liveops titles where if you're you know selling skins and you want to ensure that players who are appearing a certain way in the game have actually like purchased that skin um are all have all moved towards a dedicated server model now the downside of dedicated servers is it ends up requiring a lot more compute and bandwidth on the server side and that's something the studio has to like decide if it makes sense for the business model that they're um going for um and so in a lot of cases um for like

Indies and even like you know some super successful titles the unit economics of dedicated servers don't make sense um like it would cost you more to provide dedicated servers for all your players than you would ever make from each player in that game and there is one other component that that that's jumps into my mind about that and that's about the scale right so you think of fortnite you think of Apex Legends where the this the pure number of players could actually overload the capacity that would be available if that was a peer-to-peer installation right um yes we've made great boundary uh great jumps in bandwidth available but there's there are still locations which have typical dsls so you're talking five Meg up and you simply couldn't provide the capacity for a 100 players if it was being hosted in a peer-to-peer situation and that was one of the drivers between um Titanfall

um one of our original titles that we supported um uh and that was they wanted Bots they also wanted that not only the the ability to to run large player numbers like they did with Apex Legends but uh when it comes to bots they wanted the compute to be able to augment the experience that would otherwise not be uh possible um within the a peer-to-peer or direct connectivity experience interesting I see yeah because in the peer-to-peer Dynamic you're basically relying on the players infrastructure and their home country their neighborhood their house specifically um which is think really interesting because you're kind of Outsourcing that to the player um to front um and very interesting it's kind of like vat tax um uh okay so we got some grounding on all of these different types of Dynamics and options you know bare metal dedicated whether on you know where

you're providing servers as you as the developers responsible for that experience peer-to-peer hybrid Solutions um but I would love to get a little bit of an overview of who are the actual providers of these services and they're all the big names that we know you know in the cloud space like AWS Azure gcp is public Cloud tensent as some may know is investing heavily in its Cloud division to provide platform tools to its developers and these are General Cloud providers um Stephen can you tell me a little bit about like four games specifically who are like the big you know kind of like the the companies that work with triaa games who are the companies that work with double A and who are the companies that work with Indie when it comes to um cloud services I think you know from the cloud perspective it's all of The Usual Suspects that you mentioned there AWS gcp um uh uh Microsoft suure and 10 cent if you're looking further or forth particularly if you want any kind of support in China um you know the the

China firewall makes it really difficult to be able to provide good Services um from uh AWS or gcp um within the China region itself um and it doesn't really make too much of a difference whether you're a uh an India daa or a triaa when it comes to those pieces um you know they all provide uh a great experience you know one of the more kind of standout uh items that comes you know how do you orchestrate those Solutions right so does the cloud provider have something where you can just go and turn it on like uh Sid's company does um so if you're of the smaller Indie and you don't want to have to go and create that orchestration solution you are going to be looking for a cloud provider that has a native Native solution for it so for

example Amazon gamelift you know you can go integrate their SDK you can go and put it all together um where there uh the gcp solution is more of an open source uh solution which is going to take you a little bit kind of more effort and you don't have somebody that's going to uh be able to answer your questions at a drop of a hat so you won't be able to as a as a AAA have that Enterprise support agreement so when something goes wrong that you can reach out and get kind of that that level of support and if you've built your own kind of solutions you know that's on you to support it's only going to be the infrastructure side of things so you know to answer your question I think um you're going to see the same providers used across all of the range it depends on exactly what that particular customer is used to being created what Solutions do they have and then yeah guess what about the

bare metal scene um I don't know anything about those players uh Sid I know that you're working with a lot of them yeah so Alex when it comes to Bare Metal providers there's a lot of different varieties that come in but you know equinex is one of the biggest names in the world um and they handle the full spectrum of of services like you could R col space from them where you can like purchase the hardware yourself and just they'll just re lease you a rack um or you know as you get further along the Spectrum you can start leasing just a a server in a given rack um on a month by month basis so you're kind of like you know it's it's closer towards the cloud model but slightly longer terms but at a better price then funnily enough like the cloud vendors also offer bare metal instances which are basically the same same thing as you know they just don't have the the the hypervisor running right but so some of the in between options I would say are things like are companies like i3d servers.com volter

Zen layer you know all these companies they have their own Colo spaces and like they purchased the hardware and racked it and they're ready to lease you a server at a time for a month-by-month basis and those are the most interesting ones for a lot of game studios because you know it it's kind of that sweet spot of like operationally they're not as hard as going and leasing every uh like purchasing land and building everything but on the other hand you know they still give you a significant Savings in terms of like you know bandwidth and compute got it okay that's some really good context and then briefly would love to touch upon you know you you spoke a little bit about what haora is doing and Stephen you kind of pointed to oh you need a partner like haora to turn on the bare metal um where does haora specifically sit in this market and then Stephen we're going to go to you and talk about where Rock size specifically sits in this market um so Sid how about

you kick us off cool yeah so you know we we're a very targeted part of the stack right like we don't we don't have a lot of like broad services for example like matchmaking identity and authentication that's all stuff that like we integrate with like other providers but the part that we focus on is all right once you have a pool of players in a given region that are ready to enter into a match and you want to tell them hey go connect here and you can start playing that's where we come in right so your matchmaker is going to go through the whole flow of saying like Okay these five players are you know ranked approximately the same and given their like Geographic spread like Tokyo is the best place where I want a server spun up for them so the Matchmaker reaches out to haora and then haora basically says all right like you want something in Tokyo and like I know what you've uploaded and what we need to start here is that server started up in under

5 seconds and here's the connection details that you can pass back to your players to connect to right so that's like the rough flow of of activity but when it comes to like behind the scenes like what's the magic that that's actually happening there's a lot of concerns around like I well where do we schedule this game right like is it on bare metal is it on cloud like do we have enough Cloud instances do we need to scale up more Cloud instances um you know and when it comes to Bare Metal actually like provisioning and like procuring the the metal and like enrolling them into our fleets and patching them and keeping them updated that's all of the responsibility that haor takes so to our customers or our you know the game studios or the developers at those Studios it's this super seamless experience where all they're doing is they're uploading their build over to us telling us okay this is a rough like percentage split of bare metal versus Cloud that I want and then

integrating with their integrating their Matchmaker into haor and suddenly you know they they get all of the player experience benefits that we've been talking about and the cost benefits of you know running this like hybrid environment interesting okay got it so you're kind of like a almost like a a little bit of a middleman between some of the the cloud service providers where you know as a studio would work with haora and then haora would help them basically get to players in the best way possible whether that's through a combination of bare metal and cloud or just Cloud only because it's in Tokyo or something like that is that summarily kind of accurate yeah yeah it's it's pretty close all right um the word I'd use for it Alex is orchestration solution right they're taking all of the complexity at the lower level bringing it up and providing that level of SDK abstraction to be able to just request a capacity where it is

needed what what do we do over at rocket science we're more around um custom Solutions than what's it uh does over aora so um if you think about publishing platforms which has got all of the components from Identity to economy to matchmaking um that's where rocket science really kind of specializes will come in we'll help build some of these kind of custom infrastructure components in a way that we know will scale to the ultimate demands of uh of a game and one of those kind of those that you only ever get that one attempt at a game launch right you don't want a solution which is going to have any problems at launch you mentioned boring launches and and that's kind of where rocket science comes in but we're actually a a company that is is is has got two components to it so rocket science has two sister compies so to

speak underneath it which uh what terminal velocity is the one I've just articulated around publishing platform backend Services whereas Atomic Theory actually sits to the other side so that's more about uiux ingame components um being able to work with the unities and the unreal engines uh to to create these game experiences so that's where we very cool okay I got it so haora is like for people who want to do General stuff and rocket science maybe for people who want to get a little bit more particular a little bit more specific and you guys kind of would work alongside and be that kind of backend team partner for another game Studio yeah absolutely awesome all right so we've got an overview of cloud we've got an overview of the providers um and now we can finally shift to our favorite topic which is business models um and so I'm going to take a little bit of this here because I worked on a uh

Enterprise deal with Google Cloud um at Activision Blizzard King a three-year multi multi- business unit multi-vertical deal um and so I unfortunately know a good bit about Cloud credit programs burst models p&l and cost classification of different types of cloud um and you know we talked about this triangle of predictability cost and flexibility and I guess I'm going to give an overview of the standard models that exist though these are not necessarily tailored to games um and I think one of the really interesting things here is that because so many of these companies like Google like Azure like Amazon um are public Cloud providers A lot of them are actually not optimized to be a win-win solution for businesses in games um and it's actually rather inconvenient for games at times um but the standard cloud computing models as I know them are obviously infrastructures of service this is something like AWS and ec2 you know the provider gives um a basic

Computing infrastructure IT professional setup for your instances and network structure and you can remove or add them as you need the benefits are obviously better security network um Electric System reliability and the cons are though that you need a skilled IT staff and they aren't your servers but you have to act as if they are um then there's platform as a surface which something like um Amazon elastic bean stock which is the Computing platform is provided for me my company is responsible only for the application software running on that platform and the benefits are that you don't have to set up your own network or config or your own computers and you have much less need for an actual onhand IT staff but you don't have access to many of the operating systems involved I guess to me it's maybe more like blackboxes um then there's server lists um which is I don't have any virtual computer inst I just have the functions I want to execute um I pay for the compute time that I consume um and the

cons for this is that it's not really very good for anything with strict real-time requirements um so games probably don't use this obviously because we just talked a bunch about having servers um and then there is the software as a service um model which is the company's providing software to people as a service obviously this are something like Google Docs but probably pretty self-explanatory probably should skip this um um so putting it all together here uh Stephen you pointed out at the very beginning of this episode there is this Chicken and the Egg problem um if I'm a game developer I don't know how successful my game is going to be I don't know if it's going to be a hit I don't know if it's G to underperform so can you tell me a little bit about which of these business models I should prefer and when should I prefer them exactly it very much it does depend but if we were to focus in on the scaling of that virtual AstroTurf where we we we've we've talked talked about um quite a bit in in this

episode the key one is being able to control your cost it doesn't matter whether you're an indie developer or an EA running Apex Legends you need to know at the end of the day that you're going to be profitable in the solution so understanding your costs is the key piece now um if you are um on a very very small scale and maybe you're just current developing your game um being able to spin up a cloud instance for 5 10 minutes to run a test and then turning off is likely to be one of the most coste effective Solutions but it only it only stays cost effective um when you're only using it for a small portion of a running Deb for example um now we mentioned earlier around the 7030 split and that's you know when you are using compute consistently over time um

you want to be committed to uh that compute because you can for that you can reduce your your overall costs now there's a couple of different ways that you can do that um one with bare metal where you're committing to a month by month instance or it even in compute there are committed usage models within that um at the end of the day it is that kind of balancing thing that you need to do you need to understand what you're using for it how much of it you're going to be using and what time period of it that you're going to be using so if you've got your game spikes up to a 100,000 players at night but it only stays there for two hours and then it's back down to 50,000 well that doesn't make sense to put in a large amount of bare metal to cover that because that bare metal you're going to be paying for for 24/7

365 or or 24/7 for the month right yeah you just use that to burst yeah use cloud to yeah yeah exactly so you do want to look at that overall profile uh uh of when you need those resources but similarly the the other complicating factors like shipping your updates now one of the components of the orchestration system similar to Hora is going to be that distribution of those images so you're not just talking about the computer running it you're talking about the network bandwidth you're talking about the dis storage you're talking about shipping your game title to each of those instances and being able to maintain the updates as well yeah I think one of the things that I think is pretty interesting is you know you're talking about you know bare medal kind of being that first line of defense um and I think one of the other interesting things this kind of folds more more into the um how how profitable a game p&l appears is that capex

dedicated servers right are typically considered um below the gross profit line whereas Cloud fees are in cogs so they directly impact your margins um and so as a game evolves to becoming a little bit more stable consistent you know you talked about that 50k of concurrence spiking sometimes to 100 KF concurrence once you have that reliable data a lot of times the strategy would be to shift off Cloud to a dedicated server model so that you can put that stuff in capex um increase your margins to float a better stock price Etc because that stuff matters to the street um and all Cloud costs are considered variable and so I think this is actually something that's pretty interesting because in the beginning of a of a game's Journey variable costs are probably the best because you do not know what you are going going to need however if you OverWatch League of Legends Apex three years out from launch

now you have a very consistent reliable ma or da that's predictable you'd shift off cloud and you'd move into capex um to improve you know basically p&l Health um so I think that that's actually really interesting and that's why I love Cloud because it's a very cool touch and go between Finance business and and Technical infrastructure um no it's it's really funny what like you're saying and how much it resonates with our journey at haora so when we actually first started the company we wanted to be the serverless platform for these multiplayer games right and existing options like inbs Lambda or Google Cloud functions like like you're mentioning they have limitations that don't make them great for gaming and while from a technology perspective we were able to solve all of the problems and Studios loved it while they were development there was a major set of concerns that started coming up right around the

launch time right and a lot of that comes down to a lot of these serverless options are extremely high unit costs right if you look at how AWS Lambda is priced compared to how ec2 the underlying ec2 equivalent of that is you're paying almost like eight times more per vcp hour if you're actually exhausting that node consistently right the the advantage of server list is if it's bur SE great like you know someone else is taking on the risk of like the unutilized compute essentially but someone has to pay for that somewhere right and so when we started talking to Studios about like launch today volume and then you know starting to put together estimates on our serverless offering they're like oh my God that's so expensive and we're like okay well what if we shift to this more like standard um like compute uh reserved compute model and everyone felt happier in that model we felt happier as a business because we knew we had a more

predictable amount of compute that we have to spin up and we weren't taking on any of the risk of underutilized instances and because we weren't taking that risk on we were able to fairly price the the the the product to our own customers um but you know imagine a world where we didn't know and we we spun up like you know 100,000 instances anticipating a big launch but you know it's a it underperformed and we only get paid a fraction of that right that's the scary situation for us that we need to protect it we needed to protect against and so to do that you basically overcharge for what you're offering essentially yeah when you're when you're looking at the the scaling equation you're always looking for this safety buffer at the top right if I've got 10,000 players online I want capacity for 11,000 players right or 10,000 uh 20,000 players depending on the velocity that happens right so one of

the challenges when it comes to uh building an effective orchestration solution is giving users the right level of flexibility to go actually from my title I know these facts about it right I know that I'm going to have short match times so when I've got short match times that's going to imp impact how many instances that I'm going to cycle through on an hour by hour basis and what does that mean um so providing that level of flexibility um both from the technical solve but also from the business model solve um is really really crucial uh and it does come on to some interesting additional business model that you'll see from people like Google Cloud where they do offer the decaying discount so to speak so where if you have used a larger

amount over a period of time you start to see a discount kicking in um and that's uh a really interesting component that does play uh into the complexity of the equation yeah yeah and it's also hard to track um there's all of this you know I think when the more complicated the business models are something you have to consider is actual just governance of that business model um so you know ratchet scaling fees um different types of tiering um stuff that you know kind of gets very complicated and then you need like a full-time you know you need a full-time deal team on one side and then the other side to manage it and be like okay well you paid me X but I actually should have paid you y and this is the data that I have collected on my side that sees this and then the other side says something else um so it's actually quite it's really quite interesting that the um although some of the business model models need to be tailored to games to make them you know the you know most financially fair to both the server

provider and the server orchestration side and for the studio um that does come at another cost which is it's complicated um but our final yeah management right um so our final topic though is you know talking a little bit about the bare metal business models um instead I'm going to lean on you here to close us out you know you've been working with a ton of them to lay out stuff for Thora um is there anything really here that's like you know very uh interesting to you as guess as a server business um that you think about pretty actively in terms of which ones that you like to partner with and why yeah for sure so first and foremost as a company we put player experience as the most important thing that we should care about right and when you think about bare metal providers there's a huge spectrum of them and there's many dimensions of these spectrums as well but what of the spectrums is like well how much does ESS bandwidth or how much do they charge for bandwidth right some providers are like hey you buy the

compute you get free unlimited bandwidth but the subtext there is you know the bandwidth is capped to you know a one gigabit per second Uplink or you know it's actually they're just dumping traffic onto the cheapest provider that they can find and it like the it hops over all over the world before actually arriving to your players or the routers are so poor that it drops half the packets that that are coming their way right um so we've done a crazy amount of benchmarks on network quality and we've basically have like a ranked list of providers that we feel excited working with and we've only really worked with the top three in that list when it comes to like what's actually enrolled today so Network performance is like one aspect and then there's the other piece which is like the the term length that you're acquiring these leases for right so some bare metal providers have actually straight up gone into like

competing with Cloud vendors and they're like you can rent by the second from us right like equinix acquired a company called packet and launched this offering called equinix metal and you can hit an API and be like hey launch me an instance in this region and start billing me by the second and if I don't need it two hours later stop billing me for it the problem there is the the compute costs are almost as is just launching on AWS or gcp with not a much better savings on bandwidth either right so then and then on all on the other end of the spectrum you could literally go acquire the land you could go you know build a data center cooling power internet cables but there's like two other like kind of intermediaries between either extreme right so one closer towards the like I'm going to do everything myself and I want more capex instead of Opex is the I'm going to rent Colo space what

that means is some other company has done the leg work of like building the the warehouse and like the server racks themselves and it's set up such that they have people on hand that if you ship them Hardware the people on hand will go rack them for you so you will pay Intel AMD you know all your like RAM storage um directly and you have to pay that up front unfortunately whether you get use for that over a three five year depreciation period or not but what that means is you know you can get them if you do have good sustained use for it you'll end up saving a crazy amount of money the challenge with that still is even if you have these like remote hands is what it's called like actually kind of racking them for you and if there's any hardware failures they'll go fix it you're still on the hook for like managing the software that runs on it everything from the operating system to like you know all the packages etc etc and it's a lot more operational work so

then the next sweet spot is kind of like the providers that we mostly partner with um and they're the ones Steve has been alluding to as well you know the i3s the G of servers.com of the world and what they do is they kind of like go through all the racking they purchase all the hardware as well they say hey here's this server this one costs you $300 a month to rent um and at the end of the month you can say you don't want it anymore we'll take it back but for the period of the whole month you can't give it back to us MH um and that usually is the most popular it's the thing everyone thinks of when when they hear bare metal nowadays because most companies have kind of lost the the muscle for kind of like like racking Hardware on their own essentially I see got it okay so it's very uncommon now for companies to be doing um I'm just thinking of the Silicon Valley episode where they build their own ser and they like bring it in a truck somewhere um no one's doing that anymore

basically is what you're saying very very few companies are okay got it yeah like obviously the Amazon and the Googles and the Facebooks of the world are but it it's become very hard to compete against those companies for the talent that you need to like build an inh house from your you know how your accounting works like you know you're you're you have all these like assets on your books that's not directly tied to the alpha of your business and there's all these o Open concerns people have that made Cloud a lot more appealing but there is some like push back or some like you know rethinking of like hey is cloud actually giving me the savings that were promised that's starting to come up again yeah it's it's definitely a question for sure because it is expensive right and actually when you're talking about you know the all the way to the right you know I'm going I'm getting a land lease I'm building the warehouse I'm racking this I'm racking the racks I'm putting the servers in there um heard about there's actually a

pretty interesting company that is it's actually a real estate company that's renting out the empty space inside Office Buildings ele think elevator shafts um ceiling Hedges right to hold servers that have like nothing to do with that company's building so let's just say I'm uh the Uber office and Uber office has a bunch of empty space for like airvac Etc and can you rent space inside of a building to store server racks um so that's kind of like uh you know you mentioned that and I started thinking about that um Stephen I don't know if you have any final thoughts on bare metal business models yeah you know at at the end of the day we we did that multiplay right uh we did do the racking and stacking uh of Hardware you know we were purchasing super micro oneu servers installing um Dell blade units installing them um we had great success um running Minecraft

servers uh and all of those kind of Bey with that model you know having our own dedicated 10 gig lines Here There and Everywhere Frankfurt Amsterdam London with all of the peering Arrangements that go with it you know Sid mentioned you still need a really great um networking solution at the end of the day you can't be handing it off to the cheapest provider because because they will root you badly they will put you on an oversubscribed Network Le between two points um and that will result in in a poor player experience so yes there's definitely that real kind of range of equipment and where the industry sits today you know the I 3DS the servers.com of these worlds is if you want that level that's really where you need to be playing it's uh a lot more overhead than anybody would imagine to do the thing yourself from the scratch like we did

many years ago got it um well that's great context guys and team we are we are out of time and honestly I wish we could keep going um this has been super interesting to go over the overview of all the different options across basically getting games in the hands of players in the Modern Age um really appreciate you both coming on and sharing this um and your expertise with the audience uh if anybody wants to get in touch with haora or rocket science for services uh how can they do that yeah um they can message me uh my email is Sid s.dev yeah and from the rocket science side you can contact us on our website or my email address is Steve rocket science. awesome and guys uh so that is our episode um as always friends if you've got feedback or ideas you can hit me up at Alexander nvic.org we're out see you guys next time Sid and Stephen thanks for coming on thank you

cheers if you enjoyed today's episode whether on YouTube or your favorite podcast app make sure to like subscribe comment or give a five-star review and if you want to reach out or provide feedback shoot us a note at podcast nov. or find us on Twitter and Linkedin plus if you want to learn more about what navic has to offer make sure to check out our website . Co there you can sign up for the number one games industry newsletter navic digest or contact us to learn about our wide ranging Consulting and advisory Services again that is www.nic.edu [Music]