An Executive Guide for The Backend Jungle

[Music] welcome welcome welcome to the deconstructor of fun podcast and in this episode we're going to talk about something that I have no clue of which is backends and and to educate myself and the audience we have uh Rafael Cordoba a principal developer at Bandai Namco Barcelona formerly Tech lead for Anger Birds 2 where we actually met at roio Rafael welcome to the podcast hi Mishka pleasure to be here hope we can explain all this uh in this podcast yeah back end for dummies no this is at least the first section will be back in for dummies and then we have Chris maloian chief executive officer and co-founder of heroic Labs who knows a lot about backends Chris welcome to the podcast hi yeah thank you for having me Misha good to beome welcome back to the podcast you were here before last year with Ethan yeah yeah yeah yes let's see if let's see if I can do you guys

Justice So today we're going to talk about backends as a service and and I'm taking the angle of of something that I've experienced in in my work as a head of Studio where I was often the one who would sign these backend and service contracts and sometimes when when you would do m&a like that was a part of an m&a when you're looking at the backend Services of of the uh the target company and I knew very little and still know quite limited about backends but I every time I knew when when something failed when the back end service didn't scale when um as as Engineers would put it was spaghetti code and everything was was uh very much to me like a black box and so I've spent some time understanding about backends and backend as a service and this podcast is more meant for for people to to understand better what backends do what is a backend as a service uh whether you should be investing in building something of your own custom or whether you should be

taking backend as a service and what are the limitation of these Services what are the drawbacks when you're choosing between different options so um without further Ado let's kick off with the most basic of the basic of the questions is like what does a back end in games actually do and Rafael if You' be willing to uh to to start there yes of course let's start answering your question what is a client okay this is the client your device is the client and the client talks to the back end now the back end is somewhere in a cloud basically a server it's in Amazon or Microsoft or Google there are multiple providers that client you know you will have a game so I'll give an example you install a new game you know in the App Store you play some levels you get some rewards let's say you get get 300 coins and then you want to go

and upgrade a character you want to go from level one to level two so now you have a character level two you are at level five and you have you know 150 coins in your wallet you want to be able to actually save and persist that information so if you decide to uninstall the game install again or you get a new phone or you are playing multiple devices maybe you have a tablet you want to be able to transfer that you know information between all these devices and and this is what we call persistency it's one of the first functions of a back end and there are two ways that you know we can persist things in a back end there is the easy way the shortcut where usually games like hyper casual will choose where whatever happens on the client or your phone we just inform these back end and say something like hey I have a

character that is now level two I am at Saga level five and my wallet now has 150 coins that backand is not going to know about anything of your game logic is just literally going to take that info save into a database and doesn't care that is not good for some of the more casual to meet core games and hardcore games because you want people to not be able to cheat you know and not manipulate the memory or say oh now I have 1 million coins so the option two is the back end is actually aware of your game design so it knows that to upgrade a character from one to two the cost is 100 so it's going to perform that logic on the server and deduct the same amount of coins you know uh from on your wallet so this way the players

cannot cheat and and it's more similar to you know like I compare to a bank you go buy something and you pass your phone or your card then the bank has to say minus 50 EUR you know and if you could say oh it's actually one Euro then you don't want this to happen so depending on the game you're doing you can choose one or the other and they both have pros and and cons you know so that's the first one yeah the persistency that so that's a that's a thank you that was a very uh clear explanation of the basics of of the backend um but it seems like as of not of as of late during the last decades like more and more functionality has been added to the back end uh and you know the teams if it's an internal backend development have been growing significantly and it's not only just uh just making sure sure that people don't

cheat or that they can play on multiple devices it seems like almost every year there's new functionality at it especially if you've had your internal uh Tech teams they've been always eager to to add anything from liveops capabilities to to to things that can boost um even your your marketing effort so uh maybe maybe going to to uh to Chris like how has it G like grown from uh what Rafa explained kind of like the basics the the the most important parts that we need from backends to the sort of a en all-encompassing service from matchmaking to scalability to to all these elements uh during the pretty much like last decade or so so it's a good question I would say that the time I spent recently talking to some of the engineers Activision King they gave me this extremely useful um representation of two different strategies around back end right you've got what we call solo plus meta that's what Raphael's talked about that

information about your progression in the game World it lives on the server and the server decides rather than the client then you've got social plus meta and I think what you're talking about michig is what we'd call social plus meta right where it's not just about where the decision about progress for that player lives it's about how to play with other players right it might mean multiplayer it might mean event leaderboards it might be competitive features that might involve matchmaking it might be friendly L and Guild systems and social mechanics all of those things they lift the bar for what players are expecting year on year for what every game needs to include within it and we've been on The Cutting Edge of helping Studios with that for quite a while and there are simply today very much some of the best games in the world they will not a new game coming into the market will not be as successful as those games without doing at least as much as they do and they are

always pushing the bar out and what we would call that social and competitive side and you can't do that on client only right if you go to that solo plus meta you could make the decisions happen on the client but you can't you know you can you just can't build the kind of gameplay experiences without the server today um that you see see in some of the best games in the market yeah yeah very very much so and the games have evolved quite significantly in the last 10 to 12 years especially on mobile like going from quite basic you know the the the candy crushes of the world to uh to massively multiplayer titles um yeah I mean if you take if sorry to interrupt you take if you take um Candy Crush that's your quintessential example of solo plus meta if you take toun Blast from Peak games that's your example of social plus meta right they're both built around similar core game mechanics but one is fundamentally about interacting with other players and grouping your progress together with other users online and the

other one has taken probably because of his history in the market a route towards that where now they're social but they started with sort of solo and meta features first yeah and because the the games have evolved uh we have seen pretty much a a massive rise of backends as a service I've um I've actually you know educated myself talking to uh and doing some surveys through deconstructor fund with uh with dozens of uh of CTO and tech leads and and asking them actually like how many of you are using backends as a service and people were coming from multiple different um size of Studios from small Indies to actually uh AAA Studios and what I found out is that the vast majority if not almost everybody not depending on the size of a studio is using backend as a service so so my question is kind of like what has caused this where where Studios have willingly move to Outsource essentially not

Outsource but uh how would I put it yeah like maybe it is an Outsourcing of a develop maybe yeah maybe maybe starting with with Rafa I think Rafa you worked both with the with the in-house back end as well as uh back end as a service yes uh what I think explains the rise is a couple of things okay one is there was also a rise in startups Indies people that actually left big companies you know including you like you you have been there you left a company you've you've built a startup and there are many others who done this you know I have friends in Stockholm doing this you know right now you know them and when you are three to five people you know it's very hard to go from an in-house tool that you were used to work but you can't just take that tool and use it because you know you don't have the right to do that and coming up with something front scratch setting up a a tech team you

know creating everything from scratch for your first game is very time consuming you need resources and you need to to have people to maintain the tool so that makes it much easier for startups to be focusing on the games and and most importantly time to Market is very important these days if you have a new idea you got to be really fast you want to be the first you know you want to get the kpis as early as you can and the you don't want to waste too much time uh with that infrastructure or creating all these and once you have two or three games you also need to to have the people to to handle these you know and to to to support the your inhouse too in two or three games so you make sure things don't break and uh you know and and you and the two you've built can scale because one one of the first mistakes usual developers do is I can build the best too I can do the the you

know like it always feel like you can get something better but the reality is some of these tools on the market they have been there for many many years and all those years are learnings so I think right now companies like Bai you know we are not small as a startup but we are smart with our time so we are focusing our resources into the game and what makes the game fun and then we have you know tools uh as a service like back end is a service to to support us do those games as fast as we can and be able to scale uh Chris what what are the other elements that are that you're seeing are influencing uh the you know basically people becoming your clients so good question I think Rafael hit a extremely strong point which is that when you are a small team trying to compete even with a huge amount of accumul ated experience inside larger

Studios you understand quite how large and undertaking it is to achieve the kind of infrastructure you need to be able to compete with some of the biggest games in the market so how do you get there you need to work with some kind of partner to do it uh it makes the most sense um I think that's one element I think the other element is that the industry is may be going through a similar kind of Journey that it did with game engines right um for a decade or more every studio built their own engine in house and they invested a huge amount internally that was budgeted in and that was worked out um and then as the requirements for engine development grew and the requirements for Hardware the capability of Hardware grew we ended up shifting in a direction where well there are Partners in the space you know Unreal Engine Unity engine who make this their entire job and wouldn't it be better to be able to leverage that cumulative progress that they've made

and build on top of it rather than trying to build from scratch catch up to them and then and then own internally everything that goes on with the maintenance what what I call that total cost of ownership of actually taking on that burden inside the studio so I think that's kind of a similar Direction at the rest of the industry is going in for kind of game tech for Server based game Tech yeah it makes all the sense like raf's kind of key word there was Focus so time resources everything that goes into that and then understanding it's not only that you're going to build the back end and now you have it but there's a lot of a maintenance there's further building on it uh and also Focus means that as Rafa said as a game Studio you need to be focusing on getting to Market and that means you need to focus on building the product and that means you need to focus on iterating the core gameplay iterating on the fun and putting in you know the the main efforts as a as game developers not as as you know uh Tech Builders um and and something that that

I've heard from from other you know talking to other tech leads like the the challenge of a internal back end if you want it to be competitive is that the uh the feature set is constantly evolving and that means that you need to put more and more people on it because as as Chris as you said there's the element of not only the the solo gameplay but also the meta so you have to put in those and the socials the leaderboards the guilds the matchmaking like you you name it like those elements are con evolving and they're coming more and more of those so you need to be putting them as well and I'm curious to ask is like how important are the elements of of uh of crossplatform play scalability uh having all these analytics sets like are those important for when you're looking as a at at backends as a service um maybe Chris as you are involved in many clients good question I think the analytic side of the problem space is something that it is its own um domain that needs solving right I

think any any centralized uh set of tooling that tries to solve too many problems at once uh especially in a single product is just going to end up doing somewhere a disservice to what you really need especially in the sophistication that you expect in the games industry so I think um analytics is its own problem space sort of the um uh aggregating the UA data putting it into a data Lake and analyzing monetization efforts introducing uh liveop systems and how they interact with those monetization efforts are almost a a domain problem of their own um and then you've got that core infrastructure the game backend itself that Raphael mentioned that covers persistence you've mentioned social you've mentioned multiplayer um that are their own problem spaces um so I would say that um solving them all I almost you know unfortunately is necessary to re reach escape velocity in

the industry today um but solving them all in one place and expecting them all to be done to the highest standard um is is going to be difficult to find and I think for us the way that we solve these problems are we dig down into specific areas the core backend what we think of as core infrastructure we solve those problems and then we move to solving other problems in different products in different tools because is it won't all fit in one place in in my humble opinion understood um Rafa do you have something you at or or should we start talking about disadvantages of backend as a service we can start on the disadvantages yes well let's talk about it so you having worked with both um what are the disadvantages when when you have an external service provider in this critical part of of of your studio yes I think the biggest disadvantage and you know we we talked

uh in between the coders uh at Bandai is ownership of the code being able to have access to source code you know being able to customize things if needed uh not all of the providers have source code available and this can be a deal breaker for many of the you know bigger Studios with that that comes from you know inhouse tools and and so owning the source code having access to it it's really important uh with nakama it's open source so we were very confident of the quality you also have a community you know uh looking at the code as well as you are uh so the source code is one of the disadvantages the next one for us that was pretty high on the list was customer support and response time how fast can I talk to someone and they can

understand my problem and we can fix it because when you're developing a game and you're not live you can take as much time as you need on once you have a game that is making you know 100K a day or more you have like a top dring game and there is something broken you need to be very fast you know so uh that jumps into the service agreements and resolving issues every provider have different tiers and pricing if you want them to reply in two hours or 24 hours or 48 Hours you know so depending on on who you are dealing with you know and and the contract you chose uh it can be pretty slow or it can be very fast you know and you have to balance the cost with the speed and and and consider if some of these are deal breakers for you and and it highly depends you know provider to to provider so this is my our take was basically this ownership

source code customer support and getting things fixed uh fast if needed how about things like uh like feeling that you're locked in to a to a service provider because this is an essential Service uh I feel like it's not a plug and play to just switch a backend service provider like is is there a fear of dependency yes there was like there was a few there was one Tool uh I don't remember the name that it was a complete Black Box so that means I don't know anything that is happening there I have no access I can't really take over and continue so if they go bankrupt or if they get acquired you know or or if they shut down I wouldn't be able to do anything I would have to find another partner and do the whole integration you know and and we we we can talk more about these

uh later but basically yes we when we we chose to go with heroic Labs we knew that because the base the Bas server is open source it's ours so even if they go out of the game we can continue our own game without having to uh change dramatically you know so so yes this was on our our list for sure and and finally like when when you're thinking about costs like how are you so you talked about the uh the cost of customer support versus the speed of customer support but overall when you're thinking about the cost of back end is it like again talking to a lot of uh tech leads like what I understood that the even though the service providers like heroic and others are very transparent in how what the cost is in the end the uh the size of of your game the size of live operat the amount of

calls like how server heavy it is will play a significant role in the overall cost of your game and so I'm wondering is like uh how how big of a how big of an element the cost is we did some many spreadsheets you know uh with the cost of you know you have 1 million da 2 million 3 million so to us the pricing today you know of the market now they were pretty competitive they were pretty fair that you can make a top 100 game and still have a reasonable cost you know I think the the advantage of saving cost will come longterm but this is going to come when you're very successful you know you're not going to feel that uh before you're hitting you know I know let's say a 100 million a year so you got to have quite a lot of success to to be able to you know and

more than one game to to to say yes we're saving a lot so for us uh the the advantages they they were much bigger than the hit we would have with cost you know it wouldn't make or break our game uh because of that cost because they are pretty fair these days so Chris when you're talking to potential new clients like what are these elements that they bring in as as potential disadvantages in back end as a service so cost estimate is tricky um rapael did mention that you typically do need to spreadsheet it out um we will help gain teams with that problem as well help build them a cost estimation model that fits a kind of ma or da or CCU depending on the game type um that they're looking to build and that kind of gives them a stronger sense of where the cost moves as they scale um so that's important um depending on who you talk to this providing game Tech some of the cost estimation modeling can be really hard I

mean we strive to be as simple as possible but some of it can the devil is so in the details that you're unsure how you could possibly know ahead of time how some of that cost is going to materialize um but Raphael is right that I think generally um the market is is is doing the right thing going in the right direction U on that front I do think that when you're evaluating a game back in I think something that can often be perceived as a disadvantage but it's just difficult to figure out in general if you have what I call known unknowns and unknown unknowns right so when you're kind of at that phase you know we could refer to it as window shopping you're at that phase of you have some sense of what your game is going to be but like any creative process it's going to morph and change over time right as you try to find the fun right try to find that sweet spot that players are going to love playing and as you do that you have a sense of what you're looking for that you know you need and then what you don't know yet is things that you might need in the future and that might be

where you get tripped up later right you start working with some kind of game Tech provider they have what you need today and you hope it will evolve fast enough with you and depending on who you're working with it it might not it might be a very tricky proposition and then later how flexible is where you're at to be able to bolt something else on to achieve that outcome bolt some other feature on that they they couldn't otherwise provide to you um so that can be very tricky yeah I think that's one of the the things that comes up what we speak to Studios about is they look for this as wider pool of things to be solved for them from day one immediately now because they're just so worried about what direction they might need to morph in and it makes makes sense it's a creative process yeah that's that's the uh that's the part of the uh the customization and how important it is as you move forward yes uh I'm this is actually interesting so you know in the news we've we've seen a lot of games having challeng is when they're scaling uh nowadays especially

on uh on on triaa front um and I was wondering and again not not only wondering just talking to to a lot of uh tech leads and and trying to understand where these come from because it's so easy to in the end say that oh it was it was the problem with our backend service provider but to my understanding when we're talking about these big project it takes years and years to actually go from idea to concept to playable to actually shipping and scaling that game how important it is to keep that dialogue with your back end of service provider because what you had in your design document uh you know having been at game studios for about 15 years is pretty much never what what you ended up shipping and so when you design your backend to to uh to answer those questions and and support that game that's actually a different game that that finally May May launch and and begin scaling so how do you kind of like

avoid these type of a problems that that occur throughout the production so for us we try to be as involved as we can with the game team as early as possible um we just simply try to offer as much time as we can because as you say things are going to change things are going to evolve and the more we can be a partner to the studio where we're operating more like an extension of the develop team than simply a vendor who you communicate with for support tickets right you know the closer we can be to the studio team um the better sense we get ahead of time on the direction they might be taking some approach they're taking to matchmaking or something else and try to sort of advise or steer or adapt as we need to that's quite hard we find it quite difficult because uh some game teams they simply don't they don't really want to talk to Engineers who are not in the studio team um so we actually find that a bit difficult at times to be

able to help but that's what we would prefer to do as much as possible wherever we can we'd love for Studio teams to see us as an extension of that team uh in a way that means that we can help them uh when they evolve their ideas and morph what they're building make sure that we can help steer it um to to the way that we have seen work without game Tech yeah ra do you have you seen this as a as as a challenge where what you were building is not exactly what the uh the end product that you're going to ship and then that information hasn't been portrayed to the back end whether internal or external team and so problems are are starting arise there or even the costs are higher because it was uh designed in a different way than it was originally um thought about yes like it's very hard to predict what is going to happen at SC you know when like it's only experience that

allow you to to to to have a good feeling of you know potential issu so as as a programming team you really do your best to make applications both on the client you know your computer or your phone but also on the back end but despite having some tools to do extensive you know load testing where where you you throw millions of requests and and you see if everything is going well well certain things are are just going to be learned when you're life you know so for triple away it's really complex if your game is very hyped you know like blizzard games and all these games that are in day one you have several millions of of players playing the game at the exact same time so so that is a much bigger challenge than a mobile game where these days the scale is very slow you know you you release a game in a country you have technical launches you your UA campaign you know

the money you put is what brings the traffic so you very unlikely a company is going to spend you know so many millions in UA so you have so so so much players that you can't handle so on mobile is much controllable and and you can you know scale and you can react and you can fix things on Triple A yes it's very hard even blizzard has been having issues you know they they've done this since 2004 on World of Warcraft and every new game you have survey issues so uh there is no answer to that it's like it's uh you do the best to prevent it but at that scale uh yeah it can break things because millions of people are different than a few hundreds you know play testing your game and Chris what action should like you you of course you work with with I don't even know how many clients but I

would assume hundreds exactly so it's impossible to to be in tune with with so many and that's why you have the U the open source approach and so forth but but how do you like what could be the ways to mitigate the the the issue that may arise by the studio not communicating with the back end of the service provider um before it becomes an issue like like is there uh is there perhaps um some kind of a design or meetings in advance I think design overall is is is the the place that we strive for I think you're absolutely right um for us we've tried to design our technology uh such that as to as large an extent as possible uh the way you use it it's opinionated right the way you use it means that you will land into the most successful outcome the outcome where you didn't have to think about the scaling conditions the Locking impact the you

know performance implications of when you're Live for using that feature um so we've strive to design at a technical level the the structure of the code that a studio writes so that they have less to think about um uh when they're actually ending up scaling in production so that's one route that we take um the other route is we've seen a lot of different types of games we've worked on a lot of different games sort of uh Mac merging Grand strategy midcore RPG we we worked on a lot of different game genres and so we have been able to see a lot of different emergent patterns at scale in those games and that just gives us an opportunity to know that even within a particular genre of game design that a studio is working in the core mechanic might change but we've seen the genre enough to know what scaling patterns end up emerging and we can actually help them uh get the right fit for that understood

okay well let's talk about uh in-house backend so uh what are the uh the advantages of of having an in-house because you know every everything is there so uh Rafa you worked with both uh talk to me about uh in-house yes so inhouse at rille they had this tool called Beacon you know which is the in-house uh back end as a service there are the biggest advantage is you are owner of the entire code base so you have nobody to ask for anything you own everything uh so this is one of the big advantages everything you do is your is is within your company and you don't depend on anyone uh the other one is Case by case again because you know at roio you have a Central Tech that needs to support multiple games so I was working on Angry Birds too if the game is doing good revenue

and you have an issue and the issue is not in the game but is in in the Central Tech you raise a ticket and it's all hands on deck you know everyone is going to jump on this immediately because you don't want to lose money you're literally losing hundreds of thousands a day and you want to fix this as fast as you can so you give priority as you wish if you have a smaller game with less Revenue maybe your Central Tech team is not going to need to react that fast you know so this is also something that um being inous it's quite easy to to reach people and just like fix and debug issues uh and the other one I think as an advantage is the the whole thing with data privacy and security some companies are very strict some companies they do not want anyone holding the data of their clients

and and this I would say is the biggest deal breaker for many of the big companies where they have either some legal things you know they must be in charge or they they they have very high secur security standards where they they don't trust a partner to do this so in terms of advantages is is this uh in in this advantages is kind of the opposite you know is we talked before extra time to actually maintain and create it extra people you need Cloud Engineers you need Central backend engineers and and you need to invest a significant amount of money to reach a level where this is useful and this is you know very good uh for multiple games uh when you're hiring Talent programmers especially is your inter is having an inter internal back end A Plus or a

minus it depends quite a lot uh in the past people they were really interested in joining companies to have access to inhouse technology and learn from those tools you know these days there there's quite a lot of people you know especially with layoffs in the industry you have Studios that are laying off by hundreds and thousands and those people were working sometimes in inhouse things that they do not have in another company so there are many developers that want to be doing Unity unre you know so I think with the back end as a service as it grows as it gets you know to a level where it's like Unity it's used by thousands of developers the trend is going to be oh I want to work with this kind of tool you know because that will allow me to switch jobs or find a new

job and transfer my knowledge you know so it depends on on the developer but I think there is a trend to be using cross platform and cross Studio Technologies uh Chris What What In Your what what have you seen that that are sort of like advantages of internal backend system when comparing for example to your services so I think I think Rafael hit hit on most of them um what is interesting that a lot of those uh positives can also have that double edge s of being negatives as well um you know a lot of studios in particularly Publishers that we've worked with um they have a Central Tech team they have a Central Tech Team there is a collection of microservices internally this is their pool of um internal apis that any game team can pick up and adopt but then the ability to modify those things adapted to something that is going to be

a new game project and maybe even a new genre of game that they're going to be experimenting in becomes a lot harder because you're talking about changing something that now has a lot of other games or teams depending on the way it currently works today so you actually get this slowdown over time uh with the way that an internal team is able to to rapidly make change and support a larger number of games of different genres as the studio and publisher grows so um but I think Rafel hit the positives is that if you want total control over every aspect of the development process you have the time and the budget to be able to do that um you have perhaps uh more focused or more limited set of genres that you want to Target maybe even a limited set of platforms that you want to go after Mobile versus say you know hybrid um you know crossplay type uh gameplay experiences then it can work um

it it can definitely work there are plenty of teams that are doing it but there is a trade-off tool of that yeah and so well back end like building your own back end is a considerable investment as I've said and it's not only an investment upfront but it's ongoing investment so you're looking for a return for that type of investment and sometimes what I've heard uh from not sometimes quite actually quite often from the tech lates they're saying uh it makes sense if it gives core competence in a specific genre they often would say MMO RPG being the the primary genre where they would s they would be very serious about actually building their own backend versus using a third service provider what does that mean in in practice like why mm RPG and what does a core competence in internal backend mean in practice so um so I'll pick this one up um mm RPG is probably one of those areas where I completely agree where you've got the simulation server running

the game state authoritatively with large numbers of concurrent players online interacting with each other um and how that simulation binds to the physics inside the game world how it mean you know how it plays out for the good kind of the quality and the feel of the gameplay experience and the latency of communication for players all of those things have a massive impact on the overall player experience of that game type and so I think building tooling and Technology that's core to the studio in that area is definitely something every studio is going to do and I can't see that particularly changing okay but what's interesting is that you can still think about the collection of services that go around that core simulation the inventory systems the friends list the guild systems and actually if you if you step back a little a lot of those features and and those uh functions of the game

world are quite common to what you'd see and interact with in in other games so what I would tend to suggest when studios are thinking about sort of that MMO landscape you're definitely going to be building your own infrastructure and server-based um simulation systems for running these kind of distributed gameplay but consider not building everything because Engineers like to build and once you get halfway in they just want to go all the way and build everything and if you take a more measured perhaps a hybrid approach to that problem a little bit of internal Tech a little bit of a partner of some kind on the game Tech side you probably ship a game an MMO game faster than anyone does in the market um and as Rafael mentioned speed to Market is is really important and and one thing that I want to touch upon what what Rafael said was the the privacy so when you own everything you are the owner of your own

data and that is very important for I would assume every Studio especially the the larger ones that are more prone to hacking um how do external backend service providers like like what is the way to assure them that you are in ownership of your of your data like do they get their own um service their own environment or like what what is the way to to answer to that need so the way we solve that is we don't run a multitenant service when you are storing your game data it is stored in your own environment in in much the same way that you describe these servers the hardware itself is not being shared with anybody else okay nothing on the disk is being shared no player account information is being shared there's no crossover right they're entirely isolated sandboxed from each other right um that's the first piece of it the second piece of it I think is that we have the opportunity because of the way we distribute our game Tech to run it anywhere so we don't have to run it on

our uh Google cloud or AWS account for you right we can run it inside your account as well in which case we can adopt and conform to whatever information security policies get mandated within that studio or publisher so we can meet those requirements so I think that helps a lot and I think generally because of things like the EU data directive and mandates around PR privacy shield and things like that and the way player players own their own data even if it's remotely stored by a company means that all of these things are tightening up anyway from a privacy perspective and as the Privacy portion tightens up the security model around it tightens up anyway thank you makes sense um all right let's talk about disadvantages of of your own back end so Rafa you you know we've we've said multiple times that the cost and the ongoing investment is is part of that but what other disadvantages have you experienced when uh working with internal back end as a

service in the end it's a similar problem because despite me working on Angry birs okay I was owner and and I'm talking about knowledge I knew the Angry Birds part of the code base you know so so you know the game specific stuff the inhouse it's is still something you don't know so the company owns but I still you know I need someone in my case I was in Stockholm I needed the guys in Finland to actually do things when needed you know so the the ownership and I think we need to clarify here that the back end we are still building a back end okay it's not that you you go with a back end as a servic provider and and you're just doing client work so right now at bundai we are doing a a game it's you know we producing a game I would say we are

50/50 50% of our back end is written by us we are actually doing features talking to designers you know but as we keep on developing and as the game grows and as we have live Ops and I I can easily see our entire back end being 70 80% our own game specific things and then the 30% is the service you know which is facilitating the other uh 80% so you you as a game despite using inhouse or a provider you still have to write code you still have to put the game logic in your own game you know you have to design the rules so you have quite a lot of uh code that is your own and the provider is just the 20 30% that makes everything you know uh fit together nicely and and and

reusable got it and and something that I've so of course I don't have experience directly with the backends but something that I experienced as a product manager with analytic systems so using internal versus external maybe a little bit of a different comparison but with analytics when using an external Services I always felt that they were more polished they were easier to use they were thought about from the user perspective and um then again whenever we ended up using our internal analytics tools they were never quite as robust and they lacked a lot of features that I would have wished to have does this type of a uh you know thing happen with with internal backend development where you're like well we were just using this other service that is you know actually sold to clients and now we have this internal version and I have to ask for tickets and my game and perhaps is not the biggest one in portol and things are just not looking as you know as nice yes

definitely like you sometimes you're forced to use the in-house tool knowing that there is a better analytics you know chis said before no company will tackle all problems so with us you know we have chosen another provider for analytics dashboards so yes if you have inhouse and everything is mandatory to go through the in-house you're also putting quite a lot of limitations on what your engineers can do and maybe your in-house is not moving as fast as the market you know these providers they are fully dedicated to do these and they have hundreds of clients so the improvements they will do will very likely be much faster and better than than your in-house too so yes definitely this can be a a drawback and and even a limitation you got it and Chris what what other disadvantages are have you uh seen when

when using an internal backend service versus an external one uh let's see so I can't speak to the customer um but um I know from a lot of the engineers who we work with at some of the the larger Publishers um they experienced a lot of resistance from from even their own Central Tech teams um they simply put it on the backlog it might it might happen one day but there's a lot of compe priorities uh and things get left behind because you know Rafael mentioned that you know priority zero critical issues will get resolved exceptionally quickly but when things can't be communicated as though they are critical issues because they aren't right they might be you know a new direction that the game is going in they just get treated very differently because there's so much pressure on that small pool of engineering resource internally um that things to get they yeah they they essentially have

something has to give and it is usually um something that affects the game team's bigger Direction they want to take the project in uh it can it can kind of uh act as a as as a Blocker on that progress we we talked about you know the the provider may be going bankrupt but what if your Central Tech Engineers leave the company like imagine you have five people and two or three of of them leave at the same time it's going to be a big hch you know so this is also something that in-house has to deal with and and it can also be a disadvantage you don't have the knowledge replacement that fast so that's also one one thing to think about let's let's move on to discussing like when when your CTO Tech lead backend lead are making the decision of using a backend as a service you know how how is this

decision made like what are those important elements like I know how decisions are made with with other services that that go through but it was always to me a blackbox on how we ended up choosing a certain backend especially because I have relationships with certain backend as a service companies and I would say hey uh maybe we should do maybe we should use this service and I would get an answer like no no no we're going to go to a different direction without even uh a conversation and I would also know people from other backend as a service companies who would tell me don't use that I work there and it's not worth it they're not supporting let's say mobile uh they are moving to a different direction trust me I was there don't go there and then I would have a um a tech leader would say no we're going to go there it's uh it's I've used them before it's fine I know there's drawbacks but it's uh it's it's the service that I know so let's just let's just go with with with that one there's no much different so I was always in

this situation where I didn't know how the decision is being made but I was never the less who would pay for that decision in terms of in terms of the cost so maybe Raa starting with you is like how do how are these decisions to choose one back end over another made cool yeah at Bai we don't really have a CTO in Barcelona um because we have this flatter um you know structure but we do have three principal Engineers that we basically I'm one of them we basically do a short list so we do a research and see everything that is out there and we come up with let's say a top three or top five candidates then each one of us actually go through download the tool you know do the whole integration and we actually want to develop a little feature we want to do something you know that you would

potentially do in a game you know it could be upgrading a character it could be you know consuming some duplicates some coins something simple you want to you want to look at your setup how easy it was from downloading it installing and then you know booting the server running this you know uh in a continuous integration uh system where the builds are being built you know automatically and you you go and you document pros and cons on each of these you know you also say it's open source it's not open source we have access we don't have access customer support so we do this with you know three to five uh candidates we have it documented and basically we go in a room with the engineers and we start talking you know discarding let's remove three of them and then we have two finalists so so we start to back and forth cost we also have to talk with the head of Studio because you know he is the one that is

going to pay for for the cost so we got to say look maybe we like this one it has i' say 10 20% more expensive but you know in the end this is going to be so there is a mix of engineering you know things where you want to make sure that you can debug and you can work very fast you know you want to develop features very fast but also you need to talk to to the stakeholders that you know that will provide you the financials to make sure like okay we're willing to pay this we understand the cost so for us we did exactly this uh for I would say about a month a month and a half that we took to dive deep into a field uh until we chose to to partner with Nama and we had these you know conversations with with our head of studio with with the game leads and so on so on so this is overall a CTO

would never do this by himself if he's smart he should have a team of Engineers and then come up with a a decision you know so it takes a good while and it takes time from your lead programmers or lead Engineers so that's not ex that's not cheap that's uh you know they should be building the product but now they're they're doing this this this work um how how do you get that short list like you said you start off with five how did you pick up the five well I it's a mix of going through the promises okay so the first thing is do not trust marketing material because everything on a web page is always beautiful it's always perfect they will never tell about the problems you know nobody will advertise issues so I did talk with some people that you know had work worked with different tools and I said look did you work with these what

was your experience you know pros and cons so I did consult with two or three colleagues in the industry and and got their feedback about one of those but in the end you know you need to do some research see see what is there uh look at the promises and see if they actually deliver on those promises uh but yeah the short list is a mix of uh what is there asking people and then you have to do the work that that one month or a month and a half it pays off because you you know you you will be stuck with that for a very long time so you don't want to take it wrong got it so references over any marketing that's that's clear so you're basically getting your your checklist from other engineers in your network and they will give you hey you should check this one out and that one out and we tested them but they weren't for us but you could test them again and that builds your a list of of three to five services that

you're going to test against each other and then the second part that you mentioned is you talk to the stakeholders uh game leads as well as head of Studio who is who only says how much is this and then you give him a ballpark number that that is going to be so obscure that uh that nobody should trust that number in any way because it's dependent on so many things uh nevertheless you talk to the stakeholders to understand what are they looking from the back end and then you comply a list of things you're testing against is that correct yes and I also forgot the service agreement with your legal team is also it takes a while so you want to make sure that everything in your legal department you know understands the things what happens if they go bankrupt if they get acquired you know like all these transactions uh you need someone else to to do that uh so yeah this was also one of the rest is your correct got

it so so that all of these uh stakeholders legal team being one of them is going to be part of uh or their requirements is going to be part of the what are you testing against when you're looking at these different service providers um you said a couple of times during this this uh this podcast is like what happens if you know if the backend service gets acquired or gets um shut down or whatever it is um how how high up is is that on oh actually let me put it this way a lot of these backends as a service providers like if I take gamespark and and playfab that's Amazon and Microsoft so I would say they are relatively secure I don't see e neither Amazon or Microsoft going bankrupt anytime soon uh most other like pretty much everybody else is a startup I mean Venture funded oftentimes um is that a risk that the legal team views or like how is it um yeah just just out of

curiosity yes that uh that is pretty high that is actually a deal breaker and if we cannot come with an agreement if you know the agreement is not maybe customizable to your needs if it's you know this is what we do and it's exactly the same for every client and we're not budging on changing any of these it is going to be hard you know so it's a negotiation and there is Clauses that you can you know discuss with with the providers to handle those situations so it is possible to have that in contracts what happens if you get acquired you know so it's Case by case but it uh in the end it was the last uh the last deal breaker thing for us to a point that if we couldn't come up with an agreement we would say wellow we're going to have to do it ourselves so got and and that's why access to the

source code is so important because it's uh it's a fail safe for for a for a studio uh in case this type of a thing could happen and it and also one thing that I've I've heard of and like this H this is in almost in every business is like if you are the CTO who makes the decision on the back end uh they would say well you can never go wrong by choosing Microsoft after Amazon even though you did the wrong choice it was not the right back end maybe not nearly as robust as a as a as a more newer from a startup would be but nevertheless nobody could say to you afterwards like well why did you choose that over them it's like you can always say on on the the security excuse and um yeah I was just just curious because that is something that I've heard and that is something in my own experience when when when we were going through a lot of different backends it's like uh uh this sort of a uh um security of a big company uh played played a role but but

Chris like what are what are you hearing when you when you talk with uh potential clients and existing clients like what are the worries that they um they bring forward when when going through the selection process so from our experience the biggest um point of friction is scaling actually it's usually not necessarily I mean I think it helps it because of our open source approach to core infrastructure I think we probably have less conversations about what it looks like uh in terms of access to source code in some kind of eventuality right um so for us a lot of it is scaling and that means simulations uh benchmarking throughput testing that kind of stuff um and we have a lot of existing examples of having done that with real games and so it that's how we tend to navigate that one just help them see just by what is in production today

with game teams who use naroma and then through that they can understand what they can expect from the system at scale are there any elements sorry are there any elements that come in like as the most important when when Com when studios are reviewing back end like I know we talked about the sort of a safety being one but are there other elements that that you keep hearing that is like on the uh on the Forefront when when choosing a back end um so it as I mentioned before you've got the feature mapping uh as I I I tend to refer to it so that you're thinking about the type of game that you're building the type of meta you want to to provide to players is it social is it solo play first and then think about how that that affects what amount of code might need to be developed by the studio team themselves just as Raphael mentioned you will no matter what game Tech you build on you will be building um tooling and Tech on top of it it's it's just the nature of

any software development so with that in mind they'll start there transition to scaling and then I think Rafael is right I think in some uh conversations that others have had with the the the market it will be what happens in sort of change of ownership events or things like that for us we tend to to see less of that um and then I guess the last piece of it is then pricing really because as Rafael mentioned a lot of the market is is kind of converging to a a stable point on price at a level of scale anyway yeah got it well uh last last question is is something that I've also experienced in that in in job is um switching back end and there's there's been situations where uh the let's say in a few different ways like one is when you're building a game and the game from

original design keeps changing and during the development there is a window because it's not live yet it's not in full-fledged production where the backend team wants to actually change the backend service provider uh because the initial Choice done by somebody else is maybe not as flexible maybe it's one of those safe choices AKA black boxes that are not actually answering to to what the uh the this game needs to um needs to deliver and the other part where I've seen uh and why I find this so interesting is is in Acquisitions so sometimes going through an acquisition and uh the uh the acquirer is looking at the acquired company and then they're seeing how they could grow this game with much more improved or this portfolio with much more improved life services and and um you know more effective product management user acquisition any you know modern modern uh modern ways of of running

games and as they're going through and even have gone through an acquisition they only then realize that somebody has made a very safe choice in terms of black back backends uh implemented a a blackbox solution that is actually not functioning uh with their services and so they have to go through uh process of changing back ends which can last up to a year or even year and a half uh as they replace the these back ends and suddenly the whole acquisition is not performing as they intended because they can't even get to work and start operating the games like they want to so so I've seen kind of like these uh these backend Services bite uh the uh uh the business targets because they were just not surfaced or wasn't even thought about it was like what I didn't even know this was a thing so um so going to my question is like how do you switch a back end and like what what goes into that thinking of like now we have to we have to switch our back end as a service

so Rafael maybe we start with you yes what what you said is is pretty accurate you know like if you're having Services shut Downs persistent issues you know you're having these over and over or your game team is slow like they your engineers is saying it's taking us too much time you know there is quite a lot of things like it say low process to to use this tool or you have some legal reasons you know something came up new new leg legislations or compliances all of these are pretty valid uh you know things to to consider a switch if your game is live it's much harder it's much more challenging to to go from some A to B so my recommendation you know and and the way I would do this is in phases so you try try to break down your game into parts and see if you can transition one

thing at a time you know maybe you have a service that have analytics and some other you know Inventory management and some some other things and then you say okay let's do the analytics first so you do that you do that transition it takes some time you go live or sometimes you can even run the game with two servers so you have the original provider and then the new one and you can slowly uh get some players you know and you can transfer this to to to to the to your newly built inhouse too or or your your next provider so you have to be very uh cautious because you don't want to break a live game you know and and then your revenue is dropping or your retention is is not there because people are having more issues with the new thing so it is possible uh it should be reserved for

more extreme situations so if you can avoid it it's better you know it's not like oh the new engineer joined he doesn't like it he likes the other one you know but that is not really a a good excuse so and you will have to deal with this you will have to hear engineering say I don't like the way this is done you know I think the other one is better but that is is not really an extreme situation for a live game so to understand correctly when you have a live game it kind of is what it is and you can I mean you can switch a back end from a live game but it's going to be very costly uh it will take a lot of time and you will only do it in extreme situations where as you said there's limitations on scalability there might be severe performance issues um you know the that kind of stuff or the the companies acquired are getting shut down and so forth uh but when you

are in early phase of development switching a back end is is is more possible and that's why I think the previous conversation is important because I feel like it's it's very because it's so much easier to do it early hand and you can avoid so many mistakes down the pipe you should really focus on making the right choice from the beginning rather than trying to fix something because this is a very very tough issue to fix when it's actually on the live game and it can make or even break your business case if you have made the wrong choice because you didn't just deem it as important early on um but but Chris kind of like have you gone through uh or have you worked with companies that have switched their backends and and switched to heroic from whatever they were using before yeah we've actually we've actually done quite a few of those strangely enough um the largest one we've worked on was making is making about 190 million in annual revenue so it was a big gain um it took almost two

years wow to transition it over um but obviously because of the revenue and opportunity and and future plans that they had for the game made it very much worth doing it um it felt a little bit like trying to replace the engine Wheels shell of a car while it's still on the road racing on the racing track so um it's a big Endeavor and you have to get every single edge case right um so it's a tricky one um the other motivation that we've seen uh that Studios will move backend um some kind of internal Legacy Tech where some of the internal team have gone and so they simply have new Engineers coming in who just don't know how a large portion of the code Works how even a lot of it is still running at that point and they just need to transition off it they need to take some something new they need to go somewhere more modern

and that they they involve us I think you do need a very Hands-On approach to make that work because I don't think the engineering team internally uh if they're going to a game Tech partner are going to be able to buy um reviewing just the internal uh back and forth between the client and the server side of the code is going to figure out every Edge case I think you're going to need to involve somebody on the game Tech partner aside to help make that happen um but yeah we've done we've done a more than two dozen of them um for significant games that was worth doing that for yeah but it it's not a quick process and I don't think anyone who tells you it it is is telling you something that's truthful really at this point there's too many too many unknown unknowns there just Legacy to the way that the system is is then running yeah yeah yeah no no nobody has ever said that it's simple most often it's the uh it's the reason not to change a back end and rather suffer with uh with a much

worse option but of course in cases where you can see the potential of the game or if it's already showing its potential it can be even larger and they still undergo a process of two years of changing a back end that's insane so that's how bad the previous back end was that they were like fine we'll just we'll just eat a 2-year cost of saving of changing the ches and the tires while the is driving full clip so uh so it it just it just in my like the way I'm hearing it is like there must have been serious problems with their initial backend choice to undergo such a massive operation and that's why it's important to really discuss these uh benefits and the drawbacks of both external and internal backends so that you make this Choice early and you make it properly not based on well I know this back end so I'm just going to work with it or it's not as we'll figure it out out later down the line it's like no no no you have to be incredibly um measured

and uh put the the work in the D in the diligence of choosing your back end even though it sounds like not the most important part because you're building the product at that point and um and um yeah just you have to live with it the the internal stakeholder that we worked with uh on this particular project he was he was widely referred to as a Maverick uh inside the studio stud because uh they expected it would not work they expected it was going to be too hard to do yeah it was just it was it's that it can be that big an undertaking yeah insane insane but they're very happy now yeah well well I would be very happy with where they started with the Run rate of almost 200 million a year that's uh that sounds like a happy team yeah yeah so um thank you for uh I mean you a ton of information on back ends for for people like myself who have very limited understanding of it and who have only seen the uh the issues of a backend not

functioning as it should and um I hope this podcast gives um Founders directors Executives a little bit more tools on when having this conversation so it's not only a question about which one is cheaper uh but but more questions around around or more more about what type of questions you should ask your team who's making the selection and making sure that they don't take the comfortable option but they actually do the due diligence and test different options and as well keep communicating whether with its internal backend team or external backend of the service as they are developing the game so that the issues won't arise when it actually launches and starts scaling because they haven't updated on their initial plans so on that note uh thank you so much Rafael thank you so much Chris for for all the knowledge and all the insights and um that's a wrap thank you thank you thank you pleasure to be here

bye bye bye