Fireside Chat: Multiplayer - The Power Chat

now there's a paul who's gonna join me uh in a few minutes so for those who don't know paul bettner is uh has been in the industry for a very long time uh so more than 20 years so we work at a microsoft game studio we help on hill awards uh war sorry and age of empires as well and then there's probably a game that you know more which is word with friends that had huge success and propel your company a new tar and corporation to the stratosphere i think most dads and moms out there played your game which is fantastic and uh so i'm so happy to have you here today because there's so much that we can talk about both based on your expertise but uh where things are going so that's going to be super interesting thank you

very much for having me uh just making sure you guys can hear me okay yeah it's all good you look great that's fantastic so i've been watching for a while and uh you have a busy day obviously helping with all these panels i appreciate it yeah definitely well the goal is to share information and you know and have fun and obviously if we were physically there right now we might have a beer around us uh because i always like to do fireside chat while being cozy or i like that we have a little fire in the warmer we can feel the warp definitely with winter coming and halloween soon the fireplace is going to start cracking um so paul yes like i was saying you you you've been there for a very long time uh you saw you know many challenges uh both in the 90s and early 2000s the topologies changed the libraries changed

the medium changed then you went mobile and now going all over the place so uh maybe you can tell us a little bit like um uh obviously how did it started for you and uh what were the first challenges and um and and developed on the age of empires and where this thing led afterwards yeah absolutely so i uh as you mentioned i've been making games for i guess almost 25 years i was one of those kids that that kind of knew what i what i wanted to do already when i was you know 13 or 14 and so i was already programming at that point and really teaching myself and that was back when there were it was like right at the beginning of modems uh so there was 2400 baud 9 600 modems and you know i'll never forget my first experience connecting my computer with my brother's computer and playing a game that that you know was a multiplayer game that connected those two things and i just thought it was the most amazing magical thing i've ever experienced and

from that point forward i wasn't just drawn to games but i was specifically drawn to network games multiplayer games what's interesting you know is looking back then and now it's not i kind of expected if you'd asked me then that some of the challenges would have been solved or would be easier or whatever but they're not necessarily i mean i think that the challenges inherent in building a multiplayer game uh you know are gonna probably always be with us because it's a little bit like you were talking about in the last panel there's this there's this inherent sort of connection between how a game is designed and how it works on the network that i don't think is ever going to go away necessarily we always need to be thinking about it so anyway back in that time i wrote a an emulator that ran uh that emulated a lan network it's called ipx at the time over a network of modems and i sold that it was called sir ipx and it was if you guys remember bbs is bulletin board systems i ran one of those in south florida it was one of the biggest pbs's we had 200 phone lines coming into my bedroom actually i had

all these it was kind of insane but at the time getting modem lines into an apartment was a lot cheaper than doing an office and so i kind of like hacked the system and got the the phone company to install 200 phone lines into my bedroom and and i had all these modems and a huge air conditioning because otherwise i couldn't sleep because it was way too hot with all the hardware in there anyway i did that we ran you know several different games on that software i wrote doom and warcraft and learned a lot about about the the challenge of uh taking a game that was designed for a land because warcraft doom those games were not designed to run on modems and figuring out how to get them to think they were running on a very slow land when in fact they were running on a network of modems uh that that work that i did um actually helped get my first real job in the industry uh when i came to dallas to work for ensemble studios and on the age of empires franchise some of the guys on that t and i was only 19 turning 20 at the time

and kind of crazy to apply the studio had just was just finishing age of empires one actually when i applied but some of the guys on their team had actually used that bbs and then used that software that i had written and so they're like wait that was you you know you must have written that when you were 17 or whatever i was like yeah i've been you know really fascinated by this stuff for a while and so i joined that team and was there with ensemble studios working on age one h2 h3 halo wars age of mythology um and and really built the network system uh you know when i joined um i joined mark toronto who is uh who is actually the founder of hidden path entertainment so he's gone on to do some amazing stuff and him and i were the network engineers and then i took that over from him and uh we had this immense challenge of figuring out how to get uh an rts with thousands of units moving around doing all these things to run on at that time i think the lowest that we had to target was a 14-4 modem

and you know and this is peer-to-peer so this is like you know 14-4 modems connecting over the internet uh communicating with each other and that was a crazy challenge um we built you know i think one of the first deterministic lock-step based networking engines um and and what's fascinating is is you know now now right today i'm actually working with uh and i don't i'm a huge fan of photons and of the extreme so i'm this isn't a commercial i'm literally just a customer but i love the stuff that you guys are doing um and i'm working with now the the next generation sort of evolution really many years in the future of some of those deterministic uh techniques that we use which is called quantum and is an amazing networking engine uh by by you guys but anyway so i was there i i helped build that and then um and then i also built something called ensemble studios online which was kind of one of the first sort of like uh well it's it's what became xbox live

actually a lot of the technologies that were in that were were the things that became what is now known as xbox live um and so it was friends list and and you know identity and uh presence and all these other technologies that really allowed people to find and find a match make with other people in age of empire so anyway did that for 10 years then uh left that company um right about when a little bit after when we sold that company to microsoft and started a mobile game studio back before anyone thought that was a good idea i mean like in hindsight everybody's like oh yeah of course but i swear when we went and started to make iphone games everybody all my friends all my colleagues were like i think paul's having a midlife crisis why is he going to make game for mobile phones if he if she first started to make like bbs and have 200 phone lines going through her group yeah people would know you a bit better when i said well

i remember camping out for the nintendo wii because i thought that was the coolest thing and most my friends are like nobody's gonna buy that it has like a weird remote control and i was like no but it's so neat and different so yeah i've always been that guy but um but yeah so we started uh new toy and we built uh actually chess with friends first then we built words with friends and that is an asynchronous turn-based game so uh you know it's it's built on the same passion that i have of you know creating multiplayer games but that was just so crazy because you know when we shipped age vampires we actually thought of the multiplayer as a little bit of like a niche feature like we had built this single player campaign for age of empires we're so proud of it and then we built multiplayer because that's how we wanted to play but we didn't really imagine that that would be the way that a lot of people would would fall in love with the game but you know it turned out to be the thing i think that that still today uh has led to remasters of age of

empires and why the game is so popular so when we built words of friends it was a little scary because it was only multiplayer and we made this key decision right at the beginning to not build any ai or any single player into that and and our thinking at the time was like well we're only a couple guys we don't have time to build an ai so we just have to pick we're going to pick multiplayer but it turned out to be one of the most important decisions we made because you can't you know when you first launch words with friends there's no way to play it unless you like either hit random match make or you go and bug one of your friends to play with you so the product was inherently viral like one person had to go find another person to even play a game and you know it it was just this incredible thing you know we went from and it was probably it was built on top of facebook as well right yeah at one so back in the days when facebook came to be popular notifications and all these things were so much different in fact we didn't even have push notifications when we shipped boards with friends people had to just launch

the app and see if they had a move or not and we were like hoping like oh i please i hope that apple and google ships this feature but we kind of believe that they would have to do that it just made sense to us and they did but that was an amazing thing you know we went from from nothing to 12 million dau at the peak in like two years and so just trying to keep that game running was like i mean i think one time i i was up for 72 hours without sleeping like just because i thought the servers were going to explode i mean it was crazy so so what the so what was your topology uh regarding words with friends and what were your challenge to make this game scale so we you know from the very beginning of words of friends because of the fact that we were uh you know a small team and because of the fact that we didn't really know the kind of challenges we would face because this was obviously our first mobile game nobody built a game like this what we did is we looked to other technologies that uh that supported

similar kind of user interactions uh asynchronous push based and we actually looked at web technologies and we ended up building a stack based on web technologies things that are now you know um kind of well-known web sockets and those things but back then they weren't called that and they were just like you know these more raw technologies but we built it off of http ruby we had a ruby on rails server we used sql um and then we used basic web communication techniques um and then eventually when we got push notifications we were able to add that in as as kind of an asynchronous real-time connection but otherwise it was all polling based which is was kind of the nightmare because you know it was pushing pool oh my god yeah so like when you've got you know 100 users logging on pulling the servers off no big deal when that turns into a million it's a big deal like so we had to write all these front-end proxies to like you know just kind of scale out because we had this core sql uh based

database that would store all everybody's moves but accessing that with all these real-time players coming in and trying to push pull from it we had to write all these like um you know high performance memory based caches in front of it so that when the users came in to hit it they would only hit that instead of actually hitting the database so yeah i mean for those who who don't understand like a push and pull is really a service where if you want to know if there's something that happens on the server you have to ask the server did something happen did something happen well when you have a socket usually you have an event that just bubbles up to the client and that's it you don't have to ask so and then when you're thinking about you know server configuration back in the days every request was an intense transaction per se because you have to validate did he do something no okay i'm going to reply no that creates a lot of loads on the architecture overall i'll tell you the thing that saved us there and this is just kind of a general

thing that um that i would recommend or kind of offer as advice the fact that we built that we decided early on to build words of friends the technology off of existing technologies is really what saved us because the kinds of problems that we ran into were the same problems that other websites were running into you know high traffic websites like facebook uh and twitter were also push pull you know at that time it's like you would just refresh the page and it had to quickly spit out your timeline because they didn't really have any you know real-time connection stuff at that at that point and the fact that we were using those same technologies meant that the breakthroughs that were coming out of and a lot of these were open source uh you know packages that were coming out of the web browser world and the browser world and the web backend stuff we were able to take that stuff and use it so these were things like redis which is a real-time memory database and just front-end caching so that that push-pull that mark was describing was the same problem that these web

these traffic websites were running into and the fact that we had used the same technology meant that we could not just use that technology but also that we could reach out to other developers who were already familiar with the technologies we were using because we hadn't just built it all ourselves from scratch uh so yeah definitely and and for those that don't know redis yet i i highly suggest you have a look at it it's still i mean it kept deteriorating on the product it's extremely good today and if you ever want to go go into the blockchain world and do high frequency trading and stuff like that where this is still top of the line and uh but yeah like um back then you if you were making php you ended up with some memory uh management on top of your sql that was the lamp set up back in the days and uh and then you had a new database like members and couch base and yeah there were all memory based and transactional still and and as paul said like um uh

and still today there's games that use sql as the source of source truth uh obviously things change nowadays uh but uh but there's games like for example like eve online as far as remember it's a huge sql server farm behind it that's why the game sort of like at one point just goes very slow and there's too much thing happening um so yeah so thinking about those things and and and how open source these are it makes sense i find that um that oftentimes when i'm talking to other multiplayer developers there can be a big focus on the real time component but a lot of games uh actually need a some sort of persistence also right and if you're talking about persisting anything that's shared across the network you have to start thinking about these things you have to start thinking about well where is my player data stored where is my world data stored how often do i have to update those things and and those become uh crucial decisions that are as important as what your network protocol is for

just players talking to each other yeah that's right and if you if we go back like to the age of empire days also there was not as many players the challenge was more about like making something that is unique that doesn't exist yet and uh obviously you came up with various problems that were not seen before so my question to you what is your zen approach toward like a situation where we're like okay i have no clue what's going on but we have to figure it out um well i really start by working with uh with our game designers to understand what they're trying to achieve you know these days i'm running a studio and so i get the chance to work with lots of engineers who are smarter than me lots of designers that are smarter than me and there's this is kind of what i was talking about like what was true 25 years ago is still true today that there's this crucial connection between what the designers want to achieve and what the the director has in mind for the game and what topology what network decisions what

things we need to figure out to empower that vision and it's always a back and forth it never works if one side is winning and the other side is losing you know if the design team is like i want this and engineers say it can't happen but i don't care just make it happen that's that's not going to work and if the engineers are saying sorry designers you can't have what you want because it has to be like this and we've already made these decisions about how the network is going to be architected then that doesn't work either you know so so i feel like the key is um to listen very carefully to and if you're just a one person developer this still applies listen to carefully to what the game is trying to achieve and then once you really understand that start looking at various technologies that already exist and see if there's something that solves the problem like like what you have in mind almost always there will be but it may not be something that is like just packaged up and ready to go like here's a network engine that does exactly what you need you might have to take a part

of this thing and a part of this thing and put them together in an interesting way um but it's been my experience i mean there have been times like you're saying where we and it still happens today of course where we need to invent something brand new um and in those cases i try to reach out to other folks and see if there are other people doing similar work somewhere uh you know i recently reconnected with you guys for this reason because the game the thing that we're working on now the top secret brand new playful project it's uh it's running right into one of the hardest problems in still today in multiplayer game development which is real-time network physics you would think that this would be a solved problem but two players just running a character around kicking a soccer ball back and forth is exceptionally difficult to get right i mean you just saw in the last talk you know talking about picking up a leaf can be like a really challenging problem and so these things are never easy and and and they're very much

uh you know like the last speaker was saying like you were talking about they're very much based on the decisions that the game design wants to make with it and so a lot of times you even if you're using an out of the box networking engine you still have to be really creative about how you're solving those problems and so that's what i try to do i try to i try to be creative and i try to let the game design push on the on the network engineering and also let the network engineering push back on the game design if if the network part of it says this would be really hard there are there are lots of clever ways to work around things you know like we were hearing in the last talk even um just you know ways to hide uh some of the inherent um kind of challenges of the network so in age of empires as an example we because age of empires is a deterministic networking engine and we made that decision we had to make that decision because there's every player is talking about the player and there's thousands of units we couldn't possibly transmit the

data for every unit moving around in the game all the time and so we ended up with was a network engine where the only thing that's set around is the player inputs and the timing signal and everything else is deterministically computed meaning that every computer gets that input and has to uh has to compute the same exact simulation on every machine at the same time and this is fantastic from a networking perspective because all you have to do is send around the same inputs but it has one challenge at that time which was we had no sense of prediction so when you went to command a unit and say hey i have a villager and i want him to move over here there was an inherent delay before that command was recognized because the command that input had to be passed around all the players and they all had to acknowledge that command effectively and then finally it would get back to your machine and that time would roll forward and you were able to execute that command so initially this felt bad because it

felt like the villager was not listening to you you know like you would click the mouse nothing would happen and then a half second later the villager would start moving and we worked we we said okay we got together with the designers we got together with the artist even our audio folks on the team and we said uh this is not really something we can solve on the network um because just inherently because the architecture but what's the real problem here the real problem is the player's command is not being immediately acknowledged would it be okay if if we just showed the player that their command was acknowledged but we didn't move the villager yet we didn't actually advance the simulation but we just visually and audio you know responded to that input and that's what we ended up doing and so when in age of empires when you click on a unit and you click on the ground to tell them to command immediately on that player's machine the villager says i hear you and there's a like this arrow that shows up that's like oh yeah you clicked there but if you watch the villager doesn't actually

he'll just keep doing what he's doing for a half second and then finally he'll move over there but that completely solved the problem and so this kind of thing is you're gonna run into it when you're doing network game development all the time but if you're creative about solving the problem it doesn't always have to be oh we just have to fix that with the network somehow a lot of times the solution can be around how the player perceives what's happening not necessarily what's happening that's right it's all about player perception and then um so obviously through all those years you you've probably came uh with the number one subject once your community is vibrant and you have a lot of players it's about cheating yeah obviously cheating in hr vampires cheating in water with friends is totally different uh and in virus aspect um i guess we work with friends it's about going on google and typing some words trying to figure out the letters which is you know it's some stuff he cannot really uh handle but then with when it's

affecting the gameplay per se like knowing the letter of someone else what he can write or in age of empire spawning troops everywhere and um so how what what were the issue or like the number one thing you you that happened in your games that was you know it was very clever hack and uh and then in general all about hacking so if you go back to age of empires um it had a has a really interesting feature of the network decisions that we made which wasn't we didn't at the time even think about hacking when we're first developing and unfortunately this happens a lot like i i've had many projects i've worked on where the question of wait what if players hack or cheat doesn't actually get raised until like the last month of development and i'm like oh dang and it's kind of too late so you see this happen all the time with big games like they'll usually in the only the first or second patch will they actually address cheating and hacking because it's like it just doesn't occur the team until until pretty late in the development but for

age empires we got really lucky because the fact that the the simulation was deterministic meant that if any player decided that they wanted to cheat and do something like uh give themselves unlimited resources or something else like that then all they could do is change that on their one machine and their machine would immediately desynchronize with every other machine it would go out of sync and so it was it you know they could still like cause the game to end like because what it would do is it would crash the game on everybody's machine and say out of sync but that's not a that's not usually what a hacker wants to achieve you know they they want to like try to cheat at the game so so we were protected against that however and so when we shipped the game by the way we were like great awesome nobody can cheat in this game this is great that only took like a month before somebody realized wait a minute but my machine has all of the data for all the other players in the match because the way again that a deterministic synchronous simulation means that your

machine is running the exact same has all the same data as every other person's machine and so they're like why don't i just turn off the fog of war and show all the units on the map and let me see all the resources and everything else and so that immediately became a problem like the the hacks that were basically is a different kind of hack than you would see in a normal multiplayer game but it was a you know basically see everything and and be able to you know just kind of look at anyone's village and just kind of go anywhere you wanted on the map um so we addressed in that case some of those things by you know it just becomes it becomes this kind of escalating battle you're trying to outsmart your hackers and so you're seeing what they're doing and then you're and then you kind of figure out well what could we do to try to combat that and so what we did in that case is we actually every once in a while we would take a look at what your screen was showing and we would send around something to all the other clients that said i see this and if those clients disagreed with what

you should be able to see then it would throw the game out of sync and we did this in a way that was like out of syncs are really hard to figure out where and why they're happening and so like this was something that that stumped the hackers you know they they were like uh well my my turning off the fog award no longer works just causes the game to go out of sync and it was really hard for them to track down why that happened so every game is different like you're saying in words of friends um there's really no way for us to combat dictionary hacks you know hacks where people are looking upward but what we tried to do is design the game so that that wouldn't necessarily allow you to win we knew that would always be a problem so we actually changed the layout of the board so that playing the biggest word doesn't actually always win the game in words of friends a lot of it is about blocking the other player from you know like there's a lot of board strategy around like taking over certain areas of the board compared to like you know other games like scrabble which don't quite have that and so that helps because a great words offense player is not really the one with the biggest vocabulary as it turns

out and did you make tournaments with the words with friends or or you state it with a global leaderboard and because now tournaments are getting more and more popular obviously um yes uh yeah there's there's so many things that we started to work on that game is now we sold that company uh back in 2012 or something so it's been a while um and they they have introduced new features that have to do with global global leader boards and and having real-time tournaments and yeah there's a ton there's a ton of cool stuff there now yeah definitely and now with the uh super uh lucky cal so i think you made the decision to not make it multiplayer and and then revise it and and and and and but obviously like um you how do you choose to make a game not multiplayer versus multiplayer is it really about the game design or was it for you to make something that was um out of your previous games so without without giving too much away about what we're working on now um

the lucky's tale franchise is we kind of have a long-term strategy with this franchise and with the work that we're doing in terms of building our expertise as a studio that can create uh platforming genre games like we kind of started out and said you know some of our my absolute favorite game in the entire universe is super mario 64 or just mario 64. when that game came out it changed my life it changed what i thought a video game could be and it just was just incredible i mean it just was just blew my mind and ever since that day i wanted to work on a game like that but of course as i mentioned earlier i love multiplayer development and multiplayer games um so but we decided that we needed to start off by really just trying to understand and teach ourselves how to start to build a platformer and and a game like this that could start to reach the quality of a nintendo game and and also we were a very small team initially so we had to make these hard choices and

so the choice we decided to make was we're going to focus on building a beloved character we're going to focus on building an ip that is bright and colorful and the children will love and just all these things we had never done before and we were really focused on just getting the character controller and this actually took us three games like it wasn't until new super lucky still if i'm honest where we started to really approach the quality that that we were hoping for in terms of how the character controls how the camera works like these really you know when you play a game like mario odyssey uh it can look simple but but it's just the the complexity the the the nuance of what goes into making one of those games great is just intense and so you might imagine that we would love to take some of that expertise that we've built in creating platforming games and bring it together with our passion for building multiplayer games so i would just say look you know stay tuned to uh to the future that's right now kcc which is uh uh uh

what do you call that again like uh yeah that's right for character control and and diverse animation on the players it's super hard especially when you have physic you want to make him go up a hill down the hill and go to the tight corner and and then when you think about obviously um having all of that in the multiplayer world then there's there's not many ways to do it properly that's for sure i agree so i would say like what would be like the ideal team structure uh both like we will talk a bit about game design earlier you did give a few tips regarding how to discuss and how to push back as a developer but sometimes how you have to crunch times and making sure that you can um that you can deliver something right and uh and um so how do you how do you manage like the uh your camera board right for example like both regarding

before and after the game so after the game would be okay now you have like customer support and then you have people like complaining about stuff and you have the community asking for new features and and then you have this huge crowd that is yelling at you and sometimes the server's all down you're like okay no it's okay it's coming and so how do you handle input about what needs to be built now that's a great question so there's kind of two parts of that question as you said there's what you do what you do during development and how you build a great team during development and then there's how do you handle that once your game is is live and launched so let me take it two parts so firstly um my my solution these days to uh to building a multiplayer game that integrates all the various aspects of the team is is really to to get everyone to understand that we're working together this isn't silos this isn't engineering going and doing engineering and design going through design and art going doing art especially on a multiplayer game but really on any video

game development there's so much uh delight and joy and and wonder that can be created when those disciplines are working really close together so what i've tried to do when i'm building a team like is is find people who are cross-disciplined so this would be an end this would be a designer who also has a background in engineering or an artist who's also an engineer or a designer or you know or an engineer who who also you know understands design or has been a designer in the past these people are are difficult to find to be honest in the industry you know they they call them unicorns or whatever but but they're crucial because they can speak multiple languages in a way and they can act as translators between these very different schools of thought and these very different different disciplines that need to all function well you know when building a game if you're working as a one-person developer it's different and in that case you know it's really important actually that you understand at least a little bit about game design a little bit about art a little bit you

know um but if you have if you're working in a team uh there's nothing more important than those folks speaking together and being able to compromise um and so so my solution to that has been to try to find folks who are very um empathetic to the other disciplines on the team they're not just like super focused on their one discipline but they're willing to listen and to to figure out solutions together with people who are working in totally different parts of the game than them because really that is where the magic happens there's this example i use on our team all the time which is zelda breath of the wild okay huge game um amazing what they just just one of my favorite games of all time but you can even see the places where they had to make compromises because doing something would have maybe added another another whole category to the kanban board to use your example or added a bunch of new tasks so they had to make these compromises and i love when i see those compromise so the example in breath of the wild is when you go to sleep in breath of the wild

all they do is they fade to black and they play this little like sleepy song they don't show the player like have to get down on the ground because you could be in anywhere where you sleep you could fall asleep in the forest you could fall asleep in an inn you could like there's all these different places and they would have had to add all this animation and different they would have to add a ton to the design and animation to make that feature work and somebody on the team was smart enough or had the foresight to be like you know what it would be good enough if we just fade to black and play this sleepy song good enough and then you just wake up the next day those decisions those things only happen when the different people on the team are communicating well then they figure out like because otherwise what can happen a lot of times is an engineer or designer will assume well i think design needs this so i'm going to have to write all this code to support it and if they would just sit down with the designer and say what are you really trying to do here the designer says all i really care about is this and they're like oh that well then i can only just do this thing that takes 10 lines of code and it's no big deal so that's really crucial is to try to get that communication to happen that way

because it will save so much time and effort and end up the game is much better so and then finally the second point is about the community we actually do the same thing you guys do which is we empower our developers to get out there in the community and i know it takes time i mean i always feel bad when i when i've got eric or or you know or uh peoria or something from the from the photon team in discord talking to me about helping me out with some stuff because i'm like i know he's got a backlog of 10 000 tasks and here he is in discord helping me with my problems but when we've done the same thing with our developers and it takes trust you know you have to trust that somebody's not gonna that you have a company culture where everybody's gonna like be a good person on discord and not be a jerk but if you have that then that connection between developers and the community and the players who are experiencing the game that's magical that then the developers all know what the community wants they know the direction to take the game once it's live um and and so you know just getting in there and getting getting together with that

community even before you launch is so crucial um you know and and also just realizing that when you're building and launching a multiplayer game the day that you ship it is not the end of the journey it's the beginning of the journey so be committed to that because it's what you do from that point when you launch the game that that ends up making it into something that could you know be played 10 years from now yeah definitely soft skills is often put on the side and people just look at the resume and oh yes this skills okay i'm going to interview that person but being passionate is definitely one thing and having multiple ads and i agree with you like it's super hard to find someone that both understand for example how the engine works and then the networking side and then the game design regarding the appetite of the gamers and what is possible to be done it's extremely hard and in school they don't necessarily teach these things to students as well they just you can be a programmer be a

3d artist and then you just learn that little thing but gaming you know now we talk about the metaverse and which replace the world so okay so what do you need to replace the world well you know we need a lot of things so so obviously it's a lot of expertise and um so uh so obviously like um like i was saying also in my talk uh earlier regarding like isps and internet service providers and the actual status of internet today so we do have like super high bandwidth we do you know we have faster connections obviously than in the age of empires days but it doesn't mean anything or almost to multiplayer game because the network like internet is just badly built like period isps isps are are reselling their own bandwidth sometimes you will go with one isp for download another isp for upload after the the second hoop of the connection that is happening so

but yeah my question for you is um how do you how do you what is your your your overall view on the world state of the internet and um and how do you treat for example like asia versus europe versus north america or america as all and and what do you do with with those constrained that's a great question um first i did want to say based on the comment you said at the beginning um one of the things that has really helped us by the way find these unicorns or these folks that uh that have those soft skills is the fact that the industry itself has just become so much more diverse and inclusive than what it was when when i was making games 20 years ago it's amazing now you know the opportunities that we have to work with creators from very different backgrounds from different places of the world and and just you know i'm really really happy about that and i would invite anyone listening um who who is excited about this stuff to get involved because the ability to make a game in your own bedroom on a computer and find everything you need to learn on

youtube is insane it's all free and it's amazing and like you know i just i wish that i'd had those things 20 years ago but you have them today and i would highly recommend that that you look into that okay so your question about uh about the internet basically it sucks um it's it's it's you know i said this at the beginning but the challenges that we faced um making multiplayer games 20 years ago that i thought for sure would be gone and fixed by now are not fixed in a lot of cases the ping the latency of connections between players that i'm having to deal with is the same as what i had to deal with 20 years ago you know if we're trying to connect players across transcontinental links you know between different countries um so so what i would say is and i see this happen all the time multiplayer developers will take for granted the network conditions that they think their players are going to experience and you know this happens because they're playing locally on their land and they're not experiencing that one

during development or even because they have a small set of play testers and they're all in the same region or close to them and then as soon as their game goes live everything falls apart and it's like what happened you know and there's just such a crucial difference between a player at a 20 ping and a player at 120 ping and taking for granted that your players are just all going to be you know sub 50 millisecond ping that's just not the case and especially not the case if you want to target lower end devices and you really want to have the broad appeal of something like fortnite or minecraft that's on every device out there you're going to experience every network condition that exists in your game needs to gracefully handle those things so my recommendation and i i think this was mentioned on the last talk is um there are all these wonderful tools out there for simulating crappy internet you just need to have those tools running from the very beginning and you need to you need to verify your assumptions like this gets back to what we're saying earlier about design if your design is

is absolutely hinges on you know a player is aiming at another player and it's precision aiming and they're gonna they need to see exactly where their crosshair is over that other player and they shoot and they can hit that other player great now test that on 150 millisecond ping with 10 10 or 5 packet loss and jitter and if it doesn't work then you you have two choices you can either uh come go back to the drawing board with your design and say is there a way for us to get the same gameplay experience we want without requiring this level of precision the second choice is diving really deep into the various network solutions that are out there many of them now have been well developed they're documented there's there's white papers there's network engines like like what these guys like what the photon guys provide that solve for this um and you got you have to educate yourself about how those work and what the trade-offs are because it's none of it is automatic and the

fact is that the limitations of physics you know just like the speed of light are never going to change and and you know as we talked about earlier player perception and balancing player perception against the limitations of of just the physics of it is always going to be something you have to deal with and so the the thing to do is to get those things out of the way early like test as soon as you have a game running on the network throw one of these you know network latency simulators at it and see what happens um and especially if you are intending to allow players from different regions to play together in our case in almost every game that we've ever made except for words of friends although that game is language based so it's easier but um but we we tend to to aggregate players by region what we found is that that actually there's other problems that that show up immediately when you start letting players from different locations play with each other whether it's time zone differences language differences there's all sorts of things

that you have to deal with and if your game can can be successful with enough players playing in just one region at a time together then that's that's obviously the easier answer because latency is going to be lower amongst players in those groups so we tend to try to split out that when we can um and when we can't we try to work around those things we try to like get rid of chat so that it's you know and and and just assume very high latencies and things like that so yeah that's been our approach yeah definitely and for the the viewers like the the brain itself when it sees something like the time it takes to compute is between 12 and 80 milliseconds right so if you if you target something that is 25 milliseconds you know it's already like so damn fast right yeah so so she said to yourself okay let's aim for that 80 milliseconds already like it's it's it's very good and if you go into 60 obviously it's very very awesome and um so yeah so we have a few questions from uh uh or viewers so uh

the first one is from dan lazar which is what first step would you suggest to someone who with no previous knowledge wants to learn about multiplayer network systems well these days i would suggest to get onto the photon exit games discord and join the join that's where i spent so much of my time right now because we're working on a multiplayer game with these guys and there's just so much help there and it's a great community but actually there's several of these communities so i've been on the mirror discord there's a ton of people there now you know uh unity bought the ml appy guys mlapi guys there's there's they have a discord that's really great and these communities are very helpful and they're full of the same people that are that are dealing with these challenges and even from the they're full of a lot of beginners actually also uh and and people who are just asking that question how do i get started what do i you know and so i recommend that and then i'd recommend youtube they are just phenomenal channels of of people saying let's build a multiplayer game from scratch um i here's something i would recommend

to everyone which is before you go make your magnum opus your brand new original game that you came up with if you're a beginner just go copy someone else's go do someone's tutorial go like go through the process the couple hours or even a couple days of let's build the same multiplayer game that this youtuber guy built or or this tutorial built you will learn so much by even just typing in the same things that they typed in you wouldn't think that but it really does so so i'd recommend starting there and then jump into the community and and we'll all help you out that's fantastic paul and and to finish out for you i would say uh what uh what are you working on how can people get in touch with you and and obviously it would be amazing for other developers to join your team so i will let you tell a bit more about what is up with uh your current work and status thank you mark uh so our company our studio is called playful we are a fully remote studio right now

i'm actually on the mountain in utah right now we normally live in dallas texas but we are everywhere now our developers are spread out so anyone could get in contact with us we'd love to talk to you the vision the missions of our studio is to change the world to make the world a more delightful place one video game at a time so you know we really believe in the power of video games to bring joy and connection to people i will say too that multiplayer is such an incredible thing for this you know i when i when i was first joined zynga with words of friends when we when we sold that game to them i remember i was in this conference and somebody asked me like why is words of friend so popular what is it about the game that's so popular like is it is it the fact that you change the board is it the fact that the art is a certain way and i thought about it for a second i was like um people don't love words of friends people love each other words of friends is just an excuse to to get together with others and and this is the amazing thing about building a multiplayer game is um you know when you're building a

single-player game you have to focus and worry so much about about you know do my players love this game design do they love these characters when you're building a multiplayer game you inherently have this level of engagement of player engagement that is based on the fact that they're just engaging with others so the more that you can make that an amazing thing anyway that's what we believe in um you can reach us at contact us at playfulstudios.com and we'd love to hear from you we're going to be launching our own discord soon so look forward to that we'll send out an uh you know a press release and and more information about that we're doing that in conjunction with the launch of our new product which i mentioned is top secret right now but we're really excited to tell people about it soon and i think that all of our fans and players are going to be excited about it also so uh so look forward to what's next from us um and we even have something for this holidays coming out that we'll be talking about very soon so anyway thank you very much for having me also marcus my pleasure oh it was great to have you a different people and um

uh so yes check out his website and also don't hesitate to write down an email sorry playfulstudios.com is a really great place to start and um and then obviously the multiplayer games has a bright future in front of us we're all augmented by our phones and uh by vr and sued by augmented reality so games will always be around us because it's all about having fun so thank you so much again paul for being with us today thank you very much for having me mark and now i see sofia so i will give you back the stage and um thank you for uh being with us as well today oh no thank you and thank you mark for leading us through this master mastering multiplayer and thank you to photon for sponsoring