Emergency Meeting! It's the 'Among Us VR' Postmortem

[Music] thank you and hi everybody first off a quick video explaining what Among Us VR is all about um this is me clapping my hands oh come on oh come on we just started the game come on Piggly what is it what do you want all right welcome to the among us we are postmortem so along with the overview of Among Us VR development we will highlight some wins and challenges that we've faced so uh don't worry about taking pictures or recording the talks there'll be a QR code at the end that you can use to download the entire deck oh and yes please do silence your cell

phones so my name is and I love multiplayer games and I'm also the design director on among svr and I am Sean Patton a principal game designer at Shell Games and I worked on both mini games and play testing efforts for among svr and I'm Jennifer a senior production manager at Shell I've been there for five and a half years but unlike Miho and Sean my background was not in games uh previous to this I had over 10 years of experience in project management in various Industries so I certainly brought a lot of transferable skills to the table but truth be told this has been my most exciting gig yet now before we zoom into any juicy stories I want to zoom out to our high level schedule it's a bit of a beast but this will give you the overall project in numbers and what this represents is 15 months from kickoff to launch and to break this down further we had eight weeks of pre-production 37 weeks of production and 18 weeks of

QA that not only included our internal QA testing but localization meta's fqa submission process and four external stress tests which you'll hear more on later by mid-september we had what I'll call overlap where for eight weeks we had we were simultaneously wrapping up QA prepping for launch and kicking off a new team for post launch and liveops work thankfully this was all with the same director's group I cropped this image from our launch Team contact list which no matter which version you'd pull up these five positions remain the same and while there are times that it's better for the team to have a shift we had the benefit of not weathering a change of project leadership no matter the face and while I'm talking about Staffing we had 11 devs at kickoff at our height we had 28 beans I mean devs working concurrently and over the course of the launch project we had 49 developers and in-house QA folks to contribute to this effort this number however does not include marketing product external

vendors in-studio play testing support and well you can see the credits VR is no easy feat even with a well-established IP so now that I've given you some numbers to chew on we can zoom in to where it all started pre-production excuse me how long it should be and what does being ready for production look like has varied greatly from Project to project I like to take the approach of establishing the criteria based on how many questions you have to answer for Among Us VR we had two major questions to get through how are we going to network this in VR and what's even possible the path for one of these was a little less Rocky than the other so to warm up let's go over one of our first big wins our beautiful corner we knew we were going to go with rebuilding the skeleton 3D it was one of the first major decisions shortly after kickoff to quickly give some context on that front Among Us 2D has four maps you can choose to play on with this skilled being the first map inner sloth released

in June of 2018. and nearly five years later it Still Remains the most popular and most played map it would then serve as not only the map that people would be most excited to be immersed in but also most familiar with and while the engineers had to sort through networking Solutions or was set on to on choosing a room to bring to near polish this would be our beautiful corner and would serve as our Benchmark and basis for answering what's even possible now a lot can come together when creating a beautiful corner part of creating the broom also included creating a character this gave us a chance to work through size and scale for both bean and room and once we started to narrow in on that we replicated our character until we hit our set limits for performance we discovered that 10 players is where we should max out at and while the decision for max players was rooted in performance it made sense from the game design end as well and just so happened to match the amount of players the 2D game launched with

now while I don't have a headset on hand for everyone to really do this Justice here's an image from the early Beautiful quarter we chose the oxygen room it had a good mix of mechanical and organic components and it was small so it could help us to guide the size of the beans better it also had an opportunity for VFX so we could start playing around with those elements as well it just seemed fitting to breathe life into the VR version by starting with the oxygen room when we wrapped up pre-production we had a delivery of a room that you could get in headset and experience our beautiful Corner would also help us to clear alignment with the look and feel with inner sloth and robot teddy and with a medium they were so unfamiliar with and an art style that we wanted to stay sacred to this was imperative it was our first chance of introducing the developers of among us to the top-down World they created now rebuilt in first person little do we know when we delivered this as a milestone having this handy would

actually go so much further we got word that Jeff Keeley video game journalist and host of The Game Awards was a huge Among Us fan so talks began if we could announce at the game awards December of 2021. which would only be four months in and almost a year out from launch the request came in hot and we didn't have a lot to work with so we had to call in the best trailer maker Derek Lowe and with our beautiful Corner in hand he made 27 seconds of magic to let the world know Among Us was coming to VR oh [Music] still gives me chills but as I foreshadowed it was an all smooth sailing while big win for half

the team was underway the other half was grinding we pulled on any resource within the studio that had networking experience it was all Tech hands on deck Unity had been decided as the engine of choice and while we knew we weren't inventing something new I mean multiplayer VR has already been a thing the options were still limited really Limited the solution was a product by exit games Photon they pointed us to their newest product wasn't even to market yet Photon Fusion part of the pre-production goal was to have a multiplayer demo complete with role assignment and some basic forms of minigames but documentation and support for Fusion was sparse which is not ideal when you're trying to set up something new so after weeks of spinning wheels we wanted to go backwards they had previous powerful game networking middlewares Photon pun and bolts we started researching in that direction only to find out our timelines did not align they were deprecating and heading into maintenance mode and we had no choice

in the name of stability and sustainability we had to go with Photon Fusion in the early challenges of this Choice was unfortunately not limited to only pre-production we were developing right alongside them and with a product that hadn't even hit beta we were just crossing our fingers and hoping for the best but operating without their timeline while all's well that ends well we didn't account for the time sunk into debugging and Engineers were constantly battling and researching bugs trying to even figure out whose bug it was we did a lot of QA along the way but I digress and I don't want to exit pre-production just yet we have pillars to talk about after all thank you so having these important Tech questions answered we needed to focus our design approach so we turned to project pillars now in case you're unfamiliar with the concept project pillars are kind of like guiding stars for your game they are principles that you agree to stay true to throughout production

I'll go ahead and make the claim that we established pretty good pillars that are serving us well until this day so how do we do it three steps first off it could be really easy to make a long list of things that are important for your game but you got to remember that the more pillars you end up with the more likely they are to end up conflicting with each other and getting blurred which goes against their purpose so to battle this you should try to think about what is it really that makes your game special and try to focus the pillars on that as well as combine your pillars into maybe a smaller number that is a little broader and covers the same basis second product pillars really only work if both your team and stakeholders believe and understand them so make sure that you involve them in the process of creating pillars as early as you can this is where you'll learn what is important to different people in different teams and the sooner you learn that the better

so now that you've established some small amount of good pillars that everyone's on board on and bought in uh you need to use them so make sure you always evaluate Your Design choices against the pillars and scrutinize any feature that doesn't seem to quite align with them as you do this you may find yourself at odds with your pillars this isn't necessarily a bad thing this didn't happen a lot to us but it may be that you needed to get your hands a little dirty before you realize what your pillars really want to be so again the sooner you do that the better so what kind of pillars do we have on I'm on svr the first one we came up with was to bring the best of what Among Us is into VR among other things what that meant is to preserve the visual style the overall tone of the game and to keep a low barrier of Entry when it comes to learning how to play it and getting into matches with friends or strangers our second and last pillar was the one that I feel like we ended up shaking at

each other more than the first and this was people play over gameplay what that meant is that our design decisions were to be in service of social deduction and players actually interacting with each other it also aligned well with our accessibility goals okay so making good pillars is really hard right like it takes a lot of work it takes time you're arguing but once you have them it feels great to be equipped with them as tools so we're ready for the fun part of OMG we're making Among Us in VR this is going to be great uh we're gonna get two new crew May roads at least one alternative imposter two new game modes some ghost only activity so now board and like a shooting asteroids minigame that makes you feel like you're in the Millennium Falcon yeah sorry I'm gonna have to interject here there's certainly a double-edged sword to Blue Sky brainstorms and while I don't want to be the web linkage of production you always want to have an airtight MVP before exiting

pre-production and I'm not saying to have all the questions perfectly answered but you need to start solidifying that sandbox that you'll be playing in as best you can so what do I mean by MVP so this is the most the minimum viable product and it serves as the absolute smallest game that you're willing to launch with and truly it needs to be slim you want to end pre-production with this document as defined as possible this isn't the time for Unapologetic optimism we've seen scope Cuts generally go one of two ways a death by a thousand paper cuts approach we're at the 11th Hour you've overscoped and you're trimming down piece by piece usually cutting down team around and everyone's Soul right along with it or you come out the gate to cut until it hurts and to stay in line with my graphic metaphors the ladder doesn't actually leave a team bleeding and wounded how you might think there's acceptance when realistic expectations are set the team can now digest and buy into what's doable versus having to recover from lofty expectations that might get crushed or

lead to Crunch and how do we apply this to Among Us VR in particular well we had all sorts of ideas to work with but we did have a timeline to adhere to and we wanted to prioritize getting this to Market so we stripped it back to what we thought would be the best gameplay in VR staying true to our pillars and not veering too far from the magic that the 2D version had already discovered we affectionately called it our vanilla version now don't underestimate vanilla she's not boring and she gave us a lot of power by way flexibility we knew our community would have a cross-section of players Die Hard Among Us fans without headsets meet people with headsets that have never played the 2D game before to allow this to have more of an organic growth and allow for the new community to have a hand in driving one Among Us VR should be we needed vanilla and I mean it's America's favorite ice cream flavor after all how could we go wrong oh I want to talk about how we thoroughly didn't go wrong

um admittedly when we came to realization that we're gonna have to focus on fundamentals and stay within our scope uh there was a moment of hesitation we were are we just boarding is this some kind of color by numbers game making situation well short answer no longer answer we still found plenty of design opportunities to make things interesting there's totally room for creativity when you're translating a game from one platform to another let alone when jumping all the way from 2D to VR so consider which of your new platforms features are strengths that can help you capitalize or what what is it that makes your game great and maybe there are aspects that the original game shipped with that your translation process can improve upon or smooth out so on Among Us VR we came across quite a few opportunities like this which was a win for us and I want to highlight three of them in particular so there's proximity voice chat that we added to the game there's this magical crossover of optimization and simplification

and some balancing shortcuts that we took so typing into proximity voice chat while this already existed for the 2D game via mods to it bringing it to first person perspective VR with specialized audio really was a game changer and it did all the right things to a game about social deduction and arguing so even though this was not going to be easy to implement we knew we can't leave that out of our MVP we didn't immediately plan for any additional fancy features around voice chat but some came up naturally and we ended up adopting them as just gimmies so queuing imposters invents or forcing imposters to stay quiet while they're hiding inside of the vents was a bug at first but we realized it's actually way more fun to do it and it doesn't really unbalance anything another thing was seances which is where ghost players so the dead ones can still hear what the living are saying and they can also open the doors by just

approaching them so it opens this like Supernatural line of communication though you can be like hey ghost did Red kill you open the door twice if you did it right and it seems like such a game breaking problem uh and such a huge advantage to the crewmates but actually not so clear-cut there's plenty of room for error there's Dead imposters who are messing with this right so it actually ended up again being just this fun new thing that was now a feature in our game finally we didn't quite predict how much noise and heated argumentation there's going to be once there's someone calls an emergency meeting so the ghosts who now can talk among themselves but can't be heard by a living player are trying to talk where everyone here is arguing trying to blame each other and so no so the ghost couldn't really hear themselves which made the ghost experience worse so what we ended up doing is applying his audio effect to the living players voices as they come through the veil to the world of the dead so that the ghost could tell who's speaking that is you know alive or dead and this worked really well to solve this problem while being this cool

reminder that you're dead like you're in a different world now of course voice chat also means that that much more opportunity for players to be less than excellent to each other especially in the game that encourages you to lie deceive and literally get away with murder now we knew that this would be a problem it's not a New Concept that people get toxic in multiplayer games but we didn't manage to squeeze all the toxicity mitigating feature that we wanted to before launch so those will have to come in future patches one of them launching really soon all right moving on to this one simple trick for game design that probably needs a better name that I failed to come up with but here's the gist of it so you're at the summit you know how important it is to optimize your VR game when you're developing for a phone strapped to someone's face it's not just embarrassing to have your experience be dropping frames it actually makes people sick and therefore they can't play your game now at the same time simplifying things

is just a good game design principle because it's usually very easy to make things more complicated if you actually need to so this struck us as this win-win combination right like we can now leverage this in Among Us VR and Sean will give you a quick example of how we did in the wild yeah so one example of this is in the task design right tasks in Among Us are not actually meant to be a challenge in fact we aimed for players being able to figure them out without any tutorialization whatsoever draw calls were also at a premium so simplifying some of the 2D games mini games made it not only perform better in VR but also made them less prone to player VR air where you accidentally interact with the stuff that you didn't mean to in the example of start reactor which is a Simon game we ended up with a four button layout that made the task less frustrating and less expensive

performance wise while still remaining one of the most challenging and focus requiring tasks in the game thank you Sean so this idea of simplification and optimization being at combo isn't only applicable to your game's performance it also helps protect and optimize scope and schedule in general so in alignment with their pillars we decided to greatly simplify the matchmaking process by removing our lobby browser and letting photons matchmaking put players into games this was also photon's preferred way of doing this and what they recommended to us so more on this later now this was a lot less work and a significant reduction of cases for the for QA to cover but it was actually also better for the player experience making it quicker and less confusing to find people to play with the small caveat here is that we do get occasional requests for a Lobby browser just like in the 2D game but it's pretty hard for us to justify that this would

actually benefit our game finally there's a crossover between accessibility and simplification so in the same vein uh when things are simpler they are likely to also be more accessible and we found a couple of little design rules along that supported this so all mini games must be completable completable with a single hand basically demands that they are less complex and easier to implement so under win-win dexterity should not be a major factor for player's success does the same thing but notify notice the qualifier major that we stuck in there that still leaves some space for player mastery all right moving on to the balancing shortcuts so balancing is super important in multiplayer competitive games so it's not like we could really leave it out of our MVP or it can be really time consuming and requires a lot of large-scale flight testing so we had a problem what we decided to do is

basically copy as much as we could from the original game to preserve as much of its original balance as possible now discussions of what is the correct balance for an Among Us game that didn't fit in the MVP now the first thing we copied was the layout of the scout the it was basically copied one to one wherever possible with a notable difference pictured here so on the left you see the picture of the reactor room where you have the two sabotage stations highlighted where two players need to go and touch them to kind of save the ship from blowing up and not lose the game now how it works in the 2D game is your camera Limits The View so when you're at one of the stations you can't see if there's anyone at the other one trying to fix it and you kind of live in this uncertainty of not being sure if there's someone there helping that someone just get murdered they're over there and so on so in VR what we had to do is build in alcoves to where those sabotage stations are so that these limit your

field of view and your line sides to accomplish a similar effect this was definitely an exception to the rule though of preserving skel's original layout all right the next thing we copied was the Run speed so that we measured how long it takes into the game to run from one end to the spaceship to the other and we copied that into VR it actually ended up making the speed pretty quick but we realized that it will it should work we also copied the default match settings from the original so things like number of imposters and cooldown durations and then we didn't let players mess with them which was both a scope win and a line with our people play over gameplay pillar but it did mean that launching a balance game was that much more important because now we didn't give the players the tools to self-adjust so because we copied all those other things though it wasn't difficult to arrive at a balanced experience with just a few simple tweaks so that's how we launched a balanced game without

having to do any heavy lifting now all of these design wins were great but how do we know that whether it comes to balancing or mini games or control schemes ultimately figuring out what works and what doesn't can't happen without a lot of play testing I love play testing after all one of our Shell Games principles is play test constantly it's the only way to know now while I did a talk about play testing VR back in 2018 today I'll dig into two big differences between then and now those games I talked about well they were all single player and of course Among Us VR is multiplayer the other big difference well we are still testing games during a pandemic so let's dig into that first I'll talk about challenges pandemic and multiplayer are on there but I will also cover challenges stemming from testing a game with a popular established IP

I will also cover how we tested and decided on our Locomotion controls and finally how we made sure we could handle thousands of players all right so how do you test doing a pandemic well for us usertesting.com was an invaluable resource for finding folks that not only could test games remotely but were able to cast and stream while they play we made great use of their platform for I expect you to die too and our relaxing cooking game lost recipes both of which were developed fully in the heart of the pandemic quick note while we certainly focused our efforts on usertesting.com they did not ask me to mention them and they're not the only game in town if you need the same resources all right so process wise first we'd send out screeners questionnaires to make sure people had VR gear and could cast which basically meant Quest 2 because while it's

possible to do other setups quest to use market share and relative ease of use of setup and capture made it the one we focused on interesting tip when you're making these screeners you need to keep in mind that folks on these testing sites they're usually testing generic websites uh games are a treat for them right so we needed to be we needed to keep people honest or maybe at least paying attention so we'd collect folks that could actually test our games to that end we had questionnaire that included questions like this one meant to keep potential participants on their toes which of the following virtual reality headsets do you have access to right now all right let's look at the answers that we had in the system you'll notice some interesting answers here like the famicon 3D that's not a thing the Google Crimson which is a code name for the daydream the PlayStation portal s also not a thing

then of course there's the sensorama which is from 1962. and then the virtuality the classic early 90s VR headset featured among other places in the 1995 cinematic Masterpiece hackers it's actually also the first headset that I developed VR on back in 2001. and then lastly we had this Xbox the Xbox Moda we made it up but it still managed to cause a 132 common thread on Reddit did Among Us VR just leak an Xbox VR headset no I'm sorry we made that up but under the hood these answers have actions associated with them in the user testing platform certain ones that actually exist participants May select that was okay anyone who selects a fake one they get rejected and there in the middle is the only one that matters the quest 2. if

participants select that they're in all right so you think the only thing we have left to do is actually carry out the test right nope user testing platform does not handle multiple people at once and as we all know humongous VR is multiplayer so we needed a workaround our solution involved three tests per player still conducted through user testings platform a pre-test to record Base information like familiarity with VR playing habits and game preferences that test also asked participants to follow a process that made sure they really had the correct hardware and the capability to cast and record a lot of folks who say they do don't now that test ends with an agreement to meet at a certain date and time in a Google meet at this Google me kind of like a VR Speakeasy participants take another test this active test runs in parallel to the

meat and has installation instructions for the game and also serves to record the cast of their gameplay through user testing's own platform and finally folks take a post play test questionnaire which includes our base play test questions and anything unique to the gameplay that they just tried but more on those later now while functional this process was not without its issues there's the aforementioned Liars who despite our screeners still slip through and surprise they don't have a quest too then there's more mundane issues like devices not being charged or updated side note meta has an uncanny ability to push updates right before our tests or at least it seemed that way to me so how do we work on these well we would send messages to players the day before reminding them to charge their headsets update their devices

and we would allow a full half hour of test time to troubleshoot with folks in that Google meet all right next are audio issues since among svr has voice chat once players are in the game we want them talking there and that audio of course would then conflict with the Google me both volume wise or Worse depending on the computer in their setup the casting would sometimes not be allowed to record audio and that would ruin the test so again more time allowed for troubleshooting and an issue checklist helped with that and then there were plenty of no-shows and how do you combat that do you do it like the Airlines and overbook well what we did is we booked the full 10 players our Max that could play and then we had some shell folks who were naived to the game sort of in reserve and arguably not as not as good as fully naive people and it was also harder and harder to find those shell folks as the project went on and if that didn't work out then we would just play with less than 10 players

now this last one is not really an issue with the testers more of a heads up for your development team you're going to need a lot of debug options you'll want something special for play testing we had things for Among Us like make the host not an imposter restart the round mute everyone things like that but we actually built into our build process that any bills that were going into the play test Channel were actually disabling that debug menu for anyone except the host because no matter how clever you are you or think you are in hiding that menu people will find it all right let's talk about the challenge of an established IP we had to always be aware of the level of familiarity our play testers had with the game 2D Among Us is being played by AOC by Jimmy Fallon for God's sakes it's in the cultural Zeitgeist so in that screener that I mentioned earlier we would ask if they've ever heard of Among Us had they played it how much but again as Gregory

House has taught us everybody lies so we had asked follow-up questions when anything seemed suspicious but at the end of the day we would be looking for three groups to intentionally play test with all naive to the game heard about it but never played it was okay all familiar and a mix now hold on I have a suspicious Choice here that I want to talk about so deciding how to react to playtest data from all these different player groups and knowing that there's even more player groups we had to make some difficult not to say impossible choices and you know we had to pick our favorite kids so at first place most favorite were players who are both new to VR and Among Us those are players that we knew are gonna need most help and we didn't want to sour anyone to our game remember the low barrier of Entry this Choice led us to spending a good chunk of our development time developing making an

interactive tutorial that didn't exist in the original game add favorite kid runner app we have players new to VR only but maybe familiar with Among Us so at chill games we always cared a lot about making VR newbies feel comfortable and welcome so we ended up committing a lot of our time to providing a plethora of controls and comfort features to that end in third place we have players Newton Among Us only but familiar and comfortable with VR well social deduction games can be actually a pretty strange concept to someone who's never played them and we observed that we had to tweak some of our settings for example making your smooth turn speed match faster than what any of us was comfortable with because players out there are happy spinning really fast uh we also we're close to defaulting our turn modes to be head based so this is basically where you move your head that's what determines your forward Direction which again was something that experienced players play with but thanks again to this ranking

ultimately we ended up going with what we felt will be the better for players new to VR so kind of the twin stick movement coming at least favorite was everyone else so basically players who are both among us and we are veterans who we just weren't super worried about having to do anything special for now unfortunately that did mean that we probably disappointed that player group the most and that's not great those were some of our big fans like I said impossible choices all right I'll let Sean get back to his part ranking your children Mia how dare you all right let's talk a little about Locomotion when we first started to think about reimagining Among Us for VR there were two main Locomotion camps on the team direct thumbstick move you push the thumb stick in the first person and you move and third person which is an interesting method of movement that really you'd have to see to believe yeah I got this one I can show it this happened actually

in the pre-shawn pre-production times and I can't bring up Locomotion or you can't bring up Locomotion without sharing more details of how about how third person movement was known as penguin mode we had an idea to eliminate any discomfort from moving while also keeping the character Pace the same something teleport doesn't do when you push the thumbstick you see your body move and you are sort of remote controlling it it seemed to work but we needed to play test and in the name of play testing constantly and answering questions early you can see from the clip how primitive our early play tests were yet it didn't hinder us it's also important to note plate testing with established IP takes additional steps and strategy not just the not just vetting play testers prior to the test beloved IPS and characters have influence that you might not want skewing results so think through similar sizes and shapes to your well-known characters and pick one to implement and anonymize the play test with that way you have the same feel and your findings can be accurate

with the early timing of our play testing we needed to consider marketing if any leaks occurred prior to our announcement trailer it'd be a whole other host of issues to deal with so we used a penguin our honorary first space Bean rest in peace penguin that's awesome yeah maybe you need a penguin hat in the future so what did all this play testing show well in short it showed that the third person or out of body play testing worked really well for curbing any feelings of discomfort but and this is a big butt the fantasy of Among Us VR the pillar even is the best of Among Us in VR and what does VR do extremely well immersion simulation fulfilling the fantasy of being a crewmate if I'm constantly leaving my body who even am I so even if we thought we had time to do both control schemes which we didn't we shouldn't because it breaks that fantasy so with a first person movement decision made well that also means we needed to

add a bunch of nausea reducing options which we did what everyone else is doing it to me I'm just resetting the kill cooldowns here folks so we made the settings but we found in our testing that even when we explicitly required players to open the settings menu in the tutorial folks were still not understanding well which settings were for them or even that the settings existed at all so we bought a costume we channeled our inner Tick Tock and we made a series of videos this was the first one in this video we'll cover a movie and Among Us VR moving is achieved the use of one or more controllers many settings can be customized move with your left hand is the default try pushing forward now right hand is available as well as single-handed mirrored mode for both hands do the same thing and this is my favorite let's learn about moving the map Among Us VR need

the map closer pull it closer if the map is in your way when running simply grab that white bar and move it up or down there an unobstructed view now you can relax that is pretty good so settings videos not only helped build hype the weeks before launch but they got to educate players a win-win all right the last challenge we need to talk about is thousands of players first let's talk generally about these tests we needed to be sure that our game could handle lots of players at once while the observed testing that we've been doing up to that point was all about tweaking gameplay these tests were all about free play letting people play unsupervised also it would be the first time we really pushed Photon Fusion stress testing at its finest testing matchmaking at scale and it all pretty much worked so that was great but we

also had marketing considerations we had kept a pretty firm lid on testing up until this point with ndas some strongly worded emails if need be but now we'd had a lot more people to test with and a lot more people to handle so we figured there would be some leaks and we needed to be ready for that honestly we probably should have started stress testing a little earlier but we were afraid of those leaks we did our first test with hundreds of players on August 11 2022 and remember we were launching on November 10th so this was late in the game also these tests weren't uh weren't really going to be a design test but they still kind of were insights gleaned from these tests helped inform our first patches and updates lastly we needed to keep in mind this feedback was from a very specific audience these were the very eager the Among Us fans who owned VR already and were very active in our Discord all right so how did all this testing go

well it turns out that stress testing was quite stressful um let's talk about those issues we had partnered with an entity that was going to help get all the testers under NDA in the Discord ready to go but it didn't quite work out that way there were Communications issues throughout and we sort of ending up ended up having to do a lot of the work that we'd pay them to do ourselves we actually also had trouble getting people into the in you know the bills onto Player devices for the smaller 10 person tests it was relatively trivial to add the correct email addresses through meta system so players were on the right Channel and could find and download the build so now here we were trying to paste hundreds or thousands of emails into time and if just one wasn't valid that whole batch would be flagged and wouldn't work so we had to do chunks of the emails at a time by hand it was not great quick tip the site to add emails to a channel claims the limit is a thousand but that's actually not true

but thankfully we had a great relationship with meta we worked with their folks to make the process better and it should now be better for all of you too here's an interesting anecdote we ended up testing the logic that decided how many imposters there would be during one of our stress tests normally for four to seven players there is one imposter for eight to ten two but we did one stress test where we thought you know maybe if you start with 10 players there could be three imposters how would that be we thought it might be bad but we wanted to test it and oh boy it was bad um but folks quickly caught on to the logic and we would hear lobby chatter as soon as the ninth player entered quick quick start the game start the game everyone would shout to start so they wouldn't get three imposters so we got important information from that but yeah it was hard getting thousands of players into a secret game built on a new network infrastructure with new tools I guess that's why they call it being on the bleeding edge of Technology yeah

all right um it was also totally worth it we were very glad we did these tests we weathered the storm and knew our infrastructure could stand up to the onslaught some imbalances with the Imposter win rate also became visible with all the new data so we are able to tweak some cooldowns and which mini games were visual that is which mini games other players could see the results of and we ended up balancing the game even better and we had the joy of seeing thousands of players enjoying what we'd worked so hard to create giving validation to our team that we had succeeded in reimagining a beloved game for a whole new medium so while a challenge VR play testing was also a win as well now testing with the smaller groups was great and uh and while testing you know it was it was this the stress tests were also a win all right now as I covered in my 2008 time talk on play testing there are four

main questions they can just watch your talk on the GDC Vault and we're almost out of time meanwhile I do want to mention another bump we hit right after launch so basically as soon as we stop celebrating our launch we noticed something that didn't come up in the stress test or play tests and that caught us by surprise basically every game you joined already had about eight people in it and it immediately filled up which seems good but it actually wasn't for a couple of reasons so the bigger problem with it was that you know you have that one other friend who also has VR and obviously you want to play among us together but you can't you will join the game at this Lobby that just opened and immediately filled up back to 10. so by the time you could relay the lobby code to them the lobby was already full also this would be joining the lobby at kind of the most chaotic and Loud part of the life cycle of an Among Us game so as soon as a match ends and everyone's super excited and like freaking out about what just happened and who

backstabbed who and how you know crazy lies we're told that's when new players will be immediately put into that Lobby and exposed to all this excitement and loudness so what was the problem and it was our matchmaking now because we didn't fully predict how our matchmaking will look in the wild and we couldn't really observe it even in our large scale plane test it had a negative impact on our player review scores right at the launch Rush so what happened so in a nutshell it turns out that the oldest first matchmaking scheme which was what Photon recommended and was default with was just not the right type of matchmaking for the type of game that Among Us is so I'll show you some analytics data as I explain why so what you see here is relatively how many games start with a different number of players minimum is four maximum is 10. so at launch you see that most of them are starting with 10 or 9 players so far so good but because our lobbies

can get old and therefore be you know lasting for a long time as soon as that game ended it would get pushed into the beginning of the matchmaking queue to scoop up all the players another Quirk of Among Us is that while the game will start with 10 players it often ends with about six or seven because players will immediately immediately quit when they're not picked as the Imposter or when they get killed or frustrated or Tilted in any other way so a game would end in this Lobby that's been going on for a really long time because we have host migration right and it just exists and immediately get pushed to the front of the queue where it scooped up all the players trying to match make so if this excuse me if this circle represents the amount of old lobbies that were almost full jumping to front of the matchmaking Line This is you know the amount of new lobbies being created by players trying to let their friends in now if you did that you would end up waiting there for maybe like 10 minutes without nothing

happening and then suddenly your body would become old enough to get all these players in most players would immediately give up and think something's broken which they weren't wrong so we kind of scrambled and we decided this has to be fixed as fast as we can which you know isn't that fast when you're deploying a multiplayer VR game so we talked to Photon and our options for immediate action were actually fairly Limited but we decided to go with trying a sequential matchmaking scheme instead where lobbies will just queue up at the end of the line no matter how old they are and players will be evenly in sequence distributed to them so we didn't really know what to expect but what we saw was almost the complete inverse of the previous situation where most matches were now starting with four players and as the player number increased the number of matches started with them would decrease with an audible bump at 8 for you to get the second imposter entering the game so our first thought was wait a minute our fourth player Among Us games even

fun like what's happening in there so we went in and we started playing to check it out and it turns out really like just because they may not have the same strategic depth as matches with more players it's still fun to chase each other around and have the whole thing be over quickly whether you were the first one to die or not so it's kind of like jumping to Among Us end game straight away so is sequential management matchmaking the ideal solution well not quite but just with a few more charts and bars I'll explain yeah but really we're running out of time and I want to make sure that we land this spaceship and make sure that you call all of our wins and challenges so for wins we talk through going with the beautiful Corner approach in pre-production which had a lot of benefits nested underneath that next was setting up pillars which help to guide our project as a whole and served as a foundation for the MVP and ensuring you're not over scoped right out the gate mihao then shared the translation into VR and Sean went into

details on our multiplayer VR play testing our challenges were networking specifically working with products that are also in development moderation features target audience sacrifices matchmaking and again VR play testing which really took up the entire Spectrum so what does the future hold well from our admittedly privileged perspective the time to make multiplayer games in VR is definitely Upon Us multiplayer games thrive in VR and the more your experience relies on interactions between players the better it will feel in VR there is still so much open space to explore and 2D games to bring into VR so good luck out there [Applause] yep uh thank you thank you so much for coming for listening uh grab the slides if you like they are actually up there on the website and while our talk time is up we do have time for questions and we'll also be in the wrap-up room in 204

afterwards but yeah any questions and oh fill out the surveys too thank you very much yeah thanks really entertaining and informative wondering what XR interaction framework did you guys use so the question was what XR interaction framework did we use the XR interaction framework we basically we basically just use what we had from our previous VR games and it was in-house custom built stuff okay yeah yeah within Unity one other quick question is there a reason you didn't go with mirror you went with Photon did you consider Mira as a networking framework at all I'm sorry could you say the name of that uh mirror mirror so it's another Unity framework no we did not go down yeah our Engineers looked at a couple different networking Frameworks and they chose Photon and then there was some there was some issues but it ended up working out real quick but it was it was a yeah it is what it is great thank you thanks hi

thank you for the thought uh so as designers when you want to validate uh Your Design you get your built-in engineers and in a non-vr application you can have two computers like usual message stab yourself in the back and see what that interaction is like in VR you've got one headset so do you just put one on put one off what kind of challenges do you have there and what workarounds did you find yeah so question about uh handling multiplayer in VR that's definitely something we've dealt with on this project and others um a lot of our a lot of our developers will have two VR headsets um we actually have gone so far as to buy mannequin heads that we can attach to our desks and we'll keep the headset alive and also point it in a reasonable Direction but our Engineers also made tools within Unity uh we call it the photon Fusion simulator and it'll open multiple windows and so you can then swap in the different bodies of different beans to test some of those multiplayer things so great tool development and support and also 25 mannequins off of Amazon thank you very much hi um how did you handle uh motion

sickness during play tests like if a play tester got motion sick what was the approach for did you just end the play test right there or did you find some sort of alternative to let them keep play testing basically yeah no great question about how do we handle motion sickness and play testing uh a lot of it is in that Google meet before folks would start we'd say Hey you know if you if you feel you know bad or discomfort at any time uh feel free to jump out let us know uh so we sort of you know set that expectation early that people could jump out we also had all the settings that we talked about uh you know vignetting uh things like that and so we also made them aware of that in the play test where we weren't testing to see if they knew about that and uh yeah it's all about setting expectations at the beginning of a play test and I will mention the 2008 uh 2018 talk play testing VR because we go into that and how to like because it's a very isolating experience to go into VR right and so you want your play testers even if they're on their own home with user testing during a pandemic you want them to feel at ease and so setting

expectations early and sort of having a script that we run through to make sure people know that they can get at that it's okay don't force yourself to keep going that's all part of that and it's different than motion sickness too so I think just helping to educate especially early on so they understand when those symptoms are starting to even occur can just help to again just ground people better and just make sure that they're aware of like oh maybe I should get out now before I do get myself just really sick because again it can start sometimes it's just you know a headache or things like that so just helping people to understand how different this is yep yeah great question thank you so first congratulations on your game I'm really excited to have another multiplayer game out there uh as a multiplayer game developer myself I see when I go to the store that Oculus has submission guidelines saying that we need to have moderation tools and the like in order for our game to get approved I was wondering if you could maybe demystify that a little bit and give usify that a little bit and tell us kind of what they're looking for and what kind of teachers we should build in ahead of time yeah

yeah um they have so any multiplayer game they do have just a standard API package that they have to give us and we Implement on behalf of them uh I'd say we are working to build on top of that as well uh I guess I don't know how much we can get into details about all of that but as soon as you enter that space they certainly will tag you and make sure that these are the the things you'll have to just Implement yeah the current thing is are pretty bird bowling so it's basically an ability to report other players through the quest platform right now because we were also called across platform we wanted our own means of reporting people so that's what we're launching soon uh and we also are looking into other solutions to mitigate toxicity and you know basically make the environment a little friendlier but you know meta is definitely in the process of developing this stuff and like what exists is really just a way to report other meta players through their system yeah maybe

it's a good follow-up question uh in terms of post-launch support how much resources should one you know kind of forecast on uh allocating for supporting moderation and like yeah I I was not necessarily involved in those discussions the question was how much resources should you allocate for a post-launch player support or customer support um you know basically it depends on the volume of players that you have and the amount of issues right we certainly had a bigger influx of tickets uh early on and we worked with a company called keywords for support and they're good at scaling right so we don't know the answer is we've kind of trailblazed a lot of this so we didn't have a lot of benchmarks for saying what does a giant multiplayer game in VR with 10 people in a Lobby you know cause in terms of support tickets and players player submitted requests so you know it was basically about let's be flexible let's be prepared to scale up if we need to we also had you know post launch we had a bunch of issues that we didn't catch

right which were repeatedly showing up in our support and then after we released our first and second patch that addressed a lot of those that definitely all quieted down a lot thank you very much thanks yeah I have a different question I actually teach game design at the high school level I teach with unity and we're actually getting into making VR games with unity so my thing is I'm just different from you guys sorry but like what should I be bringing to the kids and what what should they be prepping for because so many of them want a career like you guys have like I know it's a broad question but like maybe one or two things like what should we be bringing to kid what should I be bringing to the kids and then what should they be doing on their own what should they be learning yeah so the question was uh how do we bring uh you know game development to this because even if they're in high school what are the most important things uh I think we probably all have opinions I I would say honestly uh game development is very much a team uh effort right you've got your disciplines uh art design engineering sound uh you know QA all

these different people need to work together production I'm so sorry uh but yes no it's like it's all about how can these uh people work together communicate learn enough of each other's languages so they can you know uh propagate the ideas and then just work all together to make an awesome product so a lot of what you should be teaching is team projects uh make sure that the kids can work together uh can hash out these problems figure out what they're going to do and and also just like have a way for them to Showcase it right maybe there's a website for the class where they can put their website or put the games that they make or or something at the end of a semester where they get to show off their games to other classes I think that would be really awesome sort of like energize people to work towards something and the final thing I'll say is ask them to keep the scope as small as possible when they propose something like have them do the MVP right yeah yeah and that's huge on that note I think it's awesome though you're diving into you know VR development but it all

those principles about working as a team and being a different role on the team apply when you may when if you make them uh develop a board game sure uh so you can easily teach all of those things you know by having them modify Monopoly to be a better game or whatever it is make a new game from scratch yeah exactly don't be afraid to lower that area of Entry um in the Pittsburgh GDA we have a board game jam and we have the global game Jam which you know video games and board games and like you're making games and it doesn't matter the media and board game certainly has a lower barrier of Entry although VR is also thank you uh hi uh I'm Richard from USC and um the question I want to ask is that do you mind sharing some data on the procession user plate length for anonymous VR when I say procession it's like the first when they first enter game and then by the time they finish like a lot of games and lock out of VR because uh I'm also doing a multiplayer VR game and what I found is that when they have like friends entered together with them like in the lobby for example

they have three or four friends together the Playland will kind of increase I don't know if that's like a validation because we don't have any big bigger or play test data so do you mind sharing some of that information uh yeah I I do like the analytics uh we don't have an Insight I guess in what you described which is really interesting right so the question was hey how long is a player session in a multiplayer game like this and does it vary if they enter it with a bunch of friends and play versus they're just matchmaking with random people so I don't have that insight for our game like I don't know right if you're playing uh in uh you know with friends or not um but you know our average play session is I think around 20 something minutes uh and I think around and there's multiple sessions like that per day but some people it seems will jump in play a couple of games and jump out right uh the duration of an Among Us game varies wildly because if the Imposter gets caught red-handed right away it's over

and you know in the first meeting right or it could drag out so so it's a little tricky on average I think we're clocking around 20 something minutes got it thank you hi I just wanted some clarification on the minimum viable product so is that solely documentation or does that also include like a vertical slice play testable build that the team can get their hands on before pre-production ends yeah ideally I would say slice in there in there with our networking things you would like so the documentation certainly is Paramount right so even if you aren't able to meet that on that technical gameplay side of things having that documentation having that be kind of your Bible if you will uh North Star for the project is still important to you and we made it up for it right like then we're working towards you know figuring out the rest of those milestones and how to plan that out thank you yeah yeah the MVP helps set the team expectations yes correct yeah I

really enjoyed the talk thank you I was just wondering what it was like working with someone else's IP and uh because this was inters lost IP how much of a hand did they fly into production yeah no it's interesting um and again there was so much that we wanted to make sure that we stayed sacred to in this uh whole uh story uh so with intersloth we met up with them just right at the beginning and not very frequently throughout um they had hired a company robot teddy that as essentially their acting production uh and product side of things now uh and so they just made it very clear from the beginning as to what was important to them and what that what they wanted us to stay sacred to uh so it was just about uncovering that from the beginning so that allowed us to understand how much flexibility we had how much we were able to adapt things and what we had to again just stay very beholden to uh so I've worked with several big IP holders and making sure you kind of uncover that right up front it just makes that really good for us to understand like okay how much of this is ours and how much is yours how much creative input do you

actually want to have and how much Freedom do we have and so we just had that real conversation right very real come to Jesus conversation right up front to establish that early on and um yeah again they just really wanted us to you know take this to what we felt is almost like a consultant so shell games has a history in VR intersloth does not so they were like here's our established IP we want to stay true to this art style we don't want this to be a horror game uh what else did they say I think there was maybe just one other thing that they just really wanted us to see a secret to um you can say a certain word uh it was great I mean it was great setting these vacations and then it also helped that they were great to work with that was really nice yeah it gave us a lot of freedom and again with just the amount of time that we had though we just knew that priority needed to be getting it to Market so we just wanted to you know stay true to they just allowed a lot of that so thank you but people are immersed so that

things and to like whereas that doesn't happen nine choices new workflows much more collaborative and taking having one person take people in there it's a night did we do any design choice playing a VR game which is all right then mobile the game times of not that long and so I naturally saw that that people lobbying uh you know feeling it wasn't really a problem and it

there's a mode to play without voice chat so if that's too much for you and if you don't want to be needing to listen to other people constantly you can do that and then there's a chat like there's a text you know uh pre-canned like text responses stuff and uh trying to think there's something else but I just got slipped in my mind it's definitely something to keep in mind though it's a good question yeah but in easy game just like the 2D game which we've actually called the flat game that you can just kind of jump in and out of right so on the mobile game like I feel like I'll join and then you know just maybe do one match and I'm out and then I can jump back in whenever right we set it up similar in that way where right again choices do not have a browser you know a server browser to not have custom matches at launch so you don't have to think about oh no are these Pro players doing crazy things you know with settings that I would want to understand you know we made it really easy for you to just hit play and start playing all right I think we are out of time but yes we will be in the wrapper room again thank you all very much for coming fill out your evals

[Applause] [Music]