Spatial Audio in 'Budget Cuts': A VR Stealth Game

[Music] hey everyone and welcome to our talk spatial audio in budget cuts uh ironically we will not be playing back any sounds but you have to use your imagination and speaking of that you will all receive a feedback form after the talk so use your imagination and pretend it was super good and give us a good review so my name is jonas jalbari i'm cto of a company called yesterment and we make super secret unannounced music middleware but more importantly i was the composer and audio designer of the game budget cuts and i am lakulesh anthani i am a valve person and in my valve time i do valve things like spatial audio um so this talk is really going to be about our experiences in integrating spatial audio into budget cuts

via valve spatial audio tool called steam audio so before we get into all of the all of the details about all of our experiences just a quick overview of this talk i am going to be talking about the technical side of things jonas is going to be talking about the artistic side so kind of like good cop bad cop here so first off um spatial audio has been getting a lot of attention in recent years but already there's so much confusing terminology out there that it's kind of hard to keep track of what someone means when they say that they've implemented such and such spatial audio feature so i'm going to give a quick overview of some of the concepts and the first one is hrtf head related transfer function so in the real world the shape of your head affects how you hear things the sound bends around your head before it gets into your ears so the effect of your

head your torso these flappy bits of your ears they all affect how you hear things and where you perceive a sound to be coming from and an hrtf is a set of filters that models that hrtfs are typically measured from dummy heads or real people and they're measured in a bunch of different directions and all of this information is used to resynthesize the positionality of a sound the other concept is ambisonics ambisonics is kind of a surround format but unlike your typical 5.1 or 7.1 it's a spherical representation of a sound field so it includes verticality as well uh one way to maybe think about this is it's like 360 video but for sound again ambisonics can be measured i mean recorded uh using a bunch of different kinds of microphones you can also encode a monoclip into ambisonics given a

direction and you can decode ambisonics using hardware software decoders and you can spatialize ambisonics using hrtfs i'm going to hand it over to jonas who's going to discuss a few more spatial audio concepts so what you have to take into account when working with all these things is stuff like distance cues and that means a sound will get softer and more muffled the highs will get rolled off the further away the sound is playing from the listener also if the sound is played back in a reverberant space you will get much more of the reverb and the late reflections then we have stuff regarding the rooms so uh room size q would mean that the way a sound will bounce around the room the reflections will give an indicator of the size of the space stuff like how long the reverb tail is but also which materials

are being used in the room that will affect the coloration of the sound and this thing is super important for a presence and selling the illusion of being in a different space we also have stuff like sound propagation and that in effect means the way sound flows through a space so sound will be bouncing being blocked being scattered diffracted going through different materials and being colored by that going through openings doors windows just how the sound is bouncing around basically so uh this is pretty cool but uh it's nothing new and uh coolish why are we seeing this uh talk about special audio right now that's a good question actually um all of this tech or much of this deck is really several decades old so why is this all relevant now

there's a few reasons one is that computers are way faster now you have multi-core cpus gpus you have a lot of compute you can throw at problems like spatial audio so that's now computationally feasible the other thing is vr with the rise of vr in the last few years the importance of immersion has gone up and spatial audio is a as a very important way through which you can add immersion uh to your audio and a side uh benefit of this is that with vr most of your users are gonna be using headphones which is the ideal delivery mechanism for spatial audio so all of these factors together are kind of why spatial audio is all the rage these days so that's an overview of spatial audio uh so now i'm gonna hand it over to jonas he's gonna tell us more about budget cuts which is what all the kids are talking about these days

so body cuts it's all the latest rage uh since like three years ago uh it's a rhythm game where you have two lightsabers and oh the wrong game uh so it's it's a vr spy game basically there's a snippet of our trailer um and it features stealth deadly robots uh you have to hide from the robots uh and trying not to get seen or you could choose to kill the robots the only way possible with knives and scissors so the game is designed around two core concepts it's all about room scale gameplay so you'll be crawling on the floor hiding behind cubicles crouching ducking and weaving stuff like that and most interestingly it features a very novel teleportation portal mechanic that works like this so you fire a

ball with your hand cannon it opens up a portal that you can look into and preview the new location and once you're ready you teleport to that location and the teleport portal will envelop around you it feels very immersive and it also aids in gameplay because you'll be able to pre-survey where you're going so i'll be the first to argue that spatial audio should be implemented into every game coming forward from this point it's super nice but most importantly like for a game like body cuts it's even imperative for a stealth game because spatial audio will give us clues about where the enemy's are situated where they're patrolling we will get cues about the space that we're in if we're uh

hiding behind safe cover or not and um we've all been uh talked uh talking about like immersion for uh ever in vr but it's really about that so audio normally is used to sell the illusion and even more so in vr so it makes sense that we actually use this tech to get that extra mile so we've been using reflections for instance as one of our core gameplay cues which allows the player to properly form a strategy based on what they're hearing so while you normally in a game would be able to hear that ah there's a sound emitting from the next room with special audio technologies we can hear the size of the room that's on the other side of the wall we can hear if the robot is walking away from us or coming towards us just based on the way the sound is being colored

so that helps the player to correctly assess a threat and time their strategies accordingly also a core gameplay cue we use occlusion which means the way a sound is being blocked essentially it's a ray trace from the audio source back to the listener and if there is geometry intersecting that the sound will get muffled according to what kind of material the geometry intersecting has applied to itself so this can tell us if we're uh behind safe cover or if we're in a direct line of sight to an enemy um also there are this telephone similar to the matrix where you'd [Music] pick up the phone to get new information about your mission um and then you can hear quite clearly if it's on in a different room even though it's far away uh you can pinpoint which room is coming

from so all was not well working like this and la coolish will uh go over some of that all right so for the next few minutes i'm going to talk about some of the things we had to do to get spatial audio working well in budget cuts what i will not be talking about is hrtfs or occlusion because this is 2019 and that's not really as much of a problem as it used to be so what i'm going to be talking about here is sound propagation so this is a quick overview of what we did we split sound propagation into per source early reflections and a single physics based reverb the early sound paths for every source were guided using a custom portaling system and in addition to all of this we faked some of the spatialization for ambience and music using the steam audio ambisonics reverb system

so let me start with the real-time reflections and early pass so ideally and this was our starting point we wanted to model sound propagation from every single source individually trace rays all over the place apply convolution reverbs for every individual source and everything would be great but obviously that's a lot of compute there are ways to mitigate this you can use multi-threading you can use gpus there are options but those are not always suitable for every title and in particular they were not suitable for budget cuts so we decided to do the following we said let's split uh let's split sound propagation into two parts for every source we're only going to compute early reflection so that's like one or two bounces and then the reverb will be applied separately and i'll come back to the reverb in a bit so we implemented this it worked more or less okay uh performance was

more or less okay but it wasn't perfect and in particular we had an issue where sometimes we encountered sudden frame rate hits which were kind of hard to track down what was going on so we did a whole bunch of investigation and what we found was that there was something quirky about how unity's audio engine was working and what it was doing was every time an audio clip was played from an audio source it was re-initializing a whole bunch of spatializer state now this is interesting because spatializer state is not really tied to an element in your mix graph it's more tied to the physical game object that's moving around in your scene so we obviously reported this issue to unity and they began to work on a fix but that fix was not going to come out in time for budget cuts to ship so we decided to you know approximate things further

and our solution to that was to first take a step back and really think about why we cared so much about these early sound paths and the important cue here was we really only care about knowing which doors have robots on the other side which windows which vents have robots on the other side so that you don't go there and so what we did was let's manually mark those up and then use that information to guide the steam audio ray tracer so that it doesn't have to calculate as much it can run way way more efficiently so that's what we implemented um just want to point out that this solution was developed specifically for budget cuts and is not part of steam audio at this time so that's the early paths the other issue was reverb so we obviously couldn't apply reverb to every source so what we do is

we apply a physics-based reverb for a sub mix so rays are traced out from the listener's position gathered back at the listener's position after a whole bunch of bouncing around and that is sufficient to convey a sense of room size room shape materials that sort of thing but there are issues well before i go into the issues one of the benefits of doing this is that your compute cost is independent of how many sources you have you're just applying the reverb to a sub mix so now to the issue which is that because of the way physics-based reverb works because of the way rays are gathered when you move close to a wall what happens is the size of the room begins to appear a little bit smaller and the solution to this problem is to use a directional reverb so you model how energy decays in different directions and so if you do this correctly what will happen is when you move close to a

wall the tail of the reverb will shift away from the wall and you'll get a smaller shorter echo from the wall itself and so that's what is implemented uh in steam audio and in budget cuts but there's a gotcha there as well and that is latency so physics based reverb is going to update at two to ten hertz and that kind of latency is fine if you're just moving from one space to another but that's not fine if you just turn your head around and have to wait like half a second for the simulation to update uh based on your based on where you're looking and the solution to this is ambisonics so one of the nice things about ambisonics is that you can take any ambisonics buffer and rotate it to any arbitrary orientation and so what we do is we run all of our physics-based reverb simulations in world space so independent of where the listener is

looking we do our convolutions with these world space reverbs and then once all the convolution all the audio processing is done we just take that one frame of audio and we rotate it uh to match the listener's orientation so we get decoupled orientation updates that are calculated at audio frame rates so those are the two issues that we solved with reflections and with reverb and now jonas is going to talk about ambience and music thank you lecoulish i hope all that was crystal clear so spatial audio is essentially a whole new paradigm when it comes to how you approach your mix things that are still stuck in an old paradigm is how you treat music in games or stuff like ambiences and when mixing like spatial audio together with the a flat stereo sound file uh

it's feels very disconnected and doesn't feel part of the experience and when it uh comes like music and games are supposed to propel the player further into the experience not pull them out right um so there are different approaches you could do with music in vr and one would be to always uh from the design stage uh use diegetic sound sources so that the music that is actually always emanating from somewhere in the space we did some of that in body cuts there are a bunch of radios scattered across the world and if you move closer to them the music will fall down into a specialized radio and that goes gives a pretty nice effect [Music] another approach i used was to essentially when mixing the music use binaural panning plugins and uh

move slowly automate movement around the player where the instruments will be slowly facing around in the sound stage which doesn't at all line up with what is going on on screen but the facing of the of the sound sort of allows it to blend more properly and together with the spatialized sounds the preferred solution would instead be to play back an ambisonics music mix so that you would mix your music in ambisonics and then when playing it back in the game you can choose to have it rotate together with the player's head or independent of it so that for instance drums would always be coming from that direction or whatever depending on are you moving another similar issue comes with ambiences so you need some sort of like uh to to to convey a space you need to have uh some kind of ambience and

the preferred way to go about it using spatialized sounds would be to just use reflections and have a ton of bounces and that in turn would create a dynamic room tone right but since we had performance issues with reflections we couldn't do anything like that at the same time the game world is quite empty because it's an abandoned office building with patreon robots so you need some kind of sound uh to be actually to to to get back something from the room so uh the way we did that was to fake things up with piping like a mono a sound file that played back filtered white noise uh i attached that to the player so it's always moving with me and as you're teleporting yourself into say a small vent that noise will then fill up the space and you suddenly get a sense of the size of the room the same goes when going out to big space then it would be something bigger

[Music] so spatial audio is super great super cool uh but it comes with a lot of caveats so for instance all of these plugins you can get for free more or less there are many that are not i suppose but what's not free is performance and you need to budget for performance before starting with this like especially the more of the features you add in the better it's going to sound and you will notice this drastically but it will will also tank your performance so you need to rain your expectations in a bit and try to not use all of the good stuff just yet and of course being a beta kind of software there is no community if you stumble upon an issue you need to directly talk to the developer

and i'm not sure that everyone is so lucky as we've been to have them like on speed aisle and say hey could you add this thing um but yeah you need to uh keep that in mind that it's not like any of the big uh things like fm otherwise that has a lot of community around them and the documentation is always changing of course and it is not without bugs and sometimes you can question your own sanity and it turns out it was a bug like with the unit thing all along and a bug like the unity bug did cause me to have to redo the whole mix several times because we had to cut features so that's like just a couple of weeks just doing that the last time two weeks before release even so it's not it's not a plug-and-play solution it's what we're trying to say so you need to really think about this from the get-go you can't just activate the plug-in and it's going to sound nice and spatialized

so giving back the word to la coolish it's going to talk about what a summary of what we did right so for those of us who just zoned out through most of that um this is what we talked about uh we split spatial audio we spit we split sound propagation into two parts early reflections for every source and reverb for a sub mix based on the listener position we guided the early reflections using manually placed portals and we used ambisonics reverb to spatialize ambiences and music so you know things didn't go exactly as we planned so what would we have liked to have done better you know what are some ways in which things can improve so one is obviously we would have liked to have had more success with the early

reflections the bug that we reported to unity that was leading to a lot of the performance issues that i believe has been fixed and a fix is shipping in 2019.1 or thereabouts having said that the portal system that we came up with is great however it may not be suitable for all kinds of titles it depends on what kind of scenes you have and it worked well for budget cuts so we would ideally want to try to [Music] maybe come up with something more semi-automatic something more flexible that can solve this problem more efficiently than just rate racing all the things the other thing we would like to have is what is called near field hrtf and the what near field hrtf means is when you hold some sound source really close to your head that's called a near field source so in budget cuts for example you can hold a phone right up to your ear and you want that to be rendered using a

near field hrtf the hrtfs you normally have including the one in steam audio they're far field so they're recorded from a few meters away from the listener so you want to get near field data because that adds a more visceral kind of feel to that specialization so those are some of the technical things we would like to improve uh so jonas what are some of the less technical things we would like to improve well before getting to that i'd uh want to take the opportunity to say that we are going to update buddy cuts to the new unity version so that we can put in all the nice things back in i'm not sure that we're going to do that but i'm saying it here so that the team can then be forced to do it essentially it's a promise so what we want to do more as well is so the reverb works great

but like i said before it's all about selling the illusion and sometimes uh realism is not as good as what you can imagine up so we want to be able to tweak the reverb further and like cheat a bit and make reverb tails longer for instance or use the reverb calculations to control a third-party reverb or something similar also since we're using this hand cannon thing as soon as we fire the cannon we already know where the portal will land so then we can start precomputing the reverb before even going there um yeah so instead of having a half a second delay of going to space until you start hearing the change of the space now we can do it instantly if we do it like that um using ambisonics for ambiences so uh

recently steam audio added ambisonics playback it's already available in the other specializing suites as well and that means that you can have an ambisonics recording of say you know a city or a room and play that back and have that rotate together with your head that works super well for far-field sounds but for close sounds you'd want to not do that because you can see that they're not moving or that the sound is not moving the distance with those files anyway so conclusions uh it's super expensive uh performance-wise so you need to account for that when working with any bleeding edge tech be prepared that everything is going to break all the time um but as a vr developer you're always working with like

the vanguard of that stuff so you you're probably used to it spatial audio it's extreme extremely important for vr immersion or rather like room presence uh to get you tricked into being in a space um imperative for any kind of stealth game and it's truly a next-gen audio experience me as an audio designer i'm super happy about the advancements of audio technology in the past few years within games and it's in due time that all games start to sound much better yeah that's me and what about the bad cop and as the bad cop it is my duty to inform you that if you want to use spatial audio for your next project start early it is not just something you can tack on at the end flip a switch and be like yes this is awesome you're going to have to change a lot of your workflows it's

going to affect a lot of the different aspects of your production it's going to touch level design artwork everything so just keep that in mind and that's all from us thank you and we're happy to answer any questions [Applause] i thank you that was great and great game as well thank you are you biased um what can i how can i ask us is there other packages we should try before we try the steam one or in addition to the steam one uh i think you should try all of them like they all have different uh timbre if you will the filters sound a bit different the hrtf panning it's sort of this same with most of them but they all have different ways of d of uh dealing with certain issues uh so uh yeah

try all of them uh they're some are better for some things like google resonance is better for mobile but it's not as powerful feature-wise thank you thank you more here we go hey do you want to talk a little bit about the um ambisonic reverb that you used and like what like what product you found there or products out there that we can use for our solutions so in this case the ambisonics reverb that i was referring to was the one that's built into steam audio okay so all of the reverb calculations happen in like ambisonics so you can rotate the reverb freely and did you find it impacting performance or was it just smooth so for sure the it depends on how many channels you have

right so the the performance impact scales linearly in how many channels you have so first order ambisonics is four convolutions second order is nine and that sort of thing so but overall as long as so we did a listener-centric thing so it was just being applied to a single sub-mix so since it wasn't scaling with how many sources you have the impact was not that significant thank you hi thanks for the talk um just wanted to thank you as someone who also lost a week and a lot of hair for the unity spatialization reinitialization bug uh for opening up a channel to them yes give a round of applause um yeah that was not great veterans of the unity rheospatialization war um but yeah so i will be i took a picture of that screenshot i will be sending it to my team thank you awesome great talk guys i just had a question

about the wall materials and reflection absorption coefficients did you use that a lot did you use lots of different wall materials or so being an office space i used as many materials as you would expect in an office so carpeted floors uh i'm not used to cubicles that's right so what i'm not used to cubicles right right right yeah and some of the materials i had to like since we couldn't uh suddenly use all the nice reflection parts i had to redo all the materials like to allow them to transmit more of the sound through them to fake that so yeah it's mostly the standard materials but uh tweaked thank you i'm definitely more uh curious about uh your approach to that kind of uh ambient layer that you said you kind of had like a for like room tone you put white noise

in and then you sent it through the reverb did you actually stick white noise in or did you kind of design like a room tone for each room and like what was your kind of thought process and how did that end up serving as a bed for everything else right so uh about the faked room tone uh so essentially it's really just a filtered white noise uh so not all of the high frequencies are still there but i figured um since the reverb is used in using impulse convolutions um i just want wanted to like trigger those uh frequencies uh so that's why it's a noise and uh it's not really a tune at all that's why it's so neat it's just something that's easily added and you can hear instantly that you're in different spaces and then my follow-up question is when that reverb was kind of um like following a listener for that kind of situation where you're like near the wall like how how closely did you parameterize

that to like distance from the wall was like a gradual rotation that happens or was like a focus like what was the kind of parameterization you you thought of there so uh originally the reverb was not directional and i had that issue that if you were close to a wall the reaper would think that you were in a smaller space but la coolish kindly fixed that so now it doesn't really matter if you're close to a wall you can still hear that it's the sound of a space so you weren't actually rotating it in that situation it was just happening naturally because it was already a spatialized right okay cool are we done for time so the last question so i'm not a sound designer but this talk was fascinating because i'm interested in spatial audio when i've been messing with it uh i found that open-backed headphones are the best in terms of sound reproduction for spatialized audio but when listening back on speakers it sounds very thin but there are certain times that you do want speakers over headphones for certain games is there a way that we can

kind of make a middle ground that does have specialized audio for those who do want to go with headphones and a special mix or something for headphones or is there like a way to make sure that those play very well together or no you want to talk about decoders um sure so there's a bunch of different options that you have one is obviously to just assume that everyone has decent-ish headphones and say that this requires headphones but if that's not an option uh you can then rely on external decoders so there are some uh some systems that have ambisonics decode in hardware and they can handle spatialization they will typically do things like crosstalk cancellation to improve the spatialization over speakers or alternatively you could just do what we did for counter-strike which is like you turn you have a you have a special setting for stereo with

headphones versus just stereo and the user explicitly makes the choice am i using headphones or not and if they're not using headphones the hrtf goes away thank you and to add to that uh for more uh human uh intelligible speech uh no uh well so the the nice thing about ambisonics is that uh obviously we're using uh uh hrtf decoder but uh you could essentially make a decoder that says i have 200 speakers in a sphere around me played back to those or i have stereo played back through those that kind of thing so the format itself is very versatile like that thank you thank you uh we'll be in another room i suppose answering more questions thanks