Ambisonics and the Great Outdoors

[Music] thank you hello everyone um my name is Blair my pronouncer she her and this is ambisonics and The Great Outdoors uh so as I mentioned my name is Blair um I am an associate technical sound designer at Team audio and my credits include Modern Warfare 2019 and warzone but really I'm just a nature lover and spatial audio fanatic so what is spatial audio well we can break that up into three main categories uh spatial audio is the sound's position the space the sound is in and the ear that hears the sound and we're going to go through each of these categories one by one starting with the sounds position but in order to talk about a sound's position in a virtual environment we

have to understand how we perceive the position of a sound in a real environment so how does the human body interact with audio well we can break that up into two main categories uh the human body interacts with audio with the head related transfer function or hrtf and the folds of the ears and these two concepts combine to create something called binaural audio but the head related transform function uh to clarify what that is is it's really just the difference between when you play a sound on one side of your head and that ear hears it versus when you play it on the opposite side of the head and the same here here's it um it's how the shape of the head interacts with audio the folds of the ears as I mentioned that's really just positioning sound um a little bit more clearly

but that combines with the hrtf to create this concept called binaural audio so when we talk about binaural audio we're talking about simulating the shape of the head and the folds of the ears traditionally however we don't really use binaural audio in video games most games use a channel based audio approach so that means that for each speaker there is a channel of audio there aren't any differences between the channels of audio that are being fed to the speaker system and the speaker number themselves um Dolby Atmos for example uh is uh generally 714 I believe uh at least that's what this image is here on the right um what that means is it's got seven uh seven speakers on the horizontal plane

four speakers above and one subwoofer what however when there's in this traditional game audio signal flow the spatial resolution of the player's audio is going to be limited to the amount of speakers that they have and this creates an equipment inequality with our players because ultimately we can't all have setups like this we can't have a million different speakers for hyper positionally accurate gameplay instead what I would suggest that we use is something called ambisonics ambisonics is a technology used for the simulation and or capture of an environment's audio including height by the way um what does that mean well when we break up this sphere

this uh sphere of potential listening positions that sound can come from uh if we break that up into let's say four uh points uh that is first order ambisonics uh nine points would be second order 16 points is third order up to fifth order ampasonics which is 36 points sample points or each um each point on the sphere rather is broken up into sorry let me rephrase um uh the sphere broken up into 36 sample points is fifth order ambisonics it's effectively how many speakers virtual or actual we break up a sphere into and that's ambisonics ambisonics came about around the 70s but

it wasn't really widely used until virtual reality and digital signal processing um there may be some research people who might beg to differ with me on this um really Digital Signal processing uh allowed for widespread use of amazonix um and virtual reality in general allowed for consumers to use ambisonics in their well in their gameplay experiences so ambisonics is a technology that provides for varying amounts of spatial resolution based on the order of ambisonics used so if we go back to that sphere of all possible positions that a sound could come from we would sample that at four points for first order ambisonics up to fifth order ambisonics which would sample that at 36

points with them and that would result in a much higher spatial resolution or spatial accuracy for listening so when we talk about that spatial resolution uh generally that's referred to as something with spherical harmonics and I'm not going to go into that because quite frankly it's a little bit dense but this is generally the um this is generally the diagram that people will point to and what I want you to take away from this is just as the order of ambisonics increases the more specific each sample Point gets the less amount of audio information that sample Point has to communicate but the more uh sample points there are and ambisonics allows for us to immerse our players more fully and equally

because ultimately when we use ambisonics in conjunction with binaural audio we're able to really mitigate this equipment inequality that comes from one player playing on a five one setup but another player playing on headphones um because we as game developers effectively need to design for multiple different speaker setups but ultimately amazonix allows for us to do so a little bit more easily so I believe that ambisonics provides a good solution for the sound's position providing good high quality positional information and I believe it can also provide a solution for the space that the sound is in because ambisonics if we're using ambisonics to simulate the position of

sounds that opens up the possibility for highly spatialized precise and immersive convolutionary convolution Reaver is a type of reverberation Reverb Echo whatever you want to call it that takes a impulse response or a frequency sweep or really even a gunshot in space a sound recorded in an environment and it differentiates that between the sounds that was played so by differentiating say a frequency sweep file and the frequency sweep played in an environment we're able to come up with that sound information that difference that Reverb what because of that Reverb

sorry let me restart that sentence um there are no high order ambisonic impulse responses of exteriors on the market at this time because convolutional Reverb while that opens up the possibility for highly spatialized and exciting information to be carried within the Reverb that we have right now there aren't any exterior uh impulse responses that are in high order ampasonic think about what we could be hearing in our games we could be hearing sound bounce around us um like a cityscape from downtown San Diego or the trees carrying and bouncing and muffling sound in a really interesting way we could be hearing all of these environments in our games but because there aren't any uh high order ampasonic impulse responses of exteriors out there or not

so I'm trying to fix that by recording third order and Sonic and pulse responses and exteriors across California so ambisonics is a technology that allows for the simulation and or capture of the 3D sound of an environment including height basically taking that sphere of possible positions and dividing it up into n amount of points but when we take all of that information we need to turn that into something that the player can listen to because the player isn't going to have 36 speakers they're going to have two or they're going to have a 5-1 set up or maybe even Dolby Atmos so converting that format to the ear that hears the sound is done through something called binauralization plugin generally

however uh the PS5 accepts amazonic input or object-based input from the game so we don't need to use a binauralization plugin for that particular console but for the Xbox series X and PC while they do have native Dolby Atmos support it's still a channel-based approach so it still depends on the amount of speakers that you have and ultimately for developers shipping on the PC and on Xbox series X by neuralization or the converting of let's say fifth order ambisonic audio the converting of any form of ambisonic audio to a listenable format that conversion or binauralization must happen on the game side via a plugin luckily there's a lot of plugins out there for this especially THX spatial

audio which is something that I'm very excited about personally um it uh Oculus audio SDK is a tried and true um method that's I believe phasmaphobia uses this But ultimately it struggles with handling more than two speakers atmoki ears allows for customizable hrtf based on age um that is to say it is really useful for adjusting for if you're a child or a teen I believe are the settings that it has but THX spatial audio is something that I'm very excited about because it allows for multiple speaker setups to be converted to that is to say it allows for the conversion of ambisonic audio

to a 5-1 setup or a headphone setup or a 714 setup but all within the same plugin with binauralization we can terrify because binauralization allows for all of that positional information to be simulated speakers no longer have to be physical they can be virtualized it could be right behind you and you wouldn't have to have a speaker there because ultimately with binauralization we're able to take the positional information from ambisonics and convert it into a smaller Channel amount whether that be stereo or Dolby Atmos so there are a few limitations to most

binauralization plugins um for instance the Oculus one is only stereo and everyone does have a different hrtf so uh or head-related transfer function so what might sound really positionally accurate for one person depending on the hrtf that is used in the plugin it might not for another so these are just things to be aware of however I believe that ultimately modern games can use ambisonics and binauralization to achieve high quality spatial audio that is to say ambisonics allow for us to simulate The Sounds position in a positionally Accurate Way we can take an environment and instead of breaking it up to let's say a 5-1 setup we can sample it at up to 36

different channels and from there we can simulate the reverberation of that space on those channels and convert that all into something that the player can listen to regardless of their speaker setup thank you yeah questions [Applause] um you mentioned that you capture impulse impulse responses in the city using a third order ambisonic microphone so I wanted to ask what I mean where do you place the chirps the sweeps that you play how many trips would you play and where would you place them um I am in the process of recording and pulse responses I have yet to

um I have yet to practically record in the city at this point um I'm in the process of making that happen uh but if I um off the top of my head um I would be careful with placing it directly in the center of the street because that would result in potential um phasing issues that's really something that I want to be careful of when I do record in the city is I want to be careful of facing issues so I would probably place that speaker in roughly off center uh like just a little bit off center enough so that like if this was a um if this was a two-way street I would place it in one lane of traffic um obviously not wild cars are going by but um that's how I would do it because

I would worry about facing otherwise and that way we can say okay this is how one uh one area kind of sounds and from there you can kind of build out through different positions um you can build out different types of impulse responses um you could try it on the sidewalk you could try it um in a tighter City Street yeah there's really a million different ways to do it but that's just how I would do it got it and one last question maybe you can give your uh your thoughts of how to transition from three Duff capturing to six stuff rendering I mean how would the ambisonic uh static place that the ambisonic was captured from being utilized in a sixth of gameplay

um in terms of converting from uh various singular captured um impulse responses to uh six doff gameplay um I would say that I honestly don't have a great answer to that question unfortunately um yeah I just would record as much information as I could with uh individual sample points and kind of like ambisonics try and mitigate the um I would try and take those sample points and build from that a particular image if you will um I don't have um I don't have a great answer for you

unfortunately in regards to uh the six degrees of freedom um but that being said um I would say that if you were doing six degrees of freedom try and record with multiple um multiple microphones at the same time potentially that uh I know Azalea has a uh six degrees of freedoms like time-coded link setup that uh they use that's something to look into potentially thank you so much hi thanks for the talk um our uh is ambisonics or binauralization um impacted or Reliant upon standardizations of like other audio protocols or standards in volume and format and platform like

do a number of things have to be aligned to make to maximize the effect of ambisonics and binauralization is that making sense yes um are you basically asking if ambisonics is a fit for every game yes every game and every platform and um every sound format like are there external influences that really throw a wrench in the gear of this having the desired effect yes very much so so I find that um building a game for PC and current gen consoles um especially if that is a first or close third person uh game like a RPG type layout that is where ambisonics really shines because then

there's uh because there is a little bit of a performance cost as I understand it um with ambisonics because it's handling that much more information but that's not something that as I understand it is much of a problem unless you're going into say the PS4 and um potentially going into Mobile and things like that so if I was building a RTS I would maybe not use ambisonics really I'm presenting this as a tool uh if that makes sense I'm because I'm finding that in the types of games that I have worked on and am working on in the future I have found that ambisonics has provided me with more information and more

more gameplay information really um for the player so this is all to say no ambisonics is not a one one-size-fits-all solution perfect thank you hi Blair um uh I'm wondering as an indie developer how amazonics might be able to be integrated into my work um and if there are any cool ideas for gameplay that would be really awesome with amazonix that you have and would like to share yeah um so ambisonics is ultimately um that would be where you would set like you would set up your listener to use a an ambisonic format uh if you're say making a first person horror game let's use spasmaphobia for example because I love that game and I die a lot in it

um but uh if you're making say phasmophobia um you would set your listener to third order fifth order ambisonics whatever you deem gives you the right amount of balance between performance and um and audio information and from there um you can take that um and when you play sounds uh whether they be mono or um well I think they would have to be mono in this particular case um but when you play sounds and it's picked up by that listener uh that would then get converted down to the uh the listeners format through the binauralization plugin um I'm so sorry I lost my train of thought can you please repeat your question uh yeah I asked as an indie developer

how I might be able to integrate ambisonics and if you have any awesome gameplay ideas that would be absolutely perfect with ambisonics okay um I think gameplay wise um ambisonics is really good at emphasizing positional information so if you want to be able to say hey there's a sound exactly there ambisonics is great at that um so really in terms of gameplay um it's it's really limited by what you want to do with your sound design um and in terms of integrating it as an indie developer um uh kind of like the previous question was saying uh it is a bit limited by the platform and the um yeah really it's limited by the platform and the genre of game

um so ultimately I don't see any reason why an indie developer might not be able to use ambisonics um if they're shipping on the same platform as a larger developer might be um yeah I hope that answered your question yeah thank you [Music] hi Blair thanks for the talk um it's cool to see interest in spatial audio there's a lot of folks here so um when I was looking into spatial audio maybe five or six years ago it seemed that the variation in hrtfs from person to person was a big hurdle at the time and I was I guess hoping to see that with a lot of the VR you know steps forward that have been happening that there might be some advancements in that but it seems like that's still um kind of an obstacle are you aware of

any recent research or steps forward to combat that um a lot of spatial audio developers uh like taking the th-track spatial audio for example um they're focusing on they provide an out of the box uh in their words high quality um hrtf but there's also people are trying to implement a the personals the personalized rather um hrtf of uh each consumer so um I think with um with the THX spatial audio the goal is to Implement like you would take pictures of your head and from different angles and then send that off to THX and they would generate an hrtf for you

um that is really the only big advancement with that that I've noticed I think a lot of people are trying to make a one-size-fits all um and I if I recall correctly um there are different companies that are trying to work on customizable Solutions like with preset type hrtfs But ultimately that is still a hurdle um customizable hrtf if someone makes that plug-in like like a plug-in with uh swappable hrtfs based on your um based on your age based on uh your head shape and you can pick the one that's best suited for you that that is something I would be interested in seeing But ultimately I think the closest we're getting is uh the promise of personalized hrtfs from THX at this

point [Music] thank you