Finding Space For Sound: Acoustics in 'Avatar: Frontiers of Pandora'

[Music] so welcome everyone this is uh a movie game called Avatar froners of Pandora so naturally this talk is a sequel to the talk that I did in 2020 which was Finding space for sound environmental Acoustics of uh Tom Clan's division 2 uh I have a couple of caveats to point out one of them is I will use the term emitter a lot and what that is in Snowdrop engine world is a virtual render context that has a position and an orientation that may or may not be used by the sound designers and uh all of the videos I've made are in Stereo fold down so unfortunately you won't get the full benefit of object mixing so just imagine in your minds when you're hearing the stereo how great it would be if it was an Atmos but it isn't other spatial audio systems are available

so my name is Robert Banton my pronouns are they them and my job title is snow Drop Audio architect uh what does that mean I'm an audio programmer by trade uh I steer the road map for the AUD Tech within the snow drop Pipeline and snow drop of course is an engine within Ubisoft that we use for making lots of different games my background is in EG codex originally Music Tech and cinemar audio and this year is my 10th year in the games industry indry so to summarize what we're going to be talking about today two things bubble space 2.0 in the last talk you talked about Bubble space I just want to show some things that happened the last five years that might be interesting and slapback 4.0 so to be clear what we shipped with division 2 was slat slatback 3.0 two other versions before that never got shipped or saw the light of day so I'm keeping track in case you're wondering I don't remember there being a three or a two you heard three

okay let's get started so bubble space 2.0 uh what is it so what we're doing is is that we are devising a system for finding the free air around the player so in an analogy like this with a division player it's finding the best fit cylinder in a 3D space and that's the version we shipped in uh 1.0 was shipped in division two um the tightest fit cylinder is about the azoth plane and the height of it is from above the player's head uh and it uses Ray casts it uses them radially and above the players position uh so how it did that and I they have a very sort of simplified diagram here it would do out three radially and then find a new position do another three and another three and another three and then it would rotate round so it's kind of like a radar uh that way you can do effectively 12 you're doing over four frames which AIDS in mitigating the CPU cost and then

whilst it does these measurements by the way these dots the green dots are a ray cast to hit nothing and a red dot is a ray cast to hit something so that gives you an idea and the Outer Circle is the maximum range of the raycast and this is a plan view of typical American city like uh Washington DC for instance and of course that means that when the player moves around the world the green circle is shrinking and expanding constantly and from that we can do some intelligent mixing so here's a video uh this is like pre-shipping I guess it was a year before we shipped division 2 of it working [Music]

f [Music] [Music] so there's some examples there what you were seeing there was controlling weapon Tails so each bullet has the initial hit and then spliced onto it's a tail it has some R bed into it and it was switching sizes of Tails depending on the size of the environment and also mixing in some kind of like local uh ping pong delays maybe when you're in Alleyway mix out

large ambiances that were in the street that kind of thing so what do we do with bub space 2.0 so the thing that we did before we still do it's an option is called the Minima azoth estimation and I call that the blue ring so if you look here the red dots are a rcast hit and the green one is where it's hit nothing and the yellow ones is where they both hit but it's going to ignore one of them and the reason why is is that whenever you have three hits It's sorting and showing the two shortest distances so the furthest one on the end of the that sort gets ignored so it's only averaging the two shortest ones uh and then the overall effect is this uh blue circle so that would have been our asth estimate for for bubble space back then uh and it works great except that sometimes when you're getting into cover or going up against the wall it shrinks very very small and sometimes come out shrinks very very very large in some cases you want that sometimes some some cases you don't so uh I have had to look into it like can

we have a less aggressive um azoth estimates so we came up with this thing called the Maxima one what's the difference the difference is the Sorting order is reversed so it's exactly the same calculations but now we're taking looking at the two largest points and averaging them so in that case the two red ones are being used and the green one is being ignored in the yellow case yellow dots one of those is being used and the red one's being used if you look there the actual circle is larger but is actually more like the average between in this situation where you're in a in a large rectangular space if you look at the total distances you average between to it's more it's more close to that rectangle than the other one was the other one's more like closer to the the minimum width this is more like the average width so it turns out this is preferable for outdoor environments where you're coming in out of a cave you're out in the forest and uh it's less extreme and it means obviously that problem when you got into cover doesn't occur so here's an

[Music] example I forgot to mention also a lot of these videos look kind of L because I often put the graphic settings right down so I don't get GPU bound so if you're wondering why do the game look so ugly that's why but you should be losing your ears anyway so who who cares so uh that's the asouth estimation done so we have two modes and it's basically the same calculation so depending on how you sort oh hi say um depending on how you you you sort them you can you can get the the two largest or the two smallest and that's it um for the Zenith estimation what we were doing originally was we're taking three Ray casts going straight up and we were taking the the ones that agree the most the reason why you need to do that is you might be standing under satellite dish

lamp posts or something if you had one Ray cast you might get a really weird Edge case where suddenly it feels like the whole world has come down but it's just because you're standing under a lamp post so the idea of doing three is this make it a bit more de Democratic one of the things about it is is that the minimally the two heights have to marginally agree for the to affect change so once it takes a measurement uh two Ray casts have to agree again for it to change so you don't get sudden changes it basically everything has to agree before uh something happens it's intelligent mix makes it a bit more stable and the tolerance is based on the sample area above the player so that that actual radius distance is what gives you the tolerance The Wider the tolerance sorry The Wider the radius the less tolerant it is to things that imges above it so imagine in this uh in this room here everything would agree because it's a flat ceiling but maybe if we're in a cave or in a forest canopy they'd be constantly changing so if you make a smaller radius they're more likely to agree in a larger radius they're less likely to agree so that that was how we controlled the tolerance

and of course edge cases are easily dealt with as you can see here with this hypothetical block hovering in the air so what did we change so it it kind of got interesting because the Zenith estimation now has a variable tolerance this is something Dave atino asked me about he said look I want to be able to fly one of those flying lizards called Ian under like a a stone Archway this huge Bridges uh but I want it to kick in and out the the the sparkly sweetness of the rain immediately if you fly under the bridge but if you're 50 M below don't kick in immediately and in fact at some point maybe for the perspective of the rain the bridge is kind of out of focus so maybe don't change anything at all so the sample area can get smaller or larger with height that's how we deal with that therefore the in intolerance can taper off with increased height so I have a diagram here so imagine you start rting with a radius like that and then you taper them in the further you go up the more they tend to agree So eventually get to a

situation where they will always agree and that's how you deal with that if you want to have the tolerance be very very tolerant at high distances but that's not actually what he wanted he wanted the opposite of that which is fine because we can taper the other direction we can go from out wider and wider in this situation they're more likely to agree closer to the player's head and less likely to agree further away uh and of course uh therefore you can then have the idea of the rain not the main rain but like the sparkly high frequency stuff mid-frequency stuff maybe cut in and out just as you clip under that that bridge like when you're driving a car on the motorway when it's raining but if it's really really high up don't really notice so here's a video showing that [Music]

[Music] [Music] [Music] so hopefully noticed the fact that when you were just clipping under the bridge it kicks in quicker and then lower down hardly makes any difference so the summary conclusion for this is with the bubble space is that having both minimum and maximum estimates makes it more useful and the

variable Zenith tolerance is very cheap to do because all you're doing is really tapering those raycasts right they're not actually adding any extra calculations and that's very easy to do with a coned radius uh which is much more applicable for a planetary scale type stuff large organic uh uh Landscapes and so on and the weird things you find on Pandora there we go so let's get into slapback 4.0 what is slapback so picture of Elvis Pres slapback was originally this uh delay for a sing Single short repeat Echo effect and it was simulating essentially the immediate Echo from a hard wall near a loudspeaker so when you had be playing in a Speak Easy you might have had this effect and when you went to the recording studio it's gone and you were used to hearing that so they simulated it it was popularized in the 1950s with singers such as Elis and so on uh so the slapback system in snow drop started out doing something vaguely similar so that's the name stuck basically and we're still calling it that uh and in fact uh there's other

variations like Mountain slapback and stuff like that which is sometimes used in movie sound design so we'll have some of that in a moment so in the Inception the version 3.0 as I said was the one that was development came out with division 2 it was enable a few months into the initial launch of the game because we had some problems with uh memory fragmentation with the lay line calization actually and this is what aims to do I stole this from a textbook I didn't make this up so what you're doing is you're Imagining the source position and then you're essentially creating a phantom image of where the mirror image would be and then you can reender the sound at that distance and Direction add the propagation delay flip the phase and then effectively you have a very simple idea of an effect additionally of course if you know the material of that wall you can add some filtering to the delayed version if it's rock or concrete or wood or Earth and then it sounds starts to sound like the material that it's bouncing from uh in division two what we were doing

was we were raycasting in six directions it's data driven but we can choose six and then what it was doing is the raycast hits which are the red ones green ones is where it hits nothing they have a surface normal and then from that surface normal you can calculate where the Phantom image would be so it wasn't uh the ray casts themselves are kind of like feelers working out where they're because you don't really know where where the materials or surfaces are so you have to kind of guess and then you can extrapolate from there um and then what it did was it placed emitters around the player remember the word emitter I used earlier and the placement uh the raycasting was using this Phantom image technique and what we did was in order to avoid having to do piercing rayar that very expensive we had our own physics world so this image here which was taken from the last talk it's very small but just because it's not that important the left side is basically the render view what the player sees and the right hand side is what the audio Ray cast would see which is as you can see a much more simpler geometry uh and cheaper because you're just doing a single hit rast you're not

having to pierce you only see the materials you care about so what you would do is you'd mark up these materials as acoustically opaque and then when they streamed into the world they'll get added into this into this uh physics world and it would also shut down when the player was indoors because we had another system for indoors with our procedural reverb system and um also because the echo are very very very close you get sort of a flamming effect when they're very close it's accurate but it's not cinematic um so what we did was we select audio from the player and root them to a fake device driver and then what we do is we feedb from the device driver back into the mix Matrix so it's like a a um it's a loot back system essentially and at that point you can add on the delay and you can spatialize you can move the direction and you can also compute the U attenuations of distance and all that good stuff and the transit delay was added with a bespoke DSP effect because we wanted to have delay you could change as it's

playing so here's a little demo of that [Music] imagine how great that would sound in atos so um the overview of setback 4.0 was we want to extend the system so a

more versatile system we wanted to be able to tackle Landscapes and natural phenomena and weird stuff that you get in world of Pandora um it works in two functional layers with distinct behavioral differences which I'll explain what they are in a moment and it allows for both early and late reflection effects what we were see seeing doing before was just doing early Reflections first order now we're having kind of that plus some pseudo Lake reflection effects and uh we also implemented the material idas for filtering which the filter always the the delay effect always had filters in it but they weren't doing anything now they are uh so here's a demo I'm going to show the system off firing the gun randomly in the forest system on and then a more realistic thing where you're running in and out of a building firing it stuff at random and you'll hear The Echoes uh between early and late reflection balance change over time that's off [Music]

[Music] [Music] all units be advised perimeter may have been compromised [Music] fun fact what you saw there was that exploding Barrel why was that part of the slapback system the reason was it's being mixed into it does that make any

sense well think about it like this the further o it is the quieted it is so it will be pick up less than the system the closer it is the closer is the player position and therefore more accurate The Echoes so it's kind of a it's kind of a cheat but I think it works great it's why I put that in there so in order to achieve this you need several things first of all you need to have a rast andit manager that um places reflection points on designer set rules you want an audio signal router for capturing source sounds and playing them back through multiple met positions you need some DSP Digital Signal processing in case you forgot what that means at um each Amit destination that handles uh flexible delayed times and uh filter EQ and for the late reflection effects the emitter destination needs to be rooted through an environmentally defined Reverb generator with no pre- delay I'll explain how that works in the moment so let's look at Reverb as you might have noticed I stole that from the AUD kinetic website but it is a good diagram of what Reverb can look like so

the red spike is the initial Source excitation the green ones are the early Reflections the ones you get from the immediate surfaces first second third order and then the blue stuff is after it starts bouncing repeatedly and starts becoming this late reflection diffuse reeverb effect okay so what is early Reflections let's just Define it immediate Echoes Echoes of Echoes and so on distinct phase and they can interfere with each other both constructively and destructively so very chaotic and collectively they have quite irregular attenuation over distance uh so here's the thing when you're calculating uh a room space designing room space like this they talk about things like the direct field and reverberant field and figuring out that critical distance to make sure that people are sitting in the reverberant field because then it's easier to predict what they're going to hear but when you do those calculations with the formula you got from college he might get the wrong distance and the reason is is because you're not take into account the early Reflections being like their

own sources so not just these speakers but if imagine if these walls were hard they would also have be like additional sources and that pushes back the critical distance so therefore I will say that since we're not ever in games in an ideal acoustic space that the early Reflections especially first order ones are really part of the direct field the late Reflections uh nth order Echoes you know echo echo echo echo stuff uh they diffuse largely phas in variant kind of like noise really and they have very predictable loudness attenuation over a distance so quite different behavior and therefore they're part of what we call the reverberant field so everything beyond the critical distance reverberant field so for instance right now probably guy there is in the direct field and um are the GDC staff are in the reverberant field for sure hey um so let's look at it like this if you were to capture that if you made that impulse response yourself with an omni mic what are you getting uh you're actually getting a bunch of

different Reflections coming in at different angles and they're being recorded equally and then they're all summing together so if you were to place that in a volume you go into and have that Reverb it would sound very very good the only problem is it doesn't change as you move around it's very static so how do you how do you avoid that well the way I figured was you break it up so you're componentized so rather than going with the final impulse you're breaking up into smaller chunks and then when when they resum as you move around the world They do change and in fact even if you make a lower quality Reverb than a very expensive impulse respon license from penguin audio somebody uh the fact that it's moving around actually makes it much more satisfying and in many ways probably is superior because it's reactive so I'm going to Define reverberation time t60 okay defined as a time it takes the reverberant sound pressure level from the impulse to Fall by 60 DB uh interestingly a tile bathroom can and have can have the same t60 as a

gothic church they're quite different in sight so what what's happening there clearly the church is a much larger space so why is the t60 the same well distance Reflections how much energy has been dissipated uh so how your brain estimates a scale of reverberant space is from processing the times of the early Reflections you learn this as a baby growing up you start to recognize the early Reflections are super important that's how you know you're in a bathroom what if you're blindfolded compared to being in a gothic church or in a train station Concourse for example therefore the delay times in the direct field are critical to immersion that's my point that's why it's worth mentioning so the late Reflections are essential for colored noise that indicate what the surfaces are made from and that kind of thing so they know the difference between marble and wood and that kind of thing so stone or concrete versus wood or carpet as I said here like they're clearly very different materials so the ray cast and management we have two Ray cast functional layers

working independently the ray cast have a minimum distance radius so rather than going straight from this the the sorry rather than going straight from the the center of the player position they can be a little bit further out uh this is needed for a thing that I call the Ring of local Origins uh I'll have a diagram for this in a moment reflection points um are rendered VI emitters of course and the layers have different emitter cannibalization Behavior so what you're doing is you're you're you're budgeting your emitters for these two layers and how they cannibalize and decide to go to a new Reflection Point changes because for what you want for early Reflections and what you want for late Reflections can be quite different so the early Reflections layer is optimized around response enough to oncoming geometry so what's happening is is because you're moving around it's they're moving quickly quickly quickly and they whatever they were playing a moment ago they keep playing in the new position the late reflection stuff is optimized around longtail Fidelity so what will happen instead is a new point will start forming the other one will start fading out because it still has Reverb in it so you don't want it cut out immediately and if you moved it to

the new position it might might sound weird so that's how we get around that you can actually change the um the the Fade Out times and this is how you tackle the direct field versus the reverberant field stuff okay so I'm going to use a division character again because there was no clip art available for Avatar it's player Centric so it only happens to the actual uh player and not to any of the other characters yet mathematically it still only performs first order reflections so that thing I showed you before Hitting off the walls it still does that so for the aouth raycast if you imagine you rast out radially what it's doing is it's hitting like an imaginary surface and then you can work out where the Phantom image is and if it duplicates then you know what they're duplicates so in this case from three hits we go okay two of them are the same as the other so don't just kill them off and then you end up with one uh and also we can do things like we can have an Asma height offset which we can then correct for when we resolve the uh position that might be handy for not accidentally hitting things that are

just outside of your minimum distance but they're still too close maybe a wall cover you know video games right and then upwards or downwards so depending on the layer for the early Reflections we we raycast upwards because we want to hear sailing stuff and when we're in the air flying we want to hear the late Reflections from the ground so we R cast down for the uh for the for the uh late reflection layer so here's some map but visualize graphically Don't Panic reflection points are derived from the ray cast hit positions and the surface normals so the reflected images can be positioned either by the full 3d Phantom image so you start off with the white circle you Ray cast down you get your ray cast hit it's red and from there you have a position and a normal and what you do is the reflected distance scale is two times the dot product of the direction Vector from P to q that Ray cast Okay then the reflection vector I got move my mouse somewhere where's my

mouse pointer G the reflection Vector is then the surface normal multiplied by the reflected distance scale and then you can then shift across so from the extruded position if you subtract that from the original Source position you get the blue dot that's your uh Phantom image position uh you can do it fully 3D you can lock it in the horizontal plane if you want to do that um it's the same in terms of like looking uh overhead and interestingly enough there was another mode that Alex asked for which is what if we don't bother with that at all and we just double the distance so we call this distance only that's another mode you can do interestingly enough it's less correct but it's more stable usually actually because you think about it if you're running along the side of a a mountain wall those Reflections are going to be constantly shifting around where if you want something stable that's always at the right kind of distance maybe that's better to just do distance only super cheap right uh so here we go this is what the Ring of local Origins is about this is how we got our minimum so we have the player position we have a radius around

it which is the minimum distance we rast from a local origin around that radius when we get our hit we then calculate the rest of the stuff you just saw as if it came from the center called the hits and then we compute the reflection points accordingly and so you get the same calculation but in this example you would have completely missed that little wall on the right there which is what you want for something that is further out so without having to do any kind of additional calculations by literally just moving the origin away you're getting later reflection points and of course if you do the distance only you get that effect so let's focus on the early reflection layer as old reflection points invalidate emitters move immediately to the new position uh reflection DSP uh effects play interrupt uninterrupted so if you happen to be firing a gun at that time whatever gunshots still playing out just move to the new position delay time and filter settings are

updated to the Reflection Point on the Fly this is why it's important to have a delay time that can move as you're rendering and filters that can move stably whilst you're rendering so you don't hear any weird pops or glitches okay let's take it through step by step so imagine we are in a some kind of Gully this a plan view player position is the black dot it'll raycast out to say the right there find those those two hits realize it's the same position so it will can cannibalize one of them and then you end up with that Blue Dot that is our ER position okay player moves forward oh sorry I did the wrong thing player moves forward what it'll do is is it'll make a new position but it realizes it's close enough to the old one to just shift the point so it just keeps playing moves immediately like that uh keep moving again it's going to find a new position which is not strictly correct because as you can see the wall isn't there anymore you could actually sanity check that if you reast that position and realize you're not getting the same information you could kill it completely completely but video

games um and then when you go to here you get another position over there but it's so far away from the last one that it decides it's going to kill off the older one now nor other's moving it it's going to tell it's going to fade that out and start a new one over there okay and that's how we deal with early Reflections late reverberation so here we're going to do something a bit different as old reflection points invalidate they fade out and new reflection points can then begin in new positions there's always a kind of a fade happening uh it allows for long Reverb Tales to play out so if you've already committed to a rather than cutting that off that would sound disturbing right let it play out and then fade so um he uses the Ring of Lo ring ring of local Origins quite extensively because you really want to have that minimum rast radius quite a far bit out to prevent hitting valid materials too soon that's really what it's for uh it's much more efficient than using piercing Ray cast so what you could have done is decide to do a piercing Ray cast and

then walk the number of hits you got back just keep shimmering down that um array until you found one that was at favorable favorable distance but super expensive this it's just one raycast and you're done uh and then you add Reverb to the delay line so the Reverb uh engine is working normally but it's getting a super delayed copy uh and therefore when you hear it back you're getting it spatialized at that distance with the Reverb that's defined by the environment and therefore the the shift you see here the yellow line is varying with the delay time that's what you're shifting so it's the pre- delay on that Reverb is what's affecting and it's happening in all these different positions around your play player's head so let's look at the the same Gully situation but with this wider ring of local Origins let's start here so now the player stood here and you have a bottom left top right bottom right three Ray casts and therefore we have these Phantom image positions we move forwards they invalidate and they start to fade out because they're

invalid and new ones are found so momentarily you actually have more uh more emitters playing than than you've actually budgeted for so that's you have to kind of like double your budget for this and then you move again and then the other ones that were just started fade out again and the new ones start further on and that's it so in terms of like filtering I did some research and uh got some real acoustic data back about what happens in in uh different environments I have a chart here from um somewhere in uh the Pacific Northwest and it's the different plants that someone's found the acoustic absorption Spectrum for and I use this information for the clouded Forest uh Don't Panic baby tears is a plant not actual baby tears let me to clarify that because when I showed this to somebody they panicked they went no it's um so what you do then is is that this is the absorption Spectrum so to make the EQ you just basically flip that it's power based and then once you add

uh logarithms you get a kind of decibel shape so that's all the EQ settings I based upon based on that and I don't think they ever changed I think uh I think Alex was happy with them so cool um so every valid Reflection Point has a material ID and we take the material ID whatever that material is and then we have a list of known EQ settings and we apply it materials are mapped to filter parameters so we use the raw filter parameters uh and then we send those in to wise through realtime parameter controls and that's what the reflection effect is getting for every single emitter and of course they can be different for every emitter right um so this is a this is a little bit of a window screenshot from uh Snowdrop so it's kind of small but basically there's a couple of materials there and basically you can decide to have there's two types of filter that in the delay you can have one of them on on both of them on whatever uh if one of

them is off it doesn't it just disables and it's not part of the it doesn't take part in the processing so you have this idea of uh Center frequency and gain and um some kind of balance and this is because the filters have different um uh profiles one of them is a peaking filter so therefore we can change the cut off frequency and uh the gain uh I I set the Q factor to four just so I don't have the Q factor changing constantly um the other one is a high low shelf kind of filter where we can move the the center frequency like that so you create a list of materials that you want to Target and if the material is not on that list it just ignores it filter turns off so therefore it's just completely flat frequency response for each material it links material ID used in the game it's a game specific now right uh enable each parametric band crossover filter as as required and these values are posted to the slatback emitter as it moves or is reactivated any unlisted materials disable the

filters when they are encountered so and one other thing I want to bring up we're sending the propagation distance but the sound designers choose the delay so here is an example of one of the the curves they used if you think about it 450 M should be about 2 seconds and it's not it's about 1 second that's because it sounds cool that way entertainment the signal rerouting stuff the guts of that is essentially that we have this broadcaster system very similar to the one in in the red engine finally enough uh where you're taking uh each time the the wise engine is is doing its render pass you get a call back there's two buffers and they're switching constantly so you imagine after the second callback the next one will look like the top image you either have one buffer recording and the other one playing but they never they never meet we can actually have up to eight of these so they're broadcast channels and

inside of those they have up to eight audio channels uh so think a bit like a digital Tye Loop if you ever used one of those you have a record head and a playback head and the loop gets moved on from a callback we register within a rendering chain so we know when it's happened it does add latency though so by doing this you're adding an extra buffer of latency so if you're rendering 1,24 samples by re-rooting it this way you're adding 1,24 samples which is 21 1.33 milliseconds at 48 khz sample rate however we're going to delay even more so it's fine um so the recording side we have basically the sender sync plugin in Wise which is a fake audio device wise doesn't know that it's not an audio device it's actually writing to a a common memory and uh it sends the audio always to the Record Head it doesn't know which one every time it writes it gets whichever one the broadcaster tells it to remember because they're switching uh and it controls what the broadcast channel that's using and how many audio channels is allocated so you

might if you're if you're deciding to make a game like exify where it's only stereo that maybe reduce the allocation to two channels uh called once at the end of each render pass as I said so every time it renders call back switch uh from the playback side we have a uh receiver Source plugin and that just takes whatever's from the recorded in the last pass right because that will be now the playback buffer and that's triggered by a play event so the emitter needs to know there's a play action and then it will start reading from that buffer uh it's faded out by the stop event so we always have a stop event to say the faded out as I said before depending on whether it's an early or late reflection that Fade Out might be very short or quite long uh and it controls what the broadcast channel reads from so you can actually have a on to many relationship so if you record once to one channel you can read many many times for that same broadcast channel um it get it's actual audio Channel layout specifically whether it's 7.1 or

5.1 or whatever it is it gets that from the sender sync plug-in so that's defined by the actual fake audio device which it actually gets from the properties set up by um the playback engine and then as called once at the beginning of each render pass okay the filtering part uh each Reflection Point receives a receiver instance okay then we add the delay so why is this playing a source plug-in with a single parameter the broadcast ch pel and then the audio signal from The Source plugin needs delaying and filtering right we need to get that propagation delay in there we need to filter according to materials so how it works is it's kind of like this you have a circular buffer and the record pointer is spinning around constantly imagine going to the right and when it gets to the end goes back to the start again uh the output side is going to be behind it and the difference between those two positions is the actual delay that you hear now if you shift that audio position suddenly it'll hear a click or a pop or it'll not

sound good so what we do is is we actually uh create a nice power complimented cross vade between the old delay and the new delay this is not actually the High Fidelity delay with dopl you would expect what's actually happening is it's just editing uh that way you can get around unnecessary dopl depends how fast you're running I guess but um if you're in a car you would definitely hear that in a racing game right but this is it we're now jumping to the new part of the delay buffer and doing it whilst removing any kind of weird artifacts and then we do phase control which is just basically flipping the phase and then we do the filter EQ and that's the output sample so uh do I need to yeah basically I've just said all this stuff haven't I there we go so uh let's have um some kind of like technical points in case it comes up you need specialized memory management to do this because you're constantly needing these delay lines and they're half a Meg in size right why half a Meg well for byes per sample is 13172 samples or 2.73 seconds at 48 khz

sample rate why does it need to be up to 2 seconds because that's the size of the world you're dealing with those are the kind of delays you might have to deal with uh speed of sound and air is typically 340 m/ second in Pandora they never told us we never have found out what the speed of sound in that atmosphere is but let's say it's like air therefore the maximum Transit distance could be 928 M the uh you need to have multi-threaded access because this is being rendered in multiple threads uh and you want to be able to prevent memory fragmentation so therefore we need specialized memory management right so the first filter looks like this if you're a DSP programmer you should recognize this this is called the B quadratic filter direct form to uh and what happens is you change five multiplies and you can get lots of different filter effects in this case we're just using a percal formula for those coefficients uh we're not doing any kind of musical modulation type stuff so this is actually not a very stable filter for

that kind don't use this if you're trying to do something you know DJing don't but uh as I said the Q factor fixed to four that makes it simpler to do the calculations of the the coefficients actually and then you can then do things like you can attenuate or you can boost you can move the frequency left to right and this is the kind of the overall range that that filter works at okay that's our Bode plot for the other filter this is a little bit more unexpected for some of you I guess the crossover Filter part looks like this so if you look at the inner core of that that is a low pass filter modeled on a capacitor resistor time constant that discharges exponentially and the K value that multiplier is 1 minus E to Theus Omega T that Big T is actually the the sample rate inverted it's a time of one sample the e to the minus Omega T if you calculate it exactly like that it's kind of expensive so you don't do that you do a polinomial fit because that's obviously a very good candidate for Sim optimization then anyway uh the outer

line is the same input that subtracted from this output and then you get like a high pass characteristic kind of and then at the end here we have a voltage divider and if it's exactly in the middle they will perfectly cancel each other out and be completely Phase Linear if you don't believe me there you go so the inner core part as I said that that time constant low pass filter is the blue line and then when you subtract that from the input you get the orange line with a bit of a bump but don't worry about that and then when you sum them together completely Phase Linear so you can have this on and not hear it but of course you can then change the offset between the two and so you can get things like this so you can boost it uh you can boost the other way so therefore you get your high low shelving in a nice efficient way that sounds very natural are you all ready for a white noise test all right here we go here are the filters in that ction just with some white noise you can hear them working

there we go how your is hopefully that's that's tells you how the filters are working right so this is when we when we're taking the echoed sound we're filtering it that's how it's being applied so the combined result I put this all into one picture let's go back to that Gully so what we're actually getting if we're in that position we're getting two early Reflections one of them is quieter than the other CU it's further away and we're getting four late reflections with the Reverb applied it's defined by the environment with different filtering and

because of their angles they're specialized at different angles around the head and different distances mean they have different pre- delays there we go does that picture make sense this is the this is the decomp tized version of that nice impulse response you saw at the start um and then uh I want to show an example where it's realistically in the game the one other thing that's missing from this is uh obstruction inclusion tests because sometimes you'll have um Phantom emitters that are behind a wall and you want them to be filtered that way right so here's an example ing game very chaotic but because of the um propagation system you can therefore control what you hear depending on the geometry around you [Music] [Applause]

I got you now space security reports arm restrict all soldiers be ready there we go so let's summarize then the Reflection Point cannibalization needs to be different you need to have different modes for that to optimize around the requirements of early or late Reflections right that direct field reverberant field uh Ray casting on the Ring of local Origins really useful it just incentivizes doing something where you're maximizing amount of information and minimizing the CPU cost and filtering materials is Big Win obviously because it sounds you got lotss of variations much more natural

sounding okay and you need robust memory management uh I did a more elaborate description of that in the 2020 talk so if you're curious you can go back to that it's on YouTube and one last thing uh some people will think we should hire an audio program and do this so I want to explain to you how big the team is that's working in snow reference Branch right now uh in Sweden we have me Martin Valene who's the lead we have um Martin lren who did the emit manager uh cunes who did the uh amazing uh propagation system that works in the GPU we have hangang we also have Dave Driggers in Stockholm in Germany we have Hannah kler who's working on rapid prototyping stuff and we also have in Canada we have Oliver snade and Andrew Smith neither whom are Canadian but for some reason they live in Toronto so what I'm trying to say with this picture is as well as credit where credit due you don't need an audio program you need a platoon thanks for listening

so I guess some of you have some questions yeah um yeah thanks for that that was a buffet of delicious ideas for audio program um I had a question about the um the Ring of local emitts so is that based on Bubble space at all or is that just a completely separate thing so uh the Ring of local Origins is just like another Circle that I just visualize in my head to be to be specifically the bubble space thing the reason why I call it that is because when I originally drew it out on paper it looked like lots of bubbles on the page right it's not really bubbles it's cylinders but anyway um The Ring of local Origins is just a a imagine a radius you've created in your system where you then create these Origins you already cast out from and then you use the original position for doing the Phantom image technique that's all so it's just a mathematical concept to avoid hitting things too soon okay cool can I ask a quick followup question on that do you you um do you do any SLE over time for the bubble space

calculations as well there's lots of um smoothing going on yeah okay uh again there's a better description of that in the last talk 2020 so I won't repeat it here that's why I didn't put it in this talk because it seem like money for old rope but uh yeah so there's lots of smoothing going on and that's why you can avoid weird glitches like walking past a bunch of pillars in a train station uh because it will pick them up but they won't be uh there won't be enough coherency if they take it effected to the overall calculation which happens actually over 16 frames thank you hey thanks for the great presentation um you mentioned that it was always player or listener based uh all these systems so you could do it listener based okay you could do that um what I found was is that because of the way we make games listener is usually the camera position or something close to it maybe some people you interpolate between player and Camera position but guy doesn't like that we had a whole

chapter about why that's a bad idea right don't do that um we do that in division two so anyway um so uh what it turns out is is that uh if you base it on the player position it's less correct but what you can do is is that you have a threshold for when you start making the calculations happen again because you're not doing every frame if you're standing still you do the calculations once and you've stopped and then you have a threshold and then you can set that in data but let's say it's half a meter you move half a meter and then it does the calculations again if you were to do that with the listening position you'd be probably be doing lots and lots of extra calculations and just be heavy on the CPU so that's the trade-off so it wasn't actually my question sorry um it was did you ever find a situation where um you would have liked to have uh emitter so another emitter than the player position to also have its calculations and are you considering that for the future the we have lots of okay 10 minutes to go we have um tried a

lot of different things and some of it sounds really cool but it's just very expensive and if you remember we originally targeting Xbox One and PS4 so this is why it ended up with the way that it is now however now we're in the Next Generation and we're using the GPU and stuff like that the lots of possibilities so we will see maybe in a couple of years I'll do another talk and I'll remind you of this question and go hey look guess what we did so we'll see I look forward to it yeah I thank you for talk and I have a question about geometries and I was wondering who is simplifying the sorry uh I was wondering who is simplifying the geometry for audio and how hard so how is the geometry simplified um that's actually done by technical artists so what they do is is yeah they they they start off with the render mesh and if it turns out to be too complicated they simplify it so if you get like a fallen down tree trunk they might turn that into a box that's what ends up in that physics world and that's what we see with the raycasts okay sense is it hard for them to do or is it very

super it's just in the data okay I had the same question so great talk have another question all that's okay um Anthony uh just a quick question about the late Reflections are you are they rendered uh like IRS or are they some sort of dynamic thing here's the funny thing when you have a environmentally uh defined Reverb it can be whatever you like so uh in this case we have a lot of nice impulses uh you could use uh an fdn Reverb or something else something algorithm that's the part that doesn't matter that's the modular part of it so that's up to you to decide and maybe it's uh you could have more Reflections uh more and more LR type late reflections with a a cheaper algorithm perhaps if you wanted to do that but that's I purposely avoided talking about that because that's not part of the system that is decided further down the

the it's actually the environment element manager in snow drop that actually defines that right thank you uh with having the the uh Reflections be driven by this uh kind of like ring buffer kind of thing uh was that uh sing plugin where you were collecting the delay line was that did you have just like one of those or did you put them on multiple different buses I'm woring about like so the sync plugin part is one per exit destination if you come to uh the audio kinetic booth at 115 I will show you that slide sweet thanks but yeah so the idea is is that they're completely so you have you have one per send so if you want to have you want to characterize different types of uh uh broadcaster it doesn't have to be about delaying but you can have different categories I think at one point we're experimenting with a a plant that repeated what you did it didn't end up in the game but it was a nice idea I think it just didn't quite work but yes that would have been a separate Channel and so therefore we

have a different device and then a different auxiliary send and therefore you read it off a different channel on that way back again and then do something else with it thanks uh thank for your talk I think in your system all fraction and light reverberation is fully 3D portioning but yeah isn't the light devation should be more omnidirectional sorry what was last part of the question a late liation should be only only directional it's non directional yeah so so it's a good point so the question is is if you're positioning these emitters yeah and they are being spatialized when you're adding Reverb to them how does that affect it yeah well in actual fact the Reverb in of itself is working already to render out to channels and so what you're sending to that has already been spatialized in some way so you've already worked out the distribution based on the Amit position the delay and then what however that Reverb chooses to do do its work that's what ends up going to the channels so they're actually

completely decoupled okay thank you uh did I exhaust all the Curiosity in the room should you go get a coffee all right thanks very much [Music]