Finding Space For Sound: Environmental Acoustics in Tom Clancy's The Division 2

[Music] all right let's get right into it here's the agenda for this talk we're gonna introduce ourselves we're going to talk about the audio direction for division to the snowdrop engine the raycast based runtime audio systems in snowdrop we're gonna cover procedural reverb and ambient systems for interiors we're going to talk about immersive weather sounds and dolby atmos support and we'll finish off with a short conclusion so my name is Simon cough cough I'm an RD director for the division series I've worked on division one two and farcry Assassin's Creed and World in Conflict hi my name is Robert Benton an audio programmer on the snow job team and similarly I worked in Division three with Simon and previously I've worked on other games at least hip 4 and Guitar Hero live as examples so we work at massive entertainment which is a huge soft studio it's been open since

1997 located in the south of Sweden in malmö and became part of the use of family in 2008 last year we grew beyond more than 670 staff including 50 different nationalities currently we're working on a division to an avatar the audio team during peak production were around 25 people on division 2 you collaborating across observe studios in the UK stays in Europe but the core team is around 12 people at massive so this talk is obviously based around the game we shaped last year called the division 2 and I wanted to kind of first show the kind of core pillars of the game which is that it is a modern Clancy shooter RPG it's grounded in reality set in a collapsing world where you are agents of change and it's always online and code friendly in simple terms it's a third-person online co-op shooter in a huge open world where you get sent in as a last resort to try and save the city and fight the enemy factions that seek

to take control amongst the chaos so whilst the first game was set in New York we decided to have the sequel to be set in Washington DC and why is that well we wanted to give our players a fresh play field IDC is obviously a vastly different compared to to New York City it's an iconic location with plenty of recognizable landmarks that helps ground things in reality and timeline where 7 months later after the events of the first game it's also hot summer instead of winter in New York and it's also has vastly different nature and wildlife which also gave us a lot of opportunities with sound so here's a small teaser from the second game [Music] [Music] how could this have happened here of all

the capitals of all the nations on earth none more vigilant [Music] safeguarded prepared was it vanity [Music] bad luck or something more sinister one thing is beyond question this is where we push back this is our defining moment if we fail our nation Falls we are the free world's last line of defense [Music]

so the audio direction for division - obviously we're a Tom Clancy brand so we wanted to keep things as grounded as possible and we can't really go too much into sci-fi land the world in our game plays a huge part so making it dynamic responsive and immersive was key we really wanted to have polished mix that adapts to player actions and location so the player can really tell what's going on just with help of audio the goals for the team were to improve on they establish that identity that we did in the first game but with better assets improved workflow and tools and again the worlds that we're building are huge so we need to come up with a way a smart way to implement and utilize the world that's already built in our advantage yes we also wanted to increase variety and quality of our assets and the snowdrop engine so the snowdrop engine

is developed in malmo by massive it's a cross-platform engine it has a graphical editor and most of the behavior is node based it's flexible and has a modular architecture which allows you to make different types of games with it and we also have an abstraction that connects us to the wiser audio middleware which allows us to do all our sound design and there's an example of the different types of games that you can make with snowdrop here are some things they're already shipped obviously the first few divisions are very similar but then you have Mario rabbits versus Kingdom battle and fractured but whole and StarLink battle for Atlas these are many other soft super soft titles that are in production they're using snowdrop so the question is why you snowdrop one of the main things that we try to maintain is a quick iteration speed we want to empower the end user which is why we have the

node base editing better real time processing based on the physical properties of the game world leverage of offline background processes so we have nightly builds that do a lot of data processing for us and better reuse of in our case custom DSP algorithms so we're not constantly authoring you plug-ins for every game and so before we go into the meat of this talk I wanted to go back and mention the two problems that we wanted to solve going into the division to it is how do we increase the quality and variety of the audio in our game and it is how do we decrease manual labor we found that Ray casts for audio was a good solution for us so before we go into the different systems based around that let's learn about the basics for recasts so ray casts what are they used for this diagram just demonstrates a planned view of a very simple l-shaped room and the black dot is where the player is supposed to be and I'm demonstrating a

tray cast pointing out and where they hit tells us where the walls are so the raycast can be used to determine space around the player acoustics is about spaces of air between materials estimating their spaces helps us make sonic decisions that match the players view so in a case where you have different types of materials I've just adapted this diagram to show you that we have metal we have concrete with brick and we have some chain link fence and here we had the same ray cast but ideally we want to ignore the chain link fence because that won't actually have an effect on the way the sound is propagated so the important thing is we have to contextualize the raycast concrete walls block sound chain link fences do not block sound how do you differentiate materials so we have this concept of single hit ray casts piercing hit ray casts and material IDs so in an example where we are doing piercing ray cast what happens is is that we've go

from the point of origin so again the way the black dot is where the player is standing looking at them from above and then we're a cast out towards the brick wall but first we hit a chain-link fence we look at the material ID and we go we don't care so we then have to re raycast we hit another trailing thinks we don't care and so on until we eventually hit something we do care so the thing about heavy use of Ray casting is it's very CPU expensive so vital that's done efficiently as you can see from this diagram we've done a bunch of raycast when in actual fact ideally we would have want to do just one have to be very conservative about how much we use them because we're working towards for CPU budgets piercing raycast a material Turing as a huge performance hit this is a problem so I just want to show you this is a screenshot I just took from inside Washington DC the left hand side is the normal rendered view with some gizmos from the editor and the right hand side is actually showing the triangles are actually drawn in the renderer marked up according to material ID so those different colors mark up

whether it's concrete glass steel that kind of thing the triangles are marked up with a very specific bit mask so it's material based bit masks used by red cast to ignore certain mid triangles so as the red cast goes through the world we can basically check the triangle go do I care about this triangle or not what we did instead was was that we ended up essentially marking up the world according to what we collide with being whether it's actually acoustically a peak or not so here we have the same diagram again only now I'm now demonstrating what actually blocks sound or what doesn't snow drops audio record only see acoustically Ipek materials this helps reduce complexity most of the time single hit ray casting replaces the piercing ray cast that you saw before and there's no need to use material IDs there's no material ID filtering anymore so by reducing that complexity we can enables it to do more ray cast than we could do before it's available as a filter and their litter for debugging

which is how I made the screen shot so one of the first systems were designed around ray casts was bubble space and the idea is to basically determine the average width around the player it's a real-time geometry based array cast system that determines not only width but also if there's anything obstructing above the player it's used to blend it's basically to ensure that the audio experience of the player is varied and the Bay is varied on based on the surroundings of the player and it's used to blend between different type of ambience pads reverbs random effects and even switching weapon tails so we have a video to show how the system works keep in mind that this was really early in development

[Music]

so let's get into the core module so we have these high-level requirements we need to get the sound designers a metric for the free air around the players horizon a metric for the free air above the player bearing in mind in Division two the player can't leave the ground unless they are go through an animation the metrics must be consistent this is very important because the sound designers need to hook into those metrics how they mix sound and they need to be reliable metrics not must not be unstable or appear glitchy as the player moves around the world we don't want any weird edge cases the only active metric sorry be the only active when the metric is invalidated by the players view position so we measure and then we stop until the player moves again we don't remesh that way we can save as a bit of CPU time so here is a another plan view and I just want to explain four phases so one two three four the black dot is

the player's position the green dots are a ray cast that's ended and hit nothing and a red dot is where a red cars had hit something and the black outline is the wall so what we're having here is is that at each or these phases the Ray casts are going after three evenly spaced directions and then they rotate a little bit so the over the course of four phases they've covered a full 360 so we've actually got not just three points but we actually have twelve the raycast horizontally around the player it occurs once per frame three ray casts outwards evenly spaced around the player position the distance of the two shortest ray casts are averaged so if we have hits that is new average is then added to a four element short history buffer so the idea is we're recording the last four measurements we took and then we're constantly working out what the averages now then the result of that is is that way this diagram is showing how the player can move around the world so if one two three four this is the player

moving around and you can see the resulting horizontal space growing and contracting depending on the space around it so I'll give all the updates from an average of three smallest values in the short history buffer the raycast directions rotate 112 returned as I demonstrated in the diagram beforehand right raycasting begins again and then the final metric smooths out over 16 frames approximately half a second until it reaches the new target value the moving average updates the real-time parameter control in wise if the Delta is significant if it's a very small value less than 5% we don't bother changing it and we also spare some CPU on that final metric keeps updating while the player is moving or until 16 frames have passed the reason why we do this is if a player stops moving we still have a history buffer to work through so essentially once the player stops are moving we keep processing for 16 frames and then it stops for the

zenith plane for above us we do something quite different so here we're a cast vertically above the player as you can see there are three ray casts occurs once before aim three ray cast upwards evenly spaced around the player position all three height measurements have to approximately agree with their average if you see in this diagram I've placed a cube above the player which means that one of the raycast hit it and the other two did not in this instance the red dot represents the ray cast that we don't agree with because it doesn't match the average so therefore we would only be looking at the two green ones to actually use that as our height metric when they do agree the target value is updated the final metric smooths out over four frames until it reaches the new target value yes and here are some examples of how we been using the value that we get from the system at the bottom there you see our switch container which contains three random containers with our weapons for a weapon and we have large street narrow street and open field and as you

can see that switch container is actually controlled by game parameter which is called GP bubble width so when the value is zero two zero two point four five I think roughly there we use the narrow street tails if the value is bigger than zero to five then we use the large street tails and if the value is larger than zero point eight we switch our weapon tails to the open field tails and this gives a nice effect where the weapons sound like they adapt to the environment in which they're being shot another example of us of us using the value is modifying the random effects in the ambience to be louder and triggered more often when in a more narrow space [Music] finally also here's an example of us blending between a different exterior reverbs so depending again on the width value we fade in and out different convolution reverbs everything from

alleys to streets to cross roads etc so the next system we're going to talk about is slap back and this is a player centric outdoors system and the goal here was to really add more variety to our weapon tails it creates emitters on buildings on building walls around the player and it replicates the player short pattern with a few distance based effects and folds around so this gives the player more spatial awareness it gives the guns kind of a nice feel again helping with spatial awareness

okay so what's actually going on the following diagram is a plan view again so the black dot is a pay position and we're ray casting in six directions horizontally around the player the red dots represent when we hit something and those blue lines are the surface normal we get back from that that ray cast hit green dots when we hit nothing and then blue dot is where we think there is a reflection point so the rule that we use is similar to optics where we essentially use the surface normal to infer where the reflection point should be so what we do is we use a 3d emitter pool we compute once per frame if the players movement exceeds a certain

threshold that's important so we don't keep processing ray casts around the azimuth and upwards of the player position we use simple 3d geometry to infer new candidate reflection points from Ray cars positions and normals so what we're actually doing is is we're creating candidates we first have already asked out those six candidate positions and we may get some reflection points some blue dots we then compare the raycast position again so we're a cast out from the black dot to the blue dot and see if we actually get the same result if we do then that kind of date is good if we don't that means it's just an anomalous so we basically remove it from the candidate list any duplicates also because you might end up with blue dots overlapping each other they also get removed and this is what we call a basically a series of sanity checks there are existing reflection points that we had from the last time we processed if they aren't updated they get cold and the idea is is that what they're they're streaming something so they fade out with a 200 millisecond

fade-out and then any kind of light reflections left over become a new point and they start playing again so they start streaming it completes a series of sanity checks as I said to remove erroneous reflection points duplicates or near duplicates based on threshold so sometimes they don't line up exactly but they might be close enough to cause a climbing effects and remove them anyway so what is actually causing the streaming to occur so we have here the idea of a double buffer so the broadcaster module is this place where sound is recorded and played back it's supports multiple broadcast channels up to eight each broadcast journal consists of up to eight audio channels so when I talk about a broadcast channel I just mean a brought like a radio channel essentially or some kind of not sound so what's happening is is that we've registered so every time Y is finished is render it calls back to the system and it says please swap these two pointers we have one pointing at the record buffer and one of the playback

buffer and they're constantly swapping frame by frame that's what I mean by double buffer system some of you'll be familiar with this some of you will not be in a sense it's like a digital tape loop so we have a record head and a playback head so whilst one pointer is recording in another one is playing out the loop gets moved on from a callback register in the rendering trainer as I already said this actually adds a little bit latency so if our buffer size is a thousand 24 samples that means the latency we add by creating this um this recording loop is a thousand 24 samples but we're not so bothered by that because we can add a bigger delay in a moment anyway so we just take that on the chin so how do we take sound into the record buffer we have a sync plug-in inside wise this is a fake audio device so it looks like an audio device the natural fact it's communicating with the broadcast module what it's doing is is essentially taking when Y is is essentially completing its render passes

and all the master mixer graph endpoints it then sends sound that has gone to that particular bus to the record buffer based on the record channel it's been setting the plug-in of which there are eight sends audio to the broadcast record head as I said and then controls what broadcast channel it uses and how many audio channels it needs it's called once at the end of each render pass so everytime we render out a thousand 24 samples this gets called at the end now the receiver plug-in is reading out from the from the broadcast system it gets audio from the broadcasters playback head it's triggered by the core module play event so essentially when you this actor mixer Graff every time there one of those nodes that has this plug-in on it is called to basically play out it starts rendering a bus it's faded out by the core module stop event as I said that was that 200 millisecond fade-out

that's what causes it so the control is what broadcast channel it reads from so the idea is is that you can record out to one channel but you can have multiple things reading from that so we could actually have multiple receiver plugins reading out from the same playback head it gets his audio channel layout from the broadcaster by way of the sync plug-in so whatever the the channel layout was if we said it was 7.1 that's what we receive its called once at the beginning of each render pass and as you see from the diagram I just basically did why I mean is is that when the academics our graph is calls it's no surrender that's when this is called the actual effect is then applied to the receiving end so after we get sound back from the from the broadcaster we then apply a delay effect so we had to make a custom wise effect plug-in the reason why we did this is because we need the flit solve the problem of flipper flexible delay time so just to explain that we have a circular buffer so what happens is as each sample time

progresses we have two pointers tracing each other and the gap between them if the delay we actually get in this case is about two seconds of delay because that's not what we need to actually have a realistic delay effect in the world the input sample buffer is constantly being updated and then as the output samples are pulled out we actually do a power complementary interpolation basically because we are moving so quickly we have to be able to essentially move out any artifacts and this is what deals with that problem we also have phase control because reflections usually come back the opposite phase they went in this is actually an oversimplification but it's a good optimization and we also have some filter EQ in the plugin so that we could actually modify the sound that gets reflected back and then that becomes the outputs our port which goes to the output buffer and that's what gets sent on from the source plug-in into the actor mixer node as I said it has built-in EQ and phase inversion there's one instance per slapback emitter so as we play through a slapback emitter this effect is at the

end of its DSP chain before it gets sent back into the into the mix engine of wise okay so just to demonstrate here the top left one is the that's the Sigma plug-in so when I said before you set the draw cross channel and the number of audio channels we actually need to describe and that basically gives us what gets recorded the bottom left is the source plug-in so that's what's coming back from the broadcaster and then all we do there is select what channel we want to read in and then the actual emitters of course a 3d so therefore we position them accordingly and then what we do is is then we have the delay time is predicated by those raycast we were making the distance from the player to the raycast reflection point is what we compute for the delay time and as you can see here the actual the conversion scale we use isn't actually a realistic one we wanted to

have a kind of like a movie type dramatic effect so that's why it's curved like that yeah and here's an example of us using this lap back bus here on a weapon a shot sound at first we tried it on the whole weapon sound and quickly realized that we maybe don't want to have the weapon mechanics being reflected off of walls so that's why it's applied to the shot only so the next thing we're gonna talk about today is procedural ambience and before we go further into this I just wanted to do a quick shout out to Ubisoft reflections audio team especially Septimus and Andy Mucha who were integral for basically designing and coding the system but basically the idea here was to kind of add more variety and remove the manual labor from the sound designers for covering all of our interior spaces in Division two we had over 2.5 thousand

rooms what we call interior rooms the problem the problem that we wanted to solve is that we didn't really have enough time for applying manual sound design and reverb design for each of those spaces so we thought about how can we use the already existing geometry and data and basically design the system a procedural offline system to generate impulse responses for each room and those impulses would be based on each room size and the materials of those of those rooms and also coupled with that we designed the custom wise convolution we were plugin to load those impulse responses at runtime and we even took it one step further and use the IRS that were generated for each room to create an ambience sound for each of those rooms as well combined we did this using come the IRS combined with brown noise and the web wet signal of the props of

that room but I'll go into more detail about that shortly so the convolution implementation is active when the player is indoors we know the player is indoors because of terrain markup which is also used by the graphics people its instance partition completion engine so we have multiple engines and then we can appropriate them as needed the actual processing occurs and snow drops when we talk about a custom convolution plugin the actual plugin part is best feeding sound back from snowdrop the total number of convolution engines is scalable as it happens that we used for this time but that's that's a couple according to how much CPU bandwidth we had to play with the customized effect calls into snowdrop via a dedicated interface so that's how we've done it we have basically a callback interface to pull out the audio that we processed inside snowdrop okay and this is a debug view of the kind of thing when you start ray tracing inside a large space like that all of those white markers are

reflection points generated by the ray cast what we do is is that we generate an impulse response that is static for the room space so this would be one single impulse response but because we purposely filter out the other reflections first all you hear is of the relate reflections and in actual fact they don't really change that much so it doesn't really matter that the player around that space with our three pretty changing let's hear it he's trying to get behind you cover the target get back [ __ ] off cover won't save you important thing to mention is that we didn't use this system everywhere of course where we wanted to use ambience

to kind of help the narrative of the game or if the space was a very particular space we of course design manual ambiences and reverbs for those spaces and simply did an override for whatever was generated there by the procedural system but we were really happy with the way the reverbs were sounding and so we wanted to basically find out if there was any way of using those reverbs to generate an ambience pad for those rooms as well and since our game basically takes place in a scenario where most spaces are abandoned we wanted to avoid making create the sound of an empty office tasks for our sound designers so we're thinking about what actually defines the room tone and we're experimenting in wise with noise and reverbs and here's a little example of how we reach the conclusion that reverb is actually a very integral part of a room tone here's just using noise

in wise a short example of this [Music] you [Music]

[Music] so whilst researching the this topic of acoustics and rooms and room tones another interesting concept we found is critical distance and this is basically where the level of the direct sound equals the level of the reflected sound so in the graph here you can see the black sound pressure line flattening out as its passes the critical distance point this knowledge gave us more confidence to use generated IR s from the rooms to create the actual ambient sounds so in the North line process there were two virtual microphones placed in each room at 22 centimeters

apart which is roughly the average head size and they recorded the wet sound of the props blended it with brown noise and produced perfect 15-second loops for each of those spaces here house how is the results actually sounded in the game and obviously they are not meant to be played this loud [Music] [Music] [Music] [Music]

[Music] [Music] so now we're gonna talk a bit about immersive weather sounds we knew that weather was gonna play a huge part in our game and so therefore I wanted us to see if we could create a system that gives the weather sounds more depth so we created an emitter manager to control all of the emitters on all of the props in our game and this allowed us to basically play back the sounds of rain or wind playing on the actual props it's of course material based so depending on what material the prop is that would be

taking into account when playing back the sound and it's also a cool thing because it's props that otherwise don't make any sound like tables or road signs but we can actually play sound on the NAM on them now if it rains or if there's a heavy wind nice to see you agent there

you so it was actually when I first heard what we did in the prototype for the first weather system experiment we did when I heard sounds of rain above me that I was thinking that Atmos would be Dolby Atmos would be a really nice fit for our game and as we kept developing the game the systems that we designed whether that's bubble bubble space slap a slap back or as I mentioned the weather sounds or even like the in game things like helicopters and drones it just made sense to kind of try out this technology and the first time I tried it I felt not only that was enhancing the mix in the vertical plane but also in the horizontal one where I could feel

better perception of enemies behind me and things in front of me just felt like we gave the mix that much more clarity so it was a pretty obvious choice and Dolby were also very helpful with integrating this into our game and of course not everyone has Dolby Atmos Theater at home but most of us have headphones which Dolby supports with Atmos for headphones so yeah method was a really good fit and this brings us to the quick summary of the talk today so we talked about us wanting to create a dynamic audio experience for our players and we utilized already existing geometry and ray casts to analyze the surroundings we then use that output of the raycast to modulate and change the soundscape the procedural system helped us with greater variety more quality and less manual labor and Atmos was a great match for the systems that we designed and it opened up for more space in the mix thank you very much for listening

thank you oh by the way if you want to find out more about the sound of the division 2 you can search for for it on YouTube