The Gangs Bite Back: Music and Sound of 'Crackdown 3'

[Music] this was a difficult game to make but we are happy to share the stuff that we think is cool we're pretty proud of the audio we worked on this for a long time and we put a lot of thought and hard work into it so this isn't going to focus on the hard parts and the challenges of development we just want to share a handful of things we think are cool that would be useful to other people making open world games or want maybe wanted to take a risk on some of the things in their game the audio for this game was done by a coalition of developers people from the publisher which is me and freelance people working out of their houses or studios my name is christopher melroth i'm the head of audio for publishing and i was the audio director for this game i'm joined by finishing move the brian's who did the music for the game and we're going to walk you through a whole bunch of details so thank you um

next up uh i'm going to hand it over to brian's and they're going to do the first half of the talk all about the music so without further ado the brian's from finishing move hey there i'm brian troyfon this is i'm brian wait and so the we're gonna talk about the music of crackdown three um so the topics we're going to cover is the composing strategies for a large open world staying organized across tons of assets and then the benefits of composers implementing their own music we implemented our score for it um so um essentially the task at hand with crackdown 3 was scoring the island of new providence that's where it takes place the new providence is controlled by the evil terranova corporation they've got their hierarchy of corporate gangsters that are essentially the bosses that

control various zones and assets throughout the island and so we had to figure out how do we approach music for this um so these are the challenges we face the first is it's an open world you can kind of do literally things in any order that you want you can enter in a boss battle and then run away so we had to figure out what is the music based around um how do we make it support the game play what are the you know what are what are we trying to tell with it um the next challenge is interactivity and it's to figure out how much interactivity do we want where do we want it how much can we get like given the limitations we have um and then third we have to consider the musical style which is super important and so it's what is this crackdown world the island of new providence sound like and with that consideration do we need some licensed music to support that or are we going to do all original score so i'm going to dig deeper into the musical

style that we did for it so i think the main thing that we're trying to achieve with the music was to just get that feeling of like you know you're terry crews you're you're jumping around on the top of buildings you're blasting people like the visceral the fun aspect of that the futuristic sort of nature of the world so stylistically we came up with what we called cinematic cyber trap that was sort of um the combination of influences and so what that ends up meaning is it's basically we took a lot from like modern hip-hop trap rap music so like big 808s lots of like attitude and swagger and vibe um some of the cinematic and large than life nature that you get from film trailer music and then like we were inspired by a lot

of the uh like just highly stylized musical sound design electronic music and bass music like people like noisia amon tobin um two fingers that that sort of stuff so um with the that music and that style um the way we approached it and brian white's going to talk a lot more about this um we themed a lot of it around the bosses and um and their different zones and things like that and we'll get into the details of that but we really wanted to capture the unique and distinct personalities and quirks of these these different bosses and so the way we approach that with this cinematic cyber trap style was essentially for each boss we would write a linear piece of music like a three or four minute piece that has all of their themes and like just totally captures their vibe and um then we would digest those into the interactive assets later so it's like creating a sketch that just totally

covers the vibe has the the feeling the melodies associated with those characters um that we can use throughout and then then you know break that up into the interactive assets and create more stuff based on those sketches um with the music we decided to do an all-original score so there's no licensed music it's a lot of music i think it's close to four hours that's that's in there um and a big part of why we did that was we wanted to achieve coherency throughout the the musical world uh in the game so we wanted the music to sound like what you might hear like on the radio or in the club in the future so to be futuristic but also to have the cinematic and narrative elements that you can only get with the score of like light motifs and character motifs and you know um being able to have the music react or foreshadow things like you get in uh you know a film or a very cinematic sort of game um

and uh we also wanted to avoid some of the risks that can be involved with licensing music like one of the issues that's happened in the past is you know if a game came out 10 years ago and they didn't secure the rights in perpetuity for a song because they never thought it would be released on xbox one or come out on game pass or something they'd have to go back and that can be a lot of headache could be expensive or sometimes it can be just actually impossible so we wanted to mitigate that with just all commissioned original score that's we did so we're gonna show you a little montage example of this cinematic cyber trap sound uh this is with no uh sound effects or dialogue it's just cut to the boss battles so you get the flavor of like the visual style and the musical style and how that goes together [Music]

[Music] [Music]

[Music] so [Music] so [Music] [Music] so [Music]

[Music] [Music]

[Applause] so how did we get all those sick beats to work in an open world game where like brian was saying you can basically do anything you want in any order so you can basically start the game and go after the final boss and you could be in that fight and you could run away there's so you load once into this huge open world map and and and that's it you can kind of go where you want do what you want or sort of not follow any prescribed thing at all and just throw cars around at people so how do we sort of take and use music to support uh the narrative of the game support the story sort of of direct players uh towards things and and

what brian had mentioned is we sort of decided that we'd theme things around these bosses these corporate gang bosses and so i say it's all about that boss base and so each boss controls certain assets in the world in the island of terra nova or new providence and the idea is these assets can represent certain missions that the player can take on and if you complete enough of these missions let's say you destroy enough turrets or or mine machinery you can actually draw that boss out into a final boss fight to defeat that boss and sort of work your way through the different bosses in the game and like i said you can do that in any order you can actually take on the captains before the lieutenants and it's it's pretty crazy you'll probably lose but uh you totally have agency towards that and the reason we wanted to do that so music is not persistent because a lot of

people play crackdown uh where they they don't actually want to go after the missions all the time they just want to go run crazy in the world collect orbs level up kind of just do their thing kind of do stunts in the world so you're not hearing music all the time and so we wanted to make music special in that it it either informs you that hey this is this is something important this is a mission you can take this on and sort of move through the narrative of the game we also wanted to use music to reward the player or essentially say who won or who lost did you die or did you complete the mission uh did you level up and you hear sort of a little musical reward and so we have sort of this boss themed music as well as uh agency themed music for when you level up and accomplish things so i've got a little example here of how a mission how you would encounter a mission from sort of no audio no music

in the world to encountering a mission and then hearing how that progresses and completes enforcer operation you bust up loosens their grip on the island dismantle enough low-level troops you'll force their top dog lieutenant to come out and play and that's when you drop her then the whole enforcer brigade goes bye-bye [Music] nicely done

things are getting dusty in over chemical hq or a man who calls himself a genius did i miss that last slide oh there we go that's just a picture of some of the captains and so you can see kind of how music comes in to inform you that you're in a mission area then combat music kicks in and then you get a sort of reward themed to that boss and you hear some of their themes kick in when you complete that mission and the breakdown of this like i said there's nine bosses that you can encounter in the game and there's quite a few assets uh for each boss so we started like brian said with sort of a sketch that was two to four minutes that sort of defined their swagger their flavor their their themes that helped us get buy-in from chris you

know make him happy stoked on what that sounds like and then we'd go and make almost like 20 25 minutes of music sort of abstracted from that as a palette for uh those bosses so the ambient loops uh there's combat loops there's three sets of combat loops there's low health death down but not out loops and stingers there's also a whole bunch of just notification one shots so it's not looping music it'll be things like when you enter a bosses area for the first time you'll get kind of like a little thematic 10 second stinger when you see one of their assets show up in the world you'll get a little eyes on singer that's sort of themed towards that boss so there's all these little sort of light motif moments uh where you're hearing the uh boss's theme kind of trickle in and those all sort of culminate in this final boss battle so when you draw the boss out you complete enough missions to draw the boss out

that's when you get these sort of scripted boss battle moments where it's it's it's loops sort of train card and we're sandwiching those between actual uh scored two linear uh runtime sequences and i have a little example of that and and i'll annotate on the screen when it's when it's actually scored to picture to the runtime sequence versus it's in the interactive music system and running the state system oh this place is seriously messed up well when it has come to this i can take the pressure questions can you kate is trying to redirect the pressure to release valves across the plant we need to hit those valves and purge the whole system the pressure we sent down the line

should have opened up the valves but they're locked tight all units defend those control panels with your lives do not let the agency activate them that's it hack that panel to free up the pistons and blow that valve we need to hack the control terminal and expose the next valve agent that's one down free rotting pressure to ancillary outlets protect those fires you fool looks like another valve is vulnerable hack the family take it out i'll make this single kill the agent or everybody dies

you have no idea what you are doing we know exactly what we're doing we're flashing you you piece of [ __ ] let off some stinking duck the queda and the chemical plant blown to smithereens terra nova's industrial division is on the ropes the pipeline sending chimera to vargas have been cut off at the source be a lot easier to reach her agent so the boss battle is really that sort of the when you get the full enchilada you get the full beat and it's this really cool visual thing you actually hear it all kick in and you've been hearing these themes kind of throughout and and that's kind of your reward as you're fighting this boss this like super lit trap music there's some additional interactive systems i'll just go through uh really quickly there's some things called reinforcements uh based on a hate meter

so if you're just causing trouble in the world the captains can come after you and there's a musical system that sort of says like oh okay these guys are coming at you um the kingpin who is the lead boss she can actually lock down the whole city so that's like sort of like a big uh retaliation reinforcement from her and those you either win them or you die to get rid of those and then we've got some side missions in the game propaganda towers and uh these different races and those each have their own little interactive systems to support those so i'm going to briefly talk about some of the diegetic world music so we created a whole bunch of stuff for there's lots of businesses in the games and like you know nightclubs and things like that and so we wanted a way that to just one just fill out the world and also to add a little bit of humor and just more

color to the environment so um there's a bunch of these businesses that have holographic advertisements and so we made non-looping jingles for them uh and then for like night clubs and there's like boom boxes that you can pick up throughout the world we made you know cool looping pieces of music we did over 50 uh unique like little jingles and and uh and and pieces uh for the diegetic music and by the way in case diegetic that means like it's coming out of something in the world so it's you know coming out of a speaker um or a holographic advertisement so uh it's funny because our background was actually started in commercials so we were able to flex our muscle doing that so i'll show you an example this uh is for the cash cow loans business and uh in the actual game it would be futzed but for just the sake of this example this is just gonna be just clear um so that is [Music]

so just you know low rent television commercial just to add like some more humor and color to the world so next brian white's going to talk about how we stayed organized across all of the tons of assets yeah so this game was like almost five years in development and we did four hours of music so just like lots of assets and then lots of time uh putting that together design changes and and that kind of stuff so we really had to kind of uh stay organized and we use some some typical things like spreadsheets but really the most interesting thing i want to show you guys is something that that chris actually developed and it's these vizio flow charts uh that that really sort of take all the interactivity and they put it into this

really nice visual so that everybody from us who are doing the implementation and writing the music uh to the coders to even the designers the level designers they can sort of understand uh and either comment or or or know how the music system is supposed to work and this actually uh these things got sent in in emails numerous times like this is how this is supposed to work but it actually they're super effective it would have been very difficult uh to just dive in and create a music system from scratch and wise with without looking at these and uh yeah so yeah oh i just want to make sure that was on uh wise doesn't have a good way to visualize your interactive system it's a bunch of menus and lists and so for our brains to be wrapped around how this was all going to come together uh vizio is a great tool there's other tools like that but i get vizio for free so i work at microsoft so uh i use that

one and i wish that the interactive music systems would actually use a visual scripting system like this so that we could see how the music's going through as opposed to these sort of dumb lists that are impossible to debug once they're built but we debugged the music system before we built it this way because i had to be completely honest you know about how many cues i was requesting so this is the entire interactive music system yeah in the fully zoomed out view and then so what we did is we we mirrored this with an actual linear uh spreadsheet sorry can you go back anywhere it says campaign game place uh no no score is playing so instead of having like one big block where everything comes off of that it's you're just basically understanding campaign gameplay is uh just a simulation of the world and then you can get in and out of any state by following the flow graph and this really helped with the transition uh matrix because there's

like you can literally go anywhere to anywhere so our transition list and wise is it's a bit insane uh it it takes a long time to load the interactive music system this is just a linear list so that we could sort of have a list of everything and then make notes on that and sort of check that in and out of of per force so that we could make either design notes or the coders could say how they wired it up or we could just keep track of how many minutes we were writing for the different queues and transitions and then this is just a quick example of just some wise organization we're uh very very organized uh with such a big game and we had very specific naming conventions on everything that sort of lasted uh throughout the development there's even typos that chris made that we just we kept those naming conventions so that everything stayed through

true all the way to ship and i really like to use a lot of folders and all the labeling capabilities that wise has it's really useful in the interactive music system especially for transitions like transitions to and from folders instead of just like having to make one for every individual asset and naming your transitions because we had we probably had at least a few hundred maybe like 500 plus transitions in there that needed to be accounted for and then finally i'll just talk really briefly about uh composers as implementers so we built the whole music system in wise so it wasn't a system that already existed that we just threw assets into i actually took those design documents and i built the entire system myself so i knew exactly how it was supposed to work and how everything wired up how the states worked how the stingers work how the transitions worked uh and and really the benefit of this is

just being able to provide instant feedback that that instructs the composition process so we could iterate ourselves just by testing either in soundcaster or testing in a build to know like yep that's not going to work or that transition needs to be twice as long it needs to be half as long you know just those basics and and i think that's you hear that a lot like oh yeah you know composers should should implement because then they can make better music and i think the the biggest uh one that we ended up figuring out late in in the the development cycle was just being able to own the polish and find and fix bugs a lot faster so uh in the last two months of of the game before ship i was playing every day probably three four hours a day uh just playing the build and then back in wise and i was catching bugs that testing was not catching nobody else in on on the sound team was catching just because i knew exactly how all the specific little systems worked and then i was able to go in and fix

those bugs or in the event where there might not be code support just come up with creative solutions for making it work you know like oh i can i can tweak this i can make this and and we can make it work with what we have and i think that uh is is pretty difficult if you're also relying on a sound designer who's implementing the music and they've got a million things uh it was amazing for me as the audio director because at the end i'm just chasing a million bugs and especially perf on this game was a real problem and uh brian white like just hey i fixed this today okay sounds awesome buddy it was far too big a game to play through uh on a daily basis so yeah so and then just a couple of the challenges with that we were working on this coast and there was also uh teams in the uk so there's just your typical you know sink time differences like yeah i'd lose per force access uh

in the middle of the day and they'd all be sleeping so just those kind of things and keeping up with with those changes but uh in the end it was actually super valuable um and it and it wasn't that hard to sort of embed in the development process and really add a lot of value uh to the team you want to do this last one so we're going to hand it over to chris just the key takeaways just in terms of so composing for an open world you have to have a unique approach and think about like what is the music based around um with obviously the organization as brian showed it's like having different ways of communicating that information beyond just a single spreadsheet really helps keep everybody across teams all on the same page and then you know the composer's implementing their music then you can react to things very fast and just get it going so now we're going to hand things over to chris for his portion of the talk all right time check i think you guys

are right on schedule yeah yeah you got 30 minutes hang on all right i got 80 slides to do in 30 minutes i'm gonna not micro machines it but i'm going to gloss over some of the details that are up here and you can always get the slide deck later and dig into the numbers if you would like to see more details all right i'm going to focus on some sound design systems and some mixed stuff that we did um in the front when we get to the destruction section james nixon nixon and dan krislip right here on the front row actually made all these slides and they did all the hard work on this and all i had to do was copy and paste them in here and make it sound dumber for myself to present thank you guys and they did an amazing job on this game um all right so uh we wanted kind of cool acoustics and we needed to do it super cheap and super fast so i'm going to show you how we did that because i think it sounds alright um the if you don't know what crackdown is there's two modes there's campaign and multiplayer the multiplayer hosts uh 10

players in a fully destructible world you can take down entire buildings piece by piece like literally piece by piece so in order to play the physics sounds for all that it was an incredible technical challenge so i'm going to share how we do that we have a individual system that is also part of destruction but turbulence so it's something that i personally have always wanted to get into a game because i hate it when a physics object's flying through the air and it's just silent i want that movie whoosh as it goes by so i'm going to show you how we did that just a real quick short thing on lfe because um as being in the publisher side we work on a bunch of different games and i see a lot of inconsistency about how people treat lfe and i'm a huge fan of lfe um i'm going to share at least like how we how we thought about lfe on this game and then i'll show you some i'm going to try to get to the mix stuff as fast as possible because there's a lot of detail there and mixes on everyone's mind all right uh so the acoustic system um i did some sessions with a blind gamer

named sightless combat and he is incredible he's an incredible resource and like this game shouldn't be playable by someone blind but i i took it on as a challenge of anything that would be good for ben to be able to play this game would be great for everyone me if if he doesn't get a piece of audio information to understand where he is where the bad guy is what's happening then it's actually a failure on the audio director's part to not identify that for all users so he gabe was generous and he gave me some of his time when he was in redmond and he played the game with headphones on and i would just ask him questions and one of the first things that came up was his ability to navigate the environment was pretty rough because really he just had to listen for either ambiance or bad guys or traffic or something to try to get his orientation uh but a big chunk of the game is platforming so the ability to know where a building is or some sort of traversable object was important and the best way to do

that is echolocation so echolocation is really hard it's a complex physical interaction you've got a wave that propagates out into environment bounces around scatters and comes back to you and your perception of that delay and the loudness and the texture of that sound that comes back is sort of acoustics the whole game takes place in an urban environment so we didn't have to worry about forests or caves or really anything like that it's pretty much all city street and rooftops it needed to be fully dynamic because i didn't want to build two systems i just wanted to build one system and have it work for both single player multiplayer and then multiplayer you can blast through walls and take down buildings so the building that used to be there is no longer there so we couldn't like pre-process the environment and make some sort of detail map of the acoustic layout we had to do something that was fully dynamic we didn't have any time to make a custom plug-in for early reflections and we started on this right before wise reflect came out so we basically built wise reflect but insider wise without wise reflect

um uh that that it's an awesome plug-in and that actually does more than what we were able to do but for how cheap uh cpu-wise our system was um i think that it sounds pretty good um we didn't have any time to do manual markup the city's too big um so we couldn't like go in and draw zones or make uh you know low-res structures that mirrored the game's geometry and then also acoustics are just noise uh reverb is noise it's um and so the more of that you want to feature the more noise you're adding to your game and there's a whole you know like uh insane design ethos that i pushed on everyone no noise no distortion this has got to be tight clean punchy and i want to hear tone and i want to hear power but without using distortion and noise and which is as we all know it's impossible but the choice to push that as hard as we could meant that the game sounds nice and clean and i didn't want to muddy that up so

when you decide to use distortion or noise it's intentional you're not just using it on everything and then trying to figure out how to get the noise and distortion out of your game once it's all baked in so if we're going to add acoustics into the game we didn't want to undo all that hard work that we had done cool um all right so the technology that we employ to get the data back and and you know make a sense of the acoustics are pretty simple we cast raycasts in five directions we do five rays per frame that's it we do them on the corners and we do one overhead um and uh it's all listener relative so it comes out from the listener which in our game is the camera because i think camera is the best listener i don't like to fudge with it and like move it in between or put it on a spline or put it on the character i just like camera because what i see is what i hear we then take the data output by the raise and we do two things with it we turn that data into rtbc parameters that we use to manipulate all the

different sends and do some other trick stuff with but that data also under the hood determines how we interpolate between the delays so we go out to 50 meters because that's kind of what we you couldn't really afford more every game to find send and wise that you add is uh extra overhead per voice and given how the scale of this game and how many bad guys we can throw at you and stuff we're often running you know 900 1500 cents with this system so if you even just add one more you're multiplying out and it gets crazy so i would have loved to have gone out to 250 meters so that we could have done procedural echoes and everything but 50 meters was like a good compromise then we take both the actor mixers and the output of the game defined sends and we route those through some reverbs one of which is close and one of the which is distant both of those are using impulse responses that were recorded in city streets and uh we i'll show you how we do that stuff so um i'm gonna skip that

i think you can barely see it because it's a single pixel that's just moving around but um a a single ray cast is far too rigid uh so what we do is we jitter them and we take the output of the jitter and we average that over time then i take the output of that and i also smooth that in wise just because it was still a little too twitchy um and we stopped raycasting at 50 meters beyond that point where we don't have any delays to go to so it's a slight optimization all right so the rtpcs that we create are each direction we we measure the distance between the ray from the listener to the object that it's striking we use that for attenuating the delays so we're interpolating to different delay lines so that farther away there's a longer delay before the slap comes back different longer time between the slaps and then we use that distance to attenuate it as though it's getting quieter because there's no object out there it's not actually in 3d it's just

sending to one that's hard panned a bus that's hard panned here here in each direction matching the orientation of the rays we also get in order to further reduce noise reverb sounds pretty bad if you have all these reflections and all this reverb and then you're folding it down to stereo so we detect the end point that the person's listening on and if it's if they don't have those speakers we just mute those in the reverb so i mute the bus like if there's no rears rear reflection is gone rear reverb gone overhead same thing um and that way when so you're not folding down reflections of things that you don't have speakers for back and doubling them up you might even hear in a few of these captures i forgot to toggle back to stereo because the video capture actually folds down internally and it does a terrible job at it so if you hear any weirdness with like some flangy stuff it's actually the delay is getting pulled down folded down i can speak um all right and then and then we just take the rudimentary values that we're

getting back from all those rays and we create a super low res version of the volume of the space that you're in and that's not meant for any detail other than big space or small space and between those ranges up to a million cubic meters we can determine how much we want to send to the close reverb and how much one we want to send to the far beaver the reverbs are just stock wise convolution reverbs and then these are some of the parameters that we use to control them and shape them so if you're in a smaller space it's got less spread and it's a little bit hyped and loudness and as you go into a larger space um it opens up and it gets uh into the wider space that comes up in volume and the close one comes down super simple like nothing special about it just trying to solve problems and be clever with what we had all right so i've got a series of videos to show this off uh hopefully it reads in here it's a big echoey room but this is the dry signal of the carbine in

a test level all right next one is early reflection soloed so if you hear something that means that there's a reflection coming off that geometry super simple it is filtered just a little bit because we don't want to send so much low end and we don't want to send so much high end so it's band passed and it gets more band pass see the farther away it is from you um this is only the distant reverb so as you can see as i get away from that geometry the space volume increases and uh that was hard to hear all right all right and then this is just the acoustics not the dry signal

so up to this point now we've got everything playing together and hey it sounds kind of city-like um we didn't want to over hype uh the acoustics you know we're not doing like battlefield style gun tails this game is all sound design forward so the sound design of the gun itself is key but we still wanted to glue everything together and give you that sense of being in an urban environment so this is uh with everything including the dry signal sure that's it that's our acoustics um the side benefit of spending some time on early reflections in that urban environment is that it sounds pretty good on voices some of them can barely walk it also helps you judge distance

[Music] and just like a lot of people like the battlefield talk they did everything like they most of their system was player-centric same with ours our ai is really stupid it doesn't really know much so what we would do is we would pass a token when you do something to the ai system and say like hey player has fired a gun so someone should scream and that kind of stuff it's better than actually having the ai try to know what's going on that's expensive uh there's also propaganda so the this horrible corporation that is totally not microsoft uh this horrible corporate corporation is in control of the of the whole island and um they blast you with propaganda these huge holographic blade runner style propagandas and um sounds pretty good on that too a vision of brighter tomorrows a vision powered by human potential

together we will forge a bold new future together we are terra noodles all right so that's our acoustics in total all right cloud destruction this is the hardest stuff that we had to tackle i mean it really took honest honest to god it took a technological miracle for audio folks to figure out how to do this um all of the physics for the multiplayer sessions are calculated on a server it there is no data for any of that destruction none of the physics interactions anything exist on your game client so uh we rely on all those values to determine a piece of concrete was flying this fast and hit this other thing and i need to play a sound for all of that and they're like cool well i can give you the position of it which doesn't help um so technologically it's it's quite

difficult and sophisticated i'll just do the simple version because that's how i understand it there's a cloud server that all the players connect to it's hosting an instance of the game where all of the physics are being done all of that data is then sent down to your local server your local server speaks to your local client those are both on the same machine and that then is rendered so here's basically what it looks like with no audio on this is the level of mayhem and chaos that we have to try to make sound decent and sound cool and hopefully if we do our jobs not fake because it's it actually is all real physics um all right so uh just real quick on some of the definitions of how all this stuff works um we've got the cloud server you got local server some of the things are cpu processed and some of the things are only on the gpu client-side prediction or csp makes it

so that things you know if you drop a packet or something that something doesn't just stop and then update over here it'll smoothly interpolate and if it has to rewind and fix itself it's less noticeable than like not updating per frame then we've got some things that some really awesome i you know it's really cool that they figure out how to do things only on the gpu unfortunately there is no data for us to tie into so we had to write all these new systems to get you know basically query what's happening on the gpu and pass that to the cpu so we can do something with it there's a whole bunch of behaviors and then uh that we need to cover for each of these pieces you know when the whole building's blowing up and then there's a tremendous scale tens of thousands of objects are updating across the network they all have a different state they have a position in orientation and there's a huge range from i think there's even smaller than 50 kilograms now but 50 kilograms to like a billion or whatever is the max it's just we need to make it's hard to make concrete sound huge so we have to do that all in real time

all right so there's physics chunks gpu chunks static geometry which is like the ground that can't be destroyed and then we've got impacts destroys and we've got a value of impulse we do it all because of the nature of how the team is so distributed we try to do as much as possible and wise so every physics sound in the entire game for impact is a single event call it's play physics impact and everything we do underneath that is driven by data that we query from the game and set a parameter to drive a switch so it's i we hate the switch view like with a fiery passion we wish that wise would fix it because it's terrible it's extremely powerful and you can do a lot with it but actually dragging things and having to reorder things and the fact that you can't sort to actually see what you're doing is like nightmare inducing but the cool thing is you can do layers and it'll mix stuff on the fly for you it's got a lot of functionality built

into it so um we nest the switches so the first thing we do is the material and then we do the mass so it'd be like physics impact is the action then we do carbon dark as the material and then we do the size carbon dark huge and then we go through all the impulse levels under that switch and that when we define what sounds can play at each one of those so that as you can imagine our switches i should have done a zoom w but our switch view is is enormous fortunately the adding materials is extremely expensive for every department so they kept the number of materials down to a reasonable amount because there are so many actions in so many mass ranges that we had to cover um because we have to make actual sounds to represent the stuff we can't physically model or procedurally generate it we have tried that in the past it just doesn't sound as good as a sound designer putting their creativity in and making it sound the way you want it to sound as opposed to how it should physically sound like physically correct

we take ranges of mass and we assign them to a single sound so there's tiny small medium large huge and then we even added enormous because we're running out of words to describe the mass range but it's better than like zero one two three four five so um by taking a range of masses and saying this is what we say is a medium sound we know what sound to make for that size um then within the range the min and max range of that mass we track another parameter which is the percentage and we can use that to sort of pitch it up or down slightly to give it a little more gradient so it's not just like you know some of the ranges are actually quite large so being able to pitch it a little bit and give it a little more variety is is much appreciated so we do the same thing with impulse there's a range we have to bucket it because we have to make a certain a finite amount of content um then we've got a bunch of behaviors it can be damage destroyed it can move

through the air um it can scrape on the ground it can hit stuff and then uh they cleaned it up and we call that a d-rez just like tron um because it looks like tron uh i'm gonna skip all this stuff because you know what ballistic damages this is so destroyed is not when it's actually deleted that's a d-rez destroyed is when any physics chunk breaks into two or more smaller chunks um and so it's basically if you can think of it as something fracturing that's it's just called the destroy turbulence is cool and i'm going to get into more detail on that but basically as havoc bodies are moving through the air you want to hear a moosh and we do full contact sets for every single mass range and speed you can have an impact is if a physics chunk collides with either physics chunk or static geometry um then play a sound that sound that system is designed to be two hands clapping so i've got this thing hit something and this thing

was hit by something now you get a sound that sound is not a single sound that we say like oh it's concrete versus glass at this speed and everything it's the sound of concrete hitting something and the sound of glass being hit by something and we play them both at the same place same time so it's actually two both objects register their sound that they should play because the same place same time you get an aggregate sound that sounds correct um we have a bunch of different layers that's so we can get perspective on the sound and tune it differently impact resonance which would be like if it's a piece of metal it's the ringing of the metal um sweetener would be like hey it's a piece of concrete and some chunks fly off so we want to get a little bit of dust and grit and debris on there and then lfe we do for sizes above medium medium and above yep okay and uh and the huge ones we really just kick your room with a lot of air that's awesome if you have a five one system this is a great game to showcase it with uh because we went crazy with lfe lfe is fun

all right uh scrape is so if a turbulence is in motion and not colliding with something then this grape is it's in motion and colliding with something pretty simple conditions it's really hard to tune that was actually our most problematic system because on the server getting data back from the server to say hey i've done like a bunch of collisions in a very small window should i play a scrape um is uh is still our most problematic system um and uh it's just okay but it's kind of nice when something is hitting and sort of rotating and falling that you get you get the impact and then it's got a little bit of a scrape and then until it falls it's cool all right so this is an example of with all that working just destruction audio so no weapons and no building level stuff

huh right the game designers were super nice and and they decided that when anything should break they should just play an explosion which was really nice for audio um and so that's why you're seeing so many explosions um so that's just destruction audio only here's with uh everything weapons and everything all in one

and this is all test level stuff in a normal match there's 10 players all trying to kill each other and shooting stuff so the audio director in me gets obsessed over the details and i'm like there's stuff i didn't hear you have to kind of go yeah but you're never gonna actually hear that that isn't playing that one little thing that rotated and phil so we did try to get all that stuff just sometimes we have to make some hard decisions about what's most important i'll get to that um oh and then we just play we use the uh server cleanup as a sound design opportunity all right so buildings so those are all the destruction chunks that's all the little pieces that was probably the hardest thing to get then we can query

the state of a building a building is a collection of pieces fused together through constrained joints and once you damage it enough it will enter into different states those states are below we have sound designs for each one of those states here's sort of the definitions of how those building states work it was really simple we were like hey at 33 percent and 66 percent can we just start triggering some sounds and basically where there's constraints we can start to play creaks and we randomize where they are so when you're in a building you can tell that it's starting to take damage so so that you don't hang out in there too long and get crushed to death is the real idea so there's critical the server can handle uh one building collapsing in its entirety at one point in time so if two buildings have been damaged enough to come down one of the second one is queued and so it enters into a special state says like dude i'm totally going to come down as soon as the other one's done and we have a system for that

and then we have a building collapse so here's just building destruction all the physics sounds are muted in this one [Music] cool so we can't really cheat in in this game because you can [Music] go all over the place like we don't constrain the player at all and especially in multiplayer so we actually had to figure out a way to get all the sounds positioned on the structures and do everything correctly um when the server reports that a building is collapsing we have that shepherd tone that keeps coming in so it always feels like it's falling um and then when when the server says hey if stop that's when you hear it fade out um just a simple little trick we do a lot we used a lot of shepard's tone in

in crackdown 3. um so this is um sometimes a building can be a building a building can be attached to a building what's my time check i'm at five minutes oh crap all right this is breaking off a part of a building from another building so it's a building and it breaks off now it's not a building now it's an enormous piece [Music] cool uh and then this is a full simulation just a whole building coming down so not perfect but i don't know pretty awesome considering the technical

challenges to get there um and then the last part i'll blast through this performance tuning was super hard so basically with tens of thousands of things happening the first when we first started this we were like can you just send everything to us and we'll work it out on the client and they were like uh sure we'll turn that we'll turn on audio and pass the parameters that you asked for and we completely took down the server and so then they were like man you guys took down the server you don't get any data so and we're like well we need something so between some and none or all and none we needed to figure out what was the right amount of stuff to to hit server performance and also give us enough so that we can do something convincing for the player so we do a whole bunch of relevance checks um i'm going to have to go through that quickly because i talked too long but the way that it's done is we take everything and we look at we start bucketing things and we say in this bucket you can have this many and here's how we determine what's most

important and they can graduate out and we filter filter filter filter and down to what is most important to send to the player because server bandwidth is is precious in a multiplayer game once it gets to the server uh so oh sorry one of the ways that we do that is we merge collisions as they get farther away from you so you need a lot of precision like i'm standing right there and one little chunk come you know falls and hits the ground i want to hear it like right then i don't want to wait the more you merge the more latency you're introducing to the uh sound because the server has to accumulate a certain number of impacts and then say hey here's a merged impact for all this stuff so as it gets farther away we increase the radius on those that's a really nice optimization we got a ton back from that and we also bias that towards the viewfrustrm of the player which has helped a lot so if you basically it was like when it was just a sphere around the player things behind me were totally taking precedence because they were huge or fast or something over stuff that i wanted to see

but that all had to be all that logic had to be done on the server so on the client side we do all that whole process again because even the stuff that can come to the to the client can swamp wise um on the y side we do that again but for voices so we still like okay only this can come from the server only this can get to wise and then once wise is there it's like i can't really play back all that stuff so i'm going to do my you know the most important stuff of what's left i'm going to play sounds for those and we aggressively limit and throw out sounds we don't want we have a priority system for every single voice and so we and it's a gradient between how heavy it is and how much impulse it is and what material it is if you want to dig into this later these are the actual numbers we use for um how we limit how many instances come through and then here everything

and then this is just to show you how extreme it can get so this is a building with all the physics stuff simulating and the players still interacting with it like still smashing it up as it's collapsing all right and you can do that at 30 hertz on an xbox one which is the miracle part so it's pretty cool um all right so turbulence if something's moving and it's not colliding with anything with anything it's turbulating um this has a lot of physics in it bad

guys throw physics objects at you as weapons you can throw stuff you can blow things up into smaller parts and they fly all over i'm going to quickly get to some examples so real quick this is the scale of which every single little section you see here has props in it there was a million billion um this is an example of the spreadsheet that james uh managed to track it i'm going to do the zoom out yes there's that and then oh god why did they make so many props and for most of those we made unique content it was 700 yeah somewhere around 700 unique props that had a full set of everything from impacts to turbulence pretty cool and then that's outside of the destruction materials and all that kind of stuff and we dynamically load those banks that's uh you can imagine so here's what you get with turbulence

[Music] pretty cool guys like ben sightless combat can now tell when physics objects are flying around here's throwing a pipe right here's a really cool one with uh that shows off how we take the rotational velocity of an object and we modulate the sound damn that last tip over i wish we i wish the tip-overs are the hardest because it's inco it's in contact with something and so havoc doesn't give us the right data but um and then uh oh this is this is probably the coolest sounding ones piece of sheet metal and as you throw it you throw it like a discus and so it spins and this shows off what the rotational boss lead pretty cool it's super simple it's actually not expensive um and we have all that content now so it's good

because other people can use it um uh this just shows that we also do it for destruction objects [Music] i'm out of time yeah okay can i go five minutes over okay i want to get to our mix stuff i'm going to cheat lfe we play some lfes all right so the mixed stuff um we wanted uh player feedback is essential but we wanted the music to be nice and up in the mix that creates contentions you've also got narrators as you heard they just talk all the time we need to make sure that you hear everything they say um and then we just wanted to reduce cacophony noise we use wise hdr i know some people are saying they don't like it i love it this game would not have been possible without it but it's not easy it's really hard you can't just grab a fader and be like i want this to be quiet you have to do all the math in your head and think of it in three dimensions it's horrible but um

but the good news is once you get it right the game just stays clean sounding um we do frequency domain processing much like um the just cause guys uh where depending on what's happening we'll carve out some frequencies so if you can think of hey i don't want to duck the music when someone talks super you know like 6 to 9 db like you would need to and you have the music nice and up in the mix but it competes with the melody so i can bring the melody down leave the perk and the bass because there's no frequency contention there and then all i need to do is on the percussion just really the hi-hat is competing with the um audibility like the sibilance so i just use a notch like i use the meter on the output of the speech to pull these things down so you it's if we did our magic right you wouldn't notice you can notice a little bit but it's not as bad as like some games i've worked on where we've had been like just dump the music so you can hear the voice we also do some category category things like hey if i'm shooting a gun an explosion goes off the gun can cut the lows on the or the

explosion can cut the lows on the gun and the guns can cut the highs on the explosion super simple and they just trade off and the meters are really fast we own fast meters on those peak meters and they trade off very nicely hdr is great for sandboxes with chaos every single sound in the game is 3d it was a decision that made because i think 3d sounds are awesome and we wanted to make something that could be a showcase for spatial audio which we're still wrestling with some bugs on so if you play it today you won't get that but it's coming um yes we use meters too uh here's some information you can dig into later about just some of the peqs and ducking that we do performance was horrible you're not the only ones that's why this slide is here we aggressively throw sounds out if they're at max distance there is zero priority we don't care about them we just care about the stuff that you should hear and we limit all of our sounds to 50 max this game with all the things that can happen you only ever hear 50 maximum sounds

and the way that we do that is through hdr for audibility and then priorities believe me you can't tell that we only have 50 voices playing it's crazy i was inspired at i was challenged by bungie because not challenged by them but challenged in knowing that they limit their voices to somewhere in the 40s for destiny i'm like how i played that game like it's impossible so i just want to thank everyone on the if anyone here worked on nimbus there was a code name or or our friends overseas that that worked very hard on this game this wanted to thank you it was a long journey and there was a lot of dedication from the other team on this i love you guys love you all thank you [Applause]