Building a Mixing Sandbox for 'Just Cause 4'

[Music] my name is dominic vega i'm a senior sound designer at avalanche studios and i was on the sound team in our new york city city studio for just cause 3 and just cause 4. among other areas of content i was responsible for the creative and technical mix of just cause 4. at avalanche studios games are made in our proprietary apex engine and the audio middleware we use is fmod studio while we work in fmod a majority of what we're going to talk about today is openly available and applicable to other audio tools on the market now in designing just cause 4 some new ambitious features were added that changed both how we needed to functionally approach the mix and how we were going to join everything in an aesthetically engaging way before we go a lot further i want to contextualize a bit of what we face as sound designers on a just cause game for those of you more unfamiliar with the tone of the franchise and where just cause 4 was headed here's a little peek into the world of just cause and what we were aiming to create they want to bring the thunder

let's bring it [Music] changed [Music] [Music] i'm just [Laughter] so while the gameplay and the tone of our franchise uh is really bombastic and over the top we do actually put a premium on nuance in our approach to sound design subtle details and dynamics are really important to us and something that we feel makes worlds really great to be in

now to bring all these elements of an open world together we place a strong emphasis on the game's mix for just cause 4 we wanted the mixed planning and prototyping to start at the design phase and grow and improve with the rest of the game what we didn't want was lots of huge last minute changes going in as is unfortunately common in a game's production these sort of last minute mixes risk masking fantastic sound design and music moments to better suit bigness or finality to do all this we approached the mix on just cause 4 as its own separate system instead of as an afterthought in the finishing process we started establishing pillars which would become guidelines in the framework of our mix now out of the gate we prioritize balance this is an early first step setting this overall static mix of the game and determining aesthetically where we want specific packages of audio to be set at we have thousands of assets we want to deliver in an intuitive way and we want to control and maintain balance of the overall sonic weight of the mix we want to emphasize clarity and adding a layer of focus to what the player may be most interested in or interacting

with we never wanted to lose the punch keeping the sound of the music hitting hard while retaining dynamics and economy of scale and finally style we wanted to infuse a little bit of the avalanche dna into the overall audio experience delivering something unique and that highlights the fun of the franchise we also wanted the mix to have brains and modulate more accurately based not only the sound being made in the game but by the game's design itself so now that we have our pillars defined let's look at some of the problems we encounter when trying to mix a just cause game firstly we're never able to gate the player's experience to deliver more nuanced content we have to be completely prepared to make the most intimate dialogue sequences be audible in the most chaotic scenarios we could find now in just cause 3 which was using more of a dynamics driven mix structure sound designers working in assets in low priority buses often fought against the mix to be audible this led to really hot asset designs and audible pumping as the dynamics have essentially been removed also relying on compressors for a mix is not only expensive in terms of cpu and processing they're often one look we often found

ourselves either over or under compressing but worse the mix is rather unintelligent in making space for assets as compression can only yield some very small specific results now a new project with a fresh start you guys would have the benefit of starting your mix just needing to imagine what your last day of production looks like what's crucial in designing a mix is deciding what framework you want to build and ask necessary questions of what your mix can do to better serve your game so for us to better serve the design of just cause four we knew we were going to have to rethink the mix process we had only just finished make the mix on just cause three a few months earlier when we moved into planning for just cause four so instead of putting the mix off and relying on what we had for a few years we decided to attack the mix head-on let's take a little bit closer look at our first pillar of the mix now early in the project we have this really long runway without a lot to test out we took the opportunity to suss out where we wanted basic levels of mixer groups to sit in the final mix and investigate how to remove layers of processing between a sound and the listener with so many assets in the game we

needed to parse out in silo content that was designed for player interaction from what was more diegetic in nature we could set up pseudomixed priority buckets then leave room for distinct sounds the player may be more locked into experiencing like driving a car and so on in doing so though we exposed some issues with traditional audio signal flow that arises from our gameplay model here is a simple vehicle engine event from just cause 3. now due to the fact that you can hijack a vehicle mid-drive and just cause unlike other games that might allow some more creative content swapping we allow the player to hijack a car while it's running throw the driver out and resume driving in 3d space we may want ambient cars in traffic to be in a lower bus where guns explosion and destruction will push down on them in the mix with side chains this is completely logical until you enter that vehicle in just cause 3 we solve this by just boosting the level of the engine based on a parameter called is player driving um this however causes low end build up it still pumps against louder sounds and the max distance is stuck being as long as the furthest

camera distance from uh from the player in order to make the sound audible in 3d space the solution in order to silo content in just cause 4 was to send it around the signal flow based on those same parameters moving an fmod return bus up in priority so audio is sent and bypasses output ducking to a higher priority group allows greater control less dsps and easier intelligibility since the player can only ever control one vehicle at a time we can easily set up mixing dsps reserved for player vehicles and resume and bail out of those states as quickly as the player chooses now let's expand the idea a little bit further by doing the same thing with the component parts of vehicles like tire interactions chassis rattles transmission winds physics interactions and exhaust you now have every level to all assets involved with the vehicle in their own returns we can then put a snapshot on every vehicle's event just checking for the condition is player controlled to become true which then triggers a micro state for that individual vehicle should the player enter it this snaps the mix settings for all available layers and groups we want to

mix uniquely for each vehicle this also allowed us to take most areas of vehicles excluding engines and use more global content saving on performance and memory let's take a look at some examples of increased sonic variety with more general use complementary sounds just using modulated mixes [Applause] [Music] [Music] now next here every layer except for one is exactly the same as before what's different is pitching spatialization and mixing of the component layers of the vehicles quite commonly we were able to get away with making extremely small changes in a singular layer of content and achieve really drastically varied sounds [Music]

routing this way with precise control over the output of individual layers and using microstate snapshots which we're going to dive into a little bit later for each specific vehicle reduces the authoring time of a single vehicle mix from several hours each to only a few minutes if need be the content being siloed available and largely unprocessed gives the mixer the necessary freedom to focus solely on making this singular instance sound great without needing to worry about the other 100 plus vehicles that need mixing down the line once content was more siloed and accessible should the player interact with it we started cleaning up layers of the mix through imaging where we situated sounds in surround space has as much impact as amplitude does on output since we're mixing in 7.1 clearing up that space has trickle down effects through the fold down mix as states become more balanced and controlled cleaning up sub states and behaviors to disallow output on speakers let previously masked audio come through the mix to show some examples of outputs to major groups you can see how more specific output paths enable space for a more detailed soundscape let's take a closer look at some examples

now while a really low priority sound world audio and ambiences play a big role in games like ours and said a lot of the audio tone they're a persistent character in our games but in order to make room in the mix we keep them extremely wide there are two layers the core ambience and the details details are spawned at random points in the point cloud so the player has this effect of moving through them and the core is a quad ambience rotating around the player's head making the image feel more varied and alive so

land vehicles present a challenge for a lot of games as they are very noisy and full frequency source their placement at the end of the mix chain is almost a full 7.1 image of being driven by the player excluding the center channel if broken out though they can have up to nine component player specific sends that all use different outputs here's a deconstruction of how all the individual layers of a vehicle sound are imaged and how they sound all put together to increase clarity and get rid of audio clutter [Music] [Music] [Music] [Music]

[Music] and lastly player weapons are 5.1 with discrete subwoofer content they're big character sounds we want to reserve quite a bit of space for them however to retain clarity keeping layer separation a priority kept the weapons from sounding muddy or just simply loud we only allow the mech layer of guns into the center channel the close and medium layers are front heavy stereo spread based on whether you're aiming down sights or firing from the hip and tails or stereo asset that expands and spreads to quads the shot blooms out

[Music] so now with a more granularly compartmentalized mixer and channel separation and mixed packages we could tackle improving the perception of size and scale in the game and make the mix punch now a really obvious statement about to make to everybody in this room you're probably well aware is that low end management is important in any mix now especially for us and just cause four it's something we paid a lot of attention to we have to accommodate barrages of explosions and gunfire and keep it from becoming muddy or too bottom heavy we also want to maintain a decent dynamic range in the mix that a lot that doesn't feel overly compressed now as with guns we opted to use discrete lfe for surround mixes in just cause 4 and not force sound designers to concede on their setups we design discrete content for medium and large gunshots and small medium and large explosions and in the end we still

reserve the option to send any sum of an events mix to the sub should we need to removing all subwoofer content makes mixing on a surround system more informative as it's easy to rely on the sub as a tool in your handling of your base designs but by removing it and especially removing all sums to the subwoofer you're forced to decide where it's desired and manually add it back in but why did we opt for using discrete lfe content instead of just a simple frequency roll off to the subwoofer for us we found it cleaned up the subwoofer immensely by not allowing more in than we wanted and we could design cheaply by making lfe effects for weapon types instead of per weapon most of all though sound designers were never forced to make concessions in their designs to accommodate all playback systems in the past we've had issues where a sound worked great over headphones or full range speakers but with a subwoofer it became muddy or maybe not pronounced enough you typically only have an lfe slider in most middlewares in events and output buses so you can only increase or decrease its overall volume send to the sub now in this case we sat with a consumer level subwoofer and designed our sounds

playing through that simple low-pitched sign-based sounds without a lot of upper harmonics worked really well through its low-powered output and sounded really crisp and tight now a lot goes in to maintain the low end of a mix which could be its own 30-minute talk but first accounting for where low end could and wouldn't be handled on the highest end of systems paid dividends as sound designers had more reasonable expectations at the sound design phase of content this just simply empowers more people in the audio team to positively contribute to scale and a punchy mix without over complicating the process now our last pillar was style style was actually where the mix became most fun with the previous priorities in place we had a really solid foundation to now modulate the mix in really creative and unique ways find space for assets previously deemed too low of priority and contextualize the surrounding audio in ways that best fit our game's design this would lead to us designing our adaptive listening mix now i said earlier that we wanted to make mixing simple at this stage the mix has largely been an exercise in deliberate voicing and subtraction

these are important stages but with games we're also accommodating for focus and feel these are harder to quantify and as we continued with the mix we wanted to strike a balance between prioritizing simple locked-in experiences where the player's action is more easily known and maintaining interest based on the surrounding context of these locked in states with the existing framework and a shippable static mix in place mixing had now become simple but that doesn't necessarily mean that it was easy now working with designers and teams on game features opens up a lot of avenues for discussion and debate in the early stages of remaking the mix before the idea was fully realized extreme weather was being worked on really heavily i had these scheduled sound design reviewers reviews with the product owner of the feature one of our lead designers and he kept wanting more from the tropical storm in this review i was a bit stuck not wanting to overpower the player with wind and rain and lightning to an exhausting degree a bit frustrated because this review was not going very well i also said offhand i don't know how i'm going to get enemy gunfire over this mix at this point

um i had now lost his attention completely and he was on his phone scrolling through emails and he was walking out the door and he sort of just said i don't know i don't really care about that i just want the storm bigger it was actually a really great point was kind of a light bulb moment for us we needed a smarter stylized mix to deliver a great sounding experience that could break some more traditional rules of priority but before we move into some of the more creative solutions we used let's talk about how we functionally began the process of reworking our mix logic to support style an early course of action we took was to remove all bespoke narrative mixes we then started prototyping dynamic states and microstates that would allow us to blend between mixes seamlessly and deliver on a unique and always changing soundscape we then removed most compressors and dynamics based mixing dsps from our mixer along with side chains and previously that previously determined priority audibility now since the inclusion of front lines and extreme weather kind of rocked the way the player perceives things like enemy gunfire we took wider liberties with re-prioritizing enemies firing at

you whenever we could in favor of something more threatening or interesting that we deemed worth forcing your focus now to clear up some terminology states and microstates are just mixer snapshots in fmod studio like other mixer states available in almost all middleware these are just simple terms to account for any player interaction or action and how we want them to sound now when we talk about states specifically we refer to settings triggered more globally things like players inside tropical storm is a state triggered when you enter that storm some other examples of states are car chase combat exploration dynamic combat a micro state is a snapshot that's triggered with sounds in the world in 3d space they can be equally context driven but most likely the intensity is simply faded in the closer to the sound source you get an example of a microstate is enemy ai is targeting rico this micro state sets some overall levels quickly and is played when enemies are firing their guns at you microstates are really effective in creating space more intelligently and making important or loud sounds in the world stand out adding logic like

direction simply lets us make vehicle explosions more focused and clear as we disallow the micro state if the player isn't looking at it this isn't possible with just sidechain compression as that looks at a pure amplitude output and ducts accordingly with this taking no compressor dsps no side chaining and no metering we get improved punch with better performance now keep in mind as the mix is now very separated and routing our granular control lets us even make changes to eq and balance if certain moments become too overbearing some other examples of microstates we made where ai is targeting player enemy vehicle is chasing player enemy helicopters engage with the player so we have a baseline for what states are and some examples for their use how do we use states to attack the problem with the tropical storm we want to enforce the design that storms are all-powerful while not drowning the player in sound to an exhausting degree where the mix became most stylized was when we exercised one of the most powerful tricks in setting unique mixed states modulating between states or within states themselves simply because we can since data we use to set the mix and other aspects of audio is already

available we can insert some chaos and randomness into the experience to account for fatigue varying play time or scale for these persistent storm events we opted to use time as the main modulator in the tropical storm we set the level upon entering the storm quite loud we want it to feel overbearing but after 15 seconds we start fading the base of the storm down by about 10 db we trade off and start mixing the thunder and lightning up to compensate and after another 10 seconds we lower the lightning strikes and only then do we allow enemy gunfire to duck anything inside the storms mixing like this lets storms feel chaotic and oppressive when you enter them they're extremely disorienting when you're getting struck by lightning but eventually some more traditional room for hierarchy can resume in the mix in sandstorms we mute the player's gunfire tails to make the storm feel claustrophobic boost the barks of the enemies should you not be in combat as they're yelling searching for you and track the direction of wind to balance the weight of the sound design of other noisy assets to keep the mix image even when the player is nearing the tornado we mute the player's car engine as the

tornado base is really full frequency we want it to feel like the player's cars are losing control near it now one state that was most complicated and dynamic was the dynamic combat mix state easily the backbone of our mix it managed streaming asset counts and handled various sizes and types of combat experiences one of the first functions it performs is it scales the mix of ambiences and surrounding low priority sounds the more ambi the more enemies the less ambiences this gets noise out of the quad image and in a compression based methodology would be ducked by gunfire anyway we have the ambiences completely muted if the enemy engaged count is greater than five this also dumps the need to stream those quad files make space for the music and lightens the load on audio performance so if the players to get into a car we now from the mix perspective consider that a chase the mixer takes one second to fade to this new second layer of combat and changes levels accordingly in the case of car chases we quickly duck the vehicle mixed package down to alert you to gun fire lower the rears the land vehicle's group until the chase combat is complete measure the distance between you and enemy vehicles chasing

you and inversely lower your engine and boost the chaser then we raise the level of the rear speakers of the enemy helicopters bus this really quick state change makes car chases less frustrating because you can hear all threats as they're out of camera view and briefly modulate the mix to be more rear speaker heavy now in attempting a major rework of our mix what did we learn and where did we maybe fall short now one aspect of this mix approach that makes it difficult is it's heavily reliant on feedback since we're still day-to-day game developers i can't ever spend all my time and focus on the mix uh it was great that iteration time dropped drastically and sound designers content was being complemented by the mix structure but we needed play tester and developer feedback to address areas of the game we may not be looking at aside from the sound and music being a synthesis to the visual and the experience the mix can be a vital tool in aiding the game's design but communicating that option to developers can be very difficult sometimes we had to dig through feedback searching out ways that sound could aid in areas of game of the game being labeled frustrating or challenging

listening and conversing outside of our audio walls was crucial in gleaning insight into what developers and players expect beats and gameplay to feel like and lastly one thing we can't avoid is that by reducing the layers of processing between a vehicle engine event and the listener the amount of vehicle-based groups and returns alone in the mixer multiplied 10 times over processing wise this makes no difference and in fact more granular structures actually contributed to where most performance savings came from in the mix but the mixer group account grew exponentially over the project various areas of the game were similarly restructured like vehicles were making the mixer quite large with hundreds of groups and returns the authoring of snapshots becoming so vital to the mix itself meant that an increase in workflow efficiency directly correlated to more states to create and manage on reflecting we still felt all of this is worth the benefits and shows the extent to with which granular control on a huge scale can look like so we've removed layers of processing between our sounds we found space in our mix image to better complement and declutter the soundscape and enhance the sense of power and economy of scale

all these steps act as apparatuses in our sandbox to reconfigure the way we prioritize and feed players sounds based on the surrounding context and our own internal stylistic leanings open world sandbox games can present an enormous challenge to delivering a subtle and interesting mix the principles with which we mixed though can be practically applied to any type of game but not treating your mix as an afterthought and pushing the limits of the tools you have you can better integrate your sound design and music into the fabric of your game with that whether you work on a large audio team or a one person shop the foundation is set to use your mix as a powerful instrument for reinforcing your players experience now the things we haven't gotten into today like loudness mastering dialogue mixing design changes tone all these are important and worthy topics but the framework and questions we've established can better pull solutions to the above problems into focus i truly believe that mixing should be simpler and perform both a functional and aesthetic role what hope is taken away is not only the importance of your final mix or improving your skills as a mixer but the importance of

the questions you ask when you're designing your mix ensuring that you're always mixing for the games that you're actually making now before we break for questions i do want to take a few seconds to thank the entire development team at avalanche studios for helping get this game out the door and with this presentation both in stockholm and new york city uh as well as the entire audio team that was uh really really important in helping us ship this game uh thank you so much for taking the time and coming to this talk have a good gdc um if you have any questions uh we have a few minutes now to to go over stuff that's it [Applause] hello well first of all thank you for the super interesting talk uh i was wondering uh well you said you mixed it in 7-1 yeah i was wondering if you uh did like uh separate mixes for stereo 5171 headphones this kind of thing yes we did um we we mix and review in every

format but since 7-1 is kind of our master that it's the back of the box feature that's what we're kind of aiming for as far as fold down there's obviously fold down steps at every layer all the way down to stereo that are quite complicated to go through and a little bit of hidden magic in middleware so we do that stuff does matter but uh we didn't really the spatialization and imaging thing starts at seven one and moves down the mix okay and another question uh does the uh i didn't play the game unfortunately does the player has a lot uh have a lot of control over the mix like uh do you expose like dynamic range settings this kind of thing no it's something that i think is really cool and something i'd love to do in the future the only controls in the game that we have for sound are the more typical vca sliders sound effects music volume stuff like that there's other areas of mixed things that didn't make it in that that would be good like accessibility features that are on there night mode was not in there right those are all

just states and microstates that didn't get made but yeah no cool thank you yeah thanks uh hey uh so you mentioned that um going from just cause three you had a lot of um like heavily compressed assets kind of on the way in which made it a little more difficult to mix did you have any like guidelines for like dialing that back do you have like loudness standards for the assets on the way in to kind of help it pre-mix a little bit before you even got it to the engine yeah i think um i may have failed to explain that well enough uh i think what we what we experienced in just cause three in the in the final mix and something that i've experienced in games i've mixed before you start seeing dynamic range disappear as you're finishing the game as you're doing these mastering passes and things are getting pushed together but also the fact that when you're doing sort of a strictly priority based mixing structure where dynamics are involved if you're a sound designer stuck working on sounds in the lower bus you can never hear your work so they start pushing it up in volume because they're trying to focus on it and make it sound really good uh so that's what we were trying to

solve there i think it's not really an issue of the content not being dynamic it's that they're pushing up against a mix that's fighting them all the time um i've seen that in quite a few games where you'll see the side chain almost always active on the lowest bus because there's always something higher up pushing on it so i don't know how anybody could ever actually sit there and reasonably mix their content if you're the ambience person i don't know how you could do that okay um yeah cool thank you oh and we did not do content mastering guidelines that's not something we do yeah any other questions we're coming out [Applause]