Breathing Life Into 'The Last of Us': Part I and II

thank you I'm Bo hi uh so I got started in games in 2014 started at a company called wabi-sabi sound on uh or in the blind Forest the witness and moved on to Bungie worked on Destiny 2 of some of the DLCs then made my way to Naughty Dog uh helped ship Lost Legacy and then Last of Us Part Two the lovely beautiful awesome part two and then now my PlayStation just shipped God of War Ragnarok so yeah been uh it's been a couple years um of a lot but yeah and I'm Jesse I started my career pretty much early like early on in commercials and then I came out here and worked for outsourcing companies like Formosa and Technicolor and things like that um I actually got to work with naughty dog on Uncharted 4 as an outsourcer and then came to naughty dog back in 2018 to help ship Last of Us Part two and then most recently part one so I've gotten a

little bit of both sides being an Outsource of with companies and then being in-house with the company as well so in this talk we're going to be diving pretty deep into the breathing system we developed for The Last of Us Part Two which was later brought into classes part one um so early in the in the production of part two there were uh you know we were hearing about Whispers And murmurings of uh this bleeding edge uh prototype and all many breathe uh bleeding edge prototypes and systems from Upstream departments like animation programming props and rigging departments that frankly made us a little nervous uh it was becoming very clear that the studio at naughty dog was gunning to release a technical Beast uh with The Last of Us Part Two a title that would Dazzle Gamers with this interactive props set pieces and systems so the audio team understood early on that we needed to evolve with the studio and reinvent how we handled certain audio challenges specifically for player enemies and Buddy characters after some time spent in pre-production

our needs solidified into two key pillars the first pillar being a creative need The Last of Us a static merited a robust breathing system The Last of Us is gritty it's realistic and insanely detailed it has a story with characters you can feel excitement with laugh with and even cry with the high dynamic range soundscape bags for a robust breathing system that really telegraphed the player's level of velocity fear pain and exhaustion all through sound if there was a game that required a detailed breathing system The Last of Us pretty much fit the bill the character you play as are they're vulnerable they're broken but most importantly they are human the goal for sound is always to support the story we knew if we could hear their breathing reflecting and supporting these truths that we could be achieving a higher level of detail that would support both of these goals The Last of Us features extensive Stealth gameplay which is a break from the Explorer and combat

States because of the dynamic nature of the audio in the teal Universe a breathing system would be an audible welcome addition for immersive storytelling and gameplay feedback and the second being a technical need so starting with The Last of Us Part Two the studio adopted this new technology which was later added to part two and it's called motion matching I'm sure many of you have have heard of it um so it's a motion matching is a system that programmatically checks the desired character trajectory based on inputs from the stick or from character pathing every game frame there are dozens if not hundreds of animations being searched to find the right animation to play in a given context it's an extremely detailed and elegant solution to achieve a realistic move set for a character and with this addition to our animation pipeline audio now has a new Beast to overcome in terms of how we traditionally tagged animations and just for some history in The Last of Us for PS3 uh Circa 2013. infected vocal

efforts were hand tagged to animation frames the animations played back in a less granular way and more and were more long form so this this method was far more feasible and less error prone for audio the vocal efforts were also streaming and not loaded into uh memory and the cons of using streaming sounds meant when a sound triggers there was a possibility that it may be delayed based on how hard the i o is getting hit at the time of the audio triggering so you know if if the game is trying to load a bunch of textures all over it's the you know sound streaming sounds are also trying to load the I O's getting hit the Met sound might get delayed a bit that's kind of the last thing we want with a game like the last of us where you know if a clicker is looks at you really quickly you want to feel like oh my god there it is and you hear a sound associated with that visual you know so looking forward to part two we knew that the old method of tagging animations was a non-starter with this more granular spliced animation technology

we felt that all characters needed two things the first being a looping sound concept leveraging game data so for this it's sort of like a you know a single looping sound event that plays on a character's jaw joint or face right where we can type in game data to our audio middleware like player velocity Health any you know game data that would be helpful to make this a more modular setup so we can align uh so we can play back modular bits of audio to align with the animation and the other need was utilizing in-memory sounds for frame accurate triggering this is so we can ensure our sounds are completely married to the pixels so when a clicker does that you know whoop what was that right that that head turn um that contort uh when you hit that with a shriek that you know that clicker inhale that's the sound glues to the animation you know it's it's merging the pixels and the sound and it becomes this thing that that's super heightened

um and it and it just feels more next-gen you know and it pushes the story in a more a threatening interesting way all right so we got to thinking what kind of Technology do we need to expand our audio systems into something that we can manage for the scope of this demanding title after much work and collaboration with dialogue programming and other disciplines we landed on the beginnings of a system that would snowball into something expandable flexible and something that we were all really excited about it can be summed up into two Core Concepts and we call those memoration and heart rate murmuration is a persistent Loop sound on a character we call it murmuration because it's basically the sounds under the sounds for Joel Ellie and Abby it's immersive breathing for infected it's groaning mumbling clicking shrieking frenzying things like that and for dogs it's like mouth sounds licking things like that um and we even used it on the baby on baby JJ as well to be able to play

individual like cooing or you know um things like that that babies do um it's kind of anything that we wanted it to be or needed it to be and that's kind of the power behind it let's watch a demo of this in action huh all right so murmuration is the lowest priority sound possible within our dialogue system there's a big laundry list of Vox priorities where the higher priorities will basically stop the lower ones for example a player leaping over maybe a short wall or like a vault the player may play an effort this effort has a priority of what we call effort low this effort would stop the murmuration loop and once the effort has ended the memoration loop would seamlessly return now that we had a functional looping sound event implemented into dialog

system we now have the unique challenge to create a game variable that would relay that theoretical heart rate of the character to this looping sound at runtime this illustrates their tiredness pain exhaustion and fear or really any emotion that encapsulates what we needed to Telegraph to the player and not surprisingly we called that heart rate heart rate yeah so ultimately we needed a base way to express the tension of the game which manifested as the danger level for the player which and we chose to express this values of floating points ranging from 0.0 to 1.0 with zero being the most relaxed and one being the most intense the basic tensions of the last of this combat Loop is what's called ambient Explorer and this is where you know Joe and Ellie are having a light conversation maybe there's some Scavenging maybe there's some narrative lines that are playing um you know pretty low low energy and then there's stealth so stealth is

where the player is you know sneaking around enemies but the enemies uh do not yet have a full awareness of the player um so there's obviously a lot more tension than ambient in that in that sense and then combat of course is when the enemies have full acquisition they've acquired the player and are now actively engaged in combat actively trying to shoot Mall whatever this is a brutal game shoot Mall whatever you know uh so yeah uh with these three base AI Concepts we broke up the logic in that there were different heart rate states for for pressing L1 so running or not running in each of the three base tensions so if you'll get the you know the uh this is a kind of a simplified version of it but essentially you see like there's ambient high and low unaware low and high combat low and high and the purpose of that is high is basically it's that AI state but you are deciding to Sprint your sprinting and then low is you're not sprinting right

so you're either walking idle um things like that crouched so after we found a potential solution for the heart rate states we worked with gameplay programming to create a way for us to interpolate or slew the values that heart rate was sending to the audio middleware this was so we can get a more hand authored realistic sense of growing exhaustion or the opposite that recovery in the breast after having sprinted for a long period of time so here's a video to demonstrate heart rate active with debug to actually see what's what's going on

so as you can hear when when Ellie started to Sprint her target heart rate immediately snapped to a destination value and the current heart rate then slews up over time to to get to that destination uh target heart rate and so using current tart rate as a basis for driving all the sounds in the middleware that gave us the ability to add what we call you know stages to the breathing so for example we broke up the combat Sprint into buckets of three stages uh with stage one selling the least exhausted and stage three sounding the most exhausted so actually show of hands who's played part one like the recent part one maybe or just part two part two all right so I mean maybe you get it you get the whole you know combat Sprint the AI That's great uh so um

well now I'm like um so yeah uh yeah so they were broke up in the three buckets so yeah as you're running I'm I'm hoping you've kind of noticed that there is that tension you know Ellie and Joel when you're sprinting with with them you start to hear like oh my gosh they sound exhausted they sound beat um so that that was our effort right and that in hearing that really like glues you to the character you feel like you're in that headspace you know with the bullets whizzing by the dogs mauling whatever it is right so these stages and this is actually really important uh uh point was the all of this was recorded with those principal actors with the system in mind so with that what you get is this you know a very natural seamless representation of a growingly exhausted or decreasingly relaxed breath in the character's breathing cycle and so here's just a bunch of text for you to squint that so on the left is an image of the heart rate states and the settings sound designers were able to modify

the Delta value is the difference between the target heart rate values so I'm going to do my best here but so this is at 1.0 combat high so this is when you are sprinting in combat and the Delta for the growth duration just basically means you know if you're coming from uh combat low AKA you're not sprinting in combat the Delta would be 0.3 so I just you know I threw an 18. so over 18 seconds it's going to you know increase and hit that Target um so yeah uh so we were able to say and what's really cool about this is we were able to say like you know for a character like La maybe she's more light on her feet maybe she's you know done cardio more uh so let's make it so that she sounds more tired after a longer period of time but for Joel maybe you know maybe Joel hasn't been as strict with his cardio workouts right maybe he's drinking too much coffee uh so his his Sprint stages you know progress quicker AKA he sounds more tired quicker um this was a great setup for us to not

only get a feel for what sounded good but also to tell you a little bit more about the character you know it's a really interesting subtle way to add storytelling just to breathing you know um and to speak on the audio implementation side of image on the right is from our proprietary PlayStation audio middleware scream uh so what you're seeing is a small slice of what the looping player murmuration is doing under the hood so the audio executes from top to bottom and what happens first are you know what I call like the initial checks so it's just checking a bunch of game data um yes yeah so these checks are for variables like you know is the player injured is the current heart rate greater than or equal to 0.5 or 0.85 and so with these checks you know that's the way to Route the logic to go oh oh you are injured so go here and then play this bunch of logic that will Pepper in the injured sound so that's what's amazing about screams it's it's the scr is an acronym it's

scriptable it's a it's a essentially a scripting uh middleware which is really empowering um when you get a hand on it you know um and lastly is the uh sound execution stage where the middleware will will randomly pick a breath asset to play and this Matrix of logic on the audio middleware side is is the core of how we dynamically change stuff like you know the volume the frequency what assets to play and the overall mix of of the breathing contextually and here's it in action

you should have done more cardio yeah so yeah I I really hope I wished Troy was here so he could just sit here and watch himself you know breathing um it is hard to say apologies but I did you know try to you know uh superimpose of that text but essentially what's happening is all these curves all these values appear this is all game data where we are able to you know smooth and adjust and and basically you know alter the the breathing uh loops and assets to

Where It just fits in the context of the game so using variables like velocity current heart rate we were able to do things like one you know decrease subtle delays between breaths as heart rate increases which can you know suggest a more rapid breathing um we were able to slowly creep up volume and Pitch as heart rate increases suggesting larger exertions and we set a flag based on velocity to seamlessly change the idle breath to the walking uh breath buckets and this I mentioned before this is you know a a view of screen but this is uh specifically What's called the CC or continuous controller view for screen all right everyone's still with us all right so so now that we basically have solved all the technical blockers in our way right we now had a different challenge this brings us back to the main goal of the system right if you remember we said this will always be something to support the story and its characters

so how can this system support the narrative uh the game was trying to tell this brings us to arguably one of the more important features of the system which is the narrative impact so in a studio where story is absolutely King everything that we do in the audio department always comes back to a single concept how does it support the story from small drawer opens all the way to you know characters breathing to achieve control over this newfound system we added serp functions to do things like suppress the breathing Loop for a period of time so there were times where suppressing memoration helped the narrative and it kept combat tense we could be silencing stalkers during in big infected fights we could be silencing Tommy when he's playing cat and mouse with Abby during the sniper Showdown spoilers um there were key moments where silence basically just was the answer so in this next video example we're going to show Ellie is entering a new location this is in uh last was part one she's entering a new location where the player needs to feel like they basically

are no more threats they've achieved their previous combat area and they're moving on so to achieve this the ambient infected in the distance which you can't see we have we silence them until Ellie reaches a certain point the sound of the infected breathing Loops would turn on right before her body language and her efforts change basically to Telegraph to the player that there actually is danger nearby let's watch okay foreign

so two things um one if you as you heard when the infected get turned back on Ellie also has a big like right so right there we also um we also spiked the heart right there to her to give her that sense of like oh like there's danger nearby right at the very end we had a unique kind of problem so another feature at the end that we had was when Ellie is closing the door and she's invading the infected right so in this section once you close the door this whole entire previous level which is completely unload due to memory issues right something that I'm sure a lot of Us game developers are very used to so because of that if there were any infected still alive you would basically hear them abruptly just cut off right and that would completely break the suspension of disbelief so to fix this problem we were able to take the current breathing loop on any infected and seamlessly transfer it to a spawner in the world where we were able to place whatever we'd like to basically to allow for like a seamless handoff to other other objects like if they would unload so we would take that and in this specific situation

I took those and I just like lurped them those spawners towards the door to basically build tension almost like you're volumeing you know raising them up um and then as soon as she actually closes it then I do the same thing and I alert them away from it all the way far away so that they would Fade Away seamlessly right before we would basically just completely suppress them for good so this allowed a good storytelling you know idea of like oh okay you've closed the door maybe the infected now have tried to run away maybe they're going to try to find another way out but it allows them to fade out rather than to cut off in in that situation as soon as the door closes it's just those objects that are completely taking that sound and populating it onto a different spawner and then sending it away to be able to do that so another way we Elevate The Narrative is what we call spiking the heart rate which Jesse uh mentioned so we created a script function which snapped the current heart rate which basically basically was a knob for us to say you know sound scared or tired right now um and that leveraged the systemic

breathing assets that you know worked uh procedurally in a way where it just worked wonderfully in very Choice narrative gameplay beats so let's see an example of that could be a tad loud oh yeah yeah just a warning there's a bloater in it last I was too close sure but we [ __ ] did it

oh sure did you good I've been being really old ain't nothing solid not a sleep won't shake off wanna get out of here drink a lot of coffee coffee yeah so yeah this is an interesting one because um one of the cool details about spiking heart rate at the end there was um that tired breathing Loop was acting as this like really awesome interesting connective tissue between those narrative lines that were you know directed to sound tired like you just you know you just thought it bloater right so it was a really interesting a cool way to to web it all together and then uh lastly and this is one of my personal favorites we were able to override the um just the whole breathing Loop right so so far all we've talked about is like you know the base systemic breathing Loop that supports the tension

so ambient stealth combat right it's a lot of a lot of assets a lot of complication but you know what if we add you know a custom breathing Loop to a very specific uh narrative moment that displays a whole different Loop um and so yeah so this this let us play custom tailored assets and it allowed us to assemble uh a whole custom breathing set for key narrative moments pushing the storytelling through sound so here are some examples from part two where is that coming from Joel [ __ ]

laughs so in the first example the moment Ellie hears Joel getting tortured the murmur Loop changes completely to a custom breathing set and what's so impactful about this set is that not only is it emotionally charged but uh a little technical but it's kind of cool so the whole Maya X Y Zed is exposed as a global variable for sound and so as you descend go down the Chalet I was able to make it sound more you know like high tension more like it's like a climax

getting to that door of like fear and Terror and you know emotions so that was really cool um and then the second video uh yeah so I'm overriding her default breathing Loop there to custom breath assets that Ashley did for that cinematic so the breathing transitions seamlessly into the Cinematic dialogue stem and that was really cool and that was just a collaboration with uh you know the Cinematic team where hey do we have any Ashley breaths from The Cinematic that I can take and build like a a breathing set with that so that transition can happen because you know that moment's pretty quiet you know you just hear that that tone of the music with breath so it's just another cool way to add that that detail and that quality okay awesome so we now have like a hyper realistic breathing Simulator for an interactive player character so then we thought you know how can we expand this system into something other departments could Leverage

so we began branching out with an interdepartmental kind of mindset so why animate the player's facial expressions divorced from this robust and expansive like technology that we've all created right so our first collaborators were with animation so Keith passiello who was pioneered with the emotional facial system was pioneering the emotional facial system with Mike herahan Christian Worland and others um but they were incredibly excited at the pitch that we gave them that we gave them basically in regarding to a breathing solution so he was able to provide us with a myriad of custom facial animations to play in tandem with whatever breathing set we wanted for Abby uh basically Abby's vertigo featured in part two uh Choice Slough animated the shoulders so that you can hear like you can clearly see a sense of Abby's body like body tension like the terror of heights that she has um so we're going to show a video for this but for both examples like the

duration of breathing assets dictated these animations the sound was driving the animations uh basically ensuring perfect sync so we were in a way kind of animating like in a sense right so here's a video demo and then we'll talk about it laughs

oh it it's funny just showing that like yeah it happened but like the amount of time it took to for me to move the camera lock it and do that and look at him like oh my god it works this is unreal so yeah I mean this is a very brief description of how we made it happen but it's such an amazing collaboration um so yeah I mean you know I did my best to Overlay texts to give some insight but yeah it was really really cool really cool stuff you know working with programming like just as an example um you know the the fade of that uh facial animation was uh you know we

could input values of how much does the heart rate actually affect that animation you know because we can set it so all right as the heart rate gets higher um you know basically are you more exhausted right that facial animation could look more intense or her eyebrows are more down and she looks more beat um yeah it's really really amazing the next one you know yeah let's show the next one with Abby watch her shoulders and or not in particular

thank you okay so as we know you know we work on software essentially right so programming is the core of how games are made and their success or sorry and their support on these projects was Paramount for our success everything we've shown so far wouldn't be possible without their knowledge insights and critical additions to the tools and Technology so here's another thing you have to squint that sorry um so on Tila 2 programmers Jonathan Lanier and Eli omernik provided the audio team with features that radically altered how we implemented breathing assets and to go into a little bit more detail um in our audio middleware we had the

ability to send integer values to the engine that dictated one which facial animation to play of the ones that were supplied from animation and two the duration in milliseconds of the corresponding breath sound so in the image on the left we can see our breathing sounds paired with What's called the plug-in command grains um essentially a plug-in command plugging command grain is just sending game data back from the middleware um to to the game engine and the yeah so the image at the top you see there is uh the property view of this plug-in command crane so you can see into int1 into integer one expresses which animation to play in the array Or List defined in the image on the right this gave us the ability to actually cherry pick which facial animation went with which breath sound enabling us to essentially Define and polish the facial animation in tandem with our breathing work

um yeah so integer 2 expressed the time in milliseconds of the corresponding breath sound file and so I believe uh Eli was able to create a technology uh in uh that either stretched or compressed the facial animation per sound file so I believe um each uh breath animation facial animation was 30 frames everyone I believe was all 30 frames and so this you know this logic would essentially go oh it's you know if the the sound acid is longer than 30 frames then it will stretch or if it's shorter it will compress each corresponding facial animation in real time of course um which is an amazing amazing thing that that worked really well so yeah this feature working in concert with the breathing sound feature was the Pinnacle of our efforts um yeah so kind of in summary you know this is an amazing amazing thing that that was able to happen because of this team coming together and and you know working on

trying to you know push for something that we were really excited about as such a gratifying experience uh having all these different departments uh come together okay so a lot of information we're going to do a quick recap so basically what things that we've talked about right so we said the last of us really required like a robust breathing system to support a gritty realistic combat with vulnerable characters right so to achieve that we really ran into another greater need which was the technical need uh which was with our animation team adopting motion matching this birth the creation of two main pillars in our system which we call the breathing system and those were the Core Concepts that we called heart rate and murmuration the breathing system was used to impact the narrative in ways that we were never able to do before and then we started collaborating with other departments within our studio to spearhead features like facial animation that was driven by the breathing system

so at the end of the day our number one priority was to honor the characters and immerse the player in the story and give them an interactive experience that would leave a lasting impression for the players there was a lot of Pie in the Sky ideas for the system most of which were able to be fulfilled thanks really to the passion the you know creativity and the talent of the entire team so we'd like to give a little bit of special thanks for people that were able to you know work with work with us and help us on this none of this really would be uh possible without the passion and you know the just the creativity of the co-collaborators within you know within the company within naughty dog and be able to help with this so we just wanted to express our gratitude uh to some of those and that is it thank you so much for coming out thank you so we wanted to leave a good amount of time for uh q a because this is quite

this is quite a technical talk you know there's a lot of confusing potentially confusing stuff so please feel free to um yeah ask any questions yeah it was an incredible talk thank you so much um I had a question you were talking about with some of the more specialized breathing Loops um I was wondering how you account for kind of like player freedom in those situations like if someone in the Chalet went down to the basement and then turned around and went back upstairs would you maintain that momentum would you kind of roll it back how did you deal with the situations great question uh so in that case yeah it was just based on the the The Zed so the the um the um you know you know the height so yeah when you would go back up it would revert back to the less intense but that's a great point right like Maybe not maybe it shouldn't be like that you know there's yeah there's a lot of ways you can you can do that but awesome yeah I was just wondering what you guys chose to do with it yeah it's a good way to think about it because you can you know and in that way that way of thinking right is is a story way of thinking right because you're going down like well I've already heard those yells

and maybe I should stay you know really heightened maybe I should stay that way so you can change it to say no I want the character to feel this way and that's the right way to think because then you're thinking about the character and about how the player is perceiving it in that situation yes you know most people did not turn around and go back I don't know what it's a good thought a lot of it comes down the scope because you could do a thing where well if you stay at the door maybe after 20 seconds they have a whole new set of breathing for the I'm at the door tension you know breath assets like you know so you know this was already a lot obviously because we had other this was just incepted on part two like we we didn't this was not our main task this was just kind of added on top um and we got you know it snowballed like it was mentioned so we were definitely trying to manage the scope of this you know so awesome thank you so much yeah thank you thank you for the question hi thanks a lot awesome talk um I'd be curious to know uh how far you

kind of push things in terms of exposing game variables and having this kind of procedural approach um come from the animation side and there's a lot of other things happening there's footsteps kind of gear and pack and Ambience based on clothing grabbing objects uh did you expose more things or did you still put tags and animations for specific sounds Yeah so basically kind of the core of the question is you know were there animations that you would play kind of systemically that would talk to the breathing like picking up or crowd or like uh for example foot footsteps did you tag these in the animation day trades before or did you have say maybe like a a game variable which indicates when footsteps are occurring oh yeah so this is yeah what's interesting about the murmuration which is just the looping sound on the player persistent um and yeah it's literally just based on you know velocity or are you sprinting

or not sprinting we we didn't we didn't consider you know footsteps wrap the rapidness footsteps or anything like that yeah we had I mean I would say the things that we that we tagged that would cut off the breathing Loop would be things like efforts so like swinging you know a machete that would play like maybe a specific type of effort set or specific type type of efforts or a narrative line of course by narrative line going over or jumping over a vault or even death right all of that would have different effects on the breathing Loop and then bring the breathing Loop you know back when needed um but the cool thing about it is that if you were constantly running and you were constantly swinging your machete you would get all those efforts which are very like high energy but your heart rate would be going up as you did that so then your breathing Loop would come back in at a higher like at a higher set possibly so it would complement the efforts which was really one of the main goals especially when we moved on to uh the the remake of the first game because we didn't have any of those assets I'm going to re-record everything so that's why we had things like injured States and stuff like that and we re-recorded uh Troy with all that but yeah for as in like um for tags and things like that

that's not something that we did yeah no yeah that's cool yeah I'd really love to see this kind of procedural approach applied to more things like I don't know how heavy your clothes are or you know how figure boots are and what material they are and how all of these things feed in them so I mean actually you know what was not talked about was you know on part two of experimenting with um a bunch of stuff like uh when you Vault like Spike the heart rate every time you evolved or Spike the heart rate every time you get out of water like the water after swimming but procedurally it was like the nudge value was was the name of it and so we were experimenting with a lot of stuff but in the end when the game shipped all those were just zero zero zero like we you know no no Delta was applied like just didn't really work great you know and and yeah it's just about hearing and feeling and if it feels like oh that felt that in that context that felt weird but that's always going to happen right in in when you get out of water I'd it's a weird it's a weird sensation of

hearing that especially you know to account for oh are you in combat when you're getting out of the water are you in stealth when you're getting out of the water so we we actually found like the more simple approach the better we're gonna have a whole Suite of sounds for prone like when you're prone oh what does it sound like maybe the mouth is close to the grass or the dirt oh scope right like like and the bank for the buck that's that's my big thing is bang for the buck like is is there value in doing this are we just doing it just to do it or is it going to have like a res is it gonna make you know the player feel something is it going to add something of value right yeah thanks a lot yeah thank you thanks hello great talk by the way thank you I was wondering when you said that like uh each heart rate had its own Delta to like move back down between states if the player changes uh like target heart rates in between thresholds how do you decide what Delta to use um yeah well it takes your current heart

rate so yeah if it was going up slewing up from 0.7 and so it was at like point you know seven six then at that moment when you go to a new destination that's your new Delta 0.76 and 0.4 or whatever that is oh so you just okay because you talked about like how if you know Joel has like a little more hard time doing cardio exactly yeah so in that yeah in that in that instance right we had different um heart rate like basically sets for them like for specifically for that character for when you set those Deltas we would set that we could set them different for different characters so um a good example of this in uh part one with um Ellie right we had all of Ellie's like data from being a player right and she is a player at some point so what I had to do is I had to create two separate completely breathing sets and I created two separate breathing um like characteristics for her so I made her basically recover from certain types like combat when she's like we could you know from because she's younger so like she could recover maybe

a little quicker and we made Joel recover a little bit later those are completely different like breathing tables uh breathing set so their Deltas are set um for them specifically so that was nice because I was able to get a toggle that said like oh is she a buddy or is she a player and we could actually affect them differently so when she's a player she also has way more detailed in her breathing system she has catches like so when she's really exhausted and she's trying to catch her breath oops she's trying to catch her breath and things like that is way more there but we found that if I had that breathing set in while she was a buddy it was very distracting so right because at that moment who are you you're Joel so you're walking around and you have your breathing set and you have little Miss Ellie running around all over the place like she shouldn't be but that's just what she did and then she was exhausted so she was like you know breathing heavy and was like what is going on so I had to set a specific breathing Loop for that with completely different characteristics so even had a situation where she's laying on the operating table and she was breathing heavy because she's in combat so we had to suppress that for the rest of the game almost because uh you walk in as Joel

and Ellie's unconscious going yeah very problematic so so yeah thank you yeah thank you it's a fun thing about game development yeah I think I'm part two because animation will override the tension because it's an easy way to like all right now that's the combat set for animation so like what was it like that it was like a top gun game with like Abby and what's that one oh the um oh yeah yeah the um the like Nerf gun was it Nerf gun yeah so they had like you know they were it was set to like combat or underwear it was like such a combat I think because they were technically in a combat State yeah so then the breathing would like follow that and it would work and it's like gee it sounds like she's gonna kill somebody she's just playing look at the Nerf we had that in uh in part one too with the water gun in uh and uh you know the water gun fight they were running around and it really sounded like they were like you know really trying to fight but I had to make it more playful so that was a whole separate radio for that so then the programmer added this cool thing that was basically like a like lock the AI tension so it could only adjust within like a boundary which is which is really

cool yeah sorry go ahead ice no problem it was an amazing talk learned so much from this my question actually was about the accessibility features that were built in for part two and factoring in the closer we get to more realistic audio how are we compensating for that for people that are hard of hearing um as we like increase the amount of sounds that are gonna be surrounding the player how do we still enable players with that um lack of ability to hear distance and how did you guys calibrate that for part two yeah specifically for the breathing stuff just yeah the breathing and the heart rate changes um because they are a lot louder for the player um I'm just curious if you guys have made any adjustments for the because the accessibility features were huge in part two yeah yeah I know for the breathing system I was a really good question I can't say that we did that I I know of what we did for accessibility that was a different person who worked on accessibility part but um well for breathing specifically for breathing specifically yeah for breathing specifically so um I'm not sure I don't know the answer to that one I mean I don't know if you have yeah I mean just

a spiff on like you know captions heavier breathing like is that kind of a suggestion sorry or like maybe it's a good idea I mean I think my question might have been a little confusing um I actually meant more of hard of seeing where players are relying on so much sound in their environment where the latter we get around the player yeah right now they're having more difficulty actually being able to understand where the sound is coming from right so I'm curious to see how you guys adapt to that system and having to still be accessible no that's that's great yeah yeah I mean on part part two definitely you know a lot of folks were uh talking about and I agreed and I wish we added this but you know an infected slider or basically a toggle to to kind of tame the infected because you know the clicker frenzy was is it many people it doesn't sit well with them like it's painful it physically hurts them to hear that kind of quality of sound really high frequency piercing grading and yeah like that was I think that was tossed

around but I'm not sure I don't think it happened unfortunately but that's a great question yeah um learning opportunity for everyone yeah exactly and yes so yeah totally possible I could I could totally see yeah like a breathing like maybe don't have it as maybe loud or as prominent or as as a constant I'm not sure but you know something to think about for sure it's a good one thanks that's what's cool about it is like we're all we are always thinking about it and there's so you know coming coming off of God or Ragnarok like we did a lot of you know looking at the industry and going what can we do how can we be better and I think every game there's so many opportunities to be better especially for sound you guys did an amazing job it was a very impressive game audio wise so awesome thank you so much hi um when it comes to uh you mentioned the uh compression and stretching of animations based on the length of the sound for the breathing was that applied to other things or just the facial animations I

was also applied I mean it was kind of so it was applied to a lot of things um like too many animations yeah yeah so it was applied to other types of animations not just facial animations like we showed the example with Abby with her shoulders her entire shoulder set yeah for the um for the the baby which was very like last minute thing because we were trying to figure out you know with depending on the type of content you have like babies can be really hard to like put it put together um so we actually got specific animations tied to the specific audio that we made little things like okay if the baby laughs like we can have partials where like it'll like lean back as Ellie's holding the baby and will throw its arm up like this or maybe it'll throw both arms up before the left arm up so I had the exact same system being able to choose okay well which animation are we gonna play and then if it if the baby I had sneezes in there as well which is part of the breathing Loop the baby would be cooing or whichever and then when the baby would sneeze it would play that asset the plugging grain would then tell the game like okay play has played this one out of three sneezing animations and it needs to be this long so it would literally animate

the baby to sneeze and it would sneeze or it would cough you know this like disgusting child cough and it would cough you know um so yeah we're actually able to build custom animations based on the sound for that thing and the whole baby moving around and coughing and sneezing all the right time is all based off of of that system so it's cool because you can kind of do a lot of things with it other than just the facial animation you're making me think though that sounds awesome uh thank you yeah thank you what other animations oh we could do a lot stretch based on sound speaking my language oh how's it going yeah um so you have your murmur system which is just a loop that's going on the entire time which I'm assuming isn't being synced with with uh facial animations in breaths or maybe it is but um I'm wondering so like you showed later there were specific breaths that were um that were being in that were informing the animation and they were specific lengths for those breaths I was

wondering um how is if there's anything that you guys need to do as audio guys or if you just kind of send those variables over animation they just handle it does that make sense so um the first part I believe of the question when you were saying like with the murmuration loop is persisting like you know are there facial animations tied to all that so every individual breath that happens in any bucket can be you know hundreds of breaths has a corresponding uh plug and grain for facial animation that's crazy so every single one has one and we had to enter actually uh Grayson stone is right there he pain secondly entered every individual millisecond number for every individual which we are trying to streamline better but that's how it was and that's how we did it for the next one but yeah he helped out with basically focusing mouth open mouth closed and that so it is for everyone so it's not just like a like a loop a straight up Loop that you're just playing it's literally individual yeah yeah one shot I think I mentioned that but yeah each each breath asset is an exhale into an inhale gotcha okay together in one and we found that that

because it's funny like I'm actually Consulting a lot with other uh folks doing a breathing system and there's so many ways to do it you know like and I I just personally with especially with the night dog engine and the dialogue it's kind of makes sense to do an exhale bending inhale because it exhale is almost like an effort or a line for sure yeah when you're speaking or efforting your ex you're giving your breath and then you're inhaling yeah so when it we're stitching it all together it becomes more in line and becomes more seamless and cohesive yeah basically each each individual like state or each individual type of breath table within that one main Loop that's always on yeah it's constantly checking uh variables different states heart rates things like that and it's picking the inhale the exhale the inhale the EXO from all different buckets that's why you get a lot of good like variety um on it and it's it's like I mean some of the individual what one like oh just stealth breath or just combat breath maybe like a 350 line screen script that's bouncing back and forth between that and another 350 line screen script

just choosing it's it's messy I would think I would think of like just the looping sound as a just a nest of logic yeah it's just a lot of deep logic instead of yeah because usually like traditionally you know me before doing all this work for part two I would have a loop is a sound that has a zero Crossing and it Loops yeah but with Scream It's just nested structures of logic and it gets so dense and you know it's like that the systemic stuff is just one Loop but yeah we can play so many different variations of of sounds and based on all this data so yeah so sorry I don't want to take up too much time so you have like random containers filled with like this level of effort and then that's the loop that's cycling between those randomly so it plays whatever okay Waits For The Voice to finish yeah then goes back up and does a bunch of checks and then it's like oh I'm injured I'm gonna go down here right and based on like your velocity possibly of running or things like that it'll actually change a random delay grain to delay the breath

between each other so if you're more exhausted that delay can be less so that that breath it's like you're you know rather than yeah so that randomness of that delay based on the the height of the heart rate will actually allow those those like the space between the breaths to give a whole different set yeah that's yeah so cool really cool thank you so much yeah thank you thanks for the question all right it's really uh cool what you guys built I had a question regarding the animations driven by sound one of the things that is kind of uh interesting animation system is like priority based like who's driving the animation and who takes precedence how did you guys resolve that with regards to the sounds that are driving facial animation versus the dialogue system potentially coming in or other system coming in same with any of the other kind of full body or shoulder yeah I mean I think the uh the so the whole murmuration Loop was pretty bulletproof in that it will it will always wait for a voice and you will never hear overlapping breaths that was pretty much impossible from what I've seen

um and so there was no issue with the facial animations that were driven by the breathing stuff but of course yes you know there's going to be the lip uh systemic lip flap for uh you know gameplay lines and all the uh you know narrative lines and I'm not exactly sure but I assume that you know the dialogue system stopped the breathing Loop and then that lip flap took over or that other animation took over so I think I think it all just kind of worked where things would Trump in line with our Vox priority stuff like so the animations would also work in that way and a lot of times we'd have efforts with certain animations like jumping over something or swinging whatever I was saying before and they would have their own it's it's a tone animation its own facial animation so that would take priority and that would play and then it would go back to our breathing loop with our animations and it was pretty seamless um the way that would go but also it helps that it's more difficult to turn the camera around it's not perfect yeah so you know uh you're typically the

camera's kind of behind you so it was like when that would switch it would feel kind of seamless because you weren't necessarily looking right at the player's face so you know it's not like I said it's not perfect but uh it did give that that sense uh it switched I think between it pretty seamlessly cool thank you yeah thank you all right hey uh I came in a little late to the talk but I caught the end and it sounds very uh cool and it's like wow I didn't realize there was like a whole breathing system um I myself am visually impaired and I have friends that are completely blind so I want to First commemorate you and the rest of the audio team on like if you're doing such a good job at a completely blind person can play the game and especially with the audio descripted cut scenes in part one like I've never seen any game do that so that's super cool um and I also wanted to know like was it easy to how difficult was it to preview like the like you know preview and QA

tests like the like how well all the different breathing Loops were because it sounds like such a like a nitpicky thing to be like okay we have to go on the scene like like you're like Run the game and like okay we have to wait for it to run and you know like how how complex was that I'm not gonna probably say this one uh I mean it all started with um Ashley Johnson's work on the 2018 E3 demo where she was going through the the forest and then you meet the Sarah fights for the first time the gutting all that that whole E3 demo she did a basically a run of this that whole moment like linearly she just you know was in the booth and did a brief like breathing exertions efforts to that whole thing and I was able to take all that content and a couple additional stuff and just kind of figure it all out like all right with what I have how can I make this work the whole the whole breathing stuff right the whole system and then with that we were able to go all right we like we know we now know

what we need like exactly so then we were able to go with Laura Bailey all right this is what we need like no questions we we know because we already built it with Ashley's set right so that when recording actually that was interesting because we just needed like a little bit more you know because we already had so much from her we already built most of what you heard um before doing a proper breathing session um I think it was that answer your question um I I guess I'm coming from the point of uh so I do sound for I do sound for games and uh like a lot of times like when tying stuff to animations it can be very difficult when the engine I know you guys got your own engine so I'm sure it's like easier but like for example like with unity it's like you gotta it's like okay the animation event is here now to play now to actually see this I have to run it first I have to run the game and then I have to do the animation and then like oh no it's not off all right pause it like stop it move it over oh sorry it's almost like an iteration process yeah so so how difficult was

that not too bad yeah so Scream the proprietary middleware um has a lot of really great features so you basically like restart script you know if you can connect to the game be able to adjust your logic I believe you can adjust the sounds and you can hot reload Banks so yeah it'll just immediately update kind of in the game as you edit whether CC sounds things like that um also I mean the iteration process of choosing choosing breaths you know takes a little bit of time because you're trying it's all about the feel of being able to get it right but yeah um it does handle it pretty well trying to like iterate the process of iterating is that I'm not sure if that answers yeah that answer that answers and now I have to figure out how to recreate that and why it's cool yeah and thank you yeah thank you so much maybe we have time for one more yeah perfect yeah look at that that's me it's you hey um so did you guys run into any issues with the breathing animation being restricted to 30 frames with the randomness that you were um intending for what the boyfriends

or 30 frames I don't think so oh I thought that's what you said 30 frames oh no no yeah I know restrictions with them being 30 frames long oh yeah I don't think we with the randomness because I feel like I mean 30 frames was that oh I see it yeah um right with the amount of time yeah what was interesting is yeah it was literally like one breath it wasn't like oh this is variation number two or four of this one breath for the animation it was one base animation um and then you know after after a while I asked Keith for oh can you give me like a different kind of exhausted so there was sort of variations of the exhausted but I think for the most part because of that whole uh setup that the programmers were able to do where it was you know compress or stretcher or whatever that 30 frame animation it's just hard to see the saminess like it just a lot of the audio felt very varied like it was you know it felt varied so the facial just kind of fell in line and you didn't I don't know like the association I didn't look at it and go oh yeah this

feels samey or uninteresting or hmm because it seemed a little choppy and it wasn't it wasn't like a you know the criticism why there was kind of and and hearing 30 frames the other questions it's definitely not like it's it really helps cells her nose and not through her mouth around or was able to see the players it really helped with that stuff if you wait it's moving I never noticed um yeah I mean um I think that

is more and more to see if we but yeah I mean a lot specific things like we have this enemy and you're in like um um stealth that the player is breathing if so it's like you're being to the enemy so in that situation the whole thing is to sell like so as soon as you and she'll be so there was certain Tech created for that so it made her mouth will be in the correct yeah but it's it's a what we're trying to do yeah of course yeah thank you all right one more let me

do one more one more okay very cool well thank you all for coming out yeah we'll quickly answer them real quick let's do it then we'll go the last one sorry uh I have a question if you have any problems with the transition transitions between the loop system and the individual uh efforts like I don't know Heating jumpings and so on because sometimes it sounds if each individual sound Instagram has an answer and so I believe the real I think we set the release to be pretty you know yeah it's long long-ish so yeah you never get anything super jarring when a sound would stop and yeah the only time the the loop would actually aggressively get cut off was when you were sprinting or not sprinting which is why this whole catch idea happened where like right when you let go without having like a without having that it felt really like

oh it's like digitally just stopped right it felt fake so we had to like fake the realness by adding something like organic like like a catch the moment you let go to kind of blend it all but yeah okay thanks thank you thank you thank you