'Moss': Design Before Sound

[Music] hi everybody my name is steven hoddy i'm responsible for audio at polyarc this is just a reminder to silence your cell phones and to fill out your evaluations afterwards that's super important to gdc as a whole and so rehearsals have been going straight up to the 30 minute mark so it's likely that i'm not going to have time to answer questions afterwards but we can hang out in the hall or go get a wrap-up room one of the cas is going to help us with that in a little bit so um finally uh things seem to be sounding pretty good in here but if your listening experience is substandard in any way you can go play moss at the igf pavilion so uh jason graves uh did a great talk yesterday um it was awesome his music is incredible if he's played moss so uh you should go check it out on the vault sort of a companion talk to this one so and finally um if you're not familiar with the

technical jargon in this talk uh just don't worry about it this is fundamentally a talk about ideas it's a talk about finding the heart of what you're working on and thinking about design before sound this is about the perspective that emerges when you've been on one creative path for a really long time [Music] [Music] and then suddenly you're here with a mouse in a new world starting over from the beginning this is about what makes you a sound designer so what makes you a sound designer is it knowledge of plugins that makes you a sound designer no recording engineers know about

plugins is it knowledge and use of synthesis no musicians also have this down how about foley performance how about systems design or implementation or scripting the game part is coming into focus but it's somehow not enough is it your ability to craft supremely realistic weapon lods are we any closer the answer to what makes you a sound designer is in the portion of the name that gets the least attention sound designer all that other stuff plugins editing scripting it's all craft that you implement in the service of design so unpacking the term designer is a bit of a trek so let's zoom out for a second if we're doing our jobs well player experience is continuous and holistic the sum total of the many choices and details of development wash over the player bathing them in a kind of metaphorical light our job as designers is to deeply understand the light of player

experience through the prism of design intention this prism refracts the player's complex holistic experience into discrete components in service of the game's design so the application of sound into a game is the expression of this design intention so you might say that compelling sound design starts with design and ends with sound sound design starts with design and ends with sound so before you cut the first sound you're thinking about design and when you're thinking about design you're thinking about player experience you're wondering about what impact you're making on the player at a fundamental human level which becomes their reason for playing the game but early on these reasons may not be entirely clear and each player has their own reason for playing so you must listen carefully for that central design intention to emerge during development so you're still not thinking about sound yet you're thinking about human impact

and while the game is coming together there's a circular process where you're embodying player experience what is the game telling you not just on a moment-to-moment basis but after you put the controller down what does this even look like as an example i'll tell you the story of my first player experience with moss the first time i met quill was in november of 2016. during my first visit to polyarc i played a very early moss prototype that a few founding members had built over 12 weeks so i put the headset on and i saw a cute mouse which was pretty cool so far and then she looked at me which was surprisingly compelling and then i pushed the joystick to move quill around and started to sense how this could be a game i reached out grabbed her and felt her heartbeat through the controller's vibration [Music] and that's when everything changed i

caught the first glimpse of what made moss special i felt something like genuine concern for this little vr mouse like i wanted to protect her this experience felt wholly unique in all my decades of game playing over the next few days i wondered what would happen if my relationship with quill was central to the game's design what would it be like for a game to condition players to feel empathy and a sense of protection for a character if we had more games that focus on powerful beings protecting weaker beings what would this world be like and is there anything this world needs more than the cultivation of this sense of connection a simple mechanic grabbing quill with the controller and feeling haptic vibration generated a ton of complex questions tam armstrong polyarc studio director often says a game is a function of its inputs and here moss was asking me to reach out and to bond with another being reduced down to its essence the human

impact was enabled by one thing an emotional bond with quill so you see a design intention began to emerge but it didn't solidify magically overnight these questions continued to persist throughout development and three months after i joined polyarc we started showing a new demo to players and the results we observed confirm our intuitions it's sort of hard to gather what's going on here but off-screen the player's friends are coaxing her to continue on with the demo but she's content just sitting at the dock petting quill she's so happy that she wipes tears from herself [Laughter] so we understood that the emotional bond with quill the trigger for empathy protection and connection was what made moss special to players so let's check in we understand the game's central design intention

initiated by our own intuition and more importantly solidified by observing player reactions we can use this intention the focus on quill's emotional bond as the prism i referred to earlier if compelling sound design starts with design and ends with sound this brings us to the expression of the game's design using sound so how do we support this design intention the emotional bond with sound first we must clear the way with comfort we all understand intuitively that certain emotions blot out others like how it's hard to feel compassion when you're tired and hungry vr is wrestling with its own version of hangry which is vr related discomfort and fatigue poor frame rate resolution weight of the headset and the disconnection between player body movement and camera movement are some of the more prominent examples that induce discomfort in players if players are comfortable we create a space where other subtler emotions can

arise if players are agitated the beauty of the world of moss recedes players stop caring about quill and at its worst they quit the game because they feel they are physically unable to continue danny bola polyarc's design director often says vr is an amplifier and what he means by that is that we have to take care that whatever we put into the amplifier is a clean signal subtle discomforts that may not manifest meaningfully in console games in vr may cause players to take the headset off sound can absolutely play a role in contributing to vr fatigue both through physical discomfort like listening fatigue caused by loud sounds and psychological discomfort caused by intense or grating sound a great advantage of the current generation of vr headsets is that players use headphones which allows us to take advantage of actual high dynamic range in moss the range between the quietest moment and the loudness moment is

somewhere in the vicinity of 45 loudness units relative full scale which means the game can be quiet most of the time and loud only when it needs to be the sounds themselves contain very gentle high frequency content and are offered with careful use of limiting distortion multi-band compression and sub-harmonic generation so even if players are wearing a pair of headphones that hype or exaggerate high-end frequencies they're still going to have a comfortable listening experience so we covered discomfort related to the physical sensation of hearing however even sounds that conform to a gentle aesthetic can produce a grading effect during one playtest players reported fatigue because of howling cave wins during one section of gameplay they mentioned that the low persistent rumbling frequencies created a sense of claustrophobia and unease to combat this effect we use low frequency oscillators or lfos as most of you know

an lfo is a repeating figure that can be used to control or modulate the volume filtering or other properties of the sound and that by tying the value of sound properties to these moving signals we can maintain the initial moodiness of the scene but that feeling diminishes over time as the lfo value changes multiple lfos set to different rates and amplitudes can sum together to create more variable textures of modulation overlapping lfos can be used on any ambient bed with discrete layers to increase movement and to create an undulating mosaic of unpredictable textures reducing perceived repetition and providing relief from psychologically intense sound so we're on our way to supporting the design of the player's emotional bond with quell now that the player is listening comfortably we are free to focus our efforts on the world the quill inhabits let's engage design before sound for another moment

immersion reinforces the player fantasy of the virtual reality when the virtual reality is sufficiently convincing actual reality falls away and players have a chance to be along with quill in her world and moss players manipulate objects to help quill progress if these interactions are sufficiently detailed and responsive and the weight feels right in the player's hand then it will increase their sense of immersion the blue orb here represents the player's hand as tracked by a controller as it reaches into the world and manipulates objects [Music] [Music] [Applause] small delicate sounds that might not

read in conventional games end up carrying a lot of the responsibility for selling this fantasy like the crackling glue of the binding in this book [Music] or the drifting ratchet of this spindle as it glides to rest [Music] the effect of the player fantasy extends beyond the interaction and into the realm of environmental ambience in reality much of what we attribute to ambience is perceived as the as noise that the brain filters out so it's often not enough to take a single nature recording and use that as the basis for an environment to cross over the barrier between the passive neutral experience of sound and

into a realm where the player actively registers the emotion of the world we must saturate the listening environment with story and narrative context here's an example starting with a few sounds over our logos you arrive in a library delivered from another realm a clock ticks slowly in the distance and you sense that time works differently here you hear the gentle scratch of pen on paper emanating from the books piled on the tables and lining the shelves as those stories are emanating from or as those stories are forming on their blank pages [Music] opening the book it calls to you and you feel a force tunneling through the eons having waited and found you at long last

here's another example the bright forest pulsates with the life of childhood adventure a time without worry possessed only by discovery in a search for freedom birdsong pours in from every register squirrels chase each other their claws scratching the bark squeaking taunts tree branches creak and grown in the wind a beetle's wing clips your ear as it comes by [Applause] finally one of my favorite scenes it's night time now and the wind whips

through the pines unseen creatures rustle and howl in the distance you're not supposed to be outside the hut at night but the fear of what might happen if you stay drives you farther and farther away from safety so so you can see how sound covers mundane things like time of day weather biophony and geophony it tells you where you came from where you are and where you're going but it also triggers less precise phenomena like memory nostalgia joy peace mystery

and fear quill and the player characters journey together the experience that forges their bond is reflected in the emotion embedded in the world of sound but the process of getting from room to room is just as important as what the player finds once they're inside in theater there's a problem with changing scenes the curtain drops or the audience sees the vague outlines of the set moving around they're reminded of where they are in a theater not the world played out on a stage the spell is broken the video game version of dropping the curtain is loading transitions it's easy to miss what's coming up in this next example but blanket ambience or the environmental sound that does not correspond directly to level geometry isn't tied to camera position within a volume rather each room is assigned ambience metadata which is prefetched and positioned around the camera before the room even loads

[Music] seamless continuity of sound over hard load transitions creates a feeling in the player of having teleported and arrived accompanied by an imprecise feeling of the passage of a small amount of time where quill travels from one threshold to the next so these environmental sounds we just covered the player interactions we referenced earlier and all other in-world sound needs to interact with the geometry of the world in some way reverb is like lighting for sound through subtle shifts and technique it can take a simple scene and imbue it with complex emotion and like lighting reverb design is its own rich craft worthy of attention some reverbs provide a sense of endlessness others provoke claustrophobia and so reverb is not purely a functional description of acoustical space moss uses a little over 20 different

reverbs across the span of about 75 rooms and each reverb has three layers a convolution layer a traditional algorithmic layer and an early reflection layer these were designed in a digital audio workstation mixed together swept with a sine wave resampled and deconvolved back into the engine back into a single impulse response for use in real time in the engine so the room we are in may be to reverb reverberant to adequately hear the subtleties in this clip however try to listen to how quills sword and voice bounce around each space see if you can track not only your perception of the space but the emotional load that accompanies this change and just so you can calibrate your ear the first clip is dry so it doesn't have any reverb and the subsequent clips feature reverbs from different environments foreign

[Applause] [Music] with the player comfortable and immersed they are free to establish an emotional connection with quill let's explore another idea that starts with design and ends with sound one one of the ways that we connect with each other is by sharing our perspective or empathizing with each other moss is a self-co-op game where players simultaneously control a third person character quill and operate as a distinct character in the world if we only chose sounds that operated at a scale that reflected the player character's point of view and focused on reinforcing the reality of scale what might this flaming door sound like so use your imagination and focus on the reality of scale

so from quill's point of view it sounds like this door might feel a bit larger and more intimidating choosing sounds with more mass and weight doesn't simply add a layer of listener satisfaction it superimposes the player's perspective onto quills so players can hear the world through her ears you might call this empathetic sound design her mixed perspective the larger sounds also provoke a sense of danger elevating the player's protector instinct [Music] so

you guys see the screen okay okay these larger sounds also provoke a sense of danger elevating the player's protector instinct so the player is comfortable and immersed and has a sense of quill's experience how quill expresses herself in game will clearly influence the formation of the player's emotional bond with her should quill speak a language should she deliver story what about sinless gibberish or exertions should she remain completely silent how mouse-like should her vocalizations be how female or male what's the right decision here understanding this problem through the prism of the player bond can yield a lot of compelling solutions but what if the player what if we want the player to determine who quill is

or what she may be thinking that starts to narrow our options into the range of content requirements that call for ambiguity in this way the players become equal participants in the construction of her inner life and can project onto her their own interpretation of what she may be thinking or feeling having a human perform these vocalizations also gives us more access to a wider array of expression and although relationships between humans and animals can be deep and complex relationships between humans start with an assumption for the shared capacity to deeply understand each other's life experience so we knew we were successful with this fluid approach when we observed some players giving quill a unique identity and interpreting her actions in ways we couldn't have ever anticipated or planned so the player is comfortable and immersed and has a sense of the stakes and they can construct an ideal version of quill in their own minds can you see how each new approach to a

problem reinforces the goal of establishing a bond between the player and quill the fluid approach is only part of the picture quill is not a total blank slate we thought players would enjoy learning quill's backstory and growing with her over the course of the journey the story of moss serves a few purposes and mechanically propels both player and quill from room to room and it also seeds the gaps between story moments with emotional resonance that players can use as the basis for their own construction of quills in her life we found that the fluid version of quill coexisted comfortably alongside story narration so there's a field of established options to draw from with narrator direction like curmudgeonly but loving grandpa or ethereal otherworldly being or plain spoken albeit earnest protagonist but what about your mom if the narrator establishes the world in a warm safe inviting way players would

be inclined to connect with quill i don't know if you can see this okay but this is actually that's me down there we worked with voice actor morla gorandona our narrator over the course of several theater style workshopping sessions to find a sweet spot somewhere in between a maternal bed bedtime story voice and an omniscient being we hope the maternal style might activate some familiar feelings like safety rapt attention and wonder marla had about six years of practice in this role reading to her son each night so starting with design looking through the prism of the player bond approaching each new small problem with this prism how would you approach microphone selection to support the audible perspective that occurs during bedtime stories we chose a large diaphragm microphone which is probably no surprise but the large diaphragm captures the richness of her

voice as opposed to a shotgun microphone which has a more hyped and aggressive tone we positioned her close enough to the mic to achieve a sense of intimacy without the player perceiving a two-dimensional inside my head kind of sound you'll hear a subtle amount of reverb applied to the voice and some story moments to place her in the room with the player quill's hands trembled as she showed him the glass what is it she asked a look of panic spilled over her uncle's stoic exterior where did you find this roared uncle argus quill had never seen her uncle so shaken hey kid over here i know where your uncle went i can show you you're twofold now very important bring that glass and your sidekick too

you're going to need them quill called out hey starving wait so the stack of techniques used to support our design grows even higher what about sandbox sound effects because this is a game after all as sound designers it's likely you're always listening as an extension of your craft you hear a sound that affects you in an interesting way and you take note of it and file it away in your brain if you're lucky you have access to a recorder that you carry with you so you can document these moments in the winter of 2012 i decided to document my sister's pregnancy with her first son isaac i accompanied her to an ultrasound appointment where i heard this very sound in real time for the first time i was moved powerfully by that sound

there's something about it the swishing rush fluid and pumping valves it triggers a protector instinct one that is crucial to the formation of a bond but even though the sonogram technology is relatively new it reaches deep down into some evolutionary drive into a synop when it came time for us to explore quill's low health indication a fundamentally protective game mechanic this recording surfaced in my memory unfortunately as you heard isaac's sonogram recording had room tone reflections ambient conversation and speaker distortion and built into it but thankfully you can get just about anything on amazon like this pocket fetal doppler monitor which this one even has a headphone output jack so whenever quill is in trouble and needs your help you hear this

[Music] [Music] this is one example of emotional sound but it requires you to take something real and authentic from your life and to give it to the game these are really hard opportunities to find but they do exist so let's put it all together the player is comfortable they are immersed they have a sense of the danger quill is in and have access to her point of view they are co-authors of a deep relationship and are primed for the bond through story and the sound effects are sending emotional messages to the player the good news is that there are other examples of other choices that support the emotional bond we didn't even talk about music or spatialization techniques or foley when you're making choices looking through the prism of design intention these small even innocuous details sum

together and create a powerful cohesive player experience compelling sound design starts with design and ends with sound if this type of abstract human-oriented approach to sound feels awkward or weird i promise it's worth the discomfort and confusion because it's important to remember that people are giving precious moments from their lives over to the experiences that we make so we have to find ideas worthy of that gift and to fill them with sound thank you all for listening i hope you found this valuable