Game VR/AR: Paper Beast: Technical Audio in a Simulated VR Ecosystem

[Music] hello I'm c'mon you can technically do designer and audio programmer at Pixar if and I will be talking about the audio systems for our VR game paper beast I'll be addressing several different topics ranging from simulation and physics to audio based synchronization and the art specialization if you're not familiar with the game paper Beast is a VR adventure game where you accidentally thrown into a poetic world in which new life is born from big data and you will explore this world and slowly discover and interact with the Tico system here's a short trailer to get a better idea [Music]

[Music] okay so let's go now into the details of what we've seen the game relies on a custom physics engine and everything is physics driven creatures plants terrain weather the creatures have procedural locomotion this means their movements are not written in advance but calculated in real time they can slide if they are sanding slippery slopes they

can be carried away by river streams or struggle if their limp is caught by a predator the terrain is handled with a fluid simulation water and sand flow dynamic lead as erosion creature lived attracts they can dig and create small dams the weather is dynamic as well you can have rain sandstorms and wind affecting the physics it ranges from a small breeze making plants rustle to a big storm blowing creatures away and flooding the land and all these elements can be interacted with you can grab the creatures leave them in the air had them cross obstacles you also have tools to manipulate the terrain and whether directly or indirectly all these elements and features had many implications and were the starting point of the audio tech design paper beasts being a vr game with emotion at its core in an unfamiliar setting we had to put a lot of energy to make sure runtime of your behavior was accurate I'm in synchronization specialization

propagation there were all key elements for making the player truly believe in this universe no matter its weirdness then does this highly dynamic game context where situations get very unpredictable and granular with many moving parts this means audio has to be very dynamic as well and react accurately to every possible situation even when it gets a bit chaotic and on top of that does the VR medium which gives the player much more focus and discernment about what they're hearing with 360 motion head tracking and we felt this really raised the bar for interactive audio so because of all these things sound has been an early focus in the development and indeed the audio team was quite large compared to similar sized projects for 12 people into the development team three worked exclusively on audio so that's a quarter we had fryin portion she's a radio artist who specializes in pseudo natural sounds she is fully into original song creation we had Roley

Potter for the original soundtrack British composer of electronic ambient music they both brought a very fresh vision as it was both the first time they worked for videogames and finally myself I am a sound designer and shisha programmer and I took care of the audio tech from integration to system design and coding we worked with unity and wise from a Geo kinetic which was a big help to get many advanced audio features we also used the binaural audio features of the psvr as it was our main target platform will now go through the major audio features of paper beasts and we use them to discuss the more general strategies and lessons we learned about interactive audio and vr first we'll talk about the creatures sounds and more globally how we handle the physics engines audio then we'll go with the vocalizations and how we used audio to drive animation and other things throughout the projects and finally we'll talk about the environments of paper beasts and how we handled VR specialization let's start with the

creatures here are a few of the creatures inside paper beasts as you can see we have many very different creatures with different number of legs different sizes different behaviors and materials they were designed in a custom editor within unity which we called poly tulle creatures can be created and edited on-the-fly adjusted very quickly it's a very flexible tool as we said these creatures just have a code behavior but they have no traditional keyframe animations they have intense of movements but unexpected results can happen any time if the creature slips if another creature interferes if the player grabs it and the terrain is also possibly soft so creatures can be buried the ground can fall underneath them and there's also water and streams making it even more complex so for us in audio the only reliable source of information is the actual physics engines data within these features and constraints what we wanted for the audio was something as flexible as the physics edit or something that reacts accurately

and closely to the physics simulation but most importantly we wanted to give a real sonic identity to each creature so how did we approach this well physics fluid simulation and all small granularity very dynamic systems are a big challenge for interactive audio design when working with physics on one hand you have these unpredictable continuous streams of data coming from everywhere in the simulation and what you need is to trigger a very limited number of discreet sounds at a given time maybe a dozen also you can't anticipate anything that's happening next so the work here is really to interpret this data to translate it and make it usable and meaningful for audio as said earlier we could rely only on the physics engines data which are positions velocities collision information and none of this data is clean even when you see what seems to be

a clean collision it's often a succession of frames where many collisions occur with various velocities there's a lot of noise so we build small code units to analyze specific physic interactions which we call sound sources their job is to filter the noise and translate the data into some information this zone information is events for triggering and stopping sounds plus runtime audio parameters the Sun sauces are instanced and attached to specific points on our creatures and there's one Sun sauce type for each different physic interaction for example our most common some sauce types were move impacts slide tension emotion shake and bind and each was tasked to trigger a specific set of audio events and to control specific parameters in wise now let's take the case of the move Sun source as an example so the move Sun source is used for putting sound on simple movements for this one we used

three inputs from the physics engine which are speed acceleration and position each one is fetched on the points that are attached to the sound source instance first speed and acceleration are mixed into a custom intensity value which is computed power points then this value is averaged between points and with the system of thresholds it triggers and stops some loop in wise this same value is smoothed and sent to our TPC in wise affecting volume and filters depending on the case the point positions are weighted by the intensity into a centroid this centroid is smoothed over time and used as the sound emitter position by wise now we'll explain a bit more each stage of the process beginning with the mixing of speed and acceleration into this custom intensity so it's actually a basic tweakable sum and product of speed and acceleration the coefficients are set by the audio designers for each creature this allows us to treat the weights of

velocity and acceleration in some cases we wanted the acceleration to affect more for example the sounds of paper creasing or sometimes we wanted the velocity to affect more for example for a parachute pushing around then from this intensity how do we get to the actual triggering of events and setting of parameters in white we start from this noisy intensity input and we add a threshold above which sound starts playing but as we see this would result in sound starting and stopping very quickly which is not aesthetically or technically viable in audio so we add a hysteresis behavior which means there are two threshold values when going above the upper one we stop the sound when going below the lower one we stop the sound this makes much cleaner starts and stops for the sound while still being very reactive then we add temporal thresholds to minimum duration and a cooldown which ensure that wise is never spammed with instructions and that the sounds have time to fade in and out

properly in parallel this intensity value is also smooth and sent to wise other EPC about specialization each sound source is specialized independently this has been key for us in VR the moment we put one centimeter upper leg to play the footsteps was the moment they started to really work in flat-screen games it's often more permissive for example you can specialize a whole humanoid voice for the footsteps on a single position and it sounds fine in VR you have to be more accurate than that and in paper-based I would say any creature bigger that a basketball had to be specialized on several images so the emitters were positioned on the points that were attached to the sun sources and when there were many points we use centroids weighted by our custom intensity value to get as precise as we could here's an example on this creature the metallic rustle is moving along the body as the centroid follows movement intensity the small points are the one

assigned to the sound source and the centroid is the big point so the idea was that to each creature we could add any number of some sources assigned to the points we wanted here is cage one of our biggest creatures in the game it has already three Sun sources just for the sound of the body crackling one father had a neck one for the left part of the body and one for the right part this was for when you go between its legs you can hear the two sets of legs rumbling all around you also you can see the sound emitter for immersion being positioned in the middle of the immersed legs and here for a simpler smaller creature you have a different sound source sets the locomotion sounds whereas a specific

type of song sauce that manage food slides and footsteps we used one a meter per leg one challenge we faced was to characterize the gait of each creature often when you hear a real animal moving especially when it's fast you don't hear individual steps you hear a rhythm in theory that should have been automatic with our audio parameters being driven by the strengths of each step but we wanted to accentuate that to avoid the typing machine effects when footsteps unlike tag tag tag tag tag tag tag tag that's why we added static wise parameters giving to each foot different offsets of volume pitch and filters so each footstep sound would be affected by these parameters giving a rhythm to the global gait now let's listen a bit to the result of all this [Music]

[Music] [Music] in the end we would say these are the benefits of the method the workflow is rather efficient it was pretty quick to put sounds on new creatures there's a lot of room for user intervention it's easy to what other creatures give them some signatures and adjust things when they didn't work right away and it's modular structure maybe it's very flexible in the end it was used for all

physical objects not only creatures and it was easy to add new some source types for new behaviors that would follow the same pipeline but there's also one particular drawback it is CPU heavy as each object can trigger many sounds the voice count can easily go too high and there's a limit to what wise can do on that level so we had to do a system for custom high level voice management it was a bit of work but it was essential and after that it made our life much easier now let's talk about the creatures vocalizations and how to drive animation and other things with audio creatures in paper beasts vocalize when they are manipulated when they're afraid or hurt or curious so the vocalizations were a vital element in making the creatures feel alive and having the player connect with them our intention was to make them sound like nothing known also nothing too distinctively vocal Florian used a lot of non vocal material glitches cardboard flutes

pencils here are a few examples from the game the predator cage Couty on the paper bore and the tape the challenge with this sound was to connect them to the visuals as the creatures both look and sound a bit out of this world we had to give a little push so the player would recognize and accept these sounds as their voices without questioning so we thought as the creatures are very flexible because entirely physically simulated and as there are no keyframe animations and no animators why not drive their movements with audio our strategy for this was to bake sound analysis data from the vocalization assets we made this choice so we could tweak the analysis data and

so we could anticipate by knowing the data in advance it also allowed us to ignore runtime attenuations and base the analysis on me on the original sound files here's an overview of our workflow first we have the offline analysis where we extract speech and volume information then all your designers tweak and mix the analysis curves with custom tools inside unity then at runtime when a sound is played the game fetches the matching curve and fit it back to the animation to another script module or anything else through the game code API for the analysis we used open-source software sonic annotate or and Sony visualizer with these two you can visualize and batch extracts almost any kind of data from audio files in our case we extracted amplitude each centroid and spectral centroid the analysis batch was launched automatically from unity here's how it looks first you select some files you run the

analysis then you can preview the sounds while visualizing and tweaking the curves and you can create mix curves by combining pitch and volume together with a bunch of tweakable parameters to get to the desired results we also have additional options using alternate sounds for analysis if you want to analyze only one layer of the sounds or get rid of parasite frequencies and you can even draw on top of the curves if you don't get what you want here you see the raw pitch and volume curves and to custom mix curves the mix curve exists in two modes the absolute mode and the speed modifier mode that I'm going to explain a bit more here's an animation with the absolute mode where the movement follows a combination of pitch and volume [Music] and here's an animation with the speed modifier mode where it's the speed of

the movement that follows a combination of pitch and volume [Music] sometimes we use these two modes combined on single animations each curve driving different axes in the animation here's a basic example [Music] and finally some in-game examples here's the tape howling then the baby for [Music]

and finally paper strip in the end this method had passed in several ways it helped us making the vocalizations work by creating a strong audio-visual bond it offered us free diversity in the vocalization motion as each sound file produces its own unique animation variation and finally it was very flexible and easy to adapt for various applications and it could be enriched with new analysis methods new offline processes for new uses we didn't use it

as much as we would have liked but there's lots of potential now to the final part of this talk where we'll speak about environments and be a specialization the environment of paper beasts also had its share of challenges essentially because of the VR specificities in flat screen experiences audio environment can be a background illustration that freely evokes what you want the player to imagine but in VR the player has the environment all around them and they need to think they are truly there audio plays a major role in this VII nation and it needs to be accurate and to behave in a natural and consistent way then you have also less choices when designing audio VR environments with flat screen you can suggest a lot of things that aren't really there but in VR you have to be very careful because there is no off screen space if something is help the player can just look and see if it's there you can still suggest something that is in an unseen space behind obstacles but it has to be done with caution and finally there's

the challenge of outdoors in VR making natural sounding environments is somewhat easy when you are dealing with point sources for example in ontario's with a lot of machinery electric devices that's because binary solutions provide good specialization for point sources but without those you have to deal with ambience pads which is a bit less easy in VR there's the mb sonic format which works quite well if you are able to record your environments but to design ambisonics pads from scratch for non realistic environments requires a very specific setup and it isn't very fast nor intuitive so we search for other solutions that we are going to introduce our strategy for ambience pads was to blur the specialization the goal was to make sure the player wouldn't feel like the listening to just some files playing we wanted to treat the ear into believing the sounds were coming from the VR world itself to achieve this we combine several specialization techniques trying

to find the right blend for each scene and each asset first we use static non-specialized binaural recordings they sound very natural and are easily accepted by the ear as real when rotating the head they can some awkward especially if there are prominent elements in the sound so we use them for indefinite continuous textures like earth tones or sand rustling then we use specialized quads so for channel sounds there are easier to auto land em by Sonic and they give a good sense of 3d depth to the scene when rotating the head we had four emitters at high altitude for the four channels each one with a bit of spread to smooth the rotation effect sometimes we use them with the binaural plugin sometimes with the default specialization we also had an audio billboard mode to easily convert a stereo file to a quad pad we would play the stereo file twice with the second one sitting at 50% of the file they would play on Auto Canal facing emitters giving the same result as a quad file we used additional random

ethics to strengthen the 3d motion and diversify the result but we did this very carefully not to suggest 12 elements that should have been visible now an important part of paper based audio environments is the wind the world of a papist being quite barren apart from the ecosystem when was the natural choice for being the main environmental audio character on top of that it was also a part of gameplay so it really made sense to give it some sonic presence but then the question was how do we convincingly specialize something that is immaterial everywhere and yet constantly moving here again our strategy was to combine several specialization techniques to try and make it feel real we used specialized squads for the base with a lot of asset valuations for the different in that city across the game then we added the sound of wind buffeting in the ears this was done with two voices playing in each ear full left and full right we made

these voices react to head tracking with two individual individual eq's changing the filtering depending on the head orientation to simulate the wind entering differently in the ear depending on the direction we also added three wind gusts with moving point sources circulating around the player and following the direction of the wind when the wind is soft they are almost inaudible they only amplify the buffeting sound and other win related sounds but in high winds they are very loud and they give a nice physicality to the air rushing around you another feature is the global dynamic equalizer on the wind bus the idea is that the frequency and gain parameters of the EQ are defined by the ambient zones so they especially interpolate when going from one zone to another we use this for example to measure the wind when going in a covered space or on the contrary to boost the base and gain when going on an exposed Ridge so we create subtle local changes that are very cheap to do and they give an impression of continuous change this

prevents the player from filling a monotony that would break the illusion and makes him believe that the wind is actually making this noise okay now let's listen to all of this in game [Music] [Music] [Music] [Music] now for the reverbs we tried again to

convey your sense of space as much as possible we used convolution reverbs and we added specialized reflections the reflections enhance the 3d impression of the scene which is very valuable in VR but they also made it easier to highlight the different locations and different acoustics which isn't always obvious with the limitations of impulse responses we placed these reflections on the prominent topological features of the levels big cliffs big rocks and there are plenty of those in the game on the technical side we used auxilary 3d position busses in wise and we put them as children of the convolution reverb bus each of these busses carried an EQ effects and a delay effects we would have liked to put variable delays instead of static ones to have a realistic behavior when going near the reflectors we tried with heavy from NZ an audio but we didn't have time to get to the end of that but it's doable still on propagation we did a basic sound illusion system that would measure

the thickness of obstacles between the listener and the emitter it was difficult to do more in an open outdoor environments but this simple solution was a lot better than nothing in VR it's very confusing to hear very clearly and loudly something you don't see at all that's why occlusion is essential on the audio processing side we used wises equation pipeline to filter independently pre fader and post fader allowing us to filter only the dry signal and keep the reverbs and reflections when there is little illusion and we found this gives nice results and finally we wanted to make a last note about something very specific to the VR medium when we were still looking for a composer we made some tests with atmospheric music we liked and even when the tone was a hundred thousand matching the game universe and scene most of the time it didn't work in VR it felt like a via character was listening to music on headphones within the game instead of having the magical blend that happened so easily in flat screen experiences we think this comes from

dissonance between the acoustic space of the music the space of the recording studio or the space of the reverbs used by the composer and the acoustic space of the game that hills Canyon a cave so we thought of many ways to contract this effect by specializing the music inside the game and so on but gladly our composer Roley had a good intuition about this and he worked on the soundtrack by alternating all along with the VR headsets and listening and tweaking it while inside VR by doing this he always found where to place his music acoustically in the scene which was a big relief for us the result is a soundtrack that is often melded into the sound effects making it difficult to know where the world sounds and and where the music begins we still specialized the music a couple times to integrate it even more in the sound environment so we are coming to the end of this talk let's try and sum up all of this we've discussed about the interpretation of a simulations data

into audio information with some examples of our implementation then we've seen how we can use the simulations plasticity at our advantage and use audio to drive other game elements such as animation and finally we've talked about some of the VR specialization techniques we used and how by combining them and adding dynamic behaviors and details we try to trick the player into believing in to a VR illusion okay well thank you very much for listening to this talk and feel free to commend this video or DM me on Twitter if you have questions or if you want to start a conversation by