Bringing VR Experiences to Life with the Magic of Music

[Music] my name is daniel perry um i am the emperor of sound waves aka the audio director at alchemy labs um and welcome to the talk about bringing vr experiences to life with the magic of music i know it's a mouthful and a long uh talk name i hope to fill your expectations with this talk dock at alchemy labs we worked on job simulator um rick and morty virtual reality as well as the recently released vacation simulator uh and this talk i decided to divide this stock specifically into four sections um the music system requirements uh that you may want to consider when you're creating your new experience for vr i'm focusing on vr specifically a lot of it can be translated to uh some specific ar segments so

consider that as like because i know we're in a mixed reality conference um okay so music system requirements music driven by vr specific data enhancing game elements with musicality and uh creating music instruments uh in vr like actually creating those prototypes the talk itself we'll have a lot of examples uh we're listening to this talk through mono unfortunately so some of the specialization will be lost but i think the examples will still be able to give you a good idea of what i've uh what we've accomplished with those examples uh and the talk is about inspiration mainly so there's no uh code involved on or things like that it's just about the ideas uh that you may want to consider when you're working on your new experience um so let's start with music system requirements first thing um when you're thinking about your music

uh would you want to make it diegetic or non-diegetic diegetic meaning uh emanating or coming from an object that exists in the real world rather than uh the 2d ethereal music that no one can actually see and spatialize properly i think vr specifically lends very strongly to the diegetic approach and at alchemy labs we specifically focus on on the diegetic um utilizing the energetic music as our main thing but that being said we never take things for granted uh we have explored and in our experiences we do use the combination of both so making a system that supports both will be very helpful uh channel support so we know that vr specifically um is consumed by our users by stereo

not mono stereo headphones however because of spatialization we have a really cool um ability to uh create more than just the stereo sound coming out of our space of our headphones we can utilize mono sounds that are emanating from specific areas uh in the 3d field and in this case i mean i wrote mono stereo surround and stems but basically those are old concepts in vr specifically we're thinking about everything as multi-mono um meaning even if i have a stereo file we base i basically would want to separate that into like two monos and place them in the virtual environment in a way that makes sense to my experience um and because uh like vr is such a cool um like more detailed experience we can definitely utilize uh weird setups like the the concept of surround

is not uh doesn't really translate to vr because i can create as many audio sources of those mono sources as i want and place them wherever i want so i'm not bound to like 5.1 with the 0.1 being the sub or things like that so that's definitely something to make sure that your music system supports and the last thing is uh making it responsive and interactive obviously all of those requirements like those are not requirements those are more like or if you have them in your experience you will definitely be ex enhancing your experience significantly uh like specialization no specialization is a mandatory uh directionality or directivity is uh if you're thinking about a speaker uh there's a cone of how the audio outputs from it and and when the listener listens um from a different angle of that cone uh some of the sounds will be filtered

accordingly uh and that's what directionality means occlusion behind walls and things like that uh and costume attenuation since we are addressing a um a virtual environment we are not bound to the physics uh or the way the normal physical world would emanate sound and attenuate it and we definitely want to utilize that to like properly mix the experience so unlike a keyboard and mouse experience the music system has to support all of those to be able to properly mixed in the environment what you see here is a top-down picture of the beach in vacation simulator right here the black circles depict where the player can land uh we have a very specific uh interaction areas where the player can be at so the player can't just move around to wherever they

want they have it's a room scale experience but these are the rooms that the player can be in the red circle here and if you can see here the the tower um this is where the speakers are placed uh again because we um subscribe to the diegetic experience we definitely wanted to uh have the speakers located somewhere so that they can emanate from but it had to be a very um a place where the player can see visually where it's coming from from so that the specialization is not wasted on an environment that the player can't make sense out of um but that also brought the other interesting things like the directivity or directionality because the player can land behind uh the speakers and uh the occlusion where the player can actually land inside the um the dock shop in this case uh let's see a short video of uh all those elements come into play

so you hear how the it's filtered right now open oops coming right up [Music] [Applause] [Music] so so other than um the occlusion and the direct directionality the other thing that existed in this example was the responsiveness of the music system because it starts from those speakers that are behind the player but when the player gets to uh interact with

the sports uh the music has a subtle um it disappears subtly and then the next music is actually coming from the new experience or the direct direction of the music is is different in this case and the music had to the music system had to support that specifically uh this is another example from uh rick and morty virtual reality uh from the san diego comic con demo uh this was only shown to a small amount of audience only the comic-con participants you can't experience it outside it's a 5-minute or 10-minute demo where at the end of it the player goes through the portal and into kind of into space basically and there i wanted to create an environment that feels grandiose and giant and interesting and for that i utilized the multi mono six channel that are kind of in a circle around the player uh so that it's not just stereo coming

out of the headphones as an ethereal thing but still it's non-diegetic but it's uh it still gives the player the ability to look around and kind of sense how i decided to mix the different instruments in this environment so check this portal for you to out yeah you know everything's gonna die so pull the lever on the shelf system there you really screwed up this time you gotta you gotta pull that lever right walk through the portal get out of here escape escape through the portal yeah you gotta get out of here quick [Music] that's right we're making a vr game ha ha we're doing a regular futuristic old-timey vr game the 100 years rick and morty vr games 100 years we're all gonna die morty we're all gonna die in a hundred years

though you get it calm down rick you're gonna break these people off shut up morty you don't know nothing about my vr system i'm making a vr system you're gonna you're gonna live in your headset you're gonna you're gonna you're gonna die in your head yeah rick rick is a big advocate for vr if you didn't know um yeah so that's another example again alchemy does design with non-diegetic as well it depends on the experiences that you want to uh create for your player but i would definitely urge keeping like keeping it in mind when you're prototyping and when you're thinking about designing your system music driven by vr specific input data so first thing what what can we use uh the nice thing about vr is that it lends into getting a lot more information so uh we can utilize head rotation

position gaze direction uh hand rotation position uh point and interaction and um different gestures potentially can also trigger music changes uh if you'd like because uh vr is a virtual world we try to simulate uh physics a lot it's a usually most of the experience of physics based experience it would be great to uh utilize the physics information to drive music as well uh there are a lot of opportunities there that can and should be used when you're prototyping and creating your experiences and considering uh how to do music and literally everything uh because of that that's game mechanics that's uh uh objects position in the 3d world and so on and so forth um i want to show a quick video from the an early prototype of vacation simulator where again we used a non-diegetic music stereo to drive a

surprise factor in game play let's just check it out [Music] um [Music] and thank you [Music]

so obviously uh light-hearted humor with the power of music very easily achieved and enhanced utilizing the head rotation data here's another example of a prototype that i've done in the weekend so you'll recognize the steam hands in there and all that this is a very basic educational game that has buckets and it's a addition you put the cubes in the buckets and in this case i'm utilizing music to uh gauge success so there are two things that are happening actually let's look at it and then i'll talk about it a little bit [Music] [Music]

so [Music] okay um very uh simply used again this is a prototype um uh there are two really cool things that are happening here uh a the music helps the player understand if they're doing right or wrong and then the farther away they are from uh like achieving their goal the music just sounds bad uh i could have gone with more extreme but that was the the concept of trying to prove how how far you need to go to make it uh extreme but the other really cool thing is a a game interaction or a way to interact with the world that you cannot really do with a conventional game and that is to use

your head to create focal points for example if in this case you looked at one bowl it cleared off all the sounds from all the other elements and the only thing you were hearing is the sound of the ball that was why the music transitioned while uh the player was looking to the right to the music of the the ball to the right and i think it lends to another cool game mechanic that i would definitely encourage to explore which is uh utilizing a stethoscope or a microphone to point at different uh areas of the world to here to to focus on those specific areas um uh okay enhancing game elements with musicality first thing i'm by that i mean turning objects in your world into musical instruments why would we even want to bother we've got a well first thing it can be a

determining variable in the viability of the design itself meaning in the prototype stage when you first create your uh object and we'll see an example uh later on you may want to toss away your prototype just because you haven't experimented yet with the additional factor which is the sound and music musicality in in some cases um if you if you uh add that musical elements into it you can uh add life to the environment and bring more feedback and responsive responsiveness because after all vr is a like if you do it right it's a very responsive environment and should react to anything you do whether it's surprisingly or uh or in a um a measured way it offers a creative outlet for the player who are after all here just to to have fun uh and it's it can

strengthen the emotional atmosphere of the scene for example if you're doing a bike experiment experience you'd be able to have an object that has an electric guitar sounds coming out of it as a menu for example um what are the things that you may want to consider when you're doing something like this and again i'm i i feel like i'm talking to the right crowd here because uh specifically i don't want to talk to the audio people to the audio department i want to talk to the designers to the people who are creating the experiences stay i'm kidding uh just because um these are things that need to be considered early on and prototyped on because they can make it a significant difference in your experience so okay so things to consider are a potential problem between the background music and that element if the scale of that element does not fit the

background music you might create kind of a cacophony so instead of creating something that uh would be comfortable for the player to listen to you actually are making the opposite uh just on specific chords so that's something to consider you want to choose the right scale and harmony that's again that's on the musician but but still uh it's something that you know you can demand for example uh uh the um if you have a middle eastern experience you can go with a harmonic minor scale and it will sound a little more to that genre it will lend to it a little bit better and then you want to design around the ui so because we are again uh vr is a very interactive experience you have um buttons and knobs and led levers and they all respond in a very different way sometimes stretchy objects that you may want to turn musical um so there's there's a lot of challenging there in the implementation as well um that you may want to consider early

on uh because those are um implementation this this is basically audio audio uh physics system uh which is not uh obvious uh in any system that was were created where we're we're fighting hard enough against the real physics simulation uh but this is another aspect of the physics that need to be uh taken care of and then you want to consider solutions that work into infinity um uh which means you can't predict when a gameplay will end you may wan like the player may continue forever so in vr specifically you you might have to end up with other solutions and we'll talk about it a little more later uh let's see an example example from uh rick and morty virtual reality uh in the uh microverse where you're charging a battery

[Music] so so i see this as a wonderful example for uh the viability of the product depending specifically on sound design and in this case uh ramping up the the speed of it depends on musicality as well the first thing that ends up

ended up happening when we were prototyping this if you if you look at kind of the size of this board um when we uh started prototyping uh and we tested that that uh specific interaction the player had to continuously scan the board left and right to find out where uh where the player needs their attention this play this game was literally unplayable until um sounds that came out specifically from the light bulbs helped and again we're listening to everything from mono so i don't know if you can like but the concept is clear uh the player knew exactly where to look at uh right after i added the spatialization but even better than that uh even without if your sense of direction is not that great which can happen with people um the additional help was the pitch so when the player looks to the left of the machine um they know or when the player hear a lower note they know they need to look

at the left side of the machine and when they hear a higher note they know they they need to look at the right side and it's it intuitively uh becomes kind of a habit relatively quickly if you if you do that that interaction a few times um similarly this uh device had uh two modules so there was a the the clever solution that i've had with uh five um modules and then three was using the minor pentatonic scale here uh which no matter how you randomize the notes usually it sounds like a nice melody so that's kind of nice uh for that side and then without those modules it just becomes the first three notes of a major scale which also made it very easy to listen to all the time this is another example of again turning uh an object into a musical instrument uh and this is the avatar customizer in vacation simulator uh i'm specifically starting in this example from the elevator so that we can

hear kind of how the environment also responds the music changes from room to room in a very subtle transition that doesn't sound uh like off-putting or doesn't sound real like it does it sounds very realistic and natural and also how you can still hear the elevator from behind you after you walk out of the elevator let's listen to [Music] this and also the music is diegetic here it's coming from those speakers [Music]

[Music] um so uh i really love this example because it really shows like how you can get a menu to become a real musical instrument you can improvise on it as you as you saw um so an outlet to the player and how it uh adds to the emotion of the the room and the environment um uh this is a the bathroom where you customize your avatar and it feels very calm relaxing soft and in this specific

case i wrote the music specifically for like to work with this new instrument so there are no wrong notes at any point you can no matter what notes you'll play it will sound good with the music so that's definitely uh worth checking out now this is an example of an infinite interaction um just so you can see i spit sped it up in the middle of the way so you can get a little more sense of it [Music] [Music] so basically this yeah this will go on forever the player can keep playing it as long as they they don't fail they'll continue to hear it rise

and we all know that that's impossible to make happen uh so uh this requires a specific trick that uh you can utilize it's the shepherd stone the shepherd tone is usually used for um a single long pad an infinite pad that rises and uh the but in this case i used it for mallet sounds um what you see here are basically uh the four volume automation um of of four different channels that run at the same time um obviously this is all baked what we're looking at is the recording application um where and in the lowest note the notes are barely audible and it ramps up as you get to the meat area and then on the high notes uh the uh or high octaves the um the notes gradually disappear and uh about here is where uh this track jumps from the highest

note to the lowest note and so uh uh it basically creates an illusion of where um the sounds infinitely rises while in fact the higher harmonics slowly disappear the lower harmonics gradually comes in uh that's a great effect that canon should be used for those kind of experi experiences uh to keep it musical and again a musical music scale let's talk a little bit about designing uh musical instruments in vr so uh first thing um things to consider um our our perception our uh hands for example are very very delicate and perceptive we've got uh pressure in each corner of our finger and um when we're playing a guitar for example we have the ability to vibrate the guitar in very strings and very subtle ways to make them sound natural and nice and the way we like to

hear the music this is simply not possible in in vr in their in its current stage um not even when the hand is being tracked with all four fingers because it's not as precise and as delicate as the as the real world simulation because we all know we live in the simulation uh as so as the real world simulation uh is able to respond to our fingers and our hands one day we might be able to get there um the other thing is um because we're we're we um in the real world we're bound to uh the physics system if we think about the trumpet for example it has the three the three valves the uh at the top here that you the buttons that you click on and it's kind of not sequentially to rise up you actually need to create different uh combinations to to get the notes to rise up in a scale we don't have to we are not bound to

physics we are not bound to the length of those tubes meaning that we can make instruments that are more intuitive so it lends to limitations but it also lends to like creating cool things um uh the the last thing is uh wellish i'll take a shaker and uh or uh uh well a shaker for example which is an instrument that uh sounds like it's easy to imitate uh in the virtual world while in fact it is very very complicated to get it right and we'll i'll talk a little bit about it later but um the granular versus global approach because we have a lot of grains inside that uh shaker that if we want to emulate right we have to really really think about how we want to create that emulation simulation um so let's talk about uh guitar a little bit or a ukulele in the

case of uh vacation simulator uh there are a few things that we want to consider for this medium first thing how do you measure a strum and in this case we went with a single strum for all strings you cannot do each single string how do you represent in the instrument's respond because everything needs to be visually responsive uh you don't have to rely a hundred percent on audio and you can see here how we made it kind of like a graphical toy approach and the strings only come out from here to here um will it respond with one hand and alchemy specifically does a lot of work to try to make uh the games accessible to everyone and in this case uh we sacrificed the second hand choosing harmony and or changing the notes with basically a toy approach if you see the lights right here it kind of chooses the harmony or the the chords for you again we're working

with a lot of limitations and we're trying to make the best best possible experience for our players um and then how precise uh is your interaction well do you want um the plane here to be huge or small uh and again because we have a lot of uh precision situation here um the hands doesn't actually need to touch the strings to play them in our case because we found out that a lot of players will just find it so frustrating that the design is too precise for them so again accessibility is in our mind uh here's a small video i wanted to show will look nice in your garden

[Music] okay so well in this case i just wanted to show that um in our game you can shrubify a ukulele and then use a shrink a shrub to shrink it and when you shrink it the harmony actually changes pitch according to the size like we went as far as to make sure that the world responds to the instruments in the way that well the player can't expect to shrub a ukulele or shrink it but if they can do that then the world should respond accordingly and that's something that one of the reasons why it's worth adding in the prototypes because then you can respond to it properly in in your experience um harmonica is an instrument interesting uh problem because it's a uh it's a breath instrument you have to uh put breath to be able to interact with the harmonica while a microphone uh while a microphone

does exist in most of our headsets uh blowing on a microphone is not a reliable way to get any data it's a kind of um [Music] it's it's a random noise generator basically you're not going to get uh data that you can drive music out of and so you got to think of other ways to do that in this case yes there are interaction areas and then things that uh you want to consider is how big how far you'd want to put the harmonica in front of you and uh in our case it would it we chose it to be far enough so the player can see a highlight on whatever uh uh area that they're interacting with so we can actually so the player can actually make a choice uh the other thing we um is about the player's mouth uh distance from the headset because we again we don't have that information um and we want to be inclusive there are people with with long heads and short heads and we want to address all of them so we kind of got to pick a size that works for

everybody and then how many interactions area become confusing and that also lends to how we feel the instruments because uh when we're using a harmonica in the real world the fingers can feel the screws that are in harmonica you really know where you're you're holding the harmonica and you utilize your lips and your tongue to really feel on which hole you're about to blow so so that you don't have to look to be able to play a harmonica properly which just will never work in vr the last thing again the shaker of the maracas which why is it so complicated well first thing even if you choose the approach of a single [Music] impact point and not full grains inside the shaker uh how much uh gravity will you put on on the um uh on that uh thing really depends because it can be different from reality and and bounds um

the uh a very interesting effect that you can get from the shaker is the rain effect if you just rotate it or if you turn it in a circular motion because those two are different sounds by by themselves of rain that you can generate from it and these have to be custom implementations of recognizing those type of gestures and then obviously the physics aspect of it of what velocity will equal what volume uh so to sum it all up uh music gives you the ability to embellish interactions and intensify emotions in your game consider music accents in your experience early because again this can change your final product and prototype to find the best solutions for your experience make sure to support the audio team with the systems and implementation solutions that will allow you to perform all the necessary tasks because it does require more programming

power to be able to to solve and with that the final thing [Music] yes so thank you very much uh we do have some times for questions i just had a more of a comment than a question um which was when i was at magic leap one of the first things we the difference between vr music and ar music um and one thing we discovered quite early was that if you have uh non-diegetic music too loud in your ar experience it immediately masks everything else it becomes vr immediately uh music in ar if you want

to mix with the the real world has to be kind of way in the background and very soft and not very attention getting because if it's a regular game and uh game music it'll just mask everything in the room and you won't be able to talk to anybody you'll see their mouths moving yes so it's a different kind of experience between ar and vr absolutely and and i was only talking about the music today not the sound design but if we talk about sound design in vr there's a lot of details that you want to make sure are not obscured by the music either because then you're detracting from the experience as well yeah well thank you hi great really great talk nicole lazaro from zeo design this is fun to talk on this video on the headphones this is great um i had a question about well great presentation uh i have a question about uh three-dimensional sounds like you know getting a little bit more depth we just did the bose ar uh game jam couple months ago yes and we created a choose your own

adventure where we actually did audio escapes and you interact you're using audio to determine make your choices you don't have to even look at your phone i was curious in terms of design are we going to get there it's sort of like um stereo seems to me like flatland where it's just between two points right but uh we have your chicken fairy godmother at two o'clock high and then the frog she's talking to down low it just creates a much more you know interesting volume not just a you know kind of a 360. how do you talk about it being 3d but not you know i mean but being additionally 3d where you can actually like walk through so a space so uh first thing uh this is what i was talking about when i meant spatialization yeah and the main function is hrtf uh head related transfer function this is the official term and there are like quite a lot like i think four or five um big like bigger companies that already make really great spatialization tools yeah we use the resonance plugin

resonance is lovely yeah yeah and and resonance by the way have uh their activity built into it already which is also kind of nice uh uh so there's a lot of solutions they have many um things but yeah this like it's ready for everybody to use i'm not sure what else uh like the in in terms of um like i see it as a mandatory uh tool to create here are you asking if i think on like this is enough to create an experience just the spatialization well that's definitely true i think it already is i i tried the the a lot of the bose experience eyes i i feel like they have been missing out a little bit because i have heard better spatialization in nvr uh it's probably because they have to have a lot of compromises in terms of processing uh because like the bose headset it's on a mobile phone it does it does it on on the mobile actually you're right so i'm not sure why uh it didn't i i couldn't pinpoint

to the exact area like i do in vr i can close my eyes in vr and point where the sound is coming from and i would mainly be right after i open my eyes i look at it this is where it's coming from in bose i didn't feel like i was able to do that when i was little but entire experience but if there's um there's sort of like a 2d in terms of terms of art because it's helpful to have language around this yes so in my mind and i could be completely off base but you know it's 2d you then have three dots so almost three dot you know kind of audio and then you have six stuff yes where you can actually move where you can actually move through right uh i um the difference between what ambisonics for example can perform to what the 3d specialization can perform like you have a single location area and a sphere around you or you pick where you put it in the 3d space i mean i think it has a lot of potential and a lot of future for sure for like a marketplace and it all the sounds passions you walk that kind of thing thank you yes yeah thank you

i'll be really quick um thank you that was awesome uh quick question um what's your favorite process or planning tool that you use to communicate with your production team um well first thing at alchemy i'm also a programmer so uh i do a lot of the prototypes that i want to do and we have a very interesting uh language in the in the inside the company uh it's kind of funny because um at alchemy they're like daniel make the make the sounds they bring me in early enough that i can ask enough questions and that we can iterate and go back and forth [Music] just like any other prototype and i think that's the main thing it allows me to improve over time understand the vision of the specific experience and well the unfortunate consequences is that you know just like any prototypes uh let's say anywhere between 30 and 70 of it will go to the trash

and what if you were working with a remote team how would you manage that uh we do have some programmers on remote capacity and it's a lot of video conversations just you know the ability to see and sense what's happening and and constant iterations that's that's all i can say about that really yeah thank you thank you oh one more hi in working with a lot with vr that people still get uh motion sickness because primarily because they're moving their heads too fast and i'm when i'm standing over them i can go hey pretend it's a video camera it's quit jerking around have you considered or tried any kind of auditory feedback that's the equivalent of slow your head movement down in order to avoid that thing which which is still huge in the industry yeah

um i i haven't i haven't find found a solution audio related to that uh at alchemy we see that as a yes it is a problem with the speed of the head of course the faster you'd move the the the less the frames would be able to respond properly but that's why uh on pc experiences we have such a strict hard limit to make the performance of our games work at 90 frames because the higher the frame rates of the game are you can improve the feeling of of that sensation of motion um like significantly the frame drops are what really causes uh the problem as far as i can perceive it um so so it's more about the solving that really the performance you can't get the user to hype her down well it's i i wish but it also kind of defeats the purpose of the freedom that

we want to give the the players uh if we can solve it then we like as in our on our end we without rest restricting the player uh in the way they interact with the world we will work really really hard to solve it and it's worth it thank you thank you okay thank you very much