Everything You Need to Know About VR Audio Can Be Learned from Your Car

[Music] thank you scott thank you very much everyone um somebody was worried that this is a little bit too small and the text might be hard for you to read because we're not doing the whole screen this is the only text in the whole presentation so you don't have to worry about it um the idea about is that i wanted to take a common object we either all own a car at the very least i suspect we've all been in a car at some stage or at the very very least we've all seen a car literally we have to dodge them to get into this building by using a common object it just means it's it's uh i can communicate some uh concepts to you in a really really simple manner um and also it's actually i mean cars are fairly common objects in games as well so it's relevant in that in that regard um i've been working in vr and ar for quite some time and i even wrote a book on the subject and what i've discovered is that um this this field is changing so quickly that it's really hard to keep up like within a few weeks of the book basically being signed off and published and everything

like that i'm like i want to change that and i want to update that so this in some ways is a little bit of an update of some of the thoughts that i've had where i was originally was that vr is all about immersion because we are placing somebody into an environment if i if i put up the the best playstation or xbox game on this screen right now when we're playing it with our controllers essentially what we're doing is it feels like we're stuck looking through a window and the game is all the kids outside having fun we're stuck inside and we can't go out to play whereas vr allows us to actually go awesome we are inside everything so that's where the importance of immersion is and one of the quick things i will point out is that what i've found is there's quite a few people working on vr who kind of not getting that i'm finding a lot of our experiences where they've got headlocked stereo sound now if if i'm standing in a rainstorm there is rain all around me so we'll call it raindrop a here raindrop b here but we've also got raindrops c and d

my relationship to those raindrops changes dramatically when i turn my head have you got stereo rain bam you have immediately under undermined your entire concept of providing me with a virtual environment so that's that's one of the aspects but one of the other ones is scale and that's what we're going to be talking about today so um this is a representation of my car my actual car is white not green and has slightly different wheel might have a little engine in it but but the concept is the same i'm using a very very standard hatchback car nothing surprising about it nothing particularly unusual about it and what we're going to do is we're going to expose it a piece at a time so that i can explain to you kind of why a car is important especially when we're talking vr and ar and i can do this by walking down here because this screen is a little bit

small so it's not quite life-size but we can see here that a car is a large object in relation to me if a car was in a video game it's an object on a screen that's about that big and you can quite often get away with having a single sound attached to that car and the car moves around and it makes noise but as soon as we scale this thing up there are immediately two points about this if i'm over here the prominent sound i'm going to hear is the engine which the car on the car's idling let's assume that so i'm hearing the engine but in vr because of the scale and quite often because of the freedom i can go yeah okay engine that's fine i'm going to come down here and not just come down to the back end of the car but i'm actually going to get down on my hands and knees and i'm going to put my head right down with the exhaust pipe knowing that they get really really hot i'm going to be really really careful yes i have melted microphone covers getting them too close to exhaust pipes but but this is the point this car

already is now at the very least two sound emitters because i can get close up to this exhaust pipe and the thing about the exhaust pipe is not just that it should be one extra sound if i'm standing here and the exhaust pipe's down there i'm gonna get one type of sound as i lean down onto my hands and knees and put my ears right near it it's not just that that sound gets louder what i'm hearing actually changes it's the difference between a waterfall if you're at a waterfall and it's 500 meters away it's just a very very generic white noise but as you get closer to it and closer to it the resolution in the same way that the visual resolution of the waterfall changes the sonic resolution changes as well there's more detail you can see you can get up to the point where you can practically see the pixels of the audio so as i get my ear down near the exhaust pipe i'm hearing the the popping and cracking and i can hear what's going on inside the engine whereas if i'm this

far away from it it's just a general low bro hum so already what we've got here is we've our car has now been split in two focusing on the exhaust pipe getting close to the exhaust pipe what we need to do is we need to understand that there's a difference in resolution and as i say that resolution is not just it's louder or maybe a filter or whatever i i would suggest and recommend in a lot of cases you need to have a uh like an lod sort of like two or three different sounds that you're crossfading between depending on where i am now the engine is the next thing if we we sort of pulled the engine out a little bit a standard car engine is a fairly large depending on you know what sort of size current is a fairly large object but it's not again an engine is not a single sound source

any of you who own a car i really really recommend you do this you know find your car switch it on let it idle open the the bonnet up and just lean around move your head around carefully obviously move your head around you know it's to the side move around to the front lean over there you've got a fan belt going here you've got pistons going here the average car engine is made up is it is a complex sound made up of dozens of aspects so again car game you know the cars this big on the television screen then fine but if if this is a vr game where i can walk over and i can throw open the bonnet and part of the game is to find out what's wrong with the car it's like oh is it fan belt no that sounds okay or this thing's rattling over here it might be that and what we've got over here we've got this thing buzzing so the level of detail is directly relative to the scale that we allow ourselves to deal with and in vr

sometimes the scale can be huge so we look at the individual engine um and i will tell you you know there's there's pistons going up and down but you know the the the sound of the pistons going up and down is going to be very different if we're down here lying underneath the engine maybe we're doing some mechanical work or you know it's a racing simulation game and you've got to tune the engine and so you're up top playing around with the things up top as i said there's a whole bunch of components that go together to create essentially the gestalt entity that is the engine now i'll pause for a second i'm not suggesting you need to always go to this level of detail it depends entirely upon what the experience is but this is an example this is a vehicle to discuss the sorts of things we can do and i'll give you a quick practical example and if anybody's interested in this i'm actually doing a postmortem on this game on the friday it's a very last session on friday for anybody who's still awake

a game i worked on which was a sailing game i'm not going to go into all the details but one of the things that i had was you are sitting in the boat you cannot move which for me is a beautifully curated experience i can craft that entire experience because i know you're not walking away you're in the boat it was on vive which means you're actually you can lean this way and that way etc relative to the boat you've got the boom coming by but you could actually lean right down to the water so i had the sounds of the water rushing past the boat while the boat's moving and i had different ones depending on the speed of the boat so those sounds would change but it was generally just that sort of general sort of white noisy sort of sound but if you leaned down as you got closer it would fade up a more bubbling gurgling texture sound so again what i've given you is this kind of level of detail idea of as you get closer to that sound source it's a whole different sound and we can do this with all sorts of things it'll be up to you per project per thing how important the thing is what sort of relationship it needs to

have with the audience but this all is all relevant to the scale all right so here's a really really really interesting one um okay it's a vr game it's a driving game fantastic so we're driving along we've got our interior of the car and we'll get to the interior in a second but before we get to the interior of the car we're going to open the window okay so what's the ramifications of opening the window in vr there's one thing in vr that it's kind of like okay so i'm driving and the window opens so now what we've got is we've got our our enclosed box and the sound of our enclosed box has just been breached we've just opened the window so the general sound of the engine we've got a direct sound source coming through so it's not all coming through the the sound dampening within the car we've opened the window and now we've got the sound of wind going past and also if we're driving anywhere where there's any objects we've got this of trees or telegraph poles or whatever going past so you've got all of that but it's vr i can stick my whole head out the window

now all of a sudden with my head out the window everything's changed the the my entire perception of the the car engine is external right i'm not getting any sounds traveling through the body of the car but moreover i've now got the sound of wind rushing past my head and the thing i was telling you about before about turning your head well i've now got wind coming about my head in all different directions and i can do all this so now it's a single wind sound is not going to seem realistic it's not going to seem immersive you need to have multiple emitters that basically allow me to have my head inside a volume of sound so opening the window in vr is a really really significant so if you're working on a project and somebody sort of says oh yeah it's a racing game it's in vr we're going to open the window you need to very quickly put your hand up and go just let you know that's going to have a pretty big impact they're like well can't you just change the sound to win no and that is actually part of what we

need to do is really advocate for uh the more we understand about the impacts of things the more we can actually communicate that with our teams so when somebody goes oh we just want to open the window because we think that'll be cool and you can just add a windstorm it's like yeah you're going to need to give me a couple of extra weeks they're like what and now you can explain why because of all of those elements so there's a lot there can be a really big impact from such a simple thing as saying i want to open the window interior let's mask that out except for a couple of areas hey look four speakers i think it's actually got some more speakers somewhere else but those were the easy ones do we have like you know gta style do we have an in-game radio plane if so speakers there there's down here if i'm leaning across you know the speaker's getting louder the music etc etc again these are all options i'm not i'm not saying you have to do these things

but if you're trying to create the problem about something like a car whilst it's the perfect thing for this discussion today it's also a huge issue because every single person who might be using your experience knows what a car sounds like they know what the experience is like they know that when they're in their car either they're driving or somebody else is driving and they're in their car they know there's a speaker down there and there's a little speaker over there so if i lean across to the glove box you know i lean across the glove box my perception of the sounds coming out of these speakers should get softer the ones that are there should get louder if this uh is a game or an experience where there might be a passenger then the passenger going there should muffle the door speaker theoretically you could even go to crazy levels and go well you know if i put my hand over where the speaker is in vr if i put my hand over a sound source that we know in the real world as a sound source does it muffle that speaker um i'll get to it in a second but down

the bottom there's like a coin tray that's rattling if i put my hand down there does it stop at rattling or we go do we want that level of interaction so there are so many little things and the inside of the car basically has far more the more ice the more i did this i started going oh wow okay the inside's got lots of stuff um there's so here's the glove box and we know what the glove box is and we know okay well if you can open the glove boxes you know there might be the sound for it going in the the you know the hydraulic thing of an opening but even if the glove box is just like a flat texture you never interact with the glove box that doesn't mean that we as audio people can't interact with the glove box now here's where we take a step aside and basically one of the key things that we need to do at audio folk is obviously we're making audio but the reason why we're making audio is because we are here to support the narrative so

many objects can take on the uh the importance of a character a best example for me is as far as i'm concerned the lead character in star trek uh not the absolute latest one but prior to that the lead character in star trek is the uss enterprise the ship is the lead character so if this is our okay let's just say this is a transformery thing so this is kind of like bumblebee's little sister right and so bumblebee's little sister is a little bit quirk and everything like that and one of the things is that bumblebee's little sister never opens her glove box that's private i don't mean that in a bad way i just mean you know she just doesn't do that right but what we want to do to give her a little bit of character is there's always something bumping around in the glove box every time you turn a corner there's something slide clunk slide clunk slide clunk that's in the glove box that's moving around so we've all of a sudden made a glove box that doesn't even exist there are no assets for it it doesn't open up you will never see anything in

there but all of a sudden this could be something that's really intriguing and add a whole new element to the story because it's like what on earth is in the glove box and it's like she never talks about the glove box oh so all of a sudden it's like the player's gonna be pulling their hair out going i want to know what's in the glove box and i mean that's the sort of thing that i would actually just add initially without even asking and then basically have the designers going what's the clunk with the glove box like there's something in the glove box it's easy to show rather than tell and then they'd be like what it's like you know it's the sort of thing that's either gonna or not and if it works they're gonna go okay and it's like look she's got something in the glove box it's a secret and i've just given you a whole new quest for the sequel game you know find out what's in the glove box but this point there's difference between i mean character objects are really important the difference between i'm in a game i'm in an fps game and i go through six doors they all just have the normal door opening and door closing sound i go through the seventh door and it's got a really really rusty creaky sound that

sounds like it's gonna jam or break we've drawn attention to that door for a reason we can do the same sort of thing without with our vr interiors uh so this is what i was talking about before is that we've got the the gear the gear stick that you can see down there and actually just behind the gear stick it's not really very very clear that is actually another little box for coins and stuff so again um are there coins in the coin box you know they rattle sometimes and as i say you maybe you put your hand on the coin box and and it stops rattling but even things like the gear stick maybe it is a vr game and let's say we've got gloves or whatever or you're using the controllers and a hand comes down and he's moving the gear stick do we want it to be a purely realistic sound of the gear stick are we trying to simulate reality or are we trying to tell a story my thought is if we're trying to tell a story then again it gives us agency to add a little bit more character to that gear stick again if bumblebee's little sister you know maybe

third gear kind of grinds a little bit because you know somebody ran into her years ago and it never got fixed you know it's kind of like that sort of hip hip problem sort of stuff it's it's it's kind of a lint but not a limp but again we can put character into order automatic sorry we can put character into ordinary everyday or seemingly ordinary day actions but it's still a case of when we lean closer to the coins are they louder if we put our hands on them do they stop rattling you know maybe you know if i if i move the gear stick around a little bit click on click it goes into gear more easily so i don't get that rattling there's there's so many things that can be done with sound to tell micro story in the same way that many games will have people writing like little little quests and stuff like that you can bring a lot of character to various aspects and various objects in the game depending on how you approach it and i'm not saying just go crazy it's always best to consult with

your team but your team may not even be aware that this is possible they're like oh wow we did we didn't even think of this we didn't think we could do that sort of stuff sometimes that's why i'm saying sometimes you might be you know you can just show it off by doing it and if they don't like it you pull it out but if they're like oh well this is cool what's with the godbox thing and you tell them the little story about there's something hidden in the glove box and you might find that like that's a cool idea we really like that leave the glove box clunk in wind vents air vents air conditioning ducts whatever you want to call them again same sort of thing do they make any noise at all are they always considered off or is it an opportunity to enhance the sense of speed because you know maybe maybe the the user the audience the player can't control them or change them but there would be a default setting now you could just say oh their default

setting is off they're closed there's no sound there or what you could basically have is the default setting is just you know the open vent to the outside air no air conditioning no heating or anything like that it's just a straight flow through to the outside uh intakes which means this car when it's all sort of sealed up and a lot of modern cars are really pretty quiet when they're all sealed up you're going uh it's like yeah okay the engine's threshing away but i'm not really getting that sense of i'm going fast you know there's a little bit less of that impact and then you could just basically go there's just a subtle sort of wind sound that increases in intensity or pitch or whatever layers etc as the car is going faster and faster again whether you want to go you could take it to extremes and with bumblebee's little sister maybe she wheezes when she works too hard and so one of the vents basically has got this little bit of a sort of a whistly sort of sound and when she's grumpy at you they kind of like coughs cough at you or something like that maybe that's a way of communicating

i i'm not suggesting again that these are must-haves but my point is you can see already that our interior of this car has got so much potential there's so much that we can do just with the interior of a car that up until now in most games you know and here's all the other bits and pieces um this weird little thing down here believe it or not is actually the um the handbrake this is a little digital one uh that slides open and closed or doesn't as i said the handle on the glovebox the gear stick as well in its controls all the controls for the air conditioning the radio blah blah blah the horn the electric mirrors on the side do these things operate do they need to operate do you want them to operate because they might add more character so it's it's the it's not just the addition of the sounds it's we are building up an environment in the same way that we would look at something and go i want trees over there and trees over there and i want that swamp to have frogs in it because that'll be really nice and luscious and

it'll be really good and they'll be louder when it rains etc and at night time the birds go to sleep and all the crickets come out we're building an environment where building an environment and well sorry the game artists and level designers are building an environment generally what our job is is to bring the environment to life that's one of the key things we do we take things that otherwise are often just static objects or just things that just even even things sometimes that are animated you know you see somebody who's done a really awesome animation of a dragon and i've worked on projects like this where somebody's gone yeah we really really like the dragon it's looking really really cool you know it just doesn't quite have that thing i'm like hang on pop in the sound and everybody's like that's the thing the difference between any creature uh character um or really sort of effective giant you know you know ten-legged walker thing that's supposed to come here and scare damien you know if it's not making noise

it's not completely ineffectual but it has so much less impact so all of these things are potential impact and should be added into the overall consideration of what it is we're doing and how we're doing it so the point about ha is the car is the perfect example of an object we know and we understand that's a really good first place for us to start but even better than that i would suggest that everybody in this room can get access to a car to examine it to think about it it's not like a dragon where i'm sitting there going does a dragon snore you know what does it sound like when a dragon scratches or scratches her butt i mean you know is it is it chloron scales or does she get the clutter under the scales a little bit oh that really feels good you know and so you you know so those sorts of details are important but my point is is that those are a little

bit more subjective etc we can all look at a car anybody in here who wants to try this out can immediately do some research and work out what stuff do i need what stuff can i maybe play with and be a little bit sort of poetic with what stuff is going to be overkill because you can try it out in the game or whatever and then you can basically get in your car and go well i'm just going to i'm going to take everything out of the glove box and put one thing in there drive around and see if it goes conclusion how much it goes i'm going over you know speed humps and stuff and what sort of sound does it make and then you can decide i'm really going to amp this up and make it dial it up to 15 because for our product right now that's what we need so the the i guess the sub text of this session is a suggestion that not just for for vr but it certainly helps in those that

using a real world a common real world object as your point of reference can sometimes be really really good because it allows you to research really really easily it allows you to mock things up because you literally you could do something at work you could take it home you could show your your husband or wife your kids your spouse your partner and go so this is kind of a mock-up of our car what do you think that sounds exactly like our car cool that's a good place to start or no the the coin tray needs to rattle a whole bunch more so this has been a short shot quick and dirty info dump of in some ways not necessarily techniques but just a a a mode of thinking an approach where for vr specifically scale matters and i'll leave it with one final example of um if we all had uh and

vrna it works in the same way um to just get this point across if we all had ah glasses on right now and i basically you know made sure that there was a little ar robot that's walking across the table here and he's this big then he jumps down onto there and we hear the clunk and then jumps down and walks along a little bit more but then grows till the head of the robot is up right up in the ceiling me walking down here now the little robot that was up there one sound source for the whole robot is appropriate the robot here now if i'm walking along next to the robot i want to hear the servos and the ankles in the knees in the hips in the waist i want to be able to lean across and go yeah you need to get your waist tuned because it's kind of a grindy rattle when you turn your waist and all that sort of stuff because the robot is now so big that my point of reference to the robot is i can discern all of the different areas whereas when it's this big a single source is likely enough so just keep that in mind

with vr and ir that it's not just immersion and immersion is really important but scat will impact how you need to approach your projects in a really big way and by thinking in a common sort of object it will help you to sort of work your way through the potential challenges that you may have there um and that gives us a couple of minutes for a couple of questions if we want thank you very much everyone [Applause] yeah have you started using any ray traced audio in any of your development for doing occlusion or reverb related stuff no and i'll tell you why it's expensive and we don't need it at least let me qualify that a little bit we don't really need it at least not at this stage because the reality is people are still getting used to vr and ar and the

the biggest point is is that if i'm outside in the open air and i walk over into that concrete shed over there there's one one key thing i need that's the line there when i cross the line i need a point of difference for most of our audiences they are not going to understand and appreciate if that reverb is 100 accurate for that space as long as reverb is on and off now i would take it one step further if it was an important part of the game or the project what i would like to see is if i'm in the center i get even reflections everywhere when i'm close to a wall i will get one wall closer to the others i would have probably those two states so obviously closer to that all that all that all of that will but i'd basically have i'm away from the wall when i cross sort of you know 25 away so i get the middle reverb or i'm close to war because then i get that really nice proximity effect of i am near something close but ray tracing all this stuff i'm not saying we i'm not saying we shouldn't

have it i'm saying right at the moment i think there are far more important priorities than worrying about completely accurate reflections and stuff thank you no worries thanks very much for the talk very eye opening um i'm curious if you have a sound outside the car say the engine sound right obviously it's going to sound different if you're outside then if you step inside the car it'll be occluded somewhat how would you handle a sound like on the window itself that kind of is in between multiple spaces you can hear hear it from multiple perspectives at the same time cover my car with a whole bunch of microphones and i'd record all those different perspectives um whether i use those those actual recordings or not what those do is they give me points of reference that i can then go off and go actually i don't need to have 10 different layers playing at once i can have my indoor one my outdoor one and the one near the window i would have the one near the window basically on kind of like a a rail so it sounds close to the window because it's sort of the closest point if you stick your head out the window it kind of swivels around a little bit closer towards the engine and

i might filter it if i could if i could achieve the effect that i wanted with filters if i couldn't then i'd just have the different recordings and play those in the different places because yes i talked about the engine and the the exhaust they were the the two examples but you'd also get a different sound source for each of the tires you'd get a different sound i recorded a uh some stuff for mercedes and i literally had uh three microphones in the engine bay one on the exhaust pipe one halfway down each side of the car and and if you look at the way hollywood do it they go even further than that thank you all right uh roundup can we have one more question no uh i'm getting a shake of the head back yeah so i'll go into the where i'll go outside and if anybody's got any questions you can ask those questions there but um the people in red they're really important and and if i go further they just come up on the stage and drag me off thank you very much everyone i hope you enjoyed it