It's like RAY TRACING but for AUDIO | Simon Ashby from Audiokinetic

[Music] we are now joined by Simon from audio kinetic how are you doing Simon I'm fine thank you fantastic I think you've just come from a talk or you've at least done one today yes yes yes just uh 30 minutes ago so fantastic what was that all about so it it was more of a keynote uh style thing where um and it was the the the topic was harmonizing art and technology um is it possible is it is it an naturual goal or should it stay on balance and so on and doing that uh going back the history of video gam so 50 years ago up to today and having a look an Outlook at the future what it what it may be in the future so uh yeah it was entertaining people were engaged and uh that's Well audio is what you're all about so audiio kinetic tell us what is it that that is what what you do so audio kinetic is the is the sound engine

that will be part of a game engine so to to make game you need a game engines that control all the systems together so animation display Ai and and you need sound so why is this a sound engine that is meant to be integrated into game engines and there's anothering application for composers and sound designers to integrate their content to set the right behaviors the right sound properties on those sounds so that later when those sounds are triggered by the game they will play in the right manner the right fashion they will be mixed correctly so you hear the right sounds of the right model that kind of thing Goa so people might be familiar with something called a door digital audio workstation where you know you might create and mix sounds together yeah uh how does that differ from a game engine where in the chain do these two things come yeah that's an excellent question so so the um are mainly main for linear production right you use a DOT to create music or to create the sound score for a film that kind of thing where where it's

linear and in the process dos are used to create assets to create you know dialogue and clean those dialoges and music and ambients and all that and then you output those sounds you render those sounds as wave file and you integrate those audio files into wise and that's where it starts now in Wise that's when you add your behaviors that's when you add your thing things like for example distance atation you know if you do a fill you know exactly when the character goes away and you can fade out the footsteps but when you're in a game you don't know how close you're going to be from that character and so on so you need basically the game to tell what is the distance of that character with regards to the microphone capturing those sounds and then the artist will mix the distance attenuation curves like how much volume Decay over time how much um low pass filter can be applied High Pass filters and you know all those fancy stuff so that you got something believable the illusion of a threedimensional space exactly and

because a million different player playing the same game they will not be you know they will not have exactly the same soundtrack they might never get close to a certain uh character walking for example another can follow them and so on so all the mixing aspects for game is done dynamically is done at run time so that's why before the game is run that's why you need a tool likewise so that you can set okay how far should I'm hearing some characters of that size for example and if there's a you know a big waterfall and so on if you want to hear it from afar and as you get closer that waterfall will extend and being you know more present and ominous maybe so those things have to be created in advance and you never know exactly how long and how close they will get to so do you primarily work with mobile and mobile games or do you work with any and all is the platform used everywhere so so game developers

like the budget you invest into making games is so important that typically game developers will try to Port their games on as many platform as they can right so so there will be mobile game developers where typically they will do Android and and iOS you know but if you are on the console side of things it's quite possible that you will release on the Nintendo the Playstation Xbox on PC on Mac maybe on mobile platforms as well just to get your game on as many platform as possible and reduce your Financial Risk right so that's why wise literally supports all the gaming platform it's really essential that we do that cuz cuz sound has changed in in so much as you know mono was a thing back in the day a single transistor radio with one speaker then you get this stereo so you can suddenly pan left and right and now people have you know Atmos surround sound systems in their homes where you can get elevation and you can get you know an entire three genuine 360 experience so when you're mixing something for a small iPhone speaker

which may or may not even be in Stereo it's more likely to be stereo Now versus doing a mix for something that supports round sound is it a very different production workflow or can you bake it all in and then it can just you know depending on what it plugs into it just recognizes and changes the mix yeah it's a good question and and again the concept of baking stuff for games doesn't really apply well uh for for the same reason I was mentioning before you never know exactly when where will be those objects when will they be tried and so on and you will have that game being played on so many different platforms so many different sound systems so the best practice or the way the way it works with wise is you're actually mixing around a listener so you're mixing your sound in a spherical thing right so it's not even 7.14 like Doby atmas which is a hemisphere where we're mixing on a sphere with the highest Precision possible and then we Tred to keep that Precision down to the Final Mix and we send to the endpoint so

to the console or the phone that information and where we are at today so back in the days you know when we started recording albums it was mono eventually we had stereos and so on um so you were mixing in a studio with the best environment possible now we're at a stage where the endpoint knows best about the environment and the endpoint will be your phone or your console or your PC or Mac so literally what we're doing with wise is you're you're trying to send all those objects with all their 3D coordinates and all the sub mix and you know all the content you send that to the endpoint and the endpoint knows how to play that content it knows that oh currently that person is wearing headphones no so let's do a biner all mixed with all those position uh distinct and with independent streams then we can do like really better ball effects hellblade being the example of a console game which really EMP Iz that

binaural effect with putting headphones on they recommended that you did it for the 3D SP yeah yeah exactly and that's a game using wise good to know good to know well that's that's a good sort of showcase for it there a lot of people when I you know when you show it to folks they kind of freak out look over their shoulders and you know that the effect is working well exactly so this is something that you plug into existing engines you download an additional is that how it works yes so so you know the the there are commercial engines like unity and unreal for example and godo now and and and a lot of and there's still a lot of proprietary game engine some game companies develop their own game engines so why has this been designed to be easily integrated into any kind of of game engines but because of the ubiquity of unity and not real we are also providing integration for these engine just to simplify the work for the developers and you know having them more rapidly able to integrate their content and and get and work on making the best the best

experience possible it seems like in some ways and this is happening with Graphics a little bit when you hit a plateau when it's like you can't really improve the thing that you're doing anymore cuz the jump from mono to entirely 360 spatial audio is huge but once you've kind of conquered that how how do you improve that you know it does it become about efficiency does it become about Clarity or quality how do you make it better for me it's uh I think it's uh it it there's not a single Silver Bullet for that right so it's a combination of having more clarity with your mix and again it's Dynamic you you're not you don't know exactly what will happen at any given moment so it's about building the mixing rules in a way that if you have a critical dialogue to come in it has to dug the music down and the explosion and so on so so you can capture that thing so so Dynamic uh mixing is really important another aspect that is emerging more and more is to leverage the geometry around around

you to do Dynamic early Reflections and to have sound propagation you know bending around walls or going through the walls and things like that a crude example of those Atmos speakers that fire off the ceiling it could be to render that but but this is at the end of the chain where you render you know on that emisphere for Dolby at mus but you know before that if you if the gain if the gain sends to wise where the walls and the openings are now we can calculate you know how a sound an emitter from outside can go through a window within the game geometry with the game directly and you know the effect when you let's say if you have an object in the in your doorway for example so you're going to hear it clear but as soon as you move and you don't see that object there's some sort of low pass filtering you know happening on you so those type of things can be created and automatically trigger so that enhance the the realism that enhance the the feeling of being present in the room I'm

going to say something gross now but to explain it to graphics people kind of like R tracing in the sense that it follows the the actual path of the light and it's doing the same thing to sound for dynamic ear reflection we're using R tracing for that yeah and we uh much less than for graphics because we can get away with it to a certain point audio doesn't need that level of precision to render something that is really convincing but yes we're using R tracing and other techniques to optimize you know the memory usage the CPU usage because it's all in the end it's all running you know on on on a mobile phone or on console and we're all sharing the same resources so doing arts or doing sound you're kind of sharing the same CPU the same memory and and so on [Applause] [Music]