Data Driven Granular Synthesis for Video Games

[Music] um i'm really glad i can be here actually i thought this time i might not make it through immigration because my pakistani name but i did and well this is the first time for me here gdc it's been really enjoyable so far and yeah i hope you'll enjoy this talk as much as i've enjoyed all the talks that i've been to but before i start to talk about data-driven granular synthesis i need to oh yeah first of all i need to remind you to turn your cell phones on after the talk and um before i start to talk about data-driven granular synthesis um i just want to introduce myself really shortly short my name is sujat siddique i work as an audio researcher at square enix in beautiful tokyo in japan

and yeah so audio researcher i've been asked by lots of people what does it actually mean and well to tell you the truth i'm really bad at sound design and i think that's not only because my ears are really bad but also because i think it's more because of my bad taste and i'm also really bad at programming so people people who see my code laugh at me people who want to use my code rewrite it so calling myself an audio researcher just turns this um how to say deficiencies into maybe well into charming qualities at least that's what i like to believe past projects that i've been working on as a researcher are many projects where i do audio synthesis for video games for example i've been working on the car engine sound for final fantasy xv i've been also working on the sound of

destruction for some other video games given some presentation about that at the computer entertainment developers conference in yokohama and japan i've been also working on morphing of sounds and also morphing of granular sounds the method that i'm going to talk about today and just to be sure that we're all on the same page because i know the title title can be a bit cryptical so data-driven granular synthesis is well to explain it really shortly it's a sound synthesis method that can be used um for video games and it's based on small waveforms and those small wave forms are called grains and that's why we call it granular synthesis right i'm going to explain that in more detail later of course but before i do so i also want to let you know what sounds you can actually use it for so that the talk maybe gets a bit more meaning for everybody my favorite sound and my favorite example is the sound of rain

i think the method can really like explain very easily when you when you think about rain so that's why i'm going to use rain also in this presentation but you can also use it for the sound of running water like like a small stream or something or like running water into a basin or something like that it can also be used for different sounds like flooding clothes or paper or something like that for correcting fire and lots of other sounds you'll hear some examples later on now if we talk about the sound of rain then imagine you have a video game where you where you want to have the sound of rain and now of course you um know that there are lots of different brain sounds and the thing that you probably have to do in the beginning is to decide what kind of rain you want to have in your video game and if you use the traditional approach to implement the rain sound then you would go with samples that you play in a loop and those samples will have to have a certain length because you don't want

the player to notice that they're in the loop right so like i don't know you're talking maybe 20 seconds or something like that and um yeah even if you decide that you're only going to do to only going to use two different sounds you still need those two long samples for your video game and that means if you use high quality and samples that are like about 20 seconds long you end up spending around 7 megabytes per sample now of course we as audio people know that this is nothing in comparison to what graphics people are using right but still at my company i've been told this is too much we need to reduce this so yeah this was one of the problems that i think it's not a problem because i think we should be given more memory oh sorry but um there's another thing that i think is a problem with this approach and that is the transitions for example between different signs of terrain between different sounds of rain or it's really hard to apply some

variation to those samples you can of course do fading but if you just apply fading you are not going to be able to have very natural sounding transitions or for example if you want to start the sound of rain during the game how would you make that just fade in the sound well maybe that will do but it doesn't sound very natural right so to summarize the um challenges or the motivation behind the project that i've been working on it's because we looked into ways to save some memory and i wanted to have something where we can like have lots of variation and play around with the sounds and yeah change the way it sounds during the game so the solution that i'm proposing is data-driven granular synthesis and yeah well first of all i want to explain granular synthesis in a bit more detail

i guess lots of you are familiar with granular synthesis like raise your hand who knows the term who's been working with it cool okay yeah well definitions of granular synthesis it's a very flexible method a very powerful method and that's why definitions vary a lot like between different people and between different implementations or ways to use it and yeah it's a bit difficult to find a definition that fits fits all of the usages of of general synthesis but the definition that i came up with is granular synthesis is a synthesis method used to synthesize granular sounds i do have to admit this that this definition is absolutely useless because if you don't understand granular synthesis then of course you don't understand granular sound right so let's talk about granular sounds think about the sound of rain the sound of rain is actually the thumb of a lot of

raindrops falling onto surface and like we're talking about 100 thousands of raindrops or something like that right and those raindrops converge into a single sound and the reason why i call this single sound or i call this sound of rain a granular sound is because those single sound events like in the case of rain that would be the drops hitting the surface can be called in the context of granular synthesis they can be called grains and a granular sound consists of grains so a granular sound is a sound that consists of a lot of single sound events in the case of water it would be air bubbles trapped in the water that are resonating at the resonance frequencies this may be an example that's less straightforward to understand but also in the case of water it's actually like the sound of water is or consists of a lot of single sound events that are also quite similar to each other

to synthesize such sounds we use granular synthesis and the method is actually really quite simple because all that we do is we need recordings or we need waveforms of those single sound events and we just take a lot of them and we mix them together in practice of course because we have limited memory and we cannot like record an unlimited unlimited number of sounds we limit the number of grains to say for example a few hundreds onto and to increase the variation during synthesis what we're doing is we're applying random attenuation to them so every grain would be like um once it's played like at full intensity and then the next time it's going to be used maybe at half of the intensity and yeah so on b variate the intensity randomly and then we place them like the attenuated grains we place them into the final mix at random points in time

and we do that for a certain number of grains per second and like for example for heavy rain we would use lots of frames per second say what like 100 000 and for very light rain reduce i don't know only like 100 or 1000 grains a second or something like that the random attenuation that we are applying to the sound is um very important in giving the sound a certain character and for example in the case of rain we would use or our mix would look like this we have like here on the horizontal axis you have time and on the vertical axis you have the amplitude of the grains so that's sort of the random attenuation that we are applying to the grains you have a lot of grains with low amplitude and you have only few a few grains with a high amplitude the reason for that is or the reason why this gives us a natural sound in the case of rain is because if you think of the

sound of rain there are only a few raindrops hitting the surface close to you but the number of raindrops hitting the surface far away from you of course the surface is a larger area so that number is huge and this is why there are more grain or we need more grains with the low amplitude and um one interesting thing about sounds like this sounds of rain and other natural sounds is the fact that actually those grains with a low amplitude all converge to noise so we can update our granular synthesis and instead of taking a huge number of grains per second that we have to place into the mix we can also just reduce the number of grains per second that we put into the mix and add some noise in the end that is um that gives us an optimized

version of granular synthesis which runs much much faster and the sound quality is still the same so yeah for video games that's i guess the most intelligent approach yeah but now of course the question where do we get those grains from where do we get those single sound events that we need for granular synthesis from like for rain where do we get the sound of raindrops hitting the surface from of course what we can do what we can try to do is to record those sounds i've tried that and i failed but i have to admit i'm not really good at recording so i'm sure that if you do it the proper way maybe you're able to catch the sound of those raindrops if you have a nice stud studio and you do good setup and stuff like that but also um if you go for recording you should keep in mind that different

like rainfalls on different materials so you have to do like recordings for different surface materials and stuff like that and maybe drop sizes can be different and stuff like that so you have to um well i guess recording is actually a lot of work and also if you want to go with the recording recording approach then you still need to know or you still need to yeah you still need to know what your noise looks like if you want to use the optimized granular synthesis what kind of noise do you put in the output to like simulate all those raindrops that fall on the surface very far away from you and the most difficult thing maybe is to find a good random attenuation distribution so i told you before that lots of rain jobs need to have a small amplitude and if only a few raindrops need to have a high amplitude but what exactly is the ratio between those two so you still need to like to i guess

just experiment and figure it out by playing around with the parameters um yeah i've tried that and i found it very well it did work and um i also got like i i love sound synthesis so what i did i i managed to synthesize the sound of raindrops falling onto the surface and with that i actually got quite nice results like i got sound that was sounding well decent i guess quite natural but the problem was for the synthesis of the grains and also for all the other parameters of the granular synthesis i had so many parameters that when i showed it to the sound designers in my company they were just like um what what is this parameter for and i tried to explain it to them but they were so abstract that it was actually really hard to explain and i guess if you didn't build the system yourself then it was really hard to use so i realized that's not a good way to to go and then i thought we already have

all these beautiful samples of rain sounds lying around in our service and we should make use of them so what i try to do is extract all these data that we need for granular synthesis from existing recordings and that actually worked quite well and this is why well this is what the presentation today is about data data-driven granular synthesis so we take data and that data is in the form of a recording of existing rain sound for example and we extract all the data that we need for granular synthesis from that sound the method that we're using to extract the grains and the noise and all the other parameters um well i'd love to talk about it in detail unfortunately i won't have much time for that so just to give you a quick summary or like um yeah a quick well a quick summary of

how it's working first of all we estimate the noise in the recording of the range sound we do that by cutting the sound into frames then we convert all the frames to the frequency domain so we get the spectra then we look at the spectra and of course the loudness will be different between all the different frames and we estimate the noise by looking at say um just the 15 or those frames with the lowest amplitude with the lowest energy because we just say that okay maybe in these frames we just have the noise there are no like loud drops in those frames this is what gives us the estimated noise spectrum and then we subtract that noise from the original sound this is a technique called spectral subtraction that's i guess very commonly known in

signal processing and after doing so we are left with the clean signal it's not really clean but let's say the signal with reduced noise and we kind of we cut out the grains just from the loudest frames that we have in the signal this again is actually quite a simplistic approach but i found it to work quite well actually so we just we do it in a greedy manner like we just look for the loudest point in the waveform or in the spectrogram and this is our first grain we cut that out and then we continue to look for the second loudest and so on and well the problem is that of course if there are a lot of grains per second then of course those grains are overlapping right so if you cut out part of a grain then the tail of the grain is buried under the next grains so it's

not possible or it's not easy to get the tail of the grain out of the recording so what we are going or what we are doing is we estimate the tail of the extracted grain if you want to more to know more details about the methods that we're using then yeah as i said i don't have much time today but please have a look at these two um well the first one is not published yet i'm going to do a presentation at the ace convention berlin about the technical details of the method but there's already paper which is um well it does not contain the estimation of the tail for example so it's well it's not the most up-to-date version of the algorithm but still if you look at it you get an idea of how it's working and also if you want to know more you can drop me an email i'll i'll show you my email address later and maybe i can also send you a

version of the new paper you can find the second paper on the website of my company so yeah feel free to have a look at it okay now of course we want to know what it sounds like so well i've some sounds here but actually i think i'm going to jump straight to the demo so here i've already loaded the sound of rain this is the recording and now we're going to do an analysis so in this analysis first of all we estimate the noise and now we're extracting the grains that take some time because like we have to convert all the frames to the frequency domain then we have to estimate the tails and stuff like that so yeah we need some um processing time for that and the next thing that we're or that this program is trying to do it's trying to find an optimal

configuration for the loudness parameters of of the grains that are placed randomly into the mix and the optimal configuration is supposed to be a configuration that gives us a sound that sounds as close to the original as possible so it's still running but with the parameters that we've already found sound is like this and it's still trying to optimize it further so these values are going to change but i don't want to wait for that so i think we're just going to play around with the values ourselves for example i can increase the number of frames per second or the amount of noise in sound or the parameters of the loudness

and then yeah i was talking about transitions before so for transitions um what you can do you can create different configurations of these parameters here and then place them onto this map and then you can sort of morph between different configurations let's try that okay maybe less noise

yeah and all these like different variations of the sound can be created from just just a few seconds of um recording that we're using like i think here i use like five seconds or something like that okay let's try a different sound for example oh i like the paper one let's start the analyzes this is what the recording sounds like [Applause] and now oh yeah well this the sounds that i'm using um that i'm

analyzing for um to get the grains and everything they're mono sounds but since we place the grains into the mix randomly we can actually also apply random panning to the sounds which means that even though the source sound is a mono sound we can create sounds with an arbitrary number of channels by just applying some random panning to the grains that we place into the mix okay let's listen to this sound the paper [Applause] and now my idea was that for example here you have a slider that goes from zero to one you could attach that to a parameter in the game where i don't know you have for example some movement of some thing with with lots of paper on it or

something and then you just go from movement is zero or very fast stuff like that right okay let me go back to the presentation in powerpoint there are well i've already of course i've talked a lot about the advantages of the technique but and well there the flexibility i've already told you can use an arbitrary number of channels we also need less memory [Music] so 2.5 megabytes is really the maximum that we need but actually we can without an audible loss in quality it depends on the sound that we're going to synthesize but we can reduce it to one megabyte or even less because the size of the grain that or the size of the grains like length of the grains well for some sounds they can be shorter and for some sounds they have to be a bit longer but for example for the sound

of rain it doesn't matter if they're really short so for the sound of rain i think i could i could shrink it down to like 500 kilobytes or something like that and unfortunately there's also some disadvantages i probably can guess that because um we are synthesizing the sound in real time so we need to we need some processing power i think it's not too slow it's it takes us about 19 milliseconds to generate one second of high quality stereo audio so yeah it does run in real time easily you've seen it in the demo yeah this um nearly concludes my talk but yeah future plans the first one i have to warn you is a bit embarrassing i actually want to use it in video games i think everybody hears gdc has lots of experience to use stuff like in video

games and i'm talking about a technique that's not in any ship titled yet but we are working on it i cannot tell you any more details but i hope that you're going to see some stuff soon you've seen the analyzes process take some time of course i want to speed i want to accelerate that and there's also some scope of improvement for the grain extraction algorithm but yeah those i leave for the future so yeah this concludes my talk thank you so much for your attention if you thank you if you want to get in touch here's my email address drop me a line anytime um yeah well i think we have some time for questions so if there's anybody who wants to ask something yeah please there's a mic here step up to the mic

hey thanks for the talk thank you my main question was you parameterize gain and variance for the gain for your grains so are you just applying a gain to your recordings or do you actually pick from the data loud and quiet grains and select them based on a sorting or something like that no i'm applying the gain only to the grain so i'm not sorting the grains but actually you make a very good point because by applying a high gain to grains that had a low gain in the original sound you sort of distort it in some ways maybe some of you heard that and that is actually a very intelligent idea like to sort the grains um in terms of loudness and then pick the loud grains for where you want to have loud grains and the range with small amplitude where you want to have grains with a low amplitude that will guess i guess make a sound it sounds even more natural yeah cool so i was going to recommend this thing by air cam called cathart oh okay yeah i know qatar it's like a 2d plane system yes let's use it

it's wonderful i love katard all right thanks thank you hi thank you for your talk is great um i wanted to ask uh how long did it take you to develop this and how big was the team that was working on the team working on it is they're all here because it's just me um it took me not too long actually so you've seen that i published one paper about it in 2015 but yeah don't think that i've been working only on this since 2000 and whatever 15 until now like there's lots of different projects ongoing like going on at the same time i think if i add up all the work that i did for this it would be like three months or something like that but i was really also playing around a lot with like different parameters for extraction and stuff like that cool thank you thank you hey thank you so much for the talk thank you um so when you're implementing these into a game are you writing your own

tools for that or is it working as like a plug-in the plan is that so that i develop it as a library and that library is sort of in some ways no it's not really plugged in so yeah it's more like writing on tools for that okay thank you thank you any more questions please step up to the mic all right hi terrific demo and uh explanation um you you touched upon uh different materials rain falling on different materials obviously starting with small samples on the different materials would allow you to localize you know a pond and you're under an awning and pavement etc do you foresee in the future being able to somehow model or use some other type of synthesis or

convolution to pro to create that in real time as well i actually did something like that because i think i mentioned before i was actually first doing synthesis of the sound of for the sound of the raindrops right and when i was doing that i was doing convolution like with the sort of impulse that hits the surface and there could be different surfaces and i different impulse responses for different materials and stuff like that i did play around with that but the problem is most of the time we actually i think it's a terrific idea i think it's a terrific technique that we should use but also we need to start things slowly right so first of all try this and then like go on and yeah but i think it's a terrific idea cool thanks thank you yeah so if there's no more questions or well if you're too embarrassed to ask you can always drop me a line and yeah anyway time's up i guess if you want to talk to me i'll be around and yeah thanks again for coming

you