Peggle Blast: Big Concepts, Small Project

[Music] right hello audio peeps friends in GDC goers how's everybody this morning making it here all right good deal well today we we plan on taking you on an audio adventure of sorts an archetyp archetypal hero's journey a journey packed with well Adventure excitement risk but also fraught with perils and trials and adversity and finally overcoming the odds not with brute force or anguish but with finesse and fresh as well as recycled ideas the setup is an all too familiar scenario for audio teams the world over lack of resources lack of understanding lack of love so let me introduce you to the Intrepid trekers on this journey we have

Jacqueline shoem mate audio lead sound creation wizard able to Jedi mind trick even the most stubborn producer we have RJ maddingley technical sound designer and scripting ninja whose audio systems and logic are making us wonder if those cybernetic enhancements were too much I'm guy Whitmore composer musical Trickster dreamer and disruptor who surrounds himself with awesome people to make myself look [Music] good the team is rounded out by uh Becky Allen sound designer Ace and organizer extraordinaire Damen cbow wise Savant and smoke jumper and a cameo appearance by Mike cisel on sound design there he is right on the whole team's here we are not going to waste any time I'm going to

jump right into some gam play and uh now the basic idea in pegle is that you shoot a little metal ball and you try to hit as many pegs and clear them uh as you know clear them especially the orange ones now there's a difference in mechanic between the earlier pegels and pegle blast and we had to do some different design here here's our meta map and um here we just set the mood we're just making it sound Pleasant you know here you are and a little soundseat air to get you moving and we're going to go to a turtle level here you know one of our tenets for all of our games is that things move and transition smoothly from one area to another this one of our overall [Music] HS so here he's starting the game let's see what he's going to do [Music]

nice shot yeah notice how the bumper was going up and Pitch as it was hitting little tension Builder [Music] there Turtle now has voice over an addition we did after launch so his special feature is that I can catch the ball yeah every master has their own power oh good his voices oh yeah [Music]

all right not all shots can be excellent let's do a fireball all right Fireball there's these powers that you get and it just clears path for you [Music] so notice how the music is progressing as you're moving along here okay one peg left it always comes down to the tension shot and going to get it no okay near Miss we got plenty of opportunity here

all right so there you have it and now we're going to jump over here continue on our story The Adventure so to set the stage for you we were given a 5 megabyte total asset limit for our audio footprint on a mobile platform to give you some comparison we had just shipped pegle 2 with 783 megabytes on Xbox One um wow yeah thanks why 5 megabytes why 5 megabytes guy well the game as a total was only shipping with 50 megabytes so we got five um it's

common for Freeman games and we were given a really hard limit with no wiggle room so we had to make it work had to do with download not Ram or any other issues right exactly it was the first unity game from the studio which meant that we got extremely limited Dev support resources were put in other directions um in addition we really believe that Mobile Audio should be and can sound good and we had pegal two sized expectations for the project um I'm a sound designer I really like to make content and I need space like I just take up a lot of room so this was one option we could have made a 30second sound Loop five sound effects ship it and went on vacation but then we heard a knock at the door it was Igor Stravinsky and he said The more constraints one imposes the more one frees oneself and the arbitrariness of these constraints serves only to maintain the Precision of execution

wow did we feel that on this title um and in the end I really feel that our limitations actually gave us Focus purpose and a drive to innovate and I I really don't think our aesthetic uh I think our aesthetic would have been wildly different had we not had these constraints and it probably wouldn't have matched the game as well either go ahead so to start out with we uh looked to our mentors our inspirations I know that some of you recognize this sound and this one and I don't even have to tell you what games they're from right they're fantastic audio so I looked to classic games and slot machines too actually for examples of audio that convey a lot of emotional resonance with a very small footprint yeah and for my part uh when I got the 5 megabyte memo um I actually thought small what is small that could still be

compelling and a music box came to mind and uh so I wrote this demo piece to kind of try it out you I mean you could almost fit a music box in an iPhone and think about it so I wanted the idea you're fing this little little thing that's marionette like so we put it in the game and the game team kind of had a complete freak out we loved it yeah we loved it we we we had a sof SWAT for it but the game team didn't like it they we they wanted the orchestra back they wanted you know o to and all of its Glory uh so we had to readdress that but uh that's part of the exploration process yeah totally um so from a technical standpoint uh even though pegle blast uh is much different from pegle 2 in a lot of ways obviously there are a decent number of similarities so the first thing that I did was kind of take a look at how audio worked in pegle 2 um and tried to find out how we could optimize some of the functionality for the mobile platform obviously we had much stricter um guidelines that we had to fit within so try to figure out how

we could get everything that we wanted in there um and you know we started experimenting looking around at what we had available but uh but really it it came down we we just needed to make a decision commit and uh and move forward and and that decision there was a specific moment and it had to do with do we take on this midi technology that was in a beta version of wise and we knew that it was possible we might even have to ship with a beta version although we didn't uh so there was a Moment of Truth and boy do I remember that moment Crystal Clear sitting in a room talking about the options and finally deciding of course to take the red pill so we did we went down this journey and uh here's the rest of the story yeah and once we did that you know once we decided we were going to commit we went back and started seeing what this Tech would let us do um and we started experimenting with it and that kind of brought us back into the creative realm

of like oh we can do this and this and all these different things which once again brought in new technical challenges and we kind of found that that was this cyclical nature of our entire process was uh creative leading to Technical and back to creative just all the way through so I'm just going to give you a quick one run through of how we made sounds for pegle blast our question became how do you make sounds without taking up memory our daily challenge was make something for free what we ended up discovering was you can make the best of what you have on hand within wise we had World Time synthesis with sound seed air and impact the tone generator we had lots of dig Digital Signal processors creative playback behaviors and a tiny bit of memory to play with that we could use real wave files um very specially chosen assets so the basic process looked a little like this um

as you can see here you have your Sound Source which could be soundseat air impact or a tone generator the power of this is that all of these features can link into real-time parameter controls on your sound this is all within wise right this is all within wise you can add effects at any any layer that you want to and again link any aspects of the effects or not any but many into real-time parameter controls and then play around with different playback behaviors also within y's which also link into realtime parameter controls so basically it becomes layers of created sounds with various properties combined with layer of effects and with layers of playback instructions the individual elements are really simple but the combinations can get really complex to create a varied and interesting palette and those stay real time in the game right they don't you never mix them down right real time everything is real time time and that is the beauty of it

so now I'm going to give you my real time demo which is always a little dangerous in a um presentation but in the spirit of real time I felt like it was the right thing to do so within wise I am going to make a sound for you it's sort of set up already this is actually based off of a sound that Becky Allen created for the game there so the general workflow would be I would look in unity see what needs to be made and then I would get sort of a basic sound going um within y's that I thought would fit that and then adjust afterwards so here we have an explosion which is one of the real assets that we used So within that I can add um a swoosh let's just see what this is going to sound like and then I can make it squiggle a

little because that'll be fun it was all about playing and experimenting and trying new things and turn the frequency up sure that's pretty cool and then I'm going to also add um a sweep sound from tone generator I I've seen this tone generator and y's a lot but I'd never even thought about creating a sound with it because making Assets in a dah is really the way that we do things these days or the way that I do things these days I should say that's kind of cool except I want it to be really short and I want it to sound kind of angry so I'm going to change the duration and make it a a saw tooth wave cool and then you know what I'm going to add an effect why not um I think this is going to start all right no okay I'll take that um and then here I'm going to add an rtpc again

everything that I'm doing I can link into a real-time parameter control to any aspect of the sound at any point because it's all generated from within wise so here what I'm going to do is utilize pitch which is something we often played with I have a time this is this sound is for a little time bomb I have a counter I'm going to um have the pitch of that sound sound raise over time uh the more the time bomb is hit if I wanted to I could add another control um in rtpc into this uh just the tone generator sound and adjust the pitch of that as those as those time bombs go off as well so there's really just it's just so powerful there's so many options and then to um to audition it in unity and continue to twak my sound and actually make it match the game I could use profiling I've I've always used Prof filing in the past for mixing and for bug tracking and

things like that but never for actual asset Creation in the same way it's basically everything that I'm doing in game which is pretty awesome and so now I'm going to go back to Unity and let's say that um I thought that actually this sound should be a lot longer it wasn't doing what I wanted it to do I'm going to change that to 6 seconds so that the tail um sounds cooler we like cooler and there you can hear it was a little longer so that was basically the whole process was just going back and forth between [Music] those a it was really an Illuminating process to work this way there were a number of advantages first of all there was no Daw that was necessary I'm mad at Avid a lot

of the time so it was really nice to not have to spend that much time in Protools I didn't have to make game capture bring it into my um DAW and then spit out a sound and then rinse and repeat that that meant that iteration was really quick when an animator would change an asset and I would need to change my sound I could just go directly into wise change the timing of a few different layers of my sound effects and come out with something that matched the new animation animation um the biggest and most powerful part of this I really felt like was the easier implementation of sounds controlled by rtpc part of that was because of RJ which you'll hear about in a minute and part of that was just because everything was already with it in Wise you just had so much control over every sound and how it was playing with the game and it became this really malleable element of gameplay um it also then led me to feel like I had no barrier between me and the game because I was always working in the game I knew it was happening all of the time on this project I couldn't be as integrated with the team as I usually am and that was

really disappointing but because I was so integrated in the project and working within the game I didn't feel that disconnect at all it was really fantastic actually and um more than anything I feel like there's so much potential to use this on games big and small and none of that I would have discovered had I not been faced with this limit to begin with there were some challenges um there was a limited palette I really wanted to work with more scents and just have more things available having a bigger memory footprint would have helped as well so we could use more wave files um I wanted more real-time mastering tools and the ability to use more realtime mastering tools with more CPU limited CPU meant that I had to be a little careful with some of that and voice limiting was tight because instead of playing one wave file at a s at a time which maybe has three sounds or four sounds that you've created in your dah um that takes up one voice with these maybe there was eight voices in one sound at one time so we had to be really careful about how we

were designing assets but all in all it was a really fantastic way to work and I look forward to using it on more projects which takes us to the implementation side of things um so as guy mentioned we we really we were lucky enough to have a lot of control and autonomy um kind of across the Spectrum for audio for this project from creating sounds to getting them in and creating our own systems um and it was really great I've heard a couple of other talks already this week about that kind of style and bringing more control into the audio team's hand and I think it's a a really great direction to be headed um so just real quick what enabled us to do what we did for the project uh we used unities uh our main engine was Unity um and in case anybody's unfamiliar with that it's it's a very game objectoriented and drag scripts onto um different objects kind of um Pipeline and uh and this was great it really kind of blurred the lines between what people

call like deep engine level code and the kind of highle scripting that people are familiar with doing an audio um and that was that was really crucial it enabled us to go Way Beyond just symbol play sound events um on again on our own we able to create and Implement entire audio systems from beginning to end uh and then manage those systems within our own code that we didn't have to burden the rest of the team with um a real quick example of that I remember uh kind of early on in the project Jaclyn and I were talking about um one of the character special abil abilities beorn he has this super guide kind of a rainbowy thing that helps you aim um and we were thinking wouldn't it be cool instead of just having you know obviously a static looping sound if we had it kind of dynamically react to uh the player swiping uh movement and um and it it seemed like a a great logical way to go oh I didn't even go on that okay anyways um so I started digging around and seeing how we could do that and sure enough uh within the game we had this object called the shooter

object um which turns out was that cannon that you control um so once I found that Unity has a lot of buil-in functionality like the get rotation call um and we were able to just get the position of get the rotational position of that um and map that to a variable uh and then getting the speed of that was as simple as measuring the change of that uh position over time uh and then we just ma that to an rtpc value down there and since we had the position in order to make the speed anyways we might as well put that as its own rtpc value because rtps are are great and they come in handy all over the place I think we ended up using the position to do some left right panning on top of that sound effect yeah one one thing I want to throw in here this was all done outside of the team scrum the team feature list and all that we just decided hey wouldn't this be cool walked down the hall made it happen it's in the game the game team goes wow that's neat yeah no friction yeah it's a really

relatively small feature and it's nice to not have to make a big deal out of it to be able to just I mean as you can see it's a few lines of code pretty easy to put in um and if I recall correctly my cisel actually ended up making this sound so if you hear it and it sounds especially awesome that's why um but uh so it quickly became apparent throughout during the project that with this great power came a a pretty easy ability to overstep our bounds um as we offloaded the memory sounds and put them kind of in the cpu's hands um we we found ourselves kind of doing too much without uh without holding without keeping anything back um and so we started looking for ways to come up with uh some performance improvements to try and get those voices down and um over some conversations with a couple sound designers from other EA Studios Nick lamartina and Nick Von canel um they kind of pointed Us in the

direction of this like multi-tiered quality device um idea where you know if you're playing on an iPhone 4 it obviously has much less capabilities than say an iPhone 6 and we wanted to try and make sure that each device was getting the the best sound that we could get out of it um so we created this audio quality manager that just looked um at startup about uh what the what the device was and what its specs were and we mapped that into an rtpc for kind of a low medium high quality setting and from there we were able to do things like mute or kill um some of the less crucial sound events as jacn was showing you some sounds could some sound events could actually be made up of maybe like 10 sounds in a blend container um which on an iPhone 6 would be great but as you get into a four it's uh maybe we' make a chug so we kind of said okay well if you could do without two or three of these would you still be able to get the point across and we were able to mute those and and uh just save some CPU we can do things like disable aable or swap out effects plugins we had two different

reverbs um one for low devices and one for the medium and high with just some more conservative settings uh and then the obvious one changing bus level voice limits I think on the on the the high tier we were allowing maybe like 50 voices for sound effects but as you get down to the low quality we had to clamp it down to maybe 15 or so um so that was pretty effective yeah that was really really something otherwise I think low-end Androids would have been our least common denominator and everything would have had to have been at that bar despite the iPhone 6 abilities really be able to take advantage of what each device could do um and then throughout the project one of the biggest things um as a lot of people will say was the the ability to Performance monitor in real time on device um we could do things with the wise profiler like measure the CPU of each effects plugin as as Jackin was showing you we used a lot of effects all over the place and it become it came important to find out which ones were kind of taking more CPU than we could spare um and get those clamped down um

also we we were we used a lot of like really really short looping sounds that uh we found if we had you know a 0.01 second sound you know looping within 1 second you've got 50 voices that are stacking up and uh so the the profiler was able to help us find those and take care of them like right away hey RJ what did you do for easy auditioning of all of these sounds across different specs oh that was a great question um yeah so you know since we were doing this to test different device uh settings we wanted to come up with a way that you wouldn't have to test on an iPhone 4 and an iPhone 6 each each time you made a sound obviously at some point you would be doing that but right away we wanted a way to kind of simulate what those settings would end up looking like so within the unity editor we just kind of created a a fake uh version of that rtpc so that you can in real time change it to low medium and high and see which sounds are getting cut off and make sure that it's still sounding like what you

want um and that is pretty much the main overview of uh implementation the text side of stuff so awesome thanks RJ um so now we're going to talk about the music side score the scoring aspect not and we all participated um and I have to say uh scoring pigle blast was [Music] definitely um and nice so I had a lot of fun you know imagine sitting in front of a professional choir going it's pronounced pegal ises you know so we know how to have fun here so the first part we had to think through I did was the pallet a lot of times when you're starting a score what sounds do I want what what tamber do I go for and this image is apt because that's uh this is Aaron midor uh art exploration of Bjorn and you know what colors do we use what shade and all that um and once it was decided we were going with midi and Sample Banks uh

there were a couple of elements that became critical to helping decide this so those three points that that I decided to work with are custom sample Banks and instrument choice and real-time DSP now custom sample Banks over pre pre-created ones which don't even exist on wise right now uh have an advantage in that by myself creating them recording the the the sounds and and editing them myself I can tailor these sounds specifically to the score I want for pegle blast they're not generic um samples like you would find in a in a typical Library um and also instrument Choice was important too after it was decided Well we really need to bring back the orchestral sounds how do I do that in you know part a section of 5 megabytes um one of the tricks there was to use a lot of uh shorter sounds that were staccato or Marcato or picado plucked sounds Mallet sounds all of which have a lot a character with their attack but are very short samples uh

that combined with short Loops created a sample bank that was really really efficient um and then real-time DSP any is critical for anything uh that has multiple sounds being mixed into one so for the music piping it into a real-time Reverb bus allowed uh the whole thing to kind of come into the same world the same sphere and uh so here we're going to take a quick look at the pallet on here you can see basically that's my full uh set of sample banks for the music there you can see the string stuff the music box the choir uh clarinet brass tany Etc the Opera bat which is Becky doing her Opera vocal and a little a little bit of syn there which you heard in the beginning um and here's a setup for essentially how a sample bank is made in Wise and you can see after I imported the string tocados here um you know you set up their their midi tracking note and you can see there that uh there are

two notes per octave and I either did two samples per octave or three samples Proactive or one sample proactive because proactive that's all I could afford uh but it worked out really well single layer uh sample Banks uh worked really nicely in this case and over here are the peg hit sample Banks harp is the general Peg hit that comes across everything and I chose harp partly because I knew there' be these nice glisses across pegs that would sound like a harp um and we also had a a sound effect element that just gave it a little physical impact no matter what happened there and then the marimba uh gets layered on if you hit an orange Peg because orange pegs are more important to the player so we wanted something to accent that and over here uh we're looking at the rtpc that are applied to Any Given sample bank or for the peg hits here we have pan left and right so if a peg happens on the left you hear it on the left Etc uh transposition which RJ will

get to in a minute uh voice volume which is your adsr envelope and um and velocity sensitivity you did all that with rtpc is here that's how they're built so it looks like wise but it's a sample Bank here's a quick look at the you know attac Decay sustained release that you can build for any given sample bank so the notion of a spotting session is something that's probably familiar to folks who've done film scoring television work and linear media this is something I found uh you know the Planet of the Apes Jerry gold smith plan of the Apes thing here and you can see each Q has a very very specific timing uh lists the the instrumentation and so forth and in his case even the arranger that's going to be working on it um and it's really common to see film Q sheets go kind of in a a spreadsheet like fashion like this but what about a game and games aren't linear so we can't do that we have to have something that's more Dynamic and a spreadsheet will not work for this so um the elements I needed to consider

were harmonic flow the section flow and the progression the range of gameplay fast players versus slow players Etc and Stingers or which are now Stingers are accents that I can drop in over the top of of the music anywhere so those are the elements and here's what my spotting session looked like on our whiteboard and you notice it's not linear it's circular in a way um and start up here The Meta map it was where you start the game you enter the game here it's in the key of A so that's part of my uh harmonic map and loading is also in the key of A and as after it loads here's the first thing you encounter is the this section of Music in D here now I I wanted again to treat uh each uh Pig I'm sorry each shot each shot of the ball as kind of a new scene cuz you know the player spends time they set up a shot they shoot it and then

okay there's another thing now a common technique in the linear world is to do kind of harmonic shift over the course of a scene change either right on frame or just after frame or preceding it and so I wanted to mimic that here so every time there's a shot it moves to the next uh thing around the circle so this circle represents um a gameplay Loop and it moves uh clockwise here and rotates around they keep going around and you can't predict how how many shots are going to be fired in any given game sometimes it's five sometimes it is 12 sometimes it's 24 but this uh circular form allows it to kind of keep continuing you'll see it's a circle of fifths so the progression can kind of just keep going there now that can lead at any point from any of these uh sections of Music here can lead out to uh some of these completion scenarios and starting with last ball or last Peg um not listed but they're both in the key of A and the reason is is that fever

is in D so I've got my tonic dominant relationship there when I end up hitting fever but you can also go to What's called a point of loss you you don't get that last Peg and I want to hold you there for a little bit uh while they you make a purchase decision whether to go back into the game or to just leave the game um so there's these these points are are critical in that the player can sit there and Ponder and you have to think about that and there's not a fixed amount of time so they have to kind of continue before moving on um and you know the goal is to make that as smooth as possible so last Peg you hit fever bada boom it's all great you say yes Next Level it will load the next level and start at the top again and there's your Loop so any game has a a game Flow or what a lot of times in the business now is called the game Loop what's your game Loop you know so pay attention to what that is um and and make the designer your best friend so you can figure out what this is so you can map the music to that so compositional workflow um was

composing in Pro Tools uh for this project for the most part but by the end I ended up switching to nendo uh for various reasons and if you haven't seen it yet the nendo wise combo thing that they've been talking about is probably going to be really cool to see I haven't played with it yet but I can't wait because I've been experimenting with this kind of loop when I know what needs to happen um but the first step is I'd create a temp sample Bank in contact in nendo or Pro Tools and create music the oldfashioned way just kind of create all the sections and the cues I'd write all the cues here uh to gameplay captures so I could see the game at least uh and once that was kind of in a pretty good place um I would point the midi tracks in nendo to the custom sample banks that I built so treating wise as if it's a a sampler right next door and that allowed me to edit with all the fine tuning edit capabilities I have in nendo or Pro Tools and mix it make sure the velocity is working well and once that's sounding exactly the way I know it's going to

sound in game I export the midi tracks and import them into y's and then y's can take over from there and I can um take the MIDI files and arrange them into the segments and playlists as needed so that General workflow worked really well so by the end you were effectively using wise as your sampler yes wise was my sampler and here it is this is what wise looks like with midi looks like a piano roll so just like you would you can do some basic editing but not a lot yet I'm sure they'll be adding to that here in the profile you can see samples just flying by in real time I have my voice count I can see my uh CPU that I'm using I can see the routing and effects that I'm using so all this tracking that I can do in real time as I'm arranging and writing even when I'm piping in the midi from uh from nendo I can see all that stuff going on so it was really really valuable to have that um have that going on there so any given music segment

needed to be set up for the peg hits because as you know the peg hits drop down and play over the top of Any Given music and as you see there's 12 different key centers that are possible how do you do that well in any given um segment I need to consider the key Center the quality and the range now the specifics there have to do with these custom cues I would drop into y's and the key Center in this case is a FL the quality is mixian and the range is a midi range of 59 of sorry yeah 56 to 99 midi note values so I could drop those in at any point in the segment usually at the beginning and then that would tell the peg hit system which you'll hear about next uh what to do and with that uh RJ will take it from there cool yeah so to back out high level for a quick second um the uh the goal for uh the peg hit system what we had a couple of goals the first one we wanted to take

advantage of the midi since we were using it um elsewhere in the game um which inherently would allow us to use fewer wave files um in pegle 2 each music segment uh corresponded to a separate playlist of Peg hits uh with you know a onetoone wave file um corresponding to each Peg hit um and then that also help would caused a little more imp implementation time right every time a new music segment would come in you had to create a new uh playlist and then fill it with all of the appropriate pay kits um seems like it would take a lot of time um so we wanted to reduce some of the implementation time for that uh also the other consideration was pigle blast being a live service game we had no clue you know how much music I would have to write uh over the course of the game and we didn't want to have to keep going in and maintaining this system uh every time we added new content um and then the last thing was to bring the peg hits into the music system which kind of to me felt like a

natural thing the game the the peg hits are very Musical part of gameplay um oops hang on we'll go back we'll go back yeah um so to go back to where guy left us off um he would um he would start by dropping the markers at various points uh in the music segments of gameplay that corresponded to new keys uh that that he was switching to um those strings were passed uh exactly into the unity game engine through wise's music callback system um where Unity would parse those out into the four separate segments the key the scale uh and then the start note and the end note um so that we could generate the new key um on the Fly we we also decided that we didn't want to instead of having to create some giant lookup table for all of the key scale combos that had all of the possible midi notes that seemed very comp licated to create maintain um again we we wanted to avoid all of that stuff

so we created the um the C the new keys uh on the Fly based on the parameters that Define the key um let me explain what that means so uh basically we converted all scales into their you know numerical equivalents everybody knows that the major scale is whole whole half whole whole whole half hard to say uh but if you just switch those to twos and ones or threes um it's very easy to see that you can Define any scale uh with its numerical equivalent uh and as guy pointed out you could even create your own as long as they all added up to 12 um so that was very cool and then uh this is just a little example from within Unity so then once we had those lists um we would parse those out uh in a little script which is basically this is the part you're looking for um it's just kind of grabbing each one of those numbers and building a list out of those so that when it came time to generate a

new key we just looked up whatever the key was if it's major we grabb that um we start on the first midi note um if it's a G scale that or a g key that he's calling we know that that midi note is number seven um so we start there um set the first note and then we just start iterating through those scale degrees that you saw us set earlier um either until we hit that last note until we hit 128 which is the The Logical end for aidi scale um so so what we just did there is we built out you know our full scale of all the possible midi note numbers that the play playback engine could play um so then uh once each Peg was hit all we had to do is increment to the next midi note number that's in that list um and you know theoretically fire up the midi event the one caveat there was that currently uh the wise implementation of midi requires that all

midi notes all samples be played by an actual midi file uh you can't programmatically say you know give me a give me an 87 midi note with this velocity um so what we had to do is we had to actually Point these events to real MIDI files now instead of bringing in one single midi note 128 times for all the possibilities we um we use the transposed feature um that comes with the the new implementation amidi so we brought brought in a single c0 midi note um and then just uh used the midi number that we assigned before in that last slide uh set that to an rtpc which you can see maps you know one to one right here um and then we send that off modify the the c0 um midi note to its appropriate note uh and then fire the event so for example uh here's a quick look again at that um sample for the midi or for the the peg hit the guy was showing what it would do is if it's a an

F note a 65 it would set that RTP rtpc to 65 it would properly select that the closest uh 62 D note and then it would move it up three half steps um before playing the proper note um so kind of complicated to get to where we got but once we got this all set up uh it really was uh pretty much self-sufficient and every time we had a new music segment uh that would come in it kind of took care of it because we didn't need to fill anything out um yeah we have a perfect example of that when it came time to writing Boss music uh the regular music is mostly a major Mixel lyan but the Boss music has a lot of minor scales in it so we had to uh kind of create a few new scale uh options in there was super easy and it just kind of fell together didn't have to change anything in the tech yeah I think guy added something like 15 more minutes of music for that feature and I I didn't even go back in and look at this system so really really helped us out again with the game as a live service to be

able to kind of uh let this system be self-sufficient so yeah so that's how the peg system worked so just to round out the music system here one of the biggest takeaways was it's more midi is more about the memory Savings in a big way of course the insane memory savings is kind of what took us down this path in the first place without that you know we might not have even tried M but it led to this big open-ended potential for what midi can do for us obviously the peg hit system but also for how you can manipulate music on the Fly uh in general melodic uh harmonic and rhythmic flexibility there just doesn't exist with wave files now that said it's not an either or situation there's no battle that needs to be raged about midi versus wave files these are two different tools in the toolbox and I plan on using them in conjunction with each other and even layered so mix your awesome Daw you know set of St and put some midi over the top to make sure you're hitting the Precision that you need to for certain

cues and you have kind of a really dangerous system uh that we're looking forward to so once this is all done the next you know every every hero's journey comes with the the road home you know so we had to come home and uh for me uh the biggest takeaways obviously that midi isn't just for breakfast anymore is kind of the one of the biggest takeaways from this whole thing um but perhaps even bigger was the idea that the technical sound designer and this the the role that that RJ played having him be part of our team uh gave us an autonomy that I've never had on a project before so even if we had this Tech if he were living on on the game team and we had to go through Gatekeepers for any given feature we never would have been able to do this so definitely huge for us I for me I think one of the biggest takeaways was that as as um we can integrate more realtime

synthesis and more more of these different um synthesizers and everything even within yse like different plugins maybe third party plugins that we can use you'll really be able to take create assets quickly and effectively in new ways and just um make an easier process did you say third party I said third party whoa like third party s maybe third party since then software or just less barriers between different programs in general it's frustrating you have to open this and open that they can't talk to each other working this way everything was talking together and working nicely and it was awesome yeah yeah um one of the things that I I was really interested in was you know media is kind of just this data component right I mean obviously the first the first use for it is to just play your music segments through midi but um really you you know you've got this very complicated game engine doing nothing but just spitting data out uh in like really really different ways and the idea that midi can be used in more ways than just music you know I mean really all it is is a way to interpret data so it'll be it'll be cool to see how people

uh start using midi for you know anything that they can imagine you know I mean I think we kind of scratched the surface with this Peg hit system but I I think that uh I think that it can go a long way further I don't know if we mentioned but we ended up shipping with 1.3 Megs of sound effects and how much music did you put in well just the balance of the five so like three no but it was even less than it was less than five you ended up like 4.5 or something like that so yeah I was the I was the hog I took over two and a half megabytes and she had a me and a half so you know so good but I'm the manager that's why I I get to see yeah you get we know our roles yeah it works out yeah it was kind of good and it was hundreds of sounds and lots of music and I didn't feel like we skimped for a mobile game yeah oh you know I'm while we're kind of adding points to this one of the things we asked for and I think we may actually happen is the ability to you know compress files for download but then decompress them on load as you know Aug and MP3 files take a lot of CPU so you

can't have too many of those flying around in a mobile game but if you could decompress those as wave files all of a sudden you got this this package that feels like a lot more audio you know you can pack a lot more into your 5 megabytes that way we looking ahead so with that we would like to take any questions you might have just one pointer uh one other thing we're uh we're going to be be showing um good our some demos at the wise Expo Booth both today and tomorrow from 1: to 3: so if you want to see some more in action and try and break our build uh come on down and uh we the three of us will be there at various times so yeah yes hello I just wanted to say that the uh the music sounds really good and obviously a lot of that is the effort that's gone into new Endo in terms of sequencing and velocity and all that jazz but um how much effort went into the sounds you actually put into your sample Banks I mean obviously you create

your own sample Banks I mean obviously a lot of effort went in but how did you go about creating them did you record new musicians or did you use libraries yeah no um this the sample Source uh thankfully we did live live orchestral sessions for pegel 2 we had a budget Pagel 2 was a larger project um and we had orchestral a lot of it we recorded uh short little phrases or more specifically single note uh elements for the peg hits for pegal 2 and I was able to kind of like Harvest those sessions to create sounds now when I'm doing a sound sample bank that I know needs to be Memory efficient um one way to make sure that it's going to work is that each sample has to just be rich when in and of itself you hear that sample Raw it's got to sound Rich so maybe there's layering that goes on May I'll take a couple of sounds and layer them make it sound like if in your doll you might take a couple of string patches and layer them to thicken it up well I do that with samples to make it you know a little more strong sounding so that when

it's in the context that it still holds up but the other end of that is creating custom samples again I know the music I'm going to be creating so I tailor it in that direction okay um how big were the samples you created like for individual string hits I mean I've done similar stuff before and I always run into this problem where everything kind of sounds Jagged because I got these tiny little hits and I'm trying to make it sound like a rich Orchestra and I've got these Tails but people want stuff to be short right the Tails like end up right so the key there you have like let's say a um a picado or a staccato boom has nice attack it rolls off I'll do it your way and you try to make that as short as possible without sounding like oop that just cut off but the key there is Reverb really helps flow that through that was the glue oh using DSP oh yeah real time Reaver so it's all being busted real time Reaver yeah we didn't have any baked reav either either sound effects or music yeah you can't make it on the um yeah so but there was also uh I did a lot of mastering and pre-processing to the samples themselves you notice the names on some like ptech EQ or whatever where

I'm like treating them as if it's like a final you know sound because I know I don't have a ptech in Wise you know I can do that kind of treatment prior but it's still a dry sample once it's in there and then the Reverb gets added in real time okay excellent thank you yeah Vault last year what's it yeah yeah if you want to know more about how that works there's a Pagel 2 talk on on the Vault watch it from last year thank you and that'll kind of give you some backstory on pegel 2 and leading up to how pegel blast came thanks yeah okay so first we're not worthy we're not worthy he's our biggest fan I'm literally tears in my eyes listen to you TI talk with this is unbelievably great um one I got a bunch of questions well first what about latency now when you're doing stuff in Unity on the machine everything's fine and cool but when you put it on a a mobile device um you can get latency where you know and in a game like this where the Peg and

the sound hit have to be pretty close did you have running any trouble with that and if so how did you deal not really we uh we ran into some issues uh on Android devices a little early on where there was some latency um but I think that as the as the rest of the game team started optimizing other stuff um we everything started sinking up pretty well yeah in in the early days of mobile games Android in particular had a lot of latency and lot but I think the unity wise connection and and the the improved drivers on Android all help that H yeah yeah that's a little disappointing cuz the unity fod that I've been doing latency is horrible hor it's like 150 more about syncing up the two I don't know yeah we well I mean I mean so like when I do a I'm doing a Unity fod implementation the animation the cannon goes and then the the sound happens and it was so bad that when I you push a button uh the sound would happened so much farther later that it was almost disconnected we didn't see

any problem we didn't run into that no we never run into that interesting yep not at all well there you go um another qu question about uh the samples in your sample Banks any looping in the samples so do you do GRE and then have a loop no I had I did kind of a tricky thing where um I had like string staccato it's for example and I had just a string Loop or a different bank that was just string Loops so you'd play the staccato and then the loop and sustain the the thing under the loop was it was always and first to last you never had any looping in it's mimicking what you know a play in the middle Loop would do but with the strings okay cool um oh and the other question was at one of the things I used to do with the beatnick engine which is obviously similar was to record uh chords and things that I would want to do like I'm going to play this chord and that chord and that so instead of having a bunch of of that would save me uh voices yeah so instead of having a bunch of string samples that would play

for course I would record just the chord and then just play that thing obviously you did that for the nice and the awesome Coral stuff but did you do any of that for any of the other instruments no everything else is single note samples but I I do think that's a technique you know obviously with the nice and all that those were recorded as chords rather than played that way um but on earlier in the '90s when I was doing midi in a previous life um I would often record little riffs sometimes like if you have a guitar riff it sounds better just to do the right or to have something play a chord and just play it like a DJ would right so those techniques are really valuable though I didn't do that here though except for the exclamations okay well I could I could ask questions about this for days we'll talk later than uh thanks uh I don't know if you guys can answer this question but it was sort of occurring to me as you were talking about building the sample Library up and wise I I'm wondering if you based it on a

commercial sample Library if you would run into licensing problems yes you would okay yes you absolutely would there in the '90s late '90s there was a because direct music used midi and Sample Banks there was a big discussion about what was legal and what wasn't and the general rule is don't do it um you can sample like analog synthesizers and things that are acoustic in nature yeah uh no problem but you can't pirate other libraries is the problem that's why I think it's time to actually start having conversations with uh folks who make sample libraries and see which ones are amenable to this Market because it's a different Market it would be uh much like your lease DSP for a game it would be that kind of a business model and I think it's time to bring that back right yeah okay they'd have to be smaller sound sets and geared for games as well so right um yeah so anybody any sample Library makers out there all right right we got two nice I three there's the word

word is out um first of all amazing job guys really inspired by the talk and um I definitely think that the hybrid of you know real waves and maybe is the way to go for the future um curious about your guys' audio output settings um when you're doing your compression we I assume you're doing selective compression between all the different areas um oh like data compression yeah data compression uh let's see I mean you had to go into that a lot with the sample we all kind of worked in our own areas and the samples I experimented I one time I tried all a cuz I could get more out of it but it just brought the system to its knees and in unexpected ways asymmetrically it wasn't like oh it's just crawling it's like boom randomly would you know so we've quickly took that out um so it literally was a combination of sample rate mono and uh ADP PM where I could stand it and I even did some posts on some sounds where 80 PCM makes things

all crunchy I did some uh real-time post EQ to kind of roll some of that off so you kind of play around and keep keep massaging it or even going in and re-editing samples so all of those methods were combined yeah it's a lot of listening and trying and seeing what works and what sounds and you had to do that with your sounds as well and you did a lot of like taking one little sample and making different use of it right how did that work yeah oh for one one wave file manipulating it a lot yeah well for for those um wave files we I think what were what did they end up being I think they were 80 PCM uh yeah 80 PCM no I think we used a couple of waves I think we just down CU we were manipulating them a lot and there there wasn't that much of those once we added um some emotes at a later time period that's when we really had to get creative with what we were doing and it really just came down to auditioning in the assets choosing what sounded best and took took up the less space and trying them on device and seeing if it worked yeah yeah I I think no different

than any project only we had you know the added thing if we had to make sure that it was uh performing okay you know uh we couldn't just wanly use uh vorbis anywhere we wanted right we were pretty careful about the vorus but I hear that stuff is getting more efficient so cool and if you don't have wi's I don't have wi's for the projects I work on so do you know if any of the other middleware software companies I know I know fmod supports a whole slew of ATS um I think I saw in a Unity 5 talk yesterday that they just added ad PCM and vorbis support but not even just that just the whole midi oh the mini stuff you know there I I've hear there's some plugins for um Unity is that correct but those are those would be kind of garage people someone in a garage making a little midi plugin for Unity right uh there's nothing as formal as this that I've seen that's commercially available obviously you know the Fantasia guys use midi all over the place but um that's is that wise try to try to get your studio to use

wise all right and maybe other now now maybe other uh engine makers will add it yeah if they see the success that folks are having with it or the if people are calling for it if we are calling for Midi um the other engine makers will follow suit it makes so much sense for mobile thank you thanks guys thanks a bunch yeah think that's it that's it thank you so much thanks everybody [Music] uh