Genesis Noir: Bringing Jazz Improvisation To Gaming

[Music] hi i'm jeremy abell a technical lead and audio programmer at feral cat then and i'll be talking about our game genesis noir and how we build a level around jazz improvisation so genesis noir is a poetic adventure game with an emphasis on exploration and a simple palette of interactions you play as a cosmic god in a world where the big bang is a gunshot fired at the woman you love and the player's goal is to try and stop it the gameplay loop consists of entering the universe and exploring a series of standalone short stories and each of these stories tells a story about a human character and illustrates some aspect of humanity for example a hunter character illustrates humans mastery over their environment or a scientist character who shows curiosity about the universe for the level focusing on human

creativity we really wanted to tell a story about a jazz musician due to the noir style of the rest of the game and how much jazz music kind of infuses the rest of everything else and we kind of thought that the most interesting way to interact with the jazz musician is through music itself and we really like the idea of learning how to play music and having the human character teach the player about their musical world i kind of thought that was pretty cool and i have first-hand experience in this having played drums and jazz bands uh all throughout my school years learning to play jazz is different from learning classical music with classical music you're expected to play exactly what's written on the page the sheet music tells you how loud to play how fast to play when to press the piano pedals how short the note should be and this is all done in the purpose of expressing the composer's intent it's way less about the performer's expression it's more about the beauty of what the composer has created but with jazz the focus is much more on

the individual playing music rather than the composer if you look up a hundred performances of hungarian rhapsody the differences between them will be subtle especially if you're just a student if you play it wrong your teacher will probably yell at you but on the other hand you can look up 100 versions of all of me and there will be a lot of clear variation in speed feeling texture instrumentation and everything else the sheet music doesn't even tell you what instrument to play or how fast to play it it just kind of gives you a melody uh and some chords and then you expect to just kind of do whatever you think is good the other prominent aspect of jazz is the solo which used to be a big thing in classical music but eventually kind of fell out of fashion i think it was called cadenzas in that context but teaching how to solo is really difficult and in school they'll give you sheet music like this where it just says repeat for solos and then there's some chords and a bunch of slashes and the slashes are just they

just mean play whatever you want um but that's only in school in the real world you won't even get that you'll just kind of get these bad xerox copies of handwritten sheet music and you could buy them in a book called the real book but the people who made this book didn't have the licenses for the songs to print the sheet music um so the only way you could get them was from someone's car truck outside a jazz club or if you know who to ask at a music store and the other thing is you can't let anyone see you have one of these at the jazz at the jam session because you were expected to learn all these tunes by just listening off the albums and reading off the real book in a jazz jam session would get you kicked out you're supposed to memorize them and listen to the original tracks and then figure out what you're supposed to play from that so we felt that uh learning to play jazz seemed like the perfect story for a video game well you have a mysterious community that's distrustful of outsiders strange rituals to learn and a big book of secrets

so you're kind of becoming a jazz wizard i guess so how can we make this fun and interactive for someone who has probably never played jazz before well let's let's look at the traditional path for learning this stuff step one like everything else is to copy the professionals uh in jazz school this is done by transcribing solos to learn how a solo was constructed and i guess it's kind of like copy pasting code from the internet you would usually spend you know a couple hours just listening to a song over and over again and writing the notes out and then you would get graded on it i guess on accuracy so turning this process into kind of a simon says game made sense i'll play a bit here you can see how this works in the game [Music] so as usual it starts out with only a couple notes and it adds more as you go on [Music]

we're going to have this rectangle style to evoke piano keys [Music] and after a few rounds of that the player gets kind of blown away by the human [Music] this is meant to kind of evoke the common feeling of being inadequate as a musician but our character has an idea what if we reverse this interaction it's inspired by my experiences in school where the teacher would say stuff like improvise using only these four notes so the goal of the interaction is for the the goal of the interaction is to pick notes and have the human bassist uh kind of play a cooler spicier version to kind of uh

as a call and response kind of thing so now we get to the big question uh how the heck do you make a computer play jazz music in the first place and this is caveated without machine learning because we don't have the time or the budget to really go down that route so i started doing some research and i found this great video by 8-bit music theory that analyzes a saxophone solo from a mario kart 8 tune and one of the things that he mentions is a technique called chromatic enclosures where you start with a single note and you add notes uh semitone above and below the initial note and this sounds like a really great potential algorithm i guess for taking an input set of notes and spicing that up making it more interesting and at the time i was really excited by the just by this discovery i didn't really think fully through why you would want to do this what's the benefit of enclosures i

was just jazzed about there being a potential solution to the problem of starting with four notes and coming up with a cooler version of those four notes so you can even make really long enclosures you can start with one and then you know suddenly it turns into five by you know kind of weeding in and around this note that you're targeting uh so i got this idea of making improvisation mutators where you would take a pair as input notes run it through some algorithm and you would get some spicy output notes so the first one i wrote was this enclosure one where it would randomly add enclosures to players notes so if our input sounded like this then the enclosured version might sound like this [Music] and yeah it doesn't really bear much resemblance to uh to the original input

you can see that the targeted notes are in red and they're just kind of enclosed by some other random notes that are above and below the targeted note uh i also wrote one called double notes um this one allows us to kind of here i'll just play it [Music] so if our input is this kind of allows us to feel the swing rhythm more which is a critical part of jazz uh here you can play it called straight which is without swing and then with swing it's like this you can hear how has way more of a bouncy sound and it's kind of a quintessential part of jazz um and then i wrote one called adjust durations where it just kind of randomly changes the durations of the player's inputs so if our input is this [Music] we can adjust that to

which sounds interesting but i decided that we kind of needed to know the durations of the player's input notes which is called quantization where you snap some inputs to a temporal grid basically assigning sensible note directions to what the player had played and it turns out this is a really difficult problem to solve especially if you're recording inputs from someone who's trying to play a video game and not actually perform music um but at this point it's probably like i don't know 10 p.m and i'm way too in the zone i'm too busy writing code to stop and test the code that i've already written so i come up with this a cheater solution um so if this is our input uh we can find the maximum duration between two notes divide that into three buckets assign note durations to each bucket and then measure each inputs duration and sort them into the three buckets and this turned out to be good enough i still haven't really tested any of this but i'm pretty proud of my chidi

quantization method i commit at 1am and i come in the next day to test and it turns out that all of this sounds kind of like crap it's not very interesting uh it doesn't really really uh spice up the player's inputs too much it just kind of sounds like you know random uh random stuff it turns out uh this is a pretty difficult task a lot of work has been done trying to make a computer generate a jazz solo some a lot of papers with lovely diagrams and great titles like recombinant jazz improvisations and there's also a few open source tools like this one called improvisor with some pretty great ui options like this button that says prepare critics and these check boxes where you can rectify and color your solo i have no idea what those mean but it's it's fun that they exist so at this point i realized that i'm

making a video game not a four year phd research paper and i need to kind of step back so i take a break for a few months i work on some other things and eventually i come to the realization that a generic solution is overkill uh it just needs to be fun for five minutes the player is isn't going to be playing this for hours it's not it's just a small segment in a larger game it just needs to be good once um but can we find anything in the work that we've already done that might be useful uh so let's let's take a look at that one more time um so here's the adding closures like we said before you don't really hear the player's input very much that gets transformed into it doesn't really flow it doesn't really have much of a cool rhythm it's just

kind of oh great i added it's two more notes at the beginning of all all of this input um the double notes was interesting in that it helped us feel that kind of swing beat but it's not super interesting as a spiced up version and our adjusterations idea was also interesting but our original implementation was dependent on knowing the durations of the player's inputs which you know came up with this whole quantization problem that i just didn't want to deal with so i decided at this point to actually emulate what i wanted the game to do i wrote out a sequence of notes for the player's inputs and then set up the keyboard and made up my own spiced up versions and eventually i realized that since this just needs to sound good once why not just pre-make the improvised responses and then build them to adapt to whatever the input is and this is what we ended up shipping with so let's see how this plays out in the game first and then i'll break down

what's going on [Music] [Music] cool so what we do is we record players notes into an array and we don't really care about the timing since we'll be applying our own rhythm uh we just record them in a list and then we have these things called response midis which are used to impose a rhythm on the player's inputs and this is what one of them sounds like

and it doesn't really sound that great because but that's because we don't use the actual pitches instead the note pitches are used as indices into the player's input array rather than actual pitches so midi middle c which is midi note 60 maps to the first index in the player's input array so for each note we replace it with the player's pitch from the input array and let's see how that sounds so if this is our input then our remapped response sounds like this and that's a lot more interesting we can hear the player's input and the response um and we have complete control over how it sounds what the rhythm is what notes are picked um instead of little instead of letting the computer pick randomly and chanting the fact that it would sound bad and one thing that we found was that it only really sounded good

when the order of the player's notes was mostly kept intact especially at the beginning of the response so you can see how we start with one two three four and then we just step back down that scale and this is the first the first one that we saw in the video they get more complicated as it goes on but if we start in the middle of the player's input array it gets difficult to hear that the player's input was the source of the response and it doesn't really sound like the human character is riffing on uh on what the player is doing so there are three responses in total and we let the player enter more notes each time so we mirror the increasing complexity of the npc led simon says game at the beginning so let's see how it plays out this is the third and final round [Music]

so we end up getting blown away once again [Music] so clearly we need to practice more so the four blocks from the simon says interaction kind of expand out into this full array of buttons like a piano and now we have way more control over what the player can play so they can hold the mouse button down and drag across the screen to play a quick string of notes and this is kind of meant to represent the struggling practice phase of learning to improvise where you just practice some scales for hours and you'll notice that we're not using the sampled sex anymore because playing fast notes like this allowing the player to play fast like this leads to problems with our sax samples where the note to note transitions sound super cheesy like a really bad sampler so this is what our first test sounded

like [Music] so we clearly needed to find a solution for this just because it just sounds really bad uh first thing we did was we tried to sound font but that ended up sounding like bad 90s midi [Music] and we also considered physical modeling which sounds really great but it would take way too long to implement uh and yeah i mean again we're making a video game not a a simulated saxophone uh so we kind of decided to lean into the standard synthesizer tone built in to unreal engine and we kind of enhanced that hesitant i'm not good at this yet feeling by adding a lot of pitch wobbling and modulating the time of a delay effect and that by itself without the backing track sounds like this

as you can hear it just kind of trails off into this uh drunken kind of style so let's continue and see how the rest of the interaction plays out [Music] so once you satisfy the basis with your practicing uh it's on to the next step of the jazz learning adventure [Music] so instead of just pressing buttons now the player is making this jazz sculpture a completed piece of art and we kind of represent this as a brush interaction where the player draws their notes in free space as a geometric path

of lines and blocks at this point there's no notes to click no buttons or whatever you just hold the button down and it automatically makes a note and we use the vertical axis of the screen as as pitch and this is a look at the prototype that we had built originally where it kind of starts to look like a city but eventually we kind of evolved into this more characterful style where there's windows and everything kind of pulses to the beat and we also switched to a way more prominent backing track so this was designed really intentionally to make the solo fun to play so let's talk about how we designed that so in jazz the backing track or the track that you're soloing over serves as the scaffolding for your improvisation you improvise using the chords and structure of the song that you're playing over and this is what's called a lead sheet uh which is just the basic melody and chords and nothing else and an entire

band the entire quartet or whatever would be able to play the entire song with just this amount of info and in school you'll get these lead sheets and will occasionally be given scales that you can use over each chord a scale is just a series of notes that increase in pitch so like a c scale or a d scale or whatever um and this is an example for miles davis's all blues um and it says stuff like g mixolydian g dorian etc and that doesn't mean much to me but it makes more much more sense like this uh where it just hears the notes on the piano that you can play that'll make it sound good and when i was working on this slide i got distracted for like an hour just uh just playing over the song and this is kind of how i figured out originally that i wanted to make a level about improvisation as i was just messing around on my keyboard while listening to some albums and realizing that oh this this particular

song is really nice to play over so i recorded a bit of my own improvisation to kind of talk through what makes this song interesting to play over [Music] i apologize for subjecting you to my horrible improvisation um but what you want to kind of do when you're making a solo is kind of have this interest curve is is what it's called where you kind of take the listener on a

journey with sound um and the song that you're playing over can really help you do this and that's kind of the aspect that makes all blues so much fun to play is that you don't even really have to think about you know oh am i you know do i want to increase intensity here do i want to decrease intensity because the song kind of does it for you and that concept is called tension and release so in music in western harmony at least certain chords feel like they need to resolve to to go somewhere we can hear that in this short example here i'll play it again so we're stable tension release like that um and you'll hear that all over the place um and it's really prominent in all blues so i'll play this segment again without my solo from

the actual album um and we'll see i'll talk through it pretty stable playing through the melody [Music] tension back home again [Music] oh it's a lot of tension but a lot of tension adds to bigger release [Music] and that's the whole song it just loops like that for five or six minutes as people take solos over it so returning back to this kind of home sound adds this inherent sense of progression like the song has been somewhere and now we're going to come back this provides an inherently interesting structure to play a solo over so with this in mind we kind of bake

this same kind of tension and release into our backing track and our harmonic changes aren't extreme aren't as extreme as all blues are and that's in order to not totally blindside the player and have them kind of focus on their own playing instead of oh where's the song going so i'll play our track here stable the more you repeat it the more stable it becomes and the switch to shoot sharp and now there's some added tensions it's different from where we were [Music] repeat and we're back home again so we can also assign a scale for each chord in this case we use the c mix lydian

scale sounds like this and then we switch to the c sharp mixolydian scale which sounds like this and we encode these directly as markers in the music track in fmod and they're notated as a root note followed by the intervals between the notes in the scale so we have scale starts on c then we skip one two one two just one one two one two one again and that gives us c two two one two two one and this allows flexibility in changing the scales later since the backing track hadn't been finalized by the sound team when i was working on the interaction and an example of this is switching to c-sharp uh just one step up is only one character change in in the marker so let's check this out in

[Music] you context we're stable here [Music] now we've gone up to see shark [Music] eventually we kind of build up this whole cityscape and the human character uh will join you so at this point the players kind of gained the respect of the bassist and i was trying to figure out a good way to represent this through gameplay and around this time i had heard ornette coleman's free jazz a collective improvisation which is an album

by ornette coleman where he has one quartet playing in the left side of the speaker and another quartet playing in the right side of the speaker and they're all kind of playing collectively and i was really inspired by this conversational nature of this playing kind of sounds like a big discussion with their sharing musical ideas talking back and forth and i'll play some here [Music] you hear different instruments playing all together kind of improvising together like a big kind of shouting match and this seemed like a perfect way of communicating that the player has gained an understanding of human creativity from his music character but now we're back here again uh how do we make the computer respond to the player's inputs um without machine learning again um but we've learned a lot so maybe we can implement this kind of conversational duet idea uh in a simple and easy way the first

time instead of trying some stupid stuff for for an entire day so let's look at the data that we have we store the players inputs as geometric paths in space which are stored as splines and we can imagine those splines in a box where the height corresponds to to pitch and we store one point per frame and we store both the position of the point and the time since the last point was added this allows us to replay the player's inputs both in space and in time so we started by kind of randomly chopping up the player's input paths into slices and replaying them but the problem here was that if the path was too long it could pick stuff from many seconds earlier and the player would have entirely forgotten that they played it which would lead to a disconnect in what the computer was doing so a better solution we found was to only slice the last two to five seconds of the player's inputs to keep the computer uh want the computer played relevant so as the player draws their

path segments are replayed from the latest path data but this just kind of generated uh reiterations of the player's inputs what we wanted was the computer to be a bit more creative with responding to the player so since the player's solo is just geometric data we generate variations using normal geometric transformations we can move the curve up or down to shift the pitch we can scale it to increase the variance in the pitch and rotation uh looks kind of weird in this context but if it's just a small segment the notes will kind of start out the same and then gradually diverge which you can see in this looping section here so if we combine this idea with our smarter path selection it looks like this um to add a sense of progression we kind of can increase the variance in these transforms how much they randomize the longer the player plays with the interaction and we can also move these points around in the x and y axes to have some more visual separation without adjusting

without affecting the pitch so let's check this out in action [Music] hmm [Music] [Applause] [Music]

so this geometric approach works really well because it always maintains the player's timing and rhythm where if you play short lines the computer will play short lines if you play longer held notes the computer will play longer held notes and while the pitch is changed the feeling is maintained and at some point this kind of all stops mattering anyway and the player is just kind of vibing in this moment so the last important thing we do is that we don't take this interaction away one of our core design principles was to never take away the player's toys um if we have something fun that they're playing with we don't want to have them accidentally click something that will take this moment away so the trigger to proceed here is this train where dragging it to the right will progress through the story and we

indicate that it will do this by gradually fading in an ending chord to the song and also these city sounds as the train is dragged this tells the player that if they keep dragging the interaction will end and this avoids that situation of the player clicking something by mistake when they don't want to continue yet [Music] [Music] so with all this in mind uh let's watch the whole interaction from the beginning and i'll shut up [Music] [Music]

now [Music] so [Music] [Music] so [Music] [Music] [Music] so

[Music] so [Music] so [Music] uh [Music] oh [Music]

great [Music] [Music] uh [Music] oh

[Music] [Laughter] [Music] [Applause] [Music]

more [Music] so i'll end with showing some feedback from some players um we had released the jazz level as a demo pretty early on in development and one of the functionalities of the demo was a email feedback form so it was really cool to see people enjoying the interaction but some of the coolest messages we got were from educators who were inspired by what we've made and i i can't find it but we did get an

email from a teacher asking if we could add some features so they could use the interaction for teaching their students stuff like being able to jump to any part of the three stages of the interaction and stuff like that which i think is something that we'll add in a future update but this really kind of highlights the potential of making a game not just with a jazz soundtrack but making it about playing jazz music and i feel like this similar kind of concept could apply to any other kind of music as well um so i'll conclude with thanking a huge thanks to skillbart our music and sound design team uh for being just outstanding partners in providing me with so many good sounds and ideas for the interaction i can't imagine building this without them so thanks uh thanks for checking out the talk you can find more about the game at genesisnoragam.com it's on steam for windows and mac xbox and switch you can reach me at jeremy farrell catden.com at my email address or on

twitter as uh jablesjables so thanks again and take care bye [Music] you