From 'Spyder' to 'Sackboy: A Big Adventure' in Interactive Music

[Music] hey everybody welcome to my talk from spider to sac boy a big adventure in interactive music my name is winifred phillips and i'm a game composer some of my projects include games from the franchise's god of war assassins creed total war little big planet and the sims i've written about all of this in my book a composer's guide to gay music published by the mit press and in the future articles that i've written for gama sutra back in the summer of 2019 i started working with sumo sheffield on music for two very different games it was pretty intense work over the course of many months on two projects with very different musical needs and i spent a lot of time ping-ponging back and forth between them after they both hit retail in 2020 i realized how eye-opening an experience that had

been both games required complex musical interactivity but each met that goal in very different ways today we're going to be taking a look at those two projects sacboy a big adventure for the playstation 5 and spyder for apple arcade both projects use some common dynamic implementation models over the course of this talk we'll discuss how each project incorporated these implementation strategies pure horizontal resequencing pure vertical layering a hybrid horizontal vertical system and a hybrid linear dynamic structure as we know dynamic music differs from linear music in some fundamental ways linear music is a single contiguous unit like a straight line moving in one direction with a definitive beginning middle and end but dynamic music is more like a maze that can move in many directions with lots of divergent

possibilities the art of interactive music creation and implementation is not just about understanding and deploying interactive music systems such as the ones on this list it's also about looking at the nuts and bolts of these systems and seeing all of those divergent possibilities sac boy big adventure and spider both did a great job in taking dynamic music implementation techniques and expanding on them in creative ways so let's explore that but before we jump in i'd first like to introduce you to our two heroes of course everyone is well acquainted with that international superstar sac boy his daring escapades and craft world have become the stuff of legend sackboy's latest big adventure is a smash hit action game that pits him against the evil vex who's bent on destroying craft world with his terrible topsy-turvy machine of course sackboy must act fast to save the day so let's watch him in action

[Music] [Applause] [Music] [Applause] [Music] while sac boy is a legend many of you may not have heard of our other hero that amazing covert operative that grew the agent of mystery that globetrotting defender of truth and freedom agent eight it's understandable to overlook him he is less than an inch tall the spider game is an apple arcade exclusive in which agent 8 undertakes stealthy spy missions to save the world

many times over though the british spy agency would disavow any knowledge of his actions should he be captured or stepped on so let's see agent 8 in action [Music] more [Music] working on music for these two projects at the same time was an intense and rewarding experience for sackboy big adventure i joined a diverse music team and was assigned the task of composing

music for sac boys underwater escapades i was also one of the team members responsible for creating cover music that would fit into sac boy's world for spider i was the sole composer creating original 1960s and 70s inspired music for agent viii sneaky spy missions both games were developed by sumo sheffield and both included ambitious music systems using similar dynamic strategies there are a lot of interesting differences though and that will be the subject of our talk today coming back to our list of implementation procedures let's start with pure horizontal resequencing both games used to this technique so we'll now focus on the most straightforward example like the rest of the games in the franchise sac boy a big adventure is a celebration of music incorporating original compositions licensed pop tunes and cover tracks along with the original music tracks i

composed for sac boy big adventure the team also asked me to compose a cover song that would fit into sac boy's world a brand new version of material girl my version reimagined material girl as a classical orchestral composition while also retaining enough elements to make sure it was recognizable the final mix was separated into segments for implementation now this is where horizontal resequencing comes in when we compose music in segments that can be triggered in different orders that's horizontal resequencing shuffling things around makes the music more responsive to the game and also it helps the music to feel a little bit less repetitive material girl is broken into segments according to its song structure which is actually a pretty straightforward approach songs typically have an intro a verse a chorus a bridge maybe an interlude towards the end and an outro now for

this interactive version of material girl we get all of these song structure components implemented as segments in a horizontal re-sequencing system they're all separate fragments that are activated with their own trigger points now as the level starts we hear the intro segment which only plays once this transitions right into the verse and chorus which is combined into one segment that's essentially the main loop so let's take a look at that [Music] um [Music] [Music]

after a while we get to the bridge it's implemented as a short loop that triggers right before sac boy goes flying across a chasm now once he lands the music jumps right into the interlude section so let's see how that part worked [Music] [Applause] [Music] [Applause] [Music] following the interlude we go right back to that main loop for a while and then we finally get to the outro segment which winds things up at level completion [Music]

you can see how the music provides a sense of reward as sac boy progresses through the level the music advances to reveal new content a dynamic music system such as this one has the primary goal of indicating progress through the level in the case of material girl the advantage of song structure is that it provides well-defined segments that are inherently dissimilar so jumping from one segment to the next feels very dramatic we're meant to notice what's happening it's actually part of the reward

now around the same time the developer sumo digital was also asking me to compose horizontally resequenced music for the spider game and that's actually a very different approach spider is a pretty stealthy game it's not about running and jumping it's about being a secret agent so the music had to be sneakier and it had to leave space for players to think and to plan for puzzle solving and exploration the music i composed for spider is horizontally resequenced but those segments are designed to ebb and to flow and even to settle into silence periodically there are also a lot more segments involved every level of the game had over 30 music segments in order to enable divergent and unpredictable music during puzzle solving gameplay and the order of those segments is randomized so that players can't anticipate what they'll hear let's check out an example of this

system in action [Music] [Music] thank you [Music] [Music] so you can see this interactive system in the spider game isn't really indicating progress through the level the way the material girl music did instead the focus of this system in the spider game is to create musical variety and to avoid predictability a system

like this can keep gaming music from coming across as repetitive so the sackboy and spider approaches are actually quite different however there are similarities here like the material girl example from sac boy a big adventure this system in the spider game is taking an overtly segmented approach we're aware that the music has different sections the transition into each other and the dynamic music system structures itself around that awareness but what if that isn't the goal what if we want the horizontal resequencing system to completely hide its transitions now to get a taste of how that approach would work let's look at one of the original tracks i composed for sac boy a big adventure shooby doo hullabaloo the boss battle music at the end of the thar she blows up level now like material girl this boss battle track is structured in horizontal segments in this case three progressively more

dramatic chunks of music but the implementation strives towards a seamless experience with the goal of hiding those transitions rather than showing them so let's see how that worked [Music] one [Music] it's [Music] [Music]

as you can see pure horizontal resequencing can be a very flexible approach to injecting interactivity into our game music we can aim to make our music feel seamless or we can present our music in a more segmented overt way and both approaches can be effective interactive music design is highly contextual the circumstances dictate our choices no single method can be considered the best way or the right way working on these two projects at the same time i came across this idea over and over again for instance unlike spider sac boy a big adventure is the latest game in a franchise and the previous games in this franchise implemented music using pure vertical layering now when we compose our music so that our chosen instruments can be incorporated individually and then those

instruments can be divided up and manipulated separately during gameplay that's vertical layering this technique helps music to feel reactive without also having to be divided up into segments sackboy's previous games in the franchise had an elaborate system of pure vertical layering for sackboy a big adventure the music team favored other methods of dynamic music implementation however spider embraced pure vertical layering for its action sequences so let's take a look at a couple of examples our intrepid super spy agent 8 goes on many dangerous missions and one of them sends him scurrying through the innards of a massive computer the music i composed for this sequence is a funky late 60s style track with lots of vertical layers that get activated as agent 8 moves from one area to another there are no horizontal segments so the vertical layers have to do all of the work to make the music feel dynamic

so let's see how that worked [Music] [Music] in another dangerous mission agent 8 gets launched into outer space to rescue some imperiled astronauts for this level

i composed an interactive 1970s space disco track with vertical layers that react fluidly during gameplay sometimes gamers only hear the rhythmic elements of the track sometimes they only hear the synths and sometimes they hear the whole mix the vertical layers kick in to control how intense things feel during gameplay so let's check that out [Music] [Music]

this vertical layering music from spider accomplished two goals it indicated progress through the level and it created variety to keep the music from feeling repetitive so while horizontal resequencing slices up a track into segments to accomplish the same goals a pure vertical layering approach can instill a sense of progression and variety without needing to fragment a composition but what if we want to do that anyway what if we want a hybrid horizontal vertical system with both segments and layers at the same time in the same track can we have the best of both worlds well it turns out both spider and sag boy big adventure answered that question with a resounding yes however each project addressed the issue differently

so let's start with sac boy's approach for the synchro swing level of sac boy a big adventure i composed an original symphonic style waltz the three-quarter time emphasized the kinetics as sac boys swung across the level now as sac boy progressed the waltz of the bubbles used both horizontal segments and vertical layers for musical interactivity the track was broken into seven horizontal segments that progressed as sac boy traversed the level now periodically the elements of the overall mix would change to either add or subtract a women's choir as a new music layer this worked to inject some much needed variety into the overall mix so let's see what that was like [Music]

the vertical layering technique also inserted itself into the game mechanics by virtue of the iconic sac boy success melody now those of you who have played sac boy a big adventure will recognize the six notes in this game cinema hmm or maybe he was just guarding this old looking egg yeah why don't you hang on to it [Music] the music team for sac boy a big adventure used this success melody across the game whenever sackboy picks up a collectible known as a dreamer orb we hear his success melody as a separate vertical layer that marries harmonically

with whatever horizontal segment happens to be playing at the time so let's take a look at that [Music] [Music] you can see that the interactive music system here accomplishes three goals goal one the horizontal segments indicate the player's progress through the level goal two vertical layers create variety to keep the music feeling fresh over time goal three an additional vertical layer provides a reward when objectives are achieved so we can see that horizontal and vertical techniques work well together they worked for both sat boy a big adventure and for the spider game which i was working on at the same time the implementation for spider though is

very different structurally for example during one of spiders more stealthy levels agent 8 infiltrates the enemy's high security war room while pursuing his mission to obtain some top secret documents agent 8 accidentally turns on a desk radio and my 60s style lounge jazz music composition starts playing from it this keeps playing throughout the rest of the mission [Music] diegetic music like my lounge jazz composition from the war room radio is considered to exist inside the fictional narrative whereas non-diegetic music or dramatic underscore is considered to be outside of it so now that we have that continuous diegetic jazz music from the radio does this mean that we can no longer have dramatic underscore well not

at all we just have to apply some creative solutions the audio team at sumo sheffield and i solved this problem with a hybrid dynamic music system that included both horizontal segments and vertical layers now first a vertical layer would start playing that continuous jazz music from the war room radio and then the dramatic underscore would come in on top delivered in over 30 horizontal segments now like the success melody from sacboy these music segments were structured to blend with the other music that was already playing so every measure of the radio music would have a corresponding measure of dramatic underscore these underscore segments would only play when they were triggered so they usually play in different sequences this helped the music to stay fresh as gameplay progressed let's check out what that was like [Applause] [Music]

[Applause] [Music] as we can see this interactive music system from the spider war room actually accomplishes many goals the horizontal segments indicate progress through the level they create musical variety and they reward players when objectives are achieved plus the vertical layers allow us to merge diegetic and non-diegetic music together but there is another goal that dynamic implementation can address and it's actually a much more practical consideration than any of the others dynamic music can stretch one piece of music so that it's doing the jobs typically done by several tracks dynamic systems can make one piece of

music take the place of an ambient track a puzzle solving track and a high energy action track it's actually a great way for development teams to optimize music coverage for their games so let's see a couple of examples of this earlier we checked out the shooby doo hullabaloo music from sac boy big adventure while we already looked at that big boss fight music from the end of the level that same music also starts the level in a very low-keyed way it's pretty understated and calm and that same musical composition slowly builds in intensity by virtue of its horizontal segments so by the time we get to the end of the level it's gone from a peaceful ambient texture all the way to a rousing orchestral finale let's check out how that transformation occurred [Music]

uh [Music] in this excerpt from sac boy a big adventure we saw how a dynamic music system enabled a single piece of music to scale between broadly divergent levels of dramatic intensity but the spider video game takes this idea even further by employing one of the more interesting and unusual applications of dynamic implementation for its timed micro mission challenges so let's discuss how that system worked now we all know that purely linear music has some distinct advantages as we

mentioned at the beginning of this talk linear music has a set beginning middle and end linear music can more fully explore its ideas without having to worry about how the maze of a dynamic music system might impact an instrumental arrangement or juggle the progression of musical content or even truncate melodic lines now while all these issues are solvable linear music certainly provides an added breadth of creative freedom so is it possible to preserve these advantages while also employing dynamic implementation in our tracks well let's see how the spider game tackled that challenge when agent 8 isn't off saving the world he's back at headquarters where he's evaluated and he undertakes difficult training missions these missions are timed sequences they last exactly three minutes now because of this when composing music for these

timed training missions i was able to work in a hybrid music structure that combined both linear and dynamic elements i composed a piece of linear music that lasted exactly three minutes with a definitive beginning middle and end and then i added vertical layers to this linear composition so that allowed it to scale in intensity as the countdown sequence progressed so let's take a look at how that worked notice how this hybrid linear dynamic structure adds layers when mission objectives are met [Music] [Music]

so [Music] [Applause] so now we've discussed four distinct forms of dynamic music implementation we've reviewed the structural mechanics of both horizontal resequencing and vertical layering and we've taken a look at how these very different interactive techniques can be merged into hybrid forms we've also examined some of the goals of dynamic music implementation by working with sumo digital on both sac boy and big adventure and spider at the same time i had the chance to see how versatile dynamic music can be as we discussed earlier interactive music design is highly contextual the circumstances dictate our choices there is no best way or right way

by keeping an open mind and exploring our options we have the opportunity to discover lots of divergent possibilities it's one of the more inspiring aspects of our work as game composers thanks very much for watching [Music] you