"Swiping on the Six Strings": Crafting an Interactive Guitar in 'The Last of Us: Part II'

[Music] hi there my name is mark burrows and i'm grant haigst we're designers at naughty dog this is swiping on the six strings crafting an interactive guitar in the last of us part two so before we begin we want to issue a spoiler alert for the last of us part two we imagine you're here because you've either played the game or you don't mind being spoiled but fair warning so from the very beginning of development we knew that the guitar was going to be a key part of the game's narrative joel's love for the guitar was introduced in the first last of us including a promise to teach ellie how to play even in the very first story pitch for part two this game was bookended with the instrument beginning with joel playing the guitar for ellie and ending uh and gifting her with her guitar and ending with ellie struggling to play on her own journey her missing fingers embodying the cost the symbolize the bond between joel and ellie it's a symbol of life tranquility

and hope and in that same vein it was important to our creative director neil from the very beginning to use future days by pearl jam as our musical through line alongside that one of the very first cinematics we shot for the game was one of ashley johnson who plays ellie singing a contemplative cover of aha's take on me and beyond that the acoustic guitar is important tonally to the world of the last of us from gustavo's evocative guitar infused score to joel's texan country singer roots the guitar then was a narrative and tonal linchpin for the from the beginning so the question was can we put this on the stick it's sort of a mantra that we use at naughty dog especially one by one of our game directors kurt marginal we could have simply left every instance of the guitar in non-interactable cinematics this would let us control the emotional impact pacing and focus of these moments since the player cannot interrupt them with incongruous behavior after all if the player can't smash random chords at a forensic pace they

can't ruin a beautiful moment of joel playing a deeply personal rendition of future days to ellie but then again guitar is fun we knew from video games like guitar hero that playing a guitar in a video game can be a really engaging experience we could make a mini game that was at least enjoyable to play so the question we had to ask ourselves was could we make an interactive guitar in our game that fit our needs for this story well we do have an intriguing narrative arc that we could play along with joel plays a song at the start of the game to introduce ellie to the guitar and then ellie slowly masters the guitar over the course of the game but then her missing fingers at the end of the game undercut that ability and sever this connection with joel this teach master subvert structure reminded us of the parallel design formula in games like super mario 3d world where it's used for level specific mechanics if we set up our system to align with this progression there'd be a direct link between that concept and the narrative journey of ellie in the game

furthermore putting the player in control of the music just as ellie is in control in game would immerse the player more deeply than just watching and listening and when ellie is playing she's recalling one of her fondest memories of joel if done right an interactive guitar could be used to bring the emotional experience of ellie and the player in line by having the player try to remember the song that they originally played as joel so how does the tone of last of us affect this minigame well our world is realistic and grounded so our guitar game probably can't have fireworks and smoke machines there's also these are also going to be some of the most important and character driven moments in the game we need the player to feel nostalgic and wistful so this can't be sex drugs and rock and roll we also need the player to feel like they're practicing we're not providing the fantasy of having the mastery of a rock god so we can't just plop in guitar hero as is all right at this point we think it's worth trying but before we start prototyping we need to define some of our constraints

the first and most obvious thing to point out is that we will not be shipping this game with a peripheral controller much of the fun of guitar hero was holding something that felt like a guitar and having the feeling of playing one in a rock band a lot of that enjoyment disappears when you put that functionality under the buttons and analog sticks of a controller it's much harder to rock out in the squirrel stance so we know that we won't be simulating the feeling of holding a real guitar so what do we have to work with well a playstation controller has four face buttons four shoulder buttons a d-pad two analog sticks and three buttons we can't touch because they're used for meta game functions like pausing sharing and leaving the game none of these necessarily have a great correspondence or affordance to playing a guitar but there is a touch pad this is where i kind of need to step in and say that i've had a personal obsession with trying to make something of the touch pad in one of our games i've done prototypes of painting scrolling around ui maps choosing dialog options

none of these have seen the light of day but they have been attempts to utilize the most maligned input on the ps4 controller now as anyone that is listening is no doubt familiar with in their use of smartphones the touchpad has universal affordance with swiping we can use that swiping action to correlate to the strumming of a guitar right it feels like there's an immediate affordance there so let's explore that now before we start going crazy trying out strumming options in our prototype we should determine some pitfalls and mechanics we want to avoid we should steer clear of anything that goes against our narrative and tonal needs we might not be able to avoid all these things in the end but we need to understand which mechanics might conflict with our game this will help us focus our prototyping and not waste time exploring dead ends so first of all let's say no quick time events for designers qtes are great because they provide a cool unique result with a simple player interaction press square and see your characters skewer the eyes

out of a big baddie however for a player they can tell that their participation is not commensurate with the result it can feel like you're unpausing a pre-canned animation rather than performing the action yourself as the player this can result in the reduction of immersion as they watch their character perform some interesting maneuver they feel they have no control over instead of bringing the player into the action they're further removed and made aware of the gap between the characters and their own actions so we should strive for a one-to-one correspondence between the player's input and the output of the guitar no qtes no playing complicated series and notes from one strum the other issue with qtes is that they're usually testing the player's reaction time and timing in this case is not actually that important for ellie and her guitar playing so we probably want to avoid making any kind of rhythm game ellie is just learning how to play the guitar these are introspective moments both ellie and the player should be able to play at their own pace additionally the final moments of the

game are about ellie's inability to play the notes due to her missing fingers not that she can't keep in rhythm if we want that moment to land we need the player to feel like they can't meet the same metric they've repeatedly met throughout the game and so that game-wide metric should be about notes not about rhythm what else should we try to avoid well the guitar is meant to be a personal connection between ellie and joel who is dead once ellie has the guitar so we should not be playing along with something else that means no dina on the drums but neither should that mean having a metronome the player hears or even ellie humming maybe at most we can have ellie's head bop up and down to the beat but there should be nothing audible that the player might feel forced to play in time with this goes back to our avoidance of time pressure and sense of practicing but it also limits our options on how to tell the player what they should be doing since there's nothing to play to also if possible we should let the player noodle around if they can if a

player is meant to play a b major and then switch to a g sharp minor we shouldn't fail them because they're strumming on the b chord for a little bit longer after all again this should feel like you're practicing you're allowed to make mistakes so in that vein let's not have any incredibly strict fail states yeah so we know the boundaries of the space we're working in in terms of controller mechanical narrative and tonal restrictions we know vaguely what we want the emotional experience to be since we know the beats were meant to convey in the cinematics we've already recorded that feature the guitar and we have some goals we've set out one try using the touchpad for strumming two make it feel like practicing a song and three try to make an interesting experience that doesn't rely on rhythm or time pressure so what are some unknowns that we should look to solve in our first prototype well currently we don't have a way to play different notes and chords we still have all of those buttons and analog sticks available we should try to find something that is comfortable and easy

to use alongside the touchpad we want to make we want to tell the player when transitioning between two chords is right or wrong so we need to explore how to convey what chord they should play next and how to tell them when they mess up this should feel similar to playing a real guitar for someone experience with a guitar but it should also be easy enough for anyone to play so we have our constraints our goal for the mini game and some unknowns we want to explore to help us get to our goals let's start prototyping all right first thing let's get something anything into the game let's start with just the one song that we have we have actually playing take on me from a recorded cinematic so we have something that will show up in the final game that we can use as a target let's grab some temp notes i think grant recorded some midi notes from his phone for this step we'll place those notes and divide the touchpad into six string regions the player can strum through those to play a chord or touch one region at a

time in order to play one note at this point just playing the desired note when this region is pressed will be enough we can worry about how it sounds later on now in take on me there are five chords that are played in the verse and chorus for now we just have to have the player choose a chord to play from a list of five if we put them in a list connected to face buttons however players will have to either check or remember how the buttons are paired to the courts there's a mental leap players have to overcome even when the player when the buttons are placed on screen with their corresponding chords for people unfamiliar with the layout they have to check where a button is creating diversion of that player's attention so the simplest affordance is probably going to be a wheel menu on the analog stick the player can deflect in a corresponding direction without having to look down and in fact the only way to properly confirm an input is on the screen meaning their attention remains in one place they can also highlight a selection

without committing to it like a button press would let's start there now when you're learning a song on the guitar you first learn the chord progression and then you speed up faster and faster until you can play in tempo let's focus on that first half where the chord progression is key for now i will place a yellow square near the edge of the next chord to tell the player what to play next when the player strums with a chord selected we'll either flash green for correct or red for rom this feedback's going to be important for letting the player know when they are making progress so here's one of our first prototypes the ui was made with debug lines and it's pretty messy but it was easy to make changes quickly so it allowed me to make things like the divisions of the wheel what text was in each wedge the colors of the wedges it kind of worked let's take a look [Music] all right i think this is working you can see

there's a lot there that stayed in the final version as a player i feel like i'm consciously choosing a chord even though i don't necessarily know what the connections are between the chords i can follow the yellow square if i want to make progress and red tells me if i'm making an error but even when i make an error i can hear the chord that i chose and even kind of dance around and play chords in any order they like it actually sounds pretty good no matter what i do alright well let's take a look at our main song in the game future days this one's going to be a little different as future days is in a different key and has different needs we're probably going to focus on playing six notes in the melody in the key of c maybe we can show the player the notes instead of the chords this time for this song okay well that's a very different for wheel for future days with very different needs we can still make it work with six slices but we're going to have to think differently about how we use them all right hold on let's back up a little bit more how many wheels are we going to

need for the entire game what are the songs do we expect to play in this game mind you at this point in the project we're pretty early going so the answer's still up in the air we ended up needing one more song for the narrative but at the time it could have possibly been a lot more songs on the guitar maybe in our next prototype we should wonder if we can future proof this system try to handle any song that could possibly end up in the ship game but as we started discussing this next prototype we quickly realized we weren't talking what we were talking about wasn't the inclusion of any song but rather for the ability for the player to play anything now that's pretty exciting to make it work though we needed some music theory before we could answer how many wheels we'd need to let a player play anything we needed to think through the basic structure of a wheel itself as we said through iteration on future days and take on me we settled on a wheel with six slices that felt good for our

interface if we wanted to translate that to a free play or practice mode though that would beg the question which chords should make up a wheel well chatting with mark about it got my gears turning about a different wheel concept from western music theory the circle of fifths this might look familiar to you if you've got a music background but if not no worries at all we'll go through it a little in short the circle of fifths is a musical organization tool that places closely related musical keys near to one another major keys are around the outer ring and minor keys are around the inner and the more notes the keys have in common the closer they are on the wheel adjacent slices have only a single note different in their major scales and each major key is directly outside of its relative minor key which has the same key signature now i know that was kind of technical but one thing this basically boils down to is that chords close together on this wheel sound good together generally a little more easily consonant and used together in more songs than chords that

are further away that means when we are picking our six chords we could take say three adjacent major chords and their three relative minor chords and we'd get a wheel that generally sounds great with itself and would allow the player to play a lot of songs just based off the chord relations of one wheel if we generalize it a little bit and pick the chord at the top of the key of the wheel as the key in this case c that gives us a template of chords that look like this we have our one chord at the top the key of the wheel and the other chords that fill out the wheel are the major four and five chords and the minor two three and six chords now there's a lot more we could dig into there but effectively what we get from constructing one wheel this way is two major benefits first anyone familiar with guitar or this kind of theory in general has a relatively easy to understand structure for the layout of the chords whether they know the circle fits itself or are just familiar with one chord four chord v chord etc

but second for anyone that doesn't know anything about music or theory we have a wheel that sounds internally pleasant and consistent with the sorts of pop or rock songs that ellie has been playing learning and even writing for example check out this clip even as i play random chords around this wheel because of the chords we chose it still sounds good [Music] [Music] that's not bad small tangent ellie mentions to dina early on in the game that she's written some original music since learning the guitar beyond being both fun for the player in creating this introspective tone this free play wheel provided a wonderful and somewhat accidental opportunity for the

player to role play as ellie the songwriter experimenting with chords and progressions at their own pace so now that we know what one wheel looks like how many wheels do we need to cover everything well the brute force method if you will if you will would be to have 12 wheels one for each key problem there is that we have a ton of overlap not to mention how impractical it'd be for the player to need to scrub through all 12 keys or the cognitive load of remembering which chords lived on which of those wheels if there are 24 chords 12 major and 12 minor technically at a minimum we'd only need four wheels of six chords that's a pretty good start but there's one more wrinkle we want to make sure free play still lets players play the real world songs that appear in the narrative of our game how'd we do there well we have ellie's version of future days which she plays in c check similarly there's a small beat where ellie performs the intro to the song ecstasy by crooked still a key part of the dance flashback in jackson which is

in a minor still covered we have joel's deep gravelly rendition of future days which he plays in e flat still in the clear and then uh oh ellie sings take on me in the key of b and if we want players to be able to recreate that song we'd need a b wheel so for total coverage and working within existing song constraints we grabbed five wheels of six with some small overlap those wheels are in the keys of a b c d flat and e flat might not be a perfectly sensible collection musically but there is an elegance to having an alphabetical set of wheels with some flats in there so we have five wheels of six chords players can play every major and every minor chord and they can play future days and take on me if they want to in a single wheel since the player's thumbs are occupied by selecting a chord and strumming on the touchpad we grabbed l1 and r1 on the controller as sensible options to cycle through the five wheels with their index

fingers there were probably other ways we could have arranged all of this but this is what we chose as you might expect though this did have a few side effects first of all in a lucky happenstance any possible chord you could want to play is at most two taps away on the l1 r1 wheel selection cycle since the selection loops around from end to beginning that also would have been the case if we thinned it to just four wheels so in that respect our five was just as good as four second of all we have some missing caves a lot of guitarists might say why'd you give me a weird key like d flat and not a normal guitar key like g d or e well like we said since there are classic guitar keys that have a lot of overlap with one another we decided to sacrifice those keys having their own wheel in favor of minimizing the number of wheels in total and trusting the players that really wanted to play in those keys would at least be able to cycle easily enough to find the chords they needed finally we elected to include just major and minor chords that means no crunchy chords seventh chords diminished chords

anything complicated this was also on purpose while it technically restricts the flexibility of what songs you can play those kinds of chords are especially for this genre more niche and are complex enough that they'd run the risk of sounding dissonant or out of place especially in the hands of a novice we always try to think of that balance flexibility for the experienced but forgiving accessible and encouraging for the novice by the way there's a secret mode where you can play by the fret rather than by the chord you can access that by pressing l3 and r3 with photo mode turned off try it to play some guitar solos from your favorite songs well now that we have a prototype for playing one song and we have a prototype for playing any song let's dive into the nitty gritty of what we shipped with this is going to be a high level discussion of the elements of the final design frankly we could drill into each one of these uh deeper especially with all the interdepartmental work that it took but we only have 30 minutes first feel what is the experience of

playing this guitar well there are some issues we had to solve on the touchpad it's fairly small and awkwardly placed on the controller it can only register two inputs on the surface at a time there are no clear delineations between the regions where the strings are so it can be hard to predict which string or strings you'll play when you tap or strum the sides of the touch pad especially the top curve off and have unexpected detection on them since they don't lie flat plain with the rest of the touchpad and we can only detect a finger position once per frame those limitations are actually pretty huge and we had to work with and around them well most people know how a guitar is held so the natural affordance for strumming felt like it should be up and down along the touch pad however this creates thin regions it's good for strumming through a whole chord but it's bad if you want to play one note at a time for this reason we had an option to swap the axis of the touchpad from vertical to horizontal to allow for more

space per string now since you can only check your finger position once per frame we need to be careful about quick strumming if on one frame your finger was in the b region and in the next frame it's in the a region i'm going to have to assume that you swiped from b to a meaning i need to trigger the g and d the strings in between as well but what happens if you tap the same note quickly over and over again well the notes are going to play immediately on the frame the touchpad is registered but a new note on the same string is not going to play until that region has been fully exited and returned to and even then there is a slight delay before a new note will play from that string this is to prevent only the beginning of the sound from firing too quickly creating artificial sounds effectively we had to let the note ring just a little bit before we killed it and started again finally we added some dead space around the borders of the touchpad so that your fingers didn't have to slide over the top of the curve with the touchpad to count as a high e

in practice this felt more natural when playing so if that's how the player plays the guitar next up is how ellie plays the guitar our rockstar animation team helped us get hand partials for the 24 different chord shapes we needed as well as a few extras for ellie's messed up fingers here in the top video is a look at a few of them you can see that we blend between these based on player joystick direction which is their chord selection we did play around with unique transition atoms between these but the number of permutations became just untenable and we were able to get it looking smooth enough just by blending between fixed cord poses shifting to ellie's other hand we have multiple gestures for both strumming and picking you can see here in the lower video she has a full strum when we determine the player has strummed across multiple strings but she also has individual plucks on her thumb finger forefinger and middle finger to cover plucks on each individual string finally you can see we have animations

that trigger on the strings themselves both physically on the guitar and more abstractly in the ui element so the player has extra reinforcement about which particular strings are ringing out at any given time [Music] finally the sound how does the guitar play well coming back to the needs of the game narrative we actually have four playable guitars over the course of the game with joel's guitar four ellie at the beginning the optional guitar in the seattle music shop the guitar backstage in the theater and a return to ellie's guitar in the farmhouse at the end huge thanks to our legendary music and audio teams for recording and mixing everything we're about to talk about there were 30 notes in total with a couple of extra ones to accommodate those messy buzzed notes when ellie's missing her fingers at the end of the game that was an interesting challenge we really wanted to sell the loss that he experienced with terrible sounding buzz notes but we had to balance that so you could still recognize future days being played so that the symbolism would land

not only that but we had multiple takes for each note what are called round robins and sampling this is to avoid the robotic unnatural sound from re-triggering a single audio file over and over again instead having a bucket of samples for each note helps repetition sound a lot more natural that also meant setting up crossfading on repeated notes to avoid any hitching between samples and finally we have a bunch of muting and slide sounds to fill those transitions and add that extra sweet layer of fidelity so how was the guitar received and what are our takeaways well excitingly players really loved it and engaged with it a lot in the midst of all of the exciting and [Music] impassioned discourse around this game it was really fulfilling to see several articles written up about the guitar minigame specifically and several reviewers cited it as one of the game's most memorable moments more than that players started doing exactly what we hope they do covering songs both casually on streams and quite seriously

in full produced videos if you're interested search around there's various videos on youtube and twitch of people playing different songs some covers and even some originals with a couple articles compiling some of the best together and one of the most exciting and unexpected outcomes some players were inspired by the system and the emotional resonance of the guitar throughout the game to pick up and learn the guitar for real a number of people pointed out that grabbing guitar tabs online could translate to making in-game covers and the system functions somewhat as a learning tool in its own right ellie uses mostly the correct finger shapes awkwardly voiced decor design and that musically minded organization of wheels that we talked about earlier helps inform newcomers which chords work well together and which might be good to learn next so that reception was super humbling very exciting and really validated a lesson that we learned in the process of designing and iterating on this system if we think about a spectrum of ways you could approach design whether it's

creative system like this or different mechanic in general on one side is blue sky prototyping just start dreaming and making think outside the box don't get bogged down and constraints and until you have a lot of ideas that you've tried the problem with this is that there can this can be resource intensive and it can waste a lot of time as you reinvent the wheel on the other end there's rigid documentation planning every single little detail out ahead of time and wrestling to the point where you have a complete and comprehensive design in theory before you put anything into the game this creates a strong goal but the end result might not end up working in the game the way you imagined it did on paper with this guitar we found it super valuable to strike somewhere in the middle with something we'll call scaffolding in this process we have a pre-prototype phase where we call out any external constraints like the controller we're shipping with define any internal constraints like the narrative and emotional tone we want to strike and the kinds of musical mechanics that would

support or undermine that tone and define what our goals are for the prototype in this case testing chord selection and the player experience of succeeding or failing on the guitar then you can jump into prototyping you have an initial framework so you can dive straight in and start searching for solutions to the goals you define in pre-prototyping focus on that core experience for now don't go hog wild just yet but once you have your first prototype you can go into iteration this is where the scaffolding comes off and you can begin to build the whole building based on what you learned in prototyping revisit those initial constraints you can stay open-minded to other possibilities you might not have initially thought of but that still fit within and support your initial constraints and goals explore how to make those possibilities into accessible and understandable reality then test the boundaries of those constraints and surprise yourself we went from a goal to play two specific story songs to a full-fledged guitar

practice mode because we were open to new ideas as long as they stayed true to the narrative and emotional goals of ellie's journey and her relationship with joel with this there's a concreteness and a specificity to the plan ahead of time helping drive scope and vision the scaffolding but also a flexibility to adapt and grow your design as you prototype when new ideas or considerations come into play like a free play mode for example in the design process at least in our process pre-planning and adapting aren't a dichotomy they're both essential parts of creating a great holistic system thank you so much i hope you enjoyed this many thanks again to our incredible co-workers without a system like this would definitely not have been possible and thank you all so much for listening we hope you have a wonderful rest of your gdc [Music] you