Moving the Heavens: An Artistic and Technical Look at the Skies of The Last of Us

[Music] okay couple more friendly familiar faces in the audience hey you guys doing today good first day GDC is going really well yeah okay man what a pretentious title moving the heavens set the bar a little high didn't I all right well good afternoon everyone my name is Keith Garrett I'm one of the lead effects artists at Naughty Dog don't worry I don't take this title too seriously I just wanted something catchy and judging by the number of people here it worked so if you could all do me a favor now lower your expectations tremendously we'll go out and get started before I forget I'm sorry before I begin I want to just take a quick serve and get to know you guys so how many people out here are programmers good raise your hands okay how many of you are artists okay how many of you are what we call a technical artists the people that kind of fall in the cracks

yeah so you're all the ones that are good enough to be an artist or a programmer right like me no I'm kidding it's okay well today I want to talk to you guys about sky's something that's the most overlooked and often stubbed in part of your environment part of your visual presentation it's the sky it's also got a tremendous amount of power to completely transform your visual presentation yet again nobody really gives it the do justice and due diligence that it deserves so I want to talk to you guys about what exactly this guy can do for your presentation and then I want to talk to you guys about probably the most difficult challenge when you're discussing a sky which is how to make it move we have a lot of different things that I want to talk to you guys about in this I've got a ton of different images and videos to show you guys a lot of them are not necessarily mine so I've done my best to kind of sort of cite my sources the entire way through but please just keep in mind

that I want the due credit for the artwork that I'm showing to go to an artist and the last thing is I've like I said I've got a ton of stuff to cover so removing a little bit fast okay I want to talk with you about the power of the sky from the moment we began working on lost of us we recognize that the sky can change everything about our game that for us to create involved and emotionally driven narrative we had to pay attention to the sky we had to utilize it as a tool to sell our delivery let's start with a few of the obvious points first of all time of day assuming you're in a normal outdoor setting the primary light source is probably going to be the Sun or the moon as the time of day changes your source positions in the sky changes the light direction specular highlights they change the shape of the shadows change in fact your entire composition changes

everything because the time of day has changed if you're a decent photographer one of your most important skills is to see that great light can make a great photograph composing your shot around the white direction the shape of the shadows and the highlights on reflective surfaces can turn something drab into something stunning lighting has the ability to take any surface any material and completely change the character within it that you as a viewer perceive for example this is one of my favorite demonstrations of the effect of lighting the video almost appears of several different faces some intimidating some attractive some sinister and some warm and friendly are all being merged and blended together lighting also controls

the schedule of the narrative in the last of us the time of day the time of the year played a huge role in the narrative passage of time was extremely important for us to consider and you can see it through our entire production but especially during the concept phase so something as simple as the passage of time throughout the day can dramatically impact the presentation of your art this is the first component of the sky that needs to be considered when you're in pre-production the second fairly obvious one is the weather is it a clear sunny day in the tropics somewhere your sky should be cool baby blue with a soft gradient to the horizon maybe a few fluffy clouds something that immediately makes you feel the warmth and relaxation of the tropical Sun in this case I can already taste the pina colada where does your level have a beautifully dramatic and dangerous rainstorm looming on the horizon

maybe something more dramatic gloomy desaturated overcast sky foreshadowing the harrowing journey ahead you see the weather while obvious can completely change the amount of tension in your level it's also an excellent tool to see what's coming and possibly what's just recently gone you can foreshadow with the sky because the player can actually see out to the horizon so these two points are a little bit obvious essential but obvious let's go a little more abstract a little more fuzzy a little more artistic consider the sky this is an amazing photo that our art director Rob rupal took from his iPhone near his house just keep this in mind for a second while I change the subject you've all seen a painter lay out the color palette before primary colors are chosen carefully shadows mid-tones highlights the entire composition in fact they all revolve around the choice

of colors and their complements colors that resemble the intentions of the painter beyond just representing an image beautiful interesting paintings are made with very simple color palettes different colors have different impacts on humans and are often related to different moods warm colors can be used to describe the tone of energy anxiety high tension in fact it's often these colors that draw our eyes the most blue can often be relaxing cold if not melancholy is a very simple basic concepts from describing artwork and calendar theory yet they're so rarely actually applied to the production of an environment to the overall composition of the level that you're trying to sell to the player again in my opinion the sky and all the dependencies are one of most overlooked parts of composing a visual experience for players so let's go back to that photo I showed you

earlier and look at the color palette the tone the color temperature the entire color palette and emotion that it drives they're all defined by the sky and that is perhaps the single most important point that I can make in the real world the sky drives the lighting and therefore the color palette of your entire environment in games we can do things backwards but the sky the lighting the color palette the time of day the direction they all need to match because the sky is what motivates them let's look at a few more dramatics guys from a lush yellow sunrise that shows the promise of the day to come to a deep red sunset ending the day with in which blood has been spilled according to Legolas change to the sky and all of its dependency dependent components have the ability to change your landscape from one significant emotional tone or narrative tone to a completely different experience for the viewer without

touching the base geometry without touching the materials without touching the textures which leads me to this last point before we move along the tone that we present visually in all of our environments has a tremendous impact on the emotional experience of the player for us the tones we present to the player in the last of us were extremely important to the narrative structure to the emotion that the players the post experienced when he's playing through the game and you can see it mirrored in our choices for composition again all the way from the beginning whether it's a foreboding storm cloud reaching forward from the horizon hinting of the dangers to come as you set out on your journey or the desperation and loneliness as the beautiful colors are leached out of the sky as it sets or the peace of mind and comfort and warmth brought from a bright sunny day the sky can be used as a tool to present to the player whatever tone or mood is character supposed to be

feeling because we feel it too if I show you this picture again most of you are gonna feel the warmth probably a little bit of relaxation hopefully a little bit of joy if you've had good experiences on the beach the sky is incredibly powerful emotional cue for the viewer Neil and Bruce our creative director and game director respectively notice and they've focused on it from the start which is one of the things that I loved about working on the Last of Us you can see that even when they're planning out the story when they're writing the narrative script they were focusing on the skies and how to tell that story through the sky so to summarize that point an environment completely in sync with the narrative tone can be used to deliver a pointed and impactful player experience inversely you can actually ruin the

narrative tone by having a sky lighting visual tone that just doesn't match it's out of sync in summary the sky and all of its dependencies and lighting can change the entire look entire feel of your game so it's worth starting off with it's worth considering from the pre-production which is something that most games have not traditionally been doing that we're starting to see a trend uplifting please don't get me wrong I would never argue that shapes contrasts negative space modeling texturing local lighting effects animations sounds don't matter but the sky again is one of the most underutilized tools in the portfolio games that we've seen so far and there's nowhere to go but up hey anyone know all right it's not for that let's change the subject I always like having kind of

some takeaways to presentations that was the fluffy side of this presentation now I want to talk about some tech let's let's make some friends with math if you're working on skies in a game at some point somebody some jerk is gonna come up to you and ask you to make it move and it's just a bottomless pit it's a bottomless pit that we've been exploring for a long time and that with Edmunds help actually we were able to somewhat resolve for our own production if the on the last of us at Naughty Dog so first and foremost skies move I hope you guys can all agree with me on that this guys move usually in very fascinating complex ways in the past few years I've seen heard of been taught experimented with so many different techniques to fake cheat and imply the motion of a sky to list a few I've stuck clouds on a cylinder and rotated the cylinder around the vertical axis which is great until you turn around and realize that the

clouds are still moving the same direction behind you I've used a scrolling plane of clouds which you can make better by sticking an alpha on it break it up a little bit but it's still missing the vanishing point it's still missing all the cool lighting changes and the cool motion as the clouds cross the horizon I've tried layering tons of complex motion and materials to make fascinating skies the skies of Halo Reach are still my favorite in any game I've seen today so props to them I've used volume metrics which are awesome a lot of fun but they don't quite give enough artistic control over shapes and lighting and of course they're very very expensive so not an option anyway and well not necessarily technique in itself I've done time of day systems which are usually just a combination of all the other systems that I've ever tried and Rockstar has definitely dominated that I

can't even properly state how much respect I have for the job that they've done if there's guys you see every technique that I've tried comes with very different pros and cons and more specifically it's always going to end up boiling down to more time for iteration skies take a lot of time and the challenge of adding motion to skies really boils to that one simple issue the more custom assets you build into your sky the longer it takes to iterate and adapt to the skies I am a firm believer that if you want beautiful skies across your game it needs to be given the attention of a full-time position there really needs to be more sky artists out there but who has time for that we at Naughty Dog certainly don't and we also didn't really have the ability to use either of those or any of those techniques because we need matte paintings we need matte paintings because we need artistic control you see them not the focus for skies and howdy dog has always been artistic control we

need the clouds to shape the composition the sky to highlight and draw the players attention we need the world to tell our story at the perfect point of the reveal in other words we need matte paintings like this because of this the skies and our games have always been a single static matte painting the skies have polar distortion painted directly into them similar to a panoramic photo you would take off a chrome ball or most of the sky templates that you would download online we then take that texture and we projected onto a dome using spherical u these just standard Maya script Levy's and then the dome is finally attached to the camera so that no matter how much you move the camera you never get closer or farther away from the dome obviously the rotation is not attached to the camera if you're curious we always try to ship with 4k by 2k textures never

ever ever get away with it so we always end up doing a 2k by 1k you know extra memory and all that the other reason that our skies had to be matte paintings is because we iterate we iterate over and over and over again Naughty Dog is quite famous for how close to the line we iterate huge huge lighting changes last week before we go GM is no joke and so obviously this guys need to keep up for example this is this guy that was in the military city of the Last of Us maybe a week before we went GM and this is the sky that we shift with now obviously the lighting hasn't changed too much from this but if I had actually put motion to this using layers I would have been spirit I would have had a very small amount of time to iterate over all the work that I'd done in the past obviously this change was a good thing for the sky but it's something I need be ware of so here was a great challenge on the Last of Us how do I create the illusion of motion in the sky without needing to stick myself into the iteration process

in other words I couldn't split it into multiple assets and our inspiration from this actually came from uncharted 3 how many be guys remember this level okay a few hands so what we're trying to do is we're trying to have a wall of sand being held at bay by the wall of this mystical city that wall of sand is supposed to look like just this horrendous storm kind of on the outskirts of the city we dubbed it the brown donut but what we attempted to do was treat the sky the same way we treat water which is using a UV distortion technique called flow unfortunately we didn't actually have time to really get this working for uncharted 3 so it left a bit to be desired but the seed was there and we kind of took off running with it during the Last of Us so before I really dig into that I need to talk with you guys about what exactly is flow I don't have a ton of time up here I want to make room for Evans presentation next so I'm going to skim this a little bit and give you guys some more resources but I do need you guys to understand the broad spectrum of what flow actually is so it's a very specific type of UV distortion where I am able to

specify specific directions and stretch a texture based on those directions then once I get that stretching looking nice I can just crossfade between an earlier version of that texture and it looks like the is continuous for example if I were to paint directions using any number of flow tools out there this one is Maya fluids I get this texture I'm gonna pause on this texture for a second because I need you guys to understand what's happening with it I'm gonna be applying this to the U and V channels of another texture which is to say that the red channel of this texture is telling me to push the pixels in the U direction either forwards backwards or not at all based on the intensity of the red 50% red here is no motion at all pure red is forwards and you black is backwards in you the same goes for green so if we go back and look at this base texture it

doesn't look like much but all of that vector information is in the texture this is really hard to paint by the way because it's very specific now if I apply that texture you can see that we get distortion and notice the cross fading it looks like it's continuous motion here are a few more cool examples zoom a twist here it is practically used for the padding of water and here was being used for the ground particles the wind this is the slide you guys should take photos of if you want to learn more about flow Carlos who is the person who implemented flow for us back in uncharted 1 made a presentation in 2002 I did as well in fact those examples are from it and then the original resources are cited there as well

okay great you guys know what flow is but how do we apply this to the concept of a sky moving in the world how do we break down the motion of a sky in broad terms so that we can consider applying this in a faked scenario if you look at this carefully you'll notice that there's actually something happening here that hopefully you remember from your perspective drawing class 101 perhaps there's a vanishing point there is a point where all parallel lines because the clouds are actually moving in parallel are converging from our perspective or merging if you notice all of the motion is going down to that you know roughly that one point if I look straight above the vanishing point there's no horizontal motion if I look to the sides there's no vertical motion of course I can turn around and in this case I just flipped the video but hopefully you guys get it it's the same

except now instead of shrinking to everything that is growing from that vanishing point so getting closer but I need to start thinking about this in terms of a dome because I'm gonna be applying this to a Sky Dome in my game so if I instead take this concept and apply it to a dome how's that gonna work now as we discussed before we need to break down the motion in terms of U and V for it to be useful for flow you being the direction around the sides of the dome V being the direction up to the top of the dome so let's consider for a second just the vertical motion it starts at the back and it rises until it reaches the top so for the back half of the dome it's going up and on the front side once it reaches the top it changes directions and it goes back down to the

opposite pole so it's going down in the front if we consider these as colors keeping in mind that again 50% green is no motion black is down and pure green is up these would be the colors describing the motion that I've applied now obviously this is way too simple to describe what's going on in the sky but it's helpful for me to start thinking really broad terms like this so what I really want to do is I want to take that and I want to find a way to make it go slower around the sides because I'm trying to fake perspective so the clouds that are supposed to be the farthest away from me need to be moving much much slower than the clouds are the closest to me which are directly above me to do that I could paint a gradient but through Evans help we were able to figure out how to do this mathematically which is just way better this is the correct gradient it doesn't look like much right now but bear with me for a little bit and you'll see it working if you're curious how to get this mathematically it's really really really simple if we just take the wind

direction and take the dot product of the wind direction against the vine normal of the sky we get this gradient perfectly another way to think about it is if you convert the wind direction to tangent space and then extract the green channel this is what you get okay you guys feeling good if I confused you still with me a couple smiles you're always smiling I like that so let's consider this the u motion the sideways motion once again we want to separate out the vertical so we're not looking the vertical at all just sideways the motion is going around the Dome parallel to the wind direction which means that I've got positive motion on one side and negative motion the other side but again that's an oversimplification it needs to be the slowest at the vanishing points and the fastest at the sides that are supposed to be closest to me so again I can do this through math this time the math is taking the wind

direction dotted against the tangent of the SkyDome or the red channel of that previous function here they are combined which again doesn't look like much you just want to see it in action okay I'm going to take those flow directions that we just saw in the dome and I'm gonna apply this texture to the dome this is what we get we now have a completely procedural motion creating the illusion of the clouds moving out into the horizon with

a mathematically correct management point my favorite part of this technique is that I'm not bound by the content of the mat painting at all like at all at all I can literally swap the map textures and do no work or okay honestly almost no work I've almost completely remove myself from the iteration of the process of the sky playing okay here's where it breaks down like all good things in life there's always something negative about it in this case the fact that our sky is just repeating its motion means that it's vulnerable to showing that it's repeating its motion depending on the contrast of your texture you can easily see the repetition for example notice that hard spike at the top the cloud if

you look at that you're going to pretty easily see that oh look it's repeating so this is something we had to consider we have two decent solutions to get around this one is I can just change the wind direction so that it moves in a way that the repetition it doesn't show or the worst case is I can actually mask out the motion of the sky I say the worst case because we're getting a sense of depth and parallax and the motion of the sky because of the speed that it's moving so if I slow down sections kind of willy-nilly it completely screws with your perception with the illusion of this guy the other one is that there's a very large ugly pole at the top this is always a problem with skies it's made much more apparent because of the flow we naturally called this the sky sphincter we can minimize this by painting a solid color on the top of mat painting we could stick extra cards up there to cover it up and put their own distortion on it or you can do what we did in last of us and just not allow the players to look up that high you know either/or

okay so a couple of retrospective thoughts this one's a very personal issue to me how many of you guys actually noticed that the sky is moving while you're playing the game that's a lot more than I expected you guys are sucking up no you sure okay we had a lot of debating over the speed that this guy should be moving do we make them fast enough so you notice it while you're running around or do we make them slow enough so that you notice it when you actually look we chose to favor subtlety rewarding the players for actually taking the time to look around and taking the scenery but I'm not sure that it's necessarily the right choice I'm gonna choose to believe that it was but that was the end result so I've noticed when a lot of people are playing the game they tend not to even notice that the skies are moving I also still choose to believe that even if they didn't notice the skies are moving just the fact that there is that much more motion and realism in the world adds a

tremendous amount to the player experience that's pretty much all I've got right now hopefully that wasn't too much for you guys and I've got three minutes to let Evan come up here and take over for me so I'm gonna step off to the side and take questions please come up and ask me and then Evans gonna take over with an awesome presentation about its liquid from the Last of Us liquid particle effects in particular and then if we have more questions one seven starts then I'll move outside cool thank you guys so much [Applause]