Sand Rendering in Journey

[Music] all right I was told everyone should turn off their cell phones if they want it's always funny when those go off during a talk okay this is sand rendering and journey here are some images of journey and to give a little context for those of you who haven't played the game it's a multiplayer PlayStation Network game and one of the things we're trying to do with the game is recreate the feeling you get when you're out in nature hiking and you encounter someone else on a trail after not having seen anyone for a while there's this desire to connect with them in some way to say hello maybe even to journey with them for a while you want to acknowledge them on a human level which is something that is strangely uncommon in a lot of online video games we thought the best environment for this would be a vast and barren desert until we spent a long time on the technology

to make the players feel like they were really in that environment to trigger that desire to actually make a human connection with the other players so the structure of this talk here is this is an image of the shipping game and first thing I'm going to do is I'm going to sort of peel away the various hats we use to do thus and rendering one by one so you can just see them quickly and then I'll spend the majority of the talk going over the hows and whys of each of the various packs so let's get started with that first one we have here is we have a MIT map sharpening effect you can maybe see that it's a little bit smoother in the distance this is the original image again it's grainy and then it gets smoother in the distance next we have an anisotropic failure masking half here it's that kind of radioactive oatmeal on the silhouettes of the dunes next is our glitter specular hack sand has gotten less shiny

another ocean specular hack that beam of light by the players going away then switching to a Lambert shader for diffuse we have a custom diffuse here and this is just a vanilla lambert and then finally our detail height maps this is sort of a displacement map we put on the sand notice it gets very smooth and with all the effects removes it's just kind of maybe a big bowl of pudding here I'm gonna go through that again quicker this is a different environment journey is full of many scenic environments and this one is a bit sparkly or in spookier so shipping game are MIT Maps the anisotropic mask specular number one specular number two switching back to Lambert's and then removing the height maps and this is you can't even see the terrain anymore journey in the void one thing I should say I was calling a lot of these hacks and I'd sort of like to address

that before moving on with the family vacation photo portion of the GDC talk there's a good reason for this there with me early on in development and journey the whole team took a trip to the Pismo dunes which are just south of here in California and we spent a few days just sort of like running and jumping and diving through the sand to get a feel for it and that actually served as the primary reference for all the technology development of the visuals of the sand we didn't use photo references until very very late in the project because we were never going for a photo real look we were going for sort of a feel real look we wanted to generate an authentic real feeling in the player so they would behave like real people when they encountered other players in the game and they did want to sort of make a a real human connection with them okay so with that said onto the first half here this is sharp MIT maps again you can notice that with them turned off the sand in the distance is much

smoother and actually this is probably more photorealistic if you take it take a look at pictures of the sand this is photograph of real sand dunes or really more like a bland dune if you look at it you can't really see much detail at all here it has that putting a consistency and we can actually do some math to kind of see why this would be if you look at this screenshot here the character is a hundred pixels on screen and if you imagine characters about a meter tall because the journeys are short and like round numbers then each pixel by his feet is one square centimeter and then this is actually a zoomed in view of real Pismo Beach sand where the team went for the reference and you can see that maybe on average each grain of sand is about a half a millimeter wide which should mean that there should be four hundred grains of sand in each pixel by the character's feet and at least for the diffuse shading if you imagine four hundred just different grains of sand facing

different directions actually all even out to create that nice smooth look you were seeing on the photographs but the first thing you notice when you actually go out into the dunes the first thing you feel is just how grainy sand is and I think part of that is the fact that eyes are still higher resolution than TVs a few years left with that but mainly I think it's that you're literally feeling the sand on your body you know and your hair and your clothes in the places where sand was never meant to go and so we push that graininess out as far as we could which ended up meaning a different sand grain per pixel the way we did that by the way is we use this sort of detail normal map here it's just basically noise there's a histogram of the normal map on top and you can see that most of the normals are pointing straight up and then there are fewer and fewer pointing off in either direction in sort of a bell-shaped fashion and so normally if you generate a MIT map using

like a very simple box filter you'll get something that looks like this and you can see that the the histogram here is much tighter which means that the normals are more pointing straight up and you're going to get a less grainy looking texture but if you look between these two things it appears that the second one is about half the size and so we multiply the normals by two and we got something to look the same and I'm not a mathematician if there are any mathematicians if you want to explain how come this is true to me my email is John at that game coming calm for anyone else you can just if you want grainy normal maps you multiply by two the next thing you notice when you go to the dunes is just how sparkly it is and pictures do not do this justice you can see a few sparkles we managed to capture on camera but when you're actually there it's just you know sparkles leaping to your eyes and so here's another image of the game of the sparkles are turned on

they're turned off here and it's probably hard to see so I'm going to zoom in this is sparkle turned on and here's Sparkle turned off and you know it's it's maybe subtle but when you're actually playing the game it's very very important for capturing that feeling and when we were initially trying to implement this effect we sort of thought it through logically for some reason but we thought okay what's going on what's making those sparkles it's probably the sun is shining down and there are certain grains of sand that are oriented just the right way so they'll reflect the Sun directly into your eyes and then you'll see that brilliant glint and you know grasp arms like oh well we know that is that's just that specular reflection we know how to write specular reflection shaders and that was the actual initial attempt it was just a straight specular shader on a very sort of noisy normal map and it looked like this which is pretty good I mean that looks like glitter and so worked fine for still shots but the problem was in motion hello

join GDC so you see it in motion here and it's just got a very noisy look too it's like you were clubbed over the head about to get a migraine kind of look so and I think the problem is that the just fidelity like the maybe the bit depth of our normal maps and the number of pixels we have on screen and just not enough to use the realistic equation so it would just a leus too much so we thought about the feeling of this effect and really the important part was that when you moved your head the sparkles the glitter would move around like at its base it's just this glitter is a view dependent thing and so we just swapped it on the left we have the standard sort of more physically-based specular equation n dot H and then we just switch to the basically the most view dependent you know basic view depend thing we could

think of its normal dotted with the view and that ended up looking like this which may actually still look very alias to you if you're actually playing journey in honor unreal TV you can see that it's a bit smoother and it's much easier to control here are some equations which may or may not explain why this is the case again mathematicians it's John at that game coming calm there was a big problem with that last glitter effect and you can see it here it's that radioactive oatmeal kind of look and I mean this is actually such a huge problem had to give us a lot of headaches we nearly can the whole effect took there's a wild even figure out what it was but sort of lead you guys through that thought process journey has sixteen tap anisotropic

filtering on all the sand textures which sixteen taps is the maximum that the PlayStation three allows and just to get a sense of why I'm going to show you the same scene with no anisotropic filtering and it's actually a little bit easier to see in this spooky scene here so here's kind of how we shipped and then here's with anisotropic filter turned off kind of has looked like maybe a giant ate the oatmeal and forgot to use a handkerchief when he sneezed so anyway I think this might Ellis trait an interesting visual phenomenon though and that's if you have detail that is pixel size they're probably just uniformly sized the player will interpret that as graininess so if you have uniformly sized detail the pixels by the character's feet don't look like four hundred times too big sand boulders they just look like nice for a nice and but as soon as the detail becomes non-uniform people start to take it literally and so as soon as your sand

is one pixel too big by stops being a nice rolling do and then become you know giant pile of radioactive boogers so that then is explaining what's going on here in the circle section even the sixteen tap anisotropic filter was not enough to properly get per pixel detail on these edges and we spent a long time trying to improve the anisotropic filter which proved to be a task we lacked the cognitive capacity for so instead we swept it under the rug we just detected where this was failing and stopped drawing the pixels there the way we did that was using this kind of texture here this this texture this is the texture in its mid chain it has a specific property where it will have very predictable values of color based on distance from the camera assuming perfect anisotropic filtering and you can see the terrain here with the

texture applied to it we added some dithering or some noise of the texture for dithering but anyway you can see that it's very dark close to the camera and gets lighter and lighter farther away except in certain spots where the anisotropic filter is failing and so we use this to generate a mask and the white areas now are where we know that it's safe to render things that need full anisotropic filtering and the black areas are where the anisotropic filter can't quite keep up any graphics programmers in the audience at this point may be asking the question well why did you go through the trouble of generating this texture we could have just used the derivative functions in CG there's a good answer to that so that we didn't know about the derivative functions which I think this whole episode is kind of a heartening lesson in the fact that to be a graphics programmer you need neither intelligence nor competence which is a line for all the artists in the audience to take back to the studio so all right we need fix

the anisotropic fill the problem but if you look at this image a lot of the screen is actually blacked out and that meant that we still weren't getting the Glitter enos we needed so we add more glitter this is this beam of light here never actually seen it on sand in pictures or in dreams it's completely unrealistic what it is though is it's it's basically the same shader you would use to reflect the Sun or the moon off of water it's kind of like an ocean specular and we just sort of tried this out one day in our deep despair and we noticed that it felt correct and so I think maybe one of the reasons that is is because when you're actually out in the dunes it has this very fluid kind of feeling this is a video from our

Pismo trip and you can just kind of see the sand is rolling down the hill it's flowing down the hill you sink into it it feels like a fluid and when you're standing on top of the dunes you really get this feeling like you're actually in an ocean of sand which is not a unique idea there are quite a few hits for that when you search it on google nearly as many as ocean of lard by the way we also ended up throwing on a well a fluid simulation it would simulated neither ocean nor any known fluid but it still had a good feel just adding that additional amount of fluidity made it actually feel more authentic even though it's totally away from reality okay and so that's done Sparkle is done we finally solved that problem so now onto the diffuse shader here's what we used in journey here's a Lambert notice the journey shader is quite a bit higher contrast and one of the things that we

realized sort of over the development of journey and I'm surprised it took as long as it did but basically Lambert is a terrible shader for almost any situation like any literally any change you make to it is going to be an improvement and so once we realize that we started looking into alternatives and one of the obvious ones is possibly using Oran air which i think has gotten pretty well-known now but it's a physically-based shader for for diffuse lighting and we did end up using it on the cloth actually but the problem with or Nair is that it's actually pretty expensive and the thought of this being over half of our screen made our RS X's shrivel so instead we took some some inspiration from my favorite way to reverse a string which is the random shuffle verify method which basically means you just try out random stuff until it works and

so here's kind of a time-lapse view of the diffuse development on journey we started off with the ugly Lambert added a four here and led that by 0.3 and that was the shader and I'm not sure if there's a lesson in this if there's anything I think it's that especially you know as programmers and graphics programmers we like a certain amount of rigor and that's good that's admirable but sometimes you just need to random shuffle verify and and and that's good enough and and we should admit it okay now the final effect and not going to call this one a hack because this one we actually only added after looking at photo references and so I guess you know photo references are good too but late in the project we decided the Sam needed something else and we started looking at pictures and realize it was that little sort of fine-grained geometric detail

that you can see and the way we did this is well so the dunes and journey are represented by a height map and actually it's each level has a very low resolution 256 by 512 height map which when we are sort of streaming into the vertex buffer we are using a b-spline filter on which makes it incredibly smooth that's actually a second derivative continuous which allowed us to use second derivatives for ambient occlusion but anyway on top of that very smooth height map we added these high resolution tiling height maps these are the ones we used in journey there are four of them and this was just sort of by looking at the photo references we saw that on shallow sort of like not steep areas of sand there were these very tight little ripples that were kind of going against the grain and so we use the top textures for shallow sand on steep sand there were very low-frequency waves going sort of with the slope of the sand and just you know we detected sort of

the the slope and the x or y orientation of the derivative and applied one of these textures and that's actually it for the the sand effects and I don't think well I don't expect that everyone here is going to run back to the studio to implement more sand games they're cool but there were a lot of them actually in 2012 so I think if there's anything to be taken from this talk it's that it's it's the that game company process for visual development and it's sort of approaching and not from the physical or photographic but from the actual human experience going inside the player approaching it from the feel and letting that be the guide for all of your graphics development okay that's it I'd like to thank Martin Middleton who did a lot of the terrain work on journey and art director Matt Nava who's here as well they're actually a couple more

journey talks happening design of journeys happening later today creative director Jennifer Chen and Matt Nava giving a talk tomorrow oh yeah and we're hiring by the way graphics engineers can apply to we got a lot of engineering positions so check that out all right guys I that is that is all oh yeah I was supposed to say if you are gonna ask a question please go to the mics hi there so I actually did try to implement this system based on the blog post that I think maybe you wrote or someone from the team wrote and the sort of important thing I guess that I got from the blog post was these multi layer layer with a level of height maps where you had a large height map for the dunes and then the smaller high resolution

height map for the sand waves mm-hmm and I found that basically in order to actually represent the high resolution waves well I just needed to crack other vertices if it gets a mess to use so yeah this is on the PC and I was using a tessellation shader that kind of you know fell off over those of detail but still it just didn't quite do it was there anything that you guys worked to get the mesh resolution sort of in a manageable way or it's just the ps3 being awesome and being able to do that pretty much everything we did was was probably in some way in error I know I we just had a fair number of births it wasn't a crazy number we got you just need enough tessellation for to stop looking ugly and so those those high frequency waves were actually vertex made all in perfect no normals not yeah the only normal map was the actual grain texture okay everything else was geometry less impressive hey hey dude

can you briefly elaborate on how the hell you made the sand roll down the slopes when you walked in it yeah well yeah I didn't cover that here but for the actual sand grain normal map that was generated by a 2d particle system that we rendered on the SPU's so it was I think around 10,000 particles they were like 30 by 30 a few little particle with sand texture on them and we just we had a little physics simulation and a renderer for generating that texture and so when your guy walked over it he would exert forces on the particles and it was a toroidal so it could tile okay but

that's how he did it yeah cool thanks all right I think wait yeah I think that's it thank you [Applause]