Fast and Flexible: Technical Art and Rendering For The Unknown

[Music] okay hello everyone I'm going to get started it's a busy talk so so hello everyone thanks for having me today my name is James aner and I'm a principal technical artist at Sony's London Studio where I've worked for almost nine years uh the team I'm on is currently working on virtual reality for Sony's forthcoming PlayStation VR and I'm going to give you an overview of how we developed tools I'm going to give you an overview of how we developed our tools and Technology around it so I'm going to give you the mandatory reminder to set your phones to silent and fill in your evaluation forms so first I'm going to talk about the early work on on new engine uh that we started preparing for PlayStation 4 and VR next I'll have a look at the the experiences the studio has shown so far and how they've shaped our engine we'll then have a look at the tech that we developed since last year and finally content optimization for VR we don't

have a lot of time and there's a lot to cover but I'll see if I can fit in a couple of questions at the end and otherwise I'll be in the wrap up room uh just over the way so London Studio has a long history of working with peripherals we pioneered camera based gaming and augmented reality with I toy ipet and wonderbook as well as working with microphones and companion apps on singstar so working on VR is natural fit for us as we used to exploring new technologies and the possibilities they bring we became one of the first Developers working with morphus which was a Cod name for PlayStation VR and due to its name morphus became the source of all my dreams and nightmares over the last two years a dream because VR is incredibly cool to work with and being able to inhabit the worlds you create is amazing and a nightmare because there are a hell of a lot of challenges in being the first so our remit as a first-party developer and being VR evangelist is to introduce and showcase VR to the world which means both the public and other developers still this doesn't really help us to focus that much what is a VR game it could be any genre setting and

in the end we decided to work on many different games to best explore what VR can do as a result we decided the main focus on tools and Tech should be flexibility and speed so we could be reactive to whatever Concepts we thought were going to work best we developed a brand new engine called LSS DK which is fully multi-threaded and designed with the PS4 architecture in mind to keep things flexible we Ed a tied forward renderer and this allows us to add new shading models easily without having to worry about packing attributes into a g buffer and also means transparen can be handled identically to P objects we also wanted to have the option of trying out stylized rendering and lighting models we build a PC version alongside which isn't optimized but allows us to have full wizzy wig editing on PC for my part I implemented the direct lighting models for new engine based on the ggx lighting model as well as shading models for pre-integrated skin hair and eyes we put a large effort into making a new game editor called Creator Creator ACC as an interface to the game engine which is running alongside in a separate process one nice thing about this is if the game crashes for any reason crat

carries on running so you can just reboot the game and carry on where you left off you can hot reload any resource without restarting the game so models textures animations and scripts can all that data on the Fly this gives us a oneclick export for most of our assets to seeing in the headset you can also have multiple tabs open running game sessions on different platforms and this means you can have the game running in VR on PlayStation 4 while you're editing the game through another camera on PC and this ex Creator although it looks like the game is running within it it's actually a display from another process we also use a remote play functionality that's used for streaming to visa to give you a display on PC while you're editing on PS4 we use Myer as our primary art authoring tool that this is larger for asset creation so model skeletons and animations the placement and gameplay functionality are done through Creator we don't run our engine Meer but our Shader system is available here which means we can get a reasonably good idea of what an asset will look like in game for the new engine we decided we'd have all our shaders go through materials which is worldwide Studio

shared no based material editor having a visual editor helps to increase the number of people who can author shaders we write the lighting models in code and expose material parameters as an interface in a nogra personally speaking I prefer writing code to noph a lot of the time for complex effects big no graphs can often end up being right only so you know what you're doing when you author it and it all seems to make sense at a time until you come back a week later and it just looks like spaghetti we do alleviate this to a degree by creating custom noes with Shader code in so all the not orange nodes in this graph are bespoke code with multiple inputs and outputs and this really helps us to use the best fit of code versus nodes where is suited it takes us about a minute to generate a Shader for PC and PS4 but we can iterate on them more quickly in Mya as a my version of the Shader uses a single permutation it only takes us a couple of seconds to recompile so now I'm going to talk how we applied uh the tools and take to our experiences so we've been developing on VR for 2 years now working alongside development of the headset and

showcasing experiences at various shows and events throughout the engine is also grown alongside uh in reaction to the team's needs so I'm going to go through all of our experiences in order to show you how the engine is developed so first I'm going to show you some footage of our experience which has the working title into the deep it is still quite funny to hear screams coming from meum where people play that for the first time we experimented with many different concepts but the one we wanted to show first was into the deep we showed this at GDC 2014 with the first public showing of Morpheus the deep is a trip through a coral reef in a diving cage down into the depths of the ocean where you encounter an enormous great white

shark up close and we already had some pretty unique rendering requirements we needed physically based scattering and absorption to give it a realistic ocean feel um cor stick switch to fuse as you go down further into the ocean and we didn't even have a Sky Box so we have to have a custom underwater water surface Shader to refract a cube map from above the water and reflect below the surface but more than that this was really the first tech test of our engine and tools basically everyone was getting up to speed at this point and learning how to use physical based materials we didn't really have much time to rest though cuz we swiftly mov on to our next game which was v l v l is an adrenaline filled Rush down a mountain side on a lose dodging traffic controlled entirely through the headset through leaning we showed this for the first time e32 14 so it was

pretty soon after the last demo for the L we developed localized Cube Maps um as we wanted to have Reflections as you go through tunnels we also added an atmospheric Sky simulation to give a realistic Sky color and scattering on the scenery which can be augmented with Cloud textures we add decal support at this stage as we're using a forward render decals can be quite expensive so we tend to use uh steni out on decals where possible but we do have the option of Blended and our prepass also outputs at normals which allows us normal bending and the next game we developed was London Heist I'm going to question Little P I don't give a monkey FS about you I don't trust you and you know why cuz there's only two people that could have second Serena today you or me and me now you tell where [Music] I Psy so our first showing of The London Heist was last year at GDC and it's an experience where you're being interrogated by a Bly gangster called Mickey who wants to know what happened during a botch Heist you then have a flashback to the heist

itself where you can interact with objects and have a shootout with dual move controllers so we had a fair amount of time since the last game at this point so there's many new features the main ones are resolution increase um for the earlier demos we were targeting 1080p but we moved up to 1.2 to 1.3 times that that doesn't sound like a lot but it's actually 1.7 times the number of pixels being displayed and this was a huge challenge so a large portion of our time went into delivering engine optimizations which I won't go into detail on but big ticket items are things like AC and C compute like tiling and deferred Shadows we also added temporal antialising I've previously seen advice about not using temporal AA on VR and it is a trade-off and it does tend to have a blurring effect on the image but we found that to create a sense of presence it's more important to remove flickering pixels and is that make it obvious that you're looking at a game than it is to have a Sharper Image it also takes a a fair chunk out of our frame time compared with something like FXAA and you have to be mindful of the artifacts which are more visible than VR but we think overall the results are worth it

we showed reprojection for the first time as well at this year's G Toc we showed a new version of PlayStation VR within 120 HZ refresh rate reprojection allows us to continue to render at 60 HZ but rotates the rendered image based on the current headset orientation at 120 as well as smoothing out the motion this process happens at the last possible instant before being sent to display reducing latency and this was a really big leap in presence removing the lag makes you think your head is moving within the environment rather than controlling a camera and this was also our first to the character shaders and this is one area where it shows how unpredictable and Vary the development experience was we actually developed lighting models for skin hair and eyes very early on but there remained unus for a year this was also our first attempt interacting with a character in VR we use more Fe as our animation system which let Maks us make Mickey reactive to your position and act differently depending on what you do rather than being a linear narrative the London Heist went down extremely well and we showed another segment of it E3 called the getaway so that brings us up to dat so

onto exciting new stuff so the getaway shown to the public um E3 last June so what have we been working on since well we have been working on some other experiences but I can't talk about them just yet um after the get way we had a pretty good core feature set and performance so we started to work on some more Cutting Edge techniques so I'm going to talk to you about our new ambient lighting some optimizations which allow us to hit High resolutions and character shading improvements we wanted to make some improvements to our ambient lighting system the specular is okay and fairly industry standard using parallx corrected light Maps but Q sorry but our defuse ambient was captured the same frequency so we had a bounding Box covering very large areas and this gave a really flat look to the environment and didn't capture much variation in the bounce lighting so early on in development we tried using light Maps but we abandoned those for a few reasons we disliked the slow turnaround time of the bakes and also having to lay out light map UVS in addition to that the first experience we made was a deep and our light map solution didn't know

anything about the scattering and absorption that are key to the look of that game light map only also work for static geometry so we still need another solution to for dynamic objects our first thought was to Simply generate more probes by flooding a scene based on the voal representation of the level but this would still take a reasonable amount of time to update Bruno one of our Graphics programers had another idea and developed a solution called light fields and this is a voxal based Global illumination Solution that's fast enough to run in real time for VR as well as giving us a flexibility to have dynamically updating ambient light we also have the option of caching off the results the benefit of this is instantaneous feedback or lighting the solution also supports lighting from emissive meshes which gives us some cheap area lighting where we need it and so here's a Corell box being lit entirely by missive meshes and this is the uh London High scene um with our old lighting setup which is largely realtime lights for the war lights and a couple of ambient probes for DS and specular and this is light Fields version

you can immediately see a lot more variation and depth into the ambient lighting and more correct wild SC wies scale occlusion so light Fields is a good fit for VR as well as a speed it's a wel spased solution so there's no difference between the eyes it's stable so there's no flickering or shimmering and it's resolution independent in that most of the work is done off screen so on the left here there's a shot with light fields and on the right is a shot without you probably can't really notice much of an improvement and that's deliberate as this was a test to show how we can reduce our light complexity the original version used a lot of manually placed fuel lights to create interest in the ambient on the right you can see the amount of complex lighting we have in the scene to simulate it with light fields we can massively simplify this down to a few key lights while still maintaining the look you do lose some artistic control but interestingly the original version done by I was remarkably similar to the light Fields version and you do still have the option of placing field lights into the scene as well and we can set these to only affect the light field making them essentially free the big wi with light Fields has been

the speed of iteration and also giving us much more interesting lighting results without having to manually Place field lights we currently using a static version of light fields in London Heist and we need to work on a large scale version for vusion into the deep the main disadvantage compared to light Maps is it's much lower resolution and this means you can only really use it for ambient or fill rather than baking spotlights onto the environment for instance so next I'm going to talk about how we manage the resolution requirements for VR so left image shows what we rendered before Distortion for the lenses the red area isn't needed so we mask it out this saves us about 20% frame time for no visual loss I think this is pretty much covered in all VR sdks at this point the lens Distortion applied for viewing in a headset means that we have to render a higher resolution in the center of the image ideally this should be 1.4 * 1080p to get 1 to1 pixel matching and I'm going to show you a little video to explain so here you can see the final image we need for the lens and headset

so we need to render a larger image and then essentially shrink crap it onto the final image and if you notice the center remains at the same size as you can see there's a lot of resolution used on outside of the image not only is it over sample compared to the middle but generally speaking you don't really care about it as much other solutions that have been proposed in the past include rendering a high resolution insert in the center or Nvidia solution which is to have non render targets stitched together but we wanted to have a solution that didn't increase the number of targets because doing so has a number of disadvantages like extra vertex processing having to render overlaps to have a smooth transition between different resolutions and the extra CPU burden happen to render the objects multiple times so we came up with a technique which we call a resolution gradient so here's a shot of into the deep and we mask out a dther dither pattern in increasing amounts so 25% 50% and then 75% of the pixels are discarded

at the edges we currently mask out 2x2 pixel quads as masking out single pixels isn't GPU friendly but I'll come on to this later and we change these dther patterns every frame so we then fill these holes by taking neighboring values through a dilation filter and this does give us some nasty stair stepping artifacts as you can see on the vertical lines here but our temporal antialising can clean most of this up as we're constantly feeding in new information due to the varying dther pattern so this technique isn't perfect um you do get some quality loss but you're generally pretty forgiving if it isn't in a center of field of view and we save about 25% GPU time so it's a pretty good trade-off overall we're working on a technique using Hardware anti listing to only mask out single pixels rather than 2x two quads and we're hoping that this should improve the quality to the point where you won't notice it okay so the batch of new techniques I want to talk about for character

rendering characters are really important for lzed they're very close to you and they have your full attention it's important we get the shading right so you can read the facial expressions clearly so the first thing we added was vertex renormalization this is a process we do in a geometry Shader and it calculates new face normals based on the the deformed mes we first identified this as an issue on iPad on PS3 and we had an spuse process there to recalculate the normals Dynamics may always processes messag in this way so animators need this to work at the same in game if they want their facial expressions to match if you rotate joints the normals work pretty well they just rotate with the joint based on the waiting but facial animation largely uses translations rather than rotations and that doesn't update the normal at all the left section of this image shows what happens if you translate a joint on a skin mesh using a standard game Pipeline and the normals all continue pointing upwards so lighting's flat the right hand section is what happens when you generate new face

normals everything works like you'd expect so here's the standard game scanning Pipeline and this is the same measure with recalculated normals now it's difficult to say that one looks more correct than the other from a still but the important thing is that normals are constantly changing on the recalculated version making the face whe much more alive so the next thing we added was support on the skin shade for bent normals bent normals are a normal map that store the average uncluded Direction a point on the mesh when you appli them into your ambient lighting lookups you can reduce the amount of light leaking coming through areas that should be occluded restoring B normals and wrinkle Maps we ended up having quite a lot of textures to reduce the memory and bandwidth requirements we stalled wrinkle maps and bent normals as offsets on the high resolution Bas normal map we only need these these offset maps to reflect coar lighting changes on the face with high frequency detail like paws and smaller wrinkles raining on a bass map we packed these five offsets into three textures using bc7 so this is standard normal

mapping and this is with event normals and you can hopefully see the reduced light leaking particularly on the bottom of those so lastly I want to talk about um our ambient occlusion on characters we had a pretty good refractive ey Shader but the most common complaint about it is it didn't seem bedded in enough we had similar complaints about teeth being too bright now this really comes down to having a good AO solution and in this case screen space occlusion um didn't really give us the resolution or control we need previous solutions to this problem have included skinning geometry with an AO gradient onto the eyelids and lips but this is pretty fiddly um as it's difficult to manage the interpenetration on the me the meshes with all the poses so I had the idea to generate a distance field representing the isel's and mouth looking at the skeleton we already had joints fairly well place in these areas so I just create a ring of line segments using these positions and calculate the distance to the nearest one and then we can then use this to generate AO and cavity values so here here is our ey Shader and here's the eyelid

joints this shape represents a distance field from the line segments join between those joints and we can get to this in the Shader as well but better still as the eye is roughly spherical we can project these joint positions onto the eyeball surface reasonably easily to get a better result and this is the field applied to the eye exaggerated a bit for carity there are artists control values to adjust the amount and distance of AO and cavity values independently so as you can see it's a pretty large difference and this is a Shader code so thanks to inocular for the distance function it is a fair amount of math but the pixel coverage is uh generally low this is an example of one of our bespoke shading nodes I mentioned earlier I think that this is much more readable in code than it would be in an OG graph so here's the face without distance field AO and with so so the facial expression here

reads quite differently with the AO applied we also apply it to the teeth in a similar fashion but without the spherical projection so my last section today is on optimization so we obviously have to be really conscious about performance 60 frames a second in a 2d game is difficult 60 frames a second with no frame drops ever greater than 108p resolutions is even more difficult on top of that after tracking and reprojection are taken off we have about 50 milliseconds Budget on GPU and it's really important to keep a constant 60 frames per second as immersions greatly improved and low frame rates can cause nausea and it's worth the effort too VR May mean that you have to make assets that don't look as refined as in a 2d game but the VR experience is transformative but still as game developers and artists we always want to push things as far as we can there's nothing really special about optimization for VR other than that things need to have more detail if you can get close to them or pick them up but performance needs to drop to appropriate LS quickly as a result we spent a lot of time on our LOD tools

we use simpon as our primary s for generating them we found that adding the functionality to generate mesh lods which are basically a poly reduced model using the same materials was very easy to get in proxy lods are more difficult though proxy lods can bake a mesh or number of meshes with many different materials into one entirely new mesh and textures through of oxidization process the mesh can in many cases be more efficient particularly for low lot as it doesn't have to respect the existing UV layout because it bakes a new normal map mesh often lights more correctly also the main disadvantage is that you generate new textures for each LOD so your texture memory requirements go up but we have some strategies for dealing with this as I'll go into shortly the problem with implementing proxy Lots is that simplygon needs to understand how your shaders work we have a wide variety of shaders that we want to bake down and although simplygon has functionality for this we didn't want to build a system to translate our shaders over but Andrew Maxim from ndog had a cool suggestion if you can already bend your materials in mea and we can just flatten them into UV space and do some renders so that's what

we did so here we have a drone from the deep it's a relatively simple shading setup of just two materials and S textures obviously with layer shaders you can have a lot more inputs the input mesh here is about 14,000 poies so the first thing we do is to bake all the material outputs from the Shader and these correspond with the values that we pass from our material editor into our lighting models so this is a Shader that's used on dra as you can see it's pretty simple it doesn't really do much other than read a few textures and pass them values long so we've run the model through M automatic UV mapping to generate a new set of unique UVS and use the vertex Shader to flatten model into UV space so here's the process of world space normals now we go through this process for six Maps which are base color well space normals metallic gloss and occlusion packed into one texture we render that remissive but that's an HDR value so we packed this into rgbd format over two textures and opacity we also pad the edges to remove any seams and this all gets sens into simply G and we

end up with our led1 model which is about 3,800 triangles so the textures you can see here don't make a lot of visual sense as they've been swizzled we convert the base color from RGB into YG Luma and chroma format and store these textures at different resolutions so the largest texture here contains the normal X and Y the gloss and the base color luminant and these are the most important components the next texture contains the base color's two chroma channels metallic and occlusion and this can not only be stored at a low resolution to save memory but we also bias the texture sampling to always sample one midlo and usual to reduce texture bandwidth the last texture contains the missive stored in RGB pack format for this model we need aive but obviously not all models do so emissive and opacity textures only generated and used on models that need them and all the textures are compressed as bc7 so we already St a second texture as a low resolution but in many cases even that can be improved upon so here we've replaced the second L texture with just vertex

colors there's a notable difference on this model particularly on the bottom of the turbines but many models have very little variance in their chrom more metallic parameters and you can only really spot a difference in the occlusion and the result of this is that many of our proxy LS can represent the full physically based material using only one texture read and I'm hoping on our next project that we can get some of these packing techniques onto our main models as well as a lods to reduce our texture bandwidth so I think that about all I have time for today so in conclusion we had a lot to do at the same time the arrival of PS4 developing a new engine and also writing a rule book for VR as we went along was really hard our approach to making a flexible pipeline made us a lot easier if in the future we develop a title which has more known content it's likely we'll shift our Focus away from the pure flexibility we have now to solutions that are a bit more sposed for the game however we like to keep many of the same philosophies in the future particularly around keeping the iteration times down to as short as possible

so thank you for coming everyone um I think possibly got time for couple of questions [Applause] [Music] [Applause] maybe uh not very well oh hi um thanks for your talk I was wondering you mentioned that you were moving to forward renderers I was wondering if you're final with that decision and um well a lot of people in VR know why but if you can elaborate why it's important for you to do for rendering so the main reason we decided to go for a forward renderer to begin with was was really just to keep the flexibility open so we we wanted to be able to put in lots of different light models and to be able to stylize rendering um as it is I think it's turned out to be a smart decision in that the bandwidth requirements for

forward are much lower than deferred as well so I think it scales quite well to having high resolutions for VR um having said that we haven't actually tried a deferred renderer against it so I don't know whether it's the best solution we probably need to do some more work to evaluate against a deferred solution in the future the thing is I mean forward and deferred aren't as different as they used to be anyway um now that we used hard approaches on both the only difference really is like do you store your material inputs in a texture or do you just pass them straight into material so they're not so different we could probably switch over to a deferred renderer if we need to I was just kind of curious why do you guys still do a tiled approach when you're doing forward rendering because with the Deferred approach you obviously need to tile it just to because of the size of your G buffer but yeah all if you're doing forward rendering I assume you're just outputting the actual final color values so it's uh it's basically so we can support many lights so we do a

a light Tiding pass to work out where lights are in screen space and then build up a an index list of all the lights in that particular tile so that in the Shader Loop you can just basically Loop for all the lights that are in that tile so you can support as essentially an infinite number of lights so as opposed to the previous for approaches it's basically just for the number of Lights is is the short answer thank you okay thank you very much um if anybody else uh