The Latest Graphics Technology in Remedy's Northlight Engine

[Music] so many of you are here biggest crowd I've ever drawn first things first so it says start to all to lead graphics programmers so that's not me I'm Thomas poha from remedy totally still in the airplane this was him last night with rest of the remedy crew they were stuck in Frankfurt so 2:00 a.m. San Francisco time we were going through this presentation so I bear with me I haven't had a lot of time time to rehearse but excited to represent the team we got some really cool stuff to show and and and discuss and thought oh should be here if you know security and immigration and British Airways permits him to arrive at the end of the day so he can then answer questions a bit better than I can all right so let's go so this is the roundabout topic but as we all know not

really true so what we're going to talk about here today is experiments with Direct X ray tracing in remedies Northlight engine a lot more exciting right okay yeah there you go so a little bit quickly about remedy so we're based back in Finland in Espoo started in 1995 you might notice for max pain Alan Wake and quantum break and we used to be a single project studio but now we're working on multiple projects p7 across far versus our first first-person shooter and of course since for Scandinavian we do do our own tech kind of because we're you know start from the demo scene and all that so we have a sizable team working on the engine and tools ok enough advertising about the awesome company I work for so the agenda what are we going to talk about here today well I'm gonna talk you're gonna listen hopefully so it'll give you a quick introduction to Direct X ray tracing then talk about area

shadows ambient occlusion reflections and indirect diffuse ok so in order to explore we created couple of months this moody scene that's not from any of our games it's a separate scene and I'll start off with this demo and discuss the new technical aspects after it so I'd recommend to keep an eye on how the pieces fit together and especially how the different aspects of the lighting work so we have audio I hope [Music] [Music]

[Music] [Applause] [Music]

all right pretty cool huh [Applause] okay so video just saw was captured from the engine with all the ray tracing effects are enabled and with increased sample counts so kind of high quality settings if you if you will I know that's a bit vague but so let's start with a very high-level introduction on directs a DirectX ray tracing I will not go through the API in this presentation but I'll try to give you some background on what you need to know before you can start shooting rays so ray tracing API provides bottom and top level acceleration structures the idea is that bottom level structure is constructed to hold geometry while the top level

contains bottom level structures simple way to think about this is that each mesh should be a single bottom level structure and top level tree is your scene that contains bunch of bottom level structure instances with some transformation first thing you're gonna need is the bottom level structure for the static parts of your scene in this picture each red square represents bounce of a single bottom level tree building bottom level tree is very simple in API you'll have a function that takes normal DirectX GPU geometry and index buffers and returns bottom level tree built around the geometry so in here I have highlighted four instances of a small chair that you saw in the demo these for chairs share the same geometry but have separate transformations the mid-size trees are small sofas and the biggest ones are

large round sofas in order to build a scene for ray-tracing you'll need to insert these bottom level structures into a top-level structure this is again very simple thing to do the API provides you a function that takes in a number of bottom level three instances with transformations so building a top level tree is fast so you move static meshes every frame and then just build top level from scratch each frame supporting deform geometry is a little bit more trickier as the bottom level builder eats only static buffers for us the easiest way to support deformation was to create a simple computer that takes in geometry and skinning matrices and writes out deform geometry this is still very fast as you can keep everything on GPU and just be the output from the computer to the bottom level of builder and that's it now you can start shooting

race I'll start off by showing a couple of examples you can try out with data only ambient occlusion so I mean occlusion is purely visibility based algorithm that can be easily tested out with ray tracing image you see here is taken by shooting four samples per pixel using cosine distribution maximum length of the race is set to four meters and hits to the geometry are considered black and others white this is added as direct replacement to what our screen space ambient occlusion outputs so it is easy to compare results of different algorithms so split screen we got screen space ambient occlusion on the left side and ray traced ambient occlusion on the right while the screen space technique does a pretty good job on rounding things it's clearly lacking information

screen space algorithms can't know what's outside the screen or behind the surfaces that are directly visible to the camera shooting Ray's is of course more expensive that looking up screen space depth in our quick prototype shooting a single ray with maximum the stens of 4 meters cost roughly 5 milliseconds the cost scales pretty much linearly so shooting 16 rays for Full HD picture takes roughly 80 milliseconds the picture you see here is shot with 16 rays finally here you can see a picture with comparison of different Ray accounts you should note that these are all taken we just our basic temporal anti-aliasing I'm sure it would be possible to do a better job with filtering noise and you can't make it very well in the image there's a lot of noise on the left and far less on the right side of the image alright so

moving on shadows we implemented a path to replace sunlight cascaded shadow maps by shooting Ray's if you happen to feel the shadows of the directional light into full screen buffer before actual lighting it's very easy to just replace cascade map lookup shader with ray tracing kernel that writes in to the same texture here you can see comparison using shadow maps on the left side and ray tracing on the right side in order to be fair I need try to tune up a resolution on the shadow map so this is just something we get out of the engine with the default settings anyway the amount of detail you can gain with accurate shadows is pretty remarkable in addition to accuracy it's easy to do real area shadows with retracing clip here is using 8 samples sampling pattern

is basic blue noise that's changing between frames for each sample on the pixel we offset the same blue noise with Sobel sequence and rap the values on disc so here you can see a close-up of the same effect as you can expect with ray tracing the contact points are accurate and you get a nice mix of sharp and soft shadows well this is nothing new or radical on algorithm level it's very cool to be able to go and look at your existing game scenes and can't to see how much detail is lost without accurate shadows and here's one more example so looking at the amount of detail added to a simple scene like this makes you wonder if there's any sense to keep increasing the output resolution before we can sample lower resolution with good

quality so in the demo the performance from shooting shadow rays is a bit better than with ambient occlusion even through the Rays are longer resolving shadow visibility is more coherent lower than AO getting something like this up and running with existing dx12 engine should take you something like a couple of days I can say that it's definitely worth the effort even if you are just going to use it as reference so I've been talking about algorithms that can be implemented by knowing only visibility between two points in the scene going beyond visibility requires access to additional data in this selection I'll quickly introduce how we expose geometry materials and lighting to be used in ray tracing kernels on a

geometric hit the API provides you at the intersection point in scene and triangle base other geometric properties than position you will need to read in manually and interpolate based on barycentric coordinate this means that you will need to be able to access all your geometry from shader new DirectX comes with a new binding API where you can set different resource bindings for each instance of a bottom level piece in the scene so let's say I have two instances of a chair each chair has five pieces for different materials like in the diagram on this slide I can now define separate resource bindings for all the pieces so in total I have ten resource binding records each of the record can then have a custom binding for vertex and index buffer for example it would be possible to bind material data using the same API but just before

starting dxr we had completed a new material binding system that works also with basic DX 12 by packing all the material parameters into a single buffer and reserving dedicated space on front of the descriptor heap we could already do draw calls without additional bindings this also means that we can access everything related to materials in a single shader called each index on this slide refers to a single material well I think happen to be pretty easy for us since quantum break we have already had all the lighting data accessible to a single shader call this means that we pack all the shadow and projection maps in a few atlases we also keep the light parameters packed in buffers that's it now we have geometry materials and Lodz exposed in a way that we can access everything in a single ray trace kernel ok so let's get into trying out reflections next tamai is the pace the

right now going too fast ok rapt attention ok so following section is dedicated to implementing reflections with ray tracing I'll start up a quick comparison to our screen space technique and continue to visualizing how the previously introduced data bindings are used as with the previous techniques shown we implemented ray traced reflections as a direct replacement to screen space technique this way it's easy to compare the results again so here you can see the split-screen comparison with screen space reflections on the left side and ray tracing on the right side this is screen space only reflection regeneration works the exact same way in both the screen space and raytrace technique for each sample on the screen you read properties of the surface from

the geometry buffer then based on smoothness you pick a random direction from ggx distribution and transform it according to geometry normal on-screen space we treat reflection ray as a 2d line from a point on screen to a point on edge of the screen in order to figure out if the Ray hit geometry we take number of depth samples along the line it's possible to miss features along the way due to holes in sampling and due to having access to only directly visible surfaces also the hit position can be outside of the screen I don't have any backup solution in use on this picture the most obvious fallback would be the sample sum cube map when the Ray heat is out of the screen okay so getting back to ray tracing so ray tracing doesn't miss any geometry that is out of the screen behind visible surfaces or due to lack of samples on a geometric heat we need to figure out the radiance transmitted towards reflection ray

origin for this we need to know the material and the lighting at the heat location the XR provides the heat distance position and barycentric position within a triangle as in shader intrinsic with this information we can interpolate UV is from the geometric buffers exposed earlier so the UV that reflection heat location will end up looking a mess like this with UV and other data we can sample the rest of the material properties here you can see the diffuse albedo on the reflected surface next it's time to evaluate the lighting and combined the results this is lining at the reflected surface we can now combine this and material with Fresno term and here's what the result looks

like this is reflection only visualization in order to get the final image we combine this with diffuse and specular component of direct lighting the final image as you can expect with the techniques like this the end result will have some amount of noise we didn't have a lot of time to experiment with noise filtering but I have a couple of image loops to visualize the problems the following loops are taken from a bit harder lighting case and are using our basic temporal anti-aliasing so this is the raw image you get when shooting a single reflection ray per pixel without any filtering the lining here is from sunlight it's pretty complicated hard case as the lighting has high contrast and a lot of detail as we all know the variance in lining decreases when shooting more rays per pixel while shooting more rays for all the pixels is

expensive with ray tracing it's easy to pick a few locations to shoot more it's almost free to shoot couple more Ray's 4 pixels that are over 2 times more bright than the white point in the image in this loop you can see that we can remove most of the fireflies with just this will be interesting to see we're using something more advanced like radiant don't mind sampling would get us yeah you can well you can kind of make it out so in the demo we are also clamping the reflections that are over 10 times brighter than the white point on the image is of course a hack and it's not recommended you do this unless you're shipping great advice well it would be perfectly possible to use ray-tracing to resolve shadows that reflected surfaces we use shadow maps for most of the lights in the demo the only exception is sunlight

since we use cascaded shadow maps with the Sun there is valid depth for only the pixels on the screen having to shoot a single shadow ray for the Sun on every reflection ray almost doubles the cost of the reflections okay that's all about reflections so we're going to look at the diffuse lighting next so in this image I have diffused global illumination interpolated from the volume texture the approach you see here was implemented for quantum break and is in use by default in our Northlight engine voxel resolution of the GI data here is 25 centimeters we combined the results with screen space ambient occlusion as I showed you earlier we have a path to replace screen space AO with a rarest one so let's have a look how that looks so this is the same data combined with Ray traced ambient

occlusion results are better but this still looks like ambient occlusion applied on top of volume later that doesn't have enough resolution for the case scene here instead of just shooting a lot of rays around to resolve indirect diffuse lighting I'll be talking a bit about improving sampling of volumetric global illumination and resolving near-field global illumination so let's quickly break this down with a few diagrams a pre computed global illumination is stored in sparse volume texture the grid on the screen represents the data each crossing contains indirect lighting data that has been calculated with a path with path tracer the blue shape on top of the global illumination volume is static geometric that we use for calculating the data you can see that the dense parts of the GI volume are on the areas that overlap with geometry

in addition to static geometry we have dynamic geometry the smaller green piece in the middle of the picture is dynamic geometry dynami geometry is not used in pre calculations so part of it resides on low resolution area of the volume there are a couple of problems in using the lighting directly sampled from the volume first we'll miss the dynamic geometric completely we've been relying on the screen space ambient occlusion to ground diamond dynamic geometric to the rest of the scene second problem is linear filtering of the volume data on the surfaces that are not aligned to volume we get stair casing that can be quite visible due to low volume resolution finally low resolution will cause the lighting to leak through the team geometry it's possible to try weighting the sampling so that it will leak a bit less but ultimately you are

just moving the artifacts instead of sampling the volume directly on the surface we can shoot rays and sample global illumination on the locations that didn't hit the geometry positions that are not on the surface are less likely to contain leaked light in our data this will reduce leaking and results in smoother interpolation simplest thing to do when hitting geometry is to consider the sample black this is pretty close to calculating ambient occlusion without benefit that it helps with light leaking and results in better interplay interpolation of the GI data so let's take a look at the comparison between this and direct sampling of volume data with ambient occlusion though this is the gr volume sample directly and combined with ray-traced

ambient occlusion if you look at the table or a magazine on the bottom left corner of the image you can clearly see that there is something wrong going on by modifying the way we use the volumetric GI data we can fix many of the issues that can be seen when using direct sampling I'll toggle a couple of times back and forth and highlight the locations to look at so direct sampling and ray-traced AO and ray-traced gather so this is our difficult location or example for volume data look at the dome on top of the scene now that is made of team geometry so raytrace gather again so on the previous example we considered

rays that heat any geometry to be black instead of darkening the image we can do better than evaluate lighting and material like we did with reflections so the same thing I showed you earlier so here I've added a single bounce near-field global illumination the accuracy on the first bounce is something that you couldn't easily store in limited size volume also the indirect lighting read from the volume is better filtered that you could do with a few rays this is the direct lighting that was used for the near-field this is direct lighting combined with indirect diffuse and this is fine aligning with

specular and this is the final image alright so summary on this stuff so easy access to state-of-the-art GP or a tracing via the X our performance is getting there yes that's very easy to prototype algorithms that don't fit the rasterization and possible to combine with existing low frequency structures and of course shout out to the team back home in Espoo or people pond on the plane right now that worked really hard on this demo in the last last couple of months of course thanks to the folks at Nvidia as well this was a fun thing to do then the absolute necessary thing that finland is an awesome country and we are hiring at remedy

it's not called it's all a lie there's never a winter and we can answer some some questions I can but that will be here by the end end of the day but write down that email and you probably should have said in the beginning so we have the entire slide deck with notes and including the video on our website as I speak so you can you can go check that out well we try to answer some questions or no yes okay well thank you very much and questions please hi it looks really great but I think the obvious question is that all the images are very noisy so

did you think about appliance I'm gene noise feeder yes we didn't have too much time to explore to be nothing but that's a very good question and I think that for many of these effects a working and fast enough denouncing solution is needed for a to ship something real but there might be some use cases where you can go pretty far with only one sample per pixel as well you see microphone please

it's this hybrid fourth NGO versus for volumetric rate race I see some God raised over there I'm assuming those are a traced and probably the dome is geo direct rendered is it a combination of everything or is full seen either volumetric full scene is direct rendered how does that work no sorry for if I understood completely but the volumetric effects the cart race you saw here they are not not ray-traced they are done with the more traditional techniques same for the lighting effects that I see on the dome what do you mean by line the lines and the actual yeah the the bright lines drawn on the dome yeah they are atmospheric effects yeah they are not ray traced hi great talk thanks for of this I have a question

based on the video you should first and we can see there is a lot of flickering and fireflies and I was wondering if you tried the technique that you you convert normal buffer to curvature curvature buffer and then convert modifying the roughness based on this curvature to or to having less fireflies like this one because it it happens on really smooth materials and when you have a really high curvature on smooth material so you have this kind of artifacts that is basically a specular arising and when you want to avoid this kind of stuff you can use the normal to curvature and curvature to roughness modification sorry for my question seem to be yeah I'm sorry if I completely understand I

understand the question yeah basically what did you try for avoiding speak eulogizing yeah basically that there is a reasonable amount of flickering due to fireflies and we didn't do much more the dose than what was described in the presentation adding shooting additional race on a / bright pixels and we did also experiment with some denoising solutions but they were not included in this video but it's obvious that that when you try to do the reflections with with a very limited ray raise her beak so you get noise and fireflies you have to do something about them okay thanks

okay and we have this at the Nvidia both so you can go go check it out and really for like if there's additional questions like please don't hesitate to email Totten we'll make sure he'll he'll get back to you thanks very much for your attention thank you very much [Applause]