Making Connections: Real-Time Path-Traced Light Transport in Game Engines

[Music] all right uh well thank you for coming to our talk it's called Making Connections real-time path traced light transport and game engines and it's been a great day of uh cool talk so hopefully you have enough attention span left in the tank for one more uh I'm Adam Mars um I'll be presenting the first part of this talk and then handing it off to Evan Hart uh also up here is uh Jen cow who isn't here today uh but he's been really important uh in contributing to this work uh that we'll be showing you today here's an overview of uh what we'll be talking about today I'll be going through some U preliminaries and foundational concepts related to realtime path tracing before heading off to Evan all right so first things first why are we here better lighting of of course

uh to create richer more realistic uh visuals we want our scenes to have more lights ultimately and specifically more shadowed lights and Beyond just having shadowed lights we want Shadows that have accurate perum importantly we want the simulation of light transport to be completely Dynamic so our worlds feel alive and as interactive as possible and to this end our ultimate goal is to more accurately approximate the rendering equation and to do that in real time so we also need to do this in a performant and scalable way and we really need to be mindful of this even when we're on high-end PC parts so most dynamic shadowing methods oops most dynamic shadowing methods

today uh don't scale well when we have lots of lights and this is not a shadow map thing I mean memory uh is always an issue and performance is always an issue when you when you have tons of fls uh but fortunately today's gpus offer a lot more especially highend gpus offer a lot more uh exciting possibilities that we'll be talking about uh through the course of this talk so we really encourage you to explore uh these new opportunities uh that we talk about okay so you may be thinking spent the whole day talking about interesting stuff work grafts uh and I already have all of my time budgeted for the next two years to work on things like that and it's a lot easier said than done to just go and Implement all these new exciting things that you're going to talk about and I understand games have a whole lot going on on uh they demand a high level

of performance they uh demand an you delivering on an artistic Vision uh and the artists may not want a photorealistic perfectly photorealistic game they may want to bend or break the rules of physical reality and your renderer rendering system needs to be able to deliver on that and game engines that they Wrangle that comp complexity with systems and that means we typically have a lot of systems especially in the renderer and this is because we need to support multiple platforms we've had uh other talks talk about uh consoles in addition to to PC and this can be a huge range of Hardware that you ultimately have to address with the renderer and most of the time this creates uh sort of Choose Your Own Adventure uh rendering pipeline um depending on the platforms that you

support so seriously though it it becomes this tight RPP walk that you have to do uh between overfitting uh your renderer to a specific problem or to a specific platform so we've been working with Unreal Engine 5 and we chose it because it's a highquality ubiquitous great engine that has its source uh fully available able and a whole bunch of Studios including people sitting here today I'm sure have used unreal 4 and5 at uh production scale so we expect to encounter a lot of really great representative challenging uh Graphics workloads uh using this engine uh but in addition to this uh a great aspect of unreal is that we're able to Branch it and then make what we do available to you uh so you can pull it use it

uh use it as a reference and uh hopefully uh learn something from it and tell us how you made it better um everything you see today importantly uh is in or using our Branch the Envy RTX branch of Unreal Engine 5.3 and at the end of the talk there'll be some slides with links where you can go uh to get access to that all right additionally unreal 5 has great lighting Tech right out of the box um and and this is great in general it has both awesome benefits and some challenges uh the benefit is that you have fallbacks in place already really nice fallbacks in place for Content that may not be compatible with any new rendering Tech that you put in uh or workflows that might not be compatible but challenge is that good competition is tough to beat

and uh we really like this Challenge and uh we hope that we can show you some cool stuff today speaking of which our art team uh has put together a video a demo um of an amusement park scene uh and I'd like to uh show it to you right now so in this scene uh we're showcasing where we are today with path trac light transport and u5 and the scene has well over a thousand Dynamic light sources that all cast accurate Shadows with penumbra uh we have additional Dynamic lights from a Niagra particle system for the fireworks these also cast Shadows uh this includes denoising with uh dlss Ray reconstruction and it's running at 1080p using the dlaa preset and it runs at 60 HZ or above on an RTX

4090 so I'll let the rest of the video play out [Music] [Music] all right very cool uh thanks to our team for putting that together they did a great job uh so some of what you saw there might not be obvious as to why it's super cool and uh I'm going to dive into some of the foundations that underpin the tech behind what you saw

so at its core path tracing is about simulating the transport of light along paths where we Define a path as multiple connected Ray segments and for many of you in the room I'm sure this is not anything new this is not a revelation by any means but stick with me any path tracing renderer real time or not is going to focus more accurately on more accurately approximating our good old friend the rendering equation and I'm sure that many of you have seen this equation in various forms uh this is one of them uh it's simpler it doesn't include the uh emissive term but for today it's a pretty good example uh since we can't solve this equ equation directly uh our goal is to approximate it also not news to you to start to understand how we will or could approximate it

let's treat its integrand like any other function then employ Monte Cara integration to create an estimator and a simple money Carlo estimator evaluates our function at uniformly distributed points across the integration domain and then averages those results and the math looks maybe complicated right but ultimately this is essentially the path Tracer that's introduced Peter Shirley's Ray tracing and one weekend mini book and it's probably the first path Tracer that you wrote uh and it's a great starting point so before we move on um you're going to hear the words sample a lot in this presentation and other pres presentations today use the word sample to when I was first learning about sampling and rester which is something we'll talk about during this talk I got confused because the word sample was

used so many times in papers and blogs and in videos and it just confused me because it wasn't well defined sometimes it was a noun sometimes it was a verb sometimes it was almost every word in a sentence and so then the word didn't mean anything anymore to me um so if you've ever been confused um by this overloaded use of the word sample this is it x sub I that is a sample that's what we mean by sample in this this context for this talk now you might think well what is that uh that's just function I I don't know what that is uh we're graphics people so in in graphics speak this sample translates to the path of light transport and specifically the path of outgoing light in a specific Direction reflecting off a surface given light incoming from another direction or light

source that's our sample so keep that in the back of your head as uh we go through this talk and as I say sample many more times okay so we're going to estimate the rendering equation by sampling it cool but how we generate those samples is really really important and the keys to an accurate estimator is ensuring our sampling process is both unbiased meaning it Returns the correct value on average and consistent meaning the estimator converges to the correct answer or the correct value as the number of samples goes to Infinity keep in mind these things aren't the same being unbiased doesn't mean that you're automatically consistent and vice versa this is an important aspect um of rendering in general uh path tracing in general uh but less important in real time which

I'll mention a little bit more later so to summarize path tracing versus most real-time rendering today the main differences are with path tracing we simulate the transport of light with paths we approximate the rendering equation with a Monte Carlo estimator and we produce samples or try to in an unbiased and consistent way okay so we're here for real-time rendering though uh and we want to take those aspects now that we understand them well from a path Tracer and bring them into a real-time renderer so let's connect some rays build some paths get sampling there's just one big problem uh why weren't we doing this all along right it's too expensive to do this um evaluating all that light across all those connected segments and paths is

just not practical in real time today and this is where Reservoir based spao temporal importance resampling rester comes in we'll need to be very very careful to evaluate only the samples uh or in other words paths that matter the most and lean heavily into things like reuse and clever math and reer provides exactly that the algorith mic Foundation that en enables us to select the essential paths and complete light transport at a reasonable cost in real time so at the core of rester is resampled important sampling or RIS and RIS shows us that somewhat surprisingly a single sample really can provide a good approximation and a good sample can be refined from mediocre or even truly bad

samples potentially even more important to rester is weighted Reservoir sampling or wrs and remember the re and rester stands for reservoir-based and that's where wrs comes in wrs stores selected samples uh and other Associated information in reservoirs uh this is crucial to the process of sample refinement and reuse see I wasn't kidding I'm going to say sample a whole lot so with the combination of R and wrs will make an estimate compute a value and then repeat that process to refine and improve our estimate and a convenient way to configure our Reservoir is in screen space since then

we can lean into reuse in ways that are natural for us as Graphics programmers so things like temporal and spatial sharing where we merge reservoirs together either from previous point in time or from spatially other pixels on the screen and at at first glance you may say like yeah yeah like I I get it we've been doing this sort of reuse uh but it still seems too to be true can we get all the samples that we need for a conver path traced image by doing these things and I think you will and and let me show you so here's a shot of the amusement park scene that you just saw in the video and post-processing TAA and much of our rester goodness is disabled this image shows what it looks like if we take just one sample per pixel in the scene here that has well

over a thousand lights I think it's actually 1300 lights total and with so many lights to pick up pick from we're having a tough time finding the right light for any particular surface meaning our estimates are pretty bad uh those of you in the front row may be able to see that there are some flexs of light kind of uh noisy Flex throughout the scene but those of you in the back it probably just looks completely dark so if we increase our initial sample count to eight per pixel instead of one the result improves a whole lot but we're still a far cry away from a good result or a usable result even so let's go back to One initial sample per pixel but add in temporal reuse and this is a substantial Improvement so if I this is eight this is one sample but with temporal we get a lot more reasonable

lighting and we're starting to see a shadow appear underneath the character's feet but we're still far away from where we really want to be so let's use one of our other tools in our toolbox which is spatial reuse uh I've gone back to the one sample per pixel plus temporal and then added in two samples through spatial reuse so that's where we were and this is with spatial temporal and now we're cooking this seems like a reasonable image that is close to what our actual scene should look like once it's lit so if I keep this set up and bump the samples even more the initial samples to eight then things are looking even better some of that noise you can see gets refined and keep in mind this is without TAA a uh or any post-processing or D

noising applied this is just raw sampling improving the signal so we started here and we ended here just by improving our sampling picking better paths pretty great transformation I think uh hopefully this has you convinced just how powerful uh reuse and things like rester can be but as good as this is is it's still a noisy image at the end of the day and we have to do something about that and the noisy result also isn't a surprise um noise is fundamental to the paradigm shift that we've introduced into our renderer uh because the very nature of mon Monte Carlo estimators means that we're going to introduce noise fortunately for us nois images aren't a new thing denoisers are going to come to the rescue uh some important things to understand

uh denoisers need noise to function properly and specifically they need high frequency noise if you give a real time d noiser a stable image uh it's going to get confused and produce generally bad meaning blurry results there are several existing off-the-shelf Den noising Solutions uh and algorithms that you can use to handle a noise you heard about one earlier today uh from snow drop uh but dlss Ray reconstruction is an excellent one and there are a variety of te techniques based on either recurrent blurring uh also called re blur or spatial variant guided filtering or sgf uh this includes algorithms like relax uh in the nrd library that extend s spgf and make a variety of improvements D noising is a really deep topic so I'm not going to have time to

discuss it in detail today but let's go back to our amusement park scene and apply some D noising to it to see how it works so here's a different shot from the amuseum park scene uh using the same Reser settings that I showed earlier where we ended up uh this time I've turned on TAA uh and den noising here is disabled so we have our nice noisy image and here's what it looks like if I turn to on TAA and the relax D noiser that's part of the nrd library so I'll go back and forth to give you a little idea of how it cleans it up and the combination of TAA and the relaxed KN noiser clean up that noise really nicely now if you're in the front row you may be looking and saying uh there's some blur there it's it's not as sharp as it was maybe there's a little bit of energy loss in place is the sign isn't as

bright if we swap over to dlss Ray reconstruction several of these areas are less blurry they're sharper and the fine texture de details pop out in a way that they didn't previously there's slightly less energy loss oops as well uh if you compare it to the noisy nerd and then DSS uh so ultimately we get a pretty nice image out of that okay so hopefully at this point uh you understand the anatomy of a real-time uh path Tracer and you believe that rester can kind of do Miracles um so before we move forward I want to talk a little bit uh about our implementation because during this project we've had a few guiding principles the first is respect the budget 60 HZ frame budget it's okay to take shortcuts if it means that we

respect that budget and it's okay to accept some bias where we really have to uh particularly related to denoising because a denoiser inherently adds bias to the image um but since we have such low sample counts array counts to work with the denoiser is an essential tool to make a shippable image next strive to be unbiased when sampling th this one's really important because don't get tunnel visioned on we have to be unbiased all the time or else our image is terrible and we're not path tracing you can have a little bit of bias and it's okay consistency is another one I talked about a little bit earlier and being consistent is good but it's somewhat less important for real time um because we have very low sample counts and one day hopefully sooner than later when we have hundreds or thousands

of samples per pixel I'll be happy to worry about being consistent because we'll have so many samples uh to worry about converging but today it's it's less of a problem and uh one of the other elements of this is that we're going to stochastically evaluate lights for all of our interactions and this provides tangible performance benefits that Evan is going to talk about later uh but also delivers more consistent results across different effects um so for instance direct lighting versus Reflections all right next at each Ray segment along a path we're going to determine light transport with the best method we have available and this is an important principle uh this means that things like Ray tracing rasterization cone tracing Ray marching cach data they're all on the table path tracing itself doesn't require that Ray segments

all be Ray traced and determining the best method is a combination of a few factors performance is of course a very important one uh that will will weigh heavily in our decisions um but best can also include a lot of other things like the complexity the code complexity of the implementation compatibility with existing rendering systems content compatibility too and best can change based on your game the game you're making the engine that you're working within the hardware you're targeting and maybe even more importantly time because one day hopefully we might just have gpus that are faster than a 4090 so a few more notes uh on our implementation uh as you would expect we're constructing paths uh from the camera to light sources which is often referred to as backwards uh we use rasterization for primary Rays uh this

is because it's fast nanites to pass occlusion calling works great and ultimately uh nanite requires rasterization to function we use a combination of Ray tracing and cache data for secondary Rays where our secondary arays include diffuse and specular GI and we use unreal's highquality hit lighting path uh for for specular uh and we configure that uh such that it has an optional for it takes advantage of the option for multiple bounces we disable uh screen traces or SSR uh that runs as a pass before uh Ray tracing uh because this often produces less than ideal results for light or Shadow Rays whichever one you prefer to call it um we use Ray tracing and evaluate material hits uh so we can support Alpha tested uh content like foliant

and one more thing to mention that we noticed while doing this project and this work uh introducing path tracing to the renderer is that this kind of paradigm shift in the renderer forced us to continue to ask ourselves where and how much we should be rewriting reusing adding or replacing different Cod paths in the engine and I think if you do this yourself or when you do this yourself hopefully uh you're going to run into this a lot too and uh Evan will touch a little bit more on our thoughts on this uh later in the talk all right that's it for me I'm going to hand it off to Evan and he's going to talk to you about leveling up sample lighting thanks a lot thanks a lot Adam excellent job setting up uh what we're talking about here let's go ahead and get into our implementation so first I want to talk about some uh building blocks that we're using within our rester implementation

first we build ourselves a light pool that's basically the set of lights that we want to sample from this is a structured buffer we stick it on the GPU then we have this target function this is just the evaluation of the brdf against a light sample so stuff that's already existing in the engine we're going to reuse code in the engine for doing this evaluation that's important for waiting our samples as we uh reprocess them and finally uh we're going to do this per pixel Reservoir grid that we use for storing where our samples are with their current weight so these are the weighted Reservoir sampling uh buffer basically now we're going to have in our implementation uh a few different stages that Adam referred to the initial sampling temporal resampling spacial resampling final shading and for these stages we're going to read and or write from the uh Reservoir Set uh and to update it and reuse samples importantly all the we're running presently as Ray generation

shaders because of the fact that we want to take occasional visibility samples and we're using the we're using uh Rays as Adam mentioned and using the relying on the any hit Shader for dealing with transparencies and whatnot but importantly we don't shoot a ray every time we're doing the sampling um we just do it when we need to basically so let's start off with the initial candidates in the initial candidates we draw samples from every class of light source so local directional Sky Etc and that's because the sampling is sort of uh necessarily tied into the some fundamental uh characteristics of the lights and then like I like I mentioned a moment ago we optionally will do some conservative uh visibility testing and we want to do some of this because it helps us converge faster but we want to minimize it because we don't want to shoot too many Rays here and importantly conservative means that we're going to air towards visible because we can't skip the Mast uh surfaces in this

initial visibility test uh for the final for any final testing we'll have to you know take that into account but we can possibly skip trees when we're shooting these initial arrays so here's some shaky cam of me showing you an artifact that can happen uh with respect to merging the reservoirs before we do uh full visibility testing so here we're merging the local and directional reservoirs prior to visibility testing and we found that we may want an option for merge for testing visibility against all of the local samples that we've taken uh you the merged local Reservoir and versus the merged directional Reservoir and this is because the light the sunlight outside of the beastro is shaded complet or shadowed completely inside of the bro so the very strong directional light sort of is pushing uh energy into the statistics for the weighted Reservoir sampling and by testing the directional light before we do the resample we're

good so you may say okay well with thousands of Lights random sampling can be challenging because we have to find relevant lights but we'd really like to find good lights and a random sample has a tough time doing that we do have we can see that eventual convergence is going to get us there with this uh temporal and spatial resampling but games can't wait around we need to be able to produce an image very quickly so important sampling is a key portion of this and the version that we're reusing or reusing today is reservoir-based grid-based importance resampling got it correct that time uh pre-selecting all the lights helps with convergence and we're going to pre-select on two uh two sort of metrics spatial which helps with the fact that lots lots of lights and games have a limited range they'll only go out to 20 MERS or so and additionally we'll use power as an importance Factor uh because the brighter light is the more likely we want to be able to sample from it we

have to be be a little wary of fireflies with our rear implementation and this is because under sampling these reservoirs can lead to fireflies in the reservoirs themselves which can show up as boiling and importantly this can also happen if our PDF is being poorly matched um importantly multiple importance sampling is something that we do use to help alleviate this problem and that's because of the fact that multiple important sampling from you know using these different methods can sort of cancel cancel out weaknesses in different sampling methods so our rear implementation looks a little bit like this uh that's the the amusement park scene and we're using clip map style uh grid with uh some temporal feedback so this is what it looks like and each of these different colored cells is a different cell in our regear grid where we are pre-selecting a bunch of lights and we're using temporal feedback to refine them uh as we as we run through the uh the scene basically so again this is one sample per pixel using the random sampling that

Adam was looking at before but if we turn on rear no spatial or temporal reuse just the spatial and temporal informations in the rear cells you can see we've actually already converged to a pretty good image there uh you can clearly see the sh the the shadow cast by the character in the scene but in addition we want to take better samples in other ways specular is tricky um with respect to re or with respect to reer and this is because shiny surfaces have a narrow brdf and we don't need to just find a good light here we need to be able to find the location on the light to produce that highlight and I'll show you I'll show you the problem now so here we've got a very Blobby specular highlight in the upper left cor portion of the sphere and the reason this is happening is because we're having trouble finding the Lo the right location on the light to produce the Highlight so if I turn off denoising you can see you know exactly what we're sticking into the denoiser now still with denoising off let me do some material work here where I'm firing a

material Ray and finding the right location on a light and using multiple important sampling to combine that into our rester estimate way we've done this is with a light bvh it was mentioned a little bit pre earlier today and there's another talk I guess towards the end of the week on light bvhs but ours is you know using the specular uh sampling the specular array on the material and unfortunately UI out of the box doesn't have access to light source is in the bvh so we're actually going to create our own special light bvh that we uh sample separately importantly the light BBH since we're just using these uh parametric lights uh it's just really simple geometry that we can stick into them so sphere hemisphere Square cylinder that's all we need just a bunch of instance transforms to make the light BBH work and then we only need Trac Ray inline because all we need to do is get a position in which object we hit to be able to produce the uh the data so looking taking a step back here's the visualization of light bvh with the

specular highlight on the sphere so you can see that's where the light was uh was existing that we had to shoot a right to get to so let's move on to temporal resampling temporal resampling is you know the place that really starts to making as good um we're never truly unbiased with our temporal uh resampling in ue4 or ue5 um because full bias correction requires that we have a historical bounding volum hierarchy can't really do this in games typically because we're doing things like refitting uh skeletal characters and we just don't have the uh historical bosses from the last frame and it would take a ton of memory to you know do that sort of stuff additionally we default to this modest Max history of eight um and this Max history is sort of capping out the importance of samples based on like how many times they've been reused and a larger history means more stability a shorter history means that we can have uh less temporal lag on say a disclusion for a shadow or something like that that um sometimes our artists want to turn it down even lower than eight depends on

what they're doing in the scene and then it's important to possibly keep your own copy of a g buffer history uh depending on how your engine operates and I'll show you an important thing that we ran across in the last couple weeks normal buffer for a current frame for in the amusement park you know showing our opaque shading that we're doing with the bottom of a pool of water but if we look at the Historical normal buffer that the engine keeps a hold of we see that the water has been composited in this causes a bad temporal reprojection for us because we you know end up thinking we're a different surface um and the reality is we just keep a separate copy and this is because the engine wants to have a g buffer that includes this information for its own uh temporal anti-aliasing so let's move on let's talk about spatial resampling well we've done a few different things that make things better I'm not really going to spend any uh time on these you can look at these slide notes uh after the uh course but I will talk a little bit about this bidirectional occlusion uh because it's

an interesting thing that you may run into when you go ahead and implement this stuff so this is a very complex occlusion scenario that we see in the beastro scene with table chairs a whole bunch of overlapping Shadows because there's a whole bunch of Lights coming down from the ceiling and Shining down with these overlapping Shadows we can occasionally run into a problem of runaway light reuse and this causes sort of an energy runaway if we run into this runaway reuse thing so what's going on here is that we have these different reservoirs that are looking out from underneath the table and seeing sort of different mismatch Set of Lights and if they start reusing between these reservoirs then they think that you know they're getting the history of those other lights that they can't really see so what we can do is we can cast occlusion Ray uh for to see whether the sample I'm attempting to reuse from can see my own lights or my own light if it can't see my own light we're going to we're going to avoid this reuse and by avoiding the ReUse we're going to get a nice uh clear stable

image here again this is what we look at look look like with the runaway reuse and this is look what we look like if we do the bidirectional occlusion test to break that it's only necessary if we're doing ollusion re uh testing during the spatial uh s uh during the spatial reuse now great we've uh we've done all this what does it all look like in the end well after when we do the final shading we get you know nice soft Shadows from reer and they match up very nicely against path tracing but we've had this since we've you know had access to R Trace Shadows uh in uh in games but moving beyond that things like the U default uh Niagara particle lights can't cast Shadows because it's just these simple lights that don't cast Shadows um but with reer we've gone ahead and we've upgraded that to be able to cast Shadows from particle lights and including soft Shadows additionally we can fix the split integral problem so we've got the

nice W light with the red and green texture on it and we are producing yellow shadows in the back of the room along with some you know artifacting a little bit of like catching the uh around the top of the room we moved to rester we're getting the correct color uh sampled in the back with respect to the Shadows because you know only one sh one Shadow can see only the red one one Shadow can only see the green so again without with and the path racer ignore the programmer art I made the thing invisible in the in the path Tracer case um but finally because we've done all this stuff we can start doing some more interesting effects so here we've we we produced this unified framework and so now our uh Ray can uh can integrate some transmissivity along the r and then apply that to our our rester result and we can get these nice uh colored Shadows from the

balloons so I you I think that this is a nice example of how we can improve Dynamic lighting just by applying these path racing principles so we've done this for direct lighting and we've already improved things but it's not a path Tracer without some sampled indirect lighting so indirect lighting what do we mean Global illumination Reflections or specular GI emissive surfaces you could argue either way for the purposes of unreal 5 I'm going to say that it uh belongs in the indirect lighting case not going to talk about refraction today again we want to extend replace rebuild what do we want to do well this is a fundamental question um rebuilding allows us complete Freedom which you know we can do whatever we want extending allows better compatibility right so if you've already designed something to work well with Lumen then you've you know made it fit within its bounds so let's start looking at what we're so specular indirect paths we sample the material we shoot a rate and

we hit something and we light it Etc it's sounds like Reflections so what does the Lumin Reflections path do for us well the luminum reflection pass is going to just sample reflection pass um but we're going to focus we're going to talk about the hit the hit lighting uh version of this because we want to do the high quality stuff so we're sampling these reflection paths um sampling this specular pass it's a stochastic sampling it does multi bounce um and it actually has passes for water and glass uh if it if you set things up uh for that um unfortunately the light evaluation is not quite what we want because it's doing a dense evaluation or it's evaluating all lights that it thinks is uh is is hitting that surface um and this limits our light count so we're not really getting to the full level of path racing that we'd like to get to here so what can we do with enhancements well let's first let's create an Al alternate path mirroring the one that's there uh

just to avoid sort of causing people to have to run the code uh and then we're going to add some initial S our initial sampling Loop for the light evaluation and treat the reservoir sources independently just evaluate up to n times per Reservoir and with this we can get a you know nice three bounce uh image of the Funhouse mirrors in our amusement park with you know the full 1300 lights potentially affecting them so I think we've got a win here now let's talk about the diffuse in direct PA just like specular we're going to bounce into the scene etc etc um and then produce our path well what does Lum and Global illumination do I'm going to simplify things here talk about it in a little rough sketch we're going to subsample the screen to generate probes and then from probes we're you know effectively going to have Rays into the scene but the scene ultimately is getting its color from a uh radi cache look up in the Lumen surface cache and there's a visualization of what the

surface cache looks like mapped to the scene uh in the upper right there and the surface cache is going to get its light by evaluating direct light uh coming in on the surface cache as well as taking some bounce light from other surface cache cards well this this is a path this is It's you know it's got these probes and surface caches intermediaries but it's a path so how can we reuse this well things are kind of interesting because um what's missing here is that we we don't have the light scalability again like we have like we talked about in Reflections so why don't we do this interesting thing of connecting at the surface cache this looks like a really uh interesting point so let's take a look at this lit scene and then let's look at what the surface cache looks like in the lit scene in the slit scene we've got you know sort of a low reses version of the of the scene uh and but you know a low res version of the lighting now let's go ahead and do

sampled lighting with rear uh and you'll see that hey we've got a noisy version of this it's kind of extra noisy but um it's definitely doing the right lighting and now if we look at it with the filtering and denoising that's already going to be happening to it anyway applied we've got what looks like a pretty pretty decent image it's not doesn't look perfect but actually let's compare it to what Lumen actually produces and you'll see see that what Lumen actually produces is very very uh nearly identical to what we're producing via sampling so I think that's a win but importantly all we're really trying to get is this diffuse indirect and this is the sampled version of the diffuse indirect and I think that we're again getting a very very good result here so that's great we've done that but what can we do further well we looked at we've now looked at restar Global illumination this is something that's you know in development just coming online at the moment so let's utilize rester uh to go one step further

and let's reuse GI paths instead of reusing light paths and then we'll use the same spatial temporal framework and we'll do per pixel per pixel visibil we'll get per pixel visibility for higher Fidelity than what Lumen would do because of the probe based nature of it and this is going to give us you know nice fine detail for emissive sampling um and it's important to knowe that you know we're still reusing the same Frameworks here and importantly we'll still reuse the uh Lumen Radiance cache for terminating bounces so let's take a look at what we get out of restar GI it looks good you know the Cornell box scene that we all know and was just doing a single bounce however let's tie it into the Lumin surface cache and now we've got a really nice image here and you know we're getting multiple bounces out of it comparing to the path Tracer we're very very similar so um again this is trending really well now this got our artists excited and so they put together a scene lit just just by emissive materials here we've got Lumen being lit

just by emissive materials from you know our artist sort of dream scam and here we have it with restor GI you can see that we're able to have less uh energy loss by switching to restar GI and getting that sort of finer detail into it again Lumen and restar GI and there we are so need to talk about performance because we need to make this fast so performance has several aspects but you know we we made these pretty pictures we need to hit 60 HZ or better lots of optimization opportunities there's quite a bit of code that was involved in getting getting through all of this um I'm going to talk a couple about a couple things you know both on the CPU and on the GPU side so on the CPU side light management is a huge deal uh for performance because um Dynamic lighting is expensive as we all know from a CPU perspective and it's because in a rendering engine we're to do things efficiently we're

typically having to track occluders and receivers and this is an N bym complexity task and unreal is no different so the nice thing about reer is that it allows all this to be done via statistics and it's going to scale much better so effectively we're going to have GPU driven lighting and if we have GPU driven lighting and we don't have to you know track this N byn Stuff we can just stop updating this track in and we get an infinite improve Improvement in the overhead of tracking these uh these these interactions additionally we can also we can also um eliminate these extra perlite passes that we frequently have to do uh for a bunch of different effects so that's a you know a win on draw call uh compute Shader launch overhead so we've we've cut down on CPU quite a bit and we're rendering way more lights now G execution Divergence is something that everybody that has worked

with rate tracing uh knows to be a challenge and is something that you know we have to address here so um ue5 has native solution for Divergence and that's the trace sort Trace uh algorithm that you may be familiar with it's requires three discrete kernel launches where we Trace to find out a material ID we bucket sort the material IDs and then we do some retracing to actually do our shading uh and whatnot and some moderate code complexity and it pairs you know fairly poorly with multibalance because we can't you know do a whole bunch of sorts here I'm have to have a confession I'm substantially responsible for this algorithm um you know one day I said to Adam hey what if we did and then talk to some Engineers epic and well yeah it's now in the engine and has been for a while but an ideal solution would be single Shader uh single dispatch recover every bit as much Divergence as we were getting in this Trace stor Trace mechanism um but uh we're going to make it work with multi pounce as well so I

give you Shader execution re ordering now this is the only slide in this talk that has anything Nvidia specific on it so this is because Shader execution reordering only exists today as an nvapi extension we can take a trace Ray command and split it into effectively three nvapi calls uh in the hlsl Shader and we can do a trace to get to the object and identify the hit object object do a sort on the hit objects and then do the evaluation and continue on our way now we can do things a little bit more complicated but you know just doing this on line to three line transform is all you really need and this is worth you know frequently 20 to 50% impr Improvement in performance in our reflection Shader that we were looking at um we still leave the original code there because we want to be compatible with other things as well so it doesn't none of this work requires the Shader execution reordering it just makes things faster okay so our reflection Shader is faster but it's still too slow uh when I

was looking at it in the amusement park and the reflection Shader good news is it's spending most of its time shading and the any hit and closest hit shaders are not a real problem because we've already gotten their Divergence uh taken care of by our uh Shader execution reordering but 73% of our Cycles are going into uh instruction cache misses and this was found via end site that's a that's an sight uh Shader uh Trace so if we look at look down and dig dig down into like where the functions are we look we see these GPU uh instruction cache misses are um basically 6% of it is coming from just generating random numbers and 86% and that's it's spending 86% of its time with instruction cache misses um and all the other functions are pretty much the same way if we start expanding this out we see how many calls sites things are happening at and then we start kind of thinking about the fact that dxc can

inline every one of these call sites so this is what the original code looks like fairly reasonable software engineering we've got you know a few different variations of sampling samples that we need to take um and then this is what you know diving into one of those functions we see that you know hey it's got the random number generation and it's got this uh this reuse and uh sort of evaluation function that's necessary for waiting and re and doing a resample and these are called in every single one of those uh cases now we need duplicate code we need to duplicate it at a binary level um and this is because DXE is inlining the world and polluting the iach and that's just a mess we can start thinking of this as a state machine State a go to process Loop go to state B process your Loop and from that perspective we can now say hey maybe there's a curve first control flow system that we can come up with to to deal with this I'm going to give you the

infamous for switch Paradigm so we're going to Loop over all of our states and then we're going to take a switch statement to evaluate the setup code tied to a particular State and we're going to execute a loop to take our samples within the sample loop we're going to execute a switch statement for which type of sample we want to generate but now our random number generation code and our resampling code are at you know one location each within the loop and so as long as we don't unroll these Loops we have just one location in the instruction caches for them to be pulled through and poof our instruction cache is problems are fixed this is 20 to 40% Faster by moving to this uh you know State machine-based Control flow um and it impacts many of the shaders in our system and it enables addressing other performance problems that we have so we talk about you I'm not going to

talk about some next steps today fog volumetrics transparency we've done some things in this space but um just we don't have time to go even deeper into it as Adam mentioned previously all this stuff is available in our nvrx Branch so you can go look at the source code right now if you want to and then additionally we have some other rester and sampling uh talks at GDC this week I point you you know strong WR L at the Gent uh or sorry this is got the slides backwards this is uh some other resources on restur sampling uh that you may want to take a look at um we have some other talks on uh real-time path racing and games this week first talk is our art talk uh about this ue5 work uh if you're interested in seeing more about it and more of the use in a sort of in a workflow then we've also got talks on games with cyberpunk and Allan Wake as well whole bunch of people to acknowledge our artists that have worked on creating these scenes and and telling us what's

what's wrong people have worked on rester and and got the the theories together uh some people have helped us with some denoiser implementations and just general advice and feedback on the talk and finally there are even more sessions that Nvidia has sponsored at GDC so please go ahead and take a look at those and then I would like to open it up for questions as we take a look at the video from the uh uh emissivity lit uh scene yeah again there's zero parametric lights in the scene it's all just emissive materials doing the lighting so

I think it's a pretty happy result any question questions again remember to fill out your surveys hi so uh how many total race per pixel do you do for these scenes in this case the total Rays fired we had two

during sorry we had one during the initial sampling in this case we had none during spatial resampling because they decided that uh it was not necessary they had one during temporal and then I think they have maybe five in the reflections and the reflections won't fire unless you know it's a shiny enough surface basically um and then and however many the Lumin surface cach needs because that's where you know it it's off of the surface cach resolution basically thanks hi uh great work um a question about do you guys have any plan to integrate this into the main branch of unre so that um can use it um the main I mean this this is ultimately up to Epic we're always you know interested in in working out whether there's something that they can take from our code uh and you know we've had many times pieces go in and yeah this is just

now getting to a good place to like start considering that because we had to get all the Lum and stuff working if you do want to mess with it you can pull our Branch it is Branched off of Main so you'll get the same Code Ultimate ultimately hi so I noticed that when speaking about retracing most of the time is about GI or reflection and pretty much everybody tend to ignore the refractions is it because like uh is technologically more difficult you have to render like a refraction part with multiple bounces and a reflection part or is just simply because it's not worth it for so many sces you don't have so many refractions to deal with um so is the question about uh like what

what specifically makes this path tracing versus doing indirect rate tracing no the question is more like uh why uh the focus is mainly on the reflection parts and if uh there are some technically technical limitation that prevents the refraction to be investigated on yeah so we we we just haven't addressed the refraction case yeah um the the way our artists had set up the scene there wasn't much of any refraction there and we could have done it in the water but they' kind of set things up for just a distortion Shader so we work with that yeah okay thanks I think no specific reason yeah it's mostly because like this scene most of the time you just have a lot of reflection and no refraction at all yeah okay thanks yeah also I mean from from our perspective we we're kind of focused on things that you know games could use today and you would have to kind of create special art for utilizing

refractions typically so um you know we we do want this to be you know as drop in as possible for people hey great talk by the way uh so a colleague in my team is actually looking at the brands right now and he noticed that it only supports everything in the D Tex back end so are you planning to enable all these features using the pan back end as well yeah um today so I I think that the like the the the reordering stuff we don't support via the Vulcan at the moment um I think that's the only thing in there um it's something that we need to take a look at um we just haven't spent time on it uh mostly because of the fact that the Vulcan rate tracing implementation had been a little bit behind the direct X1 in ue5 from when we

were when we started building on this we started building on this work back in ue4 so um yeah we just haven't gotten to it right thank you hi yeah I have a question um could you go back to the slide with regards to how you perform sampling for specular materials um so either I heard this wrong you could correct me um is the way that you perform sampling for specular materials to directly sample light sources and bias your samples towards that or do you do it otherwise because I wasn't really sure as to how the sampling happens um spec specifically for specular Reflections yeah I think wait did you pass it oh

yeah I passed it no yeah no I didn't brdf you're going brdf part this one right yeah okay so what's going on here is that we're actually this is actually a form of multiple important sampling we're not biasing anything so we do properly multiply important sample with the local light uh and the Skylight because the can hit the guy as well okay got it thanks all right they're telling us we're out of time so fill out your surveys and have a great GDC