Real Time Rendering for Feature Film: Rogue One

Welcome to to real-time rendering for feature film. We'll be presenting a a case study of some work we at the Lucasfilm Advanced Development Group did last year in support of Rogue One: A Star Wars Story. Today's session is going to be shared between the three of us, three speakers. John first is going to give an overview of the film itself and how this real-time work came to be involved. Then I will get into a breakdown of how ADG contributed to Rogue One. I'll speak to the details of the digital assets involved and the workflow for lighting the shots. Then I will hand over to Nattie to discuss the technical details of

rendering this content. We will also touch on some ongoing work in this area happening at ADG and at ILM xLab. Then we'll close with with some time for oh I haven't been doing my bullets. We'll close with some time for Q&A, but first I'd like to give some introductions. So since 1986 John has been at Industrial Magic where he is now Chief Creative Officer and also a senior visual effects supervisor. As John is an avid computer graphics enthusiast and hobbyist himself his shows have always been known to push the state of the art in production CG. John won an Academy Award in 2007 for the visual effects work that he supervised in in of the Caribbean: Dead Man's Chest. John is not only the visual effects

supervisor for Rogue One, but he is also the creative mind behind the original idea for the film. Oh, and by the way, in 1987, John and his brother Thomas created something that you might be familiar with, Photoshop. Natty Hoffman is principal engineer and architect for rendering at the Lucasfilm Advanced Development Group. Natty has a long and storied career on the cutting edge of real-time computer graphics from Westwood, Naughty Dog, Sony, Activision, 2K, and now Lucasfilm. Since around 2004, Natty has been spearheading the transition to practical implementations of physically based shading models across the games industry. Which I'm sure many here are familiar

with. Natty joined the Advanced Development Group at Lucasfilm 1 year ago in March of 2016. Me, I'm Roger Cords, digital production supervisor for the Advanced Development Group. I've been with Lucasfilm since 2010. I'm one of the founding members of the ADG. I was the lighting and look dev lead for Star Wars 1313. I am a real-time lighting and look dev specialist with a passion for generating high-fidelity images. One of my dreams is to match ILM quality in real time. So, this work that we're going to be talking about today was an absolute slam dunk for me. We should also introduce a couple of other concepts here, the concept of ILMxLAB Um, and of the, uh, advanced development group. So, in 2016, Kathy Kennedy, uh, firmly planted a Lucasfilm banner in the world of immersive entertainment uh, with the launch of

ILMxLAB. XLAB brings together the creative forces of, um, the Lucasfilm Story Group, of Skywalker Sound, um, Industrial Light & Magic, and of the Lucasfilm Advanced Development Group, with a mission to create new, premium immersive entertainment experiences that go beyond traditional film and other linear media. As for the Advanced Development Group, we were founded in 2013, uh, to be an innovation center within Lucasfilm, focused on real-time rendering. Uh, up next we'll show a few samples of the kinds of imagery that ADG technology has helped to create. So, first, uh, at its core, Lucasfilm is about story. So, with that in mind, this first piece is a a short cinematic that, um, the Advanced Development Group put together in mid-2014 to show what real-time rendering, uh, what we thought real-time rendering technology can, uh, can bring to the creative world of storytelling. So.

Report. According to our informants, a pair of rebel droids have attempted to make contact with the local underworld. They describe the droids as a golden protocol unit and a blue and white astromech. They may be attempting a rendezvous here at this outpost. Our ground forces have just engaged a transport that blasted its way out of Mos Eisley. Proceed, but secure the perimeter first. Those droids must not escape. Fan out. Freeze! Wait for me. No.

Hello. Captain Gator, Captain. Artoo and I are ready for extraction and heading to the rendezvous point. I repeat, we are ready for extraction immediately. I've got Imperials all over me, but I'll do what I can. I've called in some backup. That doesn't sound very reassuring. I'M SURROUNDED EVERYWHERE. ARTOO, ARTOO. ARTOO, LEAVE. I CAN'T I'm saving yourself. Isn't this how it all ends? Thanks for saving my skin. Stop right there. Oh, no. Um then Oh, thank you.

So then last year, uh the advanced development group developed Trials on Tatooine, a a VR experience um with which uh Lucasfilm and Industrial Light & Magic launched the ILMxLAB brand. Uh and here we have the launch trailer.

Uh then this one this is a high fidelity rendering test that ADG did um which kind of leads us into what we'll be talking about today. Um so with with those introductions out of the way I'd like to turn the presentation over to John Knoll uh to talk to us about this specific real-time project and what it achieved for Rogue One: A Star Wars Story. Thank you. Uh so my challenge to the advanced development group was to to take this development and use it to create some finished work on a feature film. The specific challenge was to take an ILM production asset you know without completely rebuilding it specifically for a game engine and rendering it using the real-time render at a quality level that I can put in the film. Uh success story here is that the ADG shots are mixed right in with our RenderMan renders and we have a short reel of K-2SO shots and it's a mixture

of RenderMan renders and ADG renders. So can you spot the difference? Turn your lights off, sorry. Our optimal route to the data vault places only 89 stormtroopers in our path. We will make it no more than 33% of the way before we are killed. We could transmit the plans to the rebel fleet. We'd have to get a signal up to tell them it's coming. It's the size of the data files, that's the problem. They'll never get through. All right, so uh we'll play the reel again, but uh with uh ADG shots clearly labeled. Okay. Our optimal route to the data vault places only 89 stormtroopers in our

path. We will make it no more than 33% of the way before we are killed. We could transmit the plans to the rebel fleet. We'd have to get a signal up to tell them it's coming. It's the size of the data files, that's the problem. They'll never get through. All right, so Roger and Naughty will present a technical breakdown of how this is all done, but an important question is why, you know, why are we Why is real-time rendering important for Rogue One? And The answer is that in that in visual effects the the ability to iterate is crucial. Instant feedback and fast rendering mean more iterations, more creatively useful iterations per day. And a shorter iteration cycle gives us higher quality in fewer hours, and that's that's why we're chasing this. So about 2 years ago, and this was back during the production of Force Awakens, I'd seen some early real-time render

tests out of the advanced development group that seemed extremely promising. Um and uh early 2015, ADG generated a version of the X-Wing shot that was in the original Thanksgiving trailer for folks Force Awakens. And ADG had rendered the X-Wing elements in seconds per frame, whereas the ILM software renders, you know, for the actual teaser were multiple hours per frame. And the ADG rendered X-Wings were then put through the exact same compositing step as the and the results were pretty good. The two versions don't look exactly the same. They're not identical. Um and there are a number of cheats in the AD ADG shot. Like for example, the area lights are are crude approximations and the occlusion calculation's a little bit different, but you know, generally when I I saw this, I felt like, all right, well, this is something we can we can work with. Um then last year ADG uh hired uh Naughty Huffman and entrusted the future of Lucas Films real-time rendering to

him and that uh gave me even further confidence that this is uh direction to to to pursue. Um so we've established ILMX Lab in the ADG as the place within the company where ILM CG artists uh sit right alongside game industry talent to build these, you know, exciting new things. And that's exactly what I needed for this this project. So the the challenge to ADG was to take this idea of rendering production quality images. Um you know, out of experiment uh and and testing, you know, I thought uh what I'd seen was promising enough that uh all right, well, can we, you know, this looks like we've we've hit a bar that this could go into a feature film. Um let's take it out of theory and into practice. Let's do a a shot. Um so the you know, I I was pretty up front that uh that we had to hit a level of uh of quality that was as good as the RenderMan renders. You know, if it didn't hit that bar, then we couldn't use it.

But I was pretty happy um with how that turned out. I I I think it succeeded pretty well. Um and uh let's see. Um so talking to ADG about uh you know, what we what asset we could try this with, um you know, they they were looking the their ideal case was uh was a hard surface asset, opaque, um pretty much like the the X-Wings. And a And a perfect uh candidate for that was uh was our K-2SO character. Um he's going to be a hard surface asset. He's going to be CG all the way through. Um and some of the the more difficult shading challenges uh aren't actually present in the the character. So, it would be a good test case to do. Um and with the unified asset standard um having been uh designed and come into fruition between ILM and ADG during the production of Force Awakens, um

it seemed like everything was in place that to give them a a crack at this. And so, the the first first step in this process was to mirror a shot that was being done through the traditional pipeline. So, we had a shot that was being lit by an ILM lighting TD and RenderMan. And uh that kind of set the the visual bar, the the look. If we could hit this, um then we could do another uh another shot sort of without a safety net, without a backup plan. Um so, we proceeded in parallel with uh with the shot. And once we got to a place where I felt like, you know, we could final that uh that element, then they proceeded with with doing these these shots with no backup plan. I'll also say yeah uh besides uh for final shot production, you know, there's an important use of real-time rendering uh just working with uh with actors. Um so, in pre-production, we had this uh very uh sort of crude early version of K-2SO

uh that that ran in real time on our motion capture stage. And we had Alan come in for a day. And you know, since the character's very tall, he was standing on uh on stilts. And so, he had had a little time to get used to the stilts. And then uh this was a chance for him to get on the stage and sort of puppeteer the character. And he could watch himself in real time while he's uh you know, seeing what what feels right on the character, you know, is it do do more or less arm swinging, uh you know, how loose is your body, what what feels right with the character. And while this is, you know, a very crude GL um uh the because this was done, you know, um in way in pre-production before we had the final K-2SO asset built. Um the the same asset that we used for the film, we could have flipped the equation. You know, the the constraint for the film was we had to have very high visual fidelity, can't have any noise, you have to have really smooth motion blur, uh can't be any visible

artifacts, um but I don't need it to run at 60 frames a second, so I can afford longer render times. Uh but for this purpose, we could just flip that and, you know, impose the it's got to run at uh at speed, and we could live with a little bit of uh uh noise in the renders. Um so we could you know, in future using some of the the this new ADG engine, I think it'll make this experience even better. Uh and with that, I think I'll turn it back over to Roger. So, uh what what does it actually mean to uh to work on a shot for an ILM production? Um well, it means that we will consume assets uh from the ILM pipeline. I it means that we will artistically light the shot. Um and then we will uh we'll output renders um that are consumable by the compositors. Uh so, let's let's break that down a little

further. Uh what what what do we mean by by asset? That's kind of a loaded term, too. Um there there are many many many types of assets, and uh each each type of asset is often represented by many many many individual files. Um so for this Rogue One work specifically, uh, the the assets and file types that we're dealing with um, are animated geometry caches, uh, paint and look dev data, um, cameras, background plates, virtual set geometry, and uh, captured HDRI lighting spheres uh, from from set. Uh, as an ILM CG asset, uh, K-2SO himself comes to us in two primary components. Um, the geometry stored in an Alembic container and the paint and look dev data represented in a Material X description.

Alembic is an extremely powerful and extensible open standard uh, for transporting geometry between software packages. Uh, born out of a partnership between Industrial Light & Magic and Sony Imageworks. Um, if you're curious about Alembic, you can learn more at uh, alembic.io. Um, Material X is a new open standard um, that has come out of the Advanced Development Group and Industrial Light & Magic uh, which allows material and texture information to be transported between different software packages and renderers. Material X is a key component of our unified assets standards. Um, it allows us to transfer unified shading descriptions uh, between all of our packages, between paint packages like Mari, look dev packages like Katana, uh, other DCCs such as Maya or our own internal tool Zeno, um, and even other platforms entirely like game engines. Um,

if you're interested in Material X, uh, please check out materialx.org. Adding support for Material X and Alembic to into the ADG version of the Unreal Engine uh, was one of the major technical undertakings of this Rogue One project, Um, which Nattie Nattie will go into more detail there, but the key takeaway here is that because these standards are both so easily extendable, we were able to take assumptions that are baked into ILM's production implementations and reformat the the asset data to be suitable for for a real-time case like like Unreal. So, by the time ADG got our first K-2SO turnover, the asset was basically complete at ILM. We had gone from from these paintings to Landis had had made this digital asset. Um Here we are looking at the control cage for a creased subdivision surface. There are 600,000 some odd vertexes in the

control cage alone with edge creases that go up to level four. So, that means that for an extreme close-up on one of those creased edges, if you were to to uniformly subdivide the the entire cage, you'll end up with 600,000 * 4 ^ 4 or somewhere in the neighborhood of 154 million vertexes required in order to properly hold those creased edges. Fortunately, none of the shots that we were dealing with required this extreme case. We also have 1,700 separate geometries in the animation hierarchy and 63 Udims across 10 texture effects with most of those textures stored at 4K resolution all driving look dev in the unified shading model shared between Lucas Film and ILM, which we call unified surf. So, each and every component of K-2SO, every little piston and ring fitting, each of them can and does animate.

So, this is the beauty for us of the the Alembic cache. We don't have to be able to evaluate that that animation control rig at runtime when we're rendering in the game engine. We just consume the transforms from a baked geometry cache. Here's another view where you can see in his shoulders the all the little pistons how they move from from his arms moving and his knees and and wrists and elbows every little bit inside has has moving parts. As I mentioned a moment ago, K2's material X look dev data has textures for driving 10 effects in in the unified shader. That includes texture data for for a second specular lobe and and actually a third. But at the the time of this Rogue One work, um,

ADG's real-time implementation of unified surf only supported a single specular lobe. So for for the shots that ADG contributed to the film, the lighting and look dev TD Justin Schubert had to dial in a custom specular look for K2 that mixes the spec one and spec two effects. John wasn't exactly thrilled about this, but the results looked good enough and so we powered through. Here we see a comparison between the two lobe RenderMan renders on the left and the results of Justin's efforts on the ADG render on the right. I mean, obviously it's not the same shot, not the same lighting setup, so we we get away with it, right? So now that we have the the K2 asset, the CG asset for K2 himself, we've brought in the Olympic of the camera. We have the EXR textures of the the background plates and of the lighting spheres imported into Unreal as HDRI lighting environments. Now we can get to work actually

lighting the shot. So here we have some screen capture footage of contrived version of an interactive lighting session because we didn't actually capture any of this while we were doing the work. Um Uh so, you can see what what the the interactive lighting session would look like. Um I should mention here that for games industry veterans such as yourselves, uh real-time lighting is something that we kind of take for granted, right? Um but real-time light and shadow feedback uh on a full resolution, high-fidelity Industrial Light & Magic asset uh with all 63 UDIMs of 4K textures and however many millions of vertices slammed into the GPU, um seeing this kind of of real-time light and shadow feedback uh was was revelatory for ILM lighting artists. Um I should also mention that uh at at Industrial Light & Magic, um you know, the concept of CG lighting uh that means area lighting. Um and area lights means textured area lights. Uh it's a an a

high dynamic range photograph of a uh taken on location of a actual physical light source that was illuminating actors and props and sets. Uh at ADG, we have some approximated area lighting shaders for surface shading. Um we have basic rectangles and disks and also the standard Unreal spheres and pills. Um but we we we don't have textured area lights. Uh one important addition also is that area lights at ILM means area shadows and that one is uh an even tougher pill to swallow. Um once again, Justin was able to hack his way to the visual result that satisfied um uh production for Rogue One by dialing in a tunable filter radius on uh on standard PCF shadow maps. Um Yeah, at at this point, I'll I'll pass it off to Naughty to uh to dive into some technical details of rendering. Thanks, Roger. So, um Unreal Engine now

has some support for loading Alembic files, but we needed an implementation that could handle ILM Alembic files. As Roger touched on earlier, Alembic is a very flexible and extensible format, and ILM took full advantage of this extensibility and has extended it quite far. One of the ADG engineers, Ron Radetzky, did significant work to enable our pipeline to take in this highly extended Alembic format and turn into vertex buffers that could be directly uploaded and consumed by the GPU. After animation happens on the ILM pipe in the ILM pipeline, a baked take uh is cashed out in an Alembic file. This is a vertex cache of that final animated result. The baked take is a pure geometry cache. It's baked every frame. It has no information on the rig or deformers or anything like that. It's all baked out uh for that one frame. Um K-2SO is mostly rigid.

And um So so so uh most of that cache is rigid transforms. There's a a small amount of uh kind of bendy bits, basically his antenna. They are very small compared to the rest of him. And um in these particular shots, I'm not even sure they animated uh at all. So, you could basically think of it as a very complex rigid setup with a lot of rigid pieces. ILM geometry caches contain subdivision surfaces as well as traditional polygon meshes. So, Ron's implementation had to deal with both. For the subdivision surfaces at the time of the Rogue One work, we determined that we only needed to render at subdivision level two in order to hit the visual quality target. We did some tests at level three, but for these specific shots and cameras, level two was quite sufficient. This allowed us to pre-subdivide the entire mesh in main memory, and this was the most straightforward. It wasn't the most elegant or the most performant way, but it was definitely the most straightforward way to shim this data

into Unreal given the film schedule that we had to work with. Now, all of the subdivision surface rendering is built on a foundation of the open standard from Pixar, OpenSubdiv. For Rogue One, we did subdivision on the CPU. OpenSubdiv does have a GPU implementation, and we took a look at it, but we found that it has some problem cases with UV texture coordinate data, and it doesn't work well in ILM assets like K-2SO. In the Epic keynote this morning, Epic announced OpenSubdiv support in Unreal Engine. This implementation, of course, didn't exist when this work was ongoing, but we look forward to evaluating it for possible use in future projects. Rendering K-2SO uniformly subdivided level two was quite stressful for even our our most powerful machines. As Roger pointed out earlier, every subdivision level is multiplying the polygon count by four. So, uh Ron, who was working on the Alembic and subdivision uh part of the of the engine, thought there must be a smarter way to go about this.

So, in parallel to Rogue One work, Ron developed a streaming Alembic solution that did not require subdividing everything up in memory ahead of time, and he also developed a curvature-based recursive reduction algorithm for removing redundant spans on subdivided surfaces, as you can see in the slide. The The end result is quite a bit simpler. Using the redundant span loop reduction method, Ron is able to render a version of K2 that is visually indistinguishable from the full level three tessellation all while maintaining a constant 60 Hz frame rate. While we did not get to take advantage of this on the Rogue One work, we continue to enjoy this advancement for our current projects. In addition to subdivision surfaces, another facet of our Alembic geometry cache consumption is handling the motion blur time samples, which is a specific aspect of ILM's flavor of Alembic. ILM's geometry caches include motion blur sampler samples at three times for each frame when the shut camera shutter opens, when the camera

shutter closes, and the time in the middle. This includes full data for all the geometry and any cameras that might be moving for each of these three points in time for each frame. Ron's implementation enables us to do temporal super sampling by interpolating the camera engine geometry positions between in between those three subframe samples. For the Rogue One shots, we rendered out images at 64 different times spread over each frame. This is of course computationally quite heavy, but we needed to do this if we wanted to get motion blur that was up to ILM standards. We get some additional use out of these renders by also jittering the camera at the same time using a Halton sequence. This also gets a spatial super sampling at the same cost. We have some ideas for applying these samples in another dimension, for example, area light sources as well in the future. Before we talk about rendering, we should talk about compositing, which is the process where the final image is actually put together by a specialized CG artist, a digital compositor. It bears some resemblances to the post

effect passes that we get in game renders. Various blurs, blings, blooms, flares, all of these effects happen in in comp or in the composition composite phase. And this would be no different for our shots, just like every other shot in the film. So, we needed to be able to get our ADG renders output and hand it over in a way that is seamless and transparent to the compositor. At ILM, TD renders are handed off to the compositor in OpenEXR format. While Unreal does come off the shelf with EXR output capability, in in that feature, the image is written before certain post process effects happen, and that those include unfortunately some lighting features. And we needed to output the full lit and shaded beauty render in an EXR container. That was a fairly trivial change to make in our version of Unreal Engine. So, we render motion blur, but the depth-based defocus blur occurs in in the comp phase. And this is true also for the ILM renders. We knew that we needed to have an extremely robust motion blur result in

order to pass muster with John. We also needed to be able to output depth so that the compositor could have control over the depth of field in the final output. The concept of arbitrary output variables is a staple of offline rendering and compositing at ILM and at other feature film houses. Uh these AOVs are output by the renderer, and they are used during the compositing phase. On the surface, these images are somewhat similar to G-buffers that you might have in a deferred shading engine. One key difference between G-buffers and the AOVs, the A's stands for and really does mean arbitrary. These values can be stored from any point in the shading computation. It can be even some intermediate value that's only present during a particular subphase of the shading calculation. And to generate that kind of arbitrary output from any point of the shading math in a game engine render would require a lot of plumbing. So, we identified a minimum set of AOVs that the compositor would need for our specific shots.

Depth is one example of an arbitrary output variable that were we were easily able to accommodate. Ideally, this exercise would result in less compositing work than traditional ILM shot, since something closer to the final result would be visible to the lighting artist interactively. So, we were able to get away with three AOVs for most of our shots. The beauty RGBA, which is basically the final rendered color. We have the depth, which I mentioned earlier. And an object ID buffer. Some shots or AOV. Some A few shots required a fourth AOV, which was an emissive mat. Rogue One was an ACEScg show at ILM. At the time that this work was done with an ADG, our version of Unreal was still limited to the sRGB or Rec. 709 color gamut. This was another area where ADG and ILM had some creative back and forth in order to ensure that the Rec. 709 renders out of ADG fit correctly into the Rogue One ACEScg color pipeline.

So, now that we're able to ingest the assets and do creative work in fact actively and we have some baseline capability in place for all of these various technical features, we're ready to render. But, what does rendering for feature film visual effects actually mean? To quote ILM visual effects supervisor Ben Snow, in computer graphics, a lot of the time we're trying to reproduce the reality that the viewer sees with their own eyes in the world around them, but with visual effects for film, we're really trying to reproduce filmed reality, which is a little bit different. So, what does it mean to make things look like a photograph or like uh frame from a film? Well, arguably one of the most significant components is to have absolutely no visible aliasing. The ILM render resolution for Rogue One was 4K, but for us to be able to kill all aliasing in in our ADG renders, we had to use a combination of the jittered super sampling that I described earlier and the uniform super sampling, effectively rendering the frame at a higher resolution. So, for our final final output for compositing into the uh

movie, we ended up rendering at 9K in addition to the 64 jittered subpixel samples I mentioned. Here we see a portion of our beauty pass for this frame at a one-to-one pixel scale. Even with all of those samples, our shots were rendering at just about exactly 1 minute per frame. That render time is due to several factors. First of all, it was 9K resolution and it was rendering effectively 64 * 64 subsamples per pixel. And also a significant portion of the uh time was due to massive file IO and memory transfer. Since we're talking about extremely large uh output files and we're talking also about extremely large input files in the case of Alembic. This shot is one of the first test that we generated in order to show John what the results from a real-time render could look like. These renders were pretty far from real time, of course, at about a minute per frame. But, compared to multiple hours per frame which the RenderMan renders were taking, that's still quite a win. This is an example of a shot that was

also done at ILM. So, our work in this case was to match the ILM result. The ILM lighting setup used only three light sources. There was a rectangular area light for the ceiling panel, another rectangle for the back wall panel, and an overall HDRI sphere. Roger, who lit this shot on the ADG side, managed to stay true to that using our approximated rectangular area lights and fake soft soft shadows. Screen space ambient occlusion was our only shadowing factor for the HDRI sphere. K-2's eyes are a complex set of refractive lenses. We didn't support the unified surface refraction model for transparent rendering in our real-time version of the unified shader, so the eyes proved a bit problematic for us. We ended up authoring a custom material and uh we did a custom uh pass and um and end up rendering it separately and we got results that were close enough. Lighting TD Justin Schubert took over lighting duties on the ADG side starting with this shot, an outdoor shot with K-2 in the mid background. We expected that with a single CG sun

and an HDRI sphere and with K-2's eyes not dominating the frame, this would be a solid candidate for the ADG test. There you can see clearly both the rendered motion blur as well as the defocus uh blur that was done in comp. The focus pull, using an exact emulation of Rogue One's complex anamorphic lens model, is done entirely in comp using the depth map AOV output along with ADG renders. These results were very promising and got a good response from John. We used CG geometry for the physical set to calculate occlusion and shadowing on K-2 as he walked through the door. Additional darkening of K-2 as he walked further inside was done in comp as a final bit of sweetening. This was the last test shot that we did before moving on to actual production work. This shot was working so well, we actually stopped work on it on the ADG version of it before we had the chance to incorporate the final clean background plate paint work. This shot presented two primary lighting challenges. First of all, on the ILM side, the canvas canopies above and around K-2

were presented as complex textured area lights. ILM ILM's lighting process extracts textured cards from the onset captured HDRI sphere using an internal tool within Zeno called Lightcraft. Also, the choreographed animated effects lighting from the explosion needed to be addressed. Our approach to the textured area lights was to take the HDR images from the ILM Lightcraft solve. Those textures can be seen here on the top of the screen. And place those textures on cards into the Neal scene for reference. Then Justin built a lighting rig out of non-textured approximated area lights to match the layout, shape, color, and intensity of the hot spots that we should should be getting from the original lighting textures. For the effects lighting animation, we didn't have matte painting support for ADG Alembic animation system and sequencer did not yet exist. So, we had to choreograph all of this together via Unreal blueprint scripts and timelines which would be evaluated during the

render process. Once again, for this shot as well, the results were very promising. We got this render of the K-2 visual element to the point that John said that he would have finaled it, leaving unsaid if I didn't already have this perfectly good final shot from ILM. We did get a great quote from John on the shot, "This is the future." It was at this point that ADG director Hilmar Koch asked John if he would be comfortable now switching over to a no safety net approach for at least one shot in Rogue One. John agreed and Rogue One CG supervisor Vic Schutz identified not one but three shots from a sequence that was slated to be finished in San Francisco. These are the three shots that ADG contributed to the film. This was the first of our production shots. Everything we've talked about so far really had to come together in order for this to work under real production deadlines with no ILM safety net. We tried gobos or slide maps for the Imperial pill lights. At one point we had a very clear pill pattern projected and reflected in K-2's arms. But

creative direction ended up going for a softer look. Also, the lighting change when emerging from the hallway into the larger chamber was challenging. Due to the self-shadow artifacts with depth based shadow maps, the top lighting scenario in the larger chamber took some tweaking on Justin's part in order to resolve all banding issues. This one was a fairly straightforward shot. This was the first shot where we realized the significance of the object ID map, which is one of those three AOVs I mentioned earlier. This allowed the compositor Dan Elstrom to ensure that bright background elements came through the defocus blur correctly. This shot presented a challenge in that K-2's self-bounce lighting is obviously missing from our direct dynamic lighting setup. K-2's shoulders should be reflecting a lot of light up onto the back of his head and with a global illumination pathway so you can get that. In our case, we ended up cheating with bounce lights parented under certain bits of the animation hierarchy and key framed during the shot. We used

the similar timing setup for this that was used for the explosion effects lighting in the previous in the last test shot. This sweeping camera really illustrates the weaknesses of our real-time implementation of the shading model. This was the one shot where our single specular lobe really started to limit us from reaching the approved or blessed K2 look. Justin had to do some some shot specific material tweaks to make some of the more reflective bits and bobs sing properly. We are thrilled that we were able to contribute to a project like Rogue One. The work of innovation in real-time rendering doesn't stop here though. Current ADG and X-Lab projects in flight right now, such as the upcoming Vader VR project, demand continued advancement in both fidelity as well as performance. And we have a number of areas of active development. Area lights and shadows, support for Pixar's Universal Scene Description Format or USD format. We are excited by Epic's announcement of USD support this morning.

Pushing our real-time subdivision surface rendering techniques further, incorporating OpenColorIO, supporting true arbitrary output variables during rendering, and continuing to improve our real-time implementation of Unified Surf. In closing, I'd like to take this opportunity to give a huge shout-out and thanks to the rest of the team who made this work possible over the past year. David Meny, Nick Haines, Indy Ray, Vic Schutz, Dan Ingraham, Just- Justin Schubert, Ron Radetsky, and Hana Gillis. And we have some time for Q&A. And by the way, ILM, ILM X-Lab, and the Lucasfilm Advanced Development Group are all hiring. And Sarah, our X-Lab recruiter, is here in the room. Please stand up and wave, Sarah.

There she is. And thank you. Yes. Yes, so I'm curious a little bit about the soft lighting and area light approximation that you guys came up So our our area lighting shaders are So we ended up using both techniques. For some shots we because render time did not have to be you know, we weren't making a game to run at 60 h. We had uh multiple seconds to render these images. I don't know if you want to talk about the lighting rigs that you built that were using many many many lights. Yeah. Yeah, sure. So we definitely used clusters of lights. Basically old school point light technique of like casting a bunch of shadows getting soft lighting from multiple shadow samples. Um worked but it was extremely expensive. Way too expensive.

So basically I kind of went over to Roger and was like what do I do? I had basically a day to figure it out. Um what I ended up doing in the long run was taking the PCF shadows that exist in Unreal and putting a multiplier against its depth or its depth sampling and then uh that in tandem like that was then I implemented a slider into the lights and another guy that I work with pressure Christian Maturi helped me on that effort. Very good guy. Anyway, so in tandem with that cranking the bias I was able to actually shorten my depth and get softer shadow spread which ended up actually looking fairly nice. No, I think that pretty much covered it. Lights are pretty bright. Can't see all. Yes. You mentioned that you use screen space

screen space ambient occlusion. Mhm. And you just simply dodging using frame buffer or did you do something smarter than No, it was you would know the details but I believe it was applied in this to the indirect lighting. lights, yeah. To So, the question was about screen space ambient occlusion and whether we are applying it to the entire to the final image or and the answer is no, we're using it as a shadowing factor as an occlusion factor for specific lights light types which had no other shadow calculated. Mainly like really the only light that is shadowed by that is the the environment sphere because we have no other shadow term to pull from that. Yes. I noticed that you had certain visual effects elements in some of the shots as well. For example, sand kicking up from K-2SO's feet and in the X-Wing shots there was all the water vapor and everything. Did you also render those in

UE4 and if so, how did you approach those? No, no, no, no. That's all in the composite. So, the the X-Wings are that that specific element, that's what was rendered in the real-time game engine. And again, that rendering those X-Wings fast even just rendering the X-Wings takes multiple hours in RenderMan. So, that that was a win for us there. And on Rogue One, it was specifically the the K-2SO element that goes into composite. So, the explosion I mean actually in that shot that was a practical that was photographed and then the smoke and dust those are 2D elements that are just done in composite. And the same with the sand kicking up? Yeah, the sand kicking up actually was in the plate because Alan Tudyk was performing K-2 in motion capture suit. So, that he was actually kicking up sand and that was retained. Thank you. What kind of global illumination approaches are you thinking about using for the future?

Oh, yeah. I guess I can answer from here. Um, Yeah, so there's uh there are various uh global mostly pre-computed or baked global illumination solutions that you can use within within Unreal. You can bake light maps, you can bake diffuse indirect probes. You can bake IBLs from within within light mass, which is Unreal's baked lighting tool. Uh in terms of more dynamic GI, um we intend to sort of address that as as we run as we run into the problem because there's a lot of different ways you can skin that uh particular issue and um I think the the form in which we first encounter a need to solve it will kind of drive the approach that we take. Thank you. Um yeah, uh quick question. Um, so uh in a comparison to the approach that you've taken, um what's your opinions on like

GPU path tracers like for example like uh popular ones like Octane, Redshift, or Fstorm? These kind of up-and-coming GPU path tracers that are basically using the GPU to render frames really really fast. Like have you experimented experimented with that and like what what are the kind of comparisons with that approach? So so I certainly don't want to speak for John, but for me uh I the the fact that the image is resolved fully um and there is no no uh convergence progressive convergence uh during during interactive editing, um that even though things are wrong, that's a that's a a win for rasterization in my case. Um for the lighting artist to be able to see an image that is uh that is very very very close to what the final image is going to be um in you know, completely interactively and in fluid real time during working with no uh no shift no convergence required.

That's that seems huge. Thanks. I I thought that movie companies were rendering their graphics with Maya and programs like that. Now you're doing it with Unreal Engine. Why is that? Well, most of the shots in the movie were rendered with other tools than Unreal Engine and the full list is probably quite long. I guess there's the mainstream pipeline. Yeah, I mean the majority of the the renders on the show are are RenderMan RIS renders. A lot of the environment work is rendered in V-Ray. Then a lot of effects elements smoke and fire and that sort of thing. I think a lot of those are Mantra renders along with a number that were done in Plume, in-house package we've written. So it's a it's a pretty big mixture of different techniques. But the your question about why why real-time, why GPU renders was really about

about iteration time, about looking at uh at looking to the future where ideally we're we're letting artists uh really see what they're doing and and be able to to reduce that iteration cycle down to seconds. I think get down to being able to hit feature film level quality with renders that can are interactive. So you can move lights around, you can uh material adjustments pretty much in real-time. It it does shave quite a lot off of the amount of time that goes into developing an asset or Time is money. Um do you see a trade-off between the man-hours that is required to prepare a shot for the real-time versus just sending it off to the render farm. Um Yeah. That was one of the the requirements of this exercise was that um

there we weren't allowing ourselves any any custom preparation of the content that goes into these shots. We had to be able to consume the content directly from from the fire hose of the Island pipeline. And uh and that's what we executed. Yeah, I mean certainly we could have made the job uh in some ways easier for ourselves by authoring something that that's performant in the in the engine. Um but the for me a big part of the point of this exercise was we don't don't want to do any special we're going to rebuild it for for Unreal. No. We're going to take uh one of our our standard production assets kind of as they are normally built for RenderMan and we're going to hand it off to ADG and they're going to render it and make it look beautiful. Hi. Um thank you for your presentation. As these uh processes become more and more real time, I can't help wondering if you've spent any time anticipating

what will become of the artist experience. Uh I remember from my own time at ILM that a given asset might take 15, 20, 30 artists working on it. Do you see any of these processes, these uh improvements in render time and interactivity, altering that experience for those individual artists? And if so, how? Well, I I some some of the things I I see happening are one artist being able to do more work. You know, I I I'm looking forward to the day when um when you can have a handful of artists kind of do whole sequences. Yeah, I it's you know, a lot of it comes down to just iteration, you know. Usually takes uh five, six takes uh on lighting renders to to really get things dialed in, but if you don't have to wait for an overnight render and to see in the theater, if you can make a tweak and just see what you're going to get,

I think you can get there a lot quicker. Economics are huge in this business. Anything that allows you to do very high quality work and put fewer man hours into it, that's Um I had a question specifically about just motion blur. I'm curious why you decided to do that in engine rather than kicking that to comp as well, where it seems like you might be able to get a little bit smoother result rather than just kicking it out 16 times in Unreal. Uh that was specifically because that's the way it is rendered at ILM. That the compositing setup for these shots, like the compositor who's working on these shots is also compositing a ton of other shots that did not come from ADG. And so he's used to renders coming to him with motion blur in them. And so we were we stuck with that. So rather than rendering out like a vector pass or something like that, you just figured that would be the way to do it. Well, also um that would require like in order to have a motion blur technique that we could render out, um we would have to in- invent one that is as high quality as what um you can do with the the temporal and spatial

sub-samples in the Alembic. I just figured for those three specific shots or something like that where the motion is rather linear, you're not dealing with anything any crazy motions. Yeah, technically we probably could have rendered without blur and done a done a 2D vector thing. Um but you know, I wasn't wasn't trying to give them an easy way out on this. And you know, we we have sometimes resorted to doing uh motion vector renders uh uh you know, to save on render time or you know, to reduce noise. Um and it can look good in plenty of cases, but it's sort of technically wrong. And there are always pathological cases where it just doesn't work at all, like uh spinning propellers and things like that. Um so I think in general that the preference the culture at ILM is to try and get it more technically right and just let it grind a little bit longer on the farm. Got you. Thanks. Hi. Um so obviously you had to pick and

choose your battles here um and there are some compromises you had to be willing to make um to get you know good enough but still run close enough to real time that it would beat your your requirements but um what do you what do you see as like uh the prospects for being able to start handling more more broad cases like not just as restricted as you're kind of currently having to be? So so the the work that that Natty and his group are are um doing right now will um lift a lot of the restrictions on the types of assets that we would even consider tackling at all. Um you know, having a full version of the unified shader implemented in real time uh there will no longer be uh a reason to pick a certain asset over another. Um most of the compromise that we made from the what was real time and then what was a minute of frame was all about aliasing. Um so for for the lighting artist

working in in real time at his desk, it is it's an aliased image but it's a real time image. And so uh if we can consume the geometry caches from ILM and we can shade them with the shading model that ILM uses uh and it's a slightly aliased image then I think uh we'll be in business. So you think you're on a road map to be able to handle non-hard surfaces even? Well, road map is a pretty abstract uh description of that. Yeah, more arbitrary hard surfaces is a good first one. Um once you're talking about things with uh subsurface scattering, it's definitely something that we've been looking into but that is obviously a trickier one to do at sort of ILM visual standards. Right. Yeah, I think just getting support for the second specular lobe and and for the refraction and the reflection model, I think you know, opens things up pretty considerably. Yeah, the second specular lobe we already have in in our in our latest work and refraction we've definitely been investigating that as

well. Really good looking soft area lights, too. Mhm. Yeah. Yep. Yep. We have some good ideas for that. Who benefits from from this uh our real-time rendering? Is it just somebody who sees gate two and he can look at it or does the compositor have everything composited and edit it in real life and see how things change? Well, in terms of the the pipeline This I would be happy to hear. Yeah, I somebody who's who's done a lot of sort of offline software rendering um you know, I've suffered through the the bad old days where you would tweak a parameter and you'd you'd dispatch a test and you'd you know, wait 15 minutes to see a little postage stamp version to see and it's it's uh you know, when your iteration times are fairly long like that uh

you know, there's only so much finesse you can afford to put into a shot before you basically run out of out of time. And so, the faster you can get the iteration loop then the better you can make something look. You can kind of dial something to be just so and you can do much more high quality finished work you know, on on take one. So, you know, the goal here is to reduce the the number of takes that are required and how long an artist has to work to to get a satisfactory result. And letting letting an artist to get more work done means there's in total fewer man hours that go into a given project. So, that makes the the work less expensive to do. You know, there's less of a chance that it all goes to third world countries. Yeah, to piggyback off of what John's saying, too, uh the process was very WYSIWYG, like what you see is what you get. It's right there. It's fluid. You can light what you see is what you get at the end. Like

there is no iteration in between. Uh the only effect that you're not getting is like proper anti-aliasing, uh the motion blur, but the rest of the shot, if you have a backplate behind the thing, it you can light to that backplate and get your final result in viewport. Uh so, that's also the benefit. That's the primary benefit of this tool, right? Um I could have like Vic shots, for instance, at my desk saying, "I need another point light over here. I need some extra spec on a shoulder." And I can just drop that in there and look at it, and he can approve it, and I can say, "Okay, now I send it off and render out the samples." So, that that benefit for the artist flows upward all the way. I mean, John benefits from this getting more useful takes in dailies every day. I'll say I've shot a lot of live action, too, both, you know, live action things and miniatures. And I'm also very used to a kind of different workflow. When you're on set and you're lighting something, you see how the how everything responds in real time. And

you'll have somebody kind of move in a light and you watch how the shadows fall on a character and you sort of slide things around until oh, yeah, that's that's feeling about right. And, you know, much slower, softer renders, it's it's a it's a very different workflow. You kind of take a guess at it and you And having a tool where you can drag things around in real time is sort of replicating a bit of that that onset experience. And I think that changes a little bit how you think about lighting. Thanks. So, for GPUs, I'm curious. Did you guys use the consumer 980 or Titan, or did you go the workstation Quadro type route, or something else altogether? So, we were using Yeah, we used Quadros. Okay, cool. Hey. So, you have added UDIM support to Unreal? Oh. Uh we we added Material X support to Unreal, which kind of brings UDIMs with it. Yeah. Okay. Cool. Well,

thanks everybody. All right, thank you. Thank you.