A Living Painting: The Rendering and Art of 11-11 Memories Retold

Right. Uh hi everyone. My name is Alexander and I'm here to talk about the rendering of Memories to Tell or how to be Bob Ross with shaders, I guess. Before we start, um I'd like you to cancel uh to silence your cell phones and to remember to uh fill out the evaluation forms after the talk. So, before we start talking about the Memories to Tell, I I'm going to talk a little bit about myself. Uh I have feel very weird giving a graphics talk because I am actually quite sight impaired. I only have 16% eyesight. Um it's specifically because of this gene right here. And I know that because they actually used the genes uh sample from me when I was 1 as part of the human genome project. So, um I guess you're all welcome for that scientific discovery.

Um And you might be wondering, why do I end up making games which is such a visual medium instead of like continuing on this very promising uh career in genetics I I had started too early. And it is kind of ironic, but for me I've always been attracted to video games because they're these constructed environments that are meant to be traversed and be discovered and read at a glance. When um a good level designer makes a level, they will use something called a squint test where they will blur their vision and see if the level is still legible. I can't stop squint test. I do it all the time and let me tell you, reality doesn't hold up to it. So, having um games where the environment has actually been designed to be legible, um I it's not so weird I ended up doing this. More specifically with

um painty aesthetics and something that's not trying to be photorealistic, quite often it's a lot easier to read because you artists are thinking much more about like the the base shape of the objects and you don't have like um all the like noise of the real uh life coming in the way. So, for example, playing a game like uh The Return of the Obra Dinn was something I really enjoyed because everything had clear uh outlines and um was easy to to tell apart. So, when I got the chance to work on 11-11, I I jumped at it. Um it's a narrative game set in the First World War and it tells the story from uh both uh the German and uh the Allied side. It's a co-production between Aardman Animations, uh Digixart, and it's published by Bandai Namco. Team size was around 30 and the team was distributed all around the world. I

think that's like a talk in itself how we we managed to do that. Uh we used Unity uh 17.3 uh with a modified uh legacy deferred renderer. And development time was approximately um 3 months of uh Oh, sorry. 6 months of pre-production and 10 months of uh of full production. And um the game was released in November uh to commemorate the centennial of the armistice and hence the name. Um as you can see, we kind of have the release date in the name of the game, so we could not afford any delays. Uh so, that was uh very interesting experience, but we managed to do it in the end. Oh, in case you haven't seen the game, I'm going to show you the trailer so you can get a sense of how this uh painty art style looks. See if we can zoom in here.

The first time you see someone die, everything shuts down. You're left with thoughts that go round and round. Every man has his demons. I did what any father would do. And I wouldn't make the same choices again. I just did what we all did.

I followed orders. Yeah, that music always gets me. All right. So, here is the outline of uh the talk. I'm first going to give you an overview of how the painterly uh render pipeline And I'm then going to go over some challenges and smashy features that is in it. After that, I'm going to show you how the artist use the pipeline. And following that is optimization of it.

And then at the end, I'm going to have a very short section on the future improvements and at the end of this will also be some time for questions. So, um all these interstitials you're going to see like throughout the talk actually like the concept arts we were like working towards. So, that should give you an idea of like where we where we were at the start and where we ended up. So, whenever I start on a new project and I have to develop new new tech for it, I look at existing white papers. Specifically when you're doing graphics programming, if you find papers that are kind of old, it means that they're quite they're quite likely to be um something you can actually achieve like for a commercial project now because you're also going to make sure that you have enough head room for the rest of the rendering taking place. Thankfully, I found these two papers that are both using particles to approximate the brush strokes so you get

this painterly look. And oh Sorry about that. There we go. Okay, there we go. So, what you see here is two papers that are uh one is completely object-based and the other one is working full screen. Especially the one on the right showed us that this is actually an an approach that could work. And the way that it works in the game is we have the base frame the base frame buffer and on top of that we paint all these particles and at the end they are combined together. The first um innovation we did was just painting these particles on top of the the frame buffer already meant that it looked um a lot better off than what had been done before because we could like close all the the gaps in the particles.

And um uh It kind of resembles how if you're using a camera obscura, like you are you basically taking a picture and then you're painting the strokes on top of it. On the technical side, all of this is based on uh a technique developed by Garrett Thomas from AMD. Um it's all based on compute and it's a tile-based approach. I will only do a very quick overview of how it works because like he took a whole talk explaining more in more detail how each step is put together. Um but his slides are readily available on the vault if um you need to check them up and there's also a Git repository. Right. So, this is how a frame is constructed. We first have the regular frame buffer rendering and together with that we also render a specific brush buffer that contains extra information for the

effect to place the brush strokes. After that, uh we do bloom and ambient occlusion. And after that, we start the actual rendering. Sorry, my notes are I do this? No. Uh so, we generate new strokes. And after that, we update uh the strokes that are currently on the screen. Some of them are deleted and will die out. Um maybe I should do this. There we go. And then we end with um uh calling and coarse calling. Um all of this is basically to make the rendering of the strokes more efficient um at the end. And we also do blender blending with the background image as part of the brush stroke rendering. And when all of that is done, we have the color grading and

tone mapping. This is the different aspects that the artists have control over. And what you're seeing here is the different debug views that they have. So, they can control the size of the brush strokes, if the brush strokes should boil, and if they should flow. If they boil, they basically die out quicker, so you get something that kind of shimmers. It's really good for interactive objects in the game, something that you need to stand out. Since this game is about the First World War, a lot of the levels take place in the trenches, and they're like really wet and muddy, so we needed to show something that was like running with water. And that means that the artists can actually go in and say like I want this the strokes to flow along the surface. And they flow along the direction that directors that artists have also instructed the effect to to use. Something that is also quite unique is that we can actually select what brush stroke we should render with. So, that

way you can have something that approximates like a very dry brush being kind of like dapped onto the canvas, but you can also have something that's much more wet and like long longer. So, there's a lot of flexibility for the artists to use there. This is the format of the brush buffer. So, size, flow, speed, and angle each get a full eight bits because we need as much precision there as we can get. Um for brush type, I'm I'm using uh bits, and then I have uh two bits left for I'll do it over there. for boil speed. Um The reason for this layout was predominantly that we had to integrate this uh into the the third buffer, and uh gamma correction caused some of these values to be kind of wonky, so it had to be laid out this way. So, an early challenge was to make sure

that the brush strokes would actually be aligned with the objects, and that's both uh they have to respect the objects' own orientation as as you see with the magenta arrows. And they also had to um to work with uh perspective uh distortion. And it took me a while to figure this out, but what I ended up doing was that I would take uh both the current fragment's uh position in world space into the fragment shader along with um the world's tangent and cotangent. And that way I could generate an offset point and take both of them um into screen space and get a direction uh vector. And then take the get the radians from that and then store that uh as a normalized radian into the brush map. This all sounds kind of expensive, and yeah, we I was using um uh an A10 uh 24 for that, but it turned out to not be that big of an issue um as we're going to cover later in the optimization step. Okay.

Nope. There we go. So, when new strokes are created, it it happens in a kernel that takes the whole uh screen and for each pixel generates a random number. Then for um to modify this spawn chance, we look at the current uh uh brush size we want here and the flow speed along with the boil speed, and also the alpha coverage of the strokes that are already there. So, if we don't have a good amount of coverage, we want to create more brush strokes. And all of those numbers are added together, and if they're more than one, a brush stroke is created. And that is by taking a an index from an a consume buffer, and use that to index into all the uh data arrays um that contains all the information for the strokes. So, it's important that RNG is very uniform. Um to start with, I was using a noise texture, and I kept getting strokes planted on top of each other, which was a

both it didn't look good, and it was also not very performant. So, at the end, I used um Wang hash, uh which cap Cabbie Games covers in the Render Talk uh on below. It's very easy to implement, and it is producing some very nice noise. So, I can recommend using that. So, after the strokes are created, we move on to updating all the strokes that are already on the screen. The first thing is that the position is updated, and then we get the depth and color from the G-buffer. We also get the brush type, flow speed, boil speed, and size from the brush map buffer. And if the surface is flowing or boiling, the stroke will die out faster in this um in when we're applying the the lifetime to the stroke. Um a stroke is removed, that is, its index is added back to the consume buffer from before, um if its alpha hits uh zero. So,

in order to make the effect not look uh like you're watching it through uh as a screen door, it's quite important that it moves with the scene underneath. So, what you're seeing here is two um is the same scene, but where on one hand, they are they are they are completely static, they don't follow the scene. On the right side, they are using motion vectors to move along, and especially if you're looking at at the door frame, you can clearly see that there's a lot of flickering going on. Um if you don't do this. Thankfully, in Unity, you can just say I want motion vectors, and you get them, and they turn out to be of a pretty good quality. Next step is coarse culling. This happens with a box-box intersection, and testing revealed that 16 by 8 culling tiles was the most efficient number. When a um collection a con- a collision is detected, um the strokes

index is added to that tiles index buffer. And a very similar approach is used for the next step, the fine culling. Uh here each tile is 32 by 32 pixels. It's again a box-box intersection testing, but when we're done with that, we also do a step of bitonic sorting based on the depth of the particles, um to get correct alpha blending later on. And the bitonic sort is the same that Garrett Thomas used in his talk. So, the render kernel is what really is the most different from from Garrett's approach. It's um it starts by taking um uh loading by loading all the stroke data into LDS, um so it's faster to retrieve. We then set the accumulated color to to zero, and we then, for each thread, uh takes care of of one pixel.

We loop through the particles, and it's essentially a ray cast through them. So, if the ray hits the stroke, we also we sample its um its uh stroke texture, and combines it with the stroke's color, and add it to the accumulated color. By using a custom alpha blend, we can actually go front to back instead of back to front. We didn't see much of an improvement in Gareth Thomas's talk. He see he sees a huge performance boost from this, but it's mainly just because he's actually doing particles in uh world space and he shows that you can go very close to them. Um in the end I just left it in because it wasn't hurting performance either. All of this is rendered into an HDR ready uh back buffer uh so we avoid banding. And the alpha value stored here is what can um is what is used in the generation kernel to determine how much coverage we have. So,

uh strokes are stored in an alpha 8 uh 2K texture. Um we found out that distinctive strokes are actually way more important uh than like quite diffuse and soft. I personally thought that would that was actually going to work better. Um and turned out I was completely wrong. And different sprites give um give impression of different surfaces. So, for example, for wood we ended up using quite long and hard strokes and then for the cat in the game, we had some that were like much more like hairy and and and bristly. All right. So, let's cover some snazzy features. Early on, the biggest issue we had with um with the effect was readability. Which shouldn't really come as a surprise, I guess. Uh what you're seeing here is actually a screenshot from uh the vertical slice a couple of uh months into production and it this was a bit of a crisis moment because if we couldn't get this to work, we

would have to abandon the the the pencil effect and do something completely different. You might not be able to see it, but there's actually a man crouching on the ground. Um so, this amount of of yeah, blurriness that just wouldn't work. For the um for the vertical slice, what we ended up doing was just increasing the amount of particles so they would approximate the shape of the scene more. But for the like full release, this didn't look that great and it also wasn't performing so we had to find out a different way to do it. So, the reason why this happens is that we're we're painting all the strokes on top of the background image. And that basically means that if you have a stroke that is like from the background, it's still going to cover like something that is in the foreground such as Kurt's arm here that's being obscured by the particles from the seat behind him.

So, the trick was to sample the depth buffer in the render camera so we know what is the absolute depth we should traverse through when we go through all our particles. So, we get like all the particles that are belonging to Kurt's arm, but we stop when we know that the particles that are behind behind him. Um yeah, would would just lead to like covering up the uh what's in the foreground. So, this works, but you see that we get these like quite hard edges on objects which is not paintly at all. And in order to fix that we end up blurring the background with a Kawase filter. And the reason why I ended up using this is because it's quite performant. It takes a little bit of like manual tweaking. So, what you do is at each step you sample out in a cross of four samples and if you use bilinear sampling, you can end up with double the

amount. Um so it's quite easy to to use. It does require a little bit of like manual tweaking, so you don't get like the banding artifacts uh you might be able to see in the projector. Um but yeah, I recommend using it. Something that was also important was that objects would have the correct uh size of strokes attached to them. So, if something recedes into the background and it's important that it's still legible, we need the strokes on them to be smaller. And that was quite easy by just uh taking the depth of um uh in the in the fragment uh shader and um use that to increase the brush size. Interestingly enough, we actually ended up having this something that the artist could uh control with a modifier. And they would sometimes use it in reverse. So, if you have something where you want to guide the the uh the eye of the player, this is like a technique that's also

used in in regular uh painting where you will use detail where you want the the viewer to look. And this meant that we could actually like I I guess get something kind of similar in an interactive environment, which I think it's pretty cool. It also had to take in the camera's uh field of view. So, in in the game, one of the characters has a a camera um and can take pictures. And in order to have zooming, we change the camera's FOV and it's not really because you have like a different distance to the object, but like this they can still be like smaller or bigger on the screen. And I fixed that by just having um a curve for the artist to to tweak where this would be added in based on the current FOV of the camera. Early on, it was all about getting as many strokes on the screen as possible. Like we thought that like this is how we're going to make it look look really good. Like so, we had

around 2 million strokes at the end of the vertical slice. But, sort of like halfway through development of we started to realize actually it's better if there's fewer because they look quite artificial if they're like placed like right next to each other. Um it looks like this like manic robot has been like ah I need to to paint this as good as possible. So, for longer strokes because they're all they have will have the same height as uh short strokes, they will cover a larger area. So, we need fewer of them and I basically just took that into account when generating the the random probability of them spawning. And it had this had the the added benefit that it looked better, but we would also get better performance because there was fewer brush strokes on the screen. So, for example here if you look at the image to the right, uh the beams look a lot more paintily and and human-made um because there's fewer strokes on them.

The second hardest problem to solve was flickering. So, the strokes kind of act as a as a downsampling of the the base frame buffer. And this this causes the the scene to be uh become like very noisy um when you're moving the camera around and when objects are moving. And the solution to this was to uh sample the color of the brush stroke in a cross so we wouldn't just use the the the center of the stroke, but we would also look around it. Um here I would actually use the the brush type as well to basically do a bit of rejection sampling. So, you kind of want the stroke to be colored by only the object that it's supposed to represent, but we don't have that data available, but there's a good chance that it's it's only going to be the object itself that has the exact same brush type. Um so, it's like something in the in the background is likely going to be be using something else. So, that was like a cheap uh and quick way to

uh to kind of fake that. We also found out that the motion vectors from Unity are a little bit uh like noisy. So, even when nothing is moving, there would be a little bit of of like drift. So, by applying a simple threshold, that also took care of that. So, the camera would have to actually be in motion, an object would have to be in motion before uh you would start to have the stroke changing shape. Oh, sorry, changing color. So, in the real world, when you're looking at a painting, you can actually pick out the individual brush strokes. That's because the each stroke will catch the light a little bit differently because of the inherent impasto. Of course, we can't actually do um like do that um because we don't know the lighting conditions of uh each player's uh room that they're playing in. But, we can approximate it. And that's simply by

randomizing the color, and also by randomizing the angle, we actually get something that looks a lot more like hand-crafted. And it's also kind of like part of the the pointillism um which is part of of impressionism that in inspired this whole art style to begin with. So, the way that this randomization is done is um I tried to first use uh HSB, but I got a lot of uh like flickery and and edge cases where where it wouldn't work. So, in the end, I had to come up with something where because every all the other colors was in RGB, like the randomization of them also had to be in RGB. And by looking at the color space, uh like the RGB color cube, you You actually see that you if you look at it from the right direction, it kind of resembles HSV anyways. So, if you take a vector that goes from from like pure black to pure white and

use that to form an orthonormal uh set of basis vectors, you can actually use those to add color randomization. It's by no means perfect, but it's good enough for like the small amount that that we needed and it turned to also be much more efficient because we didn't have to do the conversion to HSV and back again. So, we had to keep the randomization that was happening to each stroke constant from frame to frame, otherwise the stroke would keep changing color. And the only constant set of information I had for the stroke was actually its index. So, I wrote this function you see here where I get a a like a pseudo randomized like value out that is then used to apply all of um these offsets. So, this is some of my my favorite features. We can actually

um handle particles in two different ways. Um so, one is just using an alpha cutout standard particle shader that writes into the brush buffer. The good thing about this is that it's a workflow that artists are familiar with and it also allows the strokes to be lit. So, you can see we're using that for fire quite a lot. We can also inject Shuriken particles in this particle system directly into the effect, so they appear as brush strokes. And I thought personally we were going to use this everywhere because it's damn cool, but unfortunately um in the Unity's uh deferred renderer, you can't actually access the lighting data. Um so, they they required a lot of like artist tweaking in order to sit well with the scene. So, in then we only used them in like very like rare circumstances and most of the time we ended up using the alpha cut out particles instead. They were also kind of a pain in the butt to implement because there was a

lot of unique rules for them because they wouldn't get their color from the scene. So, for example, having them work with fog meant that I had to take that equation and actually put it into the injection code all like the injection compute panel kernel. And also um uh it's why we had to split up our post-processing so some of it would happen before the pencil effect and some of it would happen afterwards. Uh which incurred a performance cost. Um Yeah. So, good idea. Could have um been better if we had uh uh more data available. All right. So, how many of you are graphics programmers? Fair amount. Okay, how many of you are artists? Also a fair amount. Okay. Um if you've ever been part of a production, maybe you can recognize this.

I will say at times it was kind of stressful. Um So, we thankfully had some pre-production on the game and I think that is what really uh allowed us to achieve the look we had in the end. Um At the start it was only me and the art director so I had like all the like I could focus completely on on just trying out the various things and get something he was happy with. Um But I also think it's like a lot of uh the reason why we ended up with something that worked is that uh I was constantly in dialogue with the artist about like well, how should we get this pipeline uh to do the things that uh that you guys need it to do? Um I would also say the artists were like really nimble to adapt to how it worked because it's it as you're going to see now, it's a quite weird workflow and it was definitely something that none of us had tried to do before. So, the artists have three levels of

control. They have uh globally on the camera um where they can adjust uh uh variables such as the maximum and the minimum brush size and how long the brush stroke should live and so on. Then they can also go in and adjust it on a per material basis um with overrides for brush type, um brush size, and so on. And finally, there's also on the texture level a texture that in format resembles very much what's in the the brush buffer. Um where they can go uh and like have like the maximum amount of control. So, we used Substance for most of our texturing work and it has a lot of um advantages. You can reuse graphs. So, if you create like a base wood material, you can re- uh use that to create a lot of variations very quickly. Um it also allows live editing and I think this is

what was actually the most beneficial for us because like I I was the only graphics programmer on this project, so I didn't have time to do some kind of like custom solutions. So, they could actually see how the models would look like in the 3D modeling tool, so they always had to take them into Unity to see the end result. And if we hadn't had this like like like um live editing, it would have taken so much longer. And I actually have an example of that. Ooh. There we go. So, what you see here is uh a Substance material being edited in real time and you can see that getting all these variables of like what directions that the strokes going in and how big are they and and like how much boil do we have on them like not having these this exposed so you could like play around with it would have taken a lot longer time to find settings that that works. There was also a lot of problems with

them. This is mainly due to the the version of substance integration that we had to work with. We had very long bake times and it often broke on console builds. It would also sometimes like crash the Unity editor when we imported them. And it it actually and then we had to um like our tools program had to spend a lot of time on developing this tool that would allow us to switch between substances and baked out materials for this um for to allow us to actually ship the game so thankfully there's like a new integration on its way so hopefully that should completely remove all of these problems. There was also like some um guidelines for creating good textures. If we did something that was approaching real life you have something that's very noisy and that noise causes

the brush strokes to change color very quickly so you get something that flickers. But on the other hand if you just blur all the textures you end up with something that doesn't really have any definition and the image ends up looking really flat. So having something where you get this kind of like posterized look turned out to be the best. And having a bit of like smooth gradients also helped. We would also exaggerate the main form slightly so they would read better underneath the effect. And this is kind of like the same principles that would be used for modeling as well. Assets came in two flavors. We would have hero assets such as characters and assets that were that were used often. They would be hand-painted in Substance Painter, whereas the rest would use substances from Substance Designer. And the good thing about those is the only thing we we needed to do was to do a

correct UV unwrap and the model would be done. So, for characters and other hero assets, this is the the process. We would do the same texture method that you saw earlier, but we would also paint in extra highlights and shadows into the albedo map to kind of make the the characters pop even more under the beneath below the effect. So, painting the like hero assets, like as you saw earlier, the format for the for the brush map buffer is that's not artist-friendly. So, in Substance Painter, the artist would paint into these like custom-defined channels that would control that each aspect of of the brush strokes. And when

we then imported it into Unity, a custom editor script would then take all of that information and crunch it down into the texture format that the pencil effect was expecting. Specifically for characters, we found that we actually needed them to pop a lot more. So, we wrote this uh kind of like highlight shader. So, it is it is a it is a a normal rim light, but it actually takes the lighting in the scene into account so it's just so it so the character sits more with the scene, but you still get like um them standing out more from uh from the background and objects around them. Which worked really well. All right, optimizations. And I'm I'm doing I'm doing well for time. That's good. Okay. So, um

this was the first project where I was a graphics programmer. And I think for more experienced graphics programmers, a lot of this advice is going to see kind of like basic um but in my experience, it's actually quite a hard to uh find uh this stuff online and find tutorials on it. Um so, I hope it's it's going to be beneficial for for people out there that there's a place where they can actually um learn about this. So, performance target-wise, we looked at um GCN as our main architecture to optimize for because it's the one used on PlayStation 4 and Xbox. We targeted uh full HP on um on PlayStation 4 and 90p on Xbox. And yeah, I would advise you don't give everyone on the team uh a 1080 graphics card because you're sitting there and I was I was sitting there as the only graphics pro- programmer being like, "Oh, we should probably start optimizing." And um yeah, uh um it's we had to to get um

about halfway through development before we started to really um take this into into account. Um So, for optimization and for profiling, I used GPU Razer quite a lot. There's other tools out there, but I really like the way that uh it's presented. Um I think Sony does a really good job with their tools. Um They're easy to read. And not surprisingly, we found out that the main thing that had to be optimized was the render kernel because in a standard HD resolution, you have around 2 million particles or two sorry, 2 million pixels. And for a given frame, we would have around 30,000 strokes. So, the kernel creating the strokes and updating the strokes, they were quite like really cheap. That wasn't the worry. And it also meant that I didn't have to worry about putting in new features into that part of the pipeline that much. But anything that affected the rendering had to be

considered quite carefully. So, something that's quite easy to stumble into when you're using Unity is that you're going to coupling your render thread to the main thread. And what this means is that you're not going to be able to have your render thread like process the last frame while while you are doing the logic for the current one. And it's a lot of it this is not really that well documented at the moment. For example, if you um use any of the methods in the graphics class, you're likely going to cause um uh going to cause this coupling. And the solutions to that is actually to use command buffers, which is not really explained in in the documentation either. It's presented as a way to extend the graphics pipeline, which is true. But if you use them, you're also making sure that all the the commands are executed on the render

thread. So, scriptable render pipelines was released all I like was starting to get previewed during the production of 11-11. And I wish we could have switched over to it, but I mean it was still an experimental feature. Um so originally um we were generating the brush map buffer with replacement shaders. And they are terrible. They basically causes the the camera to call again. And you end up with uh with more draw calls, and you also end up causing coupling with the main thread. So with scriptable render pipelines it would likely have been easier to fix this because it would have like all the calling results readily available. But I was stuck with the deferred renderer, and I was also stuck with a non-source code version of Unity. So there was yeah, I had to be quite creative with

with how I went about solving this. So what I did is that I stole a buffer in in the G buffer. So normally Unity stores the uh specular value completely separate, but that's actually only constructed from a specular value or a specularity value um and the albedo color. So by using an unused channel to store the specularity, I freed up a whole um a whole buffer where I could store all all the information that I needed. It did mean that all the lighting equations got like more expensive because that color just wasn't readily available. You had to like recreate it for every light. But in the end in the end we didn't have that many lights in the game anyways because we were like concerned about having as much performance reserved for the painterly effect anyways. So this turned out to not be an issue. Um but it did mean that all the deferred lighting shaders had to be changed. Thankfully, that's something you can actually access in in Unity um quite

easily as you can see here. So turned out to be easy. What was not so easy is that Unity has a system for um generating shaders that work with both uh the deferred and forward render pipeline and all the lighting that goes on there. But, of course, they would be expecting the normal deferred color layout or light buffer layout. So, in order to fix that, I wrote this like kind of dumb find and replace tool um so I could go through the auto like the the generated uh shader code and replace it. But, it was not a great workflow because I had to hit a button on um on the shader. You can't actually access the the generated code um like through scripting. So, you would I would always have to go in and uh hit a button to show the generated code, copy it over into a separate file, and then run the patching system on that

um separate file. So, yeah, was wasn't wasn't fun. Some objects would be forward rendered. So, if we have something like a a special effect shader that were used in in certain sequences, they have their own like lighting going on. So, they can't be rendered as part of the regular deferred pass. And this also meant that they would have to be rendered into the brush buffer separately. Um the way that we would do this is that the artist would add a script to the object and instruct what the uh materials should be should be used or should be should be rendered that way. And the effect would then generate a separate command buffer that that would run after um the deferred pass was done. And that way we would get that information into the brush buffer. This also meant that I had to inject a

special shader pass into into the shaders, which was also taken care of by the patching system you just saw. All right, draw calls. So, um most games I've been on draw calls in Unity is is like something you really have to look out for. And there are more and more tricks you can use to reduce them. So, a fairly new thing is that you can use layer call distances on the camera to uh basically say, "Well, if you know that something is is quite small, you can use you can call it out at a shorter distance than the the far clip plane of the camera." Um we also combine meshes when necessary. And Unity's static batching is is quite temperamental, so it also took our artists a lot of diligence and experimentation to make sure that we had everything batching as much as possible. So, occlusion calling is also something

that sometimes led to like better performance, sometimes not. So, we would test it on a level-by-level basis if we we should have that enabled. So, data packing is um a pretty pretty cool technique. Um what this allows is that you can fit uh more more information into less space. In our case, we used it for uh fitting more strokes into into LDS, and we also reduced the amount of uh VRAM that the effect required, which was basically freed up like texture space for the rest of of the rendering. So, it can also increase increase performance because you have less data that has to be loaded from VRAM. So, there's like less uh data that has to be transferred over. It does increase ALU pressure, so it's easy to make sure that you actually test to

see if this is actually going to like, be more performant than not having it in. The good thing is that it's very easy to implement because you're only looking at the the spots in your code where you are reading and writing your data out. So, it's something that's not going to have, like, repercussions for the rest of your logic. And HLSL supplies F16 to F32 and reverse that allows you to to do this packing of two floats into one float. Um But, of course, the lack of precision can be an issue. So, there was like some of some of the information, such as the position, where we just couldn't use this. For optimizing ALU and VGPR um usage, I used these two talks from Emil Persson, aka Humus. Um they're really good. It's like super concrete and easy to to use advice. And in our specific case, using ISP, which

is an approximation of of division, turned out to be really beneficial in a lot of cases. Uh sin cos is also um a lot more efficient than doing sin and and cos separately. All of these are HLSL intrinsic functions that are readily available. Um I also found that pre-computing part of the intersection math um and then have that available in a constant buffer led to a nice speed up. And we would also pre-compute the aspect and inverse aspect ratio um for use in in other parts of the pipeline. And this is the aspect ratio of the the brush strokes. So, a nice feature in Unity is that you can actually see the generated byte byte code by just uh uh finding the shader in in the editor and clicking this button. And what that allows you to do is to compare if

changes to your shader is actually going to change the amount of of instructions that that you generate, which when it takes like maybe 20 minutes to do a build, this can very quickly add up to a lot of time time gained. Something you can also do on some platforms is to use platform specific defines to uh basically nudge the the compiler to to optimize your shader in a different ways. Um it's not a it's not available on on on everything, but uh uh yeah. Um for the consoles that can be be a benefit as well. So, when you're starting out learning compute shaders, quite often people will use structs. Um and it's it's that terrible. Um

it's way better to use a structure of arrays instead of an array of structs. And the reason for that is that um you get better cache currency, and each kernel can load in the data it it needs itself. Like for example, for all the calling that we do, we don't need the color of the of the brush strokes, so there's no reason to have that sticking around. Um and I I this is a guess, and maybe someone in the audience will know this, but I also think that the compiler can rearrange the code so ALU instructions can happen while memory fetches are are also being done. And if it's done as as like one big fetch of of a struct, I think that's a lot harder for the compiler to um to schedule. So, what you're seeing here is the debug view that I use the most and this is the content of each calling tile. So, if you see a

pixel in there, that is because a brush stroke has been determined to be inside that that calling tile. And something that worked really well in terms of optimizing the effect without like a visual loss of quality was to actually look at how many strokes are inside the calling tile and then feed that back into the generation kernel. So, if if you the tile was starting to fill up, the probability of new strokes spawning would be reduced. So, what's going on with all the green and blue colors? Well, it's quite often that a stroke will not cover all the the pixels in a 32 by 32 tile. So, that basically means that if um that you you you end up like with a a lot of the threads not doing anything. And on GCN, 8 by 8 pixels is actually

the optimal size for um for a work group. But, that puts a lot of limitations on how much you can actually fit into your local data share. Yeah. So, you might think like, "Okay, can we reduce the size of the calling tiles?" And unfortunately, testing showed that if um if they're smaller than 32 by 32 pixels, they started being being slower. So, what I did instead is that I would keep the calling the fine calling at 32 pixels, but I would then actually run um three different render kernels after each other, each optimized for a different amount of strokes in that calling tile. And I did this first as like a quick test to see if the approach would work

out. I thought I would have to do some kind of like quick clever scheduling, but it turns out it was actually like really really fast. So, what happens is that the the render kernel that's optimized for 8 by 8 pixels will look at how how many strokes are in the tile and if it's not fitting the amount that it can fit into LDS, it will early out. And because all the threads in the work group are early out, it's it's very quick to do this. So, on average um executing a render kernel that wouldn't actually affect anything in the image would only take 90 microseconds. And I thought this was going to be like a golden like I guess it's a silver bullet Yeah, silver bullets to performance, but it turns out it was only 0.7 ms gain we had from it, but that's still still nice.

Right. So, now I'm going to talk a bit about future improvements and a conclusion. So, one drawback of using substances is that they wouldn't be they wouldn't take the topology of an object into account. So, for example, these like floorboards you see here, you would actually need smaller strokes on the thinner sides of the boards than on the rest of them. And in order to allow the artist to fix this, I actually had time to put in vertex painting of uh of brush sizes uh into the the whole pipeline, but it came online so late, so we didn't end up uh using it in the end. Unity is doing a lot of amazing things at the moment. Um scriptable render pipelines uh is uh one of them.

And if we had uh had had access to that, it would have helped with a lot of issues. Um We could have had uh lit uh injected strokes because uh in their uh high definition render pipeline, all of that information is readily available. I guess it's will be available in the lightweight one as well. And the ability to create and bind buffers uh more freely um would have meant that I would have a lot more like flexibility in what I would have been able to do. Um Something that we saw at the end of the project is because the the brush uh stroke uh the brush sizes are in an in a single 8-bit um channel, that means that it's a normalized value. And if we needed to to go in and change the uh like make make some big strokes, for example, in one scene, we would have to do that on the global level, which meant that all the materials had to be uh had to be tweaked because it was like all coupled together. So, in the future, I

would store all of this in a 16-bit um uh half precision value instead, so you don't have that coupling between all the brush sizes. Um that would have saved our lead artist uh a lot of time uh in the end. So, nowadays, you have to make sure that your game is easy to share. Uh and we found that YouTube compression was very tough on the effect. It took us a couple of tries uh in recording uh footage for for trailers before we really got a got a hang of it. Um and it's it's it's such a shame, right? Because when people actually see the game in in real life, they'll they they can see the the texture that you get from having uh uh the the brush strokes there, whereas a lot of it is kind of like washed out if you're watching it on a phone. Um

That said, I'm I'm really proud of what we achieved. So, what you're seeing here is uh a piece of concept art from early in development. And this is the scene um that you see at the start of the game. Especially like the way that the like lamp posts are like having this like nice halo around them and how um dreamy looking the the background is. Um I think that's really nice. And you can also see it here with like other other scenes in the game like having this sense of a um of a not that defined background like rushing past you, for example, when you're playing with the pigeon is uh is something that I'm yeah, I think looks really beautiful. So, here at the end of the talk, I would also like to thank uh a bunch of different people. I got a lot of uh mentoring uh throughout the project from a bunch of very talented graphics programmers.

Um we also got a lot of help from the Unity spotlight team with the figuring out some of the various ways to uh tweak the deferred pipeline to do what we what we needed it to do. Um and I would also like to thank Cody, the lead artist on the game, for helping out with the preparing uh the um some of the slides in in this talk. And lastly, I would also like to thank Adnan and um Bandai Namco for allowing me to uh give this talk in uh in the end. So, I would also like to thank the rest of the team that I worked with at Autumn um I think everyone was was doing a really good job, but like maybe even more importantly there was it's also one of the teams I've been on where we had the best uh social interactions. Like everyone was jelling really well and it's it meant that it was just a a pleasure to

to work on this game. Um yeah, it's it's one of the best projects I've been involved in. All right. So, to conclude on all all of this, pre-production is really a must if you're trying to do something that is um as weird as as we tried to uh achieve. Um And it's important that the technology has a head start uh before the rest of production uh ramps up. Also, talk to your artists and make sure that what you're developing is actually uh what they need. Um All the disciplines have to to work together in order to uh to achieve the look that you want. Compute shaders have a huge potential for photo of like um for painting and rendering. And I can't wait to see what's going to be done with them in the future. And thankfully I would say um be a novice when you start because then there's a lot of uh easy things you can fix at the end of the the project to

increase your performance. All right. That's it. So, there's microphones uh set up if you want to ask me questions. Um I'll I can't actually see the microphone from here, but uh I guess if you form a line uh that'll work. And remember to say uh where you're from when you ask the question. Hey, that was a beautiful presentation. Really awesome work. Thank you. Um I have a quick question. How did the uh how did the artists author the flow directions? So, that was um That was by painting like regular flow maps in in Substance Painter and also having custom nodes for it when it was

in Substance Designer. I think if you meet us in the wrap-up room, Cody is actually here and he can he's sitting right there. He can he can answer more specifically how that worked. Thanks. Do we have any more? Yes. I asked about the the feature about because I look is look the static the static image, I think it's very beautiful. But what when the camera is moving and then we can figure out that the paint the stroke is like a little bit figuring because they are keep appearing and disappearing. So, is that the final effect that the artist expect? Or you try to more making more stable. you try and rephrase that? I mean for the dynamic scene, you will find that the stroke is keep appearing and

disappearing far away or something like that. Is this the final effect that the artist uh expected? Uh so, if I'm understanding you correctly, you're asking if artist ex- was expecting the effect to behave as it did. Yes, because you you create a game with some concept art that is just static, right? It's very beautiful and the static scene Oh, I see what you mean. So, in motion it was it like what we were trying to go for? Yeah. So, our art director actually made a kind of like a sizzle reel of things he wanted the effect to do early on and a lot of that was inspired by um I actually think I have a a slide for that, yeah. So, we would look at something like The Old Man and the Sea, which is a really beautiful oil painted animation and he had eight different things that he wanted to see in the pencil effect. Some of them was not really achievable, but

something like boil and like slurring like some of that stuff we we we kind of got in. So, I think early on like that dialogue with him is really what informed how the effect looked in the end. Um I think there was definitely a learning curve for the artist in order to figure out how to create the 3D art that would work because they couldn't like just look at it inside 3D Max. They would have to export it into the game engine and then fiddle around with it. But we we definitely saw like at the end of the production they had a lot easier time creating assets quickly because they had a huge library of substances to rely on and they just knew the whole effect much better. Okay, thank you. All right. Hello. Great talk. Thank you. Uh did you explore machine learning style transfer to perform this effect? Sorry, uh I think is the question to if I we used

machine learning? Did you explore the possibility of using style transfer? Um oh, do you mean the one No. Um we didn't use any machine learning for this. So, if we had used machine learning, the problem with that is that you you give up a lot of control for how easy it it is to use. And as you can see here, we had a lot of artist control over specific objects and the scene as a whole. So, if we had used like machine learning, I it I think it would we would have kind of like fought what that approach would have given us. So, I'm seeing some quite interesting approaches at GDC this year where they have like used it, but they also bill it as a way to prototype as a given look. And I think that it's going to be interesting to see if we're going to end up with like some kind of hybrid approach where we we will have like some parameters that are our control. But what I'm seeing seeing from using machine learning so far is that it

it is like it's kind of like a take it or leave it situation. You either use it or you don't. Okay, I see. Thank you. You're welcome. Uh so, you mentioned a lot of in the talk about troubles with Unity and all that. Um could you go into why Unity was chosen for your project over maybe Unreal or a custom engine? Yeah, that's a good question. So, I mean, I'm happy I'm a Unity freelancer, so I think it was predominantly because Artman has used it for like other problems other other problems other projects in the past. Um so, it's there was a lot of like institutional knowledge on how to use the engine. And I think for the size of team we were, it was also a fairly good fit. Um So, I I have only worked a little bit with Unreal, so this is kind of like

hearsay, but what I've heard is that if you have to modify their renderer, like you have to delve like you basically have to write in C++ and like tear out things and like put things back together whereas Unity is much more and I guess this is kind of like an engine design philosophy. They use um um uh they they approach it much more like a toolbox um and there's like you have to build more on top but it's kind of designed that way um and then there are limitations of how far you can go with that when you're using uh their current deferred uh or forward renderer but what they're doing now with the SRP and the whole dot system you're going to see that like it's go it's probably going to be a lot quicker to do all of these things. Um so yeah, that's that's why we ended up with Unity. I see. Thank you. I think we have one more. I think That's it. Okay.

All right. Well, uh thank you for coming and um remember to fill out the evaluations uh afterwards. All right. Enjoy what's left of Unity.