High Dynamic Range Color Grading and Display in Frostbite

[Music] my name is Alex fry I'm a rendering engine here at frostbite and today I'm here to take you on a bit of our journey on high dynamic range color grading and TV display support this is what I'll be covering today quite a lot to get through so I'll go pretty quickly this is really the last couple of years and it's our journey I'll go into a few details where I think it'll be helpful and hopefully there'll be some good discussion afterwards I'll start with some history terminology though because it will be helpful we'll start with film as many things do so particularly film stock and how it responds to light so it has three main parts here and the middle section which is where most of your data will be what you're exposed for this is characteristically the the straight-line region it's what it's called and this one's quickly to light but there are two

other parts the toe which is low down I think of the toe on your body and the shoulder which is high up I think of your shoulder on your body and these respond much more slowly to light so you get a much larger range of values compressed into a much fuller small amount of film stock so really this film stock will capture a very wide range of light values into a smaller range and the toe and shoulder or the range reduction you expose for the midpoint and this gives you a very nice this S shape gives you a very nice characteristic pleasant and very familiar look now in games fully real-time CG we don't have film to develop we have got limited range TVs and so we must do similar introduction and that's time wrapping that's the presence of mapping down this very wide range that we draw into something much smaller while preserving the important detail and filmic tone mapping does this by emulating the film characteristic curve and just a small point that when you see many HDR photos on phone cameras these are more accurately tone mapped

and I'm going to use this image quite a lot this is a demo level that we've got internally and it's got some quite useful parts it's got very good contrast it's got a very wide dynamic range has a lot of saturated colors and it's also suffering from problems which are so-called here particularly the light and the hologram you can see some serious issues with this now applying for different time mappers quickly I won't labor on this but the top to a simple time not there's the bottom to a filmic time up is you won't see a huge amount of difference here but from the bottom left perhaps we have got more contrast and color grading this is the act of applying a look to your game again it's got its background in film stock for example film development you might choose to skip a process such as the bleaching stage called bleach bypass and this will produce a different look a very silver high contrast look but digitally of course we can do a lot more than this we can do white balancing we can do color replacement color

enhancement push them back we can do orange in teal if we want and yeah we can do an awful lot more here we can fix a lot more in post and TVs so I'm sure you remember these old lovely fish balls and many people were actually still keep them around these have a particular narrow dynamic range around naught point 1 to 100 nits where a niche is another name for a Candela per meter square or unit of energy emitted per area and these screens have a nonlinear response to light at its input sorry this is called the EO TF or the electric optical transfer function also known as the gamma curve and a stander was introduced in the mid 90s to bring standardization to all of these TVs and our assets that are made for these TVs and that standard and the display capability were very closely aligned and we thought uses standard today 20 years later however modern TVs are far more capable than

this they will go as dark as 0.01 minutes and go above 300 Hz our TVs will go way way higher than this and this means that the hardware is far more capable than the standard that we're using and also LCDs respond differently to see our T's but we're still using 709 o srgb and so these TVs will take on our range signal and they will modify and they will expand it to best show off that TV and we have no control over this at all we can ask people to cow right not many well and it tends to end up with too much contrast and also too much saturation I'm sure you are familiar with the very oranjee skin tones of store demo modes okay now with that let's go for our legacy pipeline this is the simplification of what we did so we render our scene in faux HDR and perform all of our post-processing BlueMotion below etc in HDR as well then we would apply its own map convert to srgb apply a color grade draw the UI on

top and scan it out this is pretty common and I'd imagine you're familiar with this digging in a little bit and these two parts the time map is chosen by the artist from a fixed set of dropdowns usually do it be a filmic time map and these time maps are 1d curves which we apply independent of red green and blue with indeed the linear - srgb and we clip the results - not to one and that's because we then throw it into a lookup table to do color grading and this is also a very standard process different ways of doing it but we would typically load a screenshot load and identity color cube into Photoshop it's a layer apply various look transformations or whatever else crop off that layer save it out as a color cube and then index that by red green and blue at one time the new color would be what was ever in the lot and it's incredibly powerful you can do an awful lot of different things including color replacement so yeah we liked it there are some troubles with all of this though now the 1d curve when applied separately to red green and blue does

actually cause some huge shifts because we don't have a completely linear midsection and I've got some screenshots of this in a bit and also as you get near the top of the shoulder the highlights really do hue shift an awful lot especially as the channels clip and it's because you fundamentally change the ratios of light because when your red channel say slams into the clip point and you're over exposing it can't go any further but green can so you change the ratios and that changes the color we also use a filmic tone map typically and this applies a look as you saw from the the four point screenshots earlier and this look is characteristic but it's also in a different place to our grading workflow so there's two workflows going on too the same thing here it's also not easy for us at least in this particular code to author new 10 maps because it's not really they too driven only selectable and when you don't get the grading part we use Photoshop to do this but Photoshop is not really a grading package so we have a non optimal workflow for color grading it's a very powerful package but it's not really a grading package and our Lots were

typically 8 bits and this would cause irrecoverable quantization mac branding hue shifts and here's a very very early shot of faith' arm from Mirror's Edge and you can hopefully see some of the mic bands going down there and that all this was a bit of contrast also we have no access in this color grade to values greater than 1 because we clipped them earlier so performing any sort of exposure change or grading the highlights is really hard and also when you think about it the tone map combined with the srgb transformation are a lot distribution function from linear lights of grading space and that's great for viewing but it's not necessarily optimal for grading because you would get a particular distribution within the lookup table and that distribution would vary depending on choice of tone map which also means that if a game is produced a really great look they can't share it with another game was using a different tone map with this different function so it looks different also the legacy pipeline was totally hard-coded to STR and we wanted to support HDR TVs we want to get to get access to that higher bit depths at dynamic range that extra color and we

really want to they give the content graters controller for this TV take the creative control back from the TV manufacturers so we could actually use this this power ourselves so there's a few different ways to do this and the simplest approach and one that I would suspect that several people have already done is to just try and look at this tone map curve and see that isn't using range compression so why can't we just reverse the range compression and get the values back so ya get the shoulder undo the shoulder whatever the original values and profit sounds simple enough and it is simple enough but it has some trouble so firstly if you're using 8-bit then there's not enough precision to reverse that for a start if you've got multiple different tone map curves and they all need to be reversible when that's hard for some than others you could only recover a limited range because those might not be infinite compression they probably got a white point somewhere for example the first bright filmic time that clips at about 500 nits but really

the order of operations aren't correct and if you look at this diagram the yellow boxes are the Troublesome ones so we were due to tone map the srgb conversion and the grading in the UI remember when we reversed the tone map we also reverse the UI which wasn't tone mapped and we reversed the grading lot which wasn't too much so we've applied a transformation that's in the wrong place and this will cause problems as well they can't be made good enough absolutely yes it can be made good enough especially we've got a game out in the world and want to patch it you could tweak the tone map curves change the distribution function capture a bit more range complete your analytical mappings use 10 bit rather than 8 bit we also any grades need to be done you could back away from extreme color grading to avoid the reverse problem and you could scale the UI a little bit to avoid the shoulder region it can be made to work but we decided not to do that we went clean sheet and I'm very lucky that we got to do this so what we do is we transform our linear data after all

post-processing into a dedicated space for grading we'll do an HDR grading LUT we then draw you way to an oscillating buffer and it's essential to use P multiplied out for if you do this then we'll run a effectively kind of a scan out shader which is the last pass in the frame where we take the HDL day to decode it take the UI decode that composite them all together and then tone map and encode for our target display and that means that we can then target pretty much any standard that's out there you could turn that for si STR we could team up to HDR or any new standard that comes along and why did we do it clean sheet well we really wanted to improve these troublesome workflow issues and so by doing clean cheap you got to do that it means that we could also do a single grade of any TV that we support so we haven't got a horrible workflow of having to do we grade every single different type of a TV we're mastering in a single place and then mapping out to the end and if we're going to redo all the grades it may as well redo them really well and any restrictions there and we have the workplace into the same place also by

isolating the UI that gets us around a couple of problems our UI tech is got a lot of legacy screens like some games such as sports games have a very large number of screens and having to redo those to tweak them to look right was not something we wanted to entertain so just putting in a different buffer made sense we also wanted a really cheap revenge our implementation because the HDR spec is bigger than the range of TVs that exists then by targeting the forsberg your game looks better and better Easi by a better and better TV and we make these grades in DaVinci Resolve and I've looked at the wrong side how do we do this we wanted to find a nice distribution function for our color grades and we did look at a few different ideas for this we're really looking at logs base because it made sense especially there's so many s logged curves seem to look really quite nice but we ended up using st 2084 or p key or the perceptual quantizer and this is a function designed by dog we fire

dynamic range and it it's a ensures that every single different value in that in that ramp is perceptually spaced from the last and that means it's a perfect grading space because every single entry in that lot is perceptually space in the last so your artists have control over that image and the darkest shadow to the brightest highlight we've also gone one size larger on the lot 33 by 33 by 33 and this is just to align us with industry standard grading tools which do the same and a single distribution function apart from being nice and accurate it also means that we've unified our space so that we can share lots of cross games and that means anyone can build a look and shape with another team and that means we can build a look library or a film star library and that's really great and this is where i'm talking about there when she was off so i'm we decided to use this industry standard grading tool we built our some lots in their view lots to make sure that it looked exactly as the game would look and that means it's a WYSIWYG workflow and we did think about trying to emulate tools like resolve in the

engine but actually it's a lot of work and didn't really have the need all the time to try and reinvent and we definitely Dhin want to have to keep maintaining it because these packages keep evolving and I didn't really want to commit myself to have to reverse engineer this and then maintain it forever also using a standard package reduces friction for hiring contractors which is really helpful for getting say film graters in to do a really nice pass and also resolve and its basic version is free however I will say that there is a really great workflow improvement here in resolve which is the resolver live mode and in this mode you buy a capture card so it's not free and you pipe your game through the package so you play the game through the grading package so you live great this is almost analogous to putting all your grading tools back into the engine at one time it's still runtime it's still life but it's kind of inverted and it's really nice so tone mapping or display mapping in our case we've somewhat decided to call it a different name it is still tone mapping but we've changed name a little bit so

in our case the HDI version is now the reference version and we've moved the tone map right to the last stage of the pipeline and we've called a display mapping because this tone up is different depending on the TV so for a standard dynamic range we do an aggressive tone map an HDR it's less aggressive but it's still varying depending on the TV capability so a low-end HDR TV would be more aggressive than a high-end HDR TV would be less aggressive it's tuned for each case it's also not just a town map curve it's also we exposed and because it's per display its display mapping for us and the main challenge of this was to make sure that it did scale across a large range of different TVs and achieved a similar look also we really wanted to get away from the hard-coded film look so we didn't want to tow we didn't want any contrast changes we didn't want any huge shifts and we didn't want to put any sort of built-in look so essentially we wanted the most neutral map that we possibly could get and if I go back to this example image focus again on the

tube troublesome areas and I'll go back and forth between off and on and if I just I like those two side-by-side and leave it there for a second you can see that the the horrible huge shifts that were there in the light from the base well that's gone that was an orange light and it's now in July and in the hologram that was a nice blue light but had gone cyan but now it's actually a blue light so this is just a shoulder the rest of the image is completely unchanged if the artist wanted to put a film look at Oh some contrast that's entirely up to them but we don't want to force that on them and how do we do this so rather than working in 1d and apply that same curve to red green and blue we split it to work in chrome Alima and we apply the shoulder only to the top end of the luma we then progressively desaturate chroma depending on the shoulder it's not at the same rate and we're using a relatively new working space for this I see TCP and this is developed by Adobe for the high dynamic range TVs and it's

got one particularly nice property which is that the chroma part follows lines of perceptually constant hue and that essentially means as you scale chroma in and out from the center it does not change here it only changes saturation and that's a really nice property for this this particular algorithm we then did the non PBR thing of totally tuning it by eye and a completely ad-hoc camana lots of different screenshots lots of different games lots of different situations and yet it's not really PBR but this is really very perceptual so it's right at the end of the pipeline and we figured that would be okay it's perception not maths and I've got some examples side by side these are split screens so on the left of each image is the old Frostbite filmic tone map and on the right is the display map and I've highlighted to tour two parts of this image well you see that the sky which was going cyan like the hologram was now in the display map it's actually just preserving that you its desaturating as it hits the top but it's preserving hue and the same is true on the right but on the from the floor you can just see the wood color that subtly changing color on

the left and on the right is not but I've got some more extreme examples here I'd like to particularly highlight this beige sofa and hopefully you can see the subtle difference than the split screen so I mentioned that we got the hue shift and we got hue shifts in the mid-tones very very subtle hues shifts in the mid-tones because that we don't have a true linear mid section to our tone map and of course as your RGB values go up and down this curve they move closer together and the parts that we changed the ratios between the channels so this page of the phone is meant to be brown is actually kind of a gray green and it turns out that once we've done this our artists have been complaining bitterly complaining for years about this of unknown and bizarre hue shift and they couldn't really explain it didn't know why it was happening but they were fixing it in post they were color grading it out and it all turns out it's because of this slight lean on an image

section where we're fundamentally changing the color channel ratios were changing hue so they were the display mapper we've actually recovered the original color and we're not getting this and that's a wonderful thing the most extreme example though is perhaps this one on the right hand side which is a classic game sunset look that's meant to be a very bright orange light but of course in this situation the red channels crept up hit the shoulder then clipped completely and the green channel is being boosted and boosted so of course we go from an orange to a yellow and rather unfortunately once you see this you can't really unsee it so next time you get a sunset look at it with your eye not actually that the Sun and then take a camera picture with your phone or whatever and you'll see this it is everywhere and we wanted to get away from it so we were feeling pretty pleased at this point and thinking it's all going well plain sailing yeah of course of course it wasn't plain sailing so if you will call the SD aspect which is roughly 0.12 hundred nits and the TVs

are more capable and they do over Boyden everything actually that hundred nits peak is not going to be a hundred this it's probably more like two hundred to perhaps 400 s or two two to four times brighter HD our TVs though with this new standard they largely do what they're told so if you ask for a hundred nits you're going to get a hundred nits so in this situation when you put an SDR TV next to an HDR TV with the same content yhwh gr TV looks darker and it looks worse well that's not ideal so how do we fix this what we chose the that the effort on to making the HTR reference and so we chose to expose for HDR and working in HDR there's a lot of latitude a lot of range to fix things in post and our SD our display mapper under exposes the SDR version so we'll master for HDR make you look good in HDR and then we're under expose it and well

under expose it so that when that SDR TV over brightens again it recovers the original range and that's kind of a pseudo HDR on SDR look it's not correct because we can't predict exactly how the TV's gonna over brighten but it does look pretty good it does kind of work that's what we ended up with and in engine the artists by we can set this value and that means that they're working on a calibrated monitor they can dial in the correct number and it will look absolutely right so it's in now plain sailing no categorically not this display mapper is a hue preserving shoulder and that really faithfully reproduces you which no hue but that means we were seeing that existing elegancy assets were changing and that's because that some of those assets were authored to leverage those huge ships present in that tone mapper of course when you change the tone mapping you change the look of the assets and they don't look good now I'd like to thank the dice effects team for giving me a loan of an effect I've heavily butchered this this is not

the dice effect as it is I've made a very contrived example by taking a fireball and basically producing it in a single hue it's kind of a burnt orange color I've been overexposed it so on the Left what you're seeing is a single color fireball that's been overexposed so that the red Channel clicks and then the green Channel doesn't and is creating yellows in the hot spots now lots of assets have been authored to leverage this particular look to create that huge shifting when you apply the display mapper and it preserves the hue you just get a very flat original orange fireball and it looks frankly terrible so there isn't really much you can do here apart from Rio 30 effects to the tunas and very unhappy artists really does a centrosome very bright effects such as fire and because they were also deliveries that we had no choice but to fix it I remember back slightly to that slide when there was a cheap and dirty approach which is to

reverse a time map when you reverse a tone map even if you can recover the original depth values you aren't going to fix the clip point there's still a white point in that white point is quite low if your effect is in that shoulder region and you can recover that value the effect will look different on every different HDR TV as well because those huge shifts will vary depending on the capable of the TV so it didn't just limit it to SD arts as in HDR - so we often the effects was basically it had to be it had to be done and we decided to use black body simulation but as I don't know you effectively alter the effect as a temperature and then you author a color ramp which is derived from blackbody radiation index the ramp from the texture and then tweak the vamp to suit you so this is the original display mapped horrible orange one on the left and on the right is the black body simulation and these are both using the display mapper so these colors are now present in the original effect and not an artifact they're actually there and it looks better to suit of course because it's not a horribly butchered example and just to compare on the right is the

proper effect with the blackbody simulation using display mapping and on the left is the butchered one overexposed you can see they've kind of got similar characteristics but on the left is an artifact and on the right is in the earth in the effect itself so now is it plain sailing well no again this hue preserving shoulder it's great we can work with it but it's not always desirable and it precludes the creation of very very bright saturated visuals and again that's effects so if you look at the center of that fireball that very very bright hot spot it looks hot but it's also white because we've knocked out all of the saturation and that means that we can't match particular film stock so if you've got a game let's say trying you came trying to match a particular film's book you can't do that because we fundamentally changed the display member so we've started to reintroduce some hue shift so we're going back on ourselves now now on the left the and on the right you've

got two different versions that's all you shift on the right and those who shift on the left in the underexposed parts that's the bottom of the two rings you can see there's not a lot of difference but hopefully you can see that in the top two on the Left where it's de saturated on the right there is now a yellow here in there again and this is what we want and to show you the two comparison images once more on the left is the original display map and on the right is one with a subtle hue shift it's not very much but it's enough and you can see the center of that light is now yellow rather than orange and just to show you against the original image which has no display mapping yeah it's a lot better still but it's somewhere between the two so yeah we have to reintroduce the huge shifts and we're still not done we are still iterating but we are starting to get somewhere and that doesn't mean that different games that are going to ship later this year are gonna have slightly different implementations and they would have chosen this is the situation that suits them best some do prefer the huge that he preserving shoulder some prefer the hue shifts and what we're likely to end up with is sums of content different

mixer but this is really neat nice and easy to do because we've moved the display map right to the end of the pipeline and we've already graded it so we're not affecting anything there we're literally tweaking this the output shader and that's really really easy to do and it's also really low cost now a few of you probably thinking what about aces because surely aces does a lot of this or all of this and Fidesz you don't know what aces is it's the academy color encoding system and this is a really gaining pace it's a standardized color management system for film and it also defines a processing pipeline and also includes of lurk and display mapping components and this is broadly speaking what it looks like the input the Vice transforming shows everything's in the same working space the look modification transform is where you apply your look the reference rendering transform is a standardized of filmmakers curve and then your output the voice transform of which there is one per output TV or broadcast standard and it's varies per TV a broadcast broadcast standard seems very similar so why didn't we use it

well we started work on this in 2014 and Asus was really only starting to get going some of the early versions of eight is also required v16 and weren't suited to put some runtime texture formats and also we weren't convinced as I mentioned in a start of the the default filmic look but however we did really agree with a lot of the principles so we absolutely used it in concept but the order of operations is the same the concept of working in a wide master working space and mapping down right at the end of the pipeline the concept of the LMT is a grading pass the output device transformers a display mapping so we absolutely agree with Asus apart from a few little things so should you use it yeah of course have a look at it please and if you're going to do so investigate Asus CC which is the color correction space and it's a cg which is a color grading space we may still also ended up using aces of course we're going to continue to investigate it because we agree in principle on so many things and because we have a defined working space transitioning to another one should be relatively trivial but also likely to

adopt ASIS for color management okay so we have some more problems now for performance reasons this display map which is quite expensive we chose to bake down into our highly accurate color grading well the order of operations are still the same so it's made sense let's make it free but the analytical tone map is on the top the display maps on the top in the middle is our baked a lot version and you may not be able to see the artifacts here they're a bit more obvious in other situations so I've magnify the difference on the bottom and you can clearly see more Mac branding there's the there's some bands in this gradient here and this really isn't ideal it's not noticeable in a lot of situations to be fair but in some like this particular one is very obvious and why is this happening well firstly we're using linear texture filtering because it's fast and linear texture filtering turns any curve into a piecewise linear approximation of a

curve but more than that we're using a 3d lookup table and we're using linear filtering and the luma axis which she saw in that particular shot it wasn't a lot of color there the luma axis goes from one corner of the lot to the other corner of the it's an diagonal and when you're in diagonal when you're halfway between two Texel centers you're pulling in equal contributions from eight separate texels that's not wrong it just means that you're getting slightly more contribution where you might not want it so you can do you fix this well of course you can just use higher-order filtering you can go from linear to try cubic and this would really help but it's also ridiculously expensive and rather than spending our time trying to work out how to make this fast we took a different approach back to the original slide in RGB we indexed a lot by red green blue but you don't have to do this our display mapper is a chroma luminous

play mapper so we thought why not try indexing axes by Chrome illumined the fastest one is YC o c g YC g co and there's a big improvement straightaway by using this we then dried y cbcr it's even better still then we figured well since we're using IC tcp for display map i will just try that and that's slightly worse than y cbcr but comparable to YC g co and here at a different side by side all of the 3d correlated life spaces are better than RGB far less error farlis position problems so this is good and why is this well it's fairly simple we've simply aligned our primary axes with primary axes in the lookup table we're still using piecewise any approximation but these are now running in 1d or 2d rather than freely we're not going on a diagonal we're going in a

primary axis and that just reduces the artifacts and that's greatest still just as fast but it looks better but of course we're baking this display map on top of existing grades and some of these grades could be quite extreme and we wondered whether this would cause any problems with that so we then plotted the access patterns of various screenshots into the different spaces to see if it would cause any precision issues or quality reduction just reminder this is the numbers are going to be from this particular screenshot these are lookup table coverage in each space from that particular screenshot so we've plotted every pixel in that screenshot into the lookup table and made sure that we cover all neighbors where we'd be touching their neighbors and red-green-blue surprisingly only uses less than 8 percent of all a volume of this lookup table not as much as we expected but the 2d correlated spaces voice eg conc BCR these years a lot less

again this is more than half the coverage of the volume that we've used which roughly half the number of texels that contribute which roughly half the precision of the or the accuracy of the grade no idea oh I see TCP is better but not much better so today we've actually stuck with RGB and those artifacts that are there which you can't really see that often we've kind of lived with them but we're continuing to investigate this ok a few notes on performance performance is absolutely at a premium there's an increased focus on 60 frames a second which is really good increased focus on resolution because no one really likes 720p and we really couldn't afford to give up resolution or quality of performance to pay for HDR and because this new path entirely replaces our legacy one we were required to achieve performance parity it's fair enough so what was the performance of

our legacy path I forwarded two parts here together because a lot for reasons will become evident a bit later at the end of the frame where our scanner shader would be scan out shader the final part of the frame we already did a high order resample because we do run dynamic resolution we do run slightly sub resolution at times or above resolution in some cases and this is a fairly quick ish dual pass we sample high order example where we do a vertical we sample the horizontal we sample an Xbox you go viral intermediate in the faster vs room but your final buffer ends up in main memory then you join a UI on top and it's in main memory because you scan out from main memory and so we're looking at about point 4 3 plus a little bit of UI so in the cheapest situation about point 5 milliseconds and I'm going to use number some Xbox one because it's the slowest platform in this case so version 1 of this the one we what we got in the hands of the office really quickly was a super naive implementation so we

rendered the UI to an off-screen target but to do that you must clear it so we cleared the UI target and that added a quarter of a millisecond within modified the second path of audio provide your pass for example you load the HDR you convert P key to linear you then load your UI target and make sure that sania composite the two-tone map or we encode - srgb and there we go well not really optimal not really very good yeah this added a large coffin this is not OK but it was ok to get into the hands of the artists so of course to take some low-hanging fruit you know it's notice that the second pass is dominated by AO u and first thing is really easy when you are UI is rendered in AU non-format for various reasons but it contains srgb data so of course you just alias at textures srgb and get the conversion for free also PQ conversions from torrent from linear are expensive and we decided because we're doing a lot of work here that in this particular shader which is Lu dominated it would be a lot faster to use a simple

1d lookup table for the P Q to linear part and so put that in and then we started to hit bandwidth limits on the second pass and so of course you just put the UI target in es room and that has to clear costs and the UI also renders T times faster than before which is great and we come down costs come down about 0.8 milliseconds now I still not very good but it's a start and then I'll get into the resample part and it really isn't anything to do with HDR but we were starting to run into lot of load balancing issues where because we had the UI in one place and then first pass of the resample and then the second part of the example we weren't getting really great utilization across both of these two passes and this was becoming a bit of a blocker so by switching from a two pass up we sample to an optimize completely sample where you can do a single pass but do you'll pass in sight the shader whose local memory this is an enabling optimization that brings two passes into one unless the GPU load-balanced a lot better there's some specific optimizations in here related to the fact that we know

our power mode because we're scanning out and we have got some GC and specific optimizations in here and this in itself wasn't a huge win but it's an enabling optimization so we saved about 0.1 milliseconds but the last final big win was from the C mask where come composition now AMD's GC and hardware maintains this C mask metadata and essentially the stores were there any block of full before pixels was rendered to or not so we neither a clear you don't clear the memory you just clear the C math data and then when you draw some pixels it marks to see mask whether it's been drawn or not and then before you read this text you back there's a fast clear eliminate pass and all that does is look at the C mask and then figure out whether you've written it to memory or not and where you haven't it waits the clear color back to memory smart and simple so since we can read this metadata why don't we just remove the fast clear eliminate with the C mask in the composite pass and then where we know if not drawing the UI we just don't read the UI and then do anything and in concept that's what what we did it isn't

quite that simple because C mask is a custom tiled and packed former and that requires that you do detailing and unpacking in software and that isn't cheap so doing it in the second in the in the resample wasn't necessarily the right thing to do however you can of course issued after very tiny transcode shader which takes your C mask and turns it into a format that can be read in an optimal manner for the composition then you choose this format to match the composition thread layout and this is a really really cheap pass 0.02 milliseconds and of course that is so much cheaper than the first clear eliminate and in the composite pass you take this you load the bit mask you unpack the bit corresponding to you which is very cheap and if the bit 0 you skip the entire operation you don't load it you don't clear it you don't composite it you don't do anything with it and that brought us back down to half a millisecond but of course our UI is now any SRAM so it renders faster so in some cases isn't quicker than the old path which is nice now what only in the SDR version here because the STI version is the one

running on the base Xbox and most people have got str tvs that was the most important path but of course you want to support HDL so when you're in HT on is a bit of extra work you have to do firstly you have to rotate all the primaries from srgb to 20/20 you don't have to encode to PQ rather than srgb and that's more expensive but not too much more expensive so in this path it's about 0.7 milliseconds rather than 0.5 milliseconds so there's a small overhead however this path only runs on the Xbox one s and is either ps4 and the Xbox one s has a faster GPU to the tune of about seven percent or about 11.1 milliseconds at sixty Hertz so this point two milliseconds overhead is not a not a drop in the ocean here and that means that the bulk of the Xbox one X performance actually goes straight back to the game for games the dynamic resolution that means you get high resolution and they each job doesn't really make an impact and that's great and I'm not focused on ps4 here because both ps4 and Xbox one at the same resolutions 1080p this is a scanner shader which runs at 1080p so for the same resolution the ps4 has more use we haven't worried about it now a little

note on the HDI standards themselves and the platforms that support them there are two primary standards at the moment dolby vision HDR 10 and frostbite can support both and because we have this final path supporting was relatively easy so something like hybrid blog gamma becomes ratified and used a lot we could support that relatively easily that's a small asterisk here that I have to say just because frostbite can support both and we do support both and doesn't mean that every game will support both because licensing agreements and everything else I find on a per game basis it's entirely done to the game team to choose what they do so I am unfortunately can't stand here and say yeah everyone's going to support everything it'll be down to each game but the tech can support both and so we're hope that everyone will but I can't guarantee it the double-vision has some pros and cons it's a 12 bit signal rather than 10 but it's also HD credible with older versions of HDMI so it goes down to 1.4 be in double vision there's

no need to write a display mapper at all Dolby does that for you and it's embedded into the team with TV firmware and they'll be say that because they built this display mapper for each display they can guarantee some source of uniformity across display some form of standardization and we don't have any control over this that's entirely down to them and they also promise that you get a good result some low end panels now there's a few downsides to this in order to feed the TV information to let it and run the display mapping you have to regenerate some metadata and you have to encode the frame buffer and the metadata metadata generation does have a cost you tend to histogram the frame and that's it's not entirely cheap it's not insurmountable but it's a small cost here and when you encode the frame before that adds a cost as well but this frame before once encoded is now non blendable so you can't blend overlays on top of it and that means that if you try and draw something like a friend's notification it can grow up the frame buffer and that's not good also double vision isn't today supported by many TVs

HDR 10 has pros and cons as well HDR 10 is far more widespread and pleaser doesn't have a custom frame buffer encoding any a blended overlays dues of work and as we showed earlier source software display mapping we used the both software display mapping can be done very cheaply on the downside this is probably the biggest downside there is no standardization at all on HDR 10 across manufacturers indeed across modes on the TV or game modes cinema mode whatever there's no standardization at all and this means that your game can look different on every different TV and every different modes on every different TV so to help try and bring this a bit more closely aligned the game really should do its own display mapping also HDR 10 is 10 bit rather than 12 bit however they share a huge number of commonalities they share the EO TF they do show a minimum and maximum mastering luma they share a color gamut and that means that if you're mastering in wider space you

pick the same space for both because it is the same space for both and the same master content will work on both and you just have a different display map target for each one and that means you can support both with the same data so it's not real the extra work apart from the implementation and which platform support which are both ps4 support HDR 10 xbox 1's supports HDR 10 and i'm peacing you can pretty much do whatever you want so HDR 10 double vision whatever but this because you have to send metadata over HDMI to make this work you do need GPU vendor extensions to handle that metadata so you've got to call a couple of functions and the x11 with exclusive fullscreen this just works because you just take full control of everything do double vision encode your frame buffer everything's fine the x12 is a little bit trickier but it doesn't have exclusive fullscreen and the desktop compositor can then or come in and perhaps scale or add overlays and as I mentioned earlier if you try and scale or IDE overlays the double vision it breaks the encoding it breaks a metadata and that can cause some trouble but we are working with various people to try and improve this and of course do

not forget SDR TVs there is a majority of the market there's a huge number out there and the SDR version needs to look great it's so easy to get carried away with an HDR TV next to your desk and get really excited but master your content in HDR and treat the HDR as a reference version by all means but because we own the display mapping we need to really tune that to me the SDR version look really really great as well and as I mentioned before you can also play to the fact that the SDR TV over brightens your image so you can scale it to fit okay what are our next steps in frostbite well high dynamic range video is absolutely one so today we don't support high dynamic range video we do supports or pseudo height that I'm at range video because we can take the video and we can scale it up just like the SDR TV would so we can make it look a bit brighter and it actually looks pretty good we can also do something cheap cheap like reverse the tone map curve in the SDR video and extract the HDR data not a large range but we can

reverse that and get something else so there's ways to make it work at the moment but HDR video natively requires higher bit depth it is a higher decode performance overhead it's got bigger file sizes it's got bigger streaming overheads and also when you've got if you've got a marketing's or ecosystem where you're taking a version of the game and capturing it and then producing marketing videos like YouTube now supports HDR for example I don't want to have to create a different marketing video for every different version it'd be a huge work workflow overhead so we want an ecosystem to be able to capture playback edit and then spit out multiple different versions of the marketing video for every different HDR format and that needs a an ecosystem in your workflows and read this comes down to managing color gamut and we don't support wide gamut rendering currently so when looking at what HDR TVs give you they give you a wider color gamut and they also give you a wider dynamic range and we figured that the wider dynamic range which also gives you better

saturation it's um it was the bigger and easier thing to do so we went for that straight away but we really do also want to support wide gamut rendering however this is a loss of work so first you have to expand the runtime gamut to feed the TV you then have to have gamut metadata on every single color asset you possibly author and maintain because not every single thing is going to be in the same gamut so today we can assume because it's a twenty plus year old standard that everything's in srgb and say hey I'm got to worry about it everything is the same space everything kind of just works you can make assumptions but when you go wide gamut you're going to want to selectively author certain things that benefit from wide gamut and that means they need separate metadata so skin tones flora fauna fire effects all these things are outside of aside ubi 709 and so you need to preserve this metadata and you have to move this metadata in and out of the digital content creation packages as well and some of them supports one standard some of them support no standard some of its manual so this is a relatively error-prone area and it's also a lot of

work because you have to transform everything into the same working space to render and this is where things like aces open color or open image I can really help because it's a it's a standardized way of doing it but not all packages support this so this is why it's difficult however we'll start by working from the TV back so the first thing we'll do is to convert the color grading which in one of the last stages into high dynamic range implements and out of gamut or debug modes to help the artists and then work back from there however our color grading it on in look-up tables and if you recall this light from before this is ham this is a percentage of the volume of the lookup table that we actually use and if we take this and move it into 2020 for example it gets a bit more horrible so yeah our RGB value which is about 8% drops to just under 5% ycg co and y cbcr they have again back

and forth couple of times but I see TCP does not change and that's because I see TCP is a natively wide gamma HDR format so when moving to wide color gamut considering using I see TCP for a grading format it starts to become really appealing and on that just a quick note on camera production expansion is trivial what will hold more will hold less and it's basically a matrix transform a linear space however gamut reduction is not not so trivial your second value is outside of your target gamma and then naively rotate it then you get out of gamut colors and this can manifest as negative numbers which when you clip to zero to output you get hue shifts and in a quick test like a vibrant red can become purple it's not like a subtle things it's really obvious so you must absolutely map your colors to the target gamut before you do the transformation and again we're back to this format I see TCP we really suggest using this as a working space because if you recall the

chroma part of this is perceptually hue linear so you could do something as simple as just scaling a saturation until it fits the target gamma you won't change hue you'll just change saturation so this is our thinking relatively simply just scale the saturation down until it fits in the target gamma and then go from there so just to wrap up I tried to distill a few key thoughts messages and learnings over this two-year journey first and foremost ensure that your colors are actually in those assets make sure they're there do not rely on a tone map or a clip point which all vary in HD to change the hue make sure those colors are in those assets and please try to master your game in the widest possible space master in HDR and then produce every version of that as an artefact so move your turn up as late as possible in the pipeline as you can and then start to play with your time map for each different display also it is worth considering using D correlated spaces I

see TCP y cbcr whatever because I want you be isn't the only way to do things and if you can please try to aim to support all standards it's in all of our interest to try and drive the adoption of HDI it is a lot better but please don't also forget about SDR because that's what most of your sales will come from and that's pretty much it I'd like to make sure there's a few thanks here especially to Thomas here how much is really the brains behind the display mapper he's put a lot of time energy and love into this and it's really important to make sure he's credited up here also I'd like to thank Bob Lee for the help several unnamed game teams specifically the dice effects community though because they probably a lot of test assets for us and Martha helping me with this talk and that's me done any questions hi hello hi congratulations I agree on everything here it's very cool I have a

couple of questions so you have your display mapping it's nice do you use it for allowing users to calibrate different ambient levels different displays all ads LEDs you know and if so how do you present that choice to the user okay so the question is about do we use the display mapper is it can the users use it to adjust the viewing environments it's a really really good question and I haven't touched at all on viewing environments I kind of deliberately didn't do that I didn't think I'd have time I probably should have gone into it cuz I have a couple of minutes yeah this is only about the end-to-end pipeline to be use the final image the viewing environment makes a really really big difference to how an image will look you can have a very dark like controlled room and everything looks great you can then have a really point in a living room and everything will not look good so yes the display map should be used used with user controls for adjusting gamma brightness contrast and things like that we currently don't but

yes that would be the perfect place for it and actually Timothy lots did a talk last year on on vdr yeah yeah that was just really cool yeah yeah and if I can a second one very quickly how do you present all these to your artists so I imagine that not everybody will have 16 TB under day so does every other 7 HDR TV no no no definitely not so the vast majority of the artists run the display mapper in SDR on a vaguely calibrated desktop monitor and these projectors on HDR so everything you've seen today is the SDR version like it's the display map to most aggressively displaying that best er version so because it preserves hue actually it's really helping effects artists to create their content even without an HDR TV so it's it's presented as it would be to anyone playing a game at home the display mapper is most aggressive on an SDR monitor aggressively do you plug in an HDTV

seems to be going pretty well and we've an intervention resolve we've run a generated a view lot for the the most aggressive SDR version so it's a WYSIWYG workflow in resolve I guess you have very good materials we noticed this that your highlights or your levels when they are crushed but yes they are the artists don't see that sometimes so when you switch to HDR you immediately know these things that ya know something you know something else the artists can do because they can sort of type in the the theoretical peak Loomer of the FDR TV if they really want to they could type in like 2000 or or 10,000 then it will stop the image down multiple times so they can actually see the original highlights there's roughly a hundred to one in improvement in available range in color grading by switching to HDR from SDR so there's a huge amount you can play with in there you could you can completely expose it in post-processing and not really lose any information it's really nice thank you hi Alex great just a really simple question you mentioned reintroducing the shoe shift and I was

wondering if you could elaborate on how you did that was it quite as simple absolutely yes it was very simple we would literally run a 1d version of the shoulder of our G and B and then the chrome illumined blend between them right I thought it might be so Mir that Thanks any more questions I have a wonderful talk thank you yeah I just wondered because you mentioned that your mastering in HDR with your final kind of tweaks and grades what kind of display are using to do that it depends so the question was about what displayed you used to master so it really just depends so for the majority of artists would a color critical or doing color grading and they would probably have a either a calibrated monitor or something like a noise over hardware calibrated monitor

with this shield around it to shield them from light and that's a color critical work it's a high bit def monitor and they can grade an HDR and they still get all the results they need however you can also buy for roughly $30,000 so many rating monitors and others are coming the ideal is that the ideal situation would be is that every game studio rather than game team but every location would figure out how its fund buying one of these and they would share it and share it between teams we've been very lucky that Dolby have a reference monitor and called a Maori which of which they're not very many in the world but they can loan them out to different people to help accelerate the adoption of HDR and we were lucky enough to borrow a few of those so for some artists they were very very fortunate to run on a roughly 2,000 net reference grading monitor but the like if you were to use one of those eyes or monitors are they they like Hine

it know the ISO standard that I'm in range so you're really great in color but you can you can wipe the exposure up and down so you can really see what's going on and typically what you would do is run on a live version of the game in parallel with that running on say an Xbox or Playstation it connects to a relatively cheap off-the-shelf HDTV just to validate side by side and because her life workflow you you would get your feedback on both and it is really important to test a lot of different TVs by the way so you would probably end up with in key way taking this version where the artist has played on one HDR TV and then QA would very quickly run it through a suite of different TVs from OLED to LCD and just test so it's quite scalable you might just work on SDR you might work on SDR and a console connected to HDR you might be lucky enough to have reference grading monitor it's fairly scalable thanks sure I believe we're done thank you very much for coming much appreciated [Applause]

you