Visual Effects Bootcamp: Bringing Feature Film Effects to Real Time VR: Part 2

[Music] and welcome to the part two of this presentation about bringing feature film effects to real-time vr experience this talk will mostly focus on the technical side well a lot of math so i will try to make it not boring and show how we actually implement these effects in the engine first a little bit about myself my name is for who this awesome guy and i used to be a pipeline ta working on triple a games like dead space and battlefront these days i'm a look dev vfx td working at x lab and during the past three years i did a lot of amazing experience including the oscar winning carne arena and multiple critically acclaimed ar vr experience first i want to talk about the shader

effects we used in the ralph brick vr experience since it's the most recent released project is need to have a high visual bar as a disney animated feature film and need to run a 90 fps double lighting vr also we have a lot of complex gameplay so we got to finish all of this in a very limited time frame and here are some expand some example samples showing all the moments that were created inspired by the movie for example you see the hub turning on and we go through the hyperspeed internet cables and we went into the dunder dome realm and it's sparkling and continue sparkling and you can see the awesome spaceship and also the binary transition between

all different worlds all this effect serves the purpose of storytelling and give life to the environment and i i think we did manage to get a feel like a disney inside inside a disney animated film well the internet is huge and each place are so different from each other which means each places got its own unique challenges both visually and technically and this new animation studio did a fantastic fantastic job in the film to give you this amazing experience and we are aiming to achieve the same and maybe more with hyper reality technology so here i'm going to break down some of the details of how we created these moments so the first one is the hyperspace we need to make you feel like traveling's

traveling through the high-speed internet cable like ralph did in the film and need to be performed at 90 fps since his vr so apparently we cannot do a thousand meter polygons inside the engine and i implemented all of this by separating all these elements into different layers and you can see the different color lines represent the netizens traveling with you at light speed and gradual gradient patterns across the cable that makes you feel traveling through a super long distance and with different layers they got different motion settings in the shaders all this combined makes you feel you are really accelerating at light speed well in your in reality you're just standing there admiring all this awesome art so on the left is the shader parameters

that i exposed to designers and on the right is the snap code that did the motion blur in one direction that created this illusion since it's dynamic we can iterate the look very rapidly to get what we want in the end by just using a simple cone and a few textures you almost got the exact same effects in the film next we came to the dunderdon which is heavily inspired by the 80s popular arcade games with a lot of fun gameplay i bet someone here still remember when you play this kind of games to match that special visual style we have to figure out a way to get the glitches and jittering and all the crazy stuff happening in the old crt screen but it's got all working in full 3d environments while in back in the old days it's just

2d pixel animation on screen right and maybe only a few hundred lines of code to implement the whole game so how we convert the pixel art style in a full 3d immersive experience and this is what we come up with just using a simple polygon lines but you can see in the right corner it's got all the connecting points and variations of the lines and skyline crosstalk and all the detailed features of the crt look screen look after applying the shaders which is the middle where all this magic happened and here's the detail of the shader parameters and by changing just one of them you already got a very interesting destroying effects on the right and you can see the x-wing goes into lines and blobs into lots of visits i

think the mission is accomplished right so from space mountains to spaceships every elements in the scene is built by the instance of these materials and the various effects can be created dynamically through gameplay as as you can see the glitch and generous effects here the benefit to do this you can use a different you can create different behavior that meet the need of storytelling and have all the effects tied to the visual looked at at the same time and you can also adapt any changes at any time and don't have to worry about performance later the other very interesting thing is the binary transitions as i said before each element is each moment's got its unique look right so we can since we cannot do a camera cut like we did in the film how we link all these different places

together to tell a full story the answer we came up with is everyone knows the internet is made up of a bunch of data of zero and ones so it only makes sense that we use the matrix system to translate you between different worlds and this shader need to have a way to seamlessly turn any object in the world into binary form and this is how it looks like i use the cell noise and wall position to make sure it works on any kind of surface no matter how it looks or shaped and here is another snapshot of the shader as you can see i enclosed all the mess and logic and only expose as few parameters as possible to satisfy the designers we all know sometimes designers can be

really yeah and all this combination can already give multiple different behaviors so here is some detail mask about how to make it randomly switching from zero to one across the surface during the transition and the key here is to you treat the cell noise value as a threshold so any input value that beyond that shows one and below is zero and on the left you can see a line crawling effects and a spherical spreading effects in the same shader and the logic on the right the secret is to use the cell noise to drive another cell noise and that's how you get line crawling that match the look in the film and this is a way to show you can actually use the noise function that building the engine to get a very complex look

and combined with hyper reality technology it serves the purpose of magical transformation and storytelling okay here comes the fun part the other big challenge is that we want to have a pancake milkshake launcher in player's hand which shows on the right side and need to have a jar to store the liquids that make both pancake and milkshake and it is very close to player so we can simply use the mask trick to fake the volume compression like we used usually did in traditional games and since okay and since it's closely tied to the gameplay and all kinds of function as real time is almost function as a real-time simulation and we can simply use the vertex animation technique that martin mentioned just in the first talk

and i need to figure out a better way to do a real fake volume in vr and with complex behavior and the trick to achieve that is use vertex shading here shows how you can change the color and volume in the fluid it's need to look like two liquids mixed into each other but not dissolving each other we definitely don't want to mix the flavor between milkshake and pancake because this really tastes bad the other one the volume decreases but it cannot leak outside the glass jar otherwise it will break the illusion and the interesting thing here is like we don't have we don't use any collision detection for the performance reasons okay what is the trick here the key to achieve this is to separate the top layer from the bottom as two separate parts and use by using a threshold

so the top part will be implemented as a procedural surface with liquid noise and here i faked the shading to simulate the translucency and the bottom just use the world z value to drive the logic to mix the fluids it's a very simple and effective and the result is that we meet all the gameplay requirements while keep it looks really good and efficient all these moments are only parts of the effects that we did for the project and there's a bunch more down to for the environment to get a feature film look into our vr experience for example you can see the cable capsule bits go through the carts electronic sparks in this hub random animated billboards in the internet bubbles going through the cool internal and high reflected fake glass cards that used by the netizens

in summary causes real-time and dynamic nature we can quickly iterate and create complex visuals through shading now here are some bonus parts and some more detailed breakdown about how we did the lovely moment in the secret empire experience as you can see the lava flea need to distort and push the lava field around him while it's moving to better integrated with the environment and as you see the lava plane is just a simple poly plane and all the visual detail here is driven by a shader procedurally because i really really hate using textures this is what actually happening under the hood first we assign a spherical special vertex shader that expand lovely to get the surrounding volumes

then we use a render target to capture the area that the flea intersect with the lava plane is almost functioned as a ctr scan we use in the hospital and then we use it to accumulate the value of distortion to drive the vertex offset of the lava material to get the trail behind the lava flea and after that we combine the vertex simulation the lava splash that martin talked about in in the part one and the glowing belly material of the lava flea and with all the other animations and audios you got a full giant lava flea attack moment this technique was inspired by ram box talk a few years back the good thing about this is this works on any kind of surface with any kind of object intact with it without any additional big data

and it can be used on other projects as well and that's a wrap the takeaway is you can really do a lot of cool stuff in shading and it's a great way to implement the visual fidelity that serves the storytelling and gameplay while give lies to the environment well my philosophy is eventually everything comes down to just pixels that rendered on the screen that rendered real time so in summary shading rules and i also want to thank for the whole team at xlab again all these works are teamwork it takes every discipline's contribution to get the final product i'm glad that we'll keep pushing the boundaries what is possible in hyper reality and produce amazing experience that never seen before and creating a more exciting future

thanks for listening and have a great night please follow us on social media and any questions oh another thing please take on the online survey yeah the other guy told me i should say so uh hello thank you um hi so in vr did you find the render target solution was a little bit expensive yes you're rendering another camera exactly so we basically to use that technic technology is extremely expensive especially when you're using a render target like in by default unreal components so basically i work with the graphic engineers just basically write the whole thing and we literally only need to capture the one channel so it works pretty well

in this case great yeah you just have to rewrite it sometimes yeah yeah hey hey uh so sometimes when making shaders in unreal i uh i seem to find something that either it can't do by default or i don't know how so i i'm trying to use the custom node a lot and uh same here yeah yeah do you have any tricks for learning how hlsl meshes with unreal and because a lot of that isn't gpu germs read all of them and get all the basically it's just computer science graphic knowledge math things like that you just do it all you know there's really no trick man it's just in all the math you get it done all right thank you i'll try that okay yeah it definitely works hello uh just out of curiosity did you run into any motion sickness shenanigans with the like screen space zooming

example that you showed yes uh that thing right um for going through the hyperspace so the trick to get not get people sick is you shorten the time so we just basically we do this design so basically you're basically a well first is a cable and a very uh you're constraining a very finite space right at that point all the interest is looking forward so one thing i forgot to mention is that we added some vignette around the corner so once you turn this way you don't really see that much interesting details it's almost just like a blur so that's giveaway to always direct in the focal point look forward and that really doesn't give any motion sickness thank you yeah it's just tricks yes

hi hello hi um i'm currently a um cmutc student oh hi hi hello um i know i graduated from our program i know hi um so yeah our team is actually currently working on um a uh discovery exploring project um about like transition like um adapting traditional film uh transition technologies into like vr so um it's like really how i'm really happy to see like other things you did um seems like the title of this talk is called bringing feature film effects to like real real-time vr yeah um do you mind like talk uh talking a little bit more about your um just like personal uh thoughts on um future potential um just like in this area like about combining like feature film effects and feature film okay in

future vr applications future er okay well my dream is i'll talk about dream and i'm really hyped about 5g technology and i think like by implementing using the 5g stuff the vr and ar technology will actually really going to take off at that moment since at that at that point we can actually offload all the computation power and we can use like high-end machines and high-end graphics like just cloud rendering it and we can stream all kinds of crazy simulation data and now we have like crazy graphics right and we can do real-time retracing back then is unimaginable so if we can shift all of this using a much higher computation power then with 5g like real-time streaming then yeah it will work and everything will be real time because

it's say save much much production and just cut all the crap about you know iterations time renderings waiting nights and nothing nice no more okay thank you