Player Interactions Visualized

[Music] thank you [Music]

[Music] so [Music]

so [Music] ah i found one i found one of the inflatable crocodiles yes perfect now get the get the get there let's see if it pops we make it pop throw a knife into the crocodile okay that is not enough to pop it's quite strong get the silenced gum yes it pops yes i'm so happy

it has a lovely little pop animation that's the best thing this was the reaction of youtuber john at the channel many a true nerd to me it's very rewarding to see player reactions like this and it is also a reward to the player for interacting with the world this feature of popping an inflatable crocodile was not needed to ship this game but i believe it helped in selling the story of a detailed game world and in hindsight of course i wish i'd made knives cause deflation as well what you just saw was work i did for hitman from about two years ago my name is joker mellergaard and i'm a technical artist at rare the studio behind sea of thieves i started this role in october last year and before this i worked at io interactive on the hitman franchise in this talk i will be covering a lot of the experience i gathered while working

on hitman today i would like to talk about my view on ways of immersing the player in games i will be using examples from tests made in unreal engine as well as my work on hitman but in general the ids i'll be presenting should be transferable to any project so hopefully you will be able to take some inspiration ideas or code with you home today we will also be looking at examples from other games there will be some theoretical elements but mostly we'll be looking at practical solutions much like the inflatable crocodile example these ants from skyrim helps in selling the idea of a highly detailed world it is an interaction of discovery

vfx artist mark tier from bethesda softworks told me that these ants can only be found in a few places around the game world but once the player sees them it creates an impression of a much more detailed experience than the one the player would experience if the ants weren't there by adding this effect mark made the player think that details of this delicacy could be seen around every corner also other items in the game benefits from this by having the player believe in an overall richer experience i believe that's triggering the player's imagination with ants and crocodiles is the key to achieving this from a visual point of view these two examples ants and inflatable crocodiles belongs to a category of effects we may label as cosmetic interactions meaning they are pretty details you as the player did not necessarily expect to see or

interact with another type which we will speak much of much more of today is passive interactions this type of passive interactions are designed for grounding the player in the game world making them feel part of it there are so many experiences in games which we cannot do like smell or touch yet anyway and that's why passive interactions must be visualized the last category of interactive effects rewards the players for their actions it's about supporting gameplay let's explain that by re-watching the slightly unpleasant scene sorry about this the target is a patient in a hospital awaiting a heart transplant and there are many tools available in the surgery room there are in other words many ways to

get to this target each one leads to the visualization of the player's decision every decision that the player may take has to be visualized so that she can experience the outcome of her actions these three categories show us what visualizing player interactions does for the player together they immerse the player more than any other element graphical elements in the game because of their dynamic nature and direct feedback whether it being a passive interaction or an intentional one we may speak of immersion quite a bit but it is valid to ask why aiming for immersive experiences is important in the first place psychology studies have shown us that the player reacts more authentically when facing decisions in game scenarios in comparison to the real world and this

is likely to be because social constructions don't matter as much in games the players in a flow of joy having challenges and skill levels met this leads to an extreme focus and as the studies show even self-actualization and perhaps this is why video games are so popular as game developers we of course want to pay attention to these behaviors and make the player stay in the flow where these qualities can be triggered this is true for anyone in game development but i think it's well worth mentioning mentioning it in the context of visual effects by keeping the version high with our visual effects we can enhance the flow of the game so how do you make and maybe more importantly how do you know where to add effects like this start by identifying things that you believe the player may interact with and this can be by being imaginative but

you will most likely get many ideas simply by looking around the game world for items or features that you think could benefit from an extra level of fidelity know the tools you have at hand and use them wisely and optimized typical tools for creating the effects as you know could be particles cloud simulations physics scripting shading often though vfx creation is about problem solving and it can be difficult to know if a certain approach will succeed beforehand it's an iterative process if we go back and have a more detailed look at this inflatable crocodile we saw in the beginning we can see that the setup is quite simple it was created using a mix of cloth simulation shader runtime modification and game logic for listening to the gunshots the resolution of the cloth simulation is quite low so it makes sense to blend in this crumpled normal map when it deflates another type of interaction is

flattening car tires in games apparently it's rewarding shooting tires this footage is from the division and i cannot say for sure how they made it there but in hitman this effect which was made by my former lead martin nelson uses physics constraints object transformation and finally a vertex shader which flattens the tire to the ground no matter the orientation of the wheel so if you're thinking to yourself now that i have no idea how to make a shader like that then simply just do the first two steps move the wheel down and have the hole being constrained to the wheel so that the car tilts over a little bit this is a good example of when it's better to go with a lower fidelity version than to not do it at all i believe that the player does not care too much for the additional fidelity

when she's been rewarded for her actions so the clipping cost here by offsetting the wheel into the ground may not matter too much to the player another example of player immersion and interaction but in a more abstract way are these computer monitors showing a video feed from security cameras in the game the purpose of the monitors is mainly to immerse the player and making the loom room look like a security room but they also come with somewhat of a hidden gameplay feature i'd like to think that the players informed of where cameras are placed in the level simply by looking at the screens and recognizing locations inside the level the player is immersed by this live feed besides the fact that there are no characters or events taking place on the screen that's because it's not really a live feed simply it's just a screenshot taken from relocations of security cameras in the level

the fact is done in three steps zoom in to the image to make space for panning apply a fisheye effect which creates a perception of a perspective and pan from side to side finally it can look kind of cool to keep the input signal on the monitor rather than completely tearing it off when it's broken so how do we efficiently add these type of effects to the game world procedural development is something that is seen more and more in our industry the main selling point for making assets procedurally is the flexibility that comes out of it it's simple to manage and change the graphics or behaviors halfway through development if the if the asset was designed for being changed in the first place also this helps keeping the graphics consistent in the

game often it aids memory performance having assets being reused for many purposes and it is also fun and challenging making assets that are meant to be reused this solution was built for having consistent water puddle decals with ripple interactions in the game using a procedural approach previously the environment artists had been baking the pits sorry i've been baking the water plane into the texture and all the different versions of the water shading affected memory performance and decreased the assets flexibility so in this procedural solution artist would instead instead simply bake out a pit in the ground and have the shader fill it fill it with water the acid supports having interactive footsteps and gunshot ripples caused by the player and npcs as well as environmental effects such as dripping

from a broken pipe it gives the environment artist the ability to change the water height and mud color procedurally you know the pla the water plane also has wind effects and a simple refraction and a very rough visual effect of sinking into the puddle by setting the depth bias of the decal high to properly understand how this effect is created it's worth taking a quick look at the engine the solution was built for the decal system in glacier the engine used to develop hitman offers a flexible way of working with decals because decals can be transformed in uv space per instance in per instance in uv space which allows for optimal use of draw calls and for having many decals inside one texture they also have vertex colors which the artists can set per decal and this is what i've been using for

driving this these procedural effects they can also be placed in a template along with other components such as scripts collisions and volumes what all of that means is that we could have the player the artist place puddle decals inside a level like they normally would and automatically spawn ripple effects when the characters walks into them and even find electricity current now i will show you in depth how a puddle can be made procedurally through this interactive demo and it will get a bit technical what you're looking at is a puddle as well as a slice of it allowing an access for visualization purposes artist can set the preferred water height by changing the decal's alpha value

we use a height map as the data for driving this effect if we look closely there's a small gradient between there being water and no water and this the creation of this gradient is the base for making this work so let's have a d let's get a deeper understanding of of how it's created i'm using something called a remap function and don't be intimidated by the code this this function can easily be created through shader graph nodes it's just not very presentable the purpose of the function is to map a certain value to another under the code you can see an example of what happens we're running the function so we start by having this gradient we pass the gradient as the first argument of the to the function

and values between zero and one as the second and third argument and the result of this is that we get a new gradient that is more narrow so instead of passing a gradient to the function we'll be passing a sample height map and then we'll pass the user input which is the value that art is set on the puddle mine is a the width we wish the water line to have and that is the same as this width to large width and we'll start seeing illiciting the water will start to look undefined and too small and we'll start seeing aliasing and don't pay too much attention to this value this just happens to work well for this application the last argument will then be the unmodified user input value and that will give us a gradient between the two that's all very cool the puddle will shrink and grow depending on whatever value we pass as this first

argument but we can do a little bit more with this we can also add mud coloring or perhaps a wet area surrounding the water and all of this just by repeating the same code but with different offsets now onto the interactive part we were inspired by naughty dog and the way they use render targets which was presented in the paper the technical art of uncharted 4 in 2016 i think so mean jonas meyer the lead render programmer at interactive we decided to try out something similar so what you see are sprite particles spawned into a top-down buffer it's basically a camera rendering a few defined assets in the game and these in

this case the ripples the camera is placed above the player looking down so why does the render target look like vomit well the clear color is set to the neutral vector in texture space which happens to be this color on the sprite particles we simply write a low resolution representation of ripple normals we also write the height value of the ripple to the blue channel of the render target we can sample the render targets inside the puddle shader as you can see here the water plane is disturbed by the ripples the water the height changes we achieve this by adding the the sample texture the the render target to the user input and scaling it by our preferred value

then we replace the second and third inputs of the water and mud levels with our new disturbed water level and and um notice note that we choose not to do it for the this wetness level and that is simply because we don't have any delta time for making the wetness sort of crawl back in a realistic fashion as you would expect when wet ground dries up you notice that if we set the water level too low the ripples will start to look a bit strange kind of like if we're as if we're adding water to the puddle which we are but if if you think about characters for example walking through the puddle that wouldn't be the case so we can fix that either by scaling the the ripple height by uh by the user input or completely turning off the effect when we think it fails

after this we can start using our gradients for something we set the color of the wet area surrounding the water by multiplying our color sample the ground color with inverted porosity assuming you don't sample a porosity map and one more physically accurate results you can replace the porosity constant that i have here with the inverted roughness of the underlying surface of the ground perhaps that sounds a bit complicated and confused but it's really quite simple think of it as plastic and concrete the water would simply drip off plastic but be absorbed by concrete and that is what we're trying to model here we then want to find out how bright the mud color is so that we can define dark colors as

transparent because as you may recall we were using vertex colors as the input for the shader and this means there is no fifth float which we could use for defining transparency so we will have to define transparency in another way and we will do that by having dark colors represent transparency basically if the if the artist would set the mud color to black we will we will read this as fully transparent water the dot product of this value here does exactly that but those values may vary a little bit depending on your rendering pipeline note now that we know the brightness of the mud color we can interpolate between the wet ground and the muddy water we also have to unpack the ripple normals since the texture the render target is red in the shader is an unsigned vector and we

need it to be sine and that just means that it goes from zero to one instead of negative one to one and we do that by multiplying the sample by two and subtracting one the set value is set here for to one for simplicity but you could recalculate a more accurate set value from x and y if you require we have now finished all the steps of this effect and we just need to composite all our layers much like you would do in photoshop and in code that's easiest done through lerps puddles are great fun for the player but another amazing thing we can do with this tech is having the player disturb the local plant life what we want to achieve here is to have the sphere bend and flatten the ferns as it rolls through the vegetation

this can also be achieved using a render target as we see here to understand what's going on here let's start looking at what we write to this render target there are two elements which we write to this render target the first one being this single quad aligned with the sphere's position on the quad we draw a vector from the center and outwards and along with this single quad we spawn sprite particles and give them the velocity of their parent these particles are shaded using the particle direction vector and the lifetime of the particle as well as a sine wave for swaying the vector back and forth like this and that is to fake internal movement and delta time for the bush to reach its undisturbed

state in a bit more real real realistic fashion we use the particles lifetime to drive the sine wave which results in faster swaying in the beginning of its life and slower swaying in the end of its life so if we have another look here at the render target you can see that the particles are changing color and that is because the vectors are swaying so to speak the blue channel of the particle will will be used as a mask for flattening the plant in the plant shader we will sample and unpack the render target in a similar way we did for the puddle now instead of using the vectors for s surface normals like we did for the puddle we will be using them for rotating the plants

so as you can see here if the sample color is red we will be rotating the plant in the positive x-axis and in the absence of red we'll be rotating the the plant in the negative axis when sampling the render target it makes sense to think about what coordinates we should use i found out for this plant the rotation vector should mainly be should be sampled somewhere between the pivot and the vertex position but this is mainly down to the shape and size of plants you can almost see this blend value as the flexibility of the the definition of the plan's flexibility basically so if i set the flexibility to zero it means we are sampling at the pivot position only and that results in quite a rigid look and the intersection between the sphere and fern becomes quite noticeable however if we samp if we sample the render target only at

the vertex position the well the vertices aren't very unified together it kind of looks like water or something in fact they're not unified at all they're completely independent as we see here so something in between these two extremes will work for most situations so however it's it's worth mentioning that if the plant was very big there would be no sense in sampling at only at the pivot position and in these cases it may be worth looking at introducing baked pivots so if the plant look like this we may want these vertices that are painted blue to share a pivot to rotate around

and the easiest way of baking this data is to write the pivot positions to two uv sets and which will leave that last entry empty but you could also write the values to vertex colors or textures it's just gonna make it a little bit more complicated to map those values out in the shader similarly we would like these vertices to share a pivot and so on anyway that was a side note now back to our plant and in particular what we need to do before we start rotate rotating our positions so once we've created our coordinates we can go ahead and sample the target the render target and unpack the vector to assign range just like we did for the puddle and we could also recreate the set rotation in this crude way of adding x and y and this is absolutely not a true rotation by any means but it makes the plant look a little bit livelier when it sort of twisting instead

to know the amplitude how much we would like to rotate the plant we will also be using x and y as a base the amplitude can be created like this it is important to normalize your rotation vector and since we don't want this to happen i added a small number to the vector however this is not the right way of avoiding none if you want to be absolutely sure you don't get none an if statement is required but for the sake of fast performance we can sometimes take that risk in this case however it's completely safe seeing as the smallest possible value of the texture is larger than our addition here so if you're wondering what the visual output would be of of nan imagine flickering or similar crazy behaviors i won't cover how to do the actual

rotation here but you could find plenty of rotation functions online or otherwise you're very likely to have one in your engine already so what else do we need to do to achieve this effect well we also want to flatten the plant and we can do that by using the blue channel of the render target as a mask to flatten the plant we need to know how tall the plant is and either define this in a very simple per plant way like i'm doing here or in a procedural way by getting the object set bounce that is the preferable way by the way next we need to make sure we only flatten the plant where the flattening mask the blue channel of the render target allows now we can add the flattening to our position offset this will have vertices that are being masked as flat to be moved in this case

20 centimeters downwards but that would also mean that vertices can pass through the ground which probably is not what we wanted the desired result is of course that the plant collides with the ground and so let's fix that in one second depending on which engine you're using and if you're outputting um an offset or an absolute position in the vertex shader this step maybe may or may not be required but in unreal engine which this demo scene was made with this is required so we temporarily want to add the world positions to our offset we can then define a ground height for the plant to collide with this is probably just above the pivot but it could also be a terrain height map if you really wanted that extra super fidelity i can also recommend adding a downscaled rb space height to the ground height because otherwise you will be experiencing quite a bit of said fighting when all the vertices are

compressed to a single height value so now that we sorry now we can make sure that the leaves will never go under the ground by using a max function which will output with whatever whichever input it's is larger and finally and depending again depending on your engine subtract the temporary position which we previously added so in summary how to disturb a plant create a render target draw some vectors and a mask to it use the vectors for rotations and a mask for flattening it now on to something a bit different traditionally we would spawn decals on a short surface and it would look something like this

yes these are not the most beautiful textures and but in general this approach looks all right and with a few texture variations it will be difficult to see repetition however the decal often appears to be floating on the surface and appears like the stamp it really is so i think we could do a little bit better by using this method which i call the destruction buffer we can get get a bit more interesting results the idea is that we draw decals again at the impact point of the surface just like we would do with normal details but in a separate pass we can later use this buffer in our materials to create more unified destruction effects and mind that changes to engine code will most likely be required

in order for this to be fully implemented however anyway we spawn and draw decals to our standalone decal buffer and this is what the buffer looks like in the wall shader we read the buffer and use the grayscale value to peel off the tiles the data describes in more detail how the tile should break is stored in a texture which looks like this what this texture tells us is that dark values will peel off easier than bright values we start by mapping the destruction buffer to the screen we then we can then sample our buffer define a width of the peel effect and finalize the peel mask using a smooth step when our peel mask is created it

we can blend between the two materials tiles and concrete in this case but we also want some normals to shade the surface where it's broken and since we don't know how the surface will break we cannot we would have to define this in a procedural way create these normals in a procedural way and to do that we will be using the hardware functions ddx ddy and these functions are really cool because they are aware if you will of the neighboring pixels in their current state by using them we can convert our grayscale peel mask into pretty decent normals and this is kind of the quality you can expect from creating normals in this way they turn out a bit blocky but it's still usable and we can have a closer look here in real time as well

you can add an additional layer of fidelity by having the concrete in this case break even more as the wall takes damage i don't know if you've already noticed but it's worth to point out that it's quite forgiving on the eye when to see these type of decals fade off so let's revise how this wall became broken we created a separate decal buffer aka destruction buffer spawn some decals into it and then peeled off the tiles using by combining the destruction buffer and a texture defining how the material should break so that could be either tiles or wood or whatever you are about to break

now on to something quite different actually not because we kind of saw this from the earlier presentation so this is one way in how to solve the problem of having liquid placed in a moving container in this case a glass there is no fluid simulation going on here it's simply a few tricks put together and these are the basics for achieving this effect start by filling the container with liquid and by that literally i mean literally create a closed liquid mesh inside the 3d model set all the vertices above our defined liquid height to that height you will now notice that some of the

vertices will end up outside the glass now we'll just tilt the glass to get a better idea of what's going on and ensure that all vertices stays inside the glass and we'll get out into how this is done shortly but basically it's a matter of finding the nearest point inside the glass and that is going to be the radius of the glass now is a good time to tell you that this effect only really works for cylindrical containers to find the nearest insider point of other shapes is much trickier you may also notice that the plane starts to bend slightly and i just didn't i decided not to fix this because it's sometimes it kind of looks like surface tension and that's a cool side effect finally the fourth step is to fade off the effect when it fails and it will fail in this angle

okay so that's the idea but let's run through it in greater detail first we must define the essentials the height marked as blue represents how fill the glass is the set bounce of the contents marked as green and the maximum radius of the contents marked as red next we make sure that the height value can never go below the lower bounds what will what we will try to do now is to force all the vertices above our height value to that height however the height that we defined is on local space so if we rotated the glass the vertices wouldn't be aligned with the horizon which is the whole point of the effect so that's why we need to con transform our height value to world space and this essentially creates an infinite plane and now we can force the vertices to our plane like this

when we've done that we must go back to local space to carry on with the next step which is to put the vertices back into the glass because as you may recall from my gif earlier some of the vertices are outside the glass and we would like to put them back in we will do this by using the radius of the glass but how do we know the radius of the glass the radius is break into one dimensional texture in this case um in this case we the basically the way we do this is to create a height map from the tip of the glass and in other words measure the distance from the plane here to the liquid and invert that result once we've done that we can create some we will sample our textures but first we need to create some coordinates

we will reuse our remap function here which we previously made to map the height to a zero to one range so in this image you can see that the set position which is this case in something like 60 turns into 0.5 in our new range we can then sample and multiply the radius texture sample with our maximum radius and that will [Music] which we defined on our first slide now before we fix our little problem of having vertices outside the glass we need to know if there are outside the glass and we can learn this by measuring the distance from the vertex to the center of the liquid and comparing that distance with the radius we are now ready to put the liquid back into the glass by creating a direction vector from the center of the glass to the vertex and scaling

it by the radius of the contents and that way we will see that happen and i mean don't i don't mean literally like this but this is just a visualization but basically what what we're doing is to place the vertex in the center of the liquid and scaling it outwards using the radius to fade off the liquid when the container rotates too much you can use this septo angle it could have been made in many ways the important thing here is that it takes the orientation of the glass into account so so that the waves are going in the right direction

again to summarize fill the container with liquid force all vertices above our liquid height to that height put the vertices back into the glass using the radius and fade the liquid off when it fails i realize that this may be a lot to take in so you can download the slides and the demo from this link today we've been looking at cosmetic interactions passive interactions and payoff interactions we've been talking theory and looking at various implementations the facts i've been sharing with you today may not be applicable to your project but i think it's well worth mentioning it in the context of visual effects so hopefully the methodology or functions will be useful to you i just want to say thank you to everyone

who's been who contributed to making this happen and thank you very much for listening [Applause] questions uh hi hi great talk by the way uh in the example you did with the do the grass uh you said you were using a render the render target where the render texture to create all the the information that the shader is going to use to deform the plants right how do you account for bigger levels for example in your example there was a small square with about six plants but if you got like a giant stage with a lot of plants how do you account for that yes so the way it works is that the render target is actually following

the player so you wouldn't see the formation of plants happening in the far distance you would only see it close to you maybe 50 meters at maximum so everything is like on a world scale uv something like that it is it is in relation to the world uh the the camera is in relation to the world yeah so thanks yeah hey so i just want to piggyback off that um with all your render buffers you you have a lot of them just reserved for my new details in the game did you at any point do you guys run out of memory for your actual render fetches so in hitman the render targets weren't very big they were about 512 pixels and we were reusing them for multi-purposes so they weren't just used for plants or

water but do many things and we don't need to have many of them basically cool thanks anyone else oh cool thank you very much