Can We Do It with Particles?: VFX Learnings from 'Returnal'

[Music] hello and welcome to this presentation about vfx learnings on return my name is risto and i'm the lead vfx artist at housemark and with me is sharman he's one of our senior graphics programmers and just a little bit about the company as well osmar was established in 1995 and before eternal we were best known for arcade tile twin stick shooters such as deadness and rezagan and nex machina and return was by far our biggest project to date and that was released in april last year and we have been working with sony for quite some time and since last june we've been part of the playstation studios as well so um this presentation is about how we utilized our own vfx toolset to create some unconventional stuff for eternal

by unconventional i'm referring to things like vegetation character effects like tentacles and volumetric environment effects with fluid simulations basically things that you wouldn't expect to be done with a particle system in addition to those things we also produced the huge amount of conventional effects with regular billboards and point particles and seeds those are probably familiar to most of you in the audience already will focus more on the unusual stuff during this presentation okay a quick overview of what we're going to talk about we'll introduce our proprietary vfx engine briefly i feel that the tool set is quite unique so it's essential to understand how things work before we present what we did with them and after that i'll present particle hierarchies and charman will present the last segment about external resources and simulations okay

so a little bit about the history of our vfx engine we have been developing our own particle system since 2014 back then we used to have our own game engine as well already for resogun the team had made some particle effects with compute shaders and based on that work we started developing a particle system for alienation we moved to unreal for the production of matter full and boarded our vfx tool set to unreal at the same time as well and design wise we tried to focus on flexibility and performance all the effects run on tpu only and everything is fully programmable in hlsl um while we're at it let's go through some definitions as well so in our system when i say a particle that's a single particle it's just a text file with some hlso code in it so in order to create effects for our ngox we have to create an empty text file and start typing code

an effect on the other hand it's a system of particles um that reference each other for example an emitter that emits other other particles the engine itself does not distinguish between particles and emitters any particle can be any meter or not and finally we have the vfx engine that runs the wholesale it keeps particles alive and discards test particles and does other useful things for you the vfx engine itself is a plugin for unreal previously we just called it a particle engine but since it does so many other things as well we decided to start calling it the vfx engine so um there's going to be a bit of code i'll try to keep it as brief as possible since i'm aware that this is an art summit so um let's create a simple particle effect for our engine just to give you an idea how it actually happens as i mentioned you start by creating a text file and inside the text file we need to define what particle is but we also need to define what the particle

does and what the particle is is defined by implementing a thing called particle struct those things that you see inside the struct are particle variables variables are the data that the particle can store and modify you can add as many variables as you want to and you're free to choose their data types and names typical variables would be what we have here like life lifetime but also position velocity size color or whatever you can think of there's no practical limit to those and also the naming is free as i mentioned earlier but uh actually none of these are required so basically you'd have to you're going to survive with just the dummy variable there as well and after we have defined what particle is we can proceed and define what the particle does this is done by implementing an update function here you can see that the particle we just created is passed to the update function and we can modify it

there in this simple example we just accumulate the delta time to our particle's life variable and when it exceeds lifetime variable we choose to exit the update function not and not to keep the particle alive anymore and we're not even rendering the particle here the render component is optional and we'll have a look at that later the system supports all hls of features branchings and all the intrinsics are there etc and this allows us a great deal of flexibility for example it's trivial for the particle to handle multiple different states and execute update logic based on a given state okay what we have here is actually considered a particle in our vfx engine so if we were editing this we could press save and the system would create an effect graphical user interface for us this is a place where other team members and artists can do modifications for effect for example you can do material assignments or you can edit animation

curves for example if you want to have a curve or let's say color over lifetime you need to declare it in the shader code and it will be available for others via this asset and also you can add color pickers custom tweakables and everything but everything has to be declared in the code so we don't assume that anything exists unless you actually explicitly announce it in the code basically what we have is a small custom graphical user interface per effect in return we didn't have this so this is part of the learnings from eternal and not having this was actually a major bottleneck for the production because all the modifications have to be done by people who understood the shader code okay so basically writing that file and editing this graphical user interface is the whole process of doing an effect in house mock in its simplicity you start with a blank file write some code and the system produces an interface that non-technical people can tweak or modify

okay okay now we can move on to particle hierarchies which is pretty much node particles in our system let's talk about a bit about them when we started developing return returno node particles were a new thing for us they were developed for branching vegetation in another project but they turned out to be very useful for ethernel as well the concept is really simple node particles are just like any other particle in our engine but they can all read the parent data that the parent actually has and parent can have multiple children but salt can only have one parent and parent and child need to be the same particle type and we ended up using these nerd particles a lot we used them for ribbons vegetations tentacles and all kinds of stuff so as we're about to see

so let's have a quick look at the node particle code let's create some of them and see how we can utilize the parents out relationship that they have our goal will be to create one of the tentacles that we had on many of our enemies in eternal first we need to tell the system that that particle is a node particle that's done in the header so that system knows to include relevant code for it we can do the particle definition as usual and then in the update code the engine will provide us a function for getting the parent particle after that you're free to do whatever logic you want based on the parent particle data in this case we're just checking if you look at the code we're just checking if the parent is more than 100 units away and then we move the child particle closer to the parent if through okay next we're going to have a look at an animated example how this can be useful

for what we can see here is some node particles moving around and spawning child particles as we can read the parent particle data we can draw an epoch line from child particle to the parent particle this looks a bit messy as all particles are moving independently let's add some constraints to their movement first we'll limit the distance that the particles can move away from their parent and that distance is visualized by those green circles around each particle now let's apply uniform velocity to the left of the screen and start slowly rotating the velocity and damp it towards the end of the chain and finally we can apply some curl noise to the movement now we have something that we could use as a base for a tentacle motion in return but it's still

lacking on many areas the rendering being the main offender here so we have the motion but we need to be able to visualize it somehow we considered multiple options like quad strips point particles and separate tube render but ended up doing it all from a particle update function so how did we do that on top of the usual render types that one might have in particle engine such as brights oriented quads we also added triangles the two key takeaway things here are we can define the triangular vertex locations freely and secondly we can render as many triangles as we want per particle in the in our engine we define render types in the particle code by implementing a thing called render struct it works similarly to the particle struct we saw earlier but the user can't really choose what kind of variables they put there i'm not going

to go too much into detail here but it's enough to say with that kind of printer struct that you see on the screen the system already knows that i'm trying to render a triangle and if the update don't function we currently have a loop just just creates 10 triangles and places them side by side okay and there you go if we would execute that code what we would get is 10 triangles as promised it's not too exciting so we'll have to dig a bit deeper but so far we have established two important things for us first one being we can read other particle data and secondly we can render as many triangles as we want per particle now we can proceed and build a renderable mesh for our particles first we need to think up particles as control points for a curve and if you have a curve you can also have tangents those are the red lines if you have a tangent for a curve you can also have a normal for the curve as well and those would be the green lines

the only thing we need to not do now is rotate the normal around the tangent we can use the blue dots resulting from this rotation as vertices for our tentacle mesh then we can just proceed adding triangles to those positions as vertex locations okay that's pretty close to what we have in the game just we just need to change the thickness a bit and swap the material i know this is a pretty standard way to do tube rendering so nothing new there but the fact that we can do all of these things per particle and on the gpu makes it perform really well like here basically you can have all the tentacles in the world and it's not gonna cost you an arm and a leg in performance and since each tentacles is basically a bunch of particles we can control their behavior and implement things like player collisions and fluid velocities really easily i can't recall how many tentacles this particular scene has but i think this update cost something around a millisecond i think this is a bit of an

excess amount of tentacles i don't think we ever have used that in the game okay so we ended up using these things in the game quite liberally since they performed well and we were able to program their behavior we attached them to enemies and had the enemy ai drive the tentacle state so that they would for example grow and become emissive before the attacks and since they performed well we were able to use them as vegetation in some rooms as well and that was really nice because we were able to use deployed sim and the player collisions to drive the vegetation behavior and each grass plate can have its state so we didn't have to jump through hoops trying to implement their behavior we have vertex offset or something in this last example we had the player head attract the pla the vegetation so it would look kind like the vegetation is inspecting the player

i think another good example of the flexibility of node particles is that we also use them to do simple destructible objects for eternal destructibles had to be pre-processed in houdini in houdini we first split the object into two pieces interior and exterior and after that we do a simple voronoi chatter on both of the resulting meshes then we export the mes2 unreal and in game we create a single particle per mesh piece each of those particles can have access to a lookup table that contains the start vertex and vertex of that piece that way we can tell each particle to loop through those triangles and render them so in game we first showed the intact static mesh when the object is hit by a bullet we hide the static mesh and so the construct the mesh constructor of particles for visualization purposes we're solving particles of screen dots here each of those particles render a segment of the mess that is assigned to it

when the destructible object health reaches zero it's destroyed and at that point we can start animating the particles we make the particles that are below the heat point fly away explosively and particles that are above the hit point get grouped together under a single node particle we can then animate that particle visualized by the green bounding box and the child particles will animate with it this gives an illusion of that the big piece is actually a solid single solid object and well let's see i'm gonna bring the video a bit again i'll let it play so when the big piece is moving we can query the sdf collisions in each of the child particles and detach them from the group on collision okay i call this method scripted destruction it's not as generic as real rbd but it's very fast i also think that there are several interesting opportunities for improving it one could

for example export the connectivity data from houdini and simulate node particle collisions with that information but basically we used it for destroying mechanical enemies and lots of statues and in the bottom right hand corner we even did a whole boss with this destruction system so it's just basically a bunch of pieces that you can move around with particle code okay that was the end of my part and next sermon is up and he will present us external resources and simulators yes hello everybody thanks risto for doing the first part of this presentation and risto didn't leave me any tentacles but we'll be looking into some other other things such as fluid simulations and volumetric fog how we solve those things we use external resources let me elaborate

by resource we mean a gpu resource it's something that the particle can read our particles are explicit in nature they don't do anything on their own for example if you want to emit something from a mesh you will need to bind that resource like mesh resource to the particle and then manually look up the information from uh from from those buffers so the buffers could be let's look at what they could be we can have 2d textures 3d volumes and as i mentioned bone buffers vertex buffers index buffers tangents and these kinds of things i think the best way is to approach this using a real example that we used quite liberally in the game we have a real-time fluid simulation running at all times in the game we've used this technique since alien nation which was launched in 2016 so

there's always at all times a grid-based fluid simulation following the player that the and the art artists are free to sample from that fluid simulation and have the vegetation for example react or point particles or what they want to have there we can also have dedicated fluid simulations for set-piece effects and throughout this the rest of the remaining of this presentation we'll be watching a lot of videos and i'll be commenting on the tech on those things so we can start with the player following fluid simulation those red arrows indicate how the velocity field is currently we can rasterize forces into the into the field we can also sample sdf and rasterize objects into the field to get interesting motions here's an actual actual example from the game where the vegetation as as well as

the volumetric fog is using all of that as you can see when the enemy shoots at you the vegetation as well as the volumetric fog will move accordingly using resources is very very simple uh our our vfx systems are such that they you can think of them as modules they are not coupled in code but they are coupled with data one module can produce some data for example a the fluid simulation gives you a volume which contains the velocities and you can feed that into the into the next next module for example our particle engine so in here can you see the mouse cursor i think you can but the idea is that you set this kind of a we call it fluid vel so fluid velocity here and we grab it in that second line you can see that that's

what we call it and we use again our meta language to expose these things to the gameplay side and we we're free to sample from this velocity field in fact we can even write to that velocity field from the particles and with this code well it's pretty simplistic but it actually doesn't take too much to create this kind of an effect uh we're just spawning a bunch of point particles in the middle and we have a constant it's kind of a repetitive fluid impulse blowing towards that direction where you have that wall and because we're automatically rasterizing all of the all of the environments into the obstacles you'll have very interesting looking motion there it's super simple to do as i said we can or i don't know if i said but now i'll say we can use multiple resources we can have for example a fluid velocity field as well as a mesh buffer

bound to the sim to the same particle and you can do these kinds of things in this video you you will see white point particles attached to this mesh and once shot at they just get detached from the uh from the uh from their positions and they follow the fluid velocity field and after some kind of a cool down they will try to figure out their way back to the original position we do have a example from the game we yeah there's trigger area you walk past that and you get these we call these murals there's a better term that i already forgot but the i always call these by the names that we come up with them from a technical point of view but well mural isn't technical anyway party goes look at them we let's talk a little bit about volumes because we use those

a lot uh in resogun as well as next machina the whole world was constructed of volumes here we take a different approach we use them using our resources and we get volumes from uh fluid simulation we also have a real-time voxelizer that will be able to voxelize static measures and skeletal meshes but we need to understand what volumes are they're basically 3d textures right in a 2d texture have pixels here we have boxes this video shows you how you can access particles each point particle represents one voxel and they have an id it's a flat number running from zero to how however many voxels you have and that can be converted into a voxel or a world position i'm not sure ken can you even can you read those numbers but there are numbers um we can use this for example for density fields so the particles again here in this case

we are writing to the gpu resource previously previous examples showed how we read from them here we are writing we are free to write to whatever boxes we just have to take care that we don't have multiple multiple threads writing into the same voxel this is an example showcasing how we use volumetric fog in our case on the right you can see a particle representation of that volume and on the left you can see a volumetric fog that we make out of that the idea is that you spawn some density based on the sdf and once you shoot at this thing the fluid simulation will advect the particles and you'll see particles higher up also uh the artists were able to place these volumetric fog components in the level using this kind of a very crude visualization and once you press play it looks something like that so the setup was pretty simple and we use the player following fluid

simulation by the way in this so it's very affordable just write your particle code that does the advection and you for you basically read from the fluid velocity and affect the density simple right another use case we can of course make a mesh out of that density field we use marching cubes to extract the extract the triangle mesh and we get that kind of a pie but we also have in return i think it was within the first five minutes if we're good you could reach this room which we call cavern there's a pit of fog and there's a sphere and whenever i watch a video of people playing this game they just walk past that sphere and it's i hate it there's so much tech there's so much tech behind this thing like there's okay i'll explain i'll just use the rest of the time on this slide there's like there's a particle effect that spawns the density

there's a fluid simulation dedicated one that drives i mean affects that density field that gets fed back to the fluid simulation that gets fed back to the marching cubes and people walk past it the gpu will perform equally well if you shoot at it or don't respect the spear shoot at it as i mentioned we have a real-time voxelizer on the right-hand side we have a skeletal mesh on the left-hand side we have a particle present a representation of that thing and then we construct and this kind of isosurface out of it and we just use a translucent material on that it's very flexible again data only and the particle code i won't be showing it here but it's super simple sample from this voxel s result spawn some density feed it to fit it to fluid velocity make a marching cube surface out of it

done and as rista already mentioned in one of the bosses we used it on that skull that will appear three two one now and again shoot at that thing you can shoot at it it has a dedicated fluid simulation there and if you shoot at it you will be rewarded with this thing it won't do any damage so you might actually die but it's worth it i'll wrap up now so the initial question was can we do it with particles i think we can thank you very much and let me say this one thing we love to talk about this our own vfx and bfx in general so we invite all people who want to come into was it

to the wrap-up room it was maybe 204 or seven four either one will be in one of those okay thanks [Music] you