Visual Effects Bootcamp: The Rise of Realtime

[Music] visual effects in the rise of real-time that's our talk if there's one thing that I want you guys to take away from this talk it's that real-time visual effects it's expanding beyond games it's never been easier to get started and let's build the future together so Keefe and I have a whole lot to say and all these topics like I said we'll probably have too much to say so we're gonna burn through it overview of the broad sections we're going to be covering why of the effects bootcamp now that's why we think the timing is perfect for this for this event for today Keith's gonna speak to building our community in ways that we can do that better and bigger he's going to talk about visual visual effects gems from our community I'm super excited for this section then I'm gonna take it in a little bit more of a narrow focus to talk about tilt brush case study taking old tricks in new places and then we'll take the 10,000 foot view and try and

get a picture on where all this is going so why have the effects bootcamp now well we're at an inflection point that's the most that's the most important feature of the landscape of visual effects right now there's this really dramatic inflection point and that's driven by a number of factors one of the most significance is this powerful common baseline for tools and engines we've got things like unreal and unity that we take for granted now at least I take for granted but if you think back a few years ago actually wasn't the case and my work in tilt brush in particular tilt brush is built on unity we wouldn't have left our job and games and pursue something different if unity didn't exist so these these tools are already changing the landscape and changing the options that are available to us every year at the visual effects roundtable we do one of these show of hands he was looking for work and who's got an open position and every year it's absurdly lopsided there's never enough people to fill the open positions and if you

believe the high-level concept behind this talk that problem is going to get a whole lot worse there's also a platform explosion and we'll be revisiting this again at the end of it thanks to Matt Radford for the awesome stylized explosion another important factor to call-out it we started this group Facebook the Facebook group real-time VFX maybe 2012 I think it's when we started it and it's blown up it's almost two thousand members it's drawn really really really fast kind of in response to that my amazing co-presenter Keith Guerette he took it upon himself to build us real-time VFX comm which is a community a place online where we can come together and we can share our ideas and we can grow together I'm super super honored to get to you know share the stage with him if you're not familiar with Keith's work he was the lead VFX artist at Naughty Dog he did some notable titles like Uncharted 2 3 & 4 he worked on Last of Us he's been

active in the community since it formed sharing his experiences and his knowledge with us recently he co-produced the game makers Inside Story he did that with Jenny Kong who's a writer from film moving into games it's a ten-part webseries about storytelling in games it's free on YouTube right now should definitely check it out and he just launched his own studio but beyond beyond effects so visual effects studio in LA it has a super intriguing business model that very much relates to everything we're talking about today we don't have time to cover it right now but if you're curious be sure to ask them about it in the QA and so now yeah it's my honor my privilege hand it over to Keith so I prepared a few notes I wanted to go over all the cool things that people have been doing in our community and sharing with us and it's it's a long run no the truth is that I wanted to take it

a few moments basically just to express all the cool things that that everyone's been sharing with us across all the different platforms that we have to introduce the subject a little bit more though I want to take a step back and just discuss where we're at with effect I think one of the takeaways you guys are going to see from this week at GGC is that everybody across every little facet of Industry is coming up with freaking insane solutions for random problems and yet at the same time our roles are changing along with that we're no longer just doing particle simulations we're also doing illustration simulations we're rigging we're programming we're compositing we're building tools we're animating we're modeling the tasks that we do daily is changing at an overwhelming if not inspiring and breathtaking rate directly in front of us and yet we're constantly expected to keep up with it every single company that I've been to

is trying to reinvent the wheel over and over and over again and maintain that competitive edge yet we haven't really been doing that great of a job sharing it and that's kind of the reason why it means so much to me that drew started the VFX Roundtable five years ago six years ago right we finally had a platform for us to come together and discuss all the pains and woes and awesome solutions that we were actually experiencing we could start to plan out the future of where our tools should go where are our hopes and dreams and ambitions as effects artists should be taking us at the same time David Johnson and a few others founded the visual effects group on Facebook and we finally had a place to share just badass references cool explosions the actual solutions and tutorials around the same time other artists like Jayson Kaiser were creating amazing YouTube channels which to this day is probably the best single resource for learning how to do illustrative effects that I've ever found one of two in fact and view effects is the only other place that I can point students do when they're learning how to do effect unfortunately bill

cloud has moved on to bigger better things with with epic but his resources from and view effects are still some of the best and so through all that I spent time sitting here wishing there was a way to Google this wishing that there was a way that I could go online and just go to google and type in UV offsets per particle in unity and have it actually returned something useful like I wanted a way for effects artists to find resources to the challenge that they were facing and at the same time I feel like businesses should be able to target us as effects artists to to run into our community other artists that wants to make tutorials should have an audience somewhere that they could go to to find it and so we created real-time view effects it's fully Google it's fully fully archives and that's freaking awesome so what I really want to be up here talking about is the fact that this is our community we made this for us we're not trying to drive a profit out of it we're trying to enhance the visual

effects industry because that's the career that we're all excited about that we're all here for in that same vein any opportunities that you guys see out there please reach out to us we want to help you make that into something that we can all use so for example Jason Keizer approached me about making a jobs board and we've launched this it's still got some ways to go but we're trying to make a resource for people for recruiters to find effects artists before effects artists to find amazing jobs Nathan Lane has been going through and actually building a community challenge concept where we're doing monthly effects challenges similar to like the 30 second animation Club but for effects for us to step outside of our comfort zone in that same vein tonight we're actually gonna be announcing our very first ever sponsored community challenge sponsored by popcorn effects and side effects so at the mixer tonight we'll have all the full details on that but the point of all this is to say that to all of you from us please get off your asses help us make this better

our entire industry is about to explode that's that's going to be the the caveat the the summary of what Drew's up you're talking about and we have such an amazing opportunity in front of us to make industry into whatever we want so we're all here to help the other part of what I really wanted to talk to you guys about is just some cool freaking takeaways that I've been seeing coming out of the community over the last few years so everything that we share today is coming from a tutorial that somebody posted online a post even just mentions in answers to questions across the Facebook group the forum Twitter everywhere I have a ton of content in here and I'm obviously skimming thousands of posts for for other things so the takeaway here is that these are all really cool innovations that we're sharing as a community and that are becoming adopted as industry standards to start us off with that just to set the scene if you think back like 2013 the PlayStation 4 and the Xbox want to just come out for us as effects artists

it was a big deal it yeah I mean obviously for the entire industry was a big deal but for us it meant that we finally had a little bit of wiggle room to start to innovate so we had you know stronger CPUs we had way stronger GPUs and we had a lot more texture memory of course we had absolutely no idea how to use any of it so we kind of lost our shits collectively over GPU particles which you know we could spawn just thousands and thousands of particles and make awesome effects really really cool stuff the other side of that that we also immediately stepped up to is that texture space like we could finally do a lot more flip books for those of you guys that know me I'm pretty vocal about the problems with pre-rendered flip books particularly if you're new and you're learning I don't really need to go into them but since you guys are my audience that I'm on stage I'm going to anyway so if you're doing a flip book you're bound by file size as soon as you are limited by a file size Oh as soon as

you're limited by a file size it's a lose-lose situation of the resolution that you're giving each individual frame of that file size or your frame rate so in this case I've got a 64 frame simulation that's lasting three seconds now three seconds is not long you're gonna be hard-pressed to find the flip book that needs to last only three seconds but with this minimal case sixty-four frames across three seconds we're getting twenty-one frames a second it's not if you're playing in a 60 frame per second game you will see that but you can see that each individual frame is fairly low res this is a shitty looking explosion right it's super pixelated it's not great so I could bias that I could go 32 frames and give each frame more resolution but then we start to really hit that frame rate this is on a 512 texture so of course if I didn't care too much about my texture memory I could go up to 1024 I could go up to 2k 4k whatever but then I'm burning through that extra memory and God help you if you want randomization so my general

feelings out flipbooks is that we just shouldn't do them unless we're extremely extremely careful about their pros and cons if you can be extremely careful about your pros and cons I do honestly believe that flipbooks are the single most powerful tool in our toolbox because there's just no other way that we can capsulate that much motion in detail so what I want to go into is just some of the cool ways people are improving flipbooks I'm pretty sure most of you guys have seen this this is from Clemen and he's showing a technique of using motion vectors to sort of blend between the frames of the flipbook so you end up with super smooth super smooth motion I know that the team at gorilla had been using this for a while and I'm sure a lot of other developers in odd pockets had but Clemens post I think was the first time that he collectively lit a fire under all of our asses to go and explore this this approach so what's actually going on here if we go back to flow and I'm gonna throw back to my own presentation just cuz I had these assets and it's very timeless GDC 2012

presentation flow or vector displacement is just a technique where you you tell each pixel to move in a certain direction over time so in this case I'm painting a texture here's me applying it right so you can see that I'm stretching that grid in very specific directions that allows me to do all kinds of cool and interesting things based off of textures but if we go back to this flip book and let's slow it down a little bit just we can see it what I can do is I can actually interpret vectors that I need to displace between these frames that they blur together so from frame a to be here the texture that that looks like is on the right obviously this is skimming but just trying to give you guys the idea there's a lot of different ways you can go about generating these quite easily Houdini has some there's a great After Effects plug-in called twixtor but right now my absolute favorite is a tool called slate editor created by one of our very own Alex Underhill so here's my little plug for him I I think the vectors that I've

gotten out of his tool are the best I've ever seen so it's got my approval if we apply that output to this when it plays finally hit a negative on this I'm gonna hop back then just to give you guys we'll go back to Clements example so what the explosion is doing and basically is every single frame of the flipbook it's getting the vectors needed to blend into the next frame and that's smoothly interpolating between them you can see that obviously the image on the right is way way better than the image on the left especially slowed down so this gives us a lot of flexibility in the way we're presenting our effects forgive me as I hop way forward now the

benefits of the motion vector flipbooks are kind of huge we can actually get away with super super slow flipbooks playing back and not kill our presentation of frame rate that in of itself is a massive solution for full ebooks the negatives however is that it does require for texture lookups because you need the a to be of the flip book and then you also need the a and B of the motion vectors fortunately those motion vectors can be small but there's some negatives associated with just sampling that many textures of course if you're looking for more resources clements tutorial on this is amazing he runs you through how to create this shader exactly an unreal so I'd highly recommend checking it out there's also a great discussion on the real-time VFX forum a little plug if you're naughty dog you can actually do this at real time which is kind of cool so you don't actually need to bake those textures you and actually instead of taking up texture space you could use GPU processing power to generate those motion vectors at real time and then smoothly have at it basically it's okay

that's that's how we've gotten around frame rate what other strengths can we use for flipbooks what about lighting I think one of the coolest notions that I kind of realized a couple years ago is that textures are actually just a way to transmit data if we take the flipbook not as an emissive render and instead consider the way that it fits into a compositing pipeline so a diffuse pass a normal map your alpha or density may be a temperature map you know take those passes that are actually used in compositing pipelines because our shader is nothing more than a real-time compositing pipeline we can start to do some really really cool things so I want to throw out to a post that the cloud imperium guys made actually mike snowden posted it but it's a video made by cloud imperium in general and it shows this incredible breakdown of like 20 different really freaking rad concepts one of them that i want to go over is just how they use data to pass into their lighting pipeline so if we just take a flipbook here which are we

starting I'll try I'll talk over this if we take an explosion texture and again if we don't take the render the emissive render but instead we actually render out in passes so let's say you render out your diffuse lighting you render out your temperature and then you render out your alpha all three of those are probably not going to need color they're just masks which means you can put them into one texture because you've got your red green and blue channels and then you can start to use that in interesting ways momentarily I'll show you guys how we approach normal mapping brutal well it's let's give for them imagine this was just an explosion playing that just has the red as the Alpha the green as the temperature and the blue as diffuse what we can start to do is well for normal Maps normal mapping of fluid is really hard right you've got all kinds of different challenges the way that I like to do this is actually not it's not a correct normal happen anyway it's it's kind of a render process but it

works really really well and it takes advantage of the like my pre-rendered tools lighting and light scatter so what I do is I render a light from the top and throw it in the green Channel and a light from the right and throw it on the red Channel and then I do the exact opposite with a secondary texture from the bottom and left and then what I can do is just using basic math which you can't do in After Effects you have to go to a new core right yourself like because you're going with negative numbers you need to be a little more complex about this but the math itself is not hard you can blend these and then once they're blended you can just compute the blue channel and you have a normal map I'm really hoping this video plays yes okay so this is nothing more than a diffuse render with an alpha and then that normal map created and you can see that I'm getting really freakin cool lighting the shadows look really nice like and that's because it's coming out of the rendered package in this case I'm pretty sure was Houdini so back to this video how would I use temperature I

think a lot of you guys have heard of black bodies or the black body shader and unreal it's basically running on the concepts that please play No okay it's basically running on the concept that the input number which means the single channel is going to derive a color value if you think back to science in the kelvin scale like there's with with energy there's a certain color output the black body is just a mathematical operation that when you drive a certain input of kelvins gives you this beautiful color it gives you really really nice explosions and you get that transition from fire into smoke accurately what we can also do is actually just use ramps so think of this like a lot where if I just take one single channel as an input just a mask of of temperature that can drive my texture look up and gives me really really cool highlighted areas on this loop so that's just the

temperature mask a single channel going into another texture which is the lot in this case it's actually the top left texture we also probably want to bloom it which is what the map on the left does if we keep going with different lighting techniques Matt Radford presented some really cool options for rain marching I know there's a lot of research going on from a lot of great posts of different ways that you can use rain marching I think there's even a really cool talk from epic this week about fluids in 3d textures with Ray marching so keep an eye on that RIM arching is a little bit more expensive but the results are astounding so I'm excited for us to keep exploring down this path one of the other cool concepts that I've seen out of the dice guys for battlefront with regards to lighting was this concept of the gnomon material dénouement texture where what they did is they actually took a pre-rendered cloud or smoke or whatever it is and

took those renders with really nice light scatter from Houdini and just baked them all compress them all into a single texture and then using these to texture inputs just lurk between them at real time based off the light direction it's kind of hard to argue with those results because you can take advantage of the full pre-processing and all you're doing is just blending based off of direction let's keep going let's talk about some vertex animation stuff Norman a long time ago posted this really freaking cool like fluid sim basically working in unreal where he stored all the animation as a texture I don't know how many of you guys are planning on doing chocolate bars in your game but this concept kind of sparked a lot of different interests and so we have people like Louis working at side effects figuring out great pipelines for us to compress animations into textures to use on a particle in this case the entire

simulation here is a texture that's running it's not an animation it's not joints and that means that these can be playing on mesh particles or individual particles here's a really cool fire soon that he did okay so these are kind of abstract Naughty Dog and a lot of other people are actually using them for allowing us to finally do what we've always wanted to do with birds actually have birds flapping their wings that aren't playing just a silly flip book right we can do full animations now so how does this work well if we go to math really quick and we consider just a position on this grid that's my vertex it's it's not really handsome enough or friendly enough so here's drew now if drew were a vertex and you were moving around playing his animation we kind of know what the outer bounds of this animation are gonna be right so if I wrap a box around this I know that the max bounds of this animation are ten and that means that I can start to do cool math of this so let's take one frame in this at this point it's it's six and X

and it's four and Y and the max bounds are ten so if I divide by ten then I get point six and point four if I normalize that I got a color value right like I can actually bake this number this numerical data into a color and a texture that means that I can actually build out an entire texture and play back a texture as an animation so in this case the vertex ID each individual vertex gets a column on this texture and time is simply the the y-axis of this texture so if we just play back this animation we can do things like goats as particles or crowd sims that don't need to be super detailed right like fish birds everything that we've always wanted to be able to play mesh animation on we can do this way it comes with one caveat and this is a caveat that I naively ran into and I'm sure most people that explore down this path do and that's that if you're moving verts

are round at some point you're going to move it far enough that you need to reconsider where the normals pointing and that complicates things so you probably gonna have to pass another texture for your normals but just as a general caveat so what else can we do with these concepts of vertex animations again the guys at dice man like they're coming up with some pretty freakin stellar solutions so the is Oliver Tech's animation playing on a single mesh particle one one single particle that particle looks like this or something similar it's just a series of quads with every single vert of a full quad having the exact same vertex color and that means then that you can do there's a term euler integrated equations on it like deterministic math basically to say that this quad here moves in this way and this quad over here moves in that way you can make all that out of Houdini and other badass tools and you can come up with some freaking amazing looking effects for really really cheap because it's all

just one mesh particle obviously your limit on interactions but as the core of an effect like explosions that have this many sparks probably look pretty good alright let's talk about some trails I've seen a lot of really cool improvements in trail technology lately one of my co-workers when my peers jae-hoon has been working with some really cool spline techniques so this is just a standard spline and what he wanted to see is can we art direct this a little bit more what would happen if instead of just the basic tangents of your spline you actually gave yourself artistic control of the entire silhouette of the curve and so he wrote this awesome tool that just gives you a profile curve that you can go in and shape now in this case it's geometry because it was an easier example for me to record but imagine this on particle trails and actually being able to break up the silhouettes and do really I don't know artistically interesting effects out of trails for once bilk lattice has

been doing a ton of really cool stuff with extruding custom profiles along geometry these he actually kindly gave me out of his presentation because he had to rip them but so you can see who's actually extruding a star shape or a human shape and ending up with really really really cool and results so that's my kind of swath of interesting new techniques that are coming out of the visual effects community I hope the take away from all that is just like share with us go in there and learn from us because there's so many really freaking cool things that are popping up everywhere that the rest of the industry is adopting and progressing together with other companies other artists but the real reason I think this presentation is going to be cool is because I want to throw back to Drew who has just a phenomenal perspective on the way our industry is changing I remember the first time that I used tilt brush

where you put on the VR headset and you pick up the controller's and you realize that your hands and the space around you is a tool and I think I spent about an hour just making something and I remember feeling really sad when I took it off and realizing it doesn't exist but it was such a liberated like an artistically liberating experience for me and so I want to throw this back to Drew to run through just his amazing retrospective on silt brush and take a look yeah awesome man Thank You Keith that's the first time I've seen a lot of that stuff that's amazing yeah it's the first time I've seen a lot of that stuff and I cannot wait to use it I'm actually gonna bring it up and some like want to happen some wish list stuff coming in this section but yeah so a beautiful beautiful like high-level view of all the amazing ways that everybody out here has been contributing and making all of our work great I wanted to switch the lens the kind of big picture I wanted to focus it

down on tilt brush as a case study so in this in this section I want to run through some more some more tricks that we use in tilt brush and really point out how all of the effects you just saw all the techniques you just saw are about to have like a much much much bigger audience a lot of different feels a lot of different places that these visual effects can go obviously for me tilt brush is the the biggest example of that so if you're not familiar with tilt brush it is a virtual reality painting application it's built for the HTC vive you wear a VR headset these controllers paint in space I'll get into what I mean by painting this is a prototype from our very first version back in 2015 it's a real fun from the very beginning it was a pretty fun project this is how it looks now I'm gonna play a little snippet of the video actually maybe I'll just play the whole thing how many of you have used tilt brush before okay awesome and does anyone not know is anyone unfamiliar with tilt brush a few

people okay cool so I'll play this it should have audio I think there we go cool so takeaway for this video is really just that painting in space it can be anything and starting tilt brush that's really the project we wanted to take upon ourselves what happens when you are in a virtual world you can wave controllers around and the paint you leave can be absolutely anything now I'm a visual effects by trade and so for me I think of what are the visual effects that you can make in the air and some of those visual effects are concrete some of them are amorphous some of them are particles it's a pretty fun playground to explore so everything that you saw in that trailer were not everything but it a lot of it was really inspired by a GDC talk I did back in 2010 called rock show VFX bringing brutal legend to life this

was back in yeah back in 2010 with Pete Demery and back then we were desperately looking for like all of you guys like we didn't know where the community was we were all just inventing wheels over and over again it was a really frustrating time and so putting this presentation together I was really fascinated to see how many of those old old school tricks apply in VR and apply it and tilt brush and one of my favorites is UV Distortion so this is a slide from that presentation on the left you have a fire that does not have UDP distortion enabled and on the right it is if you're not familiar with it you view distortion is a technique where you take one texture and you use that texture you kind of animate that texture you use it to distort the UV lookup of a second texture super simple technique it's like old school bread and butter for us and you take it outside of games and it has this whole new relevance so prai's and tilt brush our fire is done using UV Distortion you lay ribbons of mesh down with the controller and then we apply a you know

relatively simple UV distortion shader on top of it in order to create the illusion of fire I'm gonna have this shader and actually most of the shaders I talked about at the end of the section so I'll be giving those to you guys another piece I want to call attention to is the embers that are coming off of that fire particles are super important to tilt brush like I said when you can paint anything in space you want and your visual effects artist a lot of that stuff ends up being particles but there's a little bit of a catch which is that in a in a creation tool you don't have a budget like in games you're running or your first-person shooter you're running around a hallway whatever you know you've got two milliseconds you've got five milliseconds you've got some amount of time to work with until brush there is no limit you simply need to make every element that renders as blazingly fast as possible because our job is to get out of the way of the artist we do not want an artist to ever care about budgets if we can help it so we do that with stateless particles and

I threw some code up on the screen this is a vertex shader your eyes can glaze over it's fine you don't need to pay attention to this but what I do want to what I do want to point out is the Alpha channel of the color stream so this is the only state that we're passing into our particles and tilt brush we randomize each particle we give it a random seed we send it down the Alpha Channel and then we can do anything we want in the vertex shader we also make extensive use of krill noise which is another one of everyone's favorite trick a favorite tricks this is from the also from the GDC presentation and that plus you know all of our basic just a few signs if you cosign some other fun stuff and you can start making really really beautiful things so this is an overview of some of the particles that are in tilt brush this is snow you paint the particles in space but then we actually raise them up we fade them in we drop them down we drift them back and forth with some sine and cosine and as they drop we fade them back out and then warp them back to the top so it's the illusion of a simulation but there is there is ZERO simulation it's entirely

analytic this thing just runs forward here we've got stars which are random Atlas texture to get some different and then again we just are flickering them differently with that random seed embers are the opposite of snow so we start the particles at the bottom we raise them off we fade them out and then we warp them back to the bottom again smoke this is pretty subtle but you can see a little bit of a curl noise effect in there like I said it's subtle so I'm gonna I'm gonna switch to bubbles our bubble brush which is a more dramatic example anyone in this room who's because krill noise looks at that and goes ah krill noise this is an otter brush that's until brush it's just a really fun experiment I did that kind of shows how far you can push stateless particles now I got super excited with Keith's section because I hadn't yet thought of using a texture to drive my stateless particle animation so maybe I'll put some birds and crowds total brush next switching from particles to

more traditional geometry this is a fun breakdown of one of my favorite brushes which is neon this is just this kind of neon tube that you lay down colored pulse racing down and down the interior I want to do a quick breakdown on this especially for those who are a little bit newer to visual effects and who are curious about it again you get the shader code so don't worry too much about what's on the left there but I do want to just decompose these elements real quickly so you've got a really simple physically-based material you've got some specularity some opacity it's a little bit dim on the screen there but that's it and then we're gonna add a animated use animated movies to send a ramp down the length of this thing super simple all brushes and tilt brush are driven by vertex color so the color for every brush comes from vertex colors so the next thing I'm going to do is pass in vertex colors so that we see the blue that is the actual color that was picked by the artist

spoiler we joke that until brush everything is better with bloom so we're gonna brighten that up really make it nice nice and the unlucky but here it looks a little bit it's a little bit blown out it doesn't have the characteristic look of neon that we want and so we're gonna throw an edge alpha effect on top of this to really tone it down to make it feel more gaseous edge alpha for those who aren't familiar with it is you know you're using the facing direction of the normal to figure out to basically modulate the look of your of your effect in this case of an interior neon tube so that's one of our simplest brushes on more complex end of the spectrum we have our audio reactive brushes and again all this shader code is going to be available in a second so in tilt brush you can play anything on your computer you can play any music on your computer and that music will be analyzed by tilt brush uploaded to a whole bunch of shader constants and then every brush that we have could use those those attributes in different ways this

was definitely one of the most fun aspects of tilt brush to work on over the last over the last year and going back to keys point about the community I want to have a huge shout out to K Jiro takahashi who is incredible if you're not familiar with his work you should go to his github repo he's got so many amazing examples to learn from reverse engineer his stuff and you'll be a better person for it in particular reaction is a audio analysis library for unity that we used to prototype the early versions of our audio reactive brushes we've built heavily on top of it but really his work was the inspiration it was the seed it was the it was the way we could bootstrap our way into this really unusual place so a massive thank you to K 0 and again loved this community for things like this now looking at a simple waveform texture I just wanted to show off some of our shader attributes that we have access to and describe how we achieve a really simple waveform effect spoiler it's

again just using UV Distortion so we have a procedural line that's totally flat but then we're gonna offset it using the audio waveform that we've analyzed on the way in you can see that audio waveform is the kind of top rainbow texture there and it's rainbow texture because we do a bunch of different processing on the different channels and we pass high pass low pass a bunch of other different modulations we pack them all into this one dimensional texture and then we use that to distort the UV lookup we also have an FFT texture which we can use for fun things and all those blue bars on the bottom are the shader constants that we additionally have access to before I go into details on that cage ero also has an awesome GPU particle simulation we brought it in to tilt brush just because why not it's amazing of course this most definitely is not stateless and so for that reason it's probably not going to ship until brush but it's still beautiful beautiful beautiful beautiful

so I keep talking about how I'm giving you guys these shaders I'm giving them to you through the tilt brush tool kit which is a github repo that we launched about a month ago the tilt brush tool kit has all of our shaders with the exception of particles those are coming has all of our shaders all our brushes it's a tool kit to help you take tilt brush art and bring them into your own experiences and and yeah so everything that I mentioned you can you can find in here in reverse engineer it in addition my favorite part is all of the audio analysis stuff is in here so you can just drop that in your projects and go nuts this is kind of our way of trying to get back to the community a lot of this work was done by Fernando romaso so thanks Fernando for giving back and in that little bit of my experience on tilt brush pushing pushing visual effects outside of games and seeing how it applies in these new areas now I want to talk about the big picture more generally I want to talk about the rise of real time I want to talk about beyond tilt brush beyond games where I

see this stuff going and I'm really really excited about this simplest example I have to talk to you guys about is actually my badge which is over there Google what I've got Google on my badge a few years ago if you said I'd be up here with the badge that said Google I would have done a double take the big companies are recognizing the value of real-time rendering and are seeing a necessity for it and it's going to start tugging on everyone in this and a lot of people outside of this room if you look around you might spot badges that say Facebook might say Apple they might say magic leap there's a lot of eyes that are turning into our humble field and traditional real-time VFX again Keith touched on this a bit this is where we came from right on souls C's smartphones more recently this is like this is the the primordial ooze that we were all births from but the future looks looks a lot different we're still going to have like incredibly high-end

consoles and PCs those aren't going anywhere smartphones are sticking around but they're getting more powerful and again ridiculous sensors slapped on top of them like this is a tango phone so imagine a world where every phone in your pocket has a depth sensor it's going to get real crazy mobile VR is growing up very quickly right now we have mostly headsets that only have rotational tracking but soon they're going to have positional and rotational tracking for true VR experiences Desktop VR the kind of premium VR experiences those are going untethered and so I don't know what that's gonna be like exactly but I can't wait to try my first untethered desktop VR experience augmented reality is on the horizon - it's a little bit hazy err it's a little bit unclear when that's gonna land but when it does that's gonna affect all of us okay so that's some of the exciting hardware that's coming our way but what really matters to the people in this room and the reason we're here is what we can build with that hardware so now I want to rapidly run through really

where real real time the effects can go beyond games starting with creation applications so like no surprise tilt brush is a creation application we are you know see tremendous potential in this area it's a near and dear to my heart but I want to pause for a moment and talk about how the visual effects community is already impacting all these areas so tilt brush our team is composed of a couple visual effects artists from games but we also have a lot of people from film from Pixar and from ILM quill is a beautiful absolute gorgeous applicant application made by an ego quail is Inigo I'm sorry I probably just butchered your last name but if guys don't know who an eagle is he's a legend he pretty much lives and breathes real-time rendering he co-created shader toy if you're not familiar with shader toys should absolutely check it out another you know gorgeous application is medium that's created by a visual effects artist from games and an engineer from games those guys have done

absolutely incredible work and a huge shout out I think they were one of the first teams if not the first team to recognize the potential for creation experiences on these new platforms I want to move along but other applications that are up here on this page also have roots and V effects I think that's so cool this is Google Earth just showing the potential for the impact of these these tools on education data visualization science simulation all this stuff is going to kind of slowly pull us away from games and you can see that a lot of problems that need to be solved for something like Google Earth our problems that we've been banging our heads against for a long time theme parks this one was a surprise to me but Disney has a huge investment in VR the next time that you've got like toad Toad's Wild Ride you might be handed a headset and maybe it'll be VR maybe it'll be a R maybe it'll be mixed reality who knows but a lot of theme parks are turning their attention to this one mixed reality I mentioned this

this is a little bit more subtle but I'm really really enthusiastic about what's happening here this is a alchemy shout out to alchemy for putting this guy together mixed reality is where you take real-world elements and you take real-time rendered elements and you kind of mix them together and you and you use real-time graphics and real-time visual effects to feedback onto the video feed this turns out to be one of the best ways of communicating and sharing some of these virtual experiences with people super fun for us definitely you guys will love playing with this stuff music videos so back in the 80s right music videos were one of the driving forces behind the exploration of video and no surprise in the world of VR music videos and VR music videos are actually yet another place where we're pushing the boundaries of what's possible this is a chocolate which premiered at Sundance it's a film by Tyler herd and it's in gentle man hands yeah the are music videos pushing boundaries

I'll let the video speak for itself yeah good times it's kids it's a very good times in the world of visual effects show it to me is a another film another VR music video that is done by our amazing collaborators tip Mouse that's going to premiere itself by Southwest that's done entirely with the tilt brush tool kit so it's all tilt brush assets using the tilt brush audio reactive code can't wait for this one another another great example is how real-time visual effects is impacting film so film traditionally pre-rendered to these 2d content but we're even seeing really really cool ways where our work feeds back into these areas on the left you have atom which is a short film made in unity and rendered entirely in real time on the right oculus story studio continues to do really groundbreaking work in virtual reality they've been pushing the boundaries Henry and with Jerry Angelica which definitely check out during Angelica if

you haven't but this stuff isn't hypothetical like here at GDC on Wednesday we have real-time rendering for feature film Robo on a case study so so this stuff is you know happening as we speak notice Nattie Hoffman one of the speakers there yeah he wrote the book real-time rendering so people care a lot about this stuff right now but my favorite example is Perl which is the it's a short film by Google spotlight stories and it is the first VR oscar-nominated short play a quick trailer this but what I really want to call your attention to is the fact that this is entirely rendered in real time so this is the 2d film that was submitted and was nominated for an Oscar but it was actually originally built in VR so the Oscar is for the 2d version of the film and they just happened to also have a VR version of the film so this really speaks to the power of their real-time rendering engine and just the fact that people are becoming open to

our medium in places like the Oscars but the best part is best part of this is that they this is the director Patrick Osbourne this is him doing the cinematography for that film so he actually used a smartphone to drive the virtual camera that then they edited and then cut and then put you know and then submitted for the Oscar so all those things what they have in common is really the next computing platform that's a buzz word that you guys are probably going to be hearing a lot or are already potentially sick of hearing but it's the common thread across all of the different experiences I just showed it's really driven by the need to influence anything it's driven by the need to look anywhere you need to be able to have the computer understand where your eyes are where your head is where your position is and you need to update all of those things that blazingly fast framerate it's going to be bumpy this is the new technology hype cycle which is worth pointing out everyone just to be aware of like we've got all these new technologies coming our way which is really really really

exciting but some of them are gonna go through this up and down you'll get people really really excited to hit that peak of inflated expectations and then you go down in the trough of disillusionment before you crawl back up and I don't really have I'm not gonna argue about where we are on this curve with different technologies the point I want to make is that for us as visual effects artists it's smart to target lots of them because that insulates us from the bumps and ups and downs and the real this real-time rendering world is upon us so we can embrace it but you might be wary of just doubling down on any single technology okay so recapping the one thing that Keith and I are hoping you take away from this talk is that real-time the effects is expanding beyond games it's never been easier to get started so let's build the future together and thanks and yes so before we take questions real

fast I want to I want to take the opportunity to burn through just a few thank-yous for the boot camp these are the people making it happen these are the people to buy beer for Cristina one Jason Kizer Keith Guerette these guys beer they're the co-organizers that made it happen all of our speakers thank you guys thank you so much Megan boonie coordinated things from the GDC side David Johnson did the party that we're gonna have Thursday night Matt austell a the party that we're having on tonight spacing all the volunteers out here thanks you guys Jeff Hanna he does a tech our boot camp he paved the way for us so say thank you to him and of course buy yourselves a beer for being here on that note if you guys have feedback please tell us in person tell us on the forums fill out your reviews be candid this is the first time we've ever done this and we have no idea what you guys want we're just trying to like lay us taking the ground and say let's come together and let's build stuff and so next year who knows

what it'll be I want to add that it's for everybody this this if you buy this presentation then this community is gonna grow it's gonna grow tremendously and that means that every single person in here is gonna influence the culture which Keith pointed out so we want this to be for everyone we want it to be inclusive we want to take care of each other and just be friendly and build an awesome environment on a lighter notes these are our parties tonight at Jillian's 7:30 to 9:30 VFX mixer on Thursday 6 to 8 at st. Regis and now we've got a few minutes so we can take some questions we did in spite of the PowerPoint guests that was a hundred and seventy eight slides questions I got one about the bird and the goat vertex animation are you still offering that with like a rig and then exporting it yes so we're the the pipeline in that case those examples came from Naughty Dog where the pipeline is there still animating that goat in their classic 3d

package and then every single frame you just compare the difference of every single vertices position versus its original bind post which is going to be frame zero and you just store that difference as a pixel on that texture so the engine renders it in like its old position and then like the vertex tells it to move correct yes need yeah so the mesh without that texture is going to be the goat staying in its bind pose and then it so you start playing the texture it's saying every single frame off to offset each vertice here here etc that's why if you push it too far you have to start to consider the normals as well yeah that's cool yeah thanks for your work on tilt brush it's an amazing hey thank you I agree as someone who's a traditional artist as well it's fantastic to be able to move off a plane and draw in 3d my question is is there any hope for doing like in-betweening into a brush setup keyframes and yeah that's a great question so the questions about inbetweening until brush and we haven't done in

between but we have done flip book animation prototypes really sequencing prototypes and they are incredible like I'm not believe how much fun it is and how quickly you can create a sequence of three dimensional drawings that like evolve together it's super super fun we haven't tried inbetweening yet that would be like probably the next prototype to try but even without the in-betweening it is just mesmerizing to go frame frame frame frame frame and you can do in five minutes you can do ridiculous things I should caveat that with we have our hands full with our current roadmap so that's not coming out anytime soon but it is something we're playing with thanks for the question feel free to ask in general things about the bootcamp if you guys are curious about kind of how its unfolding if you want to ask Keith about beyond effects which is a pretty awesome example of the success you're having pushing beyond games or quiet groove you know I'll go get beers right now No okay cool wait we

got one yeah since you asked about I want to ask about beyond effects so what do you guys do in LA the plug so for the past two years so I stepped away from Naughty Dog a couple years ago and I was freelancing and just exploring and I was realizing that every triple a company a game company needs help right I needed help at Naughty Dog we could never find enough effects artists we could never bring in contractors even find remote freelance workers and then at the same time I realized her Drew's discussion like every theme park company every film company every single entertained Revlon has a VR studio did you guys know that the the Human Genome Project was trying to explore using VR as a way to to synthesize and process data so that humans can better represent it by walking around it and solve and find patterns more easily by being in that space like there's so many opportunities for visual effects artists

out there and there aren't really that many companies that are trying to explore it and so what we're trying to do is see if we can find a new business model that allows us to help take advantage of those types of contracts to work with remote artists around the world to build secure pipelines that artists that are willing to moonlight that are able to moonlight have an avenue to take on other cool passionate projects and just sort of inspire and drive more fun creation into our industry so we're still early we've actually opened our doors yeah about six weeks ago but it's it's all going well and there's just a ton of opportunity in front of us to try to open up the business model of our industry sorry for another question I've been doing traditional VFX forever I moved over to video games because something I've always wanted to do was procedural visual VFX or FX work so that every time you saw an effect it was something new and something different that you could

look at from any point of view your presentation was fascinating DLC like this sort of procedural approach to B effects in real time happening anytime soon I think the benefits of doing procedural work are astounding where we're struggling is just that the tools suck it requires a very very technical and a very very patient artist to learn how to use it so I think one of the biggest things that I'm hoping that the fact that we're having these discussion is that all of you guys out there are now thinking because you asked that are now thinking what ways can we build better tools to help with procedural effects like I think there is gonna be a ripple movement forward where we're gonna start to see artistic tools that allow us to create amazing procedural and Technic like highly highly technical effects using the most artistic of workflows that's kind of a dream Houdini and in fact popcorn

effects are pushing really hard and they both have extremely technical we're closed still but they're trying Luis is doing a phenomenal job with the side effects package to try to expose the power of Houdini into a more artistic workflow and I think that's only gonna benefit everybody so yes is my answer yeah thanks for the talk guys with the almost standardization of engines and the amount of resources out there for people to get into effects do you see effects moving in the future towards as an artist being able to work remotely much more easier than you are now like say something like a concept or a character artists can work ease more easily remotely or do you think it will still be so integrated that studios will want in-house preferably that's my dream I think that's all of our dreams and I think that yeah I think it benefits the studio's tremendously to standardize that that workflow as well right like we can't hire artists because it takes two months for them to learn our tools versus if we had standardized

workflows and andreas here is board for the next three weeks I can hire him because he can just hop straight in and make freaking amazing artwork and then carry on with his with his day so I think on both sides of that argument there's there's a win-win and I think that's definitely the path were all pushing for and actually I think you mentioned this but also Tyler Hurd who did the music video mentioned it as this new these new areas grow up there's also an awesome way to dip your toes into it like the chocolate video that was done he had held part time from visual effects artists in the games industry but it tends to skirt a non-compete because I'm not doing games and so if you're even if you're a company that traditionally doesn't let you do other games work on the side some of the like you're doing a VR music video that's not very threatening to Blizzard or to riot right so it's a good way of just kind of like exploring what's out there and none of those other ventures have time to build their own tools so they're using vanilla unreal they were using vanilla unity and it's just like how fast can you hop in and make cool stuff it's very very liberating

would you recommend that artists who don't have that limitation with their studios they're able to work outside they should start trying to build some sort of freelance remote work base or like have a be be ready for when it moves that direction absolutely yeah I mean everybody is searching right now for for exactly that so if you can put yourself out there you're gonna find a lot of really fun projects cool thank you guys yeah thanks so much I'm just throwing a throwing the bootcamp schedule up here perfect and I think that's it yeah that's it so hey thank you guys so much for coming joining us kicking us off [Applause]