Creating Mixed Reality Trailers and Livestreams

[Music] let's do it there we go all right we're gonna get going guys hi thanks for coming out so we are here to talk about making mixed reality trailers and live streams so before we dive into the meat of this well just give a real quick background on who we are and why we're up in front of you talking today so I am from the middle of nowhere in Canada called Winnipeg and Colin is from a wonderful beautiful city called Vancouver and Colin spent about five years sort of traveling the world and making games and doing wonderful things and exploring awesome places on his travels he made a game called in credit which is really cool and way back before that he made the 2d version of fantastic contraption and then decided to come back to come to Vancouver and VR was amazing and then decided to make the virtual reality version of fantastic contraption which we're gonna talk about today for myself I spent about eight years in the visual effects industry working on about 25 or so like b2c great Hollywood movies they

were all pretty bad except for maybe chumscrubber that was kind of okay but basically you know I decided to like try to get into something different and to make a really long story short I now make indie game trailers full time which is that crazy and thanks yeah yeah and which led me to the creation of these two trailers now just due to the time we're just only going to be talking about contraption today but basically what ended up happening is calling in about January of last year before the vibe was launched he's like hey Kurt you want to try to make in this a virtual reality fantastic contraption trip yes I am so on board with that and so Devin rhymer who worked with alchemy was just outside of Winnipeg at the time before he moved down to Austin and I knew they were working on jobs to him and what we decided to do was shoot both of these trailers at the exact same time and when I said yes to this I actually had no idea that they were going to be the two launch trailer or two of the three launch games for the vibe - so yeah pressure was on - so to speak so before we look at the contraption trailer and what we did I just want to talk really briefly about like VR and it has this like awful Communications Pro up problem

right now it's really really hard to explain what it's like to be inside a virtual reality and obviously everyone remembers the cover of time it was endlessly mocked and these are actual stock images that you can pay money for you can you you want that in your PowerPoint you can you can get that so but yeah it's like you're in this 360 world where you can touch things and move around interact with stuff and it's absolutely amazing and it's so not whatever that is right so we were like how do we communicate this awesomeness of VR so at the time we kind of looked at what everybody else was doing and when the oculus first came out everyone was doing like these dual fisheye views because that was like the raw output you got and it was like really just the easiest thing to do but it is awful why would you release that that makes everything you're doing look so bad that give there's chromatic aberration on the edges everything is fisheye distorted there's two views of the same same thing why would you even do that it doesn't make any sense so that's just really really bad so the next thing people started doing was just doing the raw output of the head mount displayed well like hey well take that we'll move it

around and people can see what the person is looking at and all this but the problem with that is your head is like literally the worst camera in the world it's super jittery you're not focusing on anything for any length of time and your eyes are kind of darting around and your brain and eyes kind of cancel out like all these little micro adjustments that your head is doing and when you look at the 2d footage it's like this shaky mess and when you just see this you can't really concentrate on anything it's really hard to look at and personally I get motion sick watching footage like this especially when it's at 60 frames a second it's really really gross to look at so the next thing people started doing it was like this picture and picture thing it's like oh this will solve it right well show the guy in his dirty room you know like going like this and try to like interact with stuff and you can see what he's doing but the two images don't link up in any discernible way it's just like conclu completely separate things that are going on so we knew that was bad too so at the time Sony's started promoting like the Morpheus which turned into psvr obviously and we're like okay here's Bini with like gazillions of dollars they'll know what to do right and so

their solution which I kind of like in one way and we'll talk about a little bit later in a little bit another little section here they do these virtual head mount displays and what they do is they kind of put the person in and you know they film them on a green screen or whatever and they put tracking markers on their face and then they remove those in pose they 3d track the head and they put this virtual thing on their face and it's great because you can see the person's eyes and you can sense the emotion that's happening and it's usually hidden behind the head-mounted display and that's really nice but the problem is is all those shots are completely fake they're just reaction shots cut into first person footage and you can tell that they're fake because you're just going like whoa yeah you know and it's not like linking up to the game in any way at all it's just like these things that are displaced in so that kind of sucks too so when it came time to do the trailer for fantastic contraption we were like okay we don't want to do any of this stuff we want to do something new and we wanted to try this mixed reality thing so well let's take a look at the trailer here you can see what we did fantastic

contraption is a game about building and using your creativity the challenge of the game is simple you have to get the pink ball to the pink goal and to do that you have to build a fantastic contraption [Music] [Music] [Music] [Music]

[Music] [Music] all right so that's the what we came up with and I really think that works really really well so why would you want to shoot a mixed reality trailer well I just think it's the best way to convey what it's actually like to be in virtual reality on a 2d screen and I'm not the only person that thinks this even valve agrees when they launched the vibe they did their own sort of mixed reality trailer there's mimikar isn't a lot away they had a bunch of people kind of sitting on the couch the kind of show is like the social thing they had a whole

bunch of different you know games in there and they were all integrated into the environment so you can see this person playing the game and Google they obviously did this amazing trailer for tilt brush when it came out now the way that they did theirs is a little different than ours it was more of like a visual effects kind of post-production workflow than what we did but the end result is basically the same and now since this has been basically a year since people have started doing this people are doing it for crazy stuff and this is like a collaboration Marvel did for a movie Doctor Strange they got all these like tilt brush artists till they come in and doing this like super crazy stuff inside a tilt brush and they did a mixed reality trailer to kind of promote that and now even hololens is getting in on this action so the five of you out there with hololens is can sew and you know you get like this crazy DSLR rig with like an extra hololens on it and you can do mixed reality now with the hololens and people are doing this live too and kahlan's gonna kind of talk a bit more about this but this is a Tokyo Game Show a couple months back now and you've got this woman she's called dressed up she's in character you've got this crazy like hammer on a jib so you're getting these nice like smooth

can't buy cinematic camera moves and if you look at the results that are like on the top there you can see it looks like she's part of the game world she is like an actor she is performing inside of this world and that's one of the really main things I want you to take away from this is that when you're doing a mixed reality video or trailer or livestream or whatever it is you are performing that game so your performance matters and to sort of like it that home not to pick on these guys but this is like an earlier trailer from this game blasters of the universe this was like when they were just an alpha and we you can see like all the pieces are there yes this is mixed reality but this game is like super fun to play but it looks like kind of bored right he's kind of this like shooting he's not like moving around he's doing his thing the compositing is not super good you're always looking at the gut you know because he's like in that way in the green screen so there's all these like weird things that you need to kind of think about now when you're making a mixed reality video because it just doesn't normally apply and I had to throw this in there because it's great someone did this like Toby rush Bob Ross will enjoy a painting it's

it's like five minutes you just gotta check it out it's like super good so getting back to fantastic contraption so what we did we use three different techniques to kind of convey what it's like to be inside a VR we composited the player into the virtual environment that's all the stuff we shot on green-screen then we brought some of the virtual objects from the game into the real world and the third one here I'm going to talk about for a little bit is we shot an in-game avatar with a camera with a physical camera move that was created in the real world and this is like this could be literally its own talk on its own and maybe we'll do it next year or whatever but it's like this kind of technology of like virtual cement ography and virtual filmmaking has been used on like like high-end movies for years and years like they shot a lot of avatar this way with a rig like that but it cost like hundreds of thousands of dollars and now you can literally do it for with a hundred and fifty dollar five controller and a $5 Steadicam so for the updated a fantastic contraption trailer this is actually what we did in my basement I'm just shooting my friend Vince who's playing the game but we're shooting an avatar rather than a live-action person and this has a number of advantages it's

like way cheaper to do this because you don't have to do the green screen studio in the setup and yadda yadda yadda and it works incredibly well you still get the sense that there's a player actually playing the game inside of this environment but you don't have all the overhead involved of actually worrying about a live-action person and like an actor and although the thing to kind of come along with that so we also did this for the space pirate trainer trailer and this kind of took it up like another level where the developer for this game had this fully rigged like inverse command at crazy avatar that was being driven by the head and hand positions and we kind of shot him mostly from the waist up just to avoid the problems with the feet but this works super well like there's no way we could have built a costume that looked this good and filmed them in mixed reality so doing it as an in-game avatar was like really advantageous in this situation and literally to do this like all you need is this like it's me and my basement with my friend Vince playing like this and I've got my phone is like a little remote viewfinder and you have five controller on Steadicam it a pretty interesting solution so I don't want to talk about this really in-depth because it's not exactly mixed reality but it's something to keep in mind that your game depending on what it is or

game or product or whatever it is might be more suited towards that than mixed reality there's also this which is the kind of a weird problem with mixer alley how do you fix that there's a giant box on the recent space right and honestly aside from what Sony was doing I didn't really think that there would be a solution this but just let's we'd like Google fixed it they came up with this amazing solution where they get this rough 3d geometry of a person's face and captured their eye movement all across the screen then they add pupil tracking inside of the vibe put a QR code on the front of the vibe and then in post they're able to map that new generated 3d face and composite it on top of the head now display and it's kind of janky and a little bit creepy but it's like surprisingly okay and it solves like that communications problem it's like you want to connect with the person's eyes so it's like this really like amazing little thing here that hopefully we'll see where this goes in the future so to back all the way up to the

beginning here so we'll take you through this entire process this all started when Colin and Sarah North we were both like contacted by unity and I guess they sent out this film crew called breakwater Studios to do this little developer diary on fantastic contraption because with being built in unity and Colin came up with a brilliant idea of just like hey let's just strap a third vibe controller on top of a DSLR we'll make that third vibe controller a virtual camera will align the positions in in the virtual space and we'll overlay the footage on top and we'll see what it looks like and it looked amazing I remember the day that this came out and I saw the video and I'm like tweeting a call and I'm like holy crap dude like this is and so then you know a couple weeks went by January came around he's like hey do you want to do this and then we started figuring out okay how are we going to do this for real and Colin kind of looked at the live streaming approach of doing it live and I kind of approached it more from a post productions kind of standpoint of like doing trailers and doing it all in after-effects so that's what I'm going to talk about in a call and we'll talk about the other stuff so basically the

trick to that is the troll trick to this is this you know you take a third vibe patrol the world now you can get like the tracking puck which is way better than just like the last offends a gapping tape here but you align a virtual camera to your physical cameras location and Colin gave me this little UI we offset the virtual camera an XYZ position X Y Z rotation you change the field of view of the lens you know I'm just sort of oversimplifying it but that's basically it and then once everything is kind of lined up you get magic and so this is the first test I sort of did in my basement was hastily thrown together I'm just like yeah the pieces are kind of there or it's working and I got super excited really fast I'm like I got to move the camera so I called my friend Vince over his like Vince you gotta come over so I can move the camera and have this play and so I'm like going through the process of figuring out what do we need to fix what do we need to do to make this actually look really professional and so I very quickly realized I needed a better camera I needed a camera that was stabilized so you're not getting this literary motion all over the place and the main thing that I started to realize is that you needed to choreograph this you need to choreograph the movement between the player and the cameraman and you need to

know exactly what people are doing and you need to practice because when you're doing this stuff live it's literally like you're acting it out it's not like you're just playing the game think and do it 50 times it's like when you're doing it 50 times with an active the time this is taking is exponentially longer so we need to make sure we had a shot list of all the things we needed to get and like practice this all and just go through it over and over again so we decided to do with shoot with a sony a7s - on a movi Steadicam this is just the gear that we had laying around there's other cameras there's other solutions that you can use this which is what we used and this is the final test that we shot in the basement there where I was like confident that this was going to work so this is me playing the game my friend ray is working the camera with the stabilizer and this feels really meaty it feels like we got a nice wide field of view so we can capture the entire contraption the camera movement feels stable he's kind of working with me to figure out where we're gonna shoot this and it felt really really good so once we did this I was like okay we can shoot this on the green screen and this is gonna work great so then it actually kind of came time to do it in production and I don't know if

anybody is here has like been on a film set or anything but there's like there's dozens of people involved in making these things happen there's a lot of people waiting around and so when you're on the green screen it's not the time to experiment it's the time to execute what you plan to do so we had this like a whole shot list and I just want to show you here this is like what a whole rough cake looks like so this is the very first thing we shot on green-screen camera is rolling cameras rolling gameplays rolling slate one good controller calibration so we had to go through that entire process you just saw there for every single take and we did 90 takes of various things and this was just kind of the first one just to kind of get in the pool of things but you can kind of see the compositing is really rough in this because it's just a temp comp but you can see the light stands get in the back his hands are kind of going over that so that needs to be rotor scope note in post-production and you know we're just you go through it you just sort of go through the shots and you put together what works in the

end to make an interesting video so just to give a really high-level overview of what this actually looks like Colin's going to go into this in more depth but what we did in order to capture this is we split the computer screen up into quads and so we have four pieces of footage to work with we have a foreground view and the upper right background the upper left and then we basically sandwich the green screen footage in between those two views and we also had a smooth head mount display view in the lower right hand corner just in case we needed to cut the first-person footage because let's just say the whole thing didn't work you know or whatever just as a backup we had that in case we needed and we did use it in one or two little shots here and there so just so you can see this all kind of slap together this is just like a you know shows a compositing pass of all this stuff so here's the green screen footage we shot with the camera and then the this is the foreground layer which we then kind of just like Kito's blue so this is everything in front of the head-mounted display then we bring in everything behind the head mount display and Collins kind of sandwiched in between there and then we do sort of like a compositing pass and we do like you know color correction light wrap motion blur and things like that to kind of smooth everything together and that's

one of the advantages of doing this in post is you're able to duke things like that to make it actually look a lot better than you would be able to that then you would if it was being composited live so that's kind of like the the rough scope of it and then it's just a matter of doing this for you know days and after effects and like putting it all together and the actual process of like making an interesting trailer is its own talk on its own but you know we just wanted to make sure we had a variety of shots that showed off the game in as many possible ways as we could and then just put that all together in a really really nice package so yeah that's basically how you do in a post and now Colin's gonna kind of talk about the nuts and bolts of this and how you do it live sorry all right so like Kurt says has introduced me I'm Colin earth way my wife and I are north way games and along with radial games in a shell input audio we made fantastic contraption this is actually the first ever live stream mix reality video ever made which i think is pretty cool so I'm gonna talk more about how to integrate this stuff into your game I'm still gonna talk a little more from a streamers perspective because we

did a fair amount of streaming and Curt covered the post the trailer post up pretty well alright so mixed realities pretty simple idea right you really have three planes you've got the plane behind the player you've got the player separated from their reality via green screen and you've got the plane in front of the player so you just need your game to output those through planes and then you need to stick them together a quick note about the rear plane you notice in Kurt's footage the rear plane actually held everything that was behind the player but now we just in the rear plane put the entire game in there there's some like smaller reasons to do that like the usually the ground is in the rear plane so if you start popping objects out of that there shadows disappear which is obviously doesn't look very good so there's a couple different ways to do those splitting up of screens there's two kind of two broad approaches one of those is to do the compositing outside of the game and one is do the compositing inside of the game so we're going to talk about compositing outside of the game first and there's kind of two broad ways to approach that there is compositing there's moving

objects between layers manually and then there's using the camera clipping plane so I call this like object blipping because objects blip back and forth between the front camera and the rear camera so this is really flexible it's good for trailers it looks good but this takes some integration to get into your game so I'm gonna talk from a unity point of view but all this applies to custom engine or also if you're using unreal unreal a little harder to do mixed reality because it's harder to have multiple cameras but in unity all you need to do here is have some objects on a foreground camera layer and your other objects on a background camera layer so the only reason that might be tricky is because the unity layer system is really overloaded and they use those same layers for collision detection so if you've already made a game and you've got thousands of objects and tons of prefabs already and you have renderers and collision on the same game

objects then it's gonna be really hard to go through and separate those two things so you can change which layer things are on because if you have an existing game you just are changing willy-nilly what layers things are on it's gonna totally screw up your collision system so this can take some integration work to get in and might not be easy to do for everybody but I think it's a really good way it's a really good way to do things if you if you can fit it into your game technique that is super easy to get in your game uses the clipping plane so this is like conceptually by far the easiest way to do mixed reality it's really simple to get into your game you literally have background camera it's fine normal camera just like the previous one the foreground camera you just set the just the far clipping plane equal to the distance from the camera to the player right and then it just doesn't render anything behind a couple of problems that this technique doesn't look quite as good as the object flipping between layers technique you've got that seam between the objects which is hard to get

rid of obviously if the previous technique there is some visual difference between objects moving between layers but it's much subtler and also there's the toes problem because the game you can see sarah's toe disappears whenever it's in front of her headset because the game doesn't know that the ground is a special object if your blipping objects between layers you can do some putting some custom logic make sure the ground is always on the background layer so like I say the compositing for this step is all done outside of your game and so this dreamer if your so there's two options to do that if you're a streamer then you will have to download a program to do that lots of streamers use this software called open broadcasting software they're really familiar with it does everything you need if you're doing a trailer then you just like make Kurt do it so OBS setup is it's not super easy and there's no way to take it there's no way to do it forward the user obvious doesn't have the idea of like you can't make like configurations and

send them to people or anything so you can write a good tutorial what it's gonna add a lot of friction to the player that's doing it they have to take your game footage and then like literally type in numbers to split up which which quarter is for which screen and then they have to blow it all up and they do the green-screen key in OBS and then smack it all together and so it's a fair amount of work for the player and there's also a really big problem with this technique of doing the compositing outside of the game is getting to 1080p footage right so you need to get two screens out of your game and the simplest way to do that is just to render them next to each other so literally like that would be the output for a fantastic contraption we split the screen into quads and then I have the four car in the background separately but obviously if you have a 1080 monitor then those are half 1080 resolution and when you blow it up it looks a lot worse than it would if it was full 1080 so a couple solutions for that is the most

obvious one is you can go out and buy a 4k screen and then all of your quads are 1080 but then you're adding for the potential streamer out there you're adding like a big monetary investment for them to be able to do this another possibility is you can do like output to multiple screens but then the setup is going to be even more confusing for the for the streamer so technique to avoid all of these downsides is to do the compositing inside the game so doing the compositing inside the game has a couple good advantages a big one is it looks really good so it doesn't have there's no like seems and objects aren't moving around they're always rendered in place with the proper background where they're supposed to be in the game and then also a big advantage of this is for streamers it's very low friction they just like start your game up they plug in their camera and then they maybe they do some configuring of green screens they set up their camera position and they're good so you'd be like well okay perfect we'll just all use this we don't have to think about the external compositing except for two problems with that of course one

this only works with cameras that unity can recognize and that is only webcams so you maybe can't read capture cards HDMI capture cards and those are how you're gonna get a really good camera into your system so a lot of streamers do use webcams but they when they're using a webcam they're really like sitting about a meter away from it like you know next to their monitor and that's what webcams are designed to do so they're good at it but when you're doing mixed reality now you're more like 3 or 4 meters away from the webcam and they're really bad at resolving image it's that far away so like one of the many reasons that my footage looks so much worse than Kurt's footage is because this is using a good webcam but it's just a webcam instead of a really good camera I mean definitely the number one thing streamers can do to improve the look at their stream is like just dump money and do camera so if you do the in game system they can't do that there's kind of some tools on the market that are starting to pop up to let you

put HDMI camera into USB and be readable by unity so that would be a good solution a better solution would be is if anybody in this room works for unity when they get home they can make unity recapture cards which would be awesome potential problem with this is has the same Toes problem as the clipping plane technique but since you are doing logic in your game oh yeah well I guess I'll talk a little bit about how we actually do it so this is one way to do it this is the kind of simplest conceptually really simple we literally just take we just take a billboard we put it in the game to where the player is so if I'm the player the billboard is like surrounding me and then we just draw the webcam on top of that we use a still here but we're you see Sarah with their arms out that would be the web cams of moving web can image and then we move that to wherever the player is we move the billboard and then you just resize it to take up the entire of the camera frustum and so right this has the same Toes problem that the clipping plane has but

you can solve it by where your camera frustum overlaps with the ground you can just draw yourself a quad there and then project the camera on that as well which looks really good I really like how this looks so I'm gonna talk a little bit about if you don't want to write all these tools the game yourself there are some tools that can help you do it and those there they're kind of popping up now they're kind of just starting to get people are kind of just starting to start release tools that are usable this is probably built in your game already if you are working in unity this Kurtz had a little bit of footage of the steam trailer for the vibe that was all mixed reality and so valve actually wrote this tool into steamvr specifically to make that trailer so it's still in steamvr from like like a year ago when they wrote it so you can't use that that's what streamers are using kind of today to do

stuff but it's really limiting it was written for one very specific usage it was written just to do their trailer and it did exactly what they needed it to do and zero percent more it's actually a particularly bad tool for streaming because when you do streaming output from your game you can capture the Alpha channel from your window but the valve tool nooks that alpha information so OBS can't read it valve solves that problem by outputting greyscale alpha information in one of the corners but there is no streaming tool that can make use of that so the best you get and they use the clipping plane technique that we talked about earlier of course they easy to make a tool to use the clipping plane technique like we talked about so that makes sense the big problem here is the alpha stuff you can see there the when the clipping plane overcomes objects that are partially transparent it looks really terrible you can also see there's a line going up that has a black edge around it because they just you literally have to key out black so it's really it's not a

very good tool it was never really designed to do this so this is like where we're looking to improve upon from now there's a company in Vancouver that is working on a mixed cast VR I really like these guys we did a lot of streaming out of their studio actually before they were working on this tool so they're kind of priorities and our priorities overlap really closely they they're out now they just released last week and they have actually three ways of doing compositing and guest so this is the in-game technique like we talked about earlier and then there is also they have okay I can talk a little bit about camera delay so another downside of the in-game technique is if you watch Sarah she the game is a little bit ahead of her you'll see the wheel bounces a little faster than she does so whenever you have however you get a camera from the real world into the game that real-world

image is always delayed a little bit and the game is always you know is much closer to real time so it'll always look you always get this like weird look of the game knowing what the future is and the person and a player being like behind in time so if you're doing your compositing externally with OBS you've got a good solution for that you can just literally take your game every game frame just buffer it and add a delay and so you can solve this delay problem perfectly but if you do the in-game technique then it's not obvious how you solve the delay because the player is like inside the game already so you can't delay it because they're also in it so mix cast is an interesting solution of this they can't solve data they can't solve it entirely so they be two options they give you this in-game option if you have allocate if you have a low lag camera or if you are playing a game that doesn't have like really fast hand motions then you can cheat it and get away with it or they also have an

in-game clipping plane technique so in game they render the background that you render the clipping cloud ground foreground they put your player in in-game with their own green screen tech and then they delay the game so you still have all the ease of setup and you still have but you get that delayed footage they also have an output they also have the split screen quad output so if you want to do compositing outside of it you can do that so I think this is a really good solution as you can see i implemented mix cast into fantastic contraption it literally took me under a minute i dragged in two prefabs and he'd go and like this is the footage coming out of it which i think is pretty cool something I kind of glossed over earlier is when for any of this to work you need a real-life camera and you need an in-game camera and they have to be in the same place in the same position space and also the same also the same rotation in space so in that Valve tool that they

made specifically for their trailer the way you set that is you there's a configuration file you uh you've got like you know a camera here and like a controller glued to it here and then you start it up you realize that oh the in game controller thinks it's like facing straight up and it's also off to the right because it's where as taped and so you go in the configuration file and you take random guesses about where that might be and then you start the game and you see that it's still off and you close the game and you make the changes when you start the game again and then you have to actually restart the game every time you make a change and it's hugely painful so in contraption we have usually used two people to do it we've got one person the way you sync these ray you have somebody with controllers in the real world and in the game worlds right because you know where your controller is in the real world and in the game world so you have one person making these like T poses and kind of like hinting to the person standing at the computer who is using like arrow keys and woz to fly this real life camera into the place and sync it up perfectly mix cast has this really nice solution

where they use the same general idea but they put it all inside VR so I put on the headset and then I grab the virtual camera and I put it over a close to where the real camera is I see the footage of me and I can see the controllers in game and in real life and then they have these nice nudging buttons so I can get it to just where I want it so I can get it looking perfect there's also you have to set the camera field of view which adds it's like weird complication because you're never totally sure whether your camera is like too far forward or too far back or if the field of view is too wide or too shallow so their solution to that is they actually just have a database of field of views of cameras so when you plug your camera in they hit a button it goes up to their database you're like okay well field of view for that is 45 you don't have to worry about it which actually is surprise simplifies things surprisingly amount they also have a really nice built-in green-screen system where you just like flop to the green screen and then like

hover over the green canvas and fade it all out so I really like mixed cast I think it's really worth looking at one more advantages this is all done in an external tool launched from Steam rather than in your game so if players said it once they never have to set it again so for streamers that's really low friction which i think is what you're looking for for streamers the last system we're gonna look at it's the kind of like crazy future tech left-field job sim mixed reality which is like the coolest they use it so these a depth camera and so in all of these systems the player is actually two-dimensional right so you have one position for where the player is and that's like where the headset is and that's where they are in space so if you try to reach over something your hand is not gonna go over it like if I try to reach over this podium my hand would actually clip behind it but with the alchemy system and see one I'll get back there you'll watch me grab the grab the donut off the tray with the alchemy system they use this the z-depth camera they use this

guy they use this the z-depth camera to actually get the depth of where your body is in real life so your arm is in front of your body and they know roughly where that is and so they can use that to composite you actually through dimensionally into the game itself and so it can't right so ideally that you can even do that without a green screen because it's separating you perfectly from the world but depth cameras are not that good so you still need a green screen this is like kind of a hybrid green screen it's kind of a hybrid like green screen depth camera approach there also since this is similar to the right this is in the same family as the in-game rendering techniques and so since you're doing it in game you have you can mess with the image in all these different ways and so they're like really taking advantage of that they do lighting effects and if you watch when I

teleport out oh yeah yeah these are always like these and if you watch when I teleport out you'll see them like push a blue kind of some blue sparklies in front of me and so you can like mess with the player and what it looks like since it's in the game they're also developing all of these they're developing tools to make all of this like lighting and player effect stuff work better and integrate better into your game remember there's that problem of syncing up where the camera is in real life and where it is in your game so alchemy has this super interesting technique to solving that problem in so they actually 3d print this like little mount that marries the Zed camera to a third steam controller and then you always if you know what that transform is then you're good you don't have to like mess with anything so that's great if you have a third controller all of

these systems work with three controllers Kurt was talking about handheld because hantum footage especially in trailer world looks a lot better than like fixed like footage so all of the techniques I talked about before work super well with a third controller or the vibe trackers that are coming out now it's like really obvious how to integrate all of that and set that up it's really just I've like a moving mount point instead of a fixed one and then when you come when you line up you're in real life camera with your in-game camera that's just lining up the offset of the thing taped to it so I think that's a really cool solution to the offset problem that I'd like to see more hopefully that'll become like a standard mount in the future those are all the techniques and the tools that I think are Laurie are worth mentioning right now all of these have such clear upsides and downsides that I thought I'd just make some slides that are like simple pros and cons so if you wanna take a picture of slides right now this is like the this is the rundown like

you'll you'll know all the pros and cons simply if you catch these I talked about the webcam thing on that one hmm yeah it's the webcam thing on the last one there yeah all right and that's all I have to talk about about mixed reality integration okay cool so just to kind of sum everything up we have we got time for a video yeah we got time yeah right yeah so we got here yeah yeah no worries yeah we just have a quick little four-minute video that kind of just goes over everything here [Music] but we are working on the job simulator and fantastic contraption trailers right now they're the two of the three launch

games that are coming out with the vibe [Music] so when they approached me I was really really excited about it I mean I just really wanted to be on board with this kind of thing my background is in visual effects and I used to work on like a whole bunch of different Hollywood movies after I got out of the visual effects industry is sort of transitioned into video games and started doing more motion graphics but these two trailers are the first ones that really merged those skills and like a really really nice way so generally speaking I'm literally a one-man shop in my basement making game trailers but for this I knew we would need a green screen studio we needed a camera crew we'd need people on set we need actors and so all of a sudden your needs are just growing exponentially so I call handcraft and they're like yes let's do this and so then we started finding okay well cameras are gonna shoot with what are we gonna use to stabilize the camera we need to figure out which camera would actually fit in the stabilizers and like you know how do we attach the third vibe controller to that will that throw the camera off and blah blah blah blah blah just keeps mounting on top right we did

that first test in the basement and I put it on Twitter and just explode if you but like oh my god it's crazy and so we knew we were onto something but today here we're recording our trailers so just kind of this new thing that people having to think about before and we're doing that with a green screen it makes a mixed reality stuff or any kind of put superimposed people into the actual game with a hope of being able to explain to people what VR is like actually showing people inside VR for the first time that's really really magical and I think will kind of allow people to understand about trying it just a little bit of what this stuff is gonna be like job simulator is a game coming out for all three VR platforms the five the oculus was a touch and PlayStation VR it's a game where robots have taken over all jobs so you're 2050 and so they created the simulator to simulate what Jobs used to be like so these future humans could experience how it is like to job fantastic attraction is a video game for VR where you use your hands you walk around and you make a contraption so

you'd make a machine use these simple tools is just reels and sticks and you basically try to get from over here over there basically every single time we would do a take on set there's like this checklist of things that we use to go through so like the first thing we would do make sure that the camera is rolling and then make sure that I'm capturing gameplay on the computer and we would have to get people to do a calibration of their hands and with the controller's visible and we needed to do that as we needed some way to align the footage from the gameplay and the camera in post-production so the screen would be split up into a foreground layer a background layer a composited layer and then what the player was actually seeing through the head mount display so we're capturing all four of these views live at the exact same time as they were playing the Ossipee Kurt's crazy as soon as like we had our first Skype call it hurt and he was telling us all his ideas I was like that's impossible and after we saw the first few of this screen tests is pretty minor people

aren't gonna realize how much work this was it's like ah they show that on the green screen it's all computer but no what we actually did is we shot the game plays live so the gameplay was actually being captured at the exact same time we didn't go in and cinema 4d or Maya and rebuild all this stuff so literally it took two days of doing rough composites of every single one of these shots I'm just layering everything and winding them all off exactly on the right frame and spitting that all out with this big collection of files then once you have that you can actually start looking at it and sifting through the footage it's really the best way to showcase virtual it's the best way to show that that person is in that environment and manipulating those objects and a lot of people you just say you have to try it and we're gonna experience it and I think that these mixed reality videos are the best way that we can get that feeling across you know in eternity without having the headset on being involved in like the first of these kind of things is like really really fun it's fun to let go through all the technical

problems and it's fun to like figure out how to shoot in VR and what things you need to do to make that look good so it was stressful but it was really exciting at the same time all right that's all we got thanks guys [Applause] we got lots of time for questions if people have any hope you have a question I think you have to go up to the mics in order to ask them there hi can you guys talk about time and cost of the first two trailers invested well it depends right it's all good the answer everyone will give you like like fur and feather I can't give you exact numbers but I mean like yeah it's it's not cheap you can't do these for like a few grand right you've got like the overhead of like a studio on production it's like you're literally shooting a commercial with all the overhead that that's involved because of all the people you

need on set because you need like anything there's like food you have to pee but you have to bring people you have to bring food you have to have people for lighting you have to pay someone to work the lights you have to pay something to work the camera you have to have like I was trying to do like three jobs at once you have to someone to insulating so it's really it's expensive but it depends on what you need to do like for a fantastic contraption we did a couple two days of shooting one on the green-screen set we did another day in my house if your game needs the four or five environments the yours costs are going up by four or five times because you need to shoot more days and that's more overhead so it's really impossible to give a number until you know exactly what you want to do right but yeah it's not cheap but that being said if you don't want the quality of like a professional trailer you could do it yourself just do it in OBS and you can get some footage out there for fairly cheap you can rent a green-screen just put it up in your basement and get something going that looks all right you know for a couple grand it just all in right so yeah also avatar footage looks I mean if you're more if you're like really more budget conscious yes I think filming in-game avatar stuff is still looks really good and costs like magnitude yeah exactly like the in-game

avatar stuff I really really like it and especially like for right now when VR is it in so you kind of you know and the developers aren't making like millions of dollars on their games they can't throw like you know five figures at a trailer right so if you can do in-game avatar all your cost is really is dev time in order to make that in-game avatar look good and fit within your world and so yeah it's definitely approach that I can see a lot of people going with to image it still looks great it's really really cool I love the movies that you do they greatly great but just a technical question why using a zip camera when it's a passive optical depth camera rather than something like a Kinect which is an active ir1 and seems more suited for that manager yeah so the reason they're using a Zed camera is because the IR cameras screw with the vibe tracking so yeah it needs to be just purely depth map based rather than IR based because yeah like I'm the alchemy guys if you find them walking around or they could rot the answer in more depth but that's the rough answer yeah it's as you could it's the IR I

followed your step by step with a DSLR but I was kind of transparent I was wondering I think it's my green screen do you guys recommend any green screen prank or anything well you know it's the green screen is really the least of your problems it's the lighting spend your money on lights you can make you can key any color as long as it's even so if you can be pasted six hundred dollars to get like the big friggin light boxes and make them is big and diffuse as possible and get two of them so they're lighting your green screen evenly from each side and then have a third light that's lighting your player from the front and if you do that you have the one light behind here you have the player in the middle and then you have the two lights on your green screen that'll create the most even green that you possibly can and then that will keep out easily and then it doesn't matter what color it is bright as possible without blowing out the power in your apartment or wherever you're setting it last year that's good some new stuff in this one as well

cool um quick question about props with hand controllers yeah I noticed a lot of your trailers you keep it basically to just the controller to show you do you find any issues is actually putting the props and give us hands when you're trying to film things in mixed reality well the biggest problem with that is the camera delay problem right if you solve the camera no problem then you're like 90% of the way there for a fantastic attraction it's actually really nice when people spend a lot of time grabbing these big wheels and that actually hides the delay a lot because you can't really see where their hand is going so that works really well I think the biggest problems with having like literally if you try to put like a gun in or an apple in someone's hand is that their two-dimensional right there hand is like going to be on top of the Apple underneath the Apple or the opposite of that and so it's gonna be hard to make that look really good yeah yeah like for a space pirate trainer like there's a number of reasons for like god I could talk about the virtual cinematography stuff that's just it would be another talk but you know one of the reasons we went with in-game

avatar over mixed reality for that is we wanted the player to look badass and that player in that thing in like honestly like I can't find an actor that's gonna look as cool as that avatar was but one of the other problems is yeah you've got this gun so your whole like in the game the gun goes all around your hand but there's no easy way to composite live-action footage where it's actually going in properly without doing a ton of rotoscoping in post to make that look right so we just solved that problem by just doing an avatar and I think like if you're having problems like that where you're holding like giant objects in your hand looks like it's swimming inside of it because you can't get that layering proper yeah that's an issue right and like some people will notice that and take issue with it more than others like for me that working VFX I see that kind of stuff and it makes me want to puke right so it's like I need to fix that and so there's there you have to sort of look at what's best for your game right and maybe a different option would be best to fix the hand problem like if you are using one gun you could probably just like actually make one you could make a prop but then it's got a lineup really

really close and you got to get a really dead-on calibration and that's really tough too depending on the amount of time you have to throw at this so yeah there's no real easy solution to that problem right now maybe you could put green gloves on or something to make your hands invisible I don't know or make the whole bike controller green that might work too someone could experiment with that maybe I would thank you very much yeah no problem yes I was wondering when you're rendering for the trailers themselves and you're outputting all four images are you doing it to a video capture card or is it just so yeah like when we did the trailer we didn't use a capture card Collins game is light enough that we were basically able to render the two views for the head not display and the four quads I was rendering at 2560 by 1440 which gave me for 720p quads and his game is light enough that we were able to do that the head this way was getting 90 frames a second I was getting 60 frames a second on the computer and I just use bandicam to record the in-game footage just on the computer and it has like a video card

GPU encoder on it and so you can hook up to it and it just worked it worked well but that being said if your game is like raw data or something insane that's already struggling to hit 90 frames a second inside of the vibe you're gonna have problems because now you're rendering everything else we're rendering it again four times and if you're doing smooth head mount display now you're rendering another camera for the head mount display so yeah there's a lot of overhead involved there the least of your problems is going to be the overhead of capturing the footage the your main problem is going to be maintaining frame rate for the footage in general right so a capture card yeah definitely can offload a little bit of that load but honestly like that's the least of most people's problems it's just maintaining frame rate is the biggest issue so from a streamers point of view I think it's a little worse even so it's a real downside of the quartered screen technique as person says you have to render your game at like 4k with multiple cameras and you do the capturing in this trivial on the same machine and so the problem with using a capture card and uploading that to

another machine which a lot of streamers already do is you lose the alpha information out of the game so OBS is if you use game mode and OBS to capture a game then it'll get the Alpha information out of unity and so that'll be your partial transparencies will be perfect but if you push it through capture cards and you lose that and you'll have to do a key on it which doesn't look as good yeah okay thank you guys thanks guys great talk if you have any strategies for non Unity game engines yeah well I mean I there are people that are doing it in unreal I think they're modifying the source code to unreal to actually make this work I remember I was talking to another developer and he was like yeah they said they were gonna put mixed reality in unity like a year ago and Merdan you know it's still not there so I don't know I I think eventually they will do it because it's like the obvious thing to do I don't know why it's not an

unreal yet I mean like they've got billions of dollars and tons of programmers it should take them a weekend or whatever right so maybe there's some fundamental flaw there in unreal that's making it harder to do but I don't know like there are games that have done it though the Everest guys have a mixed reality trailer that it was done in it so it's possible there's just gonna be more overhead involved yeah seems like they're all custom solutions right now yeah hey guys fantastic talk this was a really really great session I just had a question about when you're directing the actors when they're wearing the helmet what's your process for that do you show them the gameplay on a computer first yeah so for the contraption trailer I mean the vibe wasn't even out yet right there none of the VXR headsets were out when we shot this thing and so none of them had even tried VR yet and right and so I purposely like got friends to help that I knew would be enthusiastic about this and that I knew would be comfortable in front of a camera too and it would also take direction and those are like three really like key things and honestly that's really hard to find in people people freeze up in front of

cameras like these are just friends that I had but you know like a lot of people just aren't comfortable taking direction they're not comfortable feeling comfortable in front of a camera when there's people watching even when they got a thing on right so yeah that we rehearsed like I brought them over to my house you know we spent an afternoon each of them played the game for like an hour or two you know to get comfortable in there to understand what we were doing I showed them the shot list here's the things that you will be doing when we go on set we rehearsed them we showed them and just because you need to be comfortable doing that just saying okay do this thing do this thing live it's like no that's not working do it like this like that creates pressure on the person when they're on set you need to make them feel comfortable and I mean this is like a directing thing it's just like it's filmmaking you know basically at this point right so there's there's a lot to think about right and dealing with actors is a very very big headache which I shouldn't say headache I mean if you find the right people like which we did fortunately it worked out very very well but I mean if you don't have the right people it can turn into a big pain which is sort of another sort of point

in the direction towards like in-game avatars because then it's just you and one person not what the pressure being on a green screen set and having a dozen other people around waiting for you again you're screwing up and yada yada yada right so cool yeah that's a can of worms Thanks I actually think performance stuff and streaming is a super interesting question and oh yeah a galaxy of tools that are eventually and like techniques that are gonna be built for that yeah so like in contraption like one of the and II set up one of the like basic tools is being able to read your twitch screen yeah you're a twitch streamer you have to be able to interact with your audience there and then we also have like multiple cameras that you can switch to and like direct your own experience yeah and I think that's all going to be end up being really important it kind of shows how much we're just I'd like it's a very toddler beginning of all this and like now I figure out how to do it basically but then the real questions the real tools are gonna be how do you get expressive performance and yeah and one of the

interesting things like alchemy just had a new blog post about like the latest and greatest with their alchemy VR stuff and one of the things they've added is actually like a live in game composite of you in VR that you can see while you are in the head mount display so what it is it's like a screen floating screen that is like the the final composite of what the streamer is seeing but you can see it in VR so you can see how you're fitting into the impact in game world and that's really key too because like you're blind when you're in there right you don't know what the viewer is saying but now they've got that so they're kind of kind of piping the output back into unity somehow with not a lot of lag out it's like magic right yeah okay yeah that's yeah like that's super value because you need to see how you look to your to your audience especially if you're a streamer right yeah do you know how many people are using the mixed reality for streaming or most people just using straight output with your game or people doing this reality so there are very few mixed reality streamers right now there's not that many of the are streamers I don't

understand the streaming ecosystem as well as many people do but I think a lot of that is because there aren't that many VR headsets out yet and people that watch games are people they usually play games rather than like watching games that they'd never seen before right now people that I mean people are doing interesting stuff odds that upload VR is making like really interesting videos they do a lot of tilt brush stuff and that's almost entirely done with the steam tool and so they end up doing a lot of blood more post videos than actually streaming because the steam tool is much better as a post tool than it is as a live tool the one thing I'll say though it's like I know when contraption came out like and it launched like all I saw were people streaming fantastic contraption videos because it was this new thing right they were able to get on green-screen and they were able to do this and it's something new that your audience like as a streamer can see that the no other streamer is really doing right so is this really neat way for them to bring it viewers in there like some of the videos have like like gazillions of

views because they were the first ones to really get it up and working and I remember getting emails from like streamers like how you do the calibration I having trouble blah blah blah when people are all trying to figure it out at the same time right so that's kind of died down a little bit now that the tools are starting to gel and the techniques are starting to get a little bit easier but yeah initially like it seemed like there was a lot of people a lot of interest if for streamers jumping on board yeah when we did our like second stream we are a first stream and someone at which is like what the and they were like what are you doing this again and we were like I don't know next Tuesday and so they were like put us on the front page and we have like 3,000 people in the stream and it was very surprising to us but that tapered off pretty quickly and I think the reason is we're just not good streamers like I think to for a streamed of VR to be really compelling you need not a webcam you need like a real camera and you need yet good lighting in a good key and then you just need to you know be them like youtuber streamer a person who's like really performative yeah and that's like an extra layer right it's like that's their job that's their job like their charger

but when you're a youtuber a year or is it like a twitch stream you're a performer Yeah right and but now you're like a full-body performer it's not just what you're coming out of your mouth it's your whole body and your acting and that's just another level so it's really the entire point of the Steve the tools after you get past the basics is to give the tools for people to give good performance yeah hey guys it's great talk just a question this might be one of those areas where you go just with the avatar but have you guys ever looked at trying to put in game lighting on to the next reality avatar well that's what Alchemy's tool does so they have magic basically that takes that Z depth information that they were getting from the Zed camera and turns out into like normal information or something like that and then basically they're able to do in-game interactive lighting that way so the if you check out their latest post that they had like a day or two ago it's the like the first post they did it like it looks alright but the new one is like holy crap like the lighting is like really dead on so I

don't know what they fixed or what they improved but they've got like like it's raver setup where they got like 2 balls like this and they're going like this and they got flames and they got stuff built in game lighting is amazing and that's one of the big advantages actually of doing it in game doing that in post is a major pain in the ass and what they did actually for the top brush trailer is they had a guy with like an LED ball on like a boom that was like you know trying to like go like this and matching and they just rotoscoped it out and post to get the in-game like lighting matching to the live-action person and you can do that in a post with After Effects and things like that too but it's never gonna look as good as the lighting in-game but the problem is is with the Zed camera you're using these crappy little webcams essentially as like youryour image as opposed to like a full-frame sensor on a DSLR so you're not getting as good of an image but you're getting that really good interactive lighting right so there's like all these trade-offs with each of these methods and there's no one clear winner depending on it's all depends of what you want to do yeah and that tool brush video was done with a crazy technique right yeah Italy tracked in space where the camera was

yeah the tilt brush I could talk for like half an hour and how they deal the tilt brush trailer but it's not like what we did they did like it was more of a post visual effects approach what they like give the short version they tracked the camera and then brought the camera information into unity and then like replay that unity and it's like it's it's insane setup so yeah thanks I mean I think as a really good point is like there's still tons of room for creative problem solving yeah like these are the techniques that people are kind of using right now but I mean I'm sure people in this room are like well I don't know why they didn't just do it like this and like that and that's problem like maybe that's how we're all gonna do it maybe that's the thing still pretty early days I hope someone makes it better and easier because that would just make it better for everyone yeah okay I think that's it thanks guys [Applause]

you