Fireside Chat: Virtual Production: A Peek Behind the Curtain for Game Devs & 3D Artists

[Music] [Music] thank you [Music] Welcome To The Game Dev stories stage we've got another 20 minute fireside chat for you and this one has slides of video as well so I would like to welcome to the stage uh my guest Justin Kafka from dark slope so play a round of applause for me for Justin thanks everyone now this is a very exciting uh talk we've got for you uh this afternoon a little change of pace from the last one we're going to be talking about virtual production now um game engines like unreal and unity obviously we're all familiar with them but they're not just for building games um in fact they're using another sectors but they're quickly becoming important

in the field of making film TV and commercials and um and in fact the the best example of this probably the the poster child for this activity is the Mandalorian which of course was created on Virtual sets you'll be familiar for the from the volume that they they create there um that's by no means the only example of virtual production and we're going to be talking about that the software with an expert so I'd like to introduce uh Justin Cathcart now Justin is head of virtual production at dark slope here in Toronto and over the years he's worked as a virtual production supervisor for a large number of projects I'm sure he'll say more but they include shows on nbcu CTV V CBC and he's going to be giving us a sneak peek behind the curtain of working in Virtual production with some examples on screen and uh and talking a little bit about how it's relevant for 3D artists and other people working in in the space so Justin can you start us off um can you give us a little overview

on about Led virtual production and volume and how that all works sure sure uh thanks everybody for for being here I really appreciate it I feel like I destroyed the fire side chat part of this uh so that's me uh on an LED volume pass through that so let's talk about um how uh virtual production LED volume virtual production Works uh I feel like this uh really illustrates it well we'll just do kind of a very high level for those of you that already might have some uh you know understanding here so we have a physical stage and a giant LED screen and effectively the two really key components so there's us are uh you know Unreal Engine or Unity a gaming engine that is basically displaying a 3D environment that is being thrown up onto that LED screen and then the camera is being motion tracked so basically as the camera is moving around we get to see you know the nice Parallax and it feels

like we're actually in that volume and I also feel like this is a great illustration so when we're building these sets and doing like LED volume production uh building practical sets and making sure that the um you know the actual environment that we're building matches uh like these are like kind of the key elements that we as as we say that that sell it the best so if you have like a uh a practical set that's happening in front of the LED volume and it's like feels like the same environment then you get this great kind of uh depth and you really sell you know the scene that it's not a flat image on that LED screen so yeah that's it now you guys all know you all know how to do it that that's incredible just a quick question look at that I I guess that the the same computer control that controls the camera and the Paradox also controls the lighting for instance so it's not casting a shadow onto the screen and that kind of thing yeah and I guess

there's another key element that we I should mention is that so the in the tracking uh of that camera that uh information is being sent to a vcam like a virtual Cam that we have in unreal and so basically yeah there's a frustum which is essentially a fancy word for rectangle uh that's uh being thrown up on the screen that's like where all the render nodes are really uh kind of putting all their compute power into but uh but yes the the vcam is a is a huge uh component of of this that's where the motion uh uh you know the motion capture data from the camera goes to that's great yeah and I I it instantly makes me think of a few more questions and I think if anyone here asks questions about this we'll make some time for that at the end but let's just go ahead to sort of the development of this the industry is pretty young this is pretty still a pretty young technique in the industry yeah so can you maybe give us some insight into how the process is developed and how things have changed in recent years sure sure I'll throw up another slide which is really not gonna

it's just a big flow chart but uh so it's a it's a new burgeoning industry and you know the traditional filmmaking has been around for like 150 years so um it's uh there's some pain points that we're trying to uh get through uh because people that are in traditional filmmaking get upset when I don't know if you've experienced unreal crashing before has this happened to anybody here [Laughter] so uh basically uh simty which is uh standards or organization uh traditionally you know living in the motion picture and the television World um they have taken the initiative to uh add some standardization to Virtual production so they created the onset virtual production initiative uh this is uh a flow chart that kind of gives us an idea as to the different elements of virtual production uh and they're also really concentrating on standardizing things like metadata that's coming out

of the camera color science across like you know all different parts of uh the color Pipeline and interoperability between all these parts that were really not built for film they're essentially cobbled together uh you know from different uh like you know a game engine was not built for film it was built for uh you know for developing games so all these things uh you know will benefit greatly from having some standardization and the other thing that that's a symptom is uh which I think is a great idea is they're really all about education as well and at dark slope we also try to bring in producers and directors and try to give them you know kind of what we're doing here just kind of talk about what it's about and how you can use it as a tool in filmmaking that's that's great and I can see on this slide obviously there's a lot of information taken there and there's a number of acronyms as well I can see vad which I know we're going to come back to later but there's other terms as well in the industry like icvx as well do you want to can you give us some insight into some of these terms some of the key like Concepts oh my God I have the

perfect slide to talk about this so it's like it's almost like we prepared uh so yeah there's a bunch of virtual production terms that you might have heard of uh like I said frustum is a term for fancy rectangle uh but uh things that are terms that are starting to develop are like you were saying vad uh osvp which is like uh people generally were saying VP and you know there's a confusion between are you the vice president what's happening so the onset virtual production is osvp um and yeah we can kind of go through maybe we'll save this slide if there's q a and you want to come back to this uh icvfx is really one of the key uh terms that you'll hear it's just basically the whole purpose really of LED volume virtual production is getting it in camera so icvfx is getting the VFX in camera without the need for you know green screen and this post-production pipeline and yeah techviz previous lots of visits up there uh and

um one of the other terms that you you mentioned and it says on your website and that we've spoken about is hyper reality now now I know I know what I think hyper reality means but um but oh can you talk can you talk us through what hyper reality means at this concept and how it feeds into the version sure sure I mean I might have fed you that line and apologies to everybody uh but uh yeah so hyper reality is really uh this um further cobbling together as virtual production is like a cobbling together of different uh kind of Technologies hyper reality is uh kind of even going beyond that and and bringing in kind of XR to the game and haptics and all these other things that are not traditionally part of like uh uh you know LED volume virtual Productions so uh actually I have a video we we've developed a few shows in this uh realm uh one of them that I think is probably speaks best to the to the kind of the game developers here is a show called uh speedrun that we did together with nbcu and uh Insight Productions who kind of

uh they do like Amazing Race and all the kind of uh reality shows here in Canada so let's maybe look at the sizzle reel for that and they'll give you an idea as to kind of what hype reality is about welcome to the metaverse this is the first competition reality series that combines virtual gameplay with real life [Music] for the first time ever players travel inside the game tonight welcome to speedrun [Music] who will be the first speedrun Warrior [Music] to me and everyone else is on the best gamer in the world three players compete in an intense battle that takes grit

determination and skill in a Race Against Time I'm feeling awesome I'm ready for this I will dominate this person players choose their own avatar represents a cooler swaggier version of myself and he hits the teleporter viewers at home attempt to anticipate a player's next move jumping off their couches with each unexpected epic challenge the motion capture suit transforms players into the first generation of ultimate virtual reality Warriors let's go haptic technology simulates realistic physical Sensations outbreaking series of Limitless worlds and endless battles but very real sweat and strategy once each week as they

compete to win the title of Ultimate speedrun Warrior [Music] thank you thank you uh incredible and looking at that you can see that so many different strands of what we do in creativity come together in one place and different Technologies come place you know there's there's all the different parts of that there's the virtual production and the game itself and the haptics and so on I mean um there's a lot of work for game devs in creating that kind of content we've just seen right yeah yeah so that that's like uh my kind of wanted to show that to this crowd here because uh really that uh show doesn't happen without a lot of active developers uh kind of creating uh that kind of VR environment uh you know there was a lot of QA testing uh like you know with uh VR headsets so it was a combination of motion capture team plus there's just all these different departments uh uh you know our gaming Department our TV Department kind of melding into one uh

and uh in doing that game and showing it to other producers that are shooting on LED volumes as well coming back to LED volume production like they see the uh you know the opportunities potentially to supercharge their uh Productions where they're like oh it's interesting so it's a gaming engine we're tracking various things can we like set up triggers can we like uh like are there things that we can gamify when we're like shooting on Led Led volumes so we also bring that as an opportunity for uh producers that are shooting on LED volume as well fantastic yeah now I want to come back to one of the terms that we saw earlier on you mentioned it I said vad and then you said vad yeah so bad virtual art yes you got it uh it's uh so that's the virtual art Department uh basically I I feel like uh there's a laziness in naming that they're just like we'll just throw virtual at the beginning because you're doing basically this job so just call it the virtual art Department uh but the the the key thing really for the virtual art Department

that's uh kind of been uh like you know pushed uh to the Forefront is these feedback loops that need to happen between the traditional art Department like on Productions that uh have had you know clunky Beginnings it's always because there's not enough dialogue there uh so there's like a really needs to be an active dialogue between those two departments there also needs to be potentially photogrammetry happening on set if the set pieces are being built they need to be scanned and put into engines so it's just like this uh feedback loop that needs to be constant uh and then another key component for the the vat is to like you know optimize these scenes so their performance on the LED screen has got to make sure that uh you know these scenes can get heavy sometimes if you just kind of throw the kitchen sink in there and so you have to find that balance that's uh performance as well on the wall uh and then these AI driven pipelines are kind of I'm sure everybody has AI in their talk at some point in time

yeah yeah it's like a prereq so um I just got a uh invite to Kubrick which is this new virtual production uh AI driven stable diffusion environment creators so these are starting to come into our to our world now and then in the interim digital backlocks which are just a fancy term for environment a library of environments uh all these things are kind of important in the in the bad and actually yes a good segue can you give us a bit of a sense of what a day a what a day on set looks like working in in Virtual production like is it different from yeah well you know one of the key roles on set is uh um yeah like the volume operator so uh traditionally somebody that might be a uh 3D generalist uh can uh operate the uh the Unreal Engine normally it's uh you know Unity is in our world too but uh I would say 98 of the time it's Unreal Engine uh and so somebody that uh has the skills

in order to um kind of uh adjust things live based on feedback from the directors and camera crew uh and yeah the the thing that basically everybody needs to have on set is the ability to kind of stay cool Under Pressure because uh there are pressure moments and like I might have mentioned before unreal tends to be uh a little crashy sometimes so uh yeah and I work as a virtual production supervisor there's also uh like you know mocap texts there's various role tools that uh that are on set but yeah really the key is uh being cool Under Pressure whatever role you feel right well look we've got about five minutes left I think by my by my recording a little under perhaps um which means we've got some time for questions yeah take questions yeah so do we have any questions for Justin about uh about virtual production and there's a yeah there's one over at the back here I will run over with my mic so we can hear you oh there's one on the floor there we go foreign

so when it comes to these types of jobs like are are they starting to pop up a lot more often now because of like how technology is evolving and since since if they are popping up a bit more often um where would somebody like try to find these types of jobs well to just like start appearing on the common job boards yeah I mean uh probably the best method is to actually actively go to the people that you know that are doing production and just seeing what they have uh on their job boards I haven't actively been looking so I'm not 100 but I know in our website uh pixamondo others they have uh like you know job listings specifically the roles that they're looking to to fill and um I will say to be candid that uh the uh you know film and television production right now is like slowed down because of the uh the uh we have a couple of strikes going on so that's um yeah today might not be the best day but come back in a couple of months and uh like there there are there's a

shortage that's why sympty and others are like really focusing on education because there's really a shortage of people that can fill these rules great do we have any other questions oh one in the front row hold on I'll just bring my mic down there we go so you made a comment early on about how this engine obviously isn't made for this kind of development right um with unreal 5 coming out last year do you find that there's been more tools to make your life a little bit easier and that's something that you're going to see becoming more and more regular yeah to uh epics um credit uh they've been really actively listening and uh you know really making uh great efforts to make it uh production ready um is just such a complicated piece of software that it's still you know it's still gonna crash but yes things uh um have been more stable since five uh you know let's say after 5.0.3 then

that one was a little buggy uh but uh yeah basically uh things have gotten uh improved and we you know have a great relationship with the Epic they listen to us and I see them literally had a epic uh or sorry an Unreal Engine meet up last night and uh yeah lots of people from epic were there with open ears and uh really um not afraid to hear the pain points and uh not necessarily defending their software they're just like listening and be like okay we're gonna take that and see if we can hopefully make it better for you in the future great thank you great question we've got about one minute left we've got time for another question Chris yeah yeah any any more questions from the uh oh there's one here in the oh uh the Chap's gonna bring you a mic thank you thank you uh in the film industry uh there is a lot of uh expanding and Contracting of of of people because you have a production on and you have another production on do you expect that this to be reflected in the virtual production as well say hey

now I've got three projects running and then I need to like expand the size of my uh uh my roster of people and and then contract it again you expect that that is going to be something yeah and talking to people in the industry they've uh also said to me that they're changing their their Paradigm they had people that were permanently on staff and uh so uh in talking to other people that are running uh kind of uh Cruise uh focused on Virtual production they have explicitly said that they're going to make it a contract job so it's like basically like uh like hiring other crew on film where basically they grew up for uh the production and then uh once the production's kind of done then they're gonna like you know kind of let you find employment elsewhere but uh but yeah when things scale up uh there's a you know an an immediate need for for those people and uh yeah that's looks like the way the the direction that we're going fantastic I think I found it there

because we got a time but thank you so much big round of applause please [Applause] [Music] thank you