MetaHuman Creator | Epic's Road to the Metaverse ft. Arvind Neelakanan (Epic Tech Evangelist)

hello everyone epic games recently announced metahuman creator a cloud streamed application that enables the creation live captured animation and potentially in the future fully ai capable lifelike npcs so what are the implications of this new technology to games in terms of cost capabilities and production value here with us to speak to these issues and more we have arvind needle contact technology evangelist at epic games welcome marvin hello joey i'm not straightened doing well and co-hosting with me to discuss this technology is lila games cto avinash pandey welcome avinash as well so arvin i thought we could just jump right in and begin the discussion by talking about the technology where it is today and the current primary applications i know it was used by ninja theory in hellblade to animate the senua character

can you talk about where and how this technology is currently being used absolutely so just to give a quick introduction about the metahemi creator itself it is a cloud-based tool so you're able to access the metahuman creator on just any web browser and you're able to create a metahuman now currently metahuman is an early access which means you have to sign up in order to get access to it this tool if when you talk about implications and where it can be used uh it really opens the doors to creation of very realistic characters uh you know if you have ever been using or creating realistic characters it sometimes takes weeks or even months to create something very realistic and meta human creator removes that barrier out for you uh in terms of use cases it can of course be used in games as either a playable character itself or as non-playable characters like npcs uh all in all sorts of games and experiences that you can ever think of eventually it can also be animated used as a digital avatar

so it's not just in games you can also use it it can also be used by say youtube streamers or even for virtual production for pulling where you want maybe have a stand-in for an actor for example you could absolutely do that too uh so because it's an early access uh the meta-human-created tool itself or the assets that you create out of meta humidity itself uh it's not uh it's still being used in a few productions that that that but it's nothing is out because it's so new right uh but you know we cannot really wait to see how game developers or storytellers can really dive into this tool set because it's a fairly intuitive interface and it's really up to them to experiment to find new and interesting ways to use these directions okay in that kind of hellblade example that was on the epic website there was mention of three different companies three lateral cubic motion or four companies ai kinema and xsense are those all part of epic now and how do those how do all those technologies like

sure integrate together three lateral and cubic motion are part of the epic family now this live capture that we were seeing i would say it's probably the gdc 2016 demonstration that you probably saw that uh that was also using i can demand excellence but they were showing off the the motion capture technology uh in terms of the body capture uh more than anything else uh so ikinema and excellence are their own companies they're not authentic whereas lateral and cubic motion is part of the epic family uh in terms of how the tech itself is used it's so three lateral is of course the face scanning technology so you have this very high resolution high fidelity uh scanning tech that goes behind the scene which also connects into this digital double pipeline and then cubic motion is the one that solves this this cat the scan data so it's able to create intelligent uh or ai based animation based on the scanning that was done by

three laptop pretty carefully supervised process uh combining both three lateral and cubic motions technology because uh there is a huge database of scanned assets that we already have which are then combined into the into the metahuman creator so they're able to create uh in you know human beings who were actually scanned from real life uh so and the process itself is fairly you know that robust like we have this data preparation we have cleanup uh you also have rigging and modeling and you also have loop dev and database integration with all of these tools uh and at the same time because it's early access it's like a version one of the product itself they're also adding a lot of things on top of it like automation pipelines and whatnot in order to make sure that you know like the tool tool is uh a lot more friendlier as as the versions keep digging up right and then in terms of the the application so you know

uh and character animation is it primarily cg movies for games is it other things that you're seeing as far as how it's being used in early access right now um i think it's really up to the up to the folks who are using it the the assets that you're going to be getting out of the metahuman creator itself is supposed to optimize for game development particularly for pc and console uh in fact i i believe uh i think i think i think that there's there's been a lot of interest about mobile and dr and ar which i will get to in just a little bit of time but primarily the idea is that we are able to use the better human creators for game development not only game development purposes for high-end pcs and consoles but of course you can also use it for as like i mentioned like virtual production or even you know have digital avatars we are already seeing that a lot because meta human creator kind of out of the box supports live linking which is the technology that allows you to use the apple's face id uh on your

iphone and just you're just able to like you know have your facial expression mapped onto the meta communicator so we're also seeing a lot of a lot of uh streamers like pitch streamers and youtube streamers using the digital avatar for that we're also seeing a lot of uh animated short film shots short film or storytellers or just using the digital humans or the metahumans that that you're able to get to tell their stories that they were you know using a very small team or even sometimes a single person team because all you really need is an iphone and a mocap and then using the unreal engine uh while you're acting out all of these all of the different digital avatars the possibilities are fairly wide open uh in terms of what is currently used in production uh we don't know yet at least i don't know it because it's still very new and i would say it will be a few months

before you start you'll start seeing it in in live games or live productions and and then you had mentioned the cost and time impact if we're just talking about let's say 3d character animations can you characterize that a little bit more for us in terms of like how much of the cost savings or like like how much time yeah i mean as mentioned earlier right it's a very painstaking process for you to create a very realistic human being and it typically requires not just time but also very high technical expertise uh so each facial feature like you know because we are so used to seeing human beings and i see you you see me you know we we noticed the smallest uh the smallest in character or imperfection in your face and you know it's it's a it's a it's not a real human being for example right so the process that we have used with relational and cubic motion along with some of the internal projects that we've been working with epic games we have made this process a lot easier

like what used to take several weeks or several days or several months now you're able to create it in a day if you can do it in fact as a great example of that we recently launched a women's creative program right here in india where we invited some of the top women talent in the film industry and they use the meta human to essentially tell a shock film or tell us tell a story create a short film in four weeks time and remember that most of these women have never used until before that's the first time they've used and in four weeks using metahumans they were actually able to create short films using the metahumans uh which otherwise would not even have been possible they would have probably used like some pretty game looking or traditionally game looking art assets which shouldn't look real right um so you know and the metahuman also comes with a fully rigged an animated character uh you are able to uh also use your mocap technology just slap it on there you're able to use live link from my phone just slap it on there

uh and within like a few hours you're able to create vision changes which saves a lot of time and money and a headache for you so i would let you translate that to cost but i think uh but i think it's a significant amount of money that you are going to be saving okay and maybe i could ask one more application question before i turn it over to avanash to ask you some technical questions but the other question i have is given the technology would this also potentially have the application in terms of being able to turn let's say a game designer into an artist by allowing you to custom create like your own you know uh 3d character or model using the technology uh or do you still need need the artist to actually kind of create you know basically create your characters yeah this is a good question and i want to first align the fear that you know you're not replacing artists at all that's not the point that's not the point of this tool i think at the end of the day you still

need an artist's eye and artist vision in order to create a good looking uh the good looking digital avatar what the medahim and trader allows you to do is get rid of your boilerplate stuff right like most of the time you're always going to be making characters you're always able to you're always creating these controls for your facial animation you're always trying to add some kind of uh good looking subsurface scattered skin some good looking hair so what the meta human creator and if you if you've used it is that it it opens up this um about a template about 50 human beings like human characters not human beings better humans i'd say and and you start that as your base character like that's going to be your your base character and then you can have an artist can come on top of it and start sculpting it using the the metahuman creator in order to create the best looking human being uh meta human that's possible so uh of course a technical artist or a non-artist can jump in to start using

this uh but i think if you really want to get that you know the final uh final good looking miss or the final touch that you want for your game or your application i think that artist artists thought should definitely help yeah to extend our jk's question so i had another question but this was pretty interesting so in terms of animations and i did my own personal research so meta human animates in real time your facial expressions but i i what my understanding is is that for the full body animation you still use built-in animation clips so meta human just works on your face and rest of the animations you still have to do uh through your existing animation systems and platform is that understanding correct uh yes uh so why yes and no uh so in when you bring in the meta human the meta human is fully both the facial rig and the body is full leaving once you bring in the metahuman inside android

engine you're then able to use the iphone which kind of like automatically connects to create facial expressions at the same time you also have a control rigs which is the rigging system within unreal engine which kind of works seamlessly with even existing animation so if you have let's say the most famous mannequin character it has a whole bunch of animations inside you can absolutely map those animations to the meta human and use existing animations we have more capped animations you can also bring that animation and map it to your meta human you can also live stream uh mocap data from excellence or whatever whatever you're using and then bring that in front of you uh at the same time you can also hand animate because the sequencer which is the timing or the uh the tool that's used for telling linear story within unreal uh you're also able to create keyframes and if you're a traditional animator like who's comfortable using uh or creating animations using just the this keyframes uh you can also absolutely do that

interesting so it's fairly flexible yeah so um so if people are if if somebody looks at meta human in a way that if there is a camera and you can the way you know we are talking we are animating our hands and using these gestures can make a human do body justice as well or it's just special real time when we talk about real-time animations yeah if you have a mocap suit on on you you can absolutely also do your body gestures and bring in your finger data your body data the other question i always get asked is uh when will it come to mobile like how optimized will it be on mobile and you know i saw like you know it supports eight or eight lods you know eight head lods four body lods and i think they have also mentioned that it it works on android so what's your take on like where are we right now and is there any priority in terms of you know bringing

into a more you know mobile ar we are more optimized way sure uh the eventual i mean this question i get asked a lot uh so eventual idea of better human is to make it platform agnostic to have like as many platforms support uh the meta human creator at this point of time uh the high fidelity that you're able to get out of the better human creator is predominantly for pc and persons we are working towards supporting android and other you know lower end devices including core virtual reality and augmented reality where you do have where you have like you know a fairly tight budget when it comes to uh cpu and gpu usage the support is coming but as of as of the early access metahuman creator that we have today it's aimed at pc and functions for the most part okay all right yeah by the way their lod system is pretty flexible like you know you uh you have control over uh so basically one of the cool things about uh meteorologist is the way you render hair

so you know it's strand based it's quite basic but if for the performance you can also just convert it to context and then you can have different settings for different lod levels so when you guys focus on just mobile or these platforms some some tech is already there it's just you have to refine it for that platform correct so you know you can absolutely optimize your meta unit in order to make it run on a low end device uh but personally this is my own opinion which is that like you know better human look best when it is in lod zero uh but if you're going to bring it down to lod8 uh it doesn't look the best like you know it kind of like uh removes the entire point of creating a metahuman because it looks it looks kind of okay to uh at this right ah so you know i think you can kind of lock it and lod8 if that's kind of what you want and you can you can of course extend the other functionalities like the control brick

uh that you get out of the box with the metahuman then use that in your android or your mobile application you can absolutely do that and nothing should stop you from that yeah i was imagining if i have to make a game which game would i make with me damn first so if you guys remember talking tom so you know earlier it was so hard to achieve facial expressions like if you have to do facial experiments first of all they have to be like you know you can it's very hard to have real-time uh facial expressions but um could be a perfect fit that for that kind of game like you know perfect animation just single character so you know you don't have to optimize so with those kind of games and even the games like you know i was you know we are building a sugar game so i was thinking how can we use meta human inaudible 2 have you know meta human features enabled but as soon as you go far anyway you can't see facial expressions so that's where you know we can disable uh you know this particular feature and

just have really low version of lod which is very similar to what we have right now but if that somebody comes to you and your talks talks to you using voice chat whatever then maybe you know uh we can enable that feature so is it possible to disable better humans on other lods and just have it on on your lod0 or lod1 yeah i mean you can do that in fact you know for virtual production specifically we we have the opposite problem we don't really care about lod one two or three we want to have only a lod zero throughout so you can absolutely lock your lod saying this is the only lod i really care about so uh and and of course unreal also does its own loading automatically so they have the eight lods uh depending on how far you are from the camera uh you can it kind of automatically reduces the police uh based on how far you are uh so that also happens automatically so you can you can go both ways you can have you can restrict your character to have only specific number of melodies or you can also force your character to

have the best looking lod ever uh based on the based on where you are where you want your application to be deployed all right and you also talked about like you know your longer term vision is to make it multi-platform so when you say multi-platform you like go beyond unreal engine and give support to other other gaming engines like unity or any other that's already supported uh we don't you don't need to that's that's already there so if you go it's right now supports unity it supports maya 3d studio max all of your typical digital content creation tools all of that is supported right now right like today so if you go to the pixel bridge which is the application that allows you to download the better human uh it allows you to export the meta human to any of these dcc's so you don't need to it's not just unreal engineering perfect perfect yeah it saves lots of money for unity like no they don't have to do so much research sure

yeah and arvin in terms of like the longer term roadmap when you're thinking like three to five years out what's what's on the roadmap and when do we get to like fully ai you know avatars that do everything well uh let me let me answer that in a different in a different manner right now the vision that tim sweeney has is is of this thing called the meta verse right the metaverse is fundamentally a digital twin of our real world and you have your your digital avatars of yourself uh hanging out watching movies you're making movies you're playing games you're making games uh and i think uh zoom would kind of be obsolete in the few years of time where you'll literally be seeing you and me in digital that's kind of what the meta was vision is all over and all of us at epic games and within the unreal engine team are just working towards that vision we want to make unreal engine and all of

the tools that we are kind of giving it out like metahuman creator as as an operating system i would say on top of what the metaverse is gonna be uh so you know to kind of give the answer of where this is gonna be in three to five years that's the vision like the metaverse is the vision and all of what we're working towards like you know be it like ai related animations or ai related realistic human expressions or whatever that you can really think of it's it's towards this vision and that's kind of where we see the roadmap for meta humor and pretty much every other tool that they're creating within the games all right awesome so maybe if uh if you can arvin could we see like a demo of that uh in terms of uh anything that you can show sure uh so let me share my stream with us okay yeah so this is kind of what you see so earlier uh you know if you go to the better human creator uh you uh it's a browser-based tool so

you can sign up for early access uh and which you can if you go to pixel bridge which is a standalone application uh you can uh download a whole bunch of metahumans if you go to pixel bridge right if i go under limits here here uh we have about a preset of about 50 digital humans that you can start with and this is what i mentioned earlier this like you could start off with anything that kind of is close-ish to what you really want to have and uh and you can just dig in from there uh and look i'm kind of super happy that there's a lot of diversity to the characters also uh like you know this guy looks like me i think i'm sure uh a lot of hair of course uh but there you go so you have this guy so you can download that asset and then you can use the better human creator to kind of then sculpt the character to the way to kind of what you want it to be and you can access it but i haven't been creative by signing up you click on the start mhc it's transport it stands for meta human creator and that will open up a browser

that you can then use for creating your tool uh once you have that downloaded uh like i said earlier uh if you go on the export settings you already have export target all of these things so you can export it to 3d studio max maya umd blender od cinema it's a whole bunch of things right like it's a lot but i'm going to select unreal engine and all you have to do is select select the avatar or select the character that you have created and of course any character that you create will also automatically show up because it uses the same login um so if you logged in and created relation here also once you have there just click on it uh and just export it and it will take a little bit of time because it is 8k resolution uh but you can of course also export it as 2k and 4k if it is your own uh metahuman that you create once it once you export it will just kind of like pretty much automatically just show up in your editor if you have the unreal

editor open or whatever project that you have currently open it will just automatically import the digital human in there uh so it's usually called dp underscore dp stands for blueprint underscore the character name of the character that we just created uh kind of answer the question of of what abinash asked about earlier if i select the meta human and if i go under a load sync i can restrict or force my lod to be only a particular kind of pilot so i'm kind of right now forcing if i have i i have it as minus one then it will you know automatically do the unloading itself but if i make it zero then it's gonna force it to be always high quality uh lod based on uh regardless of how far i am from there uh now if you look at the uh the uh the metahuman itself it's very good looking like great looking hair unlike mine uh and you also have like pretty good looking clothing all of that is like you know symbolics and physics based so you have all of

that uh you also have the ability to edit the animation blueprint and make your own changes if you want to so it's uh you have that at the same time if i add it to let's say a new lego sequence let's save that in here uh let me add this guy to the sequencer uh you'll notice that this if i you'll notice that if i select the face kind of notice the the full rig that you get here so you can kind of animate those rigs so i can kind of like you know move around i'm moving the eye around or i can move my mouth around or whatever all of that is fully read like you automatically get the rig at the same time uh you have you also have the same thing with the body so if i select the body you'll notice that this also has a control rig fully ready to be animated so you can select that uh well it's probably i need to be

rotated rather mood uh so i can rotate that around for that or if i want to really move it i can also uh there's actually a little known thing in here which not a lot of people know about uh which is if i go under metahumans common controls utilities and if i run this uh you also have this very nifty little tool that i can kind of dock so i can rather than selecting it in the viewport which could get a little messy i can also select it select my rig from here and i can choose inverse kinematics or if i don't want inverse kinematics select that character and i can you know have have like all of my animations done through this which is a lot more which a lot more animators would feel at home

uh of course if you have excellent suit or any kind of low capsule uh that also kind of streams directly in here so to close that i don't need that anymore uh i can also select my character and i can use live link which is kind of available right out of the box so if i import that you automatically see this l link which stands for live link so if i go on the window and live link uh i have my iphone here with me if it's connected to the same ip uh then it kind of is available so it's kind of green right now and i can do under simulate more complete and now if i go here so you can now see as i'm moving my my face in my my home uh including my neck animations all of

that will kind of automatically show up in in front of this character also which is pretty cool so you can then also just record this which is kind of what the folks at the women's creator program did a lot they had multiple characters where they just acted out each character they recorded it using the take recorder which is a plugin that you can download and the cinematics you have to take recorded to record your animations and then uh and you can then uh have your your uh your character kind of act out and then you can record that animation then record and bring in another character record that animation then put them together in your sequencer and you can essentially just create a one-person act with multiple characters in there uh so which is pretty cool so it's kind of like just out of the box i literally did like like minimal setup for this at the same time one more thing i also want to show is uh mapping this character uh to a to let's say a third person character

like the third person template that's already available within uh um is also fairly easy so if i have turned that off should be here so you can see that this is a third person character like a typical third person character that you like the mannequin character i literally just replaced the mannequin character with my metahuman so if i click on play uh let me just select that and now i have the same character and now i'm also have like this is my this is the live link character that i had earlier but now i also have the same character who is now my playable character and it looks pretty all right so so you can do all kinds of like i said the possibilities of how you can use them when a human is family endless you can go crazy with it with games or you can also use it for virtual production you can use it for streaming or replacing your mug with uh with a with a better human and you can you know it's it's a pretty exciting world to be

yeah it looks really great it seems like we can use it right away exactly yeah i usually just like to replace my face with the metahuman because it's much better looking this might be internal information but uh uh like it is epic also working on any games using meta humor we also make a lot of games uh there's not a lot of i can i can talk about it all right let's go all right uh yeah uh avinash do you have any more questions no i'm just blown away by this deck all right well arvin in in terms of like any final message to our audience do you have any bits of advice for using meta human creator or any other message to our audience absolutely well there's a few things i can maybe kind of seed in one if you're really interested in meta human creator definitely sign up

for the early access part uh we'd love to hear your feedback on what it is uh once you sign up it usually takes a it doesn't take too long for you to get access but you know sign up and let us know feedback uh use it in in whatever kind of different applications and if you there's something very interesting do let me know do let us know we'd love to hear you out uh we uh and if you like to learn more about unreal engine and and and whatnot definitely check out the online learning portal uh we do have a ton of content in there for metahumans and otherwise for you to also start learning unreal engine uh and if you're in if you're in the region which i operated which is india and southeast asia if you're interested in getting trained also let us know we have we have been doing a lot of hand holding of hands-on uh training sessions for free uh for a number of folks uh so you know we'll be more than happy to hear from you if you're interested in learning unreal engine and finally we also have under legit five so you know download it it's on uh yeah early access also it's a that's

got i think that's a different podcast altogether but it's got a ton of new features that's very very exciting yeah and we love i will i'll i'll definitely reach out to you about that as well but yeah all right thank you very much for your time marvin until next time everybody that's it thanks alright [Music] you