IL Gigante: Michelangelo's David in VR

[Music] hi guys my name is chris evans i'm one of the lead technical animators at epic i kind of specialize in special projects which may make me an odd choice to be giving this this talk uh so over the years since my university days i've been a bit of a michelangelo file i have i've been told that i have more books on michelangelo than they have in the shtadel art library in frankfurt this is actually a plaster cast of david's face that i have at home i've studied michelangelo for a long time and i'm a big fan and big geek and part of that was you know 1999 i was at siggraph because i'm old and i saw some of the work that was done scanning the david and i've always been

interested in that so i thought it'd be an interesting thing to take a look at and that's what we're talking about today at epic we really like complex challenges so we have this kind of 20 time where we can go and do our own little endeavors and i was the games chair at siggraph last year and we thought hey you know this might be a really cool thing to do if we can work it out with the italian government and all the parties involved and in the end when it comes to like complex challenges we we didn't just want to i didn't just want to get the statue in but i wanted to kind of recreate the tribune and allowed you to kind of write a scaffold to any part of the statue and really look at it up close so to give you kind of a background on the scanning of the david the dave was scanned in 1998 by mark lavoy and his team from stanford it was a huge undertaking it took them 30 nights of hard work so basically

every night this the museum would close and then they'd wheel in their like laser scanner and set up these trusses and um work on scanning the statue piece by piece by piece uh up until like 4am and then they'd pack their stuff away and go to sleep and the museum would open again in the morning but it was a true labor of love it's it's interesting they scanned so much data that it wasn't really pieced together until about 2009 so that was a different paper but that that kind of tells you how high fidelity the data set is it's down to pretty much the quarter millimeter the scans are it's been presented over the years at siggraph multiple times many you you may have seen many geometry processing papers that come out use the david data set or at least attempt to use one of the versions of the data set because it's very hard to do anything with

the the billion point data set it is a billion points when i was in university uh they released something called scan view stanford released something called scan view what scanview allowed you to do was tumble around like a 300 polygon representation of the david or of one of michelangelo's other statues that they scanned and then it would freeze and when you stopped tumbling it it would queer their server and send you back like a high-res jpeg of what you're looking at so i was there in my dorm room like eating all my bandwidth like looking at like every nook and cranny of the david like you could see the chisel marks you could see the unfinished top of the head it was really really cool i think back then you know it was uh i think 99 or 2000. back then i even tried to like set up a little photogrammetry rig and then but they could detect if you were taking the same view and just rotating a little bit or something but i just thought it was so cool i really wanted to see that data i was really into sculpture back then

and even back then he was just one of my heroes so the data itself it has no uvs it was basically laser scanned there's no uvs it's all ranged data there's multiple representations of that data there's a 7 million representation there's a 50 million and there's a billion representation which is the one that took about 10 years to piece together so these different representations they don't really match each other they're warped in different ways so being able to use one and then reach up into another one is very complicated because the data sets aren't aligned you can't just bake the billion down onto the 50 million or vice versa a billion points is a lot of data i um one of my close friends mark alan he we were having a discussion just about data sets this size and um he was one of the original architects of houdini and he and he was saying oh we're going to work

on some geometry processing stuff maybe we can talk about this and he's like i just thought about it a billion points that's like a gigabyte just if they're if they're bulls and he's like let's make him vertices okay now it's 32 gigs of ram and he's like connectivity yeah i think you're on your own um but it's it's definitely definitely interesting that data that was captured in the 90s today we still just don't have a way to visualize it's just so so much so much data so i kind of talked about how the different representations that people had pieced together over the years didn't really match we weren't gonna so it started as myself and then i slowly started building this ragtag bunch inside of epic to take a look at this stuff adam scott who's one of our awesome character modelers on special projects you may have seen him talking um just the other day about the andy circus work we did with three lateral or uh the siren demo i kind of roped him

into this what what sculptor wouldn't want to see you know the data such higher the david at such high res but we knew from early on we're not going to be piecing together the billion point set i mean i could load individual kind of voxelized chunks of it but we couldn't so we needed to figure out how we're going to kind of cross back and forth between these different data sets there are lots of scanning artifacts so the lower res data sets people had already written test algorithms and stuff to fill a lot of the holes to figure out a lot of the artifacts as you got more and as you got higher and higher res there were some pretty gnarly scanning artifacts and stuff that i also didn't want to start cleaning up by hand and i'm not you know i'm a technical animator i'm not going to write an algorithm to to help me with that just to give you an example on the 50 million data set this is how many issues one of the geometry processing programs found with that some of the holes are larger than others some of them are

very small but it's all a bunch of issues that normal geometry processing pipelines that we use and games would definitely have a problem with so i'm going to talk a little bit about the pipeline that we went through to get to kind of shoehore the shoehorn this into uh unreal engine so we started with the the one of the lowest approximate approximations um well i think so this one came from the 50 million uh like a down res to 10 and then we took that into zbrush from there in zbrush we did a udem layout in zbrush so we tried to give it uvs and again all of this work we're giving back to the digital michelangelo project at stanford so whereas it would had this representation without these things before you know now stanford's going to have like a nice zbrush model with udem layout and stuff for future people if they want to mess with the data set so the zbrush sub sub d3 was 2.7 million

but it's pretty much undistinguishable from the 50 because we had vector displacement like with zbrush as you go down under the hood it retains a lot of those details and for me because it was a very difficult kind of non-rigid mesh alignment problem because i wanted to get into that 50 billion data set we had to really align the meshes somehow and the ways that we did that was by creating a low-res cage with zbrush but also loading kind of the 50 million into instant meshes which is i think one of the coolest programs i've ever seen it's just like magic very rarely does a paper come along and the authors decide hey we're going to make an application and then hey that application like works better than most standard software packages that i have to use every day but instameshes allows you to artistically paint well first off it picks a beautiful edge flow this is the sample edge flow that it generated for the david and you can see by default clicking one button it's already realized that you know along the sling it's going to put the edges here along

the face it's done a pretty good job you can go and you can artistically draw exactly what edge flow you want where and then it kicks out an example of exactly what it's going to build with quads and and placed pull polygons so this is kind of an example of down raising the 50 into something that i wanted to go and then use to wrap and so for wrapping i originally tried and then we finally got to work wrap three which i don't know if you guys have used wrap three it's a really great software for wrapping one mesh to another it's for kind of non-rigid mesh alignment you pick the different correspondences we use this all the time to cast different head scans into different facial topologies you can just pick a couple correspondence points on the two meshes and then it will warp one mesh into the other so it's really made for taking a

low res mesh and wrapping another the first time i fed it different kind of decimated versions of the david it was just like no not not going to work and i contacted them and we walked through some of the issues and it's just it's it's very very difficult to non-rigidly align one scan to another scan so in the end what we did is as i was mentioning on the previous slide we ended up wrapping the z brush the zbrush sub d3 itself and then once that was wrapped we bring that back into zbrush and reapply the vector displacement and then we had kind of a version that matched all of the like different i don't say they're warped just the way it's scanned um they can be slightly off just very very slightly but when you're talking about like a 20 foot high statue i mean slightly is you know that the toe's five inches

off at the base by the time you get there because there are a bunch of different scans all the way up the length of the of the statue uh so this was the the uv layout here um we used udims in ue4 to kind of maximize the uh uv space so you can see here how it's broken up we gave a lot of uv space to um to the face and then the body and you can see the legs and the base down there is like just a little bit of uv space compared to the face but that was because we really wanted to scan at least the face get the representation of kind of the head and shoulder area in in ue4 down to the quarter millimeter so we were able to get the statue represented in ue4 baked to a millimeter resolution from the millimeter data set but when it came to the face we looked

up into the billion data set and we now had our data set or our model and zbrush matching the billion model so here we're allowed to go in and now i can piece together all these chunks that make up the face and then i can bake those across but to bake those across i just couldn't find a program that could load up you know what does this load up like a just 20 million polygons just for the face and decimate that down into something that we could ingest into zbrush so one of my friends uh kevin what got a guy on our team kevin vassey she was like hey so why don't we try blender because blender's awesome and i was like look if you know if the standard if the standard mesh tools like mesh lab and all these things that you know people that write these papers use can't do it like blender's not gonna be able to do it and um i remember one one weekend we had

like one software package going all the way over the weekend and then it crashed like three days into the the down resting of the face and then in 12 minutes blender just popped out a mesh it was indistinguishable i think it was just a couple million it was significantly lower low enough that we could load it into zbrush and use it to transfer the details across to our mesh but that was amazing i mean you can see here the meshes are indistinguishable but on one side i think it's like 20 million on the other side it's like three so we had this high-res face we had everything aligned um but you know my topology was from instant or no it wasn't from instant messages at that point it was just from the zbrush downrise but to recapture all those details in zbrush we wanted the even even on the cage mesh we wanted the face to be higher resolution and around this time we were also working on the digital mic seymour and that's our in-house face topology so we used rap3 and we just wrapped mike seymour's face

onto the david because we were kind of running out of time and that worked really really well so adam uh was able to bake the details across and as you can see here at the quarter millimeter resolution mesh you can um you know in the experience you can ride the scaffold right up to date of its face and you can see the chisel marks from the rasp chisel that he used here above the brow where he left it a bit unfinished i mean he he was told that the statue was going to be seen like on the top of like on the duomo and then nobody would get really up close to it unfortunately he did such a good job it's curse of competence you know that they wanted it down where everybody could see it really up close this is a screenshot from in the experience kind of showing the um peter sumaceni he did a really good job setting up lighting exactly like so i had modeled the inside of the tribune which is where it's stored in florence and he did a great job lighting it to where it looks almost exactly like it

does there in florence here you can see kind of the the rasp chisel here there are so many details of the statue that come across uh it's kind of an annotated experience so you can click different uh you enter a mode and then there's little spheres you can click on and then you hear my voice kind of drawing on about the statue but um little things like the crack to arm like in 1527 they were riding and some people entered the palazzo vecchio like some dude threw like a bench out a window just because he was angry well the bench just like lopped off the david's arm outside imagine being that dude that's like oh hit the dusty trail you know like uh so um yeah you can see all of those fixes that were made over time it's it's really interesting um even that that's what the body at the one millimeter resolution this was the final topology that we ended up using

and then this is the the tribune here so we tried doing like in a radiance bake from the skylight itself uh pete set up like i don't know how many lights as soon as i unhit his layer and unreal like my world just became wireframes of lights pointing everywhere um but it's it's using the the dome itself to bake a lot of the illumination and then there's a bunch of little spots that they have set up in florence that just highlight the statue itself so looking at a couple things that could probably be better the experience we use the vive and we were it was really awesome that they gave us this extension box so basically the extension box would come out into the middle and then we had full five cord length around the middle so someone can walk around the entire statue but it's not the experience is not a teleport around experience it's very much you're on a

wooden scaffold and you can completely walk around the entire statue in 3d at life size and ride a scaffold up to any to anywhere you want even way above his head and look and look down we only really had time to look up into the billion data set to do a quarter millimeter bake on the face but um like i said we're gonna give this stuff back to stanford and if in the future someone wants to do that kind of bake on the entire on the entire body that that would be a pretty interesting science project it would be nice to see some of the unfinished marks on the tree behind his leg and different areas at that that resolution and it would be great to see some more statues they uh the team when they were there they scanned uh dusk and dawn and some of the medici chat chapel statues and um there were some some artifacts and stuff there's there's quite a there's three or four statues i think you can look at on scan view even to

this day but it'd be cool to see some more of the statues in vr and i could envision in the future with other data like scan the world it's a bunch of guys that go out and try to scan statues around the world be cool to have like a sculpture loader that would load it kind of at the exact size that is supposed to be viewed as the sculptor originally intended but so i just wanted to thank the people that were able to help me create the experience it was like a learning process it was the first level i ever made in in ue4 so that was good it got me out of maya for a couple days but um that's about it if you guys have any questions um be sure to fill out uh the little email they send you i think it'd be really great to see more people talking about kind of working to get sculpture or fresco or any you know ways to experience things in vr at a high quality that you

can't really experience without traveling there i mean it's pretty interesting to be able to also see things from different views maybe the different views the way the sculptor originally intended it that we put the david up on the duomo and you can check him out from there you know he he he changed a lot of the proportions just to be seen from that angle originally but uh sure questions hear a little bit more about people's feedback like artists sculpture sculpture when they see david in vr it was um i feel i almost feel bad telling you how awesome it is when uh originally we had planned to to have it here but it didn't really work out um there were some i never thought i mean i'm a michelangelo geek so i thought like oh this is pretty cool maybe some people think it's cool there are some people that took the gogs off and they were like crying and i was like wow it was pretty it's pretty uh crazy because you you just don't expect

it to be so large when you put the goggles on and you're there on a scaffold and you kind of look up at the statue and you don't expect the detail and kind of the hair and areas around the face and a lot of people didn't realize certain things he did with the eyes to cast shadows and stuff some people said oh it's little hearts or something but it's it's just really interesting to be able to see the statue up close uh what do you do for materials did the stanford data have any color and did you try to capture any of the or any of the subsurface stuff yeah so it had a whole bunch of color i think there were like 40 000 photographs of color that we had to use as reference because they were all i think it was film stuff and we had access to it but we just didn't have time so pete went in and he lit it and then we used the images that we had so that's probably the thing that is the most everything is super ground truth when it comes to the mesh but when it came to and if i can go back to my first slide when it comes to the lighting or the

surfacing we really had to kind of try as light-handedly as possible to add something and one of the most of the stuff that we used as reference was pre-cleaning so you saw like you can really see the chisel marks because it's a bit gritty they have since cleaned him with like a you know fancy process and he doesn't really look like that anymore but that was the data set we had so we just tried to keep kind of adhere to that right so it's just photographs it wasn't like registered to the points or anything no no i couldn't even get a macbeth chart so we yeah i wanted to just white white balance everything anybody else you can either walk over there or yell and i can repeat your question into the microphone um i'm just wondering if stanford or other people have continued to do this

with more statues and sort of what the time frame is now to scan versus the 30 days that it took so mark levoy and his team scanned this stuff in the 90s he's now at google um he is the licensor of the data in the us if someone wants to do something with it he told me um he he's been in the experience and he took the goggles off and he was like holy crap he's like i sure am glad i replied to that email from the animator at epic but he was blown away and they did a lot of scanning of different statues and it's in the archive but there's not really anyone using those at the moment for for anything like this there are other statues like saint anthony and i mean there's there's other statues that are more life-sized that are interesting to walk around but there's no plans that i know of unless like people's enthusiasm and this have and this has kicked some people into gear

um i was wondering if um what are your thoughts on larger scale virtual tourism like the pyramids of giza or the great wall of china or something like that would it be possible in today's or future technology to have that kind of level of detail for those kind of super large projects um i i've been taking stereo photos for like 20 years like i love stereo 3d i do feel that especially just like stereo photography i mean you feel the parallax it feels like you know like almost like you're there just visually so i do think that could be possible um i think you'll always get some kind of resistance from the people that sell all of the mouse pads and tourist stuff at those places i don't think it'll ever supplant the actual experience of going there but some people just can't you know so it's really interesting uh

so is it this also raises a a lot of questions in terms of just ip you know because it's a it's stuff that's public domain you know if you go into a museum and you take a picture of something you know that is in the public domain it's a slave reproduction so it's like you could then go print mouse pads so it was really really great for for mark and for the team back in italy to allow us to show the demo but it had the caveats of everything being encrypted and it was really super on lockdown so it's with data sets like that i mean to be able to capture some of the experiences to the level to replicate them means you're going to have to like live there for 30 days and the people that own those things aren't going to let people come in and mark had said that it was an amazing thing that that they let him and his team go in and scan it because it was at the time even newspapers in florence it was like america is just going to make a digital

david and nobody's going to visit you know it was he definitely said it was kind of just an amazing thing that they let that happen and they've had to be very respectful of the data and of the digital michelangelo project itself has to be very respectful of that stuff and keep it on lockdown hi um so i'm wondering if this project actually available publicly for people to experience with or that goes right into the next question no because i'm actually with stanford i work for stanford university yeah and uh i'm wondering also like do you know which department that mark was in when he uh processed this scandal i should have put a link if you just google digital michelangelo project there's the the whole web page as it was in 99 is still there and you can get a lot of information uh as you guys know when as soon as something hits a video car i mean with

directx ripper and it's so easy to rip meshes out of stuff uh no we would never be able to um to have it when we want to show it as an experience at a single place like at a conference we have to ask uh for uh to be granted the right to show it and then it has to be kind of on lockdown with uh encryption and um and it's pretty yeah it would unfortunately it would never work uh putting online the sistine ceiling however yes i have seen that yeah the sistine ceiling there's enough slavish reproductions of images out there that someone could piece together sistine ceiling in its public domain by themselves maybe in their bedroom but uh yeah in this case the data is only you know you're only going to find the data in in that data set and that's not something that's shared but for researchers he he grants the data all the time to be used in papers and things thank you yep

i see that the the data was uh taken in 98 and uh nothing was done with it until 2009. when did you complete the project um i worked on it for about six months uh with adam and and peter and shane helped me with uh some of the learning the engine in the ui um but it was about six months from start to finish but just in spare time like we call it like a 20 time project i mean when you in 2009 was you mean it was completed in 2009 there was a paper at siggraph i had mentioned the author of the paper where it was a new mesh alignment algorithm and they said and you know to show how well this works we're going to piece together the whole billion point date data set for the first time i'm wondering if you uh if you had the choice to do your own scans would do you think if you have the

choice to do photogrammetry would you be able to get the same results no no way not even close well i mean who knows in photogrametry you know you're just limited by how many cameras and what sensors you're dealing with and how um the the data is really really high-res and yeah i don't i don't i love photogrammetry and i do a lot of photogrammetry but the the details that are captured in this data set um yeah i it's hard it's just hard to compare even if in vr because in vr it's really you really uh the quality you're not getting the same quality that obviously the scans uh are able to capture uh so even with doing you mean like normal maps in stereo view or which like baking it to normal maps versus parallax occlusion or i mean getting the same the the final at putting the the final nvr using photographer tree you would be able to

tell to tell the difference between the scans and um if you like i said it depends on your budget i can i guess i can totally picture a photogrammetry rig that could do as good of a job with a lot of macro lenses and a big structure i just i do a lot of photogrammetry and uh yeah i i think it could get there yeah it's just the lit the laser scan picked up even the the smallest detail but it's a bunch of swatches you go like and then you have a swatch so in uh for many years at seagraph it was how do we align all these swatches because that's how cyberware laser scanners worked back then it didn't just auto align it like photogrammetry where it just figures out all the views and then does a pretty good job of making one mesh out of them it's a whole bunch of little pieces thank you yep all righty well i think we're out of time thank you guys very much thanks for coming out