Low Cost Facial Scanning Using Photogrammetry in State of Decay 2

My name is Matt Heineken. I am a technical artist at Undead Labs and today I'm going to be talking to you about the photogrammetry workflow that I developed for State of Decay 2. Uh over the last couple years there have been a number of really great GDC talks about photogrammetry. Uh companies are pushing the bounds farther every year and the quality ceiling is always getting higher. This talk is not about that. Uh instead this is a talk about lowering the floor, uh reducing barrier to entry and making photogrammetry available to everyone whether you have a AAA budget or an indie budget. So I'm going to be walking you through the pipeline that I developed over the last 2 years and I'll give you an in-depth look at the rig that I created and used to scan over 200 people for State of Decay 2. I will also be offering some tips and tricks to help you get the best out of your own scans. So before I dive too deep in this, I'd like to start by giving you some of our

project history so you have a little bit of context for what I'm talking about. Back in early 2016 State of Decay 2 went through an engine change mid-production which as you can imagine had a huge negative impact on our development schedule. So while I waited for some infrastructure things to get set up with our artists, I decided to take some time and see what I could do to speed up our production pipeline and maybe get some of that time back. And photogrammetry was one of the things on my list to research. Now at the time there were a couple games released that had used photogrammetry to great effect. But it was still fairly new to games and there weren't a whole lot of practical guides on getting usable scans. So I just started doing it to see what I could get out of it. I began by scanning some props. Uh quality ranged from so-so to completely unusable. Uh there was a lot of lighting baked into the textures. There was a lot of noise in the meshes and they just generally weren't that great. I ended up

spending more time cleaning up the mesh than I did just modeling that same prop by hand. So after about a dozen scans, I was about to write off the technology as being unfit for our needs. And then on a whim I scanned a co-worker's face. I brought this face to our only character artist at the time and he was absolutely thrilled. He turned that scan into a playable character that day and our character pipeline has been using photogrammetry ever since. So for those of you who are unfamiliar with photogrammetry, let's take a look at how it actually works. Now this isn't going to be a step-by-step guide on which buttons to push and which software. Those guides do actually exist on the internet these days. Um this is going to be more of a high-level overview just so that we're all on the same page. What is photogrammetry? Uh photogrammetry uses a series of photos to reconstruct a 3D object or of a physical object. Basically it's 3D scanning with a camera.

I begin by having my subject sit in the chair with even lighting and I take a series of photos from different angles. For a face I do four rows of photos at different heights generating roughly 40 photos. I then feed these photos into our photogrammetry software. There are a couple of good options out there. Um Metashape and RealityCapture are the two most common ones. I personally prefer Metashape. It's a lot slower but it gives more fine-tuning control. Once the photos are in the software, it analyzes these photos for key features. It then tries to match these points in multiple photos. Using these matches, the software uses perspective distortion to determine the camera's location in 3D space. When the camera locations are determined, the software projects those point matches and where the points meet, it creates a point cloud.

Next the software does this projection again at a higher resolution to create a much more dense point cloud. This cloud was about 17 million points. The software then converts the point cloud into a surface about 2 million polys. The photos are then projected back onto the mesh and blended together to create the texture. The resulting model is not quite suitable for suitable for a game just yet. Uh the mesh is too high poly. There's some noise in the in the model and some lighting information that still does remain. So at this point the model gets passed off to our character artists. They do smoothing, sculpting, painting out some of the artifacts. They also split open the mouth, add teeth, align the eyeballs and the back of the head. After that cleanup is done, it essentially converges with the traditional character pipeline. Uh decimation, re-topology, unwrapping

and projecting onto the low poly mesh to generate the textures. And this is that final scan in Unreal 4. So what makes this process so great for our faces? Well these same results can be achieved with a 3D scanner. But classic 3D scanning uses specialized equipment. Photogrammetry on the other hand requires a camera. Not many people have a 3D scanner just lying around but chances are you or someone in your studio has a camera capable of doing this. Photogrammetry is also more versatile than 3D scanner. It is very difficult to find a 3D scanner in this price point that can scan both a face and a cliff wall but photogrammetry can do both with ease. Scanning a face and cleaning up the model is considerably faster than modeling a new head from scratch. Our head pipeline went from about 1 week per head to 1 day per head. Not only is it faster but the quality results are better.

This is a comparison of our last hand-modeled face on the left and our very first scanned face on the right. The models that are sped up that by this process are the models that need it most. Organic complex shapes that take a long time to model by hand usually scan really well. Hard surface models not as much. Uh you can get a 3D model of like a pile of laundry in about 20 minutes but you wouldn't want to scan say a bar stool. You will spend more time cleaning up the model than you would just modeling it by scratch. This process allows us to exaggerate facial features much more than we could with hand-modeled faces which sounds kind of counterintuitive. If you're modeling by hand, you can exaggerate as much as you want. But when you're targeting realistic proportions, it's often hard to find the line of what is unrealistically exaggerated. A nose that may look fine on one face will be completely ridiculous on another. You can very quickly cross over the line and go full DreamWorks with exaggerated

features. So our artists tend to be conservative and as a result our models all start looking the same. But when you use scanned models, suddenly you can get these great Adrien Brody noses or Angelina lips, Lena chins. It's impossible to cross over the line into unrealistic proportions because your scans are based on reality. You just have to find subjects with these features. This is one of my favorite scans. This is Officer Diaz of the Seattle PD. He came in for one of our most recent scanning sessions and it's a great example of the kind of face we just wouldn't have tried to do with traditional modeling. So next I'd like to talk about the equipment and setup that I used for State of Decay 2. There's a wide range of hardware that you can use for this and quality definitely scales with cost. This is the quality that is achievable with the phone that is in your pocket right now. No lighting rig, no fancy camera. I literally pulled Dave into a conference room and spent 30 seconds scanning him with my iPhone.

And then there's the other end of the spectrum. Big budget studios and specialized companies like Pixel Light Effects build an elaborate camera rig with between 30 and 64 cameras that all fire in unison. These rigs can catch facial expressions flawlessly in real time. Companies like Infinite Realities are pushing the boundaries even further. They can use metered light bursts to extract specular and subsurface data to give a full PBR texture set right out of the scan. It's possible to spend several hundreds of thousands of dollars on a rig like this. 64 cameras, shutter synchronization, measured lighting, permanent facilities. It adds up. Now Undead Labs is a fairly small studio and we don't have the space or the funding to dedicate to something like that. So I set out to find the sweet spot between cost and quality. I endeavored to build the cheapest functional rig that I could and after several iterations, this is what I came up with. Six LED light strips attached to some

light stands using some clamps, a tripod to give our subject a fixed focal point and a single handheld camera. It provides even lighting without sacrificing comfort of the subject. The setup time takes about 15 minutes and the entire rig collapses to fit in a duffel bag. And this model of Dave that we've been seeing a couple times showcases the quality we are able to achieve using this setup. There's very little shadowing and mesh noise. It's not a perfect scan but it gives our character artists a huge jump start without breaking the bank. So what are the advantages of my rig over a multi-camera professional photogrammetry rig? Why not just go all in and build an expensive rig? Well my rig is cheap. The total cost of my setup including my own personal camera was just over $2,500. If you filter out the cost of the camera that I already owned, it was about 500 bucks to do this. It's portable. I can set up and tear down my rig at

will. It takes about 15 minutes to set up and measure. It doesn't require any permanent space in our tiny office. I can take my rig over to the Microsoft campus. I can bring it home and scan my neighbors. I can take it anywhere in the world. And the whole thing fits in a duffel bag next to my desk. The quality bar is good enough for what we needed. State of Decay 2 is a third-person action multiplayer game. We weren't creating any up-close cinematic Nathan Drake cutscenes. That kind of micro detail was overkill for us. This process doesn't always recreate exact look-alikes. When an area of a scan is particularly noisy, our artists have to do some guesswork in shaping features. But our characters needed to be believable, not identifiable. We weren't scanning famous people our audience would be expected to recognize. So even if there's some distortion in the model and scanned Jeff doesn't quite look like

real Jeff, it's still a completely usable scan. Now this particular method is not without its downsides. There's more cleanup time involved. The cost savings on the rig is not completely free. More artifacts in the model means more cleanup time for character artists. We average about 2 and 1/2 hours of cleanup time that we wouldn't have to do if we were using a multi-camera high-end setup. Some scans just don't work. We've got about a 3% failure rate where the scan is completely unusable and no amount of manipulating the settings can recover the data. This can be caused by a variety of reasons. Maybe the camera focus wasn't on point. Perhaps the subject moved too much during the scan. Maybe their skin was too oily or a combination of these things. We We compensated by volume of scans. We intentionally scanned significantly more people than we needed to account for scans that didn't work and to give us more options on which faces we wanted to

use in the future. Now there were some cases where the scan had to work. We held a contest for five of our fans to be in the game. And Microsoft held a charity auction where appearing in State of Decay was one of the things up for bid. Now those scans had to work. Failure was not an option. So when I was doing those important scans, what I did was I brought in just that one person to scan on that day. I scanned them and then I brought them back to my desk and sat them down and had them watch me process the scan. It was neat for them to see how the sausage gets made and it kept them on site in case the scan failed. When you take all of the photos in unison with a multi-camera rig, it takes an instant. But taking 40 plus photos from different angles one at a time takes time. Our volunteers have been pretty good at sitting still during these scans, but no matter how good you are at sitting still, eyes tend to move and blink. And so eyes are often problematic in our

scans and usually require a lot of cleanup. I did have one subject who didn't know he was allowed to blink. He spent several minutes trying his hardest not to blink and by the end of the second row of photos, he had tears streaming down the side of his face. We had to clean him up and start the whole scan back over from scratch. The comfort of the subject is definitely a concern. The lights are bright. And sitting in front of them for several minutes can get uncomfortable. And people who are more light-sensitive tend to squint, which gets baked into the look of the character. So my particular setup is designed to be a compromise between comfort and even lighting. There's a space directly in front of the subjects that has no lighting and it's got a tripod that they stare at. So they're not looking directly into a bulb. When in doubt, I prioritized comfort over even lighting. It's a lot easier to paint out lighting artifacts than it is to account for a squint. Getting a subject to relax can also be

difficult. We needed neutral expressions. And now people instinctively smile when they get their picture taken, especially by a stranger. And scans with smiles are unusable. These subjects are often strangers in an awkward situation. Sometimes I kind of feel like a dentist having a one-sided conversation with a captive stranger who isn't allowed to respond. So I find it helps to throw on some music or talk to your subject about what you're doing. Keep them engaged. Just make sure you don't make them laugh. I had one group of college kids come in all at the same time and after the first scan, they decided it would be fun to try and make the kid in the chair laugh. And even holding back a laugh, your muscles tense up and it changes the shape of your face. So I ended up having to kick the whole group out of the room and bring them back in one at a time. So if you do catch your subject moving during the scan, wait until they're done and they're ready to start again. Hold up your camera like you're going to start shooting again, but wait about 5 to 10 seconds for muscles to settle because when someone thinks they're relaxed,

they are still tensed a little bit and it takes them time for muscles to settle. We've had tremendous difficulty with scanning facial hair. It's simply a limitation of the process. The camera estimation step evaluates all the contrast points and averages them to find the camera's location. Short facial hair throws in a large collection of faulty matches. It may find one hair in one photo, a different hair in the next photo, and think they are the same hair as a match, which gives us incorrect location data. On the left, we have Dave with 4 days worth of stubble and on the right, he's clean-shaven. You can see there's much more noise in the model without or with facial hair. And not just in the beard area, but everywhere because the camera locations are being incorrectly calculated. Thicker beards often turn into a giant pillow of noise if they even scan at all. Often times they just turn into a hole in the face. And this limitation on facial hair made it harder for harder for us to find

volunteers because beards are in fashion right now. I did have one subject come in on a Saturday. I was doing a Saturday session. And he was my last one for the day and I started getting texts about 5 minutes before he was supposed to arrive. Okay, I I missed my bus, but I'm on the next one. I promise I'll be there. Don't leave. And so I ended up sitting there waiting around for an hour on a Saturday for this guy to show up. He arrives at our building. I go down to let him through the elevator. The elevator doors open, gigantic lumberjack beard. And I told him, "It says right in the listing we can't scan that." He says, "Okay, hang on. Don't leave." And he goes across the street to a corner shop and buys a can of Barbasol and a razor and comes back and shaves in our bathroom to be in the game. But ultimately this constraint is not a huge issue for us because of the game we were building. We wanted to reuse faces for randomized characters and we could add a beard to a character later a lot easier than we could paint out the beard on a scan because you can't see the underlying facial structure underneath

the beard. So let's take a look at the final results of all of this. We shipped State of Decay 2 with 75 unique faces. Every single one of them is generated using this method. And this is some of our heads in context. We swap out different hairstyles and clothing to reuse faces for different random encounters. So now we get to the practical portion of this talk. I'd like to share with you some general advice to help you get the best possible scans you can. First of all, you always want to shoot in manual. Exposure adjustments from one shot to another will make it much harder for the software to find matches and it will cause a lot of problems with texture blending. When you have two different exposures, when they try and blend together, you're going to get very patchy, uneven textures. Shooting in raw allows for global exposure correction and helps avoid

compression artifacts. I often get questions about, you know, is my camera good enough for this? Is it high enough megapixels? All that kind of stuff. Uh don't worry too much about the resolution of the camera. More pixels is good, but up to a point. The final textures are generated from a series of photos stitched together. So after a certain point, more pixels is just wasted disk space. More important than resolution is sharpness. A 12 megapixel photo that is tack sharp will provide better results than a 40 megapixel photo out of focus. That being said, you never want to sharpen your photos artificially in post. Unless you know exactly what you're doing, you don't want to adjust your images at all in post because this has the potential for nullifying point matches and can seriously mess up your scans. Shiny surfaces do not scan well. Specular highlights confuse the software

because a specular highlight moves across the surface depending on viewing angle. The software doesn't know the difference between a moving specular highlight and a fixed point. And so it throws in a large collection of faulty matches. But skin is shiny. And particularly oily skin does not scan very well. A polarizing filter can help a lot with this. On the left, I just took a normal photo of Dave. And on the right, I took the same shot again with a polarizing filter over the lens. You can see the large highlight on the forehead goes away. But it doesn't get all the highlights, but it helps. Uh polarizing does polarizing filter does reduce the amount of light reaching your camera sensor, so you're going to want to increase your camera settings up by a couple stops. You can also use anti-shine powder. We've had tremendous success with this, but it requires your subject to be comfortable with a stranger putting makeup on their face.

State of Decay 2 takes place during the zombie apocalypse, and having female characters all dolled up in full makeup doesn't really fit the narrative of the world we were building. So, we wanted to scan faces without makeup. But, we quickly discovered that not all females are super happy with someone taking 40 photos of their face without any makeup on. So, we quickly changed the listing to request natural colored makeup, which helped a lot with reducing skin shine as well. So, on average, our female scans ended up more reliable than our male scans. Before you start a scan, take a photo of your subject holding up their name. At one point, I could name every single one of the 206 faces that I scanned, but I wasn't the only one working with these photos, and we needed a way to identify these faces. Folders can get renamed, tables can get sorted. This identifier photo is a great way to keep things organized that can't get decoupled. It's also super helpful when you're processing a full day's worth of photogrammetry.

I can dump 30 scans into one folder and use the thumbnail view and look for this name photo as a delimiter for where one session ends and the next one begins. Lastly, there's a lot of debate within the photogrammetry community about whether to rotate the subject or to move the camera. And we've tried both. The first approach is to have the subject sit still while you move the camera around and take photos from different angles. The subject is able to sit very still and focus on a fixed point. There is, however, potential for blur in the camera. You want to take these photos as quickly as you can, and having the camera on a tripod or even a monopod gets in the way with all of the the legs from the lighting equipment. The other approach is to put the camera on a fixed tripod and rotate the subject. Put in Put them in a swivel chair and rotate them Sorry. Around the camera. This approach gave much more even

lighting. Uh a feature that may be shadowed in one shot can be directly lit in another, and those values average out. But, without a single fixed focal point, there's a much higher probability of your subject not quite lining up right. They may look a little higher or lower on each turn. And just the physical act of rotating your subject causes muscles to tend and shift, and they put their weight differently. So, the best scan we ever got was actually using this method. But, ultimately, we decided on moving the camera rather than the subject. No idea what that was. All right. Um so, the quality ceiling was lower, but the average was a lot higher. So, there it is. Uh photogrammetry can be a great tool for boosting your production regardless of your budget. I hope you've learned something here that is helpful for you to get your own scans. Um like to thank you all for coming. Looks like we've got some time for

questions. I can take some of those now, otherwise, we can meet across the hall uh in the atrium for further questions. Yes. Thanks for that talk. It's really cool stuff. Thank you. My question is, when you were rotating the model, did you ever use like a headrest or even some sort of brace that you could see only from the back to help eliminate some of the movement? Um we didn't go too far into that. Um we we were experimenting with a whole bunch of different setups all at the same time. Uh basically, I I took one of our swivel office chairs, I pulled the wheels off, and I fixed it to the floor. So, there was some backing on the chair up to about the upper neck. Um we haven't tried to like get them completely braced into the chair. Um comfort is definitely an issue, and having strangers come in and get

strapped to a chair is a little awkward. Mhm. Yeah, cuz I've seen setups where they have some type of not restrictive movement, but just something that the subject can press their head into and would definitely help. Right. Yes. I have a question about photo scanning, but not particularly faces. So, I'm not sure you have experience with that, but I scanned many models last year, and I ended up failing with a different kind of stuff. For example, a lemon. It's it's it's very tough because uh even colors and you know, shiny surface, I know. Mhm. I started to use different kind of markers. For example, barcodes visible on the photos. And then even using different kind of markers, red and black markers on the model itself. And didn't work at all. So, I'm just wondering if you have any experience with that. With with shiny and flat colors using markers, because none

of the actual ticks I drew drew on a lemon or using those uh uh barcodes. Yeah, um we haven't tried too much with markers. I've used a couple markers when scanning larger outdoor objects like boulders and stuff. Uh for something the scale of a face, I don't tend to use markers for it. Um if you're trying to scan something like a lemon that's got a bit of a shine to it, if it's something you don't care about, there is anti-shine spray you can put on things. Um we've tried scanning shoes that have like reflective, you know, for seeing people in the dark when they go out running. Um putting some of the the anti-shine spray on those as long as you don't care about the shoe is fine, but if it's something you want to keep, that becomes a bit of a problem. Uh a polarizing filter would definitely help in a lot of those cases. Spray, that sounds cool. Thank you very much. Mhm. I think they're all on that side today. Hi. Uh so, my question is also not

really regarding facial scan. So, um I mean, we've we've been trying doing photogrammetry for a few objects. So, how do you deal with objects which are made with glasses? So, is there any way we can scan them and use photogrammetry for that? Uh scanning with glasses? Yeah, I mean, the object, yeah. So, it's a multi-face which are pretty complex, but made of glasses, yeah. Uh generally, we have them remove glasses, and I find it helps to have them remove their glasses for about 10 minutes beforehand. Cuz like the the red marks you get on the bridge of the nose, Uh no, no. So, I didn't mean the faces, objects. So, for example, a glass artifact. Uh glass object. Yeah. Yeah, I wouldn't know of a way to scan that well. Okay. Um anything transparent is even worse than a shiny surface. The the software is not going to be able to pick that up very well. Okay, cool. So, I mean, um and does um depth sensing and depth cameras help in that or that also doesn't Uh potentially, there there might be some hardware out there that can do that kind of stuff. I don't think photogrammetry will ever work very well with that.

Okay, cool. Thank you. Hey, so I remember you mentioned uh earlier not to do any post-processing to the images unless you knew what you were doing. Mhm. Do you do any post-processing, or you just throw the raw photo into Agisoft? Uh I do minimal post-processing on occasion. Um There's, you know, sorry. The triangle of how much light you have, your camera exposure, and like the the film ISO and stuff like that. Um trying to get enough exposure on the image so that you can see what you're looking at. Um Generally, when you're trying to get enough exposure, you can always just throw more light at it, but that gets very uncomfortable for people standing under very bright lights for a long time. And so, we had to limit how much light we pour onto our subjects, and so occasionally, I had to um artificially increase the exposure, but I do it on the exact same settings for all of the photos so that they don't get

mismatched. Usually, where the disconnect comes from if you're doing um post-processing stuff, is when one photo ends up diverging from the others. So, you never want to do like exposure correction on one photo and not the others. Mhm. Okay, cool. Thank you. Mhm. Hi, Matt. Hi. So, uh my question is what kind of lenses will be needed for scan? Because I think uh if you if you use a different lens or special lens, there will be some distortion of the facial features. Uh Different lenses don't make a huge amount of difference other than you know, how sharp your lens is can definitely help. Uh whether you're using wide angle or if you're using a tighter lens, it doesn't matter so much because the software takes into consideration the focal length of the lens. So, if you're getting a lot of distortion from a single image from a

wide a wide angle lens, the software can read in that data. And so, no matter which angle you look at it, it reconstructs the the actual uh object that you're scanning. Okay. Uh sorry, I have another question. Yeah, go ahead. Uh so, will some small details like eyelid be scanned well, or you you have to manually model that after the scanning? Uh eyelashes, you said? Uh eyelid. Oh, eyelid. Yeah. Um because of the nature of scanning someone with a single camera over multiple minutes, Mhm. people are going to blink. And so, the eye is almost always require cleanup. Um there's not really much we can do about that with this single camera setup. If you're doing a multi-camera rig that all fire in unison, then you're going to get that kind of detail out of the eyelids. Okay. Okay. Thank you. Mhm. We have one on this side. Yeah. Did you have any experience with flashes or strobes? We did. Um

I We found that constant lighting tended to be a little bit better, especially with people not kind of wincing when the flash goes off. And so, having slightly dimmer but consistent lighting made the people relax more. Yeah. Thanks. Yes. Um you mentioned Last one. Hm? Okay. Uh you mentioned that some of the higher-end rigs can generate textures like glossiness and subsurface and stuff. Um what sort of pipeline steps did you guys do to get those maps? It was just up to the artist's discretion or did you have automated tools? Yeah, it was mostly up to the artist. They did a lot of painting just to recreate that sort of thing. Got you. Okay. All right. Thank you, everyone. Uh if you've got more questions, we can meet me over across the atrium.