Capturing Reality: Gathering Reference for Forza Motorsport 5

[Music] hello all right everyone my clock shows 3:30 so I'm not gonna waste anyone's time and I'll just go ahead and get started so I wanted to start out and thank everyone for coming I know that capturing reference is not the sexiest topic in game development so I'm really appreciative of you guys for coming so as my slide indicates here my topic today is about our reference gathering techniques for forts of motorsport and in particular I'm gonna focus on some of our best practices or on taking environment reference so how we sourced a lot of the tracks in our game all right okay so just as a quick background turn tense videos which is the studio I work at is one of the first party Microsoft Studios so our mission if you don't know is to help create Xbox games that solve our consoles to be frank and so our

particular studio was formed to create racing titles and so that is my destiny in life and so so you can see here on some of the titles that we've created so far and so on the left you see forces 1 through 4 that were created for the original Xbox and Xbox 360 and hopefully it's not a big surprise to you that our latest project was for its of motorsport 5 which shipped as a launch title on the Xbox one back last holiday so a little background about myself I left Disney animation after over 10 years and I joined turn 10 at the beginning of production on Forte's motorsport 4 so it was about four years ago and I operate as a technical art director there okay so for folks who aren't familiar on that much with our title I thought I would play a quick clip to introduce our game and give you a sense for the scale and the kind of content quality that we try

and attain [Music] [Music] all right that was a condensed from our launch trailer so that was just a 30-second taste of what we have going on in the game and if you like it buy more all right so what am I going to talk about I wanted to kind of intro the topic the fort's of motorsport environments that we build mostly represent real-world tracks and by real world I mean they exist in the real world and so we're trying to model after some reality that's there we also have a

handful of fictional circuits so those are circuits that exist in like a certain real-life environment or motif and so if our latest project Prague was our cover track and so we went to the city there's no race track through the middle of the city and so we kind of had to invent some circuit through it and so it was a little imaginary and piecing together this and that to try and build something that would make a compelling race course so all of us in this room when we're creating 3d art content hopefully you guys know that we have to create environments and worlds from thin air into 3d and so like in our game there's a lot of real-world environments and there's a lot of imaginary environments that you might have to build and so I hope some of these techniques um are you'll find applicable to the specific titles that you're working on so no matter where we work in games our film we have to build these worlds with textures and polygons those are the tools at our disposal so anything that we can do to author those textures and polygons faster is going to

result in cheaper budgets more content and just general fairy-dust feelings I guess from everyone involved in the production of the game so at turn 10 when we sat down to imagine this new project that we were going to be embarking on four to five it was for the launch of a new console generation and we knew that there was going to be some huge content demands and that we were going to have to push it further than we'd ever pushed it before so as we started investigating our processes we kind of became convinced that the solid reference gathering techniques were the beginning or the foundation of creating content more quickly and accurately so also just to be clear reference gathering is the process of trying to document an environment or capture everything you can about a certain place so that your environment team can create those results in 3d so that's going to be the main topic of my talk today all right so quick agenda this is one of

the few bulleted slides I have on here so what I'm going to talk about today is I'm first going to describe the old-school ways so how we used to gather reference and then I'm going to quickly switch to some of our newer techniques so I'll introduce you to our hot new texture action I'll talk about some spherical video things that we played with and use it in the production of the game I'm going to talk about laser scanning and then finally a few quick words on why we feel like this is important and hopefully you can get something out of it as well okay so when I first joined turn 10 the team wanted to send me out on a reference gathering trip just so I could see what it was going to be like so a few weeks after I started I went on a trip to the Indianapolis Motor Speedway and so there we were starting from scratch the process of building a new track so what is the absolute minimum that you need to capture reference for a 3d environment that you're trying to build its photos and so photo reference is a pretty

standard practice in reference gathering and so like many other studios we will send a whole team of three or four artists out to the track and we will arm them with cameras and set them loose on the track and we have a spec that we follow so here's the specific things to take pictures of and we will go to that track so like many artists with cameras we unfortunately get more than a few photos that end up looking like this but but in reality we actually end up with tens of thousands of photos on each trip so we have all these photos that are scattered all over the place and it's really daunting right we bring the photos back and then the artist sits down and opens this directory and it's like here's five million pictures for me to sift through and how do I piece that back into an environment that I can actually build so what's missing from this is context you don't know where a particular photo is in relation to something else and to

just take a folder like this and expect a team to build a 3d environment is asking way too much so on our indie trip one of my particular responsibilities was to go around and take pictures of buildings so it all made sense to me when I was there because I was in the physical location so I walked around I'm like hi I see a building let me take a picture of it and so I'd go off and take pictures of buildings then when I got back to my dismay I had rolls of pictures that look like this so this is a program building so where is that on the track nobody knows so people would place it randomly so what we found is that there's a big disconnect to being at a place and using our human brain skills to kind of process exactly where you are and to get that feel and there's that big disconnect to a whole directory full of images and so the poor person that is on the other end of this who wasn't there it's tough for them to get some context about what this picture actually represents here's another building that uh has awesome smoked turkey legs and so where is this

building in relation to anything else so to add context to these photos what you really need is videos as well so luckily that was part of our reference gathering spec so we also went around and shot videos and so we had some kind of method to the madness so part of our reference spec was to go around and mount a video camera or or DSLR that had video capabilities and kind of drive the whole circuit of the track and then take 45 degree increments so we would go around once with the camera facing forward and 45 degrees right left and 90 degrees and so forth and so the poor artists back at home end up with something like this so at Indy before I play this video instead of giving us an awesome Indy car for us to cruise around and take videos of they provided us with golf carts because they have a golf course that's part of the track and so we got these each one of us

got an awesome golf cart to race around each one of these took 15 minutes to get around the holes circuit so to capture all the angles that we wanted for our video respects requires eight drives around the track and it was fatiguing both to us as the driver and as the artist back at home who was trying to witness this so this is some of the awesome action you get so when you're an artist back at home and you're trying to build like this track and you see hey there's a truck in the left oh wait now I passed it so I don't know what it looks like then you got to go to the 45 degree or the 90 degree left one and kind of like get motion sickness and see this and so this was our video spec this is what was happening as part of part of the thing so also so photos and videos are kind of a given so to bolster some of our high tech redness we also lugged around some other digital tools so this is a digital level where we can go around the track and you place it down and it tells you how many degrees the track has of camber

and so that's like the angle of the track for car optimal performance so this is me diligently marking in my journal the measurements coming around turn two of the Indy Speedway and what we would do is we would have to go to several different spots along that line because the camber changes actually as you go from the inner part to the outer part and then you march down another like 10 20 feet or so and then do another thing and then so I'm making little X's on my map and saying like 4.1 degrees and then someone at home would have to kind of process that so unfit this took friggin forever so it sucked alright so the last part of our reference gathering spec was GPS so this is our track production lead Trevor and he's walking around with his nerdy GPS backpack and so he had to walk through this the entire two and a half miles of indy speedway on every every edge that

you can see so here he is on the inner inner portion of the track for f1 and he would walk down this line and take GPS measurements and then he'd walked the other side and take GPS measurements and then if he had extra time he would say oh there's a lamppost there I should probably snap that and so the end of that whole process if you have a good satellite signal and you have a good pair of walking shoes you end up with something like this so this is our 3d map of the Indy Speedway and you can see every single footstep that he took and kind of the measurements and the guides that we have to build this so that is a summary of our old-school ways when we come home from the reference trip we would take the photos the videos GPS data all that stuff and we would just hand it over to our environment team and this poor team of like you know tens to hundreds of people have to like really take this and puzzle back together and figure out like what exactly is what and so they download

internet images and like poke around and then like come by and ask like you know this port-a-potty that you took a picture of like where is it I don't know like watch my bouncy crappy video so it's actually amazing that we ended up shipping our game at all so some of the questions that we found ourselves is how can anyone tell what's happening with this reference that we're gathering a second question is how do we even manage to build anything out of all this stuff that we come back with and the biggest question of all is what if we miss something so just to touch on that real quickly when we started when we started our process for four to five we knew that no matter how complete you try to be with these old reference techniques there's always something that you're going to miss so if you have a picture of an alleyway and there's something down there and you're like well we're gonna drive by it but what's down that alley we're like perhaps nobody took a picture right there or like there's a building what's behind that building because we actually have to build it like I don't

know just make something up so as we settled on Prague as our reference or as our kind of cover track for for the game we started planning our reference needs and creating the new plan for the trip so since we are going to be traveling a particularly long distance so here's us going from the Seattle area to Prague we thought it would be really important to our studio management and executives that we plan a complete coverage spec because the worst thing that you can do is say oh we forgot something can you send us back to Prague despite the deep pockets of a corporation like Microsoft okay so how did we arrive at a situation where we try to consider our reference gathering techniques because as I mentioned before it's not a sexy part of track production and when you say well we have to build all these environments it's not the immediate thing that you jump to so this is kind of a breakdown the tech our team did some data mining after four to four and we're like well what sucks about building our environments and so I took some key figures off this but here you

can kind of get a sense for this pie chart of where exactly we spend our time so building this detailed environment actually takes a lot of man time and so when we sat down to plot out what takes the longest starting from the beginning of their very very pre-production ish kind of stage we were surprised to find that the process of laying out the shape and creating all these rough objects actually took a really long time combined so what you see here is there's two areas in particular there's this coarse phase which is our process for taking those GPS splines and kind of laying out a road surface so that we can see what the driving area feels like and then this rough phase at the bottom in in the right hand corner here that's the process for taking blocks of white box objects I guess and saying like here's some trees here's a building that lives here and so you can see those two added together actually take almost as much

time or about as much time as our refine stage which is where we're actually building out all the content and like making sure it goes properly so we attributed a lot of this kind of get up and running kind of time to artists having to puzzle these reference pieces back together you know like we can't take all these reference images and everything back to the studio and then have them start building stuff and then like you know the first week we have some awesome drivable surface it takes a lot of time to process all the data and to organize it and everything so we're like our general inability to quickly get up and running is really restricted by the kind of reference that we're coming back with even for real-life tracks so we rolled up our sleeves at the beginning four to five and we knew that we are targeting this much more content and we decided to design an updated way to gather reference which was the fundamental source of our content production going forward so what I'm

going to cover for the rest of the slide deck is how we've evolved our reference techniques how we do things right now and hopefully give you some insights that you can take away okay so photo reference I just want to note we still take photos as part of our reference gathering but we do it a lot smarter and I'm gonna jump into some details on that second point is inspired by Henry labounta who used to work at EA and moved over to Microsoft for a bit in particular he gave a GD c 2011 talk called art direction tools for photo real games and and in that talk he talked about some of the reference gathering and texture capture that goes into making video games and we discovered that the way we were taking photographic reference was honestly stupid and then the third point is that you may have heard at GDC here a lot of discussion about next-gen pipelines being based on a physically based model and so that includes material and lighting a lot of next-gen

pipelines are moving to that and a lot of the stuff at the tail end of Xbox 360 and PlayStation 3 generation were kind of headed towards this direction anyways and physically-based model means that you're modeling all your lighting and your material system using mathematical formulas instead of just squinting and eying it and saying that that's about the right color so one of the critical pieces of that pipeline is if you don't feed in good stuff then you're gonna crap out as well ok so I'm gonna talk a little about our photo reference gathering techniques so what is notable about this particular photo up here except for the awesome fact that Prague has a hippies festival that has its own website the awesome thing is this little guy right here so this piece of tech is a portable Macbeth's chart so my best chart is something that has been used by professional photographers and cinema cinematographers to calibrate color for years now so

chart is a swatch of 24 bits of color that have known RGB values and so our reference team will lug one of these around each guy gets one and we try and stick it into every picture that we take on our reference trips so how does this help us so this color checker tool it comes with this a pretty awesome software that automatically detects the chart in the presence of the photos that you have and it matches the colors that you have against the known RGB values so because they can calculate the deviation between the actual known colors and what it's actually showing up in your camera it allows you to create a custom color profile for your camera and the shooting conditions at the time of your reference gathering so if you keep a fixed exposure and ISO settings and everything and then you just say like from this time to this time I was using this particular set of settings you can create a camera profile that automatically color corrects for for the

environment that you are working in so how does that color calibration work so with that profile you can plug it into a tool like Adobe bridge and so when you pull in Adobe bridge we have outlined here where you assign the camera profile so when you assign this a color corrects your image and you can batch do a bunch of these at the same time and this ensures that the colors that we're feeding into our physically-based system are actually correct and at a known good standard which is really important as I mentioned before so the other thing we can do with this chart that we have is get some white balance accuracy so on the left side of the chart that you see in the picture here we have varying shades of white to black and so you can just color sample the white part and the black part which are different from the color calibration and you can set a good white part white balance and black balance so that you ensure kind of like that your values are in a good range and

so we try to do this for every photo that we're going to be using for a reference and then we save it that way and commit it into our repository so that anyone that's trying to take a swatch of this is already working good kind of stance so here's a quick example of what this can do for us so we actually took a couple pictures at a racetrack with varied exposure just so we can see as part of our testing as we were leading up to a reference gathering so on the top you see image number seven three three six we took it at a low exposure and then later on we fumbled around with the exposure settings and then we get the one on the bottom so this is before then after the process you end up with the pictures on the right and it's to a known good value so no matter what circumstances you were trying to take the picture and because we're capturing RAW images we can expose adjust exposure and the white balance and all that afterwards and then we end up with a

known good target right one final thing I wanted to mention is because we have this increasing reliance on a physically based material model where almost all the lighting and material calculations are done with mathematical formulas that are meant to mimic the physical world it's especially important that these images and the values are correct so if you don't put in the right stuff then you end up with colors gone wild and we've actually seen that on past projects where if we adjust exposure or bloom on something something will unrealistically blow out and we've always had to go in and kind of like mess around with a texture and be like is it too white is it too high a value and so this is an image of one of our environment artists took a bunch of gnome things that occur pretty frequently on racetracks and he kind of dialed in that the appropriate value numbers that each one should be an

approximate range and so this way if you take this image this is a 32-bit image and you start adjusting exposure the stuff that you expect to blow out when exposure is high is going to get that and that's kind of key to feeding our our material and lighting pipeline I just wanted to call out real quickly I don't have a slide for this but Nick's a goal our rendering engineer is giving a talk about the material and lighting pipeline for 4:00 to 5:00 tomorrow at thirty so you should attend that if you're interested to see how all this reference gets fed in a little bit later okay so that's reference images that's kind of where we've evolved our pipeline and the images are at least a lot better quality and to a known circumstance even though we still don't know where the heck anything is so remember this crappy video from Indianapolis the still video is boring to watch and the field of view really limits what you're able to see so we thought to ourselves wouldn't it be

sweet if we could just pause in a certain location and then look around so if we pass that truck like we can just look at it instead of having to stare this and kind of decipher exactly where everything is so we played around with things like QuickTime VR and panoramic photos and all that stuff where that you have a way of taking a 360 photo and you can just stand in one place and you take photos from all angles and then you can compress it down into something where you can standing over one place just look around in all directions so we're like it'd be cool if there was some kind of amazing tech out there so that we could do that with videos as well and it turns out that there is and we just didn't know about it because we hadn't paid attention really to our reference gathering processes before so after some extended research our tech art team came across this and this is a spherical panorama camera and in this specific case this is a ladybug camera and it's used by a lot of the Google Maps and

Bing Maps as they capture some of the stuff that's feeding into our modern-day mapping technology and so we love this camera so much that I'm gonna put a halo on it while I talk about it so we love this ladybug the these kinds of devices aren't really that common or widely used so it actually took us a while to track down something that would actually suit our purposes and be useful in production so how well this worked was actually a big surprise to us and we were stunned at how really accessible this tech had become so a couple things about this camera this camera comes with a bulletproof awesome looking black suitcase that looks like one of those nuclear code things that you carry around so it's always a pleasure to walk around with when we're in Prague in other places we took the suitcase along on all our reference gathering trips for four to five so the camera itself is really sleek this camera comes with an array of five cameras around the center core and then there's one on top and there's various modules you'll see at the bottom

that kind of dues are streaming over wires to your capturing device and also do some real-time stitching of the images together so whenever it was appropriate we would take this camera out we'd mount it on a car or like walk around with it and then we would record the 360-degree streams so real quick this is the 3d video process that you use in order to make it useful to your artists so on the left here you can see we record the video and we record a certain number of them each one has to go through an encoding pass to get it into a format that you want and then you have to have a player that kind of lets you watch it back and so I'm gonna provide details on each step in the in the next few slides so here's us there's me in the silhouette driving around Prague with this awesome contraption going outside our window so this camera you can see the wires heading into the car it's actually hooked up to a laptop that is capturing all the streams and so the six cameras on this guy

simultaneously record a huge gigabyte size streams and it stitches them together in real time after we do the recording and we have a full laptop we have some software that comes with the camera and it lets you encode it into whatever format that you want so here we are here's a video that's kind of doing the 360 mapping and the distortion to kind of make it so that it can map onto a spherical viewing kind of air kind of thing so we encode the video into a panoramic 360 stream so this is what you get at the end and so this right here looks like it's a standard video and so as I start playing it you'll see that we're driving down but the difference is I can use my mouse and just start turning around in any direction so I can see exactly where something is I can take it to any point pause zoom in on a certain location and see exactly some detail that I might have missed and

it's all the single-stream I can jump to any certain point in the stream and look around so we forgot to take pictures of some tram wires when we are out there and we're like hey that's all over the place in Prague so we could jump to a certain location scope around and really get that missed reference without having to ask for a plane ticket back to Prague and so it's really cool because this also gives you the sense of immersion and the artists who weren't able to go on the reference trip can kind of play these back everyone has access to it and then they can really get a sense for what is it like to drive down this particular part of the track what did things look like and what are some pieces that we would have otherwise made up in any other circumstance so this has been a tremendously useful tool for our artists who didn't get to go to capture the feel of that environment and make it a lot more interactive so that was a cool thing that we discovered in the course of four to five all right lasers lasers make everything sweet so

I'm going to talk a little about our laser scanning processes that we we were using on four to five so here's a quick diagram that's an overview of how we make laser scanning useful in our pipeline so first thing we do is take a bunch of scans it goes through a process called registration and then after you register that means it's stitching all the point clouds that you get back together in one global coordinate system so that you know where everything is in relation to anything else you can slice it back up into pieces that are pieces that you might model you can also stitch it together into multiple bits and then make it so that you can move pieces around if you want to take a little artistic license and so this is really useful for blocking out fake environments other things that you might do and so we kind of stumbled on laser scanning in a in an interesting state of events so one day we had a vendor come by to show us the state of the art and laser scanning for objects like a car

and so these particular vendors accidentally set the vendor to scan everything cept the car so they set it off and it was going and then at the end we're like and this is taking forever this sucks and then they pulled up the data and they were looking at each other going like oh where's the car and and then so while they were puzzling over their incompetence I was taking a look at the data and I realized that they had picked up the entire parking garage where we were we were taking this reference and including the stairwell that was like hundreds of feet away I'm like I can see the stairs and everything and so we chased those guys away because they were not fitting the bill for what we needed but it really planted a seed in our heads for when we started our reference gathering process on on Prague so when we sat down and we had a little breathing room after we ship for we're taking a look at the state-of-the-art in surveying and laser scanning and lidar and all that technology around there and we were pleased to find that there was a

number of financially accessible options to us and that there are techniques to scan like miles of area was subset a meter accuracy and and that was really surprising to us because like you know we didn't subscribe to any like surveying magazines or like go to like oil drilling conventions where they use this a lot and so so it was cool to see how much that's a that's evolved I'm gonna talk in more detail about it so you guys can hopefully be equally as impressed about what we can do so it's really fun to say laser scanning we should do laser scanning for our tracks but in practice it's really difficult to go from that phrase to try to make it accessible and usable by the artists back at our studio so I'm gonna go into detail about each of those okay so this is us chillin in Prague and we are in the old town square and this is what the laser scanner looks like that we used so nobody's around because we were scared

of carrying a laser kind of device around Eastern European police and so we would get up at 2:00 to 4:00 a.m. and go out and take pictures of like big landmarks like this with nobody around and it really helped our reference gathering process so you set this guy up you set it on for a few scans you choose some key locations and at the end you stitch it together and you get something like this and so this is a screenshot of some pricey serving software that is out there for processing point clouds so this is a pricey enough and like they use like opposite axes and like their navigation is completely opposite of what our artists won to do and so for that reason we couldn't buy this software and place it on all our artists desks and we wanted to integrate it into their toolset so one of the biggest hurdles we had was to try and figure out how our artists could use this in actual practice of to help their production process so luckily around this time when we are poking around

we discovered that Autodesk was pursuing some opportunities in the area of point cloud capture and processing and so we actually contracted with their custom Consulting Group and decided to work together on something and so what we ended up with was a special implementation that would allow us to have a plugin that would load up the laser scan point clouds inside a 3d studio max which is what we use in our pipeline and let the artists actually take chunks of this and just load it up directly in their tool and you can vertex snap to it so you can real quickly layout like a block of a building and just see exactly what it's looking like and so thanks to the deep pockets of my studio Autodesk has released this funded project into the wild with a free extension pack to 3ds max 2014 and it's also built into max 2015 which was announced yesterday here so this is equally as accessible to you guys now so we also worked with Autodesk to get some early versions of some software

that they've now released into the wild as well called reality capture or recap and so using that software you can take the these point clouds that we load up in 3ds max and kind of slice them and dice them however you want create chunks out of it delete some points just block it out to however you want and then end up with discrete chunks that you can bring into 3d studio max so after we process this like after the registration process in the laser scanning surveying software after we slice it up using recap we can bring it into 3d studio max and I'm going to play a quick video here to show you what it looks like to our artists so here's some chunks from that scan from the Prague Old Town Square I'm gonna play it so we can chop this up and now we can take some artistic liberties so that we can make the Old Town Square funner to cruise a supercar through so we try and make it a funner to drive Old Town Square so we're taking chunks of the buildings moving it so you can see how this might apply to a fictional kind

of area if you take scans of like cliffs or terrain or something you can actually piece together a compelling environment and see it in pretty realistic 3d and it's all fast and interactive and so real quickly here we can block out what a potential what a potential design for this area might look like after we've taken our artistic liberties so here's another view of the chunks pieced together in the end after we did a little rearrangement this is blocking out the entrance to the Old Town Square for our Prague circuit that actually ended up shipping in the game so once we arranged this inside a 3d studio max and again this is all like full detail like you can zoom in to the detail of the windows and get the exact ordinate decoration that was on it and should you choose to you can model it as well so after that it's a matter of turning the production crank we have a lot of environment artists that turn 10 they eventually build the set of meshes and it ultimately looks like this in our game with lighting AO debug mode turned

on so that's what we ended up actually getting into with our ship game all right so now a quick few bullet points to wrap up this tour of our reference bits so savings we found that going through this process I actually had to write all this business justification and budgeting projections and all that to say like hey we want to spend like tens of thousands of dollars on laser scanning and it's going to improve our processes so I kind of had to prove that out before we spent the money we found that in the end we've significantly decreased the time it takes to create those early rough phases of our production using some of these techniques here we ended up saving I don't have the exact number here so just summarize it as we've saved hundreds of dollars of money during these phases balancing this reference gathering with the time that it took artists to sit

around and be like well I don't know I'm just gonna make this random building here and then having people look at it and comparing it to images and say like well it's not actually there it's over here and all the adjustments that that takes so it's been a huge dollar savings for us we also got a faster initial iteration on the look and feel of our environments so you could see in the scans that we had even inside 3d studio max you could really get a feel for the environment like way before we ever generate models and get anything really useful out of it so it actually resulted in less design changes after after we really wanted to have things baked down and then ultimately this led to a lot less quality and plausibility bugs during when we were trying to shut down the track and for for our QA team and our test team to identify like things that are wrong like we have a whole team of people that sit there and look at stuff after the fact you know like this sucks this is wrong and like we got rid of a lot of that follow-up action all

right so I wanted to have a quick slide on the lessons and application in case you choose to venture down something like this so investing in this was not easier cheap we actually spent some significant spend on this and it was really tough to justify a lot of it came down to somewhat hand waving because like you can't really say well you know what amount of time is an artist like sitting there twiddling their thumbs before it gets to a place where they can actually start producing content and it's not something the studio management really wants to hear and so so it was tough to get over that initial hurdle of saying like we want to actually invest in reference gathering because it's so unsexy and it's not like well you know they're not building the track at that time so why is that important so the cost actually ended up being pretty reasonable for what we get out of it and our production savings here ultimately proved to be really worthwhile we had a

big accessibility struggle for artists I'll touch on that a little bit more in my next slide but it was really tough for us to figure out like well here's a cool tech and here's how we map it to something that is useful actually in production and then ultimately we end up with a large much larger data footprint than we ever had for any reference gathering trips before so where is this going what's the future I know we're in the early stages of like all these like really cool techniques for trying to really pick and quickly pick up what reference actually looks like so I have a slide here showing that the future is crap and and so in no particular order except for it is an anagram for crap here's where some of the future I see of reference gathering is coming from so costs are going to come down like we see day-to-day and like everything that we do the technology that is seemingly inaccessible at the beginning becomes cheaper and so like we are actually

surprised we were like oh like what if we bought our own laser scanner so we could carry it around and like it was actually a thing we're like we were like well I could go either way we could actually partner with a service or or actually do something like that so I see costs continue in a drop and so that's gonna be really good for us trying to get a good start on our 3d environments recognition of data features so we end up with a lot of point clouds and we can process it to make like normal maps and texture maps and things like that but ultimately what's going to be really handy is if we could take that and we could press a button and it would divide up into all these pieces like here's building here's a tree and so the recognized recognition of that data there's a lot of people in academia and research who are working on that kind of problem and so that would really help us as well if we didn't actually have to get down to trying to model things after like a template a point cloud data or something else that is just like slightly out of reach for the artists accessibility made

something I wanted to point out as well so some of this hardware and software is not really accessible I talked a little before about the surveying software like it was impossible to use and like I get I still get confused now from my post-traumatic stress of using that software because like I don't know which mouse button tumbles and which one goes up and down the hardware a lot of it like the ladybug camera uses firewire and like you're hard-pressed to find a laptop that has that built in there's all these wires standing around and so especially when you're in a foreign land like walking around prague and you have a backpack with i could all these wires sticking out and everything like you look like you're up to some sketchy business and so i see it becoming a lot more a lot more easy to use ultimately portability is the next big thing i look at the reference gathering that we did for four to five i'm like well how come gopro doesn't build something sweet like this where I can just like mounted on a helmet and walk around and look a little

less suspicious and so I think I think you know with all the you know iPhone apps and Windows Store apps and Android kinds of things that we have on our phone like a lot of this 360 capture and you're beginning to see it in some of the released apps that are out there the ability to stitch together a bunch of pictures like do some photogrammetry basics to create point clouds from that and just to capture something I think that's going to become more and more prevalent and the last thing that I wanted to end on is Prometheus so I didn't want to get this a huge lawsuit on me because Microsoft is concerned about my slides getting sued and so so I thought I would quickly paint up the scene I'm talking about from Prometheus so so in prometheus there's the scene where these guys land on this alien planet and they were like well who knows what's in these caves right so they unbox this thing and they have these laser flying drones that they take and they toss up in the in the middle of the cave and they fly down the cave and like real time they're getting this 3d model

built and so like I was watching that movie I'm like what where is that like you know that's ultimate and if you think about it we aren't that far from that kind of future because we're at a place where you know Amazon is talking about flying drones to deliver some of the stuff and you see that in all the quad copters and all the stuff that you see floating around and then laser scanning as I mentioned is pretty accessible and there's a lot of stuff coming so I see Prometheus future as a way like we can quickly go to a place even our halo team visited Iceland and did all this other stuff to try and capture a completely sci-fi fake environment and they can go out and take reference like this and using some of the tools that I showed earlier be able to snip it up into pieces stitch it together and real quickly build up a 3d model so this is my commemorative GDC 2014 painting and and hopefully you'll have seen from this talk that we heard that distant from this kind of vision and hopefully we can all progress collectively and move forward in terms of our reference

gathering so that that becomes not a piece that is like special to turn 10 or a studio because it's not a piece that anyone really wants to own it's a piece that is fundamental in us building our 3d worlds and so that brings me to the end and I will be glad to take any questions [Applause] there's a guy moseying up so um I was wondering if you looked into autumn automating low poly generation from your point cloud data to get quick prototypes up and running in a second to that projecting color data onto that so that you can take your data really quickly into the engine yeah that's definitely one of the pieces that were were interested in and pursuing so there is some stuff that is out there our gauge of the state of industry is it's not completely there yet so there's pieces

there's a lot of point cloud stitching kinds of things and again they're not really that accessible like you can poke around and like you'll see ten types of tools that are out there that create low low res meshes or high res meshes from from an area but you have to take it through all these different tools and then we've actually been looking at a tool recently that has powerline the extraction from a point cloud so you can import a point cloud and then it'll be like well here's the power line and it kind of builds it together for you we're like well that's not very useful for us and we don't want to pay like ten thousand for that privilege but like that's one of the things I was mentioning about accessibility I think there's gonna be more and more tools like Autodesk is going in interesting directions with their point cloud data and I really want to get to a point where we can be like well this is a tree so make a low res tree so I can just block it into the game we haven't seen anything that is exactly there but if anyone has a product that's like that like please come talk to me afterwards yep hey I might have just missed that

did you guys use this one in particular is a laser scanner from a company called Leica le ICA there's a ton of alternatives out there so I don't want to be a pitchman for them but it did that you have accuracy and kind of the time constraints that we were interested in the ladybug camera and the laser scanning completely replace your need for photos and if not did you have any way to link those photos back to those videos and skins alright yeah good question no it did not completely replace our need for photos so there's a specific need for photos which is to gather text for reference so once we take some of this we might want to have like here's a Prague ich brick texture here's some cobblestone that was on there and that's what was really important for us to capture and to be able to use as texture of reference and cut it up how we want and so like the laser scanning and the ladybug camera was pretty low res and what we showed like it's actually HD but

by the time you split it out across all those cameras like you end up getting something a little muddier than we prefer and so it doesn't completely replace it so yes we still have the need for photos and then your second part of your question was how do you relate that back to the 3d source so there's a lot of stuff like on my phone when I take photos it kind of comes with a GPS tag and I can actually look at my photos on a map and see kind of where I've been so we are real excited at that technology and we're like what if we took these and with like a GPS bit we could stitch it back together on our laser scan and like have this awesome like big data it ends up not being very usable on that kind of vision so we've tried a bunch of hardware where you have a little card that slides into your camera and it takes GPS readings and stamps it on as metadata into your pictures so the thing that you don't really tell you about that is it takes a while to acquire that signal and so like if you're standing in the middle of Australia trying to capture Bathurst like you could be sitting there for five to ten minutes

waiting to take a picture because you're waiting to acquire accuracy if you take like you're you know handy phone and do that it gets rid of the accuracy so I was I did that for for some of the photos that we were taking and then like you know some random photos will end up being in the middle of Indianapolis and you're like well it doesn't have the real accuracy that we need and so I think there's some some of the new canons and Nikon's have some GPS built in and so again it's something that it wasn't there for us at the beginning of four to five it continues to evolve so we'll look into it but yes the dream is to hey let's go on a reference trip it's easy to do and you back and you have exact photo reference laser scan reference video and it's all like stitched together in one intuitive interface some notable tracks from the history of Forza Maple Valley my favorite are gone now and the thing is is that Maple Valley has no real world

location it's based off of is it missing now because your production pipeline has has gone this route where you have to have a real world analog to get the source material or is it missing for other reasons it is missing for other reasons there are quite a few tracks that we plan to resurrect and to build up to the quality quality level that it would take as you guys know in game development you can't always hit the desired targets of everything that you would want to possibly have in a game and so when we're at a place where we can say well here's a our content looks like we're going to be able to build 15 tracks these are the tracks for the design considerations and the the thought was like you know one of the things behind porting something like that and bringing it up to the current spec would take too long for the time that we had and so I think you'll see and we've already started the process of kind of resurrecting some of the other tracks and providing it as free content as time goes on so you'll see us slowly bring stuff back up to spec and for

something like Maple Valley it is based on a real world location and we sent people there to kind of gather some of the reference and so we would want to send people and capture using the methods that we had and piece together something kind of on a more creative license formula more than we did on Prague and so there'd be a lot more creativity and design later laid out for it but we would use some of the similar techniques for sure yes when you take reference pictures in a city for example and it's sunny then you get plenty of spots in your photos with shadows and then your pictures become more or less and usable because because of that you can use it as an albedo texture and so was it a problem for you and if so how did you work around it well so Microsoft has a lot of money so we actually seed the clouds before we go on a reference trip and there's no Sun when we go so for for sunny sunny days

is obviously a consideration there's actually a lot some software out there now that does a pretty good job at removing lighting and so I guess I won't go into specifics because I don't want to be a pitchman for any specific thing but there's software out there that we have used and that we continue to look at which will let you take sunny photographs and identify an area of shadow and Sun and it'll automatically and it does a pretty reasonable job and subtracting the sunny component of it and so we did that for the most part but then but then that's only scaleable to a little bit and sometimes there's some undesired artifacts and so we will actually take it in and then use some of the MEC mask chart data to kind of dial it into an area that's more realistic and so that was one of the reasons it was really important for us to capture images with Macbeth chart because if it's sunny on the object that you're taking a picture of it's sunny on the chart and you can kind of compensate for that even though there's gonna be some slight differences thank you very much

last question I guess yeah I was wondering if you could just briefly describe the pipeline for the laser scanner basically like the data get once you do the laser scan you know how you get that data basically into Max okay yeah yeah so I covered it a little in a in a previous slide which I'll try and walk back to the wall I'm here but basically the process for it is we take to take all those scans and then we so this is the slide I had up on it so we do the process of registration which is you need to stitch it up into one single coordinate system so you know like even though you take like 50 scans because stuff was obstructed you know exactly where the top of the cathedral is from like you know this point that you were trying to take a look at and so you take all that and surveying software lets you kind of stitch that together and so it's all registered then we can take each of those pieces and we know it's correct in 3d space and then we can start adjusting it and so we use the recap software from

Autodesk to export out RCS files and that's freely available for for people to use and so you can break it out into RCS format and then as I mentioned before Autodesk has now released in 2014 max 2014 and 2015 this plug-in that we paid them to make and it lets you load in those RCS files as a max object and so you can bring it in and you have a ton of clouds and it does occlusion really well and decimation so you can crank it down to like 1% of the points if speed is a concern move stuff around and so we have a really kind of smoothed out pipeline from when we first initially started doing this so our artists can kind of take any bits that they want loaded into their Mac scene and start building a model off it and there's a couple hops in there but it's as good as it's gonna get for now yep all right well that's over I'm feel free to catch me anywhere after the show

after the show or after the session and thanks for coming again [Applause]