The Future of Art Production in Games

all righty uh looks like it's time so uh we're going to Jump Right In cuz we have a lot of ground to cover but I know there's some still some people coming in so uh looks like the session is pretty full so if you if some of you guys could Scoot a little bit closer to the center so there's more room for other people that be grade and uh without further Ado uh let's jump right in our our subject today is a future of art production and uh I actually want to kick it off with a story that I really like probably my most favorite story uh that I shamelessly stole from a speech by David Foster Wallace and uh it goes like this uh two younger fish are you know swimming at the bottom of the ocean mining their own business as an older fish swims by and the older fish goes hey boys how's the water and then just keeps on going and a couple of moments later you know one of the younger fish asks his buddy hey man what's water and while not that I'm using I think this anecdote very well describes the condition that all of us are very prone to where we got where we get so used to

the circumstance around us that we stop critically analyzing it dayto day we stop applying critical thinking and especially being a person involved in in pipeline development and Tool production I find it to be crucial to constantly be asking what is water what are the things that we're not consciously thinking right now that impact us dramatically that need improving that could make things a lot better and so this is something that we're kind of try going to try and do today with art production we're going to try and see what are the things we're missing what things are we not thinking about and we're going to do it through the prism of technology and this talk is obviously about technology right uh we're going to be talking about the latest trends in uh in all the pipeline and art Tech and sort of even Tech outside of the games industry is going to be shuffling its way in into art production pipelines uh but at the end of the day I think the most important part here is that uh technology were will force us to consider are more because if you look at

all these pipelines they're inherently reductive to the amount of work we're going to be giving things up there are going to be things that we're going to be find that we're going to have to find minimally contributing list contributing something that we'll have to let go off and uh in it on its own it's fascinating because technology is going to force upon us that conversation about what is art because when you look at it the other way around the things that we're not going to be able to let go of the things that are going to be so inherently human and artistic that they're always going to be parts of pipelines are going to essentially be art and uh to tell you the truth this is probably the scariest talk that I'll ever have to do because this is no joke right it affects all of our livelihoods it's you know years and Decades of skills that we Acquired and 10 years 20 years down the line some of them might be gone and uh the biggest reason for doing this talk for me is I feel that everyone has the right to be informed on that question everyone has the right to be be part of that conversation because

this is going to be a debate that we're going to be having for the next 10 or 20 years and that debate is not going to happen here or any other conference is going to be each and every one of you going back to your studio and having conversations about how do you want to work how do you want to create art and uh and basically hopefully giving you some language and some context as to how to do that uh will help all of us end up in a situation where we're only improving and uh so without further Ado let's Jump Right In into the features and these are kind of the four major things we're going to talk about and uh they are uh automating optimization uh capture in reality uh parameterization simulation and generation are all lumped together at one point and we'll talk about why and then finally we're going to have uh deep learning and artificial intelligence and how that is going to affect art production and so let's start with a simple one uh optimization automation that is probably a no-brainer that we're doing that already and we're going to keep doing that

but something very interesting to the subject of uh what's water I don't know how many of you folks got a chance to work during the NES days but this is the original color palette from NES and back in those times color was a technical resource it was something that you managed because you couldn't afford to display all the colors at the same time so artists carefully picked their pallets and made compromises and that was a very big optimization point but how many of you think about this today probably none of us because it's no longer an issue technology has evolved it has caught up so we don't have to worry about that but that inherently begs a question what other things in our pipelines today that we don't even think about that we spend so much time on that are going to go away and uh there's obviously a lot of examples right anything from uh low poly or high poly to low poly workflows to level of detail meshes to separate Collision meshes Shadow casters ik meshes gaml features uh even unwrapping UVS manually all of that is basically a compromise that we have to make for the sake of technology

that is still imperfect that is still developing that is catching up but inevitably we're going to reach a point where that doesn't matter anymore where we're going to be able to afford to display anything we want to display and then from the other hand we're tackling that problem on the software side where we're getting better and better at automating those things we're automating level of detail meshes and improving UV on wraps and so the more that's going to happen the more chunks of our pipeline are going to sort of fall off and become obsolete uh which is great and uh this point is easy right because we we can actually uh very easily and safely say that these processes do not have any artistic values so they're going to go because we can we want to allow ourselves more time to focus on the things that are important but that once again inherently Dex the question what is important and so now we're going to jump into something that is a little bit more controversial or probably more controversial and this is a capture in reality and uh I actually wanted to start with this slide which uh seems

very far removed from the subject but actually I mean obviously Prince of Persia a lot of us probably as kids or maybe not played the crap out of it got inspired wanted to make some games and uh you know the fluidity of the animation there made the pr so lifelike that you couldn't couldn't help but be blown away but every time I look at this now I can't help to think back to Jordan mechner's talk in 2011 at GDC where he was talking about how he produced Prince of Persia and the way this worked back then in 1986 is uh he and his brother went outside and he shot his brother running around in a parking lot and then he actually rotoscoped it and basically pixel by pixel painted it in into the game so when we're talking about all of these new and cominging scanning techniques and uh sort of capture reality and animation it's not a New Concept we we can't treat them like they just appeared and they're going to sort of circumvent what the art process was because they were there all along and and we were doing them 30 years ago and we're going to keep doing them because at the end of the day Jordan's goal was to make a game and that got him where he

needed to go with the resources he had and with the sort of allowing him the opportunity to retain that artistic vision and so we're only going to keep going with that and same thing for you know facial scanning uh first time playing Max Spain I remember being so blown away by the quality of his space I was like this cannot absolutely get better this is photo real because it was a photo on like a box basically uh but this only keeps on going and uh there's no sign of stopping and there's more and more of that coming around obviously we're doing that for environments and by we I mean predominantly dice at this point and hope hopefully you guys caught their talk last year I think it was and uh it definitely proves that it's already doable and uh technically there's hardly anything in the world that we can't scan today uh we're solving these problems one by one where we can do characters we can do organic environments there's obviously concern about you know oh heart surface very nudely kind of uh man-made objects we still can do those this exists today

right and with just with enough pictures that you take of an object you can get the levels of reality and sort of sophistication and detail and history and wear and tear that would take you stupidly long time to do by hand and so we are solving most of the problem that exist and a lot of them are solved the other one being reflective surfaces which is obviously a big concern is something that we have a hard time scanning but you can already do that there's either you're using a a cross-polarized dark room setup or you're using like a matte spray uh that already starting to appear on the market actual sprays for photogrametry that you can just use as coding for an object and then uh apply that so most of these problems are already solved and they're only going to keep being Incorporated more and more right you can scan entire environments they're going to be 100% photo real and your only limitation is memory and as we all know that is only going to grow so at some point uh once again we will be able to leverage that much more extensively than we do today but obviously there's still a lot of question right photogrametry is clunky to Implement into a regular development

pipeline where when you're augen in made man-made art with photogrametry and then also just you know flying places and getting your entire team and having them go somewhere is an expensive process and not everyone can afford it and uh you know we've made games just using Google so um sure a lot of people are are used and accustomed to that and I think by the time photogram reaches Mass penetration we're going to we're going to overcome most of these issues but I think fundamentally what's happening is uh we got used to producing everything from scratch because there wasn't anything at the world in the world at that level of abstraction where we could scan it and put it into a game because the word did not afford US low poly models that could be transported into the game but eventually and this might not even be games light right that might be Google Earth coming in and just scanning the crap out of everything and then we're going to have accessible environments from across the world at a resolution that is at a highest level of fidelity that we can just borrow and then eventually we'll basically transition into the same thing that film is doing where you treat your

environment as sets and you and you light them and you dress them and you modify them but you can still communicate art right the art is not going anywhere if you look at Blade Runner one one of the most iconic and stylized films ever it heavily leaned on the real life Los Angeles and the Bradbury Building is still around and it's there downtown LA you can go there and uh that is going to be a major part of our pipelines and most importantly it's going to be very cheap and that's why it's going to find traction because it's going to be a lot cheaper to just take an existent scan of an entire environment and use it rather than produce everything from scratch uh I'm actually going to skip this one uh but you can download it online later it's just more scam stuff but now I want to talk about uh parametrization simulation and generation and uh I wanted to start with this and uh this is an example of a visualized L system which was developed by a biologist and a botanist by the name of Lynden Meyer where he basically described a plant cell growth using

something very close to an algorithm that basically used uh types of an element of a plant as a letter that he would organize into words that would describe the growth of a plant and so this particular system is inherently organic it comes from botany but it lends itself very well to programming to basically being computer simulated and there are a lot of things like that in the world that we can definitely use and sort of apply in our current day technology and similarly right epic forther K demo had like a simulation of growing in a forest and that was 2 years ago on Unreal Engine this is basically old news but you can simulate natural processes to get art or to get the content that you need and later today we obviously going to have the privilege of checking out a talk about Horizon that leaned heavily on a lot of these procedural population technique and Guerilla being the great example of a studio that embraced that approach and sort of how that approach is going to shape our future moving forward uh same

thing with our tools right uh if you look at speed tree we're moving away from working with individual vertices or polygons we're actually working with trees now we want to treat treat virtual objects as if they were real objects we want to you know control the height and the density rather than the polygons and the faces and the same thing with the with substance painter because you no longer work with pixels per se and texture as you're applying materials your breast Strokes are abstracted to a particle simulation that will actually populate an entire sort of system of of patterns that you want on an object and this is only going to continue uh same thing with the sort of physicality and simulation because once again we're treating objects as if they are virtual but further down the line anything in our production pipeline we're going to want to treat as if it's something in the real world if you're populating objects they have physics they interact with each other they lean on each other and uh that will allow us first of all to create more believable worlds but also so save some time doing

it right because you don't want to sculpt every single fold on a curtain when you can actually just have a physics simulation of it so uh another example of that is obviously characters and there's nothing today that is inconceivable about a system where we go in and we scan a whole lot of people and we use their facial features as sort of the extremes for different for different representation of facial features and then we automate that and blend between that so I wouldn't be surprised if years from now most of say Indie Studios that cannot afford the talent is going to be that much more expensive at that point to get generate characters are going to be using an automated system like that that might be argumented with some uh deep learning or you know AI technology but at the end of the day we're very close to today right this is from Black Desert I think and it's a couple of years old probably video and uh we can do nice believable characters in a modularized procedural way and uh I think the bigger takeaway and this is why they're all lumped together is we're

seeing this abstraction of interfaces and abstraction is a programming term that basically means creating a higher level input into a lower level system so say you have ones and zeros that your processor operates on but you have programming language commands that you actually issue to execute something and so this is an abstraction layer and we're going to keep building those up to a point where the knowledge of computers is not necessarily necessary because you can interact with virtual objects as you would interact with the Real World objects and so that rection is going to keep going it's not only inherent to Art if you look at programming right unreal blueprint allow people to make entire games without having a programmer on the team meaning we're building up that functionality to go higher and higher level trying to get people to express their intent to uh do what they want to do faster without less sort of layers of execution and uh the final point for for this particular section is uh obviously AI stuff and uh we're actually going to start uh a little bit outside in

consumer Tech this is a promotional picture from Google's new Communicator aloe and the way it works is it actually has a neuronet on board that is going to track your answers and it's going to remember how you reply to questions and it's going to analyze the questions post to you and is going to intelligently propose answers and this is not unheard of right Chad Bots for most of the companies that you interact online to rely on these deep neural networks that are taught to communicate with humans and we're only going to see more of that where neural net is going to analyze say your entire game that you made previously and all the art choices you've make and it's going to offer you solutions that you can then pick or modify or interact with but once again this is uh not as far off so uh Google did a thing with their uh deep learning neuronet that they used for image recognition and they basically flipped it around right when you search for an image in Google uh it would actually scan an image and spit out words that you would then match with your search pattern but when they flipped that

system around and they put words into it it was actually able to Output images that look like this which inherently proves that A system that is capable of that kind of analysis is also capable of some form of creativity albe it driven by the information that is fed to it by humans but all of this is not as far off as it seems so uh there was a deep learning panel uh at sigar this year and folks from Nvidia were talking about how remedy used the a neuronet based uh animation solver where they just basically taught a neuronet to convert a raw video of actors talking into almost game ready animation and in general sort of deep learning becomes bigger and bigger part of animation solvers today so it's it might sound like an episode of Black Mirror but it's actually closer than we all think and uh I wouldn't be surprised if you know 20 years from now everyone has a a deep Learning Net sort of as a as a partner that actually learns how we do things and then allows us to once again deliver what we want to

do by offering us suggestions as as of what it thinks we're going to do rather than for us having to do the whole thing by hand and once again this very well feeds into this abstraction layer Theory right because essentially when say Google Alo sends a message for you you're still communicating what you want to say it's just that you don't have to type every individual letter so we're going to that higher level of communicating intent without the manual labor of typing in letters and uh same panel of C graph uh cross bite Labs we're talking about how they're looking into neural nets for the stuff that they're doing and also at steam Dev days uh last year during the keynote team Sweeny was talking about games industry being way behind on deep learning compared to a lot of other Industries so that kind of leads you to belief that there must be someone that's epic looking into this right now uh but now here's the million-dollar question so what is left when we you know scan the crap out of the entire world when we have neural Nets helping us populate in everything generated and scanned where where do people come in

what's left of the artistic process and that that is the most scary question and we're going to try and unpack this as best as we can but basically what's happening right now is we have a creative problem we come up with a solution right say this environment looks boring I want to add a puddle that was the creative problem and a solution but then the execution of it is I'm going to go to a Shader add a feature make sure there has a vertic blend on it go cut in some verts into the environment paint in the verts build the level look at it in game tweak the Shader bake the lighting and then hopefully it's done so that chunk is execution and it's by far the biggest part of any production and so what all of these things are doing uh all of the technology that we're talking about they're cutting down on execution time they're going to get you from your idea from your solution from your intent to the final result as fast as possible hopefully instantaneously at some point and uh this basically means that the core skills that we rely on as artists are still as valuable and as needed as they're ever going to be but a lot of

other skills that surround feeding a still imperfect Hardware or software might go redundant and so the interesting thing that comes out of all of this is that intent is basically going to be our basic building block of any art production but also of any interface if you're working on tools they'll have to be refactored into a manner where we don't treat content as something that pertains to a computer we have to content as something that's from The Real World and we'll want to inter interact with it based on intent just like we would in the real world and uh once again that very well feeds into the artistic process that we have today because we're all masters of intent right we don't just go in and do something for no reason if this plant has to have more angles and be sharp it's because this environment has to be scary or for boating we're trying to communicate something about it and then all of the choices we make during production hopefully all rely on some kind of artistic intent or meaning to back it up and so if we were to look at

a process as to how it might look you know years down the line is uh you would have your high level intent or Global intent where you would generate a an entire environment space and then you have a regional intent where you go oh I want plant on the corers you place a region you specify a biome and they just grow and then you can go in and be okay I want to actually added individual prototypes and I actually want to modify them and add a particular color or something but that will have to be the workflow and what's interesting is uh this is kind of the way movies do it when you think about it I don't think anyone's going to say that movies are not artistic and this is a great example from Rebel Without a Cause And this shot is without a doubt art right but it's art because the director and the Director of Photography infused it with meaning uh because the character of James Dean is is you know smack dab in the middle of the sharp corner that's dividing the scream into two halves and the right side is the policemen and sort of uh men and kind of the stern world uh of men that jam Dean's character is getting into and on the other side it's

it's women it's his love interest it's sort of more kindness and and the choice for example of the of the red code is not accidental it attracts our attention so the art part of this is not the fact that someone put a mug in the corner of that room or you know build that desk it's the fact that it was heavily infused with meaning that communicated something and that part was Art so in essence deriving from this you can say that creating art is essentially a process of assigning meaning and in that case no matter how much you optimize the process the art is not going to go away because you're still going to have to think about what are you communicating to your audience if your art does not have a meaning to back it up then it will not be as successful because there's nothing for people to resonate with and so this part is once again not going anywhere and so there's obviously the bigger question right how's it going to impact us as an industry and uh Cinema is once again a great example of that because they went through it through this they went went through the process of democratizing

their Productions they used to be way more timec consuming laborious required a lot more money but then with this great abstraction came great democratization because everyone had a pocket camera that they could shoot a movie with and uh that brought a lot of different things and we're going to try and see how that applies to the games industry or at least speculate and do our best so obviously Productions are going to get cheaper right this is understandable uh when you don't have to model all of the art but you can just import real locations and dress up the bulk of the cost of any production goes away and the same for animation you know 20 years from now you're going to have a 4D phone that captures you know holographic people you can just drop into the game immediately uh and same thing for smaller teams don't get scared just yet we'll circle around to that point but yes you will need smaller teams because you can do more with less people and uh that means that there's going to be a sort of a regeneration where we used to specialize a lot but then at some point it's not going to make sense anymore at some point all this is going to be automated we're going to be more generalized again or we're going to execute more Artistry

rather than technical communication uh production times uh so I don't believe that the production times change much in the movie industry because a good story or good gameplay takes time to mature it's not something that if you can produce an entire world in 3 days it's it doesn't mean that you're going to have a good game because good good story takes iteration you're going to have to work with your audience you're going to have to play test you're going to have to figure out what makes it great and that part just takes time there's nothing technology can do to speed excuse me speed it up uh another fun uh outcome is that photoal is going to become the new indie because it's going to be that much cheaper right now it's diametrically opposed right independent Studios do more stylized games because they cannot afford to compete with super expensive uh Productions that more AAA Studios do that are very realistic and detailed and that way they also get to stand out on the market but eventually when you know the cost of everything scan is going to become Dirt Cheap everything's going to be very realistic and everything's going to look good provided you you know you can light it and give it enough context but that

being said if I were a big AAA publisher or studio thinking of a franchise is going to last for 20 years I would probably consider something slightly less real uh and final one uh I do not necessarily believe that a lot of jobs are going to get lost same way in the movie industry when you know that democratization happened it didn't shrink right there was actually more value being generated there were more movies made because it's it's that much simpler now that many more people are coming in to do it and all of them need talent all of them need people to support it all of them need artists all of them need you know lighters and directors of Photography and what be it and I I'm convinced that that is what's going to happen in games because right now it's a very technical process it's a high barrier to entry but eventually it's not going to be and then everyone will get a chance to express themselves and uh that's just going to mean that we're creating that much more value and that many more games but it's just that on every individual project you're going to have less people so if if you take one thing from from this talk is

hopefully that this this is how I think it's going to be so take some comfort in that but then for the for the final point is uh every time you go from this big huge gigantic production that has a lot of people to just a handful of people being able to produce something of the same quality you know for for every Captain America you get a a Goodwill Hunting and this to me is is fascinating because every time I go to GDC you know I get to I get to meet all of you I get to talk to you and see this sort of cont contagious passion for art and desire to do things and uh you know the people that are going to be making the next Goodwill Hunting Games are going to be you and uh if you know I I used to model and I still do for for 15 years now and I come home after a long day at work and I do some modeling text room till 3:00 a.m. and more and more these days I'm real realizing that I might not get to do this for much longer but if this means that that many people get to have a voice if that means that

that many people get to tell their stories if that many people get to go back to what inspired them to make games in the first place cuz I don't think any one of you saw someone do a UV on repap and went holy that's what I what I want to do for the rest of my life you know we're all inspired by stories that profoundly changed us made us want to be better we're all inspired by Virtual Worlds that made us want to go there live there and then and share them and this is what I think technology will allow us to go back to and yes we're going to lose some things and you know if if that kind of future means that I don't get to model or Texture anymore then I guess I'm I'm fine with that and to be honest I think we are the species are going to be better off that way so with all that said to tell you the truth I I can't wait thank you so [Applause] much uh if you have any questions I think we have a minute but you can always just

find me after the [Applause] presentation hey man one question one question over here yes yes yes so um I'm with you I can't wait I think ultimately what we I wanted to say from your first story it's uh what is water is say why is water I think that will be the question we have to ask that's a good question when we are when we don't have have all the worlds ready at at our fingertips is like what do we want to tell in that in that world so that's what I hope you know that's what I get out of your talk I'm inspired by it because I think ultimately we just we're going to be able to be more empowered to tell stories we won't have that barrier to to tell those great stories so thank you for the presentation thank you I totally agree [Applause] cheers