Super Powering Technical Artists Using Deep Learning

[Music] hello everybody thanks for coming to this presentation my name is cesar romero and i'm a machine learning engineer but today i'm mostly talking to artists and my goal with this presentation is hopefully to maybe inspire you a little bit maybe expose you to some tools that are becoming available some techniques that have made a lot of progress uh in the recent years and hopefully you can come up with even better ideas of things that you might create using some of these tools i think it's just a matter of time before some of these tools become ready for production um in some of the examples that i'm going to go over i'll point you to tools in some cases open source in cases our websites you can just run them directly some of the cases don't have tools yet i i still believe it's just a matter of time so let's get started the first example i'm going to go over

is uh style transfer if my slides okay so just click who's familiar with style transfer okay that's more than half so that's good you probably have seen many many of these examples first papers published around 2016. [Music] google even has some examples in the keynote earlier this week the the case is basically that you have some content and you want to see what that content would look like if you paint over it in the case of a game you might be exploring whether a specific scenario or like setting environment how might it feel like how do you get the right mood or the right style does it work or doesn't it work um painting over it of course takes some time but um you could do just style transfer and get something like this and matter of clicks so remember i said i'm machining engineer which means i'm not an artist even though this is not particularly mind-blowing i would not be able to

grade this myself and i would be able to create it running a machinery model but in this case i didn't even have to run that model myself so i intentionally show this there's a couple things i want to highlight it's not production quality right there's some artifacts you can you can tell um but here we're just applying styronite from van gaal which you may have seen similar examples on san francisco which is probably familiar to you at the bottom right you see watermark it's just deep art dario so the author's original paper created a site where you can upload your own image your own style select from existing styles apply it all you need is a browser um and a couple of minutes it took me i don't know a few clicks and maybe three minutes to create this example um and if you want higher resolution you also have that option uh you can pay to make your images private et cetera so it's you know this is one of the examples of the cheap tools that are available to check it out and of course you might be interested in applying this to

um two videos so let's take a quick look we have a simple video of a beach you have waves going from left to right someone's swimming on the left side um and you might also be interested in seeing what it might look to apply a style so let's get the same style stereo nights from van gogh and apply it and then you get something that looks like this again it doesn't have to necessarily be production ready you're the artist you decide when it's ready but it might help you explore a lot more quickly um in particular this example was created using 10 lines of python code you don't even need to run the model yourself this one was intentionally created using algorithmia which is a sort of a platform for hosting machine learning algorithms um it's really a couple lines of python code and then and then off you go and just wait for a couple minutes um so then both of these examples the the style was applied to the entire image

so how about we want to control a little bit more what this style is applied to using mask or something like this um someone had this idea um alex jc some of you might know him he's not jdc this year i believe but he proposed something called neural doodle and i won't go into the details because the intention here is not to do a deep dive into the architectures and the techniques and loss functions and things like this that are familiar for machining engineers like myself the intent is to show you examples so you you get the idea of what things are possible um and this is just a sample there's a lot more where this came from so in particular the idea of neural rules is to control a little bit more what the style is applied to as opposed to applying it to the entire content um so this is some quick thing that we put together at a hack week like about a year and a half ago the paper was published in 2016 um and the idea is that then i use a shape to decide what the style is applied to

so the style comes from the left side the shape just controls what's in the output image is the style gonna look like um and uh so you could like rearrange essentially even a piece of art uh we're using a single style here for simplicity of the demo um but here you see how it actually works so if you have the style on the top left at the bottom left you just have a guide or a mask that decides where to style to extract the style from then you draw what you want the shape to look like and then the output has the shape and the style that you wanted and then you can redraw it if you don't like that shape you get different different looking trees and then you can decide whatever you want to do with that content in this case it's just generating images but the point is give the artist the control these are the stools that are learning how to interpret images um but you're the driver so

um this was this was just three examples of style transfer and let's i'm gonna give you here a couple of references um so i can move up to the next set of examples yeah so at the top here we have the face paper the first two are the are the papers for for the machine learning folks in the audience that's what you want to read if you haven't already um then deepart.io is a site created by the authors of the first paper where you can upload your own style and your own image right below it uh it's a github repo open source you can grab it that's the one we used to create the that little demo with the potato and so on um i put the license here next to it um in my experience the gaming industry is not super familiar with open source licenses in this particular case what you need to know is that http v3 means the code is totally there you can use it you might not want to release something that uses that code because this license would require that you also open source

circuit um but at least you can just grab it and explore it just like we did for that for that little demo that i just showed you um and at the bottom is the site that i mentioned earlier the algorithm is kind of like a marketplace for machine learning algorithms and they have plenty of demos and simple like sample python code that you can run yourself and that's the one that was used to create the video style transfer um so let's let's move on to another set of examples um i want to see quickly if you can think of what these people have in common and if you if you look well it's not gender or age or the hair or facial expression or clothing or colors of eyes or skin or backgrounds so what is it that they have in common someone i think knows they're not real um they never existed this are completely generated by a model

um which i think it's it's pretty interesting what what you could do if the models are now getting to the point that are generating images that look very real like it's very very hard to tell just from looking at the pixels that this is not real um in particular this is generated by a class of models called gender power several networks the machine learning people in the group probably familiar with it um javier here from ea had a fantastic talk on tuesday afternoon check it out and vault explaining how generate models in general work if you're interested in the math what i want you to think at high level is that many people are familiar with how machine learning models can classify images for example so think of a model that given a face can tell whether or not this is actually a face um and then the insight here of 2013 or so

um was what if i use that information of whether the model believes this is real to automate the process of showing another model whether the image is generating is real or not real so you have now two models that's where the adversarial world where it comes from it's kind of like these models are playing a mini max game or competing against each other what matters to us here is that the generator can get very good at generating the content that a traditional machinery model knows how to tell apart whether it's real or fake um now in that example that i had earlier just faces the generator just learns the space of human faces but for an artist it might not be enough because you want to control what the face looks like how you're going to use it depending on where you're going to use it or um so one technique it's a clever simple from machine learning point of view but it's a clever technique that i

came across uh some six six months ago or so um it brings that idea of trying to provide more control to the human and what are those phases going to be generated for the machine learning people in the audience it's it's a very simple linear model that learns a mapping from the input to the generator into the output of a multi-label classifier that takes the output of the generator if that didn't make sense it doesn't matter i have a short video to show you what the user experience might look like or for something like this so the model knows how to generate faces and what you might get is sliders or a tool like this where you can individually control what each of the attributes of the face you want more beer less beard or maybe blonde hair generator faces at random which the model obviously knows how to do you might want more hair you might want younger or older looking face you might want bangs who knows it's up to you to create it and again for the machine learning people

the audience one very cool thing about this technique is it does not require that the generator is retrained uh anyone that has worked with cans the hardest part is training the generator of course um this doesn't even require the generator's return so you can grab an off-the-shelf pre-trained generator in particular in this case it's a progressive gan by nvidia and and just learn that mapping and then you can build a tool like this which i think is pretty interesting um i'll give you references all together in a couple slides um let's look at another example of content generation this particular one is called image translation the original authors called the mystery mission section in particular i'm talking about the pix model for those machine learning in the audience um the idea is that the generate models actually before learn space and and then with the trick that i mentioned you might get some control on attributes but what about instead of

turning something somewhat random into a face get an image and turn it into a different image that's what they call image translation um i won't go into details of how that works but what i want you to take away from this is that as long as you can think and produce what this input output pairs should look like you can train a model that does this and it's the same architecture and on the side that i'll give you the reference there are multiple examples um so i'm going to show you one of them this is something i created in my browser this is running in real time and again it's not a production tool right it's a tool that might allow you to explore so here i'm just creating facades and doing very it's a very very simple ui where i'm just selecting different colors for blocks i might select doors and i might want to create an entrance or might want to create windows so running 100 in my browser in real time i'm just selecting where i might want a window and whether i want to add some windowsill or a trim

or some shutter like a curtain and and every time i want i can generate a new facade and it starts looking like fsi you're not going to put this directly in the game but it might allow you to create things to prototype quickly and it takes literally seconds to to do something like this um and then you can iterate as much as you want you can generate random examples you can add more windows add a third floor add a balcony and then reprocess and at the end you might end up with something that actually kind of looks like a facade from you know maybe 100 years ago um which might be what you're after and this model was trained with facades but if you can generate these input output pairs um then you can just you know get a machine learning friend then they'll train for you um in this site in particular there's also examples about going from sketches to cats which you may have seen um what matters is that you're translating an image

into another image um i have one more example in terms of content generation which i think is relevant for for gaming um in machine learning we call it super resolution and gaming the closest i mean is kind of like remastering so many people probably have enjoyed remastered games like i don't know the last of us come to mind um spyro more recently um resin evil is kind of more than a remaster but you get the idea like some old game that you want to kind of adapt it to a newer platform better resolution etc so you one of the things that needs to happen is that the materials need to kind of now grow and need to become bigger but a simple by cubic interpolation is very naive and it doesn't work well so if i grab a random texture from the acid tour and i stretch it four times it looks something like this this might be a little bit hard to see from the back so i'm going to try to do back and forth and hopefully you can see it if i stretch this image four times

you see like the pixels you see the artifacts generated by the bicubic interpolation um if instead we use uh in particular this model is called esr gan i came across it last year at some point you can then teach the model what an image looks like this model wasn't particularly trained on textures in the access store this model was just trained on a large collection of random image from the coco data set defined right and then you can ask it to output not the same image but a larger version of the same image that you show as inputs if you do that let's see if you can tell the difference so the second image is the output of the model so i'm going to go back and forth bicubic interpolation esr again by cubic interpolation esr again so and it's not specific to textures so you can apply it to like i don't know some random character um

jeremy also had a talk earlier this week jeremy from unity on a few other examples and the we have even started to see uh examples in the wild of somewhat recently someone grabbed all the textures from max payne and remastered using just this model without any retraining just put them out there i came across a couple examples from diablo 2 um and again this could be a tool where it can help you if you're in the process of remastering a game my understanding from talking to a couple artists is that it's also not necessarily the most exciting thing to do um it might be tedious and if this can help you get that job faster then you can get more creative so a couple of references the first one is the image to image translation that's the one that i use for the facade example the first three are just the papers for

the machine learning people in the audience or whoever really wants to get into the mass the second one is the one used to generate the faces the third one is for super resolution so things that you can look at today even if you don't have a machining background at all this person does not exist.com is where all the spaces were generated uh it took me just a couple minutes um the first github repo that's called transparent latent gan is the one that gives you this fine grain control of the attributes of the content that you're generating the example was used with faces but conceptually it doesn't have to be limited to faces it was basically built on top of progressive gen which is the third repo which was created by nvidia and then esr again so it's the second github repo it's for super resolution and the pix depicts at the bottom with the bsd license is the image translation sorry but the order doesn't exactly match

but i put the license here intentionally remember how before i said gplv3 and one this is mit apache 2 and vst all three of these licenses are open source and very business friendly actually you are completely free to grab this code build with it modify it link it you don't have to pay anyone anything all you have to do is give credit to the creator like mentioned somewhere there's your delivery licenses have a different way suggesting how to do this but you can use this this is not commercial software you can just grab it go look at the code if you want to run it you can do it um so i want to look at a third class of examples which is animation so animation is harder than the other two classes that i've shown you i've seen a couple of good talks ready to animation this year gdc which makes me happy um

so let's let's just look at the first example um let's look at smoke and in general fluids or particle would have something similar some of you may even be familiar with this paper in particular uh this is from siggraph a couple years ago the smoke here has been simulated by neo-never but what does that really mean um today what you typically have is some simulation that has an implementation of some complex physics formula without getting into the details of the math um it somewhat limits the kind of amount of smoke that you have the amount of particles that you can simulate the amount of iterations that you can run the simulation for so someone had the idea that because machinery models can be general purpose approximators and someone understood the simulator enough to identify the piece that was expensive why do we replace that with a machine learning model this is trained completely on synthetic

data which means you have infinite data to train um so then can we train a machine learning model to approximate this function and the end result is that you get smoke that looks like this but it's ten times cheaper than what you might encounter in your engine of choice um these kinds of things i think it's just a matter of time where until this kinds of tools are just in the endings in the tools that you can you can use to create your content um let's look at character animation so um the first example we're going to look like for the machine learning people the audience called pfn uh fabio from ea earlier this week had an excellent talk comparing a couple the next two examples coincidentally then you're holding who i don't think you see in the audience is the author of the first example the the key observation here is that animation is very complicated and can

you simplify the pipeline would be our goal and in particular you can use molecular data and a machinery model that takes as input let's say the context of the character the direction of where it's going and where it was recently and that's enough for the model to decide how to animate the character in a way that adapts to the terrain um there's no kind of handcrafted animation here for exactly where to place the hand or where to like slow down and scale as if there are steps uh the model is reacting to an extent uh to the to the train um and it basically simplifies the pipeline you know i'm speaking in simple terms but danny holden had a talk also on on exactly not this paper but newer work more recent work that they have on ea that even shipped um an assassin's creed odyssey sorry ubisoft um and five year from ea had a talk explaining this

and he showed some examples in fifa and so on um this is not something that is a tool that you can grab today i won't give you an open source repo where this exists but these things are coming and so we take that example and we extend it to quadrupeds with some generalization there's some extra tricks but the machining details we don't need to get into um then you get something that looks like this so this is from siggraph last year and inspired by the pfn which was that yellow solder you saw earlier and so then you get a wall in particular who's just like the soldier it's just reacting to the user input based on the terrain and the speed it just decides how the character should be animated um and it learns like you know jumps and and how to go up or down and turn and stop in a way that looks realistic just from mockup data pretty much

and um i want to give you one more example which uh some of us know a stick mimik this came from berkeley last year and it's also related to animation but in particular it's related to physics based animation and the key observation here is that if the model can learn a behavior but not by stitching mockup data together but by learning what force to apply to the to the rig how to move the muscles if you will um then that allows the model to react to changes in the environment including obstacles or projectiles or things that could be thrown at the model so using either kind of manually animated character or mockup data you can have a model that learns how to imitate that hence

the mimic in the name and then you put this character out there and affect its environment so let's take a look at what that looks like here we have a dragon animated by hand and then this is the one learned by the model so that obviously you can't do mock-up data for a dragon but then you can start throwing things in front of the dragon and the dragon adapts by itself in a way that looks realistic like it has to maybe it looks like it's turning to avoid the boxes uh you can have a quarter bit in this case it's line you could you know it could be a dog or whatever for which you could actually have mockup data and then learn how to run and then react you see how it has to work to get out of the cubes it can even get completely bombarded and fall in a way that actually looks kind of natural and none of that was animated by hand it learned how to apply force to the body so that it would actually do what

it was expected to do um so i'm gonna give you here the references unfortunately these are unfortunately for some of you in the audience these are just papers in this case i don't have code that i can share with you um but i do believe this is a matter of time on tuesday there was a machine learning track where i learned several studios using machine learning for animation um different parts of the pipeline and some of which has shipped in games some of which is shipping games it's very exciting work and i think it's just a matter of time um so with this one of the messages that i want to live with is help us machine learning people help you artists just hopefully from this you get some idea of the things that are going to be composable or that are possible today and get creative on what other tools you

might want what are the things you want to create that would be very difficult to do without tools and there's a lot more where this came from so i think with that we have a few minutes for questions so i'd like to hear from you and learn um how the last part with the animation oh great presentation by the way uh the last part with the animation where you're saying it was learning does that is that something that still takes a lot of um performance to calculate or anything or is that something that's going really close to keyframe animation great question because i should have mentioned and i did not so um this is using reinforcement learning so what it's doing is um applying starts imagine that you start

with random policy that just applies random force to the muscles and it's just going to fail to imitate the reference animation and then it gets a penalty or reward over time it learns to mimic that behavior by applying force to the right places at the right times um the process of training takes a long time um in particular in unity that's one of the reasons we released the ml agencies framework that you may have heard of it is for the purpose of reinforcement learning not for the purpose of this animation but for the purpose of reinforcement learning in general um and it takes time it takes some compute power just like many other machine learning models take so we're talking hours perhaps yeah i don't recall off top of my head how long it took to train each of those but once it's trained then you can embed that in like a mobile device and it can run in you know near real time hello

are there versions of gans or i should say any sort of machine learning approach that does image translation from say something that may be a game that's trying to look float photo realistic into actual photo realism so like a celebrity gan is an example where you can make it maybe make a face and then make it look like a human face are there also gans like that for say structures or real world or like i'm [Music] not aware of something so just to make sure that the questions it's like are there guns that can be applied to examples being terrain or content in games to make it look like hyper realistic yes um i can think of proxies of things that might be in that domain not something that you would apply directly just to like especially meshes or something like this um because i've been very successful with 2d data so with images not directly with an entire scene um but

there are um so obviously the super resolution is also again by the way uh and so it can be applied to textures individually there are also examples from siggraph where this has been applied to terrain for example a version of image stimulus translation would be from a sketch done by hand into something that's not a sketch whatever that is as long as it's an image in that sample you can sketch some like mountains and then you get a full terrain height map type of a situation um so it's more creation it's not as much refinement which is i think maybe what you're after um but you know if we talk in detail about the complete examples we might be able to even just in a conversation to determine whether or not these things apply and this is the kind of conversation that i want to kind of start between machine learning and artists to to see what problems we can help you solve for example cool thank you

hi great so thank you um the when you spoke about enlarging textures and training it beyond bicubic do you think that that could ever become a runtime solution as opposed to an offline thing to actually you know be optimized well enough without so i i can say that based on my experiments it's not real time today um but uh it takes let's say in the order of a second to to get like a 4x or an 8x address yeah um i do know people doing research in particular like i've talked to people at places like lg for example that are interested in ultimately doing this in near real time on device for when you don't have 4k content you want to do something better than just stretching the this is it i'm just thinking you could take shadow map density and all every element of the game beyond it just yeah rather than just underlying textures and i don't we're not there today um

it might be a matter of time like many of these things are possible because hardware has gotten better there's obviously improvements in the algorithms but hardware has gotten significantly better to run something like that in like a second you still need a decent gpu so it's not something you can do today on every frame some of these things might become even more viable as more game content is streamed from the cloud for example because that provides us machining people a very controlled environment we don't have to worry as much about the cross-platform and the different kind of levels of hardware availability and you reduce the latency between where the game is being rendered and when where the machine learning is happening um but the short answer is not today once per second kind of thing awesome thank you all right i think we're done with time anyways so thank you