Math for Game Programmers: The Math of Deep Learning

[Music] it's thanks for coming that's an amazing turnout my name is Alex I'm a co-founder creative AI and in general we apply lots of AI techniques into creative industries one of those is deep learning well that's more of a hobby of mine I apply algorithms on recent papers that I've read and put them onto github including this one that I did recently called neural enhance you can find the code at github slash Alex JC that's all the neural network and another project I did this one's published last year the ICCC called neural doodle where you can just sketch cool things in the neural network will paint that out in the style of an artist so in this case it's turning my doodles into hanwha painting so that's also on github and my journey with neural networks actually started a long time ago and fifteen years ago I

published an article called it was a book chapter called the dark art of neural networks and luckily a lot of things have changed since then and now they actually work and and we understand that much much better like the entire academic community is kind of converged around deep learning and adding ideas or taking the best ideas from the field and we've got a much better theoretical understanding of how they work the downside however is that deep learning is still and it may always be a dark art that part is not going to change you will never understand a deep learning model as well as we understand the rule-based system or a finite state machine and the reason for that is that they're modern deep learning models can be made up of up to a billion parameters that's a billion floating-point numbers that somehow combined together multiply divide they have to somehow understand how this works and you will get a very different sense of understanding of how such a model works compared to a finite

state machine with a few states so what I want to give you in this talk is an overview of all the different tools that you can use to understand deep learning and starting from the data side of things and understanding properties of the data sets how they can help us understand how that data flows through these different models so covering the models in the computation and then finally the more the learning side of things in the optimization there are lots of different fields of maths that help understand deep learning and I'll give you sufficient references so you can dig into it at home not too many equations but a lot of homework so what is deep learning ultimately the simple answer is that you compute an output Y based on an input X and deep learning is basically making a really heavy memory intensive overweight bloated function f the world compute Y from X but it can do things that no other thing no other machine learning algorithm can do but I

like to think of deep learning is basically taking all the ideas of neural network that work well in practice and and moving more towards the side of software engineering so it's a field that is typically called differentiable computing so not just solving problems that you'd expect with neural networks but generally building large programs that just happen to be a differentiable which means we can train them using gradient descent or other optimization algorithms and this is particularly important because these two disciplines are going to merge the software engineering and deep learning are becoming closer and closer thanks to libraries like tensor flow and so it's it's very useful to understand how deep learning works a bit more formally because it's going to become such an important part of solving difficult problems in the future so when should you use deep learning and well the simple answer to this is that it's the same as machine learning in general whether you're doing a classification problem you're predicting

outputs that classes this is a cat a dog that's your output Y based on input X could be an image or a regression problem which is basically predicting continuous values Y from the same input X then that's kind of a special case generation is sort of predicting very large images based on very small inputs so that's kind of deep learning is also shining in that department that's one of my favorite applications of deep learning so what really justifies using deep learning is actually the dataset itself so you need to have certain properties in your data set in fact I think there are three big properties of your data that justifies using deep learning otherwise you might as well just use regular machine learning linear classifiers these three properties are a very high dimensional input so if you're working with large images or videos or even a highly sampled audio signals and deep learning is ideally suited to this and that's why all the progress has been

made on image recognition and voice recognition and translation because of really a lot of data there and deep learning does it exceptionally well with it but it also requires you to have a sufficiently interesting problem so this sort of think of deep learning as as trying to approximate a manifold in n-dimensional space so it's like a decision surface that separates true from false in in your in your input space and neural networks are very good or deep learning in general is very good at finding these complex decisions surface and if you just had a very simple decision surface you wouldn't need a deep learning model for that the third thing is having a lot of examples and this is almost a cliche of deep learning that he need a lot of data for but Google's internal data set that they wrote about in some recent papers have over a hundred million images which is completely crazy and it allows you to be very lazy about how to approach things you just grab that much data and you don't have to worry so much however it

does help if you have a statistical understanding of how the data is structured so if you have these images it's good to have some sense of how the data is distributed so are the more cats and dogs and if so we need to rebuy a things or if there are very few chairs we may need to train the neural network a bit more on those variations to get it to be a just as accurate or recognizing chairs and it is animals for example so this is called data D biasing and it's over I would say a relatively standard practice in machine learning and data science but what deep learning is sort of increasingly doing is also doing the same thing but in the input space so it's a bit harder to do because typically these input spaces are very big here we see a data set that's very imbalanced there's lots of samples on the right side but we're missing information on the left side so we could gather more examples from the left and have a much better representative picture of this data set and this is more valuable than having a lot of examples that are condensed on the right side it's better to have a nice

distribution of your input data across all the dimensions that you care about and this is very active and important area of research and effects Google's data set a hundred thousand 100 million images is actually partly automatically labeled as well so they didn't go through all those images and manually assigning labels to them they just understand that based on proximity or the structure of the images that they have a certain tag so they don't have to go through that whole thing manually and that is kind of the the essence of this field of interactive learning being able to suggest questions to humans say what is this and then popping that up and getting a really valuable piece of data is better than having 10 times more data there's a very low value here are some references if you want to go into more detail the slides will be available later you don't have to scribble it down so now we have this data we can basically feed it into

neural models of deep learning models I like to think of these models as computational graphs I'm not going to use the words biologically inspired or I might occasionally use the word neuron but deep learning is progress way beyond the biological inspiration that it originally started with now it's easier to think of these models as very large computation graphs that basically take some input and then produce some outputs where the step steps of operations in between so on the left side we'd have our input X and that is typically a tensor so that means it's a a 2d matrix for example if it's an image if it's a batch of images it might be a 3d matrix if it's a batch of videos it might be a 4d matrix and then on the output we have similarly it's going to be an also a tensor which is going to be another matrix of it could be boolean values that this is a cat a dog a chair or probabilities of different classes so

within this computation graph we have even more tensors and these are potentially smaller matrices of different sizes scattered throughout the graph and these make up the model what we're going to be doing our computation way than what will be trained it it becomes this function f and we have lots of little tensors scattered throughout and some of them will be relatively small others will be much bigger and it's effectively you can think of it as a way of expressing all these different matrices that build up one bigger matrix multiplication with lots of custom operations in there instead another way to think of this is that when you specify these computation graphs in for example in tensor flow you'll specify what happens to your input X and the line might not immediately know what it is so there's some abstraction there and then it will sort of reason about these different operations in in this abstract way like multiply this input times this other matrix and then perform a nonlinear function on that at the Sault split it - and all these different

steps are specified on abstract concepts and then tensor flow or theano or the deep learning libraries will compile that and run it on your GPU as fast as it can so just two things to note about these computation graphs is that the first is that it's important to have nonlinear functions regular intervals throughout the graph if you just have linear functions throughout your your deep model you'll end up with just another linear function so it's going to be a really expensive linear function which is a complete waste of time so you want to put nonlinear functions into your into your graph to really give it the benefits of the depths and the deeper you go the more benefits you get as long as you add normally near functions so from left to right these are some of the most common the most popular ones on the left is the tan h function it's like a smooth step from minus 1 to plus 1 sigmoid looks a bit similar except is between 0 1 and they basically use this very often room for a

machine translation and sequence the sequence learning in the middle we've got the r lu it's called the rectified linear unit which is basically defined as 0 on the negative infinity side and it's the identity function on the positive infinity side and this is used extremely often in image recognition image processing and then on the right side we have the Lu exponential linear function which kind of a mix between the two goes from minus 1 smoothly transitioning into a linear function and this is used more often in image segmentation and yeah I like this one it's said it's a good one for generative models so the second thing to note about the computation we here we have lots of linear nonlinear functions throughout the graph but it's also good to have these residual connections so this is a concept that's borrowed from signal processing where you have a base signal and then you add a bit more detail on top of that so in this case we're going to have a separate branch that branches

off and then another computation is done and then we add the result back in later and this is important because it helps the data flow a bit more smoothly if you can imagine a graph like this that has thousands of layers like this is that the biggest debt that Microsoft has trained went up to 1,500 deep models the data flows much more smoothly if you have these regular skip connections throughout the graph so that means that when you're going forwards and you're computing these functions it means you can remove parts and it still kind of works but also when you're going backwards it means that the the training procedure has a easier time updating all the values in the neural networks I think this is one of the biggest insights of the past couple years actually it made a big difference in all the competitions so the tricks we've seen so far are on the general side they're reusable concepts you can apply them into like whatever function you're trying to

approximate you can use these but there are also very specific tricks that apply to for example image processing and this can be thought of as a node inside the computation graphics called a convolution filter you might have heard the word convolutional neural networks and this is what's going on under the hood it's like it's taking a really small kernel or 3x3 matrix and scanning it over the input image and then producing another output as a result so you can think of this as like edge detection or pattern detection on the 3x3 scale and then producing this sort of the output grayscale image that shows how strong those edge detector edge detector were activated so you can imagine stacking these detectors together we have more on three 3x3 matrices and just ending up with more and more output channels and they're all detecting different kinds of patterns and those patterns then get turned into more patterns and as you make the neural network deeper it learns patterns of patterns and building higher level abstractions so on the right you see

there are gaps between the blue cells that means we're sort of padding them with 0 that means you can do things like op sampling operations you can also skip certain cells as well you can do down sampling operations and you can change the kernel size and make it a 5 by 5 matrix and so in practice most image recognition neural networks use layers and layers of these operations just stack together to basically form these deep computational graphs that recognize images so another common pattern that's used often in image not image recognition translation is the LST M cell so it's a long short-term memory and if you can think of it as like a almost an electric circuit it's very specifically crafted to be assembled together so they put together like thousands of these and then they stack another layer of them on top of each other and they can learn sequence the sequence translations so they're

very good at translating and that's what Google uses everywhere so here are more details about the models but so far in general deep learning is difficult to get right when you're trying to figure out what you need to plug together it's a very intuition driven process and this is the part where the the dark art comes in you need to have some sense of what's going to work and you can do that by reading a lot of the papers in the field so every morning I try to read some PDFs from archive.org which is a preprint server for the machine learning community has sort of converged on so you can get papers that are related to what you're trying to solve and then get a sense of what architecture they're using and try it out but so in practice if you have hundreds of thousands of GPUs available to you if you're willing to spend the money on their cloud compute then you can use more experimental testing approaches trying

out all the different variations of your architecture and seeing what comes out it can be a bit expensive but that makes it more of a methodical like an experimental science as opposed to we don't yet have a full analytical understanding of how these architectures work so in practice was they what this means is it puts more emphasis on the computation side of things and understanding how the data flows through your neural networks and that's important to understand the statistics of the data as it's progressing through the network so in traditional machine learning you have to well it's recommended that you normalize your input data which means you want the mean to be around zero and you want the standard deviation to be around one so that has a lot of benefits all these nonlinear functions that we talked about if you feed in a nice normal distribution into these functions you get a mix of Plus Ones and minus

ones or of a positive or a zero and in the case of the middle function and so this means that the neural network or the deep learning model is actually making decisions it's actually deciding whether something should be true or false or a plus one and minus one where as if you weren't normalizing your data it might end up in the case of this tan edge function all the values might end up being one point 0 in which case is giving you no extra information about your input data you've just lost a lot of information and these were fundamentally the problems that the machine learning community wrestled with for about 20 years until they figured out what was happening so it sounds simple but it took us a while to figure out how how things were things were going on under the hood so as you take your data and you filter it through this computational graph like the deeper it gets the harder it is to to predict what shape the data distributions going to have if it's going to drift off towards infinity or if it's going to converge

towards zero and end up just vanishing completely so that will depend entirely on the kinds of operations that you're using it will depend on how you've initialized your data how you've initialized the parameters and so it's generally a very difficult thing to predict these distributions so the trick and this is what brought back there the deep learning craze is correctly initializing the parameters this is one of the approaches such that your output distribution is basically also a normalized distribution so if you set the certain weights and sort of biases you can sort of shift your data distribution around and then make sure that it is a normal distribution again so there's some great papers on this this is a recent paper with a new approach to doing that called weight normalization it's a well written paper and quite accessible the other approach which is used almost everywhere is called batch normalization so

basically keeping track of the mean and the standard deviation of your data set as you're filtering it through the graph and then effectively you're you can adjust it on the fly as the more data you filter through it then the better will understand how to to move the data set around and this is a technique that allowed us to really get much better results in image recognition and it made the significant difference this is one of the the most important techniques of the last five years so the result of applying these techniques things like batch normalization you get a nice distribution of positive and negative results like each of the channels inside your your deep mode making decisions positive and negative and you end up with a nice sparse result as opposed to just having zeros everywhere so and this is what so made things like neural style possible because each of these different photos as you filter it through the network will have a different fingerprint or a different unique

identifier and it looks a bit like this so you can train for this and try and encourage these results to emerge and I'm gonna skip a bit quickly through that what this means is that you have certain channels of certain neurons that have certain activation patterns and you can remove them and that work will still mostly work so we don't want to put all of our eggs in one basket there is that there's a certain amount of redundancy there and the the systems are relatively fault tolerant thanks to these representations and yeah you can train to encourage these representations to emerge and the yeah I'm going to skip this because I'm running short on time but that leads into the section on training and gradient descent so this is where we take all these the models that we've talked about and we actually train them to approximate the function that we're interested in learning and in general this is also from the machine

learning perspective they're two functions that you can use to train a model the first is mean error which is you calculate the difference between what you have and what you want and then you use that as a signal to basically train the whole network if you have non continuous values if you have like boolean's like this is a cat a dog then you'll use another error function it's called a categorical cross entropy and so that these are two they're not specific to deep learning there you'll use them in regular machine learning as well and the way deep learning takes those signals and then trains the model is use an algorithm called back propagation and back propagation combines two things it combines the chain rule with the update rule the chain rule is there to take the error which is calculated down there on the right side we calculate how far off we are from our target and then we sort of propagate that through the graph again and the goal is to calculate the

gradient of the error so we want to know what direction we need to adjust the weights in to get a better result so there are derivations of how you can do this there are some great tutorials and links to this I'm not going to derive that now but as a result of applying the chain rule and going back we get a sense of what the error function looks like locally and so we can step in the right direction to minimize that error that is done using an update rule and there's a collection of different rules that you can use as the standard SGD which is stochastic gradient gradient descent and that's the the classic it's been around for decades and there are others are more recent more modern and they have a lot of different benefits but all the bottom ones use a lot of extra memory to sort of maintain they maintain extra per parameter weights that how quickly to step so you basically get

a lot of acceleration for free but it costs you memory so all these different algorithms the ones that are used the most often are the plain old stochastic gradient descent so the simplest thing you could possibly do like just take one step in the direction of the gradient based on random data and the most second most popular is probably atom at the bottom which is used very often Eve is the up and coming these academics and their names and rmsprop is also it will come back you most of the time as a practitioner you don't need to know about these they're completely encapsulated you just call this update rule and it just works and you don't have to worry about the details one thing you might have to worry about is that deep optimization or optimization of these deep graphs is a non convex problem so you have no guarantees that you will reach a global minima but as it

turns out and this is very recent research on the pop doesn't really matter that you can't prove that the global minima can't be reached because it's such a huge parameter space with over a billion different parameters there are minima's everywhere and there's many symmetric versions of the solution and as it turns out these algorithms like the update rules will converge often to results that are if not the global minima they're pretty close and so you won't notice in practice so there are also some proofs if you simplify deep learning down to some very basic primitives then it actually is convex so if you wants what we can mathematically prove it's not quite the state of the art but the state of the art stuff you don't have to worry about it it mostly just works the hard part is figuring out the architecture so here's some more great tutorials that go into more detail about the back propagation process and I highly recommend those and again I'll post the slides online afterwards

so to summarize data is important but it's more important to understand this statistics of your data and make sure it's well distributed across all your output categories and all your input space the models in deep learning of progress way beyond neural networks and biologically inspired models their general computation graphs that you could integrate into your software to solve problems based on data making architectures and models for deep learning is difficult but you can sort of get away with doing lots of simulations and seeing which one's work best it's a bit computationally expensive but it's quite a reliable way forward and from the optimization perspective you can't guarantee for sure that it will converge but in practice it always does and modern deep learning libraries are very reliable so you get very good results from them great that's it for me thank you very much [Applause] [Music]