Teaching an AI to Beat You At Your Own Game!

let's check my baby here it is it's training teaching and III probably feels a lot like teaching a toddler every time it makes improvements you feel super proud let's just keep it alone did we keep trying at the beginning Dai is just super stupid and just tries a bunch of random moves but the more it plays against itself the better it gets in this day and age understanding the basics of machine learning can be pretty damn useful so in this video I want to share what I've learned so far and answer some basic questions like how the hell does all of this work and could you use it for your own projects this is how the entire project started out a couple of months ago the ultimate goal creating an AI that plays this game better than me the game itself is quite simple you have to push the other player off the platform and you can move to the left to the right down and you can also boost yourself upwards but that consumes this bar over your head which refills itself automatically over time so there's quite a bit of strategy and resource management involved if you waste your

energy too much you will probably lose in the late game now the main question is how do you even write a program that is able to learn I'm not gonna go too much into details here but I want you to understand how it works this is how our classical neural network looks like every orange circle represents a neuron and every neuron has an activation value between 0 & 1 the first layer right here is the input layer that's where we input our data so for this project that would be the position of the ball the position of the enemy ball the velocity of the ball the energy you currently have all that sort of stuff every purple line represents a connection between neurons every connection has a multiplier value which can be both positive as well as negative there's no limit for that one so let's say this connection has a multiplier of 0.5 that means this neuron will have an activation value of 0.5 because there's nothing else coming in here so every neuron pretty much just adds up the incoming numbers makes sure that result stays between 0 and 1 and that data gets passed onwards through the next connections at the other end of the

network are the output neurons those represent the actions the a I can take for example the buttons that can press and then you just say an activation value of larger than 0.5 means the button is pressed and smaller than that means the button is not pressed I hope you can tell that this can make for some quite complicated behaviors and the entire behavior of the eye depends only on one thing the connections or more precisely the multiplier values of those connections so what this really is is an optimization problem we have hundreds maybe even thousands of connections and we need to find the very best multiplier values for all of them now for this project I chose a slightly different architecture but the same concepts still apply each neuron is connected to every neuron before that here are our input neurons and then here at the end are our output neurons thus that structure look familiar somehow oh yes you guessed it the structure on the wall right here shows what's going on in vai right now that's the input neurons right here

normal neurons in between and output neurons with their size they are showing us what they are current activation values of course what you can't see here are the connections between the neurons but Oh believe me they are there so main problem once again how the hell do we find the best values for all of those connections there are a lot of different strategies for this out there simulated annealing gradient descent particle swarms but they all sound kind of complicated probably because they are so I decided to go for the easiest one which is trial and error or if you want to put a little more elaborately an evolutionary algorithm that means we just make random adjustments to our connections and if our new configuration works better than the last one we keep it now how do we find out if our new configuration is better than the old one easy solution let's make them play against each other so this is how our training process looks like when we get started I think all of the connections are at zero that's why there's nothing going on in the brain of vai at the top

here you can see who's playing at the moment the mutation is always the one we made some random adjustments to as you can see the mutation one year so now it needs to play the rematch on the other side unfortunately it seems like it's not gonna win this one so we don't even have a superior strategy to doing nothing here that means the program will now create a different mutation and just keep doing that until it manages to win on both sides by the way the bar in the middle here always shows what's going on in the right hand ai at the moment it's showing nothing because on the right hand side is our empty version zero just so you don't get confused about this and I think by now you can already tell that this will take a while luckily there's no need for this training to take place in real time so I can run the simulation at 100 times the speed running it even faster I would obviously be even better but the performance of my PC kinda caps it out there here we are already at version 2 which means we already found two configurations that worked better than the previous one that doesn't mean they are magically super smart now it just means they managed to beat all of their previous versions let's give them some

time now and we'll see what happens here we are a couple of minutes later a couple of versions later down the road and as you can see they already look a little smarter the mutation always takes the currently best version as a starting point this way the chance that it does something useful is a lot higher here you can see the mutation won the match as well as the rematch against version 5 so now it needs to play against version 4 and that's just to avoid these rock-paper-scissors scenarios where the current version can only beat the last version but not the previous ones so the mutation will only become the next version if it manages to beat all of the previous versions now let's have a look at how the AI played after 50 hours of training which by the way it's only an estimate I didn't measure the time but definitely something like that plus minus 10 hours now as you will be able to see here obviously version 54 plays this game a lot better

[Music] the matches are quite exciting the only question is just how does it perform against a human player so I'm controlling the yellow ball here the AI is controlling the red ball and honestly it plays quite well the overwhelming result is humanity wins if you play a very predictable style 1di is used to playing against then you have almost no chance the AI is very good at a certain meta game it captures the high ground and then it boosts whenever you boost this way it always keeps the height advantage you can absolutely counter that strategy just in this match I didn't manage to pull it off and yeah as I've said the AI plays quite well in a certain meta game the only problem is when you do something unpredictable uh I guess I just found something really set I just wrote a very simple AI it's just one line it's just this line that's nothing but move to the center ok so watch this the 1 9 AI that just moves to

the center the self-taught AI that has been training for about 3 days straight geez let's see what happens just moving to the center oh yeah what the hell is happening here it is not very adaptable it knows a very certain meta game and it's good in that meta game but as soon as it's confronted with new unknown situations and strategies it goes completely crazy so here once again the one-liner versus the self-taught AI and once again the one-liner wins da I played really well early as a why is this happening now unfortunately these neural networks are almost impossible to debug I mean you can look at the numbers you can look at all of the connection weights you can look at the activation values the problem is just you don't understand at all what's going on it looks like a bunch of random numbers just look at all of the flashing dots in the background

here do you understand what's going on well neither do I but wait it gets even better our on writing this this is a very traditional AI about this long not super complicated okay this time around the yellow one is the self-taught one and the orange one is the one I just wrote the funny part is that they are a lot more evenly matched now so the self-taught AI does better against this one than against the one liner which is kind of funny still the AI I just wrote in one hour obtains the victory pretty much every single time so did I manage to create an AI that placed this game significantly better than me no I did not I failed what went wrong I have absolutely no idea I can only make a couple of assumptions maybe I didn't have enough neurons maybe I didn't have enough training time maybe I used the wrong training algorithm there's a lot that can go wrong and not a lot of ways to check what went wrong I'd say one of the biggest bottlenecks for me is also just how fast I can run the simulation at max speed I can simulate about 2 or 3

matches per second which for a complex optimization problem like this is not enough I think machine learning is just super fascinating so if I ever make a part 2 I will definitely make sure to link it here I would love to improve this AI even further but for now I gotta get back to work on will you snail the indie game I'm currently working on this is Jonas and I document the entire journey to becoming a game developer hope you enjoyed this video hen see you in the next one