The Alchemy and Science of Machine Learning for Games

[Music] I'm Eva key I'm heading the production services studios at Ubisoft Montreal and in particular Ubisoft LaForge which is our R&D departments and in which the different prototypes that I will show have been created by a very talented multidisciplinary team thank you so unless you've been intentionally not following the news last year you have urged about AI but the word AI means many things we usually generally we use it in place of NPCs so it's worth setting some vocabulary here AI represents the concepts of machines that can learn reason and act for themselves machine learning is a subset of AI there are programs using statistics to

find patterns in massive amount of data and use those patterns to make prediction on new data deep learning can be seen as machine learning on steroids it extracts layers features or learned from complex data in order to make prediction and because it analyzes the data through many hidden layers it is called deep learning don't worry I will explain more in detail what it means later people often refer to AI disruption in reality the most recent developments deal with deep learning the hype of deep learning is at its maximum and because it's still exploratory we leave a time of excessive expectations followed by excessive disappointments so

what I hope to accomplish today is to provide with a more concrete view of what machine learning really is can and can't do and most importantly how to include in our production process I will first show a handful of examples to explain why it has the potential to shake the way we work I will also try to briefly explain what it really is and why it requires to rethink many parts of our production which is why it's not only technical a technical aspect that needs to be considered when talking about machine learning then I try to assess maturity and pace of machine learning applications and finally we'll explain how Ubisoft decided to read the change the first part of my presentation is powered with examples of prototypes that

have not shipped yet in a nun's in an announced game and the second part of my presentation will show concrete examples that have been applied to real productions from last year so let's start how disruptive is AI we've been making video games for decades nai has been at its core whether to shape the players interactions or in our production processes so why are we here to discuss about AI why is it considered as something new or disruptive for many domains including our industry let's start with a hint this is how we made the cinematics in Assassin's Creed origins two years ago so dark and scary with

white poppy you always get too scared let's go on a real hunt so as you can see this requires capturing a lot of data and many people from many different crafts logistic of such traditional methods is expensive and not flexible think about that for example each retake requires a lot of efforts and mobilization from many different people so can a I have a concrete impact that shakes the foundation of our pipelines let's ask to someone who has been heavily impacted by AI last year by AK can you explain us how disruptive AI is hello everyone creating in-game cinematics requires

many skilled professionals such as motion capture animation and dialogue experts one year ago would have involved hours of manual work to record and animate this dialogue now what if I told you that everything about me is driven by artificial intelligence my voice has been generated from a simple text file which then drives my lips animation it's totally automatic takes just a few minutes and soon we'll work in real-time and the best thing it works in any language by the exam Rapala philosophy connection cannot yet I'm not sure what the limits are but it gives you idea of the incoming magnitude of changes for many professions involved in video games such as animation and dialogue experts another example assassin's creed's is a franchise recognized for the quality of its open worlds and in such games diversity is key however the size

and the richness the world's leads some technical trade-offs as an example you can see here that the characters feel a bit on a rail if you pay attention is there a way to create more diverse and organic animations while simplifying the production process what if instead of creating animation data offline we were able to generate motion on the fly based on some high-level context this is an example of data-driven animation or in other words a system that generates the Wrights motion animation it works in real-time and instead of creating animation clips and rules to play them the AI decides what to play amongst all the data so not only does it require less production effort it will also offer a more appealing feel

for the player in the next example steel from Assassin's Creed origins please pay attention to the characters who want to sit down the transitions between different animation systems like motions and stations are difficult to manage and it leads to animations that feel not really natural we usually blend clips that were captured with mocap so what if we did not have to switch between animation systems and generate transitions in this example the same system just the same as before generates more believable animations in any context and in real time we have a lot of fun with our

characters so what is machine learning the part of artificial intelligence that has recently evolved exponentially is called machine learning I will try to explain here the core concepts of machine learning there's nothing technical here but it is important to understand the data-driven application and approach of machine learning because it's key in the change we have to lead in production just raising hand how many of you are already familiar with those concepts okay so half of you will have to bear with some simplification that we'll have to make for to explain what machine learning is but I think it's important that we are all on the same page here is a simple prediction problem if I'm asking you the value of F your brain

will analyze the data they compose it and generalize an approximation this is a generalization step and then extrapolate which is a prediction step this very simple example illustrates two key concepts first the principle of a given data set that illustrates a desired behavior and a learning algorithm that uses these data sets to create a function that enables generalization to new input and this is what machine learning does but we use it for problem a bit more complex than drawing a straight line it uses three dangers in gradients first a data set that represents the desired outcome a neural network which is a transformation function basically a set of mattresses

and also learning algorithms that parameterize the transformation function basically feel the matrices so the data set is used as a set of input and output examples for a neural network the algorithms measure the difference between the network output and the desired output or in other words what we call the ground truth the process is obviously iterative in a loop that is called the training phase that can last hours days or even weeks and based on those examples it sets the weights of neurons which means filling mattresses or in other words Bahamut Rises a neural network somewhat the machine learns based on examples the normal network is then ready to generalize as a predictive model so a new input provides a nun put but because

machine learning approximates the ground truth based on its own observation the output always comes with stats like precision confidence recall and etc it is also important to note that simple data sets for simple input treatments have been used for the case for example for handwritten character's recognitions so what if we use complex data like images or sounds or even what is apparently the biggest challenge on the Internet which is which is recognizing pictures of cats the principle is the same but the models to extract features or generalize is different the key idea is to decompose complex data into smaller manageable features and because there are many hidden layers it is called deep learning what is interesting

though is that the mathematical methods behind deep learning are not new however the first applicable results of deep learning real applicable results happen in 2010 the missing link was computing power it works exactly the same way the algorithm extract features from the data and then populates different layers of neurons each layer represents a step in the future extraction when the neural network sees new pictures it predicts whether the image is a cat or not by extracting the features from the input data simple datasets require simple machine learning problems and models complex datasets require more complex algorithms to populate hidden layers the

truths as feature extraction and this is why we say it is data-driven instead of implementing a specific outcome or behavior if these than that we focus on the right data sets that represents this behavior and because machine learning always make a prediction when new data is shown it is extremely important that the dataset reflects the future usage of your network or it will lead to very unexpected results if the neural network learns from pictures of ducks and beavers showing a platypus believe me will lead to interesting but likely unexpected results now why isn't it important because as game developers we

have many types of interesting complex data like sound animation textures models which means we have a lot of data that is relevant for deep learning and many possible usages for predictions like asset classification behaviors programming or even as a generations it was for a simple or complex data set but now what if we just don't have the data which happens often well we have simulations and we can use some specific kind of algorithms called reinforcement learning the idea is simple an agent is given a set of actions goals and rewards when it finds a way to reach his goal we also set punishments when it reaches some specific outcomes when it fails

then the agent tries many different things in the simulations or in the game to maximize its rewards after a while by trial and errors the algorithm finds the right strategy the strategy populates the neural network this can be seen as creating the Wrights datasets that map states to actions so when a new situation is given to the neural network it predicts what is the best action to take so here are the main things you should remember for this section but the most important parts is that more than a technology it is a new way to think it's less about

programming a behavior it's more about choosing the right datasets that illustrates a behavior with the objective to generalize it so I hope you get a better idea now of the implications of being data-driven it is not something that happens behind the scene or under the hood or only for technical people it's really a significant shift in the way most of us work and it has strong production implications traditional approaches require a lot of programming they provide with complete control in a very manageable environment however they are difficult to maintain and to scale up this is a very simplified version of a tree if you worked with animation trees you know what I mean conversely data driven approaches lead

to less programming and offer a offer less control huge amount of data are passed through math and intuition which requires to juggle between hard science and some sort of help alchemy but the reward is significant as they lead to unprecedent capacities to scale up with very low maintenance efforts because it relies solely on training data if you think about that when you want to implement a new behavior it only requires to add new data and retrain the model concretely instead of focusing on creating data and rules people will focus on finding the right data to fit in a system that generates a desired behavior let's how concretely this applies to some of well-known phrase

voice designers today voice designer use a script to record actors apply some treatments and associate each sample to a predefined action in the game being data-driven voice designers create the right voice inference that will generate voices on the fly based on dynamic screen programming traditionally details a lot of if this than that code that is written to program a design behavior but with data driven approaches the program learns the desired behavior from a database of examples for example to generate code or even detect potential bugs and even predict online failures testing it was to follow a test plan that documents the desired in-game behavior or even techniques like smoke tests to assess robustness but soon it

will also mean operating automated test platform that will generate turn of scenarios overnight and will analyze stats and logs to point out and is that behavior I'm now going to show some concrete examples for people who are in charge to craft and PC behaviors trading living city with traffic is difficult programming all situations and exception it is then that that's difficult and maintaining them that is daunting so what if we used reinforcement learning here in our case we're given a agents access to a steering wheel brakes an accelerator and we give some rewards when it succeeds in for trajectory and or stub and ads and recessions and punishments when it fails and then the AI will try and learn to maximize reward thus developing a strategy to drive it says it saves time

on programming and maintenance for example for new vehicles new behavior specific behaviors and even new weather conditions so it works for our characters or vehicles but we can also apply the same kind of techniques for crowds or traffic on the Left we have a traditional tally of pathfinding algorithm each agent tries to optimize its own path on the right we have developed a new type of algorithm called swarm that works like a shared decentralized intelligence think flock crowd swarm and it results as more fluid and believable behavior and providing a richer experience i'm not saying here that one is better than the other I'm just saying here that it opens new possibilities here which trained an

agent we have a specific behavior which is to drive at high speed in a city so not only does it benefit our production capacity it also increases the quality of the driving to make more believable words this approach can also be applied as an assistance to game designers guaranteed hits or what we call exploits are difficult to chase and especially when they evolve you know specific patterns or combos often our community discover exploits before ourselves boats they open automated testing opportunities reinforcement algorithms can be used with agent Explorer all combinations to check if there are some holes in complex designs this is particularly important for games as a service where you need to make sure that new content works well but also the break existing content these are only a

couple of concrete examples they the there is a technical challenge but more importantly it represents a huge change in the way we work and these for all trades and this is why we should care we covered what deep learning is and have seen how it has a potential to profoundly change all of our trades now let's see how deep and how fast the video game industry is likely to be impacted many breakthroughs arise at academic stage that are spectacular but not still ready to reach a production stage for example a I physically mission worked well in a sandbox but are still for joyous with real-time interactions still I can wait to play a game with such advanced features some of those

breakthroughs seek to simulate a player and even beats pro players as you might be aware like deep blue did for gasper of more than 20 years ago if you remember however is it robust or even simply a fun opponent not yet the game is only used as a sandbox to measure the decision quality of an algorithm against the the quality of the decision of a human bottom line is serves only some specific research breakthrough of the eye field games are used to test those research our elements which means that even if it's spectacular there is still a lot of research to be done conversely it goes way faster than even experts expects this is extracted from a paper from 2016

hundreds of machine learning experts were asked when AI would be on par with humans for various tasks and you see here the results and for example they said that go would be solved in ten years ish from 2016 and it was solved in 2017 let's talk about asset generation here are generated pictures using a technique called generative adversarial networks as you see academic results were not that appealing for us a few years ago and now what they are and they are exponentially think about that known of those faces are real and in 2018

not only did generate features it now generates videos we're entering an era in which our enemies can make it look like anyone is saying anything at any point in time even if they would never say those things so for instance they could have me say things like I don't know kill monger was right ben Carson is in the sunken place or how about this simply president Trump is a total and complete now you see I would never say these things at least not in a public address but someone else would someone like Jordan Peele this is a dangerous time moving forward we need to be more vigilant with what we trust from the internet that's a time when we need to rely on trusted news sources they sound

basic but how we move forward age of information is gonna be the difference between whether we survive or whether we become some kind of up dystopia thank you stay wolf [Applause] so it's fun and impressive but still not in real time so now the question is do those techniques transfer to our industry I'm talking about the technique as you can see another field of application these techniques yield impressive results it's far from being perfect and does not work in all situations or in real-time but this is where we are now given the present pace of academic research I am positive that next year I should be able to interact more naturally and convey more emotions let's extrapolate here is what to expect

from the recent developments it goes fast three years represent the ballpark production of one triple-a game the question is now how do we speed up our innovation capacity in this field while making games disruptive means the capacity to shake established rules often leading to newcomers think of Nokia vs. iPhone a decade ago it is less of a technical challenge than a transformation of the way we work and the development of opportunity it creates also this is why it is important to discuss it not only in technical tracks or from a technical standpoint but also at a broad level what are the challenges we must overcome to benefit

from this in our productions every technology our process has lifecycle first it takes time to provide convincing results then most problems are overcome and the recipe is improved over time before reaching a plateau and then a new technology emerges like deep learning be clear today it produces results that are globally less performance than our well-established recipes but the question is how much time will it take to be on par we think it's gonna take two to three years don't get me wrong I'm not saying that deep learning will replace everything we're making by hands or with our current pipelines in two to three years what I'm saying is that it will take significant parts of our

pipelines in two to three years so how do we explore this area because we can't build and pilots the plane at the same time the nature of machine learning brings very uncommon challenges first it goes fast very fast most of our processes and production life cycles hurts our innovation capacity it requires data expertise but also a valuable applications and very few companies have the three it produces results that are industrial interesting but they mostly show up at academic level therefore it grows new culture of openness and publications that let's be honest goes against most of our industrial habits it goes against most regulations or at the very least

requires rocket create some because it raises important questions like the authoring of generator generated assets who is the creator data ownership or even usage like voice imprints for text-to-speech or data privacy and did I mention it goes fast I mean since I started talking a relevant paper probably hit the Internet so how do we address those challenges there is no one-size-fits-all solution we all have different production realities so I will try to share now a solution that worked well for Ubisoft and maybe some elements might be useful to your own context this is the type of open world we creates at Ubisoft we offer rich interactions for the player and all these requires tons of data

textures animations models but also AI to drive the world's components behaviors those simulations offer appealing technical and scientific challenges but also interesting assets to solve them we saw that as gaming industry we have to read assets first we have our game engines that can be used to train or test apps official intelligence and second we have a lot of very specific data such as mocap data textures dialogues models creating or using these data is key to the development of deep learning so how do we make it happen when AI machine learning developments is still essentially at academic level this is a

path of innovation and this is a scale of love love being whatever you want depending on your values money recognition attention investments a good researcher is known by the quality and quantity of his or her research publication which means that the academic world puts a lot of love on the left side of innovation at Ubisoft we focus on creating amazing experiences so when we try to collaborate on academic topics such as deep learning there is in the middle what I call the terrible gap of wasted IDs and almost three years ago we decided to bridge this gap Ubisoft LaForge is an open place where we bring in our people and researchers to work on prototypes that serve both interests

make greater games while creating public knowledge researchers at La Forge have access to everything like our employees do our data are experts or technologies or facilities the forge was created in Montreal three years ago and now spreading in other Canadians to you just to give rough numbers were working with eight universities regularly Austin at each point in time 20-ish active projects so let's pretend now we have a prototype then the main challenge is to match a relevant AI results with a real valuable application not only instance the key is to create prototypes in parallel of production cycles and make them targets a well-defined scope so in other words the production plan be all

my previous examples were prototypes that are not implemented in an announced game I'm now going to show concrete prototypes that were implemented in production without being initially part of a plan next is a prototype from last year based on Assassin's Creed origins unlike my previous example it uses real actress voices it was a new machine learning based technology called sound matching that animate sleeps based on voice and these for any language was Wallace barks baccala political regime or chef children alicent tsukuda Kaduri Oh terrible economy a dot para este es el hermano necesito protección al fin del Cabo ha ha ha

so they have Sasson screed OTC team decided a few months before goal master that the prototype was worth it to implemented it decreased costs while increasing the localization quality and the game was praised for the quality of these location amongst other things here is a short seconds taken from the game yeah we won kid it to despair alone Canandaigua the co prophet o kumi facto a delivery fee Pepe the body drachma LeBron duper Marco I forgot ah qualcuno pretty poor Tasha a penny for Horner is ass worth fighting game that relies on many different opponents which are

regularly added to the game so in this game we prototyped an automated testing feature to find exploits it helps designers to fine-tune the balancing also here's the smart bot in a wrench tries to beat the regular in game warden in blue and as you can see the smart BOTS developed a kiting strategy by itself using reinforcement learning trades are changing but the way they work together also traditional programming pipelines in involves many stages to test the code and find bugs with large teams or for quick turnarounds like continuous integration it can become daunting what if we leverage the data of past bugs and corrections to estimate the probability

that a new portion of code includes a bug before it is tested last year we unveiled a prototype called commit assistance that does exactly that it has been first implemented on Randall six and is now reaching over productions worldwide on Rainbow six they're still 30% ish of false alarms and it should decrease all the time as the system continues to learn is it magical no SIL you need to add some alchemy to the science as we saw before the output depends on your data and your data sets and your algorithm and crafting all this requires a lot of intuition and the involvement

of experts from different trades as an apprentice alchemists this is the kind of result you will often get at least that's what we got [Music]

so now it's time to sum up the things we covered more than a technology machine learning represents a shift in the way we program a desired behavior it focuses on the data and raises question of quantity diversity representativity curation variety usage and etc it is deeply changing our industry both because it requires to think differently and because all crafts and processes are impacted creation assistance is often a good angle to accelerate a pipeline while in production and not miss the train it goes fast and it's still

exploratory I suggest not playing planning for it like you would for a feature conversely I would advocate you to find your own prototyping pace and a research model that is linked to your productions within your reality being production plan B is probably good idea when playing with such disruptive elements thank you I think we have time for questions [Applause] are you shy or on the mic please so that everybody can hear it you mentioned that you in a kind of deep relationship with

a universities in Canada and and it's understandable how you know academia benefits the company but how the company benefits academia what kind of what you're giving back to academia excellent question why that didn't match and mention is that Ubisoft LaForge has few rules but one of it is extremely important which is colocation in other words researchers come in our studio so that they have access to everything so as a researcher of think about animation you work in in deep learning on animation there are very few data sets that you can use to make your research so when you come at LaForge working with us you have directly access to the tons and tons and salt of data that we have so that's one aspect but also we are pushing at the arm for publications so

each researcher who comes at Ubisoft LaForge is incentivized to publish and share what we discovered actually if you go on our website life force dot should be soft calm you will see many papers or links to many papers including the the commit assistance so the a I think that you know try to catch bugs in the mind of programmers high accent all thanks for doing this I have a question for you like so it seems and maybe it's my perception of this that you have a focus on this tool of machine learning can you speak of what is your perspective because there are to me like ways of having an R&D group play you can believe in a tool and then try to apply it around or go around and shop for problems and then find the best

tools for those and sometimes you say you know you shouldn't use a hammer and then try to but what what is the advantage that you see in this yeah excellent question Oh Lafourche is not a machine learning department Lafourche is an R&D departments apparently in the academic world most of the most interesting applicable results that we have revolves around machine learning deep learning reinforcement learning which is why it's something that we should take care of as as a gaming industry but actually we work exactly the way we all you described at Ubisoft LeFort which is we bring in problems suggestions and then we try new things whether it's machine learning biometry or even social sciences hello my name is Martin climate

thank you very much for your talk I have a very specific question I was very impressed by the quality of the generated voice can you please expand on this when do you think this become a viable alternative to actual actor recording it's it's a tough one it's a tough one it read appends I've this is an opinion I think that we will need actors we will need artists to add that extra layer of emotion however there are many situation where you don't need to put a lot of efforts recording artists and and and managing the data and post process and etc so I think that is it's going to be something that will happen gradually in other words today I note that some games already used

to speech for some voices that are not that important just you add more diversity in their world but then when you need more emotions or to convey more complex you know information to the player maybe it's a good idea to keep to keep real humans and this will happen gradually I think hi so one of the biggest challenges with machine learning is the bug ability do you have any insights for for the things I get experimented that improve the ability of machine learning models the believability me the bug ability diagnosing or understanding if the model is working or not not really because I'm not an expert I would say but definitely what I would say is it offers less control definitely now the question is more to try to assess is it important or not and that's where we need our our

experts from different trades if you are there were some very interesting talk about usage of very specific usage of machine learning on boards on animation made by Daniel as an example as you can see the models are never exactly predicting the ground truth now you need this external eye of a real professional or real animator that will say is it important in that case so maybe you want to generate a crowd and maybe you know it's it's okay it works well to the eye but conversely if you need animations for some specific gameplay fighting as as an example then using deep learning techniques on or machine learning techniques might be more tricky because you might have more difficult time to tweak things and try to explain you know how they are the system will react thank

you for the talk I guess my question was kind of related but given the potential unpredictably unpredictability of a lot of these systems you have a standard evaluation pipe one together to a point when you're comfortable put it into production the most standards evaluation pipeline we have are called people actually we're not Ubisoft lavage is not meant to create products or services we create prototypes so we're very far when something goes that of out of lavage it's very far to being something that you you know it's plug and play for production what we try to do and because although those question of testing reliability predictability and etc what we try to do is we have a prototype that works well that proves you know solving a real problem we bring this prototype and then hence that off to productions

who will adapt that to their pipeline so as an example we were talking about the commit assistance which tries to address the probability that some code that is submitted contains bugs what is the right level of reliability or or probability that you want to reach is 70 percents enough only a programmer can tell right so we work with them and working with people from the trade programmers animators is always a good idea because then we'll decide what in which usage it's helped them it's augmented an and you know it's not like an additional burden an additional I don't know dashboard that they received or thing like that

so we are also learning with that a lot any more questions I guess it's a no thank you very much [Applause]