Reinforcement Learning in Action: Creating Arena Battle AI for 'Blade & Soul'

[Music] i am jinyoung zhang the team leader for this project and the first speaker of this session in this session we will share our experience of making ai agents for the arena battle of blood and soul at first i will introduce the problem we solved and the key ideas for it and then soon the next speaker will explain how we solve the problems and challenges and show experimental results including them videos let's start with problem explanation blood and soul is a mmr pitch launched in 2012 and is now being serviced globally brilliant soul has shown distinctive visual styles movie-like scenes and combats inspired by realistic

martial arts this is the arena battle of brilliant soul the arena battle is a pbp sub content of bread and soul two players fight against each other in an instant battle arena two win each should make the opponent hp20 in three minutes if both of their hp is still above zero after time out the one who deals more damaging wins for the fairness their status are standardized our research goal is to create programmable ai agents in one-on-one better using reinforcement learning the reimbursement learning has been successfully applied to many games including some atari games gold chess qx3 dota 2 and starcraft 2.

however the ai modules of many commercial games are still being composed with handwritten rules this tends to make agents simple with full of weakness so as to give limited experience to users to make barriers and flores agents we try to apply reinforcement learning to it to achieve more effectively we also try to give styles to agents aggressive balances and defensive we thought that this would give various experiences to users inbred and soul there are 11 classes destroyer blademaster assassin someone and so on to reduce complexity we made a destroy agent that could fight against destroyers only we were choosing destroyer because it's a crash that steadily appears in bread

and soul world championship their skill settings are fixed to be the same for the fairness to describe our agent we drew the blood and soul arena battle in abstract form an agent would observe the environment including its own state the opponent state and arena state the observed state include hit point skill point the distance to the opponent the skill curl time distance to the rna boundary and so on an agent would choose an appropriate action for the observation the action affect the environment and in turn the agent observes the changed state to decide an extension the agent has a chance for choosing actions every one hundred milliseconds in the radiance or innovator action can be categorized into skill move and targeting

these actions should be decided strategically since they have trade-off between costs and effects i'll explain the skills of destroyer with examples of strategic decisions it has 44 kinds of skills which can be categorized as crowd control skill damage dealing skills resistance skills and escape skills to deal damage continuously an agent should decide one of the cloud control skills before damage dealing skills if the opponent is hit by crowd control skill the agent would decide the damage drilling skills consecutively to reduce the opponent hp as much as possible to avoid such situation the opponent should use one of the resistance skills before being hit or use one of their escape skills after being hit

since the resistance skills and escape skills have a large opportunity cost of crew time you should decide to use them carefully in some situations it should save them to use later this makes it difficult to learn the optimal timing for skill decision in this project we faced four challenges high complexity real-time response generalization and guiding fighting styles the first challenge is high complexity when compared with the game of google our agents have larger space in case of goal the degree of freedom of each accident space is about 10 to the power of 170 on the other hand our dead of hours is about 10 to the power of 1 800. longer game lengths and the combined action of skills moves and targeting by the action space larger

this makes our problems so challenging to solve this challenge we constrain the freedom of skill selection and reduce the decision interval of movements the details about them will be explained later second challenge is the constraint of real-time response since the arena battery is a real-time fighting game rate response result in poor performance even if our agent decides the optimal actions this restricts the method for implementing the decision policy for example we cannot use any search based method which can require which requires huge amount of processing the dual networks are very flexible and powerful computational method to express a policy using neural networks the complex computational complexity could be easily controlled to satisfy the real-time

constraints the third challenge is generalization actually we could not expect which programmers our agent would encounter to make our agent play well against any pro gamer we needed to train our agents against the opponents of various styles for this purpose we provide our agents with inspiring partners from the open pool of sales trained agents of various styles are added to the open to pull continuously to have stronger and more diverse sparring partners finally we needed to guide fighting styles without hand-written rules humans can easily adapt to the monotonous agents and the field board the agents of various styles could reduce this we define the trainable styles according

to their aggressiveness the aggressiveness could could be guided with carefully designed rewards the next speaker will explain how we tune the rewards to guide different styles hello i am sunro ai research engineer from ncsoft i will cover the methods applied for creating the arena battle ai since our agents are trained with reinforcement learning rl i will first introduce the basic concepts of rl and then i will explain the overall learning process including features and rewards finally i will present some engineering techniques which facilitate learning process okay now let's imagine a baby trying to learn how to work he must learn by himself with numerous trial and errors he first he falls down again and again until he figure out how how to control

his body this type of learning is correspond to rl let's get into detail these three boxes shows good and bad sequence of actions if the baby somehow falls down he realized that something was wrong but he cannot figure out exactly which part of the action causes the bad result likewise if he succeeded to keep walking he knew he are doing well as a consequences the signal that tells whether the baby doing good or bad is called as a reward the objective is to find the best action sequence to maximize the cumulative reward during a lot of trials the baby modifies his acting policy to increase actions in good sequences and suppressing actions in bad sequences as experiences accumulate he learns how to get more of rivers this is the core of our rl trial and error and improving acting policy

then what is the reward in bns are in a battle what is the signal that tells you are doing well of course winning is the main key if you win you're doing well but it is very smart signal because it only happens at the end of the match we need another signal to follow and that is hp if you took away the opponent's hp you're doing well and if your hp is reduced you are doing bad quite straightforward isn't it with these two components the agents learn how to win and how to take away the opponent's hp while preserving its own however since our agents are required to have diverse styles we introduced some additional rewards we decided to guide different degree of aggressiveness into the agents we have determined three dimensions of rewards to tune the aggressiveness

the aggressive player's priority is dealing damage while defensive players priority is preserving its own hp so it is reflected in the first dimension since aggressive play tend to approach while the defensive play tend to secure distance other two dimensions reflect these perspectives you saw this plot before right here the reward signal added the agent sends an action then the environment gives you the next observation and reward i hope you are familiar with each component of the plot exact feature features represent how the agent sees the world the agent decides an action based on the feature features are basically the same with that given to the human player so any accessible information to human during your game are also provided to the ai such as hp sp distance position

skill cool time and so on now let me show you the overview of the learning process the agent which is composed of neural network proceeds combat simulations in parallel in our system 100 workers operate simultaneously when games end networks are updated from the game logs with an rl algorithm called acer what algorithm does is basically modifying the probability distribution of each actions i mean increasing actions and suppressing bad actions this loop is repeated continuously here i guess you might wonder of who the opponents are there are two conditions for good opponents first opponents should be diverse enough if the opponents are limited our agent just learns how to exploit the fixed strategy but could not be generalized

for different opponents second the opponents also should be reinforced along with the agent as both sides of the combat are becoming stronger agents can experience higher quality of battle the self-play method with opponent full satisfied boss of suggested conditions we store each agent to the pool at fixed interval so the pull gradually grows then the pool provides a variety of variety of opponents and the opponents becoming stronger with the agent from this to following two slides i will explain more engineering focused techniques and first technique is about learning the move action policy learning the move policy is rather difficult because the effect of a single move is limited a single move indicates a movement of

0.1 second considering the character's speed how far could it go therefore single move action barely affects to the state to solve this we enforced agent to maintain move decisions for 10 consecutive actions this allows the agent to literally move and lead to the change of subsequent states and rewards the method enables the agent to learn meaningful moving strategy and finally we're able to run away or approach to the opponent we also facilitate learning the skill action policy with help of domain knowledge out of destroyer's 44 skills the skill that has longest skill reach is 16 meters therefore when the distance between the agent and the opponent is farther than 60 meters we just blocked most of skills because it is waste of resource such as

skill call time and sp it helped to ignore meaningless exploration and boosted the learning process especially in the early phase of learning i also want to give you some useful findings during future engineering which is discretization of continuous feature actually helped each skill has its own targeting range and accurate distance recognition is essential to hit the enemy for example there is a skill named drag and its range is 3 meters to 16 meters but the agent trained with continuous distance feature keeps miscalculating the skill range and use the skill when the opponent is too far or too close this is because continuous feature gives burden for model to learn the exact distance treasured for each skill to solve this we encoded the distance

feature into one half vector and continued training and then the agent was able to use the skill drag precisely in this section i will present some experimental results which includes diverse learning curves our first extreme experiment was conducted with an opponent called built-in ai learning how to defeat a single fixed opponent is relatively easier task compared to deal with multiple random opponents therefore before training by self-play method we first test the viability of the method by training against built-in ai from infinity tower infinity tower is a special dungeon in blade and soul where human players fight against npc ai we've used this sparring partner as a testbed of various algorithms and tune over hyper parameters let me show you the training results

first graph is time win rate graph it shows how room rate increases as time passes the second graph is time hp div graph hp div indicates the difference of agents hp and opponent chp when the match ends as can be seen in the graph our agent achieves 90 percent of room rate in just 30 minutes of training although the learning curve was splendid human player was able to easily defeat the agent it had a lot of weaknesses and it was even easier to exploit than the built-in ai so we moved on to the next step we now enter to a self-play training phase in this stage the agent fight against the opponent which is randomly sampled from the pool of past itself since there exists hundreds of hundreds

of opponents so we displayed multiple learning curves here each plot corresponds to the training curve for each of each of the opponent in the poll as you can see the performance against each of the opponents all stably increases then we briefly tested this agent against human and finally the level of difficulty also increased and human player felt hard to exploit the agent therefore we evaluated the agent with more systematic approach the goal of the evaluation is we want to measure how the agent fight well against the opponent it never had met before it can be called as a generalization performance or robustness of the agent so we made a match with this agent against built-in ai and skillful human player note that the agent has never met built-in ai before

because it is trained with self-play method nevertheless it defeated both of built-in ai and even skillful human player maybe at this time you might want to see how the agents actually fight so i guess i should move on to some demo videos these videos tell how the agency is getting stronger as training goes on female characters are ai and male characters are dummy opponents the leftmost video shows the initial model which acts randomly we measured time consumption for ai to reduce dummy opponent's hp from full to zero it requires 120 seconds to defeat dummy model after a day of training the ai agent was

capable of dealing damage with basic combo also it used crowd control skills before dealing damage to make sure the opponents cannot react it requires 42 seconds finally within only 20 seconds ai defeats the dummy opponent with a single long combo sequence as you can see the initial model is still struggling with the dummy opponent for your information our final agents that compete against pro gamers has trained for two weeks in means that each agent experienced up to four years of real-time gameplay

now let me show you how fighting stars styles are different between the aggressive and the defensive agent black characters are ai and why characters are human player aggressive agent keeps approaching and attacks relentlessly opponents have no break to think about counter-attack and assessing their current state on the other hand defensive agents tend to secure distance and it waits for an opportunity for proper timing to counter-attack you can see that agents of each style manages the match in largely different manner just before the live event called blind match we invited two prominent professional gamers who are the former

winners of annual competition for the fair evaluation we restricted our agent's reaction time with a delay of 230 milliseconds on average which is similar with that of professional gamers as you can see in the table the aggressive agents dominate the game while other two types of agents have rather intense games according to the gamers interview after tests this is because human players often lead to some break between fights for just think about current state and high level strategy however the aggressive agent does not allow human to have a break between battles but it only pours on attacks both of gamers agreed on that the aggressive agent is impeccable and flawless they also added that aggressive agent plays in totally different way from human

while other two type of agents play similar to humans and finally with these three models we went for the apply match let me show you the event highlight video [Music] players here of course you know you know colorize he's been fantastic overnight hey i don't know who that is i don't think they don't stop a game right now a bit of a dog okay okay all right so moment colorize show this bag all right i'm going in there we go but they both still all persistence is actually alpha knight j is looking great for colorize for now amber stop is down as well

oh drag mercy there ninja already moving forward and seeing what he can do oh we can get the win another one but not if night he has better gusto even with gusto we'll see how he's going to time that one up should be in favor of colorizing the drag connecting there it's gonna force out the gust already so pretty much uh similarities between the last set as well and what hopping gets you know but no instead punish instead oh gosh he's going at it against alright amber's time is also down oh he's just again though

very nicely done this might just be it is with that nine castle only a couple more seconds away from 14 more disgusting adjustments that will be the damage wave with the wedge coming up all that grab and that will be enough for shove around to take that game on victory yeah who just lost in the third place match with his team comes back in this event picture yeah we'll have to see how this happens because like we oh my god that stun that reaction time is crazy but i was saying no the first two destroyers were able to beat knight j but this time around the choice not looking great if he's the only destroyer that loses against knight this is devastating this is it though that's it that's it oh but there we go this is it three seconds not too long but it's gonna be looking a little bit better for knight j but not quite good enough

no though we will see once again the gust is gonna be coming back and that means knight jay if you hit the weight now the heel heel's coming back now oh my god that's it wow playing with your food knight j huh wow the overall results including the blind match is as follows the aggressive agents won all of the game except one and balanced and defensive agent made competitive results against professional gamers to the best of our knowledge this level of performance is achieved for the first time in modern complex fighting games until now we introduce the overall process with specific engineering techniques used for creating ai i want to conclude with brief summaries of what we've made first we succeeded to make pro-level ai

in complex fighting games based on rl methods second we guided fighting style without any hard-coded rules aggressive style is just one example and any type of styles can be guided with rear shaping third we trained robust ai with pool of opponents last we reduced the problem space with various engineering techniques it is the end of what we've prepared for today's session i hope you've learned some useful lessons from our session and please don't forget to evaluate our session we are jin yeon chong and sun noor from ncsoft thank you and our company prepared another session on thursday 3 p.m it is about gym learning-based universe kinematics if you have interested please stop by

oh okay i'll just take one one because oh we have two minutes so if you have question i will take only one oh if not it's okay i'll jump in uh or did you want to go uh you could probably answer this quickly it seems like the aggressive style was something that the ai discovered did players start incorporating that into their gameplay in order like did they is it a discovery of a game style or is it just a unfair advantage the ai has oh it is not unfair we we have very carefully designed for the fairer match so all the aggressive agents uh acts perfectly it has delay between its decision and its reactive time in game 230 milliseconds of delay so yeah it is a fair match

i wonder if you're allowed to tell us how did you adjust the weightings of the neural network based on the reward signal that came back how do we adjust the adjust the the weightings in the neural network rates of reward yeah we like rl method we have a lot of trial and errors and to make the style obviously we tuned the weights and tried many set of weights and if if they look similar we made it far apart the weight of each reward and that thing like that thank you