Deterministic vs. Replicated AI: Building the Battlefield of For Honor

Thanks for being here today. Uh so, just a quick reminder, uh if you can just mute your phone, and uh also, at the end of the session, um there is a survey that that you can fill. Uh if you want to give any feedback, uh it will be really appreciated. It's our first talk uh here at GDC, so uh anything can be uh interesting for us. So, this is uh Xavier Guibert, gameplay tech lead on For Honor, and I am uh Frédéric Bard, AI programmer on For Honor at Ubisoft Montreal. So, the game has been released 2 weeks ago. Uh the feedback we get from the community is really incredible. Uh but few years ago, when we started working on the project, we didn't know exactly what would be the game today. And in fact, most of the time, when you work on a new IP, on a new game, you don't know what you will be released at the end. You need to iterate, you need to prototype, you need to test stuff, you need to find what is fun, and try to build around it.

On the tech side, it means that you have uh multiple needs that will emerge, and as we were able to see uh during this conference, uh sometimes one technology is not enough to fulfill all these needs, and you have to uh test multiple approaches, and maybe have multiple approaches that uh cohabit in your game. So, the vision for For Honor was to create a melee fighting game. So, basically, you control a character, but most importantly, you control his weapon. So, you really are inside the battlefield, you are inside the combat. Uh one uh important aspect is that the game uh needed to have a competitive multiplayer aspect. Uh the game needed to be uh fair, because it's player against players. Uh it needed to be skill-based, responsive, reactive, and stuff like that. Also, the game uh we wanted to create a game where you are in a believable battlefield. It means that it's not um a kind of arena of gladiators where a few fighter are fighting each other, but really a

huge battlefields with the clash of armies, hundreds of NPCs that fight each other, and you are only one soldier inside this battlefield. So, for the AI we ended up creating two separated system. One deterministic, one replicated. Why? What What happened during the process? What are the need that we needed to fulfill? That's what we'll try to show today with this talk. So, we'll in this talk we'll first show a few fundamental of the game, how we control characters, how we handle the online part and stuff like that. Then Xavier will talk about the deterministic AI system. I will follow with the replicated one, and we'll have a few takeaways at the end of the session. So, let's start with the fundamentals. In the game, we have multiple characters that you have to control. First one is the hero, as the one with the more skills, the more abilities. It's the one that the player is able to control in the game.

But we have also what we could call the map fillers. So, on the left captain, soldiers, archers, this kind of stuff. These are the characters that creates the feeling of the army. Also, in the campaign we have a cute elephant and huge wolves. We have horses and other kind of characters, but so yeah, that's the kind of character we have to control. And to control this character, we decided to use the input driven approach. It means that it's like controlling a gamepad. So, if you have a gamepad, you just press a button and it will have an impact on your character. But it means also that you have indirect control on your character. You can't say in the AI, for example, I want my character to go at 10 m/s. There is no way to do that. The only thing you can do is give your orientation of the of the left stick of your gamepad and the strength of the stick, and that's all.

So, it means that the AI is input-driven. We use exactly the same pipeline for the player and for the AI in in this game. Which let you go to one of our tool. We call it Katana. Basically, it's a tool that the character designer used to create the characters. So, it's a huge state machine in which all the states, the blue boxes that you can see here, uh define a state of the character. So, it defines a gameplay um during a certain duration. The duration is defined by the length of of the clip. Uh and they can define all kind of properties uh the the movement available for the character at this moment. Uh does he have some defense? Uh maybe he has some modifier. He has some special abilities. And they also can define all the branchings between the states. And as you can see on the bottom, they can specify some kind of conditions that allow to trigger these kind of branchings.

So, uh yeah. So, that's it. Sorry. Um so, when what does an update loop look like in our game? It's in fact pretty simple. We just generate inputs from a pad. We send them into Katana. Uh Katana updates the state of the actor. We get this into the world, and we're able to render it. If we have multiple character, it's exactly the same. We just generate inputs for the other character. We send them into Katana. It updates the state of the associated actor, and we're able to to put that in the world. So, for the AI, since we use the same pipeline, it's pretty pretty simple. Uh the AI has to generate inputs, send them into Katana, and you know what's happened after that. But, how does it work in multiplayer? I will let Xavier explain uh this system. All right. So, Hello. So yes, our game is a multiplayer game. It works online. The players are fighting against each other. And the architecture that we chose from the beginning was to build a peer-to-peer

one. And the main reason was that we wanted as little latency as possible between two fighters. Between when you're fighting somebody, he's in front of you, he's in your face. We wanted it to be as fast as possible. So, what we actually send between each of the peers in a session are actually the inputs of the players. It's the only thing with very rare exceptions that are actually sent. And the way that it works is when a peer receives those inputs from the network, it's the exact same pipeline that Frederic mentioned. We go We get those inputs. It starts driving the new actor with Katana. But obviously, there's lag. We're not receiving those inputs immediately yet. So, what we do is we have deterministic simulation that runs on every single peer of the session. What that means is that every peer is responsible to keeping in sync with everybody. When we receive an input from another peer, we every peer rewinds in time and shows

resimulates every single step to catch up. And we end up with the same conclusion. So, uh the this deterministic simulation in the game, what Frederic showed you before, the graph, it's all our gameplay runs into that that simulation. So, everything is deterministic and comes to the same same conclusion. To illustrate a bit what that means, I'm going to give an example that doesn't really happen in our game because we don't have three frames attacks in our game, but I think it illustrates what what I mean. Imagine you're fighting somebody over the net. He's on the other side. And you launch an attack. So, at time zero, your attack is launched, and we compute the frame, you start winding up the attack, your weapon is starting to to travel. The frame continues, we get to the second the the second part of the animation, and you finally hit your opponent. You're happy, you're you're trying to celebrate, and then now comes the fourth frame.

Imagine at that moment that you actually receive an input from your opponent at that moment. I mentioned that the inputs are time stamped, so we see at that moment that at time zero, the opponent decided to change his stance. So, what that means for us in the game is that that input, it needs to be taken into account at time zero, the moment that it happened. And what we do is that we completely invalidate the state in which we were. So, there wasn't there were never any hits, it didn't happen. And always in the same frame that were that that were mentioned, so the fourth frame, we start re-simulating every single step, but now with that new information. So, the information catches on, we we re-simulate, and we end up with the result. Obviously, in that case, it's it's pretty intense, but as I mentioned, we don't have four-frames attacks in our games, but it's really important that aspect because it introduces a fundamental difference in our game between a frame and a simulation step. Because simulation steps happen multiple times

per frame. They're They're directly dependent about your network conditions, when you receive those inputs, what's happening. So, imagine you have your big frame, do the couple of simulations, but as soon as you start having inputs that bring you really far away in time, your frame becomes way too big, the player is sad, the frame rate drops. So, this this was a really big implication for our game. Those rewinds, they happen all the time. We we have a way of prioritizing the inputs of the actor that that's fighting in front of you, so you won't see any rewinds with him, or as least as possible, but the actors that are around him, players that are on the other side of the of the map, they still generate those inputs that are processed more slowly. So, we always have up to three to four resimulation steps per frame. So, we it's our normal operation. And we need to be able to recover and not have a a big impact on frame rate frame rate when we have a really big resimulation. So, in that case for example,

300 milliseconds delay, which would mean 600 milliseconds of ping, it it it incurs up to nine steps of resimulation in a single frame. To be able to do this, every single object in our game has a history buffer. So, we save the states of those actors. It's it's it's in a simplified structure, optimized and compressed as much as possible. And every actor has a definition of that state that's then copied inside a a bigger list. And that history buffer we keep around up to 5 seconds for every single object. So, of course, that takes a lot of memory. And but that enables us really to when we go back in time to resimulate from that time from that moment. I mentioned every single object has that, and it's really the case. You get the idea. Ev- every gameplay element, the explosions in the background, the guys fighting, the heroes, the small soldiers, they all use that system.

It introduces also something really awesome that we did in the game that helped the production that you don't see that you don't really see when you play it. It's the fact that since we keep all those states and the our game is deterministic, we're able to have those kind of features in our editor. Let's say you're fighting, you're testing something, our designer is working in it, and he sees a bug. And he's like, this this this didn't work as expected. Maybe there's something that was wrong. Well, we have the ability to rewind and go back directly to where we where was interesting moment. And we can replay. And all the inputs of the players of the that moment were recorded and we replay them. So, we get to the exact same conclusion than the than the than what we saw before. And all our tools can enable us to inspect what's happening. We can see what's the states of the objects. And I mentioned designers because it's a really powerful tool for them. But even for us programmers,

I mean, since the code is deterministic, we can put a breakpoint in the code and what we're going to see is the exact same values happening. Awesome camera work, by the way. So, everything that's happening it's it's a really powerful tool. It it was a bit of a side effect of that architecture choice, but we wanted to share a bit about it. You probably see me coming. This talk is about the AI. Deterministic AI. So, I the deterministic simulation is where they live, what's happening. And when we started the game, we we like we knew a bit in which direction we were going, but we wanted to give a believable medieval battlefield to the player. We wanted them to be to have a sense of scale, being able to fight other fierce opponents, the other players, but also to be part of something bigger. So, we knew that we wanted to add a lot of factors in our world. We were not aiming to that tens of thousands of factors. I mean, we're not a shooter, so that the

size of our maps was not going to be that big. But around 200 actors was basically our goal at that moment. And with the the architecture that we showed you in the beginning, sending all those inputs over the network, we knew it was not going to cut it. It was going to take way too much bandwidth. So, our solution for that, as we mentioned in the simulation, was rather than computing the input the the inputs that we're sending to actors out from outside the simulation and sending them, we compute them from inside the simulation. Thus, they're deterministic AI. What that means is that when you're playing online, every single peer is is coming to the same conclusion and controls this AI in the exact same way. Everybody's Everybody's synchronized without sending anything over the network. That's a really big implication for us. It was It It It met our requirements. It worked well for that. No network traffic. That's awesome. But it had a really big implication about what did that mean on on the CPU side. Computing all those actors with those rewinds, it was going to come to a cost. So, we needed to find a good

solution for that. So, we have multiple one of them, but the the big ones that enabled us to do that is the first one is time slicing. Those actors, they we wanted them to fill the map so that the players had a sense of scale. They They did not have to be as reactive as possible. So, we cheated a bit and time slicing them and basically not updating them in every single step of the simulation means that if we get to a frame that we need to resimulate multiple simulation steps, well, those actors are only going to be updated once. So, our update time that we came up with was around 3 Hz. So, of course, it's something that we tested through through criteria, through game plays. It We We needed to see what was the the the limit at which we could bring that. But it It did allow us to be extremely resilient to rewinds in that in that case. The second steps are simplified physics. So, this is the world that you see when

you play the game. It looks pretty and all. And this is the world that the players all the the collisions the the the collisions that the players actually interact with. The collision meshes, everybody's probably aware of of those systems. But even that, it proved too much too costly for simple soldiers. So, what we do is we extract the collision edges of of all those meshes and all of that is cached in a localized matter in the map. So, the actors really only test the ground and they test those edges and that's all. It's a very dull life. They don't have They're not aware of of the rest, but they don't care. They're there to die for you. So, we're not too attached for them to them. Third of all, it's awesome, but we needed a solution for navigation. And it came clear at the beginning that nav mesh path finding in our case was going to be problematic. One of the reason was that our libraries were not deter the deterministic from the from the get-go. So, it was going to cause issues

and we had we had concerns about the CPU cost of all of that. So, our solution was to use what we call nav flows. They're basically big roads that level designers and mission designers drawing the maps. Looks a bit like something like that. Obviously, it limits the gameplay space of those actors. They don't go everywhere, but they they really have a defined role in the game. They're fighting over a certain piece of territory. So, level designers draw those those those nav flows. They define which direction the actors should be going, and but they still have parameters that they can tweak to tweak the area on those. So, we started to have something that was interesting. We wanted to see those actors fight and have some meaningful gameplay in that in that section. And our first solution was to start with a top-down approach to controlling where every single of those actors should be going. So, you can imagine the formations, the soldiers are marching together and having a clash. But, our our problem with that is So, on

further, we were extremely gameplay driven. It was our mantra was follow the fun and fell faster and follow the fun, the four F's. And we were playing that and we were not feeling it that it was interesting. We It was It started to be evident to the players that there were some higher rules that were happening that were weird sometimes with the extra exact positioning. And also we we still we were still having the merge issue. It's a big battlefield. There is the players kill those guys extremely extremely fast. So we need to fill it up as as soon as as often as possible. And having those formations, those platoons, and merging them was was proving to be a daunting task. I'm I'm not saying it's not solvable, but in our case it was causing it was starting to be extremely costly. It was not really fun, and it started to be so it was a bit of a mess. So we took a step back. And our solution when we think about it is really simple. It's it's flocking.

And I'm not even kidding our first implementations were pretty much that. They were wandering aimlessly. But the idea was to find simple rules that would govern those actors that would that when they're going to to the center of the battlefield so that it started to make more sense to the player. The rules are simple. It's easy to visualize, and the player started to see that they had an impact on each of those actors. So the complexity of those behaviors they come from the number of rules that you start adding to them. By themselves those rules are not very complicated, but the more you add the more you start having something that's that's interesting. So one of them for example was the distance between actors. When they're traveling to to the battle the distance between themselves we allow them to be closer, but if they fight we ask them to spread a bit more. There's this notion centered around the player that you we have that repulsion bubble that increases and decreases based on his actions. So they start getting being afraid of you. And all of those starting adding up. And up to one of my So when the player

fights those that bubble increases and decreases and the and the guys can the the soldiers can move out of the way. And those rules starting adding up. So for example this one the Moses effect you start moving, they move out they move out of the way. It's always very satisfying. Uh so, all those rules added adding over and over we were like, "Yeah, that's perfect. The system works. Let's give that to designers so that they can start tweaking those values." Well, they were not that happy. Uh so, we we gave those systems and what hap- what was happening? Oops. Yes. So, it's still working. So, what was happening is that the I need I need to find my own path. So, what we wanted was really the the end result, the battlefield that we in which the player was was actually fighting. But, we were getting to that point with very simple rules. So, every time we were changing one, it could affect the other and that's it's that weird moment that our we were creating emergent

gameplay, but our goal was that emergent state. So, it was finding the rules that bring to that state proved to be a lot of trial and error left and right trying to see what happened. And to quote one of the designer I had a chance to work with, that's pretty much that's pretty much what what she was saying. Like, I'm she was scared of touching those values. It was starting to do something weird. So, so, our solution for that was to create presets. In the end, when we think about it, the experience of the player in those lanes in where he's fighting was very predefined. So, we wanted something very compact, claustrophobic, or something more spread out, or something that we call the the scattered one. So, those systems if you look a bit at what how it works, so, we can see a bit of the debug display, all all those forces that are applied to every actor, the guy the the soldiers in the back that join the battle, everything was very systemic. It evolved well. It was

giving the result that we wanted. To go back a bit to the scattered part, we all saw those movies. I mean, those two armies are crashing into each other. Then there's that big camera cut and smoke, and then everybody is mixed up left and right. There's a big a bunch of battles on each side. And every time you were trying to get to that point, well, it wasn't working. The reality is that when you think about it, the first layers just start hitting each other, and then it So, it wasn't it wasn't creating what we really wanted. So, to show the kind of thinking that brought the us to there is that the simple rule that we ended implementing that by itself is not very complicated, but it brings us to the proper result is something like that that we can call the gentleman rule, which is if somebody is fighting an actor, so let's say we have a blue soldier that comes to the battle, and then the other blue soldier sees, "Oh, I'm sorry. You're doing a duel." He lets him fight, and he goes to find a new

target, which is more appropriate, and starts fighting. So, only that simple rule, when generalized, brought us to the to the state that we wanted. So, to give a small example, give something like that. It's It's extremely messy. There's a bunch of guys fighting guys and girls fighting left and right on the battlefield. And but Finding those those small rules was really one of the challenging part because because of the the intricacies and the interdependencies of all of them. So, it's fine. We have those soldiers. They go to the battle. They start fighting, but we still needed to be able to generate those inputs. So, Frederick mentioned at the beginning that we use the same character pipeline for every single of our actors, and it's the same thing for the small soldiers. So, this is really great for animators. They can take the the play the the the the small soldier and start using with them. But it's not really interesting for the

player, or maybe it's DLC character announced. And this is really a a great tool for for animators because they don't rely on setups from the AI programmers or AI designers to test something. They can build AI the the animation, they can look and control the movements, see what's happening. So, they it made them extremely independent from us. Not that they don't like to talk to us, but Uh so, the way that we used to generate those inputs is that we have an implementation of our behavior trees inside the simulation. It's not extremely complex, but it does a good job. So, you can see actually that the behavior trees that we have right now in the game are extremely simple. It's different rhythms of attack depending on the distance of of the target. But, what's interesting is that so we developed the tools that enable designers to actually debug what was happening and and the and and see and control the rhythm and the difficulty. And this can also be applied to any other actor. This is one of the other

actor that we have in the game that uses the deterministic system. We call them captains. The the trees became started to be a bit more complex, but it it's still not like our our bigger fighters, kind of like a mule, but it's it's started to prove interesting. So, our behavior trees by themselves are nodes. They're the one that are actually generating the inputs. When it runs, really the the output of those nodes is it creates a structure that are the inputs the is the character attacking? Is he moving? Is he dodging? And the conditions that the designers can put in place and hooks that enable the actors to to to influence the environment. They're basically events. So, our experience of using behavior trees inside the simulation was a mixed one. In the sense that it was extremely powerful for prototyping. Designers could come with with quick ways of testing gameplay, see what was working. But, we ended up often to once that system was validated on a gameplay

level, we needed to extract those functionalities and then implement them as services. So, we had a we had a lot of analyzing what we wanted to do, strip them out. So, it was a really good way to iterate quickly. But, CPU-wise, it was costing too much. So, we needed to extract the the logic of those and put them in separated nodes. So, we saw the actors, we had a a living battlefield. But, then on the other part, there's still heroes that needed to fight. So, that's Fred's part. Thank you, Xavier. Um so, can you hear me? Yeah. Can you hear me? Yeah, it's okay? Yeah, good. Uh so, yes, uh we have uh with uh the deterministic AI, we have um a battlefield with a lot of character inside it. We you are able to to go and play with the system. But, we wanted to also bring uh challenges to the player. Uh it's a it may it was meant to be a

PvP game. It was meant to have a PvP experience inside it. So, we needed ways to control as well as the heroes, the kind of characters uh the player is able to to play. So, um we uh we tried to create a system to to handle this kind of uh of of of challenge. So, why do we need another AI system for it? First of all, uh we have nav flows in the map for the deterministic AI system, but if the player go out of it, uh we are we can't do anything. So, we needed to have a way to be able to chase the player everywhere in the map. Also, we want to have these cool moments like you can see in the picture where you grab the player on your shoulder and you throw him into spikes. So, we needed to have a a more um uh bit a better understanding of what is your environment, what are what what do you have around your hero. Uh we needed indeed also a better reactivity. Uh, 300 ms to update your AI it's it's a bit slow. And if your player is just in front of you and

you have to react to it, it's it can be problematic. And one important aspect is that we wanted the AI to control the same kind of character. We didn't want it to add uh, because it's a bot uh, to add 10,000 HP so that the player had to just attack him all over again to be able to kill him. So we needed to uh, base the difficulty on the skill and not on the statistic of the character. So if we look back to how do we control the character? Uh, the player for the player we get the pad, we generate inputs, send them into the simulation. So for the AI so for the replicated AI system for the bots, uh, we do uh, the AI will generate inputs. So it will generate the input on one machine and send them over the network. So here we have a an interesting system. It's since we are uh, working with a deterministic simulation. So what does it gives us? Basically it gives us that we don't need to predict the state of the replicated characters

as that of the other players for example because the simulation does it for us. In fact in the simulation if you look at the current state you have in fact the latest updated version of your character. And if you receive an input and a rewind happen it will be updated and and you will be fine. So of course uh, what happen if we have a rewind? The world the state of the world can change. So we discuss with the designer and we decided that we didn't want to consider the rewinds because most of the time a rewind happen but it can be on another place on the battlefield. It doesn't really necessarily affect the fights that you are currently doing. And but if it happens that it affects the the the fights that you are currently doing, um we The only thing that you have to do is just check your the current world state and change the decision of the AI. So, for a certain amount of time, uh when you fight against someone who is

controlled by AI and the machine, you can take wrong decisions, but we decided that it was okay because as soon as you will receive the real state where you are, we will just update and take new decisions for the AI. So, and also when really important aspect is that since we are updated outside of the simulation, we are updated only once per frame. We don't If a rewind happened, it doesn't affect the the replicated AI system. So, we have more CPU than the Sims. Um So, for the system we built um a lot of elements. First one is the team strategy that will analyze the battlefield and try to assign task to to the characters to to spread them and try to win the game. Then, each actor will have to try to accomplish this task and they have a lot of services to fight, to navigate, and kind of stuff. But, in the end, what we want is to generate inputs for Katana to control the character.

So, in this talk we'll mainly focus on the fight service and the inputs because that's really a specific aspect of For Honor. Maybe next year our colleague will want to give you more information about the the other systems. So, if we look at Katana, so the way we control the characters, basically it's a state machine. So, if we want to control the character, what we want really is to control what happened into the state machine. What are What are the state? What are the transitions that we do in this in the What What is the execution of your state machine? But, the main issue we had is that we have 12 different characters. All 12 characters have a unique state machine that defines their gameplay. And also, this this data was was edited by designers. So, they were able to submit and make changes every

day. So, it was really problematic for the AI to be always up to date. So, we decided to automatically extract information from this data. So, we have two aspects that we extract automatically. The first one is standard information. So, what kind of attack are you able to do? Are you able to charge your attacks? Can you find your attack? What are the ranges, the timings, and do you have special properties? Like, for example, as you can see on the pictures, you have on the left an assassin which doesn't have some defense when he is idle. And on the right, you have a tank that has some defense when he is idle. So, to be able to extract this kind of information, we discussed with the character designers and we were able to define standards. And so, it means that they they built all the characters, all these standard information in the characters with the same kind of patterns. So, the

algorithm was we were able to extract and find locate these patterns inside the state machine and extract the information for the AI. But, you have always combos and unique abilities, unique actions that define and give the flavor to your character. So, you can see that, for example, we have an ability which look like this. The context is not really important. It's like more of the the structure. And what we want for the bots is to go to this state. This state represent, for example, the end of the attack. But, how can we do that? Basically, what we want is to be able to extract the path into the state machine. So, we go into the branching between the states. We We look at the condition, and we try to extract what kind of inputs, what are the elements that will allow the character to fulfill the condition and this trigger the branching. So, we do that for all the states until we go to the

root of the ability. So, to be able to do that, it was in fact pretty simple. We needed to find a way to give information to the algorithm to say I want to reach this specific state. So, in fact, it's some data that the character designer will have to put directly inside the data and and that's it. We are able to to locate them and in fact, since they know their characters, they are they are able to define, "Okay, this is the kind of This is a special ability of my character and I want to be sure that the AI is able to do it." Yeah. So, with that, we have all the knowledge of our character. We know how to control it, but we have to get this information and bring it to the AI. So, we decided to create simple elements that will that we call fight actions that are able to perform one specific action. So, for example, we have a

fight action for the block, one for the attack, the dodge, wall breaks, and stuff like that. And we have also one that is able to play input pass. So, the pass that we have extracted with the algorithm. One important aspect of it is that all these blocks, all these fight actions are character agnostic. There is no code that say, "For the Viking Assassin, in this specific condition, you have to do this kind of of stuff." So, they only use the the auto-extracted data. It's way easier way easier to debug. Uh it was really a really a good thing. So, if we take a look at one example, um we'll uh we'll uh just look at the uh uh block of the attack. So, on the left, you have a player. He will attack the uh the bot, and the bot will try to defend himself. So, it's pretty simple, but how do we do that uh in the game? So, first of all, we need to be able uh

to detect that the bot is uh attacked by the player. Then, at run time, we go inside the Katana uh graph of the player, and we're able to detect that, okay, he's attacking, and we know when will be the hit time. After that, we go inside the extracted information for the bot, and we get the defense duration of this character, and the delay to trigger to activate uh this uh defense when we send the input. And the only remaining part is just to be sure that we will have some defense when the hit occur, so that we can block the attack. So, we do that for all these kind of actions. Most of them are really pretty simple, but we are able to to combine them. Um but, yeah. We wanted uh So, now we are able to do all the small elements, um but, we have two need two needs in our game. First of all, is PvP. Uh in PvP, we want to replace players.

We want to have unpredictable fighters that when a player will face an opponent, he will not uh know or he will not recognize pattern in it. But, in PvE, it's different. We want it to be able to teach the game. We want it to the the the to create uh some elements that that will help the player learn the mechanics and try different things and try to find what's is the best option against a special abilities and stuff like that. So, these kind of your character are more pattern based and we also had a few bosses in the game with multiple stages and stuff like that. So, we decided to um Oh, yeah. Our solution to create this kind of factor was to create a kind of database of all the moves, all the the actions the the bot will be able to perform. So, if we look at the editor, as you can see it it looks like Katana, but it's another one. We were able to reuse a lot of of elements. That's why it look

almost the same. But, so we have three main elements. On the right it's what we call the action graph. So, we put all the the fight action blocks together. We're able to create branching between them, set up conditions that allow to trigger to activate this branching. Um on the left you have the decision tree parts. It's where you will create a tree structure that will contain all the action graph. So, all the leaf of the the tree contain an action graph and it in fact is a an action that you're able to perform. And we have conditions, filters, cool down that allow you to define what branches of your tree are available at a specific moment. So, how does it look at run time? Uh basically we do a query inside this database every frame which can be problematic. So, we can optimize. We we did we did some stuff to be able to be sure that it's not a a big deal. The

first the first thing is that we precompute everything. We notice that most of the condition are duplicated. So, instead of computing your information into your condition, what what we did is that we compute the information before and in the condition you have only to just check a value. It's really really really easy. After since we use a tree structure, if the designer organize well the condition and the branches, we are able to have a good early out. And also, we are able to do partial queries. Basically, when the bot is fighting with in when he's performing a specific action, he's performing he's doing a action graph. So, we have we define a priority for it. And we can trim and remove in the query all the elements that are less important in the tree. So, in fact, most of the time you just look

at a few parts, a really really small amount of branches in your tree. So, if we take a look at one example, so on the left you can see the bot editor in winter at runtime. And on the right you have the player in blue and the bot in in orange. And the bot will try to defend himself and attack. And you will be you can see the on the bottom the condition they are highlighted in blue in green and red when the condition are met or not. So, it was really really powerful to help the designer understand what is happening. He's doing a pretty good job. But, so right now we have the designer were able to create all these fighters, but how can we but we have to use them in PvP. So, if

we take a look at one simple example, we have six players, the matchmaking they want to play a game where you have two teams, two team of four player. The matchmaking will balance all the player. So, you know that your team are balanced, but you have one missing guy in each team. So, you need to add a bot in each team. And but what kind of bot do you choose? If you you don't want to introduce an advantage to one team by choosing a fighter which is way to to add and select two bots that are that don't have the same difficulty. And in fact, in the game we created three difficulty levels for from rank one to rank three. So, how can you be sure that your difficulty looks something like this? All the rank one are at the same level approximately and they are less

hard than the rank two and the rank three are better than everyone than everyone else. How can you guarantee that you have your difficulties are almost homogeneous? Uh in fact, we ended up uh wanting to validate the the AI. But we have so two 12 characters, sorry. We have three difficulty levels. It means that we have too many match-up that we want to test. And most of the time when we had meetings with designers, testers, directors and we were fighting against a specific bot and everyone was saying, "Oh yeah, I feel this guy is easy. I feel he's hard, but it was not it was only feeling, not science. So, we decided to implement automatic duels between the bots and we track the results. So, it's very simple. You set up an arena. You put two fighters against each other and as soon as there is one dead, you

you have your results. So, the usage was really really cool. We were able to play more than 250 duels each night for on one PC. And the goal was to reach a predefined win ratio because an assassin, for example, can should win more easily against a tank. So, you don't you don't need to aim to a 50% ratio for all your characters. But, it's not a silver bullet because if you have two bots that are broken, the result doesn't make any sense. You need to be sure that you have one bot at least that represents exactly the level of difficulty that you Yes, that you want for a specific rank. And after that, you are able to compare it with others and and so on. So, it's it's only mainly to detect eye level anomalies, but we were able to to use it

in that in that way. So, that's it for the replicated AI. We'll try to share a few takeaways about all this this process. So, we use the input-driven approach. Was it worth it? Basically, yes. In fact, we were able to share all the player improvements and testing. So, we have on For Honor a team of testers that is fully dedicated to test the player to test the characters. And they they they do that by just playing with the characters. And on the AI side, we were sure that the characters are well tested and well well and in in fact, if we send an input to the character, it will do exactly what we wanted him to do. So, mostly when we had a bug that when a bot wasn't able to perform something, it was on the AI side that the input that was sent was not the good one. Also, as Xavier mentioned, it allows the

user, the animators, the designers to prototype really easily. They were able to spawn an elephant, for example, and just play with him with the pad. So, it was no need of any programmer for that, which is a really good thing because programmers can be a bottleneck sometime. Uh And also, it was really lightweight for the networking. And one interesting thing is that since we use the same pipeline, the AI can't cheat. It's not possible. The AI can only send input like the player. And so, if the player can do something, the AI can do it. And it's in the opposite, if the player can't do something, the AI can't do it. So, for a competitive for a multiplayer game, it's important so that you know that when uh you do you are in PvP, you your AI can't can't cheat. It's It's important. Oh. So, we used also a lot of data-driven, as you as we explained. The behavior trees on

the left on the top right, you have Katana on the bottom right. You have the the bot editor. Was it really worth it? Uh yes. Basically, the designers were really autonomous. They were the owner of the feature of the fighters of the characters of anything. It was really really important for us. And but also and we were able to create a lot of characters. We have so 12 characters at launch in the game. We have I think um 100 of different kind of fighters of AI fighters that can take control of these characters. Um but it's really important to to give them great tools. You You can't just give them like as Xavier mentioned, a list of properties to set. And and that's it. You need to be able to give them good visualization. You need to give them debug tools, a way to

understand what's happening so that they are able to isolate when they have problems and bugs and they are able to isolate, understand what is happening and after that they come to see you and because maybe there is a bug inside it. And then the other side, working with the simulation, I mean it's been 5 years in total on For Honor. What was it? Was it a good idea? Did it work? And it's been a daunting task. We learned a lot, but the advantages of having something that that you don't need to to think basically programmers that are working in the simulation, they don't need to care about the fact that they're on the on the network or not. The simulation is always right. It's going to take care of what's happening. If it if a programmer was working in the constraints of the simulation in a deterministic fashion, well, he it was already working online. So, that was really a great a great thing. We didn't need to be taking into account is is it the real position? Is it is it

an entity that's owned by my system by my by my session or is it somebody else handling with replicas? It's something that we didn't need to to look into. So, that was really great. Of course, that we need to you need to take into account that you need to work in the with deterministic code. So, as it happened a lot storing something outside of the history buffers that we showed in the beginning, well, it goes it goes these things. So, obviously, once once everybody knew the framework in in which we were working it's work it's work pretty well. And I mean it's the whole talk why you are here. Two AI systems. Was it a good thing? Was it a bad thing? And for us really, when we think about it, it's it it it those systems did what we wanted. From the beginning, it's really what what were the tools that we needed to build those systems? What's the behavior that we wanted? And by having the approach of always centering our design decisions or technical decisions on the player well they enabled us to come to that to that point and in the end yes we

think it we think we we made the right calls. It I mean it's it's doing a a great job every systems both systems are different. However, having those two systems it's not an LOD so I soldiers that are going in a nav flow cannot become but and start going somewhere else so there is definitely some limitations. It's some things that we can think about evolving and it's a constraint that was definitely something to to to look into and throughout the all the production. So that covers it that covers it all. Thank you very much for your time. I think we have some time for questions and And also one point just after the the this presentation there's an awesome presentation about modifiers and foreigner that's done by one of our colleague Aurelien Chevalier. So Yes. Good. So I don't know if there's any questions

do we have time? Yes. Yes. Um yes it's not directly AI related but um in the replicated AI how do you decide in a peer-to-peer based game how do you decide who simulates the replicated AI? The way that it works is that there is there is a peer that's owner of one of that AI so there is a it can be one machine or multiple machines that that can send those inputs. The the way that we decide it's since in the the the goal of the bots when they're fighting is really to replace a player we there is no there is no specific metric about choosing who's going to be the the the owner of that bot. I mean it's going to be it's it's there's definitely going to be a difference for player for players that are fighting against that bot. I mean he's going to have the same lag as the host and the owner of that of of that, of that peer, but uh it it's it's the same thing as the player against we are fighting. So, a good choice, I would say, was probably would probably to take

the um choose the teams. So, like basically, if if if on my machine, I'm mostly ping, um I'm handling it, but that's on my on my side. I won't be fighting him. I won't get an advantage. I won't get zero zero lag, zero ping. Uh well, so by using that rule, it's a good way to mitigate that issue. Interesting. Thank you. No problem. Hello. Uh so, you mentioned specifically about AI, but I wanted to know this for the player versus player. You also only send the inputs, or if you also send some state properties on the players or health or anything? The same thing, yeah. It's the same or or only inputs, yeah. Yeah, only the inputs, and everybody gets to the same conclusion, and so so so yeah, as soon as you play on your machine, you take the your decisions instantaneously, so it's deterministic. Um and those inputs, when they're received on the other side, the other machine really syncs up and goes to the same point, but it it's the only information that we send. So, it's closer to a lockstep system than to a proper replay.

Yeah, thanks. No problem. Um two questions. Would you re- uh do the matches between the bots, like if you did balance changes for the heroes? Um and then also, did you only do that with one map, or did you do that with multiple kinds of maps? Uh can you just repeat? I just I didn't No, sorry. Uh the bot versus bot matches, if you rebalance the champions to like maybe affect their range or something like that, would you rerun the rematches between all the bots to get a new ranking? Yeah, yeah. Oh, yeah, of course. Uh at um currently, we are doing iterations and patches and stuff like that, and yeah, if we can we can use the same system to to validate, and we can, if we need, if uh really if we change something really important on on one character, we we can update the AI accordingly to be sure that we are maintaining the same difficulty, yes. Got it. And then for the bot for versus bot matches, were you doing that just on one map or all the maps? Uh currently we use it on one map, but we could use it in any map here.

Basically, we we have one map which was the first map that was created and we use it all the time. So, we did the test in this one and I think we have all the kind of situation that we want in this map like drops. So, you can just throw someone on the from a bridge. We have spikes and stuff like that. So, yeah. Thank you. Thank you. Hi. So, are there properties to the melee combat that make the deterministic and input-driven system that you guys went for more useful as compared to like other game types like a shooter or RTS or something like that? It's a it's a it's a very good question. Honestly, on our side, well, it works well. What we wanted was really to get feedbacks as quick as possible because the your enemy is really in front of your face. So, we wanted to to have the link between those two peers as short as possible. So, that's why we went into into that direction.

If your enemies are further away, I mean I I'm not sure you need such a big system because it has a cost. Like every single of our actors in the world, even somebody that's really far away, well, we if there's a lag if if we get an input that's further away in time, we resimulate resimulate that actor. So, it has a toll on CPU. So, I I don't think it's definitely it's necessarily something I would be using on a on a first-person shooter, for example. Cool. I also had a second question. Did you ever run into butterfly effects when like replaying stuff or were there every like really long latency that like you wouldn't be able to replay? Definitely, at the beginning of the game it was something that something that happened. We call them like the the vortex of death because if if if it takes too much time, then your next input is going to take more time, then the other will simulate longer, and then over and over. So, what we do is like we always ensure that our we have enough time to to catch up. Luckily, we didn't have that issue, but it's definitely something that can happen if you have something that's

applied on, for example, all the actors in the world. And if you do this and you need to resimulate a lot of of simulation steps. It can definitely happen. What we probably do is that we our system is going to if somebody is lagging too much, he's going to get kicked out before things get too bad. So. Cool. Thank you. Yeah, I'm curious about your bot on bot matches. Um don't you get like like a skewed statistic if like a certain type of AI is like just because of the balancing is just by design more effective against another type. Like how do you like compensate for that? I'm just curious like about the metrics. Like won't that like skew them somehow? Uh it's I think it's um we guarantee that at least one character is tweaked and tested by a human so that in fact the other one will only be tested against this one. Of course, we test all the characters with the testers, but we guarantee that some character are really really heavily

tested so that we are sure that it represent exactly what the level of difficulty that we want. So, all characters are supposed to be equally effective against all other characters in that case? Not equally, but that's why if you have an assassin, for example, against tank, it may may have a better win ratio against this one, but we try to just be sure that it's homogeneous. We have a we we allow a certain margin, but Okay. Thank you. Hello. Uh my question is about the rules determining the rules, and I was wondering if the what the process was to determine if a rule was a good one or a bad one and if there was some kind of formula you were using to judge the fitness of a certain rule and if that could possibly be automated. For for the rules that the govern the fighters? Yes, so you were talking about how you set a a rule and then it wasn't quite right and then you'd go and change. I was wondering about that process. Yeah, it was a lot of

so our engine since it's a simulation based engine, we can speed up the process a lot. So there were there were a lot of times where we're able to so let's try those values, let's see what happened, simulate it and some of our designers and people were wondering what the hell are you doing? It's been 10 minutes you're watching the game play on fast forward. Well, they were validating the balance and what are the impacts of what they're doing. So we have a way of really increasing really really fast. So that's one of the way. Other way is a lot of debug display tools, visualizing what's happening with the with the different forces, the different algorithms that we're putting in place and yeah. That's most of it. All right, thanks. Um I was curious on the bot matches if you had completely deterministic AI how the matches just turn out exactly the same every single time. Uh no, the bots are not deterministic. The AI of the bot is not deterministic. It's computed only on one machine and that's why we we replicate the stuff. So But it's but it's it's a good question. I mean our game it's deterministic but they're

still random like every what the way that it works is that every single legend has its seed its own random random generator that's aligned to the the current simulation step, the agent seed and then we're sure that in in this simulation step we can use random that random generator and have varying results. So even if But definitely if if we use the same thing and we send the same inputs, you're going to get the get the exact same result absolutely. So changing the seed of the session and stuff like that is ways that we use to to have a varying variance so that you're not doing exactly the same thing. Okay. Yeah, and and yes, in fact in fact, it's if you send the same inputs but because the for the bots, the system that generates the input is not deterministic, you can just have different matches every time and uh yeah. Thank you. Hello. My question is about what happens when you when you receive an update and you resimulate? Do you have a way to figure out which entities to resimulate or you

just go to resimulate everything? Right now, we resimulate everything like when the the the naive approach maybe. Uh and so yeah, so we redo everything. What we do is that when we see that there's a change that affected an object, so we have that other pattern that we call the observer pattern that's responsible of all the visual effects. For example, blood splatters, effects, sounds. So that when it sees that an an object state has changed, well, it cancels the the effects that were linked to that to that object and so so that we're we come in sync. Uh but yeah, so we're we're we're we're not separating. It's it's for foreigners too. Hello, thank you for the talk. Um so you're so the game is available for different platforms, right? Xbox, PlayStation, Windows. Can you play across those um No. No, we can't. Well, the deterministic engine is one of the problems. So, you cannot you cannot do that.

And but but we have a meta game where you control territories and stuff like that. This is the same across all the platforms but the game no, you can't play on PS4 against someone on Xbox One. No, it's not. Because of determinism. But even for Windows versus Windows, I would think determinism would be a problem if you're if you're platform dependent. Yeah, well, we we're not that platform dependent in the sense that we did have some issues for example at the beginning with Intel versus AMD CPUs we're not always getting the exact same results but it's it's something that uh I personally didn't do it but our engine engineers figured a way to to fix that. But so we probably would be able to fix it for multi-platform inter-platform consoles but it's it's not part of of any plan. Uh it seems that we don't have enough time for more question but we can meet in the wrap room if you want after that. Thank you very much again. Thank you.