AI for Testing: The Development of Bots that Play Battlefield V

[Music] my name is Jonas gilberg I am part of eqe and I spend all of my time building Bots that play games so that humans don't have to today I'll be talking about those bots in the context of Battlefield uh so let's have a look at Battlefield 5 [Music] so that was a short clip of what Battlefield five can look like now why do I spend all of my time building Bots that play games so that humans don't have to

well it provides me with fun and interesting challenges but the reason EA wants me to do this is that we've reached capacity the session sizes of today's multiplayer games already stretch the limits of what is possible for us to continuously stability test using humans especially outside of initial launch games are growing in both size and lifetime so to be able to test things in the future even the near future something that scales better than humans was needed take Battlefield 5 if if you want to test all combinations of maps and modes for 1 hour that requires 2,34 man hours if you want to test that every day that requires 288 people multiply that by the number of Target platforms classes customization and other permutations and it transforms into an enormous number no matter how cleverly you optimize it and this was at launch the number of maps and modes will keep growing other content will be added new features Etc these old things need to be retested along with the new that

is why we need the machines now this is a picture of the dice automation Farm which has over 200 machines we want to put them and other Hardware our disposal to good use which brings us to automated stability play tests this was the goal for the initial proof of concept and is still a core core pillar today for this there are a few requirements they need to run on all client platforms they should exercise code AS simp similarly as possible to a player so we wanted it to be input driven with minimal client impact we want to make sure that we're still testing the same thing the implementation should also be as far removed from the actual game code as possible if we tie the AI down to game implementation details which will shift a lot during development we spend too much time on maintenance as well as being in a worse position to apply the solution to other projects and anyone should be able to create test cases and direct the AI players it should not require an

engineer more specifically this was a collaboration with dice QA and the desire was for their QA automation Specialists to be able to implement the support for all the game modes now before I came to EA I spent my time doing loads of different things Ai and otherwise as technical lead AI programmer on Tom classes The Division I gave a talk on the behavior tree Tech a few years ago some of you might remember this um another thing I worked on then that is particularly relevant now were the Bots for Server load testing as I was hired to do this proof of concept it meant that when this all started I knew nothing of frostbite and very little of Battlefield so the first thing I did was to investigate the tech and if there was anything I could reuse or repurpose unsurprisingly the existing AI solution had been built with a different set of constraints and non-trivial to fork and rebuild for my needs if it had been a reasonable fit I

would definitely have tried to reuse existing Tech the navesh tech itself could be used but as the multiplayer levels do not feature AI they and their assets are not built with AI or navigation in mind there were however a lot of interesting bits and pieces for both input injection and player skip scripting as well as the capability to create fake players on the server without client connections I would just have to figure out how to glue them together with AI on top having a parallel implementation for the test AI is not without benefits as it makes it possible to update it without worrying about breaking the actual game and it neatly takes some of the boxes of the desired requirements but it came with the cost of building building the AI parts from scratch so what about machine learning now some of you might remember this from last year's GDC while I was doing the proof of concept the that was a research project with phds impressive Hardware Etc and I personally wanted to produce something that had the potential to run

on all platforms without significant runtime impact where non non-engineer end users could be in control within months and we would use throughout production where content and the game changes constantly so don't get me wrong I think machine learning is a great tool I just made the Judgment call that it was not the tool for this job at that time um so normally When developing gamei you spend time on functionality fun Fidelity uh you might be tempted to approach this in the same way but the end goal here is not fun uh for most part for the most part no one will even watch these Bots play so we don't have to deal with fun or Fidelity and to be completely honest we don't have to be all that functional either the Bots will de not deliberately exploit each other and if they accidentally do it would only be a problem if it hurts the simulation we don't care if they're having fun they are definitely allowed to cheat if that makes things easier the

main goal here is to get test coverage not a touring test so I started out uh starting out I focused on taking control of the player on a single player test range providing you with inputs produced by the AI so what does it mean to say that the auto player should be controlled using inputs normally when you press the button on your controller that is translated to an abstract input based on the current game context this abstract input is then what is process by player code to produce movement actions etc for a players we don't want to interact directly with the game code nor do we want to press the equivalent of hardware buttons as that would require us to correctly track the current context the input state provides us with a reasonable middle ground this means that almost all of the AI brain the output of the AI brain is produced as inputs such as yaw pitch fire Etc something that's worth noting here is that we chose not to automate things through the UI which means we're missing out on a a whole

class of potential issues now the agents combat behavior is based on the current weapon equipped a simplified weapon representation is generated from game specific data the effective range of the weapon what inputs are needed to fire should aim be used plant and detonate Etc to find hostile targets we do a line of sight test to the closest Target in our view cone it's a single asynchronous rast and we don't do it very often while this might seem overly simplistic it's good enough for our use case and extremely light in terms of performance uh as the AI potentially runs on a maxed out system we want to keep everything as light as possible sometimes it doesn't work we temporarily Blacklist targets if we have line of sight and shoot at a target for some time with no effect so by default the pathf finding runs on the server and in our case we wanted the AI to run on the client so I implemented some custom networking for

paths this meant it could we could keep the path finding on the server and not impact the client uh as the auto players are used throughout production and often on content that is not built for AI the nav mesh is often out of dat or suboptimal as a result paths will often go straight through obstacles and the Bots cannot to tackle this we monitor the progress on the path as a 3D distance to the next Waypoint if we haven't made sufficient progress within the a lot of time we try to clear the obstacle by spamming buttons that often get us out of a Jam jump interact open door and if that doesn't work we teleport that was good enough and what we used for quite a long time now we actually need to control these players uh to do this we wanted to use frostbite schematics not only because it's a powerful visual scripting tool but because it would allow us for easier adoption and potentially a peace wide upgrade process of existing scripting uh QA came up with a list of

requirements of what they would need to write the high level Logic for all game modes based on their previous test and scripting experience so schematics are used to set the objectives for individual Auto players to follow objectives control where the auto players go but objective parameters also control how they get there if they should engage their enemies remain passive have god mode on Etc examples of objectives are in addition to move and defend seen here interact action attack follow and more high level ones like Seek and Destroy you can also create game or game mode specific objectives if that's necessary now that the auto players had basic combat navigation and scripting capabilities we needed to actually transition from single machine development to many in order to actually develop these tests for a large for our for the large scale game modes or did we as I mentioned earlier it was possible to create connectionless players on the

server all I had to do was to make that work for auto players as well and rearchitecturing this local multiplayer player setup then it can be tested in the Real Environment the important thing here is that to the client the server players look like any other clients this is a simplified view of the setup looks like for the conquest game mode Conquest is Battlefield signature 64 player domination mode where each team tries to control predefined areas control points the game mode schematics expose data that the independent test schematic can read and act on when the auto player needs to pick a new

objective they pick a control point at random and then pursue that either aggressively or defensively the defensive ones are guaranteed to actually get to the control Point unless they get killed now we are about seven weeks into the project and this is what it actually look like now it wouldn't be Battlefield without vehicles they really push things so the limit so the auto players needed to support that support for keyboard turning once we had the what we saw you saw previously was easy uh now digging out the weapon information for the for the tanks and actually aiming turned out to be a bit more involved so for the first iteration Auto players driving a tank simply assume that the weapon pointed forward and turn the whole tank

when they wanted to shoot the targets I mean it's not optimal from a gameplay perspective but it exercises the important paths at a fraction of the implementation time airplanes similar scenario following the same bank for bark principle the airplane support simply meant that the othero players would spawn in an airplane get the same objective position as otherwise on the ground and crash there I mean that is what happens to a large number of our players anyway uh one way to expand the autop player capabilities uh Beyond weapons and the primary objectives provided by the game mode is something called secondary objectives secondary objective generators are typically provided by game or game mode specific code and produce an objective based on a specific context it's probably best explained with examples the first example that I'll use is the way that the auto players build fortifications defensive structures in predefined locations the area around the player is scanned for locations where fortification can be built and repaired

if a suitable one is found an action objective is generated the action objective instructs the autop player to equip a specific item tool and use a specific input action fire while all while being close enough to and aiming at a specific position it's also common to set the passive and God mode parameters of the objective to make sure that it really happens the other player doesn't really know what it's doing or why but it doesn't really have to as long as it succeeds often enough you could potentially us run this is a separate taste test case with all auto players running around constructing all possible fortifications on the map you could also periodically generate secondary objectives to blow things up or again all auto players could be configured to have Bazookas with infinite ammo and shoot at all possible destructible objects on the map

yes so here I simply wanted to show you a short more recent clip of what it looks like when the auto players are playing Conquest viewed from above with some debug drawing turned on [Music] visual inspection of the AI from the

ground or above is not a very efficient method to detect patterns over time and most of the time no one will be looking at the automated tests like this is worth repeating the tests need to produce data that you can use to validate them since you can't be sure that human end users will do it for you uh for the validation of Auto players we depend heavily on the Telemetry system used for real players uh a good way to analyze that that data is in the 3D Telemetry viewer in this video you can see the playback of a conquest session on narvik an old version and this is an old version of Auto players um I've also used the Telemetry workf quite a bit uh on my locom machine uh if you look closer at the video you just saw there were a bunch of pink boxes being generated the pink boxes represent a player being killed because they were outside of the allowed Game Area scrubbing through this again with Team callers turned on it's clear that they're chasing enemies into their home areas uh which is something they shouldn't be doing this is a really

early example where I immediately realized that I'd forgotten to do position validation when chasing an enemy I mean correcting that was quick and easy then I simply reran the same 64 player play test locally with local Telemetry and could verify the issue was fixed using the same tool that found it so Telemetry visualization is useful to get for get to get a feel for what they're doing and find anomalies in their behavior the initial validation of Auto players was done by taking data such as Telemetry crashes and a search produced by play tests with human players and comparing that to data from an equivalent play test with auto players we call these mirror tests the vast majority of these tests have been done on PC which affects the numbers you will see later for the mirror test we track overlaps and crashes and asserts but we can also look at the actual session heat maps for the first test we needed to use spare machines during the weekend and turning them those on manually and that's a lot of machines so thank thank you paral lingal and Marian kobas for

making that happen that process has now been largely automated and for each human plst that corresponding Auto PL test should be run on the farm and here you can see the heat maps for two sessions one with the plate test and one with the auto players as you can see there are some differences uh some of the hot zones shift slightly the auto players have a tendency to both follow the exact same paths over and over and they also tend to camp Camp the flags closer in general though this does not show anything alarming such as Auto all auto players being stuck under the terrain clustering only in one part of the map or some other really bad thing that could potentially happen um as the primary purpose of the auto players is to provide test coverage they also try to cover as many player configurations as possible when they spawn the kit that they used is picked at random normally for play test people will pick the default ones uh then the character will get either random or default unlocks we want to make sure we test the default unlocks more but not

only the default unlocks uh on respawn there's a chance to redo this whole procedure these little pie charts show The Kills by weapon as you can see the auto player still use the default weapons to a high degree but has a bigger spread in terms of weapons used in addition to data produced continuously analysts have done a manual Deep dive on the crashes encountered on play tests and autop play tests during 2 months to find detail on what the auto PL tsts might be missing compared to manual PL tsts the left part of this graph the green 49% are crashes from the play test the the auto players have found the yellow 15% are crashes that they haven't found but that are similar enough to crashes that the auto players have found that they are judged likely to be found given enough time the 36% that currently cannot be found have in this graph been further been split by cause since the Auto players don't use the UI the 133% missed

UI crashes are not unexpected the hardware and Os related crashes which come which come down to the farm simply not representing the crashing configuration was slightly higher than I expected 16% again this data is mostly from PC the remaining very rare crashes we might never find even if we in some way manage to cover the other areas now if we remove the uan hardware related crashes which we know the auto flares can't find no matter what they do this is what we're left with for stability testing we're currently trying out a 5050 split of clients controlled by humans and clients controlled by autop players and we will monitor those results closely so to tackle the problem today rather than to wait for the machine learning future we developed a bot system using classical game AI autop players autop players allow anyone to set up fullscale multiplayer test scenarios where AI controls the players they can be run in a full client server

setup or or all of it can be run on a single machine having this system allows us to leverage the hardware at our disposal and not be limited by humans for Battlefield 5 there are 601 game mode and feature test cases it takes 330 client hours to run all these tests on all platforms we have spent more than half a million hours running automated tests for Battlefield 5 or put in another way about 300 work years 178 of these tests contain auto players and those tests take up over 50% of that time um AFK takeover deserves an honorable mention as it's so very useful and very simple to implement what this means is that the AI automatically kicks in if the player has been inactive for a certain amount of time you can watch the center bottom of the screen there it's very useful while working on the AI but it's also can also be used if you want the AI to take over for players that go AFK in a play test even if the BS

haven't been polished for gameplay it might be better than nothing multiplayer client stability testing was the primary goal but now that we had a system it turns out people could do a whole lot more uh the following are just a few of many examples very early in the in the process well before we done any thorough validation of the crft and assert data autop plate tests started being used in addition to the automated flythroughs for performance testing all that had to be done was to run the autop play test test case instead of the flythrough test case and 64 AR players would show up and play the game as it was no longer necessary to run a plate test with human for each configuration numerous configurations were could be explored and the best could then be used on actual PL TST to get more useful data out of those this is also used for something akin to profile guided optimization the game is run and restarted over and over again in many many configurations for about 10 hours hours and data is collected such as

directx12 Shader permutations which can be extracted and fed into the next build uh as we can create server autop players and the game server itself can run inside the client process it's very very easy to test these on your local machine a lot of other tests depend on this ability some QA workflows especially so while the primary P purpose of autop players is to produce data not to look good people can see them and do play against them and as such we've added a few small behaviors and tweaks to make the experience slightly less jarring um you can we can also use those server Auto players to fill up dedicated server game sessions if there aren't enough human players for a full round this is not primarily used for gameplay testing but the mirror test for example depend on this to fill the servers up to capacity as it doesn't run the full 64 clients autop players are also used to play through the single player campaign dice has a long tradition of keeping a scripted playthrough alive the difference now is that it can be

implemented in a way that is more resilient to changes and exercises the content in a more realistic way another way to run these tests is to use them to test the servers but for these configurations you instead want to minimize the impact of the test system on the server this means that all the AI logic such as navigation and objectives Etc instead runs on the game clients clients which run in the cloud this is an example that we never imagined uh Auto players are used to Brute Force the multiplayer maps finding tons of spots where real human players would get stuck in Collision geometry forcing them to suicide and redeploy this was made possible after some improvements to the unstuck Behavior had been made where Ray casts are used by agents to find their way out of tricky situations when they get stuck now this can be done locally and at high speeds as it as it can be run with the server Auto players now these are two examples you might not even be able to see the characters as they blend in quite well at the top the character looks stuck

between two trees in a rock and at the bottom in some debris working with independent hardware vendors and Tech Partners the historical challenge has been that they cannot locally reproduce our stability issues as they require 64 player play tests on Battlefield 1 working with frost bind on dx12 they wanted 100% local Repros for these bugs as and it was impossible as it required lots of players doing crazy things simultaneously for Battlefield 5 Auto players will use extensively to test dxr you could simply run them on a single machine with an RTX card at the office or send the autop play test configuration to Nvidia and they could test things themselves uh another success story from Battlefield 5 includes tracking down an incredibly hardto Find dx12 Crash which took weeks of Auto player soaking and involvement of multiple partners to track down our partners were able to do this with only guidance required from Dice there are a lot more things that I would like to show you today and many more details to share but that's all we

had time for so wrap to wrap this up with auto players we addressed the scaling problem to enable large- scale testing we're ready for the future and ready for testing massive number of players in our game sessions we empowered end users by using frostbite schematics to unleash a lot of hidden testing potential by trying to focus on what was required we could manage with what to me at least was surprisingly simple behaviors and this is just the beginning we're still exploring the possibilities that this affords us now that we have these autonomous clients that can take orders how can we use that to improve workflows are there more ways in which we could generate useful data automatically or will an interactive collaboration with the machines yield the best results during the development of Battlefield 5 autop players have gained momentum and have been expanded to multiple projects and Studios across EA we have access to vast quantities of data and we need that need to leverage that better models better feedback loops better everything this work has already

been started and there are some interesting there is some interesting progress but um and but there is a lot more that can be done and I can hopefully speak on that topic at a future date now I knowbody short on time but as some of you might have realized the first time I showed it the intro video was created using footage of autop players playing the game not humans so let's have a look at that again [Music] [Music] thank thank you