Assassin's Creed Origins: Monitoring and Validation of World Design Data

[Music] hello everyone thanks for coming I'm super happy to be here today and I'm very happy that you are all here because they also would be a bit weird for me to be here right now before we start a shocking announcement I need to request that you silence your cellphone's because you don't want to be that guy and I need to remind you to fill in the survey that you'll receive at the end of this talk now that this is done and out of the way it's my pleasure to present to you today assassin's creed origin monitoring and validation of World Design data but this talk could also have been called how I got to listen to more music while developing this game I'm Nicola woodsy I'm a design technical director at Ubisoft Montreal I've been there for 15 years now I've worked on

several brands including Far Cry and Prince of Persia but for the last seven years I've been than the Assassin's Creed brand where I ship Assassin's Creed revelation ac for Black Flag and lately Assassin's Creed origins which I'll refer from now on as a CE o so as a technical director in Montreal I get to work with the content creators the world designers the level artists and with the tools engine and gameplay programmers to create the world creation pipeline that we'll use to create our game world I'm also the person that programmers and content creators will come to see when something breaks so you can imagine that I don't get much time to listen to music because I'm always being interrupted throughout the day but enough about me why are we here today well we're here because inicio we were faced with a problem or should I say a challenge or an opportunity this

problem was created by the world that were trying to make the scope of the game that trying to make and all the data that goes into that world to create this game how do we test all of that data how do we make sure that everything is fine and tested so we're here today because I want to share with you a solution that the ACO team it's not only me there's a lot of people involved in this develop to tackle this challenge I think there is something in that you can bring back to your production no matter the size of the game or the type of game that you're making we'll then talk about the impact that this solution add on the production target of our game so as I said the problem starts with the game that we're trying to make so who here played a CEO cool thank you so for those of you who don't

know what a CEO is it's a massive open world game set in ancient Egypt where the player can explore tombs and pyramids and the full country of Egypt from the tombs of Giza to the beautiful lush Nile Delta region with its farm and its fields and it's dangerous animals to the beautiful cities of Alexandria and Memphis all of this world is filled with persistent a new persistent AI system so as you can imagine developing a game like this requires the work of hundreds of developers hundreds of developers that create an insane amount of data every day and they make this data using new tech that we developed during the production of this game so as you can imagine developing game data and new tech at the same time brings its own challenge we also use procedural to create much of the game world so because

of all these moving parts a lot of stuff can break often and as the problem we're trying to solve we need to know when things break so before to dig into the solution that we've develop I'll explain how we build things and how we break them so here is a small village from Egypt with tons of debug data being drawn let's build this so first what we would do is well designer would come in and start placing buildings and it would place ladders so that the NPCs can go up on those buildings and the player as well would then place a trajectory all around the location to tell the procedural system not to place any trees inside this location we will then create another trajectory to tell the system to create a road going through this village and then it will continue placing gameplay ingredients like zipline so that the player can move freely through the environment and an artist will come in start placing

vegetation trees bushes and then we'll place a stacks another gameplay ingredient light sources collectibles and then we'll place all the necessary data for the AI to use this village so at the end it's a big mess so how can we break these things I'll give three examples first example it's about the zipline so the zipline is a fairly simple ingredient the player jumps on the rope slides down the rope ends up at the target position so let's break this it's the same setup from a side view you can see the rope between the two posts so let's suppose that a level designer or a level artist or procedural system places a tree in the middle of the rope right here well now we have a problem because the player won't be able to reach the intended destination so that's

a problem this is broken what if for a level design Direction reason or art Direction reason we want to make this feel more epic or we want to allow the player to see across the river well by doing this the angle of the rope is too high now so the animation system the player behavior won't be as fluid will be glitches it will impact the player experience we don't want that that's a problem the next example is about our meta AI system the persistent AI system I talked about at the beginning the meta AI system is a system that tells every NPC of the game what it should be doing right now so every NPC has its own schedule when does it wake wake wake up go to work eat socialize and go back to sleep every NPC has its own schedule so guards as a

schedule they wake up they go patrol they go back to eat they go to sleep farmers will go work the farmland and sand you get the idea so when the meter AI system tells an NPC what it should be doing it will also tell it where it should be performing that action and the where information is in the AI network the air network is a high level representation of the game world it's made by an by level designers inside the game engine it's a collection of note of images that are linked around the world and it this is what allows us to have NPCs travel throughout the world even though the player is not there because this is always in memory so once an NPC is near the player you will still use their network to move around so targeting nodes but it will use the navmesh to move from node to node so that's a very high level view and super fast overview of this system if you want to know more you're in luck because this

is GDC and mr. shallow five over here we'll be talking about this whole system tomorrow at 10:10 a.m. so tonight no parties you have a class to attend so we will explain the whole committee III system so let's take a look at that AI network this is a rampart and a fort in Egypt so you can see here the nodes and edges highlighted in yellow so if we suppose there's a guard that can spawn inside of this rampart suppose it wants to reach one of the pink nodes up in the rampart so one of this potential position how you will yet there it will start by using the AI Network edge that is on the ladder to go up and then you'll be able to reach one of the guard position so if the player comes in the material system will go alright let's fight the player and it

will take one of the guard and say use the ballista to fight the players so the ballista here it's basically a turret that the the NPC's can use so the NPC will be able to Pat fine to these nodes and fight the player off and when night comes the system will tell alright time to go to sleep and the NPC's will be able to use a network to reach their sleeping positions all of these nodes as mentioned needs to be on navmesh so here you can see in green all of our navmesh for this location and so all the nodes are above valid knavish this is what allows the NPC's to pad firing from nodes to notes so let's break this now suppose that the level designer or a level artists for level art dressing pass or to add cover information for the player cover objects have to shield props here right here on this rampart well when they add these things and we

generate navmesh what might happen is that an AI network node might be off navmesh so this is pretty bad because if an NPC tries to reach that now you'll never be able to because there's no nav mesh leading to that node so in this case who broke the system it's whoever pray place the shield problem so it could be a designer or an artist but maybe the artist or designer that did plays that shield it there did this work correctly and they did generate navmesh and made sure that the node was still above knavish but perhaps a prop modeler in another studio receive the bug about this shield prop saying that there is a navigation problem for the player and a certain when you collide with with the in a certain way so he fixes the collision checks it in make sure it's

fine it works well it doesn't create any problem for dislocation he's been assigned to but this shield is being used everywhere throughout the game so now next time we generate navmesh we have the same problem so who broke it this time it's a prop muddler by fixing a bug now another way we can break this is let's suppose an engine programmer changes or updates the fixes fix a bug and the navmesh generation system and we end up with no more navmesh and every slope of the game or every stare flag a certain way well now we have no pad fine no valid path on the navmesh between those notes so once again no one will be able to lose use these ballestas and no one will be able to go to sleep so who broke up this time it's an engine programmer the last example is about our procedural placement to replace our asset we use Houdini Engine and what we

we use it for was placing vegetation creating roads and placing cliffs the next example will be about vegetation and roots so as we said earlier to create roads the artist or designer can simply place a spline through the world it will create a road and we'll make sure that vegetation is placed next to the road and this resultant beautiful navmesh that the NPCs can freely use but as we develop the system it would happen sometime that trees would end up in the wrong location so this is pretty bad because of course then we generate navmesh and we have all's on the inner damage because of the tree trunks and so it's bad for AI network because no navmesh is not good for AI network and so no one can use this road now no one can reach their location so once again this is broken who broke it this time it could be the tech artist that tweak the

Kootenay rule to place trees but it could also be a code error maybe it's a build machine that the in the right order all the data that was required to process and build the procedural content so it ended up having trees on the road and this sometime also happened with rocks and so in this case this would be very broken so to recap who breaks things well it can be content creators as they create their location initially or as they iterate on them or fix bugs can be programmer and it can be the personal system so the answer to who break things is everyone all the time for a lot of different reasons so this is why we need a system to track all of that we don't want to every day run into these problems we want something to find these problems so usually how would we

go around to solve this problem well we would give everyone a checklist that they need to run through every time they do tax task X verify this this and that and once you're happy we're done with the checklist which I get in and then we rely on human testing we tell the designers the artists and QA people to test everything in the location make sure everything is finally every quest that can be in that location and make sure that the they work and once we know that everything works we'll lock the data and make sure that no one touches it again because we know it's fine and we'll ship it problem is this is the village we saw earlier it's just one of the 500 location we have in the game is this tiny dot over here and these are all the locations that we have in the game to give you an idea of the size of

this world this is London from Assassin's Creed syndicate so it's about 12 square kilometer was quite a large city but it's just 10% of assassin's creed origins egypt is over a hundred and twenty square kilometers so because of the scale and the complexity of our data it's very challenging to have complete coverage using her traditional testing methods so we add two mega methods evolve we needed a solution to help us tackle this huge problem so let's talk about that so in a nutshell its daily automated testing of world designed data keywords being daily and automated so every day we have a set of input that are being fed into a day data analysis

system which perform an array of tests and then we output all the information necessary to understand find and fix these errors it sounds very simple and it is conceptually but for us it was a game changer so let's talk about these inputs first they're the world data this includes all the and plays data so that's the buildings and the trees that the artist and designer place in the game that the chests that the designer place in so that player can loot stuff the ladders all the interaction items that the game features we then have the procedural content which is being built by build machine and checked it into the data and the navmesh as well which is also being built by build machine and checked in to the world data we don't have code it's weird to say that code is

an input but basically the point is that we always need to test with the latest code because what if gameplay code changes what if for animation reason a gameplay programmer needs to change the angle threshold that we can tolerate this is a point to be at well it could be that because of that new animation we need to lower the threshold and so zip lines that were placed in the world and that were valid are now invalid because of that new code if that happens we need to find out and once again if we change the engine curve if we change the way NBC spat fine through the world well now we could also break something that used to work an NP scene and was able to reach a location can no longer reach a location because of a code change so we need to know what this happens lastly we have our game design intentions and the key word here is intention basically it's an excel file listing all the location of the

game and for every location we say what is the type of the location what should the player be able to do there can the player fast travel to this location what's the type of loop that we can find so on this is the the level design director fills in this information and this is what you expect the designers to put in every location that's it for our inputs when we have all of that data we feed it in our data analysis system this is simply a collection of automated tests that run on all that data but before talking about these tests and giving you examples about them I want to talk about how we write those tests because it's really a cornerstone of our entire effort and that's animals go animal script is a feature that we have in our game editor tools which are just called anvil it's basically c-sharp script that run

inside the game editor and using these scripts we can query every piece of data in the game I can access a chest object and know what is the loot inside the chest what is the mesh of the chest where is it in the world what is it's recurrent rotation and so on and I also have access to all editor features and most game engine features so we can do raycast to query the collision information of the world we can do shape cast we can do Pat fine test on the AI network to know to make sure that people can reach their locations and this is some code because it's cool to have a code slide so here we're simply going through a list of location looking for for stuff we'll talk about that later on these novel scripts were written by TDS like myself and my colleagues so we have three broad categories of tests and these includes placement metric test AI

network tests and location tests for every test we have a checklist basically for every type of this and these tests are objective tests this is our true/false pass or fail zero one every item every object has its own list of tests and guess what computer are really good at this type of test so we decided to use them so let's take a few examples first about placement matrix so this is our friend zip line from earlier we have to make sure that nothing is in the way of the player right so what we do is we make a big shape gas underneath the rope and we make sure there is nothing place there underneath the rope that would block the player like it three so if we do find a three we don't want that that's a bug we report an error like I said we need to validate that the end goal of the Rope is within the accepted threshold so here it's fine

there it's bad super simple for a computer to do from just compare two points and you get the angle and you report an error if there is one in this case there is one so we do this type of test on variety of gameplay ingredients like navigation ingredients interaction ingredient and spoilers the type of test that we would do on on the chest is make sure that it's oriented in such a way that the player will be able to loop the object for spawners we'll make sure that there is a nav mesh if it's a boat spawner we'll make sure that it's over water this type of thing in total we have 22 object types that we test and once again every object has its own less list of tests so some of you right now might be thinking there's nothing new in this like we are we've been building world's before and we've been successful so why not just

you know use the same things as we did before just use more testers well let's do some math there is 228 zip lines in our game let's suppose that it takes two minutes for a tester to get to the location load the location first get to it test it a bunch of times and if he finds bug report the bugs well this would take one tester a full day of work so eight hours just to go through all the zip lines and that's once only one time now the kicker is that from the previous slides I talked there is 22 object types we test these result these ends up being over 200,000 instances of ingredients throughout the world so once again if it's two minutes to test one instance then it would take 20 tester

two full months to test everything once so that's a lot of people to test simple things so forget it it does not scale we cannot rely on human testing we need we need automated tests second example is about the AI network so as we said we need we every AI network node needs to be over valid knavish it's very easy to know if they are over a mesh we simply go through all the nodes and we check underneath them castle rate down 50 centimeters and make sure there is a nav mesh underneath the nodes so here there are two nodes are not over a nav mesh so that's two errors that we report we also need by definition every nodes are linked by an edge there needs to be a valid path on

the nav mesh linking the two nodes so in this case we have four edges that are not using that do not have a valid path to reach node a to node B so here we have a problem for errors and we need to report but also as we said every NPC has a schedule and every schedule every action and the schedule is associated with a station that the designer places in the world so where they should be eating sleeping working so we need to make sure that when we spawn an NPC you will be able to reach all these potential station that the schedule will assign to so what we do is we take spawner we look at the type of unit that is in there if it's a guard we go through that schedule and we make sure that every potential station you would like to use is reachable from a spawner so here in this case if the player shows up and we need to tell the NPC to go use the ballista we will tell

them to go to that ballista the problem is there's no nav mesh in the stairs the NPC won't be able to reach that so that's a bug that we need to report it's super easy for a computer to do this test it's super fast and you can do it every day but for a tester it's not super easy to notice that when I did fight NPC's in this location no one used the ballista and yesterday they were using it if your tester are able to give you that information every day keep them they're really good all right and last example was about location tests so remember the excel file we talked about with all the intentions well this is where we use it this is a bandit camp in the desert so in red around the bandit camp you see a trajectory that we call the footprint using this footprint we know that everything that is placed inside belongs to that location so everything here belongs to that bandit

camp so if our world designer places a chest inside that tent inside the footprint then we know that this chest belongs to this location so we can then cross-reference the information of the the loop type that is inside the chest inside the game with what we wanted from the the design intention document and by doing that we can see that here we found we were expecting to find one high value item but we found zero and we were expecting to find zero rare value items but we found one so here the designers simply use the wrong loop type and we can just go in and change it so these were just a few examples of the tests that we do run on our data we have much more tests than that but once we have all these tests and gather all of this this information what do we do with it well we need to communicate it to people so that we can act on it so we do that to our outputs which are Excel files emails and Atlas

let's take a look at the Excel first each test that is done writes its result into an excel file and the excel file contains all the information needed to understand find and fix the errors so here we can see the excel file generated while we are going through all the zip lines so here in the columns you can see the name of the EES applying its unique ID is it valid or not and if it's not valid we have the error written in plain English saying what's wrong and lastly we have a link that if click will take you exactly to the location of the error in our game engine so this is an example it's quite simple it's a simplification of our Excel files each test writes its own excel file but to give you an idea the location test the output to locate an excel file that is over 50 columns

because we do a lot of different tests so once we've generated our excel files we can then generate an email and in our email we create a summary of the information that we found during our tests we send that email to everyone in the world team so that's every world designer every level artist every managers technical directors like me and QC members here's a snapshot of one of the first email that we we sent using the system so as you can see it's in 2016 there's only four object types that were testing you can see the total amount of instances we found in the game world total amount of valid objects and total amount of invalid game objects this is the summary email this is not the excel file so we have a link that will take you to the excel file if you want more information so as you can see we have a

thousand letters and 75% of them have some type of problem with it so that's not good that's another snapshot from later in production five months to store shelves we have now 18 type of object that we test we did several improvement to the system but something that is very very cool and turned out to be really useful is that we have the Delta with the previous error with the previous error account from the day before so we can see in green which is good that we're fixing errors getting rid of invalid ingredients and you can see in red the invalid ones so that's bad we don't want red and big columns at the N is we split up Egypt in five regions with five sub teams each with their own managers and teams of designers and we would assign every error to the region's so that each individual team can track how good they're doing or bad you'll notice that

we are now have 1300 ladders with only seven left to fix so your team last output we wanted something to visualize or error because so far we're generating Excel files and emails that's a lot of boring text and not everyone will managers enjoy text and Excel files but most developers or more of the visual type so we wanted something that we can see visually what's going on so to do so we output it our test to something that we call Atlas atlas is a nin tool and in house - sorry develop at Ubisoft Montreal it's basically Google Maps for our game world so with it we can and the game map zoom in and look at the data that we have there the cool thing it's also where the house sorry web apps web these developed in-house the cool thing is that through endless script we

can write to 802 at us so when we find an error we can create an icon and put it on the map at the exact position that we found the error app so I'll show a quick work quick video about Alice in action so you can see them at the game world map and on the left side there's a bunch of layers every layer is a game play ingredient test if we turn that on them on and we'll see the content of these layers so you can see a few red dots and if you do click on the dots we have the same information that we have in the excel file so we have the link to the to the game editor so if you click that link it will open up an Ville there are game editor and it will take the camera and put it exactly where the error is so it makes it really simple and fast for content developers to find their bugs and to get to fix them so it makes things way faster and way easier

for everyone it's almost too easy and like this is pretty damn cool you're saving a lot of time and people were really enjoying this feature that was our inputs if we recap we have inputs which are world well data code and design intention that we then feed into our data analysis system and from there we create a bunch of outputs to excel email and Atlas and this is done every single night on a dedicated build machine and in between we fix the errors by going in and fixing the data or code and then we improve our tests make sure that we output more information so that people can fix their problem more easily we also add more tests because like we saw we started with 4 gameplay ingredients and we ended up with 22 so we keep improving the test based on how the production goes so if you can be a few stats about the

system this system runs on one build machine and it takes three hours to execute most of this time is spent loading the game world actually running the tests are super fast we generate twelve reports from there which are in our three broad categories of tests and we test 500 over 500 location making over a hundred and fifty thousand pad fine tests testing over two hundred thousand gameplay ingredient testing over a million AI network nodes and edges and five million procedural asset being placed in the world and that's every single day automatically and that's it that is it that's our solution it's simple in concept it's simple and execution but for us it was a huge game

changer and it made a huge difference in the shipping of our game and we will then talk about the impacts then why why was it such a big impact on the production of our game well first water it has big impact because a system like this has several benefits that are not all apparent on the surface but because it's simple you don't see them right away but once you start using it it's it's really a big difference so it's been very useful for the quality of our game but it also greatly improve the quality of our life as we develop this game so what are these benefits well the first one which is quite cool is that we ship this game with a relatively low number of bugs and detested systems and let's be clear we

would never been able to ship this game with the at the same scope same size same quality in the same timeframe if it wasn't for this system I'll then talk about how this system changed that we reproduced the game before and now how it gave us quantifiable data and how it led to a smaller de bug so regression every game I made there was always regression so at some point well every week it's not every day so we need to know about the regressions so basically what would happen before all right a triggers will go like this I would get into work coffee and sit down at my desk put on my headphones start listening to some music and then 10 seconds into a track someone would come to me bump on my shoulder and go hey Nick ladders are broken in my village I don't know why all right

have you read the documentation say yes I did I did read the manual really you did that awesome thanks man so I'll stand up and go to his desk start finding me errors I cannot find the errors so I'm starting to wonder what's going on but luckily there's another level designers she's standing by she says hey I have the same problem in my location can you can you help me out please and so who okay nice it's not an error only in this location it's everywhere so that's math so now ok we need to figure out what's going on so I would ask you see have you seen that anywhere please go check all errors everywhere and then I would need to start invocate and investigating looking at if maybe programmers that changes in the latter so the the the Claire navigation ability code I would look at the original ladder prefab that were placing in the

world was did someone change something yesterday that could have caused this so basically I discovered the rig at some point during the day usually very early and I don't know the scope of the error until I dig and find the the error but now what happened is I get to work with my coffee sit down at my desk put on my headphones put on some music and then I go through my emails because I have a daily report by email so then I can look see gameplay ingredients ladders plus 500 and read that's not good so right away what I do is I write an email to my world design team and I say all right there is a problem with ladders we know about it we'll investigate don't come see me so I know the scope and so I can take the right decisions and talk to the right people from my

desk listening to music so basically I'm proactive I'm not reacting to what's going on I I take the action to fix the problems because I know about the problems the system gave us more iteration time so before when we created the we integrated data in the world once we integrated all the data that need that was required for one feature we would ask you C to do a pass and test that everything is fine we'd give them a checklist please go through all that make sure everything is okay and try to break it see see see if you can break it and once we they report the bugs and we fix them and then it's solid we lock the data because we've been through this pass we fix everything you lock it down we don't want to break it again we'll ship that we know it works but now with daily automated test we're a bit more confident to change things because every day we have tests being

executed we don't have a fixed number of QC pass so we have more iteration time for content creators because we allow them to change things much later into the production I think you'll agree that more iteration leads to more quality so that's why I'm sure that this is helped this game have a good quality level and lastly finding bugs the system gave us a new ways to find bug and I'll illustrate this through the wall torch example wall torch is a simple ingredient it's just the torch that is unlit it's placed up on a wall and the player can go next to it with his own pocket torch and light the wall torch so one day late into production very late late in to debug actually should I say we receive one book by one tester and

one location saying that one wall torch instance did not work the player couldn't reach it it was just a trick just bit too high so the player felt like he could reach it but he couldn't so mad bad it's a false call we don't want that in our game so all right we need to fix it but the first thing I think when I receive a bug like this is this isn't that the only wall torch in the game there's a few of them as you can imagine so we take a look there's about 700 of them throughout the game so usually what we would do is ask you see all right cool we think this is everywhere go everywhere look for all wall torches and make sure that they're fine so once again we would do a QC pass and we would do probably only do it once because this is long to do but now what we do is we write the test we cast array down from the wall torch and we try to find a ground into a certain threshold

and this is the trace threshold that we know that the player can light these torches so if the wall torch is in the red zone we report an error and now until the end of the project its daily and automated so until the end of the project we'll know if someone adds a new wall torch and it's broken and we'll know if we have a regression about the wall torch we don't need a QC pass anymore for this the system gave us quantifiable data and every manager in the room will appreciate that because they know that what gets quantified gets managed so as a TD it's super cool for me because I can know the numbers of errors that are in the game for feature XYZ so I can go see the managers and tell them you get 500 error of this we know it's a data error we've we know how to fix it we

trained the people it needs to be done or it can also tell the managers that or a manager will tell me that please train people on this feature because everyone is having problem as you can see so it gives us the information to take the valid decisions and it also tells us there are current velocity are we fixing enough of these bugs to reach our target of zero bugs by date X it also led to a smaller bug because typically in game production most of the bugs come at the end of the production so during the bug this is where we find out about all the bugs so this is where we crunch to fix all the bugs but that's not so fun but since our system was developed during production we found all of these error during production so we were able to allow time in our schedule to fix these bugs and we were able to fix them upfront so basically we front-loaded the the the debug phase and this allowed the

content creators and the QC team to focus on three key bugs and focus on putting quality into the game rather than going through everything to find the three in the middle of the zip line so lesson learned we I would like to have this earlier in production next time around well I would like to have this on every production that I work on in the future because I know that a system like this would have changed the way our production would have been done in the past like I'm black flag I know we will have loved to have that it will also help us not to find lots of bug at the end because throughout the game we can know that we can find these air as we did love them there's a big dangers of false positive though so as Dee Dee's it's super important that we make sure that our tests are accurate that we don't report false information because if we do so the people on the

receiving end they see errors and then they go in the data and they don't find any errors so we're wasting their time and so the last they lose faith into the the system and they'll never use it again and we've built all this for nothing so we have to make sure that everything is fine so it did happen and what we learned from this is that we have to test and use someone as a guinea pig to test the test and make sure that the test is accurate and lastly we wrote way too many emails like I said every report sent its own email and we had 12 of them so that's 12 email a day for everyone and the world team so that's that's a bit spammy but as we develop this we thought done is better than perfect let's just get this out so that people can start using it this is where the return is so we just put it out there maybe in the future we should only make one email with all information or

I'll put it to a dashboard somewhere so the next step for us would be to add this velocity chart well we said that we had a delta for a few of our errors reports we didn't have it on every reports but it was super useful so we would like to have like a velocity chart on every of these report so that we can see if we're doing well what's the trend are we fixing our things correctly or not a lot of you might be thinking right now why in hell are they not preventing these errors from being created it's a very valid question we actually add a few slide about that but for time reasons they're gone even like a recreation is super important and we we have a few of these system in place but even if you have them errors will show up because like we we said like data that was fine on day one might be broken on day ten for a bunch of other reasons

so even if you have a gate had submit and make sure at save or addition to make sure that everything is fine stuff will go through that will be get broken later by another change so we still need to have a snapshot and test everything and lastly create more tests we are testing three broad categories of objects of features we have way more features than that in the game so I'm sure we'll be able to expand that and take it to other features and I'm sure that you right now in your head hopefully you are thinking of stuff in your game your own games that you might want to test with something similar to that lastly I need to say thanks and mention my colleagues well built this with me so I need to say thank you to Kevin Coughlin who's right here who built this and actually Kevin co-wrote this presentation with me so thank you

Kevin Kevin will be joining me on stage for the Q&A sessions so please have one question at least so that Kevin has not can't come all the way here for nothing and then I need to mention a lolly vikon in me to my good day who wrote a lot of the tests that we've talked about today so my colleagues and I have been in industry for a while and between all of us we've she shipped a lot of open-world games we're proud that on a CEO our designers were able to focus on the right things and work on the quality of the game never we've been so confident that the system that we tested at least were bug free and that the player won't find any problems with these systems in the game we know that the the the our system of daily automated test was a major contributor to that and it was a huge contributor to the quality of the game and the quality of the life of the people building that game so I invite

you and your teams to think about how you could integrate a similar system into your pipeline your game and your team will thank you we have 12 minutes for questions so I really awesome tech thank you for this talk it's awesome I had a question what would you do if you couldn't run the scripts as it can easily it's right on in the editor what would like what if you had like to go through quarters C++ every single time you wanted to add a test what what would be your approach there well so I guess the question you're asked is what do you do you don't have to see you see sharp running it if you don't have that animal what do you do god I gotta say that my gut reaction

is you need something like that some of the game teams that we've talked about the that we've talked to you about this at the very least they can do maybe run Python scripts to parse their XML files to extract as much data as they can but really the that easy access to the the attributes of all the data and to be able to run those pathfinding and shaped a test that's really such a key part of this that you you probably want to look at that first Thanks just a few questions did you consider instead of it just being daily like doing some sort of continuous integration of like the tests question so some of these tests they rely on data that is being generated overnight on build machines so we have we need that data to run those tests so that's why we

need we decided that we only run the test ones that we generated everything and we have all the necessary data but yes some of those tests we could execute every hour okay because the game is so big and you have all these gameplay ingredients all over it when you're testing you have to interact with the streaming systems like okay we got to stream this part of the world and then test that stuff yeah then we got to stream this part of the world that I'm I may have just said something obvious I don't know if there was a question in there well yeah you have to load everything because when you're doing collision tests you have to make sure that every collision are loaded so we need to make sure that all the collisions are I mean in the in the game world and so that we can find them yeah maybe just something to make clear the the tests that we run were all on static data so you're not actually running the game and having the player go through the world so you are then when you talked about it taking three hours to run the test but most of

it being loading in the game world is because he was loading the whole game world in one shot and then running all the tests and maybe just another thing about the continuous integration question some of the tests like the location test the level designers could run that test before checking in so they were able to go okay I got this error run I think I fixed it run the error suite and poof yep either fixed or not fixed and go from there so you may have just answered the other question I was going to ask so all the tests were what were were for what you could test statically and I was gonna ask a question of as you like progressed through the game's plot or whatever the world state changes and was gonna ask you like oh we have to like test it in this world state tested in this world so it does but it sounds like you're not doing something like that cool thanks hi great talk thank you I just want to ask do you have some way to test some dependency resources let's say

I change wine one feature one content or some ID some code change but actually you know sometimes I think designers or we want to we care more about how many other contents or stuff in the world will be influenced by that do you have any salt or I mean test of using this tool to do that thank you so I'm not sure I understood the question one thing certainly that that a nice and anvil that wasn't some of the other engines I've worked with is the dependencies are very clear there's a dependency viewer that Maps on all the dependencies you know exactly oh and this object I wonder what depends on it boom viewer you see exactly what it depends on but and again because the test that we run it's all on static data we know what the appendices

are when were during our test and we can just go from there I'm not sure that answered yeah part of death yeah thank you and because I mean because you have a lot of content sheared in the world right so actually you change one once you change one stuff let's see a collision mesh of one of a letter but actually it smarter so and you find that I mean you found the end working another please I mean so but actually I want to say oh actually this is the collision mesh calling this error so house you have me but when you will all of the I mean where you let's see this letter is we have five letters of these broken letters in the wall where you get five error report it's on the same in the in the world okay so that's thank you thank you hi I just have one question by the

time you ship this gigantic game how many red dots were left on your map obviously just to be very close to zero I honestly think it was like four or something is and maybe just an extra little thing question you could have asked also is well were there ever any exceptions to the rule right because sometimes we've got these hard and fast rules but there's this one case that you know what you're reporting an error on that instance but it's not really there we had the ability to go to that instance and say this particular test don't run it we know it's good so hi so I wanted to ask since all the objects are in some sort of a database then why didn't you made the QA team able to create their own tests using something like

queries or something on the database using scripts or anything in the real world like to help them decide on what to test and what not to so why we allow the QA team to write their own test is that the question yeah yeah well we we wrote the tests ourselves because we're a bit lucky we were all all the teenies that i mentioned we are super confident writing c-sharp quote so what happened that a tester finds a bug and would come to us and say hey there's a like some this always breaks can you help us out to find it more fast and if make faster or can you write a test for this and we would work with them and we would write the test ourselves some of the tests are also not in code they're actually data so we there was ways for people to tweak that data to get more information

cool well if there's no more questions thanks for staying until [Applause] you