'Final Fantasy VII' Remake: Automating Quality Assurance and the Tools for the Future

[Music] hello my name is fabinho and today i will present final fantasy 7 remake automating quality actions and the tools for the future but before a small warning most of the developments that i would present here have been done at the end of final fantasy 7 remake project and only part of it has been used before the master but the development teams allow us to use their game to continue to develop our tools and maybe use it for another project first i'd like to talk a little bit about the problematic of qa4game and focus on the main part of this talk that is a replay system with an overview of the system what kind of data we use how we synchronize them what kind of workflow we are using and some ways that what that we got i will also talk about uh small extensions that is expressions to

allows again the bot to explore the game and conclude on some future works but first game development is evolving more content more interactions greater freedoms a lot of play styles and also the game is not finished when you release it you have now passwords content dlc patch and so on so this will is to go into a cost and you have to test it again and again what we can do to improve that we can improve reporting tools so you have some graduates talk about that we can feel part of the bug report automatically we can completely automatically report crash and asset for example we can add telemetry to the game so we can gather data and evaluate the quality

of it we can also try to make some in some way we play or play the game automatically so with scripting or with rippling already play data or with explorations but first let's focus on replay and even in replay we can divide it into two big categories things that are done without game change so what you can do to control your input is basically the screen and you can control time you can control the game through the gamepad driver and the good things you can test the final release of your game the one that your user will be you dina

so the bad thing that it will be really hard to support non-deterministic actions and level change will probably need to rewrite part of the script so this technique is generally used at the end of the developments you can also uh try to add autoqa and replay feature on the game engine side that allow you to access the world code but probably you want to restrict you to something that is more near what the user can do and especially in our case we want to register in fact the game state we can also control everything but you want probably to restrict you on the gamepad emulator even if time two times you may want to what's the command maybe you are restricted to the test period but the good thing is that you can now try to spot non-deterministic actions

and you can be robust to some level change see that allow you to start to use this kind of techniques as soon as you have a first vertical slice this is where we will focus our talk but first a small example so i will first record a small gameplay by using my gamepad and we record not only the gamepad input that we are using during the menu but also the game stat that we will use after to absolutely here we will have a tutorial and a game a battle battle are typically something that is not deterministic and just to personalize my replay i go for one and go back okay let's stop it for now and replay it

so as i said we record not only game input but also game stats so once the game screen open we can push the button there is no additional wait time basically and here we have the tutorials keep it and now we use a battle bot to control the game so we will have different battle on the battlebot but can even test all the command randomly to be sure that we didn't break anything in the battle system when we go back to where we play and now we can go forward and back as i have done in my recording data and i finish by a replay success that the game has detected so what is the step we will have this kind of ui the launcher but we can also have command line for uh jenkins that will start

uh two process one is the autoq server that we call it that will be used to supervise and control the game the second one is a watchdog that will be platform dependent and can start to unmonitor the game the autoq server will ask the watchdog to start the game so again the watchdog will really start the process the washer can also be responsible for recording videos and so on screenshot and detecting garden on the game start you will try to ping the watched watchdog to be sure that we hang or not and the game will also send a generic game state that the auto q server will use to control the game by giving some requests

we will have also some ui command like stop and so on that we can send to the qr server inside the game in fact we can split into two parts so to qa one that will be in fact a game engine specific that will be in charge of communication some default behavior and another part that will be specific to this game we will have this kind of bot and we will say what are the game stats basically so just like take a look at this game specific part so in final fantasy 7 remake we had a few bots so just give me some examples of them in the left side you have some bots where in fact you are managing but usage with replay because the control of your character is completely different from what you are we are seeking first

either you control your robot arm or you are on the ceiling but basically you are using replay to control where to go the middle part is a battle but one of the most complex one that meets in fact both replay under a battle just to be able to be sure that some battle techniques are if you want to climb during a battle or two things during battles can be still replay if needed the last part are very small minigame that generally we implement about in one day just to be able to pass successfully because synchronizations will be a little bit too precise on this part which just we play data for this replay data okay so we have talked about the game specifically now let's talk about the generic game state what kind of data we

are using so generic game stat will be will contain where we are so the positions the level id the orientations and also the velocity of the characters when we are we will have in fact several times because depending of what kind of game statue are you care about different times you may have like we have seen the battle some slowdown that are used to activate the debug menu and so on so the control menu and time will flow in different speed during the game so we are that beauty time will be mostly used for video sequence frame count is generally the default behavior that works basically everywhere after you have some [Music] action and event that you want to know

what you are doing and also you want to know why you are doing it and this is done by knowing the game state and we said not game step but game stick stack why because we cannot replay data and for example we can start a follow mode so if you go too far from the npc that you are following you may have a one in and make you go back this is one of the game mode and you may have a battle that started while you are still in photo mode and during the battle you can have a tutorial that in terms of the battle and sound so you will have always one step but you have the other ones that are continuing inside the background so this way we have a game state stack and what we will do to identify each text is an id that will be based on the type and the cinematic cities the game generate specific that will specify

those values so game id is generally a fixed id for battle or cinematics maybe each of one of the different names something like that we can still also use this does that will go back when we are doing replay and we will cancel so synchronization flags and timeout overrides that we'll just talk from now on that is used to synchronize in factory plane so we will not explain all the synchronous flags during this presentation but let's take a look at a few of them so let's suppose that you have a game stat and you want to go to it you are replaying your data and now you are supposed to get to the game set but maybe it's not popping up so if you have a non-we look at ball flag it means that you need in fact to be at these positions to be able to

start this game state so what you will do is first set that start to timeout to say okay i i will have given time to to succeed and you will start to to continue to to do some play or motions to be able to be sure at these positions or maybe it can be either motion also maybe wait and if you reach it you can synchronize again and after because you succeed during your start time mode you have a success and you start to end time out and you will you must be sure to finish this replay during this time out before the end of the time let's see another flags

in that case is very similar we go to the system but it didn't start okay but now it's a hello catabol flag so we want in fact to continue on replay because maybe it's not exactly at this position that the game state can start so let's continue and we push a little bit the data but if we push that much at one point we cannot push more so we have also to decay to push everything together not only the start and if we start the synchronizations we can now again compute anti-mode which is based on the original recorded timer not only on the current one and we will try to replace the data so let's go on again and maybe we can go we we can go on the next

part of the replay other example skippable skippable means that when you are replaying you have a new game state that you were not expecting it can be a new cinematics that has been added to the game a new battle i don't know and what the replay system will say about if this one is keepable let's try to skip it and ask a skip request if not we may finally fail this replay [Music] most of our replay detailing fire is capable we can also have star table it means that now we have a [Music] session that the uh game stats that is inside the record editor that if it's not there in size or play we will not want it to go to move we will ask a good quest to start it to the game

this is typically used for uh interactions and so on because they those ones are very essential for the game for flow so we wanted to to be sure that we start all of them at the right time we can also have optional flag that is very simple we try to replay if we get a game set that is find the future after some optional which you skip them so typically optional flies can be tutorials that can happen only the first time for example or all the other games now let's start to talk about the move to

part so we say that we move but in fact we don't use directly navigation mesh in fact we mostly not use navigation mesh we create in fact a 3d map 3d grid map during the replay you will use both the recorded data so or previously predator so we can create a person saying that we are able to connect them but also we will have the current replay we will try to detect wall when we are not moving uh in the desired direction so for example we wanted to go in these directions and we go on the other side probably have a wall this is still probability probabilistica only and if we are lost we can try to do past fanning we will also assume at first that

the going back is possible even if it's not a necessity we will also use simple heuristics to extend the map if we are blocked we can we can even in some cases or first simple objective didn't succeed try to recast to the game a pathfinding that may help us to go out on some situations but the important part is that in fact the player may go to some place where you don't have a navigation mesh we don't assume that navigation mess is ready for us okay so we have talked about game generous text and in fact we can find out that

a lot of things can be shared between different projects so not the game specifically but as long as you have the same game engines even though the qst tools can be shared between projects now let's go to the workflow you will have someone that will try to record a replay like you have seen that will give a database of replay data we will have the game build a system that will trigger war jenkins that will ask a set of pin machines to build the corresponding executable and ask a setup of test machine or consoles to test the game we will update everything to a test with the database to do some test monitoring but we will also send it to a system to do some bug classifications

some of them will be sent directly to the qr team or to the game team because they are assert and crash and we are sure that they are relative to the game and some of them will go to some auto qa specialist so you may have some real prefigures that can be in fact a change in the data that is too strong for us to be able to uh to cope with so we need to record the data it can be also some problem in on workflow so we will try to analyze it and debug it and we will report what we need to do on the other teams so what kind of advantages we got with this system we were able to find where we are bugs wear bugs logics beauty threat something that happens even less than uh

0.3 percent of the times so 7000's of replay to get it we were able to test levels about three three thousand chapter by days basically we were able to test a large large code change so we will get some requests to test some champions and test it maybe for one week to be sure to have a stable wizard we can test the game with also bad frame rate so if you want to have a memory analyzer on the game may be really slow and really painful to play for human player but the system is able to cope with that we were able to perform tests 24 hours a day and even a aging test that were generally about 18 hours for one test by

playing the game in fact several tiny loop the main disadvantages is probably because we test only what we have recorded we don't go outside your path so if you want to do a complete test you need to create a lot of levels so now let's go to look at another example exploration example so this one has been developed on ue4 basically before and we have uh in fact two adjuncts that are playing together the game to try to create a map like i said we are able to create these 3d maps this 3d greens map and the goal now is to find where we were not where we didn't go so here once we have explored part of the room we can see that we try to find another way out and

together we can start to explore in fact the world map so the advantage of expression is that we can find the bugs outside of the normally tested probes that we add we can capture performance so fps memory usage everywhere on the map we can check for collisions the problem is that by trying randomly a lot of things we will not go very far quest and game are too complex for that we'll not be able to clear any game any level so what we have done is mix both so here we have in final fantasy some replays and explorations mesh together so currently we are exploring the map

so as you can see we try to go everywhere and we cannot try to detect where our walls the first time of the expression is random so we go there and there but as more as we are exploring we will start to fill up the map and we will need now to find where are other edges that we didn't explore and in blue you can see some pathfinding that we are doing to try to test other edge case by doing that we will finish by filling up all the map so here we are finishing the last holes that are left and when we fill all the walls we can start to explore new eras that we have not yet explored but sometimes you can have game events that pop up for example here we have a battle what we will do is when we have new game sequence at pop-up we'll synchronize

with the replay data so we'll try to go to use a replay editor of these petals if there are one and we continue the explosions until we time out of it and we will go back to the replay so we are doing again past finance to go back to the end of the replay detail that is after the battle that we have done during our explosion step because we are able to to do it and they are uh real donable so now let's try to go back to the player we have uh some cinematics and so on and we had another exploration there so how we do we do it we generally [Music] re record a replay and if we want to add respirations we look at the movie we synchronize uh we add some replay exploration event inside the play based

on when we are compared to the movies and after we can explore where to explore in fact a different part of the map okay so we go back again to the replay data now so still with pathfinding above the navigation data and now we can find out with the repeater that we need to press this button to continue again one other thing that we can have is we can have telemetry so here is an example of replay data it's a 14 replay that's the same path so each time that we are replaying the data without expressions we are basically have one only path that we follow if we use expressions that will take more time but now we can see that we have a more wider range of views

our sure expression is not necessarily used everywhere in the game but for example if you have a bomb that will explode you probably don't want to explore now you want to go down but so this way we have also this kind of events but it can be used really to increase the range of the part of the game that we can test and uh and get data from with for example this telemetry tool so the advantages is that we were presenting in fact an automation system that avoided a lot of report repetitive testing we were able to show how to share development through several projects by making this this part as more common as possible and trying to have a generic game stat

in which we can store a lot of data even some user data like what kind of objects we want to interact with so in the replay we can use this kind of id to look for these objects and not go to the recorded data but to the new position of this object this allows us to have some margin with a level change we were also showing that we have a navigation system that is outside the game that can allow you also to have something that is less game specific and with mixing expressions you can start to have also support for some variety inside the game level even if you have new obstacles during your own path you may be able to explore once and go

and go and go on these obstacles the problem is that we are not able to do any qualitative create so graphic issue and so on the replay system was perfectly defined to say that the game was working when there was a black screen all over the place which is because he was able to complete the level but we have still some graphical issues the graphical movies and so on that we can export to to help create in fact to make some decisions about that so what we want to do is in fact add more of this extract video on screenshot to in fact for example gather all cinematics that are inside the game

and have a creator that just checks the cinematics doesn't have to run the games several times to gather all the possible cutscene cutscene depending of these different actions and we want to maybe use expressions on other things at positions for example on menu or maybe on minigame and things that i have said that we the expressions may be able to detect collisions but this is some scene something that we have to work on we want also tools to find large change in level collisions okay so if you have any more questions you can always contact me and thank you for listening [Music] you