Session: Effortless Game Testing With Planning Agents - Filip Dvorak, Filuta AI.

[Music] first up we have a session effortless game testing with planning agents so uh please welcome to the stage AI Philip dorak and I'm going to be talking about uh how the planning agents and AI agents can help uh with testing of fors and generally help the industry to save a lot of money make better games make them faster and leave more money to developers and making great games uh first uh it's um pretty important to realize that we are in a shift industrial shift uh in how the QA

is being done uh we can always hear those stories of uh testers that had to test for two years straight something like riding a motorcycle every morning or uh doing regression tests every morning trying to shoot every gun and throw every grenade and do it for several years of active development of a AAA title those are not very fun jobs and uh those uh type of human testings uh and and test approaches to testing are going to disappear at this moment uh we predict that it's going to be about 65% of testing that is done by people at this moment is going to be replaced by uh Ai and it's not because it would be taking jobs from people it's because it's just so much more efficient and more practical for those areas so people can focus on areas uh where uh the subjective creative experience testing matters much more and and uh this is a

fundamental shift in the industry when we imagine that we have some large studio that has uh potentially hundreds of developers uh they produce features they produce functionality they produce content they produce level design and uh it becomes owned by the QA Department the QA Department uh takes the responsibility of the life cycle of those features and uh when we look at those features they can break in time they need to be fixed they need to be validated they need to be in the long term they need to live up to the uh release into the actual game that the consumers can play and uh this QA Department uh what is what we see quite commonly is that uh it is using uh potentially some form of automation which um when it's de developed by internal developers in the company can become more expensive than using human testers or or it is used uh or it is

outsourced to QA departments that are in some areas where the human human testing power is very cheap be it uh China India or Philippines and those testers uh do not provide the reproduction steps and insights that are good enough to be able to uh in a large scale provide the testing capabilities that we need for the games today and uh in this flow uh there is always some Excel sheet we see it quite often that the QA department has some enormously large Google sheet or Excel sheet that contains all the test scenarios they want to work with uh where we come in is that uh 65% of this Excel sheet can be just fed into AI agents that are going to do the testing they are doing going to do it at much scalable way so if you have 20,000 tests that it takes your engineers

several weeks or your uh testers several weeks to process uh the agents can scale up you can spawn thousands of agents and do it in few days this is advantage of scalability uh also uh agents do not require any onboarding once they are integrated into your system you can send any number of tests to them you can choose the scale uh it doesn't matter if they test it in days or if it takes them one week uh you just pay 40 hours and the the agents uh can do it without any onboarding and at any scale you choose and cover 65% of playr functional regression tests that otherwise would be actually pretty boring for people to do so uh this is the situation we are at right now what we see in the industry where about 65% of uh all testing uh the actual uh playthroughs through the game can be outsourced to AI how how much of

it is it at this moment outsourced uh very little it's uh few percent and there are only few companies that are working in this area we are we are one of them what is what is another advantage of uh the AI uh players that uh run through the game they are able to uh record every every change in the world in the game world that has occurred they can um precisely uh control what is happening in the game they can provide you with precise reproduction steps complete recordings audio recordings they can analyze the V they can analyze the videos they have collected this is uh so much more information you can get from the AI agents than you would get from Outsourcing to human testers and uh this is in a nutshell as a opener our product we are providing agents that can do this

for you uh I said there are planning agents uh to be to be more precise these days AI is a buzz vard that has uh many different uh sub branches within that umbrella and uh we are working within area that is called symbolic AI which compared to what is these days called sub symbolic U naral networks language models uh this area is different that it provides precise explainable auditable Solutions so if uh the agent the planning agent uh is going to encounter uh in its massive deployment of dozens of thousands of hours that it has been running through your game if it encounters some failures it is able to logically analyze what is the root cause of this failure and uh based on this analysis provide you with some minimum diagnosis minimal critical set that

tells you that probably the trouble is coming from this area of the code or that this feature is probably bucked because it has appeared it has had several hundreds of manifestations this is uh significantly more difficult to do it if you have agents that are based on language models or uh reinforcement learning or generally any any type of blackbox approach the symbolic AI agents have been motivated and built for space engineering where you need to diagnose and be able to identify any type of bug that has occurred because uh you're in high uh high stakes Mission critical environment so as a main uh I would say how how to identify our agents and uh the advantage of symbolic AI is that uh it is fully explainable an auditable logical system back to back to why the QA is such a

challenge these days so we have seen over previous decades that uh games are uh linearly increasing in its complexity in its size where uh historically you had when you had RPG game you had a character with some inventory running around with some development and uh building some skills hacking and slashing around monsters and uh you usually had only few different systems connected together to create your game uh when I uh speak about systems those are subsystems that are Min games on its own Inventory management is a Min game where you try to fit the best items in the available space and uh you are trying to be efficient about it uh when you are for example combining it with a Min game of crafting crafting is uh some type of supply chain game that can be connected with some probabilities uh probability of dropping certain item

which is close to Lottery system which is uh close to um many many other different uh problems within mathematics that make the game interesting uh but uh those those mini games that historically there only a few in a game have been uh significantly increasing these days in AAA titles even in small games you have monetization system connected to dialogue system connected to Quest system connected to RPG system uh with some physical repres representation of your character uh that is running around with some inventory system and uh this complexity of uh the carian product of multiplying all those different systems uh is creating uh combinator explosion that is not possible and it has not been possible uh to be tested purely by human testers for several decades this is a reality of the combinat

of gaming these days uh we are we are not able to test it with people and uh there has been for some time a very popular idea that early access is going to solve it because uh we just give the games to players very early in the development and they will play it they report the box back to us and everything is fine and we use that power of the crowd uh the trouble is that uh the players became a little bit spoiled they do not want to anymore play games that have some fundamental bugs that prevent them to actually uh have experience from the game so you these days need to test even the Early Access games and uh that means that it doesn't Early Access doesn't solve the testing scalability issues it doesn't solve the comat explosion we need we need some solution that is going to help us to uh enumerative put uh enough testing power into the games uh to save uh those

pressures uh developers and uh people to do uh development and making the games better and testing what matters in the games which is uh how fun it is that something uh no AI can really tell these days that's still up to us so in this situation uh where we have so much to test because the games became so large and uh the number of combinations have exploded the AI agents and our planning agents are something that uh provides incredible value when we put it into comparison with uh metrics from the industry uh working with the keywords report on how expensive it is to maintain a single line of testing code if you have test scripts for your game and uh we have we have customers that have dozens of thousands of regression

tests where each of them has some length on average some end steps you need to do to test certain functionality and uh when you imagine that you have thousands of tests that have that have certain length uh and uh based on the reports each line of those tests costs about $3 to maintain per year you get to enormous amount of costs for maintaining those Solutions those testing Solutions which provide good coverage they are uh good uh software engineering practice large software houses in Google Microsoft and uh these these large software houses it is uh quite common to have enormous amount of uh tests that are constantly running at Gates and testing every comment that is supposed to come into the code base and uh their their numbers their cost per line is much higher we work with

Automotive customers that have uh their one line of testing code is more than $40 uh in finance is more than $90 so those numbers can be uh can can vary based on the based on the industry and how expensive it is to have a failure in your system nevertheless uh it's not necessary you do not need to maintain fixed uh testing scripts if you're agent only requires the goal if you can give your agent uh let's say in a fairy tale game if you can tell the agent to save the princess instead of uh having a test script that is going to navigate it for half an hour gameplay and maintain this test script if you just give the agent what to do and the agent figures It Out by itself this provides enormous amount of saving in a long term you do not need a developer to maintain this code

you uh do not need to even if you were recording it and uh uh then replaying it as a test you do not need to maintain the script and uh this is the saving that uh the AI agents can provide not to mention also um based on our statistics so far we are about 17.5 times more efficient per dollar spans than human testers um little about a little about uh who we are uh we are uh Space Engineers or I would say AI scientists focus on autonomous systems in space u we have historically I have done my disertation uh for uh autonomous systems that control our satellites that control uh Mar Rover perseverance for example and for systems that are operating uh different robots that are running uh on the sea Flor and they are checking the cables and monitoring what is happening

uh at different uh um biomes in the sea uh this is uh area that is absolutely amazing for applications of autonomous systems but the disadvantage is that uh it takes really long time to get some feedback from space applications and uh on the other hand uh we have found out that one very large advantage of fully autonomous systems that have been developed for space applications is that uh they do not require any configuration you can throw them into the game environment you give them the goal and they will adapt uh you can change the game design you can change the configuration you can we we have customers where we integrate and 14 months later we do not need to change any integration all the tests are still running they are constantly adapting uh to the changes and development of the game this is enormous advantage not to having to do any

maintenance well so this is the uh I would say scientific portion of the team with three professors and 14 phds at this moment and uh we have experience uh with uh developers that worked for uh AAA games Tria Studios and that are helping us to build build the system as I have mentioned our systems and our agents do not do not need uh practically any afford from your side uh if we integrate into your game we love to integrate into RPGs uh puzzle games logic games about 80% of the games on the market are perfect fit uh Flappy Birds Angry Birds not not really good games for us because uh the logic there is limited games based on Fringe engines are uh less uh less uh comfortable to be modeled in a purely symbolic AI problems but uh once

we integrate our agents can start testing uh your games in a scope that you tell us we migrate potentially dozens hundreds of thousands of tests and the Agents can run through the games for you any time with any trigger and uh provide long-term analytics of operations of your system of your game even before it has been released so you also get uh artificial population if you need to as I have mentioned our technology is coming from space research it's more than 40 Years of uh research in AI planning uh it has been seen in games in uh go in form of GOP go or goal oriented action planning uh in many many different games uh the difference is that um we are we are the planning experts and uh it is generally difficult to scale up your game if you want to use

uh symbolic AI for uh characters in the game because uh those problems that the solvers and the planning reasoning is solving are very hard and uh they are not uh practical to be solved on consumer devices uh we do scale up those planning agents in the cloud at very large scale we are very efficient at operating them and we are not constrained by the latency envelope where we would have to respond within 100 milliseconds our agents are in this case they are uh just uh enormously capable of uh planning the whole playthrough through the game even if the game takes several hours hours to finish end to endend we can plan it we can plan thousands of uh actions the agent is supposed to perform uh we can uh recover if it fails uh performing the actions

and uh we are not uh uh within the QA we are not constrained by the time limitations we just put as many resources as we have uh into Computing the best paths uh we are also at this moment uh pushing the barrier of state-ofthe-art uh our technology for learning the planning and symbolic AI models uh automatically uh has received uh one of the best paper awards at KR uh knowledge representation conference last year so uh if you are interested in for example V fan leun the chief uh AI scientist of meta uh he has recently uh shown in Paris that uh learning symbolic models is uh one of the ways forward how to advance in AI it's something that we have been doing for years and where we have the state of the papers so feel free to check out research page where all the Publications

can be seen and uh I have only a little bit time left so we are super happy to start with regression functional smoke testing we can do golden path testing we can run Monte Carlo through your game if you have uh dozens of square kilometers uh that you want our agents to go through and kill monsters or shoot helicopters or do any type of uh exploratory testing think that's something we can easily do and we are happy to do it our analytic analytics are uh as good as we can provide we are basically collecting absolutely everything uh we are fully integrated into unreal fully integrated into Unity we are happy to integrate into your engine and uh you can download our case study uh schedule our demo