AI-Powered Quality Assurance

[Music] welcome to the novic gaming podcast where it is our mission to explore the business and future of video games we bring together the industry's brightest Builders investors and thinkers to keep a pulse on current events dissect emerging Trends and games share Lessons Learned and have a great time this podcast is also part of novic growing ecosystem which ranges from free and premium research to Consulting and AD advisory services for more information visit [Music] www.nic.edu zebedy provides a plug-in playay API and SDK for seamless integration of instant borderless and low fee payments using the Bitcoin lightning network with fees less than 1 cent zeid Powers over 4,000 developers across sectors processing

millions of transactions monthly you too can unlock the potential of borderless transactions to better engage and monetize your Global user base including the unbanked and simplify the way you handle payments start for free at zebedy doio integrate with just a few steps and monetize your experiences again that's z. or check out the link in the show notes and with that let's jump into the episode hello everyone and welcome to the novic gaming podcast I'm your host Alex te and you know it this is interview and insights as always I've got an exciting topic for all of you listeners today trigger buzzword AI we know that everyone and their mother are talking about um the use cases of AI and the billions of dollars that have flooded the AI ecosystem in gaming specifically startups looking to bring AI to gaming across a myi of categories have closed over 700 million in disclosed funding since 2022 in terms of funding categories they are broadly broken down into five things one

efficiency tools for creators two enabling new patterns of play three AI enabled game developers four moderation and five other the biggest of these categories is efficiency tools for creators at 272 million and moderation the most pecuniary at 42 million within the slice of efficiency tools for creators there are a few subcomponents such as art generation and animation videos game engines audio content creator tools and finally play testing and QA which from the title of our episode you can probably discern is today's topic and so today we're going to dive into the use case of AI as involved in the quality assurance stage of production for video games and to do so I'm welcoming two founders of two companies leading innovation in the space so my first is Christopher homegard uh CEO and co-founder of m.ai I'm of course going to have him share how he came to found the company but for our audience m.ai is a company seeking to redefine game development through two

products um model. test which is automated game testing and modal doplay virtual on demand players to satiate matchmaking needs um modal has backers like Griffin gaming Partners rendered VC and several others so welcome to the Pod Christopher yeah than you so much thanks for having me excited to be here W my second guest is Nathan Marts CEO and co-founder of agentic agentic and of course Nathan will explain more in depth is building AI players as a service AKA a service that allows developers to quickly train AI that can play and test their games agentic is backed by 8vc and others um welcome to the Pod Nathan hey it's great to be here thank you awesome yeah I'm very excited to have you on um I was really impressed by the demo that you guys showed at the um EC GDC event so I was like I gotta I gotta get these two guys together um yeah awesome all right so before we get started on all things QA and AI I gave you guys some brief intros but I would want to hear um and have you tell the audience um a

little bit more about you and why you founded your respective companies and specifically what inspired the names behind your company um Christopher how about you go first yeah sure absolutely happy to um yes so uh so so we started uh model in 2018 um ahead of a time I myself i' I've been in games since around 2008 professionally uh so been doing been doing that for quite a while and then I've been doing work in in Ai and machine learning for games since 2011 and what we do in in model AI is really just you know bringing those two things together as it were and uh and trying to help as many Game Dev game developers with that as we can um why we decided to found the company uh I mean there's there's a multitude of reasons for that but uh you know um the things that we're trying to address and help folks with in model AI are things that we as the founders of the company have faced ourselves when we were building games I think one of you know one piece of original trauma for me was when I had a game crash on live National Television one of the first ones that I did uh so that definitely taught me a lot uh about

the value of of good testing in QA um but more generally I think when we started model ey in 2018 we were kind of like seeing that um there were a lot of strides were being made in what could be done with gameplaying AI and some of it was in the service of just advancing AI in general and I think we're seeing some of the benefits of that you know rid large now um but because we came out of the space we thought well somebody should really try and and work on getting it out to as many people as possible and help them reap the the benefits of this technology that's why we started the company and ask for of a name well at the end of the day what we do at model we we model what players might do in your game we build models and we do it using AI it's it's as simpl as way awesome but you uh you model them without Excel I'm sure so you remove the E from model um from the finance word that's that's where I'd had to go that's death to excel that's only it's a it's that's a newbies model or space I'm sure yours are much more sophisticated I I love Excel and Google Sheets just for

um okay very cool I love that Journey um and thank you for sharing a little bit about the name I think that that's always so super interesting especially in the um you know it's it usually has a lot of meanings to to the founders that that found the company um Nathan what about you uh yeah so I like Christopher started out as a game developer in my case in the early 2000s I was a Game Dev for about 14 years at Lucas arts and double find um and then eventually went to Google was there for seven years I L developer products for AR and VR and some machine learning work and you know actually I started my career as a as a tech artist building like tools and pipelines and I think in many respects the the through line of my career has been creating technology that empowers uh it helps out creative professionals right that's what I love most is to build a technology that lets humans do what they're best at and often that you do that by getting rid of like the busy work the stuff that's tedious time consuming so that humans can focus on really being you know creative and

expressive um and so you know during the time with Google uh I started to see oh there's this intersection this Confluence of you know AI uh and game development there might be ways to apply that that really helps game developers in a new kind of way so I I met my now co-founders Stuart and Leo while we were at Google and in 2021 we left to found atic excuse me the focus of which is to create a service that allows game developers to trade AI that can act like players of their game um you know very at a high level very similar to what Christopher outlined you know sometimes you want players for QA sometimes you want them to validate your infrastructure sometimes you want them actually in your live service to be you know non-toxic teammates or to keep liquidity in your matchmaking pool but all of that was our our thesis and the thing that we wanted to do most of all was to develop technology in a way that it was you know um Ai and service of game developers it was thoughtful about the way that game developers work their like very pragmatic constraints their goals and to make sure that everything we built from the algorithms to the

infrastructure to the developer experience really made sense when your day job is making a game not working towards towards AGI um as far as the company name goes you know we uh like had our main goal is we wanted something that was that was fun that was a bit playful that was suggestive of the work that we were doing uh with AI I think it's very very easy to have a name that's kind of dystopian right a lot of kind of sci-fi AI references you know are are like not awesome um then also you know we I I personally in one area I'm superstitious is I think you you have to be very careful about your like company name don't tempt fate you know if you have something that's like Jinx you know like why do we have all this bad luck you know oh you know so we wanted something that was yeah very like cool positive you know Good Vibes and when we looked around you know we came across this word agentic which is really about acting with agency acting in a way that is goal seeking where everything you do is designed to achieve a positive

outcome and it's like oh that's cool that's like kind of what the company is about that's kind of what the tool does and then it's actually been a pleasant surprise that in the last you know year or two the word agentic has become actually kind of part of the the larger generative AI conversation so yeah that's funny yeah no it totally has and that's actually why I was wondering whether or not that informed the name but it sounded like the root of it came from something else and it's just kind of coincidental that um yeah awesome I mean it has it has agent in the name right so that's that's kind of nice but also I mean a gentic comes from like kogai it you know predates you know llms and and our company and we just oh yeah this is a you know a nice it's an easy to say word it's memorable and then it sort of is evocative of the space that we're in that's awesome well thanks guys um that it gives some good grounding for what your companies do who you guys are um and before we go down though the AI buzzword Wormhole and talk about a lot of things that you guys mentioned um you know the different types of testing formats and how could

we could improve QA through the use of AI I want to talk a little bit about the QA function in games its purpose and its history and how QA needs evolve over the life cycle of a game from pre-production to production to Alpha to Beta to ship um and one of the things I want to make clear up before we even open this discussion is that QA is often a disenfranchised discipline in games um I'm sure we've all seen Mythic Quest on Apple TV but um to some extent QA departments in games have been where people begin their careers and Rise through the ranks to become some of the most famous game directors and Senior game leaders of all time but it is kind of that like mail room equivalent in Hollywood um and another debunking that I'd like to share is that QA people sit around playing games all day which is not true um it's not monotonous and in reality QA can actually be at times a creative and Technical function so I want to set that tone today that although we're going to talk about hey how AI can enhance Enlighten QA demands for a game we're obviously in no way

diminishing its functional importance um so Nathan you're up first um I would love for you to tell me a little bit about um what QA is in games why QA is important to games and what are the various methods developers use to access QA resources currently yeah so I mean to start I think the most important idea is that like when you're building a game at least the way I've always thought of it you you're you're building this little virtual world and contained within that world is like basically a set of possible experiences right that's what I love about designing a game is like no one experiences it uh the same way and as a designer you're trying to sort of shape not just one experience but all possible experiences and when you're doing that you know there are some experiences that you don't that should not exist right the experience where your game crashes on national TV you want to you know prune those experiences out of the game right uh or if the game is unfair or frustrating you get rid of those on the other hand there are like positive

experiences that you want to have more of you want to shape the game towards those and I feel like at a very very high level the role of QA is both of those right part of the QA function is finding those kind of negative experiences and getting rid of them um and then another part of it is like actually helping refine and improve those positive experiences you know how they do that there's a variety of forms of QA right you have traditional kind of automation unit testing where you have you code do it for you usually that's doing very atomic parts of the game at the other end you have you know human QA which is actually this you know where people are playing the game in a variety of different ways usually collaborating with other members of the development team to understand you the behavior of the game and within that especially within human QA you can look at some of the work that's incredibly creative and collaborative and you know I am so grateful for like every QA person that I've worked with because you know when you have a bug that's hard to Repro or

you need a tough decision about oh should we change the game this way or that way those people who are really experts in the game and the experiences of the game are invaluable but there's also parts of QA that are can be like very very repetitive you know where it's okay there's a giant open world the player can Harvest plants make sure that every plant is harvestable you know brute for search just um and up to every plant exactly um and so at agentic you know really our highlevel goal is to empower you know members of the team especially folks on the QA team to use AI to like delegate the busy work to scale themselves to have you know machines do often I think the least interesting the least creative work and then free up the humans to do what humans are really great at which is refining The Experience collaborating things like that in that you mentioned so many different types of testing Frameworks um and Christopher maybe I can lean to you on this um in there I

heard gameplay testing I heard unit tests I heard functional testing um can you kind of like sort out what all these things are um you know kind of maybe explain the ideal test pyramid um you know unit tests all the way up to functional tests and what do those actually mean you know Nathan you talked about there are tests that are only in code and there's human players when are these sort all of those things kind of by like labor and Al time intensiveness um because I think that's I think also for our audience that's something that's sort of and even myself I'm kind of confused about where actually what are all the different types of tests that you need to do in QA to make sure that your crash on National Television does not happen yeah that's right that's right well I'm happy to take a step at it at least and and I think I very much agree with with what you were saying nth and but there are kind of like two questions that there many questions but two fundamental things you're trying to answer with the with the QA and the testing process right is like does it work and and the other question is kind

of like is it good or is it doing what we want it to do right and you can kind of like think of the testing pyramid as being being sort of like made up of that we're like at the bottom uh of the P pyramid uh we're trying to answer the question does it work at all and we can do that at different sort of like levels of scale and with different kinds of method and then we try to move the way up and saying like are we building the thing that we're intending to build and if we're not what should we do about it and how do we react to it and that's kind of like the way of it I think fundamentally you want to think about QA holistically but it's something that that permeates the whole development process and is part of every stage of of the game's development right and back to your point Alex that's that's and your point as well Nathan uh that's also why you know QA is so intrinsic everything that we do in the business and and maybe sadly has been a little underserved tool wise we we can get into why why add the case in in a second perhaps but but sort of like at the bottom um of the per when you we were talking about the unit testing very you're trying to to Really almost you know when you are producing game when you're writing Cod bits when you're adding ass sets you're trying at the smallest functional level to

understand like does this bit of my game work and that's that's incredibly valuable uh in the uh in in the day-to-day because it helps you like with every step you're taking forward to answer the question okay does uh does this part of the program this was part of a game do what it what it needs to do but then on top of that once you start putting all these bits together then you start having these emergent phenomena happening between all the different bits that people made right that can be between multiple systems in the code just two pieces of code it could also be the fact that you have somebody who's adding an asset as part of a production flow into the game and an asset is in many ways also a function you know that's also a functional piece of the game right and uh and now you have multiple disciplines that are interacting and they're all contributing to the same artifact so as soon as we set that into Dynamic play and it starts operating we need to add on these like more complex elements of testing and we start stepping up the pyramid and so that's when we start going up to things like you know smoke testing for instance like will the game Run at all does it crash when I try to operated under which

conditions uh will it will it run will it not run and for how long will it run so we also need to start doing things like soap testing so so we have to figure out you know keep this running for a long time do we have you know memory leaps or other emergent phenomena that only you know uh show up after the application has maybe just been open for a long time or maybe you know has been running for a long time then on top of that we have to think about um how does this perform under load and on different hard Hardware targets so we need to get into performance testing as well so that means we have to thinkig out how does this run on the different platforms that I'd like to deploy it on but also if we're doing something that has a multiplayer component for instance well you know will that also work if a 100 clients connect at the same time if a thousand clients connect um now at this point we've only really been addressing the question like does it work like does it even run does it even blow up uh but but but that actually takes up the the vast majority of uh of the labor that we're talking about here um and for all of those different processes I think it's also important to to say you know

you can solve some of that with automation but you still need to have like a layer here where somebody you know when something does go wrong somebody still has to a have a look at it and say okay why did it go wrong what are we going to do about it who should take an action on this and How likely how important is it How likely is it that it's going to happen again um do we even know how to reproduce this so there are all these steps of actually analyzing the information that's coming out of it right um then stepping up from there that's where we can start thinking about things like functional testing and now we start not quite moving into the to sort of like the part of it where we say like is it is it good but we're just talking about you know is it doing what we want or the Dynamics of the application playing out the way that we would like it to and so that's again sort of like where you start relying more and more on this um you know human judgment and human intelligence to say like you know is what is happening in the game making sense is really a thing that should be happening but also you know being smart about the combinatorics and saying have we tried all the things but we really need to try in order to to push this you know application to the edges and then finally you know moving up from there uh considering the game more

holistically then we get into like the area in the space of like play testing where you know your team is playing the game is it playing out is it giving that experience that we'd like it to have and at that point you're probably doing that both with your internal QA your external QA and ideally of course you know probably also with with with your players or somebody who's representative of your player base um and I I think where the the role of AI is coming in and automation is coming in at the moment is just like you were saying Nathan is that well if we can take uh some of the time that's being spent uh at the bottom of the pyramid take that away either through great processes in our organization or through automation or those to things together then you can really set the people who used to be spending their time there free uh to operate in the in the upper part of uh of that pyramid so to speak but but for me the takeaway there is just that there are so many components and so many dimensions to Quality Assurance um but that's you you know one Fe could be but that's another reason that it may have been a little underserved from a tooling perspective because it's a really really hard problem to solve without having you

know human intelligence in there yeah well I mean well first that was incredibly uh I I feel much more educated um because I think that there's all the the way that you broke this down um you know does it work is it is it does it work at all and then there is it functioning as intended and then there is like is it good right is like the three different Tres of QA testing and you actually need all of that the whole way through the whole the whole life cycle um and I think I can't remember which one of you guys said this but you sort of we're just we're talking about how testing needs change over the course of a games production life cycle you know you said different players of different intelligence and it sounds like it's not even just different players of different intelligence it's also like entirely different types of testing um things that require players and things that actually don't require players at all um and so you know before we kind of jump into the into the actual components of I think this is a great point to switch over to model. and agent's role in QA and what you guys are doing in that actual pyramid which of

the which of the things are you actually servicing because it actually might be all a lot of them um just giving some perspective for how big QA organizations are at gaming companies because we talked about people spending a lot of time in the unit tests like the more like does it even work at all um at the bottom of the pyramid and um I'm not sure what it was like at EA um and Christopher maybe you also have an interesting perspective given your background in some indie Studios but the QA orgate blizzard was like 400 people if not more uh now obviously this was for many titles but that is an an army right um and the Activision Blizzard paa Union for example stands at like 600 people strong um and so do you guys have any perspectives on the way that you saw QA org staffed um to kind of give us some perspective of you know just simply how enormous is this task in terms of headcount I think one thing is important to qualify to is that there's there's not such a thing as like the QA or you know usually you have for example like

Dev QA you know which are you know members of the you know employees usually of the same company usually working you know very very closely with you kind of committed to one game at a time experts in that game um you know just as focused on it as you know an an animator or an engineer um then often you have um you kind of a shared service QA model or sometimes it's called Pub QA where like it's a broader pool of people that may move from one game to another based on like where they are in development so often they'll have expertise like um certification or other kind of common processes that every game through goes through at certain points in the life cycle and then more recently there's often been um Outsource QA beyond that as well where they look at a vendor um who has you another body of people that you can bring on at certain times and so you know different Studios different game companies whether you're at indie or triaa it all it all looks different um but I think it's just that's the you know what even think about like tooling the workflow who does

what it depends a little bit on on which group you're talking talking about yeah yeah I I agree with that and I think um you know if we're thinking about it like from the India perspective or at least some some of the Productions that I've been part of you know um in in most Productions you're you're probably actually you know one of the one of the challenges there is that you you just don't have enough QA you don't have access to enough people uh or you have to Outsource it but you can only do it for a certain period of time and you'd like to have more when that you can get probably or you're you're riing on the publisher qway um and having those people help you out often maybe a little later in the process uh than than you might like right so it's a very very it's also just a case of like do you have those resources at all and if you don't have a dedicated resource what happens then well at that point you probably start leaning on your on on the rest of the team and everybody becomes like part of that function um so you know at different ends of the industry there might be sort of like different makeups uh in in terms of like you know in the Bigger Productions you have

team you have you know enormous amounts of Engineers and artists to build that game and if you're doing a game as a service your players are content if you have ugc you know this the state space of the game is even bigger and so I think there's been this macro Trend towards games becoming rapidly more complex and that has I think generally speaking really outstripped the ability for traditional teams just keep up with the scope of the game and so I think this is other dimension which you how many just people can you bring to bear

at any one time and then the final which is you want to think of scale but then the last piece I'll say is like flexibility where um you know game development is not like kind of a linear constant process there are um kind of es and flows to it and so often you know let's say you want to do um infr you want to like load test your servers you're doing a multiplayer game you want to know you know today we're in development today we have zero concurrent players and we want to test what would happen if our infrastructure had a million concurrent players well how you know now do you need to have a million concurrent players Around the Clock no that's that's you're not there yet um so today folks say well maybe just you wait for Early Access or something like that but ideally what you'd want to do is sort of spin up have you if you could have a million people jump on your game for a weekend you know or or an evening collect data turn it all off and then go back to making the game like that would be ideal but that that flexibility the ability to kind of

scale up and scale down you know you can't do that with you know with Live players because they w't actually play the game you can't do that with QA you know so if you look at these sort of three things like uh how much access does QA have how much scale does QA have how much flexibility does QA have I think the kind of technologies that we're building can really benefit you know all three of those areas and again not as a not at all as a replacement to QA but actually a compliment as like giving the people who do the work the ability to do things they couldn't before to work at a scale they couldn't before and to be dynamic in a way they couldn't before yeah I think if if I can latch you on to most like free Concepts like access scale and flexibility uh especially when you're thinking about scale scale and flexibility it doesn't even have to be that much before you start really enabling teams Al together if you can you know give them access to a number of virtual players I'll give you a couple of examples we've talked to a number of like multiplayer Studio where you know they're being built out by teams that are just like the core

team is smaller than the number of people that the multiplayer game is intended for so that means that not only like if you want to play test your game for instance or just like see see if it works like even if you take every developer in the studio and ask them to spend their Friday afternoon doing that it's still not going to be enough and so and so what that means in terms of of flexibility is it's very hard to get like a proper test of what it is that you're building and that in turn means that your iteration Cycles become long and uh and I think most people who are Game Dev would say but like shorter iteration Cycles are usually preferable because it gives you a uh you know a faster path to understanding again to answer that question is it good or not right and and that's also a place where I think that but these AI tools can help out a lot because we can start tightening some of those iteration Loops that are going on in in development I think everyone can you know sa save time save effort but also get faster to the creative goal that everybody on the team is trying to reach well I was going to ask you both to pitch me on why Studios should look towards AI for QA but I'm sold um what

you've already said seems to be um incredibly valuable uh shorter iteration cycles and actually even the capacity to even perform the tests at scale access and flex in and flexible ways um Nathan as you as you said um so I guess this is a great time to actually start shifting towards talking about modal Ai and agentic specifically um and you know we kind of gave we have this framework we understand what QA encompasses we understand how big orgs have been in the past or we understand how Indies have got it done in the past and we understand what the QA needs might be over the course of the project so I would love to hear from you guys about you know where are your companies starting um what development where in the development cycle are you guys you know you guys as your specific companies based on where your you are in your own product road map today equipped to help um what genres of games are you equipped to help and does that actually change um based on whether or not you're a mobile platform or maybe a third person shooter um you know Nathan maybe you you can kick us off here um what is what is

actually in the who's the first guest I guess the first QA customer of atic what do they look like yes yeah so um we are right now we basically just announced at GDC this past year which is pretty fun uh we're working with our first design Partners right now uh those are mostly look like uh doublea and AAA Studios as a company uh our big Focus you our our mission we want to serve game developers right help them be more effective and so we're looking at the areas where we think that our technology has the highest uh marginal utility uh where our where there's like a significant Improvement in using you know a technology like a gentic relative to the next best thing um and so we think that um you know certainly real-time games are we can benefit a lot more because you have this very complicated simulation and we're currently focused on avatar-based games where you control um a particular character in the world so be first person third person MOBA Etc but kind of fundamentally Within that you know kind of space of genres our

approach is that we want to give uh game developers on the team and I include the you know QA folks in that we want to give them a tool to you know create AI that do useful work for them um because we think you that the broad pattern is you know you want more players of your game you know Christopher made a great example of like that how do we like it's a battle royale there's 30 of us of the company and we have 100 players per match like that's a case where it is a QA not usually in a formal sense but you want to be able to inject more players into your game if you have that you know that big gap between you know your unit testing and your human testing where you know your unit test very little and you want to automate more you know of the busy work you know smoke tests often you want to do that by injecting more players now these players may play in a different way but it's all you know you want uh agents to interact with and play your game and so what the thing is that every game is different so you can't take a one- siiz fits-all you need you need a tool that uh can adapt itself or

be adapted to the specifics of each individual game and also the like the workflows and the the sort of scenarios that developers care care about vary from one company to the next and so our our Approach at agentic overall is rather than saying like Okay you know we will give you this like TurnKey you know you must use it this way for this purpose was like no no it's it's it's much more like a behavioral Photoshop like we give you a tool that allows you to Define and train and deploy agents that are useful for you and I can talk about the tech you know behind that in a little bit but that's the at a high level it's about empowering these teams with a tool uh especially when that allows non-programmers non AI Engineers kind of a wide variety of folks on the team to create these agents that do useful work as you define it and then once to find you can deploy the agents you know whenever you want wherever you want however you want yeah I am super excited to talk about

the the how there and what it actually looks like from the developer perspective so we're definitely going to go there but Christoper any reactions to that things are similar things are different at mod. um you know for you guys where are you focusing on the development cycle and you know what gaming platforms what g game genres um what type of studios are you guys looking to work with um first yeah for sure uh I think in terms of a core problem very much agree with everything you were saying Nathan uh you you like we're we're also very oriented towards bringing um quote unquote AI V works so it does useful work but also AI just like pragmatically Works uh to Productions so we'd rather give you something sooner that'll move the needle for your team right away than then sell you like on a big bang project and or anything like that right um in in terms of how we've gone to it I think we also start with the type of game because that kind of like determines what kind of AI is good uh for for solving practical issues in them um so yes we've also focused a lot on on Avatar or character-based games uh you know games

where there's like One Singular player character that that you can ask the AI to control um but also makes it perhaps easier to learn from what players are doing but that's part of what we do and some of our approaches is to to observe what players are doing and then replicate that inside of the build um and then the second thing that that we've been looking at is actually just the go layer and all the things throughout core gameplay but also supporting that so that could be for you know obviously we have a game that's that's that's completely made up of of like a narrative game or something might a 2d narrative game might be put together that way um but also if you have a store that you're releasing quite often that's just something that we found to be valuable to many teams because it's a you again like we were talking about before you're putting in a lot of assets you're making a lot of changes all all the time you might be releasing weekly daily multiple times a day they're just like a lot of things that can slip through the crack faere so that's like another place where where you can you can help folks out um ter terms of the studio types yeah um I think we've uh we've done a lot of our work recently together with with Double A's and and some AAA uh I

think what what characterizes those organizations is also that typically have that functional specialization so you have somebody to sit on the other side and kind of like be the the you know your Champion or your ally that receives it that's that's always good to have somebody who understands for Mission you're on who can relate to it um we do have we actually do have a product out but Indies can can can install and use on their own but take some of a functionality but generally what we're also trying to do is to provide our tool to folks know way where you know we help you get started we'll give you support to get going with it and then over time you make it yours by extending it into the functions that that uh but but but you need it to do because as you were saying Nathan every game is different so the only the only team that can really Define what AI needs to do for your game is is your team right but we can help you we can give you tools at the end of the day you you'll you'll know where it kind like moves a needle for you um and and we tend to approach it yeah with with three different things in mind one is like what what we're putting together needs to be informed by the craft of game making and it needs to understand the workflow that dabs are going through

then you need to have ai behaviors that are practically useful um but I don't think they always need to be human-like um sometimes you just want a bot but just like pushes every bot button or like tries to you know visit every part of the map or anything like that um so we tend to think about it as sort of like an there's almost like an intelligence ladder for AI agents or AI Bots where some are you know some are useful for some things others are useful for other things and then the the third thing that we bring uh is uh we we have an infrastructure solution but you can even have you can either have it with us or you can deploy it with yourself um so often if if it's a barrier to kind of like figure out how do I spin up a lot of instances to like test my game at scale that's that's something that we can help out with as well um and and Target wise I mean um we we've built out our tooling mostly for unreal and unity but we're not we we can technically support uh most targets um but obviously for the big engines we we try to have a nicer interface from folks to work with yeah well it sounds like you guys have basically from this concept I know that we you Nathan I think it was you that

were lamenting the relationship between Ai and games at the present you know games are being used to develop AI techniques versus you know AI companies being helping game developers make good games it sounds like B to Slate from both what your context you're providing is that you already have tools that are have production Readiness and are working alongside QA devs at this moment in time for some dable a Indie Studios um in a way that that's actually pragmatic and applicable um and so maybe I would love to hear maybe some a success story Nathan and Christopher about something that you guys have done alongside a studio um and how that has actually gone down and you know how did you ensure predictability um you guys talked about this phrase called The deterministic Hammer which I would love for you to elaborate upon for our audience about what that is um so yeah would love to hear about um how that's actually working day-to- day for for some studios right now yeah Christopher you want to start uh sure sure I I mean maybe it makes sense to uh to take out out just like a specific case uh but but uh but we've

worked with before it's a game that's that's shipped now uh but we worked with them for a year and a half a couple years uh and for this particular uh this particular title was built out by I think they were a 50 person team or or something uh around that size so it's kind of like where you have some dedicated QA resources but you wish you had more um and what we did with with this team was that uh they were building a you know it's a a multiplayer loter shooter uh with not proced generated Maps but they would sort of you know dynamic map loading in big areas and things like that and they were just facing that uh but it would take them like 40 minutes to play for a level also story content and everything for for a human tester right and they were shipping the game with 100 levels and then with variations it was going to be 300 and it was just like way out of scope for what they could do with that team to have continuous testing in there so so so what do you do about that well well for this particular title we uh the thing that ended up being I think the most valuable thing to them was consistent and continuous performance testing uh of

the game so so the sort of like the flow that they got into was that um every week uh their build would go to our test service we would scale it up we would test all the um we test all the levels overnight so that means that when you leave the office Friday then when you when you show up Monday uh then you basically have you know smoke tests and performance tests and functional tests across all those different maps and when that informs you like which of these do I send on to human QA which do we just know okay there's there's something totally broken here and and maybe also crucially you can look at the maps and then you know week to week to week you can say like okay is the performance getting better or actually did we make a code change that now completely broke this one map what do we do about that so it becomes sort of like a way of taking the temperature uh on your game over time um it was interesting we we can see in the usage uh from from the studio you know every time that they were getting closer to a milestone or they were shipping a new feature if they're going to do a a release that's kind of like when all that re that all that that um you know testing consumption really went

up so you can see the team using it and I think that's that's one of the most validating things uh when you're working with a team but you can actually see like where where are they in their process and what we're doing seems to be making a difference to them because we're using it you know more at least like key points in in the development process yeah a couple from our side um one there's a studio we worked with last year is a similar size Studio PC console game and this is a a game uh it's like a kind of action brawler with procedural World Generation designed for between one and eight players and when we started we started working with them at Alpha when the team was you know getting a handle on the game itself um but they were realizing that their number one risk was multiplayer especially when you had like eight concurrent players that just stressed everything more than when it was single player except they had two full-time Dev QA and so they had this basic problem they're like we want every test to be an eight player Co-op test we don't have that many people to you know

like that's actually bigger than the test team itself and so what we helped them is we we created agents that were like kind of companions where you would have a human and the human would be the kind of implicit leader um and then the agents would would join and you know you could choose whether they' you know follow along and kind of Be good buddies or just be randos and go off in their own Direction um but uh you know thing that we do is uh our Integrations uh whether it's Unity or unreal or proprietary we kind of integrate at the player controls layer so we um this is a bit in the weeds but a lot of Bot Frameworks uh kind of bypass the player controls they actually connected the game with kind of lower level systems so they they look like a player but they don't actually interact with the game the way that a human does and we think that's you know uh really important that if you want to truly exercise the game like a player you have to integrate in the right way and so in our case you know we integrated like a real player so they could could actually spit up a bunch of little head basically whenever a tester would test the game they would

kind of fire up the main game and then seven little headless game instances that had agents in them that all join the same game instance and all of a sudden lots of you know like oh my goodness you know like this here there's a crash when this happens you know latency spikes here um and in a lot of cases you know the developer they had the ability to catch those defects you know on their own they just didn't know they were there without having you know a consistent Suite of players um I think another thing that's that I think is also interesting is that even though you know QA I think is the kind of the obvious first application of this technology you know we really see it as um a tool for the entire Dev team you know other examples let's say you're you're an animator and you're just like iterating on the move set of a character you know uh your your tool of choice is you know Maya you're animating you know you're trying to work with you know real data or mocap data own inspiration pull it all together export it to the game

but in practice today you often have to just like sit there and like play the game to do a move over and over and then go back work in Maya play the game some more and that iteration cycle like Kristoff was saying was like that the longer that iteration cycle is the less productive you are and so actually just training an agent to be like do this move over and over again or like do this combo you know is actually now you could just let the agent I mean like this is AI in a very simple sense but if you can Define that and train it and deploy it without writing any code without knowing anything about Behavior trees or state machines you're like yeah hey AI do this thing over and over again and now you can iterate on your animation just look over oh that's looking better keep it oh and now you've got you know a move set that you like and you know is that like a AGI kind of a you know application no but it's incredibly useful and productive right yeah and maybe that's a lot about the you know the distinction like you said

before about the relationship between Ai and games this being useful to game developers versus some sort of AGI statement like you said before um and you know first of all that was super helpful because I guess from you know from the studio perspective right you're saying okay how does this actually work and it sounds like from at least from the agentic side it's kind of like almost like being in like a final fantasy party-based combat system where you're like okay um I'm playing Kingdom Hearts and Donald Duck is GNA I'm going to send him to player I'm going to send him to Agro he's only allowed to heal me at all times he's never allowed to attack anybody right you're kind of setting the customized behaviors of the characters or the other players in the game to be alongside you to help and Aid with that QA testing um is that generally the way that some studios can think about it yeah I feel like Christopher and I could probably get into the technical weeds for a long long time uh but I'll and I it's I'll try to hold myself back from getting too deep but uh with the Gent basically uh we provide sdks for uh

Unity unreal and proprietary game engines so you pick the SDK that matches your game engine you integrate it um and then we provide you with uh depending on your game engine different ways of defining an agent and you know agents are kind of fundamentally defined as the actions that they can take and that's like you know jump move Look Fireball you know use item whatever um and then you also Define what are called observations which is for us the kind of the parts of the game the game state that the agent cares about um and so if that's like in a third person shooter well you probably care about the enemies you might care about you know your next objective and where that is you maybe you probably care about the static world around you obstacles and things like that so we give developers tools in the SDK kind of in the game editor to Define one or more agents but then once you've defined them uh you actually need to like tell them what to do you need to train them quote quot and with a gentic we provide uh a conversational uh human

in the loop approach for defining training and deploying agents and so uh the way that you once you've integrated the SDK the way that you train a new agent is you launch your game uh we give you a you know chat interface at our SDK and you talk to the agent and you tell it what you want it to do uh and so if you say yeah I want you to go you know be a Healer and heal under these conditions uh the agent will actually uh write code against kind of a script rting language it's part of our SDK that's customized per agent per game and so essentially the AI will kind of try to write a script to do what you asked it to do we'll do some validation we'll deploy that into your game live you can see the agent actually do its version of what you asked it to do sometimes it's perfect and you can basically approve it or you can give it additional feedback like ah you know like too much healing you know or like actually I forgot to tell you this important detail so is this uh you essentially have a conversation with the agent and what

you're doing is you're building up you're giving that agent a set of skills and often skills build on top of other skills uh a lot of what we do is we uh make it we allow you to do that without writing any code there's a lot that we do to actually allow an llm to write code that's functional in the context of a game we're kind of an adapter in that sense and then as you train skills for the agent the agent can learn them memorize them and then you can deploy agents later on with those skills and you can do that conversationally or through an AI an API if you want to do it with automation but um you know there's lots of really interesting reasons of like you know why we chose that approach there's other approaches that are also compelling in different ways um but I think that some really high level things that we believe in are that um that it's human in the loop and that it's a tool right that it's it's not like a it's tempting to think of you know AI as like a magic lamp you're like test the game uh you know Genie I have free wish you know um and then somehow magic happens and you know it's like it all

works but that's that's not how this works at all right it's like really we need to leverage um what AI is really good at like scale flexibility you know writing code on your behalf and also Leverage What humans are good at which is like creativity context purposefulness um and so the way that we see that working is to rather than being like yeah here's a magic lamp here's a black box instead here's a tool that gives you some new superpowers but fundamentally because it's a tool it has an interface you know uh you can use it in different ways you get actually have final say over whether the behavior is what you want and we think that that kind of developer experience framing infrastructure all of that in service of the human is what what we're excited about and and what we've really built our product around yeah I think I think philosophically we're coming from very much the same place and also from the same starting point it's kind of like well if you have an agent in the game it needs to know two things right it needs to know like what can I do and what's going on and those are kind of like those that's the starting point for for

minia like for almost all well for any agent based approach right H we also have an SDK uh but but we take as as sort of like the starting point for the developers to get going um we do uh setup that sounds a lot like like how it works with the Gen you know we ask a developer like nobody knows the game better than the than the developers themselves so it would be like a total waste to try and guess what is this game about instead of just having the developers tell you by setting it up with the SDK um so that's that's like our core approach as well and what that gives you is sort of like a um you know a summary of like what is this game about and that's like the perfect bit of like expert information to get your to get your AI started so okay so once you have that in there I think that's that's where in the model approach we diverge a little bit I I think that's a great approach but but you're taking me I definitely see strengths of that uh I think we uh we approach a little bit with some pre behaviors that can get you going without any input from from um other event setting up SDK so that's where we try to move the needle for deps like out of the box where we say you

don't have to tell it anything you don't have to do anything just set it up Let It Go but it's it's not going to play like a human but it'll do something for you and then the way that that we interact or sort like that we that we think about informing the agent about what it's supposed to be doing inside of the game is very much based on showing at the moment so a lot of our approaches is it's kind of like rooted in the fact that you know QA QA uh and and Dev teams in general will have like a lot of test cases is defined and they will be running through them you know daily weekly monthly whatever you know over and over and over again and that's kind of like a perfect starting point for asking the developer to show me what this game is about and just like demonstrate for me how you would do this because I think we're very R rooted in the notion but if you have done something you know five five times over and over in the same day you know you it should be automatable we just have to find find the right technological way of doing it h so so that's kind of like where we build a lot of our approaches on having like leveraging the fact that you already have a team that are

executing these functions we can learn from like seeing what we're doing in the game and then you know by virtue of doing it in the game they can then also demonstrate is this what we want to happen inside of the game and then that comes with basis of building up the behaviors so so Al you know like a similar s similar approach I guess but but based more on seeing like like what we uh what what the um the people who are building the game or doing with it and then replicating that and there then once you get enough behavior of that we can start merging like that data together and then building models that are typically you know more adapted playing the game in a general fashion yeah so this is um first sounds like there's slightly two different approaches but um same kind of situation where you're kind of plugging into an STK as a studio and you're basically either empowering an agentic agent to play in a certain way or Christopher you're talking that mel. kind of has these custom preset agents modals modals agents and modals um the ages and modals come to my game and they and they help me out um and so actually I think this

is a great time to actually transition to like what does this actually look like in terms of a business relationship between yourself and the studios for you and for the studio um and so you know sensibly it sounds like you know you you'd partner up with you guys um walk me through like what that actual onboarding looks like and and also tell me a little bit about like what this costs to run for this studio um you know obviously like potentially we're saving time but one could presume that some of these large scale tests are expensive um who's bearing the cost for those tests um is it the studio is it you guys um Nathan how about you you kick us off yeah sure so I mean in a we're a SAS business so software as a service um you know it's fundamentally uh you pay per unit agent time so you know if you have a thousand agents play your game for an hour or one agent play for a thousand hours it's the same cost um with our c customers you know we're right now invite only so we're doing uh it's a the

pricing is all private but it's essentially tiered usage based pricing where you know developers they need to be able to you know increase the like Christopher was saying earlier there are seasons where you're doing a lot of automation there are other Seasons where you're doing less and so people need the flexibility to go up and go down but we've also heard from a lot of customers that the thing they're most afraid of is like some script goes Rogue and they wake up to a you know million dollar cloud or you infrastructure bill they weren't expecting so they want I mean that sounds good to me um you know but uh no they they really want to have kind of capped costs right so the ability to have predictable expenditures but really conceptually you know we think of our our agents as like little virtual contractors right they will you know you can deploy them in your game they'll be confidential right so they won't tell anyone else about the specifics of your game uh they'll learn as they uh as they you know play your game and they'll learn how to do it better um but then also that they may carry some of the

metal learning from one game to the next even though they won't carry into the specifics of your game from one to the next interesting yeah and and over on our side we're we're also a SAS business so that is uh that's that's kind of like taking the same approach here um I I think what what we want to do is you know the more you know we want to provide value to our customers by having them want to use our products and and we want like you know the testing to be helpful for them and that's also why why we take a consumption based approach we basically say you know um it's very straightforward we want but we want you to want to use this product and the more you use it probably the better your game is going to be the better off we're going to be and it's kind of like how we succeed together you know in in the industry we want to be part of that movement and that's um that's how how we've set it up so so with model A we typically set you up with a running subscription package that gives you hours of testing um that testing will involve one of our Bots and one way or the other um and and basic will'll spin up you know machines that will run for

the allotted amount of time that you'd like to test your game for and then that can happen on demand some teams like to to manage it uh manage it themselves and and just be very manual about and say we want to test it now other teams maybe wanted to be part of their continuous integration Pipeline and say you know given certain conditions we just fire off a build and it goes up there and we'll spin up you know 10 or 20 or 100 copies spend the testing consumer results on the other end so that's that's the same approach but but we're taking I think in in terms of the onboarding experience that's something we we spent a lot of time uh talking to customers about and experimenting with and trying to figure out how to get these tools to fit into to people's workflows and with the kind of customers that we have at the moment the conversation often goes along the lines of we need help but we're really busy and probably too busy to get it like on our own uh so what we've decided to do in model AI is that if you decide to onboard with us like one of the things that we set you up with as you're coming in is what we call our developer success Engineers which is basically you know it's a person who will sit next to you and make sure that you get the tool

implemented in a good way that you get the most value out of it um that they they'll help you know Define your first test you know test cases with the AI automation all those different things because at the end of the day it's it's as we were talking about before it's it's it's all about the the team and the people who are building the game and you need to help them adapt to it we've just found that if you have somebody to get you going uh that that but usually can help sort of like drive that transformation in the production and then you know when the teams ready you can hand over the keys and they can they can start building out with the with the with the tools on their own yeah I was I was going to say yes yeah in and we never go away we're always around for support right but the idea is that we get you going and then you you take it away when you feel that you're ready for it yeah we have a similar model and that's that's part of the reason why we're invite only right now where we're working with select Studios to make sure that we can provide that kind of high touch onboarding experience I also agree with Christopher that yeah there's this interesting balance that you want to strike where you know these are like we said these are this is is a tool but it's a new kind of tool and so teams need you know at minimum some help like

how do I use the tool what's the most effective right um in some cases there's more than that there's the integration of like Yeah we actually want some help not just understanding it but connecting it into the details of our game sometimes it's not just integrating it it's it's setting up things but that's I think that you know that it's important for companies like ours to be available in that way to help get people off the ground but I think we also both you know there's other trap where if you do too much of that for too long then the game teams don't really internalize it you know and ultimately what I think both of our companies aspire to is a tool that like the teams value you know not just because it's work that somebody else has done but they value because it makes their day-to-day job easier and then the more they value it the more they use it the more they use it the more they're able to own it and that's the sort of path from you know early on kind of high touch support to a sustainable scalable business yeah yeah in a way you're actually quite similar to like the backend gaming engine business where you

know it would take way longer for me to build a QA org by myself to almost to a un you know especially for smaller Studios an undoable amount so this is a little bit like of a build or buy situation um and what you guys are basically suggesting is that your business models are they should be saving these Studios a tremendous amount of money right whether or not you're on a subscription package or whether not you're on a consumption based model um at the end of the day right it should take I should need to hire my org before needed to be 70 QA people and now it can be two QA guys with partnered with either agentic or modal a yeah I I don't actually yeah and there I I slight I don't know that I would say like don't hire the 70 people I would say still hire like just those 70 people like can just do way more than smoke testing the build and like harvesting all the plants right so um and I think it's really important not to see this as like a a zero sum game of like humans versus robots fight you know but I'm like the more humans the better and even if you

have those 70 people I guarantee all of them can be more productive help you make a bigger and better game you know if you add a technology like AO or a jet Tech yeah I mean I mean historically automation has usually led to to more growth and more productivity and and and basically more more you know Prosperity being created uh sometimes I call it the spreadsheet effect it sounds like a boring story right but when when VY calan first spread sheets came out everybody thought like oh accounting is going to be automated right there's there's more accounting than ever happening in the world and that's that's kind of like the way it tends to go with with this kind of automation so so so not not too uh not not not too pessimistic about that side of it I suppose yeah and maybe that's um you know you were drawing to a point that I was going to try and close on which was you know this is the cyclical pattern of of Industries right which leads to more Leisure Time you know um I started reading a book um about you know generational change and um you know it was talking about this idea that you know you know back in the day we spent a million hours harvesting corn and now we

have machines help us harvest corn which has allowed us to make the iPhone right that sounds like a pretty good life Improvement and so you know maybe perhaps this the the the wave of AI that's happening in games is going to be very similar where it is going to allow like you said Neath then the people who are humans to do the things that they're the best at while taking away some of the the brunt work um and I think like that's you know earlier in this episode we talked about the atvi QA Union and for those in the audience that have been keeping up with the news um the Activision quality assurance United CWA formed to protect QA salaries labor protections and overall to fight for their um status rights to stay at parity with developers and other software Engineers uh this was initially start start sparked by QA testers at Raven um but it grew over time and I think that you know Nathan to your point you're not saying don't hire these 70 people just Empower them to do more but at the same time this is the appeal of things and tools like this is to allow a studio who would be un otherwise able to do full

Productions testing fit it into their budget um and so this isn't just the the QA thing like this is the entirety of the efficiency tools for the game sector's value proposition um and so I think that that's important for us to just kind of address head-on is that like it is actually going to allow us to move away from brute forcing things and but Brute Force before costed money um which I think is obviously where the business business value proposition comes from from your companies um but guys this is such an amazing episode and I learned so much from this about the different types of QA testing where a gentic and modal could really impact a studio and I have a final question for you guys um before we kind of close and wrap you know we're 10 years into the future um what do you guys think that the future of QA testing looks like um and how do you envision that relationship between um Ai and man and to be uh Nathan yeah I mean I think

given the rate of progress that we're seeing you know in machine learning at AI 10 years is going to be a long time it's going to be you know I think any prediction made today you know about 2034 should be taken with a giant grain of salt um but I think there's a few things and again I think you know I'll go back to you know our company name like agentic right helping people achieve their goals and uh my hope is that you know in 10 years or well before then that we're a technology that helps everyone on the dev team achieve their goals more effectively and so you know if you're a tester your goals are to like cover the whole game which is very very how do you which it's very hard to do given procedural content creation ugc so we'll give you a tool to scale yourself and delegate the busy work one of your goals very often is actually to be closer to design right if you go into QA as a career often it's because you care about games you want to help them be more fun well we're going to free you up to let you do more spend more of your time on like not does it work but is it

enjoyable you know for the rest of the dev team same story right you're an animator you want to focus on your craft the agents will help designer infro engineer whatever or you you want to shape the experience you know give your players a better experience you know we'll help you there but I think broad I think honestly the 10year vision is way bigger than that where we're going to see agents permeate software permeate software development and you know the same way is like right now if you do a task on a computer you use a goey right your experience is mediated by and facilitated by you know windows and icons and that helps people who aren't programmers you know who don't like console commands be more like use computers it helps all of us be more productive there are many things that you can't do with you if you want to paint on a computer you can't do that in a command line um and so you know I think that agents are really going to be the next generation of interface where more and more of your work will be facilitated by mediated by um in

collaboration with with an agent you know an agent that and I think agents that are increasingly uh capable intelligent directable collaborative and so I think what in that 10-year time frame what you're going to see is everyone on the team maybe even without even without fully realizing it is using agents in different ways in order to be more productive more creative that hopefully you know make games that are richer richer and more rewarding uh more polished you know than uh that we can make right now awesome that's a beautiful feuture well well adding on to that I I I I think um you know 10 yeah 10 years is is such a long time in terms of like what we've seen just like in the last two to three years but but I also think you know it's it's gonna be you're gonna have these these assistants or these these agents or these you know aists that are going to ask you questions like you know what would you like to happen uh so again we're thinking about the quality of a product right it's going to be less about you know press a PR you know like does this sequence work it's

going to be is is this what we would like to happen is it is your game supposed to work this way I made this change do you like it I think it's it's entirely realistic that if you think about having agents that uh or Bots that interact with games that actually play them right but when produce uh ways of seeing what happened inside of the game automatically identifying issues not just like functional things or or you know code level things like crashes or missing assets but maybe also you know you could imagine easily that you'd be able to detect things like textural glitches or you could maybe even identify with certain constellations of things in in the game scene as it's being rendered to the player are not really exactly what you're looking for and then 10 years down the line it's absolutely thinkable that that agent would then also go back to the game project and suggest a change to it right so there's like like there could be another leap where you say like what happens after we identify the bug well then we go and make the suggestion how to change it and fix it and then the the role of the tester really becomes you know you become sort of like the decision maker in that process and

answering these questions is this what we wanted to do you're directing the process of what the game is becoming over time so I think that'll be a big part of what testing looks like some years down the line and then I think the other thing is that like experiences are going like games ass experiences are going to become even larger than they are now I bet would be my guess anyway just because we now have tools to generate more content faster and players are participating in that as well with ugc and everything right um but B Ben you know also um asks for you know caus you to ask question which player sees what part of the content so what part of content should go to who uh as we're as we're bringing people into these games right and I would imagine that the testing role also becomes part of that kind of like annotating The Experience like building kind of like a map of a game that is now so large that no one can see all of it like in one go right so what's appropriate for which person that's kind of like a natural extension of like the design intent that you have that you're building a game again answering the question you know is it good but also who is it good for yeah so I was just say I think I agree this

really you know there's that commercial from the 9s about there's the the two QA testers like hold on I've got to tighten up the graphics on level three so my vision is that you know in the future you'll just tell the agent tighten up the graphics on level three and it'll happen the true the true Vision the true dream state um all right guys well I loved hearing about your visions of the future um 10 years down the line where are we with QA um and it was such a pleasure to have you guys both on um there's so much opportunity in this space clearly um if anybody from the audience uh would love to get in touch with you either because they're looking to work alongside you they want to have a further conversation they're just a fan um how can they get in touch go our website a jk. and same thing for us it's m.ai uh so model without B amazing a model without the e all right so basically uh we concluded the episode and then we got into the breakout room and we decided that there needed to be

um an addendum couple of minutes to discuss a couple of things uh around quality of life um for QA and also around the things that um modal and agentic are not doing in order to train their players and so Nathan why don't you kick us off on your appendix thoughts yeah well so first I mean I think I'll talk to the quality of life because it's a really really important subject that you know it's very easy to focus on game quality and you know saving money but I think one of the big issues in game development certainly in my professional experience is that it could be a very very hard job right and uh there's still an industry uh with a lot of overtime a lot of stress and I think if if you think about like what are the causes of that I think very often it's because you're you're making the game and it's just barely under control and you're try to find bugs but inevitably like if you're not able to sort of scale you know your testing your Dev processes then you wind up you know bugs again they stay there for months at a time other bugs get built on top of

them um you know you finally fight and fix that first bug and then more things break and for a lot of teams the the the Crunch at the end of development is just to like get the bugs to burn down close to zero right like you um and so I think that actually automation has this really great opportunity to improve people's like quality of life like dayto day and year to year and that if you could actually be consistently exercising more of your team automating the busy work then you're going to actually like your game on a moment to moment basis is going to be more fun you'll spend more of the time doing the things you like but there's really really strong evidence that you can also keep your kind of maximum number of bugs in control which means that the process of fining the game or shipping the next season is way way way more tractable and reasonable without just sort of like throwing the candle of your life at the problem yeah and and I think maybe that's that's actually something that is most keenly seen from the QA perspective right if

there's any part of your organization that understands how how how smooth the process could be if if this was like a continuous thing rather than something that was built up and then addressed at the end it's it's probably the people who are directly like involved in that uh I just remembered that qualicon just finished yesterday with which is this like great online event um that bring QA professionals together and talk about it and there's there's a lot of the the talk topics that came up there was were precisely on you know scaling and um and automating and structuring and systematizing both at sort of like the you know the social technical layers like how do we work but also in terms of the the you know the hard technical layer what do we work with so so there's definitely that um I think that movement is happening in the industry everybody understands uh that uh that there's a potential to be gained by automation but I think a lot of people are still waiting to see it to come to fruition and and I think you know hopefully companies like ours can start providing some of the tools that can actually make that happen for folks so that you don't you know sleep under your sleep under

your desk but so so so that you don't have like those super long days better been end up you know just costing a lot at personal level awesome all right epilog part two uh let's talk a little bit about uh what you guys are not doing um in terms of large language models and the kind of academic approach um from the research companies in AI versus you guys um being more production Readiness tools for game developers well we were just talking about it I guess neither of us do reinforcement learning at scale and uh and that's because you know when when you're you're doing that kind of uh agent construction where the we AI basically learns the game from a set of high level objectives and just from interacting with the game just the cost of scaling that up in terms of for how long you need to run the game in terms of like how many uh agents or how how many Bots how many sessions you need to run how much data you need to collect uh V a very very expensive process to do and and it may make sense for certain games at a certain scale but as a

general solution problem for the games industry at least in terms of like where we stand right now it's not something that really seems to be cost effective or Within Reach for most for most Productions because if you look at some of like that really really impressive work uh that's been done with um with Bots for playing you know Dota or Starcraft or uh racing games like the the the Sophie AI that that came out of the Sony team all of that is super impressive uh and it's really great AI work but the amount of effort that goes into generating that which just put you outside like both in terms of the effort like to build it for a game specifically perhaps but also just the amount of compute that you would need to expend to make it happen would end up taking up such a large chunk of your game budget but it would be out of scale for you know 90% of the industry probably yeah sorry Nathan over to you no not at all yeah and I think you just because earlier we were talking about you know the differences between you know the agentic approach and the model approach I think there it's natural right like there's actually probably more than one approach that that works and there's

different trade-offs um but the thing yeah I didn't want to go un said was what Christopher just mentioned which is that I think that you know if you if you come to the space you know from the outside and you feel like well alpast star was 2018 like they trained an AI that can play the game so how why is this like a hard problem like why do we you know like why can't you just like take that algorithm you know wrapid API around it put a stripe payments in front of it and problems solved you know and the the reality is that you know that the work of a game developer the resources that you have available the amount of compute and data is really really different than you know a research lab that's focused on AGI especially at a a deep mind or open AI kind of scale so I think much of what our you know I think that's very common to the to the two companies that I really personally find fascinating is this you know how do you deliver AI that is useful directable productive um without you know the sort of without requiring expertise you know you have to have 10 mL phds or you know tens of

thousands of concurrent game instances um and so to me that's like actually the more interesting story is like that fundamental the invisible part of like shaping your Tech and finding a path that allows you to deliver that value in a way that's know usable useful and economical all right guys um well as always friends um if you got feedback or ideas um hit me up Alexander nov. Co I'm always open um and with that we're out uh Nathan Christopher thanks for coming on thank you so much it's been a pleasure yeah thanks so much it's been great thank you if you enjoyed today's episode whether on YouTube or your favorite podcast app make sure to like subscribe comment or give a fstar review and if you want to reach out or provide feedback shoot us a note at podcast nov. or find us on Twitter and Linkedin plus if you want to learn more about what novic has to offer make sure to check out our website

www.nic.edu and advisory Services again that is www.nic.edu [Music]