How AI Continues to Reshape Game QA

Welcome to the Navit Gaming Podcast, where it is our mission to explore the business and future of video games. We bring together the industry's brightest builders, investors, and thinkers to keep a pulse on current events, dissect emerging trends and games, share lessons learned, and have a great time. This podcast is also part of Navik's growing ecosystem which ranges from free and premium research to consulting and advisory services. For more information, visit www.navic.co. This episode is brought to you by Heroic Labs, which powers some of the world's biggest games with its heroic game stack. Its composable products are Nakama, the open- source back-end game server, Hero, the game development framework for all your live service game features, including the complexity of managing offline mode, Sattorii, the

worldclass live ops product for maximizing monetization and retention. And lastly, the Heroic Cloud to provide massive scalability. After nearly 10 years of operation, thousands of studios trust Heroic Labs to help them focus on their games and not worry about game tech or scaling for success. To learn more and reach out, make sure to visit heroiclabs.com or check out the link in the show notes. This episode is brought to you by Neon. For the first time in our industry, developers can finally have their own DTOC channels to save on app store fees and build closer relationships with players. But there's a catch. Selling direct means either building complex infrastructure yourself or choosing among legacy providers whose inexperience in games or questionable reputations pose real risk to your business. Luckily, there's Neon, the DTOC platform built for games. As a merchant of record with customizable web

shops optimized for conversion and modern APIs that let you use your existing live ops tools, Neon is trusted by some of the biggest names in gaming to help you sell direct without the typical overhead. To start selling direct and finally take control over your full player experience, simply head to neonpay.com or check out the link in the show notes. And with that, let's jump into the episode. Hello everyone. I'm your host Devon Becker and today I'm delighted to be joined by Christopher Homegard, CTO and co-founder of Model.ai and Julian Toelis, professor of computer science and engineering at NYU and co-founder of model.ai. Model.ai uses AI to help improve the QA process for gamers. Uh quality assurance if you're not familiar with the acronym. Today we're going to discuss the realities of the uh QA and AI market. And for those who didn't hear the previous episode that we recorded last year, uh could you give us a quick overview of what a model.ai does and your background, Christopher?
Oh yeah, sure. Uh absolutely. So, so

Model AI is entirely focused on helping game developers automate and augment their QA processes and their QA teams. So, so what the company does is that we apply AI to player games. We apply AI to uh look at what happens when these AI agents are are playing the games. And then I identify, you know, issues or areas of concern that you would want to lift out to your QA team. Uh so that somebody can do something about it. And once those QA specialists and managers have had a look at it, decide what goes on to the rest of the development team and you know ultimately helping people do do more in terms of improving the quality of the game uh to the benefit of their players. And we've we've been doing that for uh for quite a while. We got started uh started in this space in in 2020. And then in uh in recent years we've started taking a slightly different approach to how we do it because as as people might have noticed um how AI works and the meaning of AI has has changed a lot over the the last five years.

Cool. And your background and involvement in models of AI Julian.
So I I've been doing um I came to this from academia and I'm still um largely in academia. I I've been doing research on AI for video games since way before it was cool really. I mean, starting like more than 20 years ago. Um, and back then, you know, people did not take it serious at all. You're doing what? You're doing neural networks for like car racing and platformers and stuff like this. Like, why why would anyone want that? That that's not real science. I used to get asked like, "Oh, that's fun, but what's your real research?" And then I'm like, "This is my real research." Oh, um, and then look at me like, "Oh, man. This guy is never going to have a career." Well, yeah, jokes on them. So, you know, later on I was part of the founding team of Model AI together with Christopher and several others. Um, and we were looking into because we came from a disparate set of backgrounds including game design and AI

um and couple of and a couple of other things and we had a lot of experience in in this. We wanted to bring modern AI to video games and what we were looking for what are the problems we can solve and we pretty quickly zeroed in on yes quality assurance is the thing because um partly because it's a problem people want solved um people are generally fine with like um speeding up um quality assurance um and uh it's like there's a lot of like low value testing tasks that people don't necessarily want to do. So we decided that um which is as opposite to for example character art and so on where where it's much more ethically tricky. Um, so we decided this is what we're going to go for. And yeah, as as we'll give a little bit of overview, we we tried a bunch of things to make to make this work and we've learned a lot along the way and I think we have something

really quite cool now. Um, finally after like um after many learnings along the way.
Cool. Well, it's been uh it's been a little over a year since you guys were last on uh I think in spring of last year and we'll link to that uh appearance in the show notes as well just to make sure people can catch up on a little bit more detail on the QA process you talked about. But I wanted to kind of catch up on like since the last time you were on Christopher, what are some of the biggest uh ways that AI in generative AI specifically uh has has changed and affected the product of model. AI just like catch us up on on what's gone especially since it's been kind of a big last year I think for AI.
Yeah, definitely. Um so like in the last year and honestly you know um you could see a lot of things ramping up to that uh until it happened but definitely in the last kind of like 12 or 18 months we think that it's become really visible um that we could use generative AI to achieve something that we actually talked about all the way back when we started the company but that we never attempted because we didn't think it

would be technically feasible and and what that is is to do what we would call you know what we call blackbox testing or like completely you know, non-integrated testing of video games. Um, and so, so what that means is that, you know, the major thing that we've seen enabled by generative AI is that it has enabled us to take images from games and then use those images to make decisions over what to do in the game. Now, but but you know, if you went back like a couple years ago, our products and everything we did was based on integrating very tightly into the game engine. And you get a bunch of benefits out of that. you get kind of like, you know, direct X-ray vision into what's happening inside of the game. Or, you know, if we think about it from sort of like a self-driving car metaphor perspective, it's kind of like having a super advanced LAR system and you can, you know, measure everything that's happening. That that makes it really easy to be or at least easier to be an AI that has to play a game. But it also means that you have to invest a lot of time in, you know, putting this um software integration into the game, getting these values out. You have to

think a lot about, you know, all these different parts of it. Now what what we started doing in the last year was that we uh we decided to take the route where we said well what if you didn't have to do any of that kind of like engineering and integration work at all. What if we could just play the game from the screen the way that a human player does? And what if we could just send our input to the game using the same inputs that a player does or or simulated versions of
and and we we've been wanting to do that since probably at least 2019 and probably earlier Julian in in the research domain.
Right. I I I I I I I I I recently came across a web page from uh my days as a posttock back in 2008 or so where I were like hey I'm working on this thing trying to play first person shooter games for shooting games on pixels and so on and it is true that I was working on it. It is not true that I made any progress.
That's right. But but but I mean well some progress was made but uh but but I mean um around last year

2024 I just think it became very evident that this suddenly you know the LLMs and the VLMs were big enough they were precise enough and there was enough capability in there that you could extract information out of those systems that we just didn't consider that realistic before. And and you you know the other side of it, the other thing that generative AI has enabled us to do with the product that we have now is that it enables our users to kind of like talk directly uh to the AI agents in terms of what they should be doing when they're testing the game or in in our case writing it to be honest. But you know we we could add a voice feature at some point. I don't know. But but um but what it means is like previously we would have you know interfaces and custom things that we had built to allow you know either a developer or an estad or the QA person to tell the AI agent what to do and the way that our solution works now you write it in plain English um until we support other languages I suppose English for now but so so you write in plain English what you want to happen in the game and that's kind like

also a major change in how we've been approaching this product for the last you know 12 to 16 months
it's like you know when we started this 2019 2020 the stuff we're doing now couldn't have been done there then there wasn't the technology to build on and in fact the technology it probably wouldn't have been possible even two years ago maybe one year ago you know so we kind of right now the way we're building the product we're riding on the frontier of foundation models um it sounded really good didn't it you know she should should put it on on an ad somewhere but but the good thing about this is that project only get better and um and but it's also clear that you know yeah it could not have been done um uh and uh we should have asked the investors for uh you know an 8-year runway originally right
there would have been perfect timing uh but you know better late than never uh but but you know I also think from a product perspective but now we talked a

little bit about the technical parts of things and motivations and all that one of the things that you gain when you have a blackbox testing solution and something that can you know play the game just by looking at it take instructions just in English is that you broaden the range of people who can contribute to that you know to that process of auto automation in game development and so suddenly this is not necessarily just for the you know game programmers and engineers and people who can like you know have direct access to codebase suddenly you have a much wider range of people who can start automating what happens in the game production process and they all bring different expertises into that they can contribute in different ways and so it you know that's that's something that's really exciting to us as well that we can bring it to more people.
Well, it's cool that you guys are able to make that shift because it's obviously when it's a QA tester testing the game or a player playing the game, they're not integrated into the code and they they don't interact through the code. They they watch the game, they input the controls and things like that. So, you're doing it sounds like a better job now actually simulating the end user

and that I would imagine can result in a better QA process. Um, obviously, you know, not exactly the same, but definitely something that better mirrors either a QA tester or a player. Um, but you know, speaking of like those changes, looking back, Julian, uh, you were talking about some of the the things that made progress that didn't. what assumptions from the sort of previous generation like you know maybe prior to last year uh or going through the last year assumptions from that sort of previous era of AI before the generative LLM stuff got really good turn out to kind of be wrong and which turned out to be correct especially the stuff that you've benefited from or the or the hurdles you kind of had to overcome to get here. So basically within model we've tried like a hundred ways of doing this and outside of model we've like in academic community tried a thousand ways to play games in general and I often when I sum this up you know and in in that we we are on our second edition of an AI in games book um with by me and Yorganakis who's also involved in the company um we basically catalog the different ways you can use AI to

play a game and it turns out it differs completely depending on the characteristics of the game and the characteristics of um what you have access to, what what is the information you have access to? Like do you have full state information? How is it represented? Is it pixels? Is it objects? Is it simulated sensors? Do you have a forward model so you can simulate things? Um and and all these things there are much easier and and much harder ways of doing this thing. Um and if you can if you have a simulator, it's like a forward model of the game, then things are way easier. So the chess really is like the paradigmatic like you know simplest possible case. You have like full information. You have a discrete game state that is like um easily processible um just a few bites um a fast forward model. You can simulate millions of moves. All of these things. Modern games is not like that. You don't have that. You can't simulate the engine that fast. um the the core loop is kind of like you know um

um in it intermingles with UI elements and all kinds of stuff um and it's not even really well defined what the game state is. There's like a whole list of like you know things that make it hard. Um so for a long time we tried various things of like looking at you know how could you make an integration that integrates an engine even supposing that you don't have a forward simulator just gets structured information about the game in a way that is useful for um a model um or a search algorithm trying to play the game. Um and this turned out to be really really hard because one thing is this one learning we've had um again and again and again is that all games are different and not only are they different they're different in different ways of sometimes subtle things with the mechanics and even like how the UI plays into the to to to the game loop and so on really throws you off. So in a sense

what we're doing here is just like we're like you know this is the Gordian not approach we we we we we're taking here of just basically like nope skip all of that and basically just work from pixels. Um so I think a lot of the learnings we have about like what kind of information we need um they still apply you know you need certain other information in an first person shooter then you need um for for a matching tide game for example but we are not building those sensors because it's just too expensive to do for each um for each game. We've really learned the hard way that you can't spend like um tens of thousands of euros of integration on each game. Um it um it doesn't work that way. But we do have a lot of a lot of learnings that we brought with us including how different game genres have very different kind of

um uh affordances and and and things that are important to look at. And these things scale like you know matching tile games do behave similarly in different ways in terms of what information you need and what and and and at what kind of scale of time you need to operate and so on. And they differ that differs a lot from thirdp person shooters for example. And sometimes you see people coming new to this field um and basically oh we can have one model that works for both of these and I'm like sure try that call me when you're done tell me how it went.
So I mean something that I'm I'm really curious about you mentioned you know different genres and and different affordances things like that and you like the the customers not having to directly integrate the game into the code. Um are are they still having to explain to the AI how the game works? like what kind of prep work do they have to do? And also how do you handle those genres that are maybe brand new or mashups of genres where the AI may not have any preconceived notions of how to

play it? Like you know I imagine the AI like basically understands a first person shooter and how that works and you know genres that have been around for some time that may be trained in the model at least in like conceptual understanding how do they handle this is this is new stuff like the the we're going to have to explain to you AI how this game works so you can test it.
Yeah. I I mean uh I mean concretely the way that we do it maybe maybe the place to start is that at the moment we're trying to be pretty specific about what kinds of games that we support not really based on like how much um information is available on the internet about them. It turns out that you know for most game genres unless it's like super novel there's usually some information out there that will have been crawled by the frontier models that they can talk about at least at a high level. Um, but you got to think about like all components of the system and kind like the speed of how quickly some of these models can make decisions inside of games in a cost-effective manner. And so, so for that reason, we've been spending this year mostly with mobile casual games. And they are they're super interesting to work on

because they are if you don't play them they are more complex than you'd often think and they require sort of like some thinking ahead and a good amount of reasoning and a lot of them don't require you to have you know Twitch level uh capabilities in terms when you're of of playing the game. You can take a little bit of time to make a decision or even if there's some real time aspect to it you might be able to play in a number of steps ahead and then still buy yourself some time for the whole system to think. Um, so that's that's kind of like one one way that we've been addressing this and now we're scaling it to other game genres and moving in other directions, but that's that's where we've started. Um, I would also say that uh the other way that we handle kind of like differences and complexities is that we provide the the AI models both the big ones uh that we can only get you know from uh from from large server of ours but also some that we set up ourselves. We provide them with um you know um specific context about the game in question and descriptions about the game. So you can like you provide a little description about you know what is this game, what

is this about. We might help the customer do that uh as well if if they want us to. Uh but it really is all text uh at least for now. Uh we may be adding more more more modalities later. But we add that in there. So you say like oh this is a game about this and that and you want to make sure that you keep an eye on these rules and if this happens you definitely want to let us know because that's not you know according to the rules. Um and then the other big part where we do some some game specific tweaking is for what we call or what is called I guess the perceptual part of the problem. So that's just being able to see you know what the heck is happening on the screen where is it happening and uh when is it happening and being able to do that fast enough. So we we still train our own models. Um but where we used to train it, you know, back on the real paradigm, it was like much more in the behavioral side, a lot of our training now goes in on just being able to see and act accurately inside of these um inside of these games.
Um yeah,
it's um it's interesting to point out there that um there's really is a role for like multiple different models,

different um at different sizes, right? we have like um I mean there's the large models that need to be do all this reasoning and then you have the models that you can tune for very specific um circumstances it's kind of an intricate game latency cost and so on.
So talking about how you guys are integrating it now um what technical breakthroughs I mean outside of just the general LLM uh stuff that's been going on the last year or two uh made this sort of vision-based approach like doable for you guys. Uh I obviously you said Julie you mentioned like maybe within the last year maybe the last two years was this even a doable approach. What is it specifically that made this doable? Is it computer vision stuff? Was it the LLMs? Um what what kind of tech made this this doable?
I I I think I think them coming together. So the video language models or multimodal models. So um um the large models from the leading providers including of course open AAI and um and

Google so Gemini um and also a bunch of open source um models like Quen there are like natively multimodal they can take images in as well as text and images and text and combine that. So that gives us reason because you have to think about the test they they simply they have seen so much essentially. So they can recognize so much and you can do some kind of um visual reasoning um by combining the text and the and and and input. Now it has downside as well. Um one of them is that they're slow, right? If you if you got to send your query um over the network to one of these large models, it's going to take time. So, you're not going to be you're not you're not going to be able to act within a second. You're most likely going to take several seconds to act. Um which is why you may want to combine this in dispersed with local models that are fast models you can run locally and and that that respond in much less than

a second. and also use models that are pure vision models to um identify and locate various objects. So here's like this this is a complex dance and a complex orchestration. Another issue is that these large models and this is I'm this is December 25 and I'm saying this and it's true now. We might be different. They are in general very good at recognizing things and also even understanding relationships between things. But they are weirdly bad at um spatial reasoning things as like you know oh here is a carrot and here is a flashlight and the flash is under a table and basically how do you need to move the carrot to put it on top of the table? Um, and and these things you it it often just fails. It's it's it's like it's like it it it knows about the objects, knows what they are and so on, but can't figure out how things relate

in space. Um, and then you need to sort of so basically get around that by orchestrating smaller models to locate things and maybe sort of um have like a separate little path finding routine and so on. So here's where it gets it's like the devil is in the details really. I I mean and I think like following up on what you're saying Julian like nine out of 10en times or even more often kind like the highle plan that you're going to get from some of these models if you orchestrate them correctly and give them the right information from the game is just going to be super reasonable like like that that that part like works really well if you put it well together. But then like you're saying making a timely decision and understanding where things are that's that's usually where you have to do a little bit of um extra work. Yeah. In order to make that work well. It's not at all obvious to a person who's not working a lot with these models what they're good at and what they're not not good at. That's the kind of jagged frontier thing, right?
Yeah. Yeah. And and I think another thing that um that's worth kind like pointing out is where where you can gain a lot is by also like especially this

has really come through this year but if you can orchestrate the different models not just in terms of like oh I got to call that one or I got to call that one that's important right but the other thing that you can do is that you can crystallize artifacts out of the models that you can then use really quickly later and so what do I mean about that that's with that I mean that instead of like having to use an LLM every time you have to make a decision you can use an LLM to generate a little piece of code uh that can do that every time from the outside of the game, but instead of like quering a model, you're just running something that you know is going to work, right? Or if you need to be able to recognize something inside of the game, rather than asking a super big uh VLM every time, maybe you ask it, you know, 20,000 times uh ahead of time and then you store it in something else and then you use kind of like this cache response to improve the performance uh when you're actually when you need to perform kind of like at runtime. Um, but that's also, you know, the way that we build our models and data sets for stuff is kind of like based on both before we're playing the game and when we're playing the game and after we're playing

the game, we're sending all this information uh around, you know, for different visual models and only textual models and we pull it back in and we train something. And that's also something that's only really like come together this year, I would say. Uh, at least for us anyway. So
do you guys think then um that maybe the next big advance based off of what you're saying Julian about spatial reasoning and things like that and just the the ability for you know the models to learn off the games uh that world models could be the next big leap in terms of this tech or is it something else you think?
Great question and I wish I had like the simple um answer. World models simply mean very different things to very different people to different people. Um I mean there's a tech courses like Sach Genie 3 where you basically simulate um simulate kind of 3D you you basically simulating it produces walking simulators in fanciful environments for you. That that's a way of putting it. Um the issue with it there's several issues with it. One is that it's slow and

others it's unreliable and so on. But even if you assume that um um you can get around this, you you you get the you get them coherent, you get them like you know promptable um and controllable and fast and all this kind of thing. Then you have the issue that they are making very strong assumptions on the world. Then we get back to what kind of games are we playing. So world model like nothing you can produce in G3 has anything like the dynamics you get in Candy Crush. So um then you need a world model. Then you can imagine like can we get a world model of things that are vaguely like matching tile games. Um you know oh in this world we get BJ world and Tetris and Candy Crush and Dr. Mario what whatever. Um but that's a completely different problem. I mean now now I'm now I feel I'm putting on my researcher hat here as in like I want to do this. I want to build this. Um this this motivates me. But um um I

think that's pretty far from being useful for um uh for what we're building at model. Um we we need something simpler. We we we need models to understand that if an object moves to the left then it will be to the left of where it was before and things like this. Yeah. that I mean I mean that's that's the level and it's like again if if you don't play around with a lot of these models it's kind of like it's it's it's really hard to sort of see the relative capabilities where like you know they can be amazing at figuring out like you can get a lyrical description of the background scenery you're in and like you know intricate in detail sort of you know description of like the objects and and and what they could be used for and so on. But then it still can't understand that you know um it needs to move left to go left or something.

Yeah. Well then uh I mean going back to you know how it actually is used in practice in the current products and things like that. How do the actual game studios integrate it if you're I mean we we touched on it lightly but they're not integrating it in the code level right so what what do they do assuming it's a genre that you guys are working with? Mhm. Yeah. I mean, currently the way that we that we work with and have worked with our early customers is that uh we ask people to point us to their app store build if the game is already live. And then we're going to show the first results with that in the first meeting. And that's that's kind of like a neat thing about this blackbox approach. You can show up and say like, okay, this is what we were able to do with your game kind of like out of the box. And and really and that's what integration looks like. It's it's uploading your binary. Um and then we take it and we run it in our infrastructure with the agents and we get results out of it and most of the results are derived from you know uh the video uh of the game playing and whatever logs that we can pull from the outside. Um so you don't have to do anything special. I mean for some games you maybe want to set up a particular

user account or if you have you know some some hard and soft currency systems going on maybe you want to have some tests where it's great to have like a million billion uh you know jewels that you can just like spend as you're testing the game. Um, but it's it's it's it aligns much closer to the practices that QA teams are used to having when we're setting stuff up for testing it. Um, you also have to produce some text that tells you what the agent should do. Uh, so the agent that plays the game. Um, and in that case, you can probably start from the test cases that you have already or you can come up with uh with some that align with that. And we've kind of like come up with our notion, our own version of that. We call them tasks, but but at the end of the day, it aligns, you know, alongside either kind of like you can have a task that's like a highle description of something that you want tested. And that could be like, oh, make sure all my menus work, but don't go to the store. Or vice versa, you know, make sure the store works, but don't spend time on the settings menu. Or it can be kind of like very specific, you know, lists of sequences and of stuff that you want to have and kind kind of like anything in between. Well then, um, you know, having this easier

integration obviously like it's great that you guys can do stuff ahead of time uh and and it enables a lot of like accessibility. I'm kind of curious then on the output side and on the capability side, what is this newer approach, you know, with the vision based approach versus the code integration approach, what does that enable? Uh, and and what is that maybe stronger at in the QA side than the code integrated approach, which I imagine also has some advantages as well. Um, like what does this enable? What does this let people do that they just couldn't do before?
Yeah. Yeah. Maybe I can Well, I can start at least, but feel free to jump in. Julian, I guess we could start with the strengths. And I think this is uh at least from our perspective and something that we uh that we see that, you know, studios and people working on the games find very appealing is that there's uh there's ecological validity to it. So the results that you're going to get out of testing your game with this technology are going to be very close to what you would be getting if you would just hand it off, you know, uh a phone or a device to a human tester and ask them to play the game. Um so there's

sort of Yeah, go for it Julian. Sorry.
No, no, no. I'm I'm just saying that there is like sort of just like the wider view is like it's a common problem, but like when you use various game playing algorithms that um you might get good game playing behavior that is not humanlike. I mean it's like common on like chess and go and so on. Um we've seen this in you know top level play and like ways to do some weird things and we've also seen this a lot with like this discussion about reward hacking in learning to play video games where they learn really strange strategies um through through reinforcement learning. Um but here because basically I mean the auto reggressive training of um large language models isn't exactly you can't really call it super or you can't really call it like imitation learning but um the but the way they understand it is more the VLM's

understand is more like what humans would do it and and and and the kind of action suggestions you get is or that you humans doing humans humans come up with because they're trained on human data. So that is um largely why it is true what Christopher says that it's it's more humanlike behavior. Although now when I think about it, I'm not entirely sure that this completely explains it. It's empirically true though that it it gets more humanlike than you would get with for example a search algorithm.
Well well actually well actually I wasn't talking about the agent behavior. I was talking about the game itself. Uh but you you're absolutely right but I was kind like thinking that the thing that actually gets tested the artifact that gets tested has not been modified to support the agent.
Right. Right. Right. Anyway, right.
It's true.
Yeah.
Nice. But what I said was also true. Right. Yes. Yeah. Yeah. I said that this is the after the podcast. We got to get back to this and try to sort of
I'm curious though. Does it ever

actually watch players play to to learn anything about how to play as well?
Uh we we don't do it at the moment for anything that we're we're we're we're selling to a we'reing stuff
for the game. Um
yeah,
and we have like a whole pipeline for it and so on, but right now we're trying to go to do away with it or like to to not use it when we don't need to because that's essentially integration time,
right? I'm just curious useful to generate more humanlike behavior in certain situations where maybe the LM or AI doesn't really do that normally.
Yeah. No, it could definitely be useful for Yeah. for that. Sorry, Tony. Go ahead.
Although right now good behavior is more important than humanlike behavior for most games.
Yeah.
Yeah.
I mean that's like that's actually like there like three different things. This is like playing game well, playing human like and also playing in a game in a sense in in a way that's useful for testing because the testing you're often sure you need to be able to play the game, you need to go through it, but you

also want coverage. Um, and that is slightly different. Um, but yeah, um, it it will it can definitely help to train on human behavior for the particular game, but if we can do without it, it's preferable. Can you actually tell the um the AI since you said you know you can kind of communicate with it. Can you tell it to purposely try and break the game to do things that like a a bad actor would do in the game? Uh you know I don't try and play the game well try and screw it up. Try and do things that you shouldn't try and break the game as much as possible.
Oh yeah, definitely. Like a lot of the a lot of the tasks that end up getting entered into our system go in that direction because you want to be testing for some of those edge cases or you just want to make you know you want to make sure that everything works. Uh we've also you know we've we give the agents full OS level access. So if it's like on an Android phone it can use the keyboard and it can you know could go to different app and then go to play store or whatever. So, you know, we've also had it rename itself and, you know, try to change the account password and stuff

like that. And you obviously you could put some guard guard rails around it, right? But all all of that is available kind of like by design, but because you want to see what happens when the agent, you know, turns off the internet uh to the game mid session, right?
Nice. Well, definitely definitely helpful. Um, you mentioned earlier about like this something being something more accessible for non-technical people as well and maybe even non QA people. Do you see this expanding out uh within companies and and things like that, the the sort of QA role or the ability to do QA? So, say for example, right, when um when you're developing the game and you're one of the coders on it, it's very helpful to be able to then run the code that you just built and test and make sure that that thing work before sending it to QA. Is this something where you like increase the accessibility of QA processes to other people? And what do you see that like what direction you see that going uh in terms of helpfulness for for companies? We we'd love to be enabling that way across different disciplines. I will say what we are what we're putting out there has you know the QA folk in mind. So

like the typical user for us is kind of like Northstar is like serving a QA specialist or a QA manager. Um, but there's no reason that other disciplines can't like pick this up and use it for you could be interested as a designer in like how often does something occur perhaps or like you were saying if you're like an engineer on the game, you want to make sure that we spend a lot of time in in this part of the game. Um, and any, you know, if you have anyone with an account on our platform could could do that. I suppose you're not limited to uh to to being on the QA team necessarily. Well, Julian going to uh for for more between somewhere between practical and ethical question is uh how you draw that sort of line between like augmenting what humans are doing the quoteunquote giving them superpowers and the replacing human part which I think is especially in the QA department been a bit of a big concern you know with unionization and things like that going on. So I imagine that's something on you know people's minds listening to this like where where do you see that landing especially with your product but also just the general concept of using AI and

QA? honestly is something that we think a lot about and it's something that I mean also outside of QA. So when we went for QA originally one of the reasons was that we thought that this was the one this was one of the fields where people were less worried about being essentially replaced because um fewer people take that kind of see it as a domain of their own creative expression. Right. But then again, um it's a job and I mean taking trying to take away people's jobs is not something to be done um without serious thought, you know, basically applied to it. So what what what what we want to do is basically to give people leverage to basically um be more effective testers um and more effective at QA. Now, um, and you could argue that, okay, if if if the machine

does all the QA, what what would would human do? No, but human sits on top of it. And here's the here's the crucial thing, though. Almost all games are nowhere near tested enough. I mean, we need way more QA. We just don't have the resources. Um, so what we see as what we're doing is that we're basically giving giving testers superpower. like basically instead of like playing this kind of you know um repetitive sequence 100 times, you're basically telling 100 agents to go around and go and do it and report back to you afterwards. Um and and you know and then you need to do this every every time someone changes something in the game. Well, you just click this button and there you go. And instead you spend your time basically doing highlevel analysis of the results and also thinking about what should be tested. Um like basically we're seeing this in other kind of AI augmented field like you know the the cognitive work moves

more into deciding what should be done like when you're kind of using AI tools to program for you it comes more in like what should be implemented and so on
and and if I can add to that Julian I actually think you know from my perspective there you do need a lot of creativity to do like good QA and hopefully this could kind of like enable you to unlock more of Because a lot of the like yes you have to do like a bunch of repetitive and you have to test everything works and you have to go to all the parts of the level and all that stuff right but the kind like underlying that you know once you also remember that you need a lot of kind of like lateral thinking and the kind of like creative holistic systems understanding both to figure out like where do I go to break this system what have I seen and haven't I seen and so that is the thing that I imagine the human operator would be spending more time on right and then also kind like diagnosing things. So rather than having to collect all the evidence yourself like through, you know, by playing the game, you can just you get handed like, you know, kind of like a detective

mystery case and you you like you start putting both, you know, different clues together out of the gate. Um, and that's that's part of our vision with it anyway.
So I mean, how how far can this go in terms of acting like a real QA tester in the reporting side of things, right? talk about you know the the sort of QA person started deploying these like a bunch of workers and bringing information back but how far does that information go like can it actually uh identify like reproduction steps here's how you would reproduce it here's like here's a video of it here's uh like sort of the root cause we think you know or is that left more to the human side of things of like hey I found uh bugs uh you figure out how this all works off like a recording or something like that
I would say some of it we pulled together for a user already so if If you give us a list of steps, uh the system will check for all of those different things happening. If something different happens, it will uh tell you what happened. Uh we're also increasingly working on pulling together multimodal bits of information. So we take the

video and we take the instructions, we take all the thinking that the agent had and then also you know the game log if we can access it and anything that happened on the device and then we can like make all those available in the reporting and we try and use that to pinpoint the different moments that a human uh expert human tester would want to know about and then summarize it. So it's kind of like in two levels. We have like all the individual indicators like here we saw something happen this minute go to this video you can correlate it to the log and we had a performance spike and then we start looking for patterns in all of us and we pull them together and kind of like you know highle reports for for the user and then sort of like on from there deciding what happens next. That's that's where we currently say that's where the human expertise comes in and you decide what flies and what doesn't fly what gets sent on and what doesn't. Um, of course, you know, it'd be super interesting in time and I think eventually it will be feasible to take some of these like higher level summaries and then say, oh, actually, you know, in nine out of 10 cases, this

is the typical fix for an issue like this and here's a, you know, a post that goes along with it. Maybe you want to send that to the dev team and maybe you want to send it off to some other AI agent that could make a pull request to your repo and then somebody could decide if that's what we want to do. So how do you see this fitting in then with other AI uh tools like so for example on the report writing side right like maybe you know there's a human involved in identifying a bunch of stuff and and collecting that information but then AI writing the report or uh you know dealing with tickets like how do you see this fitting into the broader ecosystem as people look for different places and different ways to integrate AI and it's almost sort of like idea of AI co-workers where there's people with sort of different AI departments handling different aspects of it. Where does this all fit in? Yeah, I I I think this is a process step that kind of like was missing before, but you know on one side of this we have kind of like the um you know CI practices and producing the artifact gets tested in the first place and that has to go together with the tools that people are using already with their testing testing expertise to decide what actually get tested on this kind of artifact and you've got you know

our system in the middle right but the output is also information that then needs to go you know needs to go to uh Jira where people might review it or it goes to some documentation system or it goes straight to the code repository. But I, you know, I think it's key for a solution like this that it's like super interoperable with all the other systems that people use and that they might want to connect to this. Like we we don't see any reason for like once you've run a test with us and you've executed your game and the agent did things and we collected data about your game. Um, and we we make conclusions off of that. uh a product like ours should just make that available for all the other places where it would be helpful for you to plug it in because at the end of the day at the end of the day it's kind of like about maximizing the leverage that this automation gives you and then let people do as much as they can with it. Imagine this going really far like in different ways you maybe one day we'll have a command line interface you know there's cloud code and model model you know test sort of right CLI
where it sort of integrates into that sort of broader envir AI environment so

to speak
yeah yeah yeah I mean we we'd love for this to be like imagine if this generates an ongoing knowledge base about the history of your game and where your game is now and you can actually query that done right into that you can ask you Oh, is level six broken? Right? And then the system comes back and tells you, oh, these are the reasons that I think level six is broken. And now you might become more interested. And then you say, okay, how long does it take you to complete level six on average? And then you start using that data in interesting ways, right? If you have something that's playing your game all the time, you collect so much information about it, right? And so you can imagine just like asking questions straight down into that and getting you know information back that you can use
and you can look at what the test you've done in the past and saying hey I saw you did these tests on level six before like you know should you do this you know you want me to basically run those again and do slight variations and stuff like this and and and while you're anyway doing something else um have it running. Yeah, this is like since the very beginning um of model we've had

this vision that like you know you should as you're developing there should be agents playing your game all the time you know keeping keeping the servers warm and and feeding data to you. This also comes out of like academic prototypes of this we did way way in the past basically that is like you know real time real time agents playing.
Yeah. I guess I'm curious where you see the management side of this going like cuz right now you know you're talking about oh this is kind of deployed by maybe the the QA team leads or some of the maybe more experienced QA testers who kind of know how to direct uh what kind of testing to do. What do you see like do you see this going pretty easily into the the a QA agent manager that's AI managing the other QA agents that are AI to like like you said run that 24/7 where it's like hey we're off for the weekend but the but this this AI over the weekend is just sitting there managing itself directing different tasks knowing which coverage needs to be done or figuring out additional coverage

checking in test like I could see obviously there's a lot of potential for orchestration to be pretty huge in this do you guys see like potential to grow this for example this product out to that or to integrate maybe with other products that help manage that as we kind of touched on a little bit earlier.
Yeah. Yeah. I think there's a lot of uh and maybe that isn't even the end game but that's that's kind of like a place where you could end up through something like with this right but something that's continuously generating new knowledge about your game. And then again we're not you know um we're not limited to just finding bugs off of it or kind of like you know concrete issues that would need to be addressed. You could also use this for balancing information or personalization information or assessing what would be good for different player groups. Like suddenly the possibility space expands a lot from being find issues to so not just talking about you know um whether it works but kind of like what works well and what like aligns with the vision that we have and you start giving people design relevant information kind of like up and down that axis. Um if you think about it that
cool a lot a lot of fun and possibilities I imagine. Uh Julian, then

u given how much has changed in the last year, 2 years, I mean I guess all the time really, but especially in the last couple years, what do you see happening in say the next two years from sort of the technology standpoint that affects all this? Whether that be like specifically your product or just QA sort of testing in general, might even be outside of games, uh but just general sort of QA testing. That's possible with with the SEC in next next level. It doesn't sound like world models probably within the next two years would be super useful, but what what else might be? You were saying you were saying before the endgame, you know, just like and I was just thinking the endgame, there is no endgame. There was a Marvel movie called The Endgame, you know, and then they just kept coming out in movies afterwards. They just kept getting more and more confusing, you know. It's like
both of us are based here. I would I would like for this to not go the direction of the MCU. Let's let's hope it's more positive than that. Yeah. Um it's um I mean the spatial reasoning

thing obviously on the technical side is is is is a really big thing. Then there's something as simple as speed. I mean speed is um a real issue for us right now. Um like we we constantly navigating the trade-off between analysis quality, reasoning quality and speed. Um and and the more that solves itself, the the better the results are going to be. Um there's at some point going to be a test um a cost um issue as well, but this goes hand in hand with the speed essentially. Wow, this sounds really really boring. I'm just like you you're asking for this visionary stuff. usually like talking speed and cost and you know but but honestly these are major issues for you for for for us and there's a lot of stuff we could do in theory that just doesn't fit in a production workflow because speed and cost. Um um we are really excited about like what we could learn across games. Um like so as we said before like you

know all the games are different but within the genres in particular things that could um that could um um generalize and questions such as like you know if you learn something about testing strategies for like um uh match three games will it carry over to merge type games or like um falling blocks type games and so on and can we use that to basically suggest the interesting actions and so on. Then another thing that you know again looking at the very rich um literature of things that tried in academia but like are far from like applicable in in production so far. Can we look at player styles? Can we have this levels tested by played in a lot lot of different playing styles like here's different player archetypes? What which kind of issues would be faced by um um a pro player or versus like an amateur? And of course like going beyond does it work or not like giving like complete experience reports of of the

games like hey I played this part of content you're proposing here and I think it this is going to be interesting for like novice players but kind of frankly a bit boring for pro players or like hey I play this and you know I think there are too many options and few people are going to look explore these things like uh there's so many things you can get you can go into here and and then you know having being able to even like what kind of behaviors would would happen, what kind of players would be attracted to this. For example, um at some point there is the um I mean and this is some this this isn't I'm taking off my academic hat, you know, because it's we might get dirty. Then you have to connect this to ad monetization. Um, can you use this play through data um uh to predict which ads should be shown or which ads you should show? What kind of players you want to um you want to attract by through um

understanding the game through automatic playthrough analysis? um almost certainly you can I mean and if you are the company doing all this testing then you'll be sitting on a lot of um on a lot of like data about like um um game play through profiles and what player archetypes will work and will not work and then you could use that to do interesting recommendations about um for example ad placements and ad buying and maybe I'm saying too much now Christopher tell me if I should um know I I think Ben's about to ring here.
It's really interesting to think about that idea of a AI focused group uh like QA testing for subjective
instead of objective problem.
Yes. No, no, but but but obviously I mean I mean it's like this thing like does it work and is does it have obvious bugs is really really just scratching the surface of everything you can get through player analysis

and and again like you know people who want to second guessess us and figure out the next moves can dig into the many dozens of papers we have in our academic selves written about the about the topics. plenty more additional recycle.
This is like a whole heap of ideas there like you know that we just barely got working for one simple prototype game but like you know at currently is far from working in a generalizable way in any kind of production environment but but we know that in principle it can be done.
Well, cool. Well, hopefully hopefully it'll be pine
and of course and of course and of course there's the integr the more direct integration into into content pipelines like you know um okay I see that this I feel that level six because we've been talking a lot about level six today.
It's my best level so far
level six doesn't have good content um for like um um for exploration focused players. Should I do you want me to

suggest this kind of labyrinth puzzle here and so on? It's gonna be like Chad JBT at the end always suggesting like you want me to make a table of that? You want me to make some slides for you? Yeah. Yeah.
Exactly. You know, which is like so so I'm thinking I'm thinking more visually just pops up like hey how about this puzzle here? Do you want to put this here?
Okay. So we like to we like to finish things off here with just just a few quick questions just to get some quick bites for fun. Um so so feel free to be a little controversial if you'd like, but we're just going to go through the three questions real quick. Um, and the first one just kind of an example you can give of an unexpected success or like an emerging behavior or something from using AI and QA that's just like whoa what what what just happened?
Well, the uh the overall unexpected success is that it works as well as it does. like who would have thought that um but but uh like one thing we saw recently was we an agent that entirely fall you know uh failed to follow our instructions and by doing so you know showed uh it exposed them as a bug in our system and then at the same time

cost triggered a bug in the uh in the in the customer's game as well. So I think in many ways their bug showed us our bug and our bug showed them their bug h and we weren't really counting on that. So the question is like who's really testing who.
Cool. Good answer there. Uh what areas and I know you mentioned the jaggedness earlier. Do you still see AI unexpectedly struggling outside of just like the genre thing?
Sure. Um I think we all I'm always surprised what they can remember and what they can't especially for kind of like long running or long horizon tasks. They'll they'll remember really arcane and unexpected things and then they'll you know forget whether red means on or off for a button and you kind of have to work about both of around both of those things. So, you know, smart, but maybe maybe not necessarily smart in bandwidth.
Some coherency issues. Seems to be a constant thing with these. The more complicated they get, the less the shorter the memory seems to be. So, that makes sense. Um, and this feel
better off they are.
Yeah. Feel free to get a little controversial on this one as well as some shots fired moments if you like. But any recent games uh that you've seen

that really could have benefited from AI QA coverage.
Oh, you know, I don't want to call. So, what I actually have seen are some games that have been out for a decade that could still benefit from some QA coverage. So, so I I think nobody goes free here. You can you can get a title that's been in live ops for half a decade and you'll still find stuff in there, but people either know about or even didn't. Uh so I think I think we're we're all in this picture and we have to decide if we like it.
Maybe you'll have to get some people use it for remasters, right? So when they go to remaster the game and know about a lot of issues, they can get make sure they get some good coverage there and have the remaster be a better version. Nope. No, that's right. We'll we'll test both and then if you want we can compare which uh which one had fewer or or or or more bugs in them. Yeah, that's uh we we'll develop one for free.
Cool. Well, hopefully it leads to better games, right? That's what this is all about. Hopefully, at the end of the day, as your product grows, as other products go, just the whole AI ecosystem grows, it leads to better games and not hopefully worse games. But I think

there's definitely some positive future here. And it's great to hear what you guys have cooking. I imagine a lot more cooking under the hood that we'll see over time here. So for anyone who wants to check it out, uh it's model.ai, but no E in there. So just modl.ai, of course, be linked in the show notes, but for those of you who are just listening, I want to make sure people know where to find you guys. Lots of cool stuff. Lots great to see the the technology be unlocked as well from these uh new changes uh in vision. So but I want to thank you guys for for joining us for second time for you, Christopher, first time for you Julian. Really appreciate the conversation. a lot of uh fun stuff in here. I'm sure we could keep going forever as this tech is continuing to emerge, but I want to thank uh everyone listening as well. Hopefully, you had a good time. It sounds like Julian is suggesting there's a lot of reading material out there for those of you want to go further down this rabbit hole.
Oh, so fun. Or have AI help read it for you. Yeah. So, it's like so but uh
yeah, but then you'll have to curate the AI subs to see what it does.

comprehension tests. See? Cool. Well, thanks guys and we'll catch everyone uh on the next episode. In the meantime, have a good one.
If you enjoyed today's episode, whether on YouTube or your favorite podcast app, make sure to like, subscribe, comment, or give a fivestar review. And if you want to reach out or provide feedback, shoot us a note at [email protected] or find us on Twitter and LinkedIn. Plus, if you want to learn more about what Navic has to offer, make sure to check out our website, www.navic.co. There you can sign up for the number one games industry newsletter, Navic Digest, or contact us to learn about our wide-ranging consulting and advisory services. Again, that is www.navic.co. Thanks for listening and we'll catch you in the next episode.