Steal This Strategy — The One Top Studios Use to Save Millions

[Music] Hello world. Welcome back to the Pocket Gamer Podcast with your hosts, me, Brian Baglo, and the paragon of presenters, the arch deacon of AB testing, Miss Peggy Ansults. Hello, Peggy. It's great to see you. It's been far too long. It is great to see you, too. I mean, we have had a wonderful pipeline because we have had tons of interviews from Pocket Gamer London. So, yes, we've taken a little bit of a break from sort of our get together show. So, it's great to connect and uh be energized. Here we are, you know, um and uh yeah, it's great to see you too and uh winding up and getting ready for all the shows ahead. Indeed. Absolutely. Absolutely. I feel like we're moving into the the beautiful daylight hours of 2025. It's sunshine in Scotland. It's a bammy 4° centigrade, you know, so you still need

a hat to go out, but um it's it's good to be back. It's really good to be back. And what what a show we have lined up. Yeah. I mean, you mentioned the AB testing. I mean, testing is it, right? We know it because everyone on the show says now it starts with a great game. It does start with a great game. There was a time when you could sort of like start with a great campaign and a mediocre game. Uh, but those days are gone. Those days are gone. And that's why we're deep diving into game testing, you know. But what is it? What does it mean to test a game? And more importantly, a first ever report that I found that is just amazing because it's one thing to say this is the way you test a game, but it's another to say this is what top performing studios do. This is how they do testing. And that's what separates them from the rest. Right, the leaders and the also rans. So we have two industry insiders who are going to tell us about this report and testing best

practice. We have Jack Dunn, VP of research and operations at Play Test Cloud. Great to have you, Jack. Very nice to meet everybody. Excited to get going today. Absolutely. And I'm excited because we have John Wright back, former VP of mobile at Quali and currently I would just say mobile influencer and overall great guy. Sean, I mean, yeah, let's just stick great guy, part-time bum. That's that's what I'm going with right now. Okay. It's part-time consultant, part-time, full-time dad. That's that's what I'm going for these days. So, I'm enjoying it. Lovely mix, John. We've had That's never a bad thing. Absolutely. Welcome, Jack. And welcome back, John. It's It's great to have you both here, John. Good to see you again. And before we dive deep deep deep into the into the testing side of things, but um you're both kind of serious athletes, you know, I'm making me think that my uh my career of sitting on my butt talking to people on a laptop may not be right for me. So

Jack, I believe you run ultramarathons and are going for a This can't be right. 300 mile trek through the desert post GDC. Yeah. And John, we know you're into your strength training. So Jack, what? Yeah. So, that's the uh that's how I'm going to decompress after uh after GDC and uh yeah, going to go run through the desert with some friends for a week. That's de like what is the temperature going to be? Cuz it'll range Yeah, it'll range between probably 30 35 degrees Celsius during the day and get down to get down to like almost freezing at night. It snowed last year uh on the same run. I didn't do it last year, but yeah, it snowed last year in the desert. Uh, so we'll be running through the Mojave. Uh, very very excited about it. Okay. Well, I'm I'm I'm I'm glad that you're here. It's I We'll have to ask you back on the podcast post run if he survives. This pray for now. I was not going to go

there, John, but uh yes. And and and John strength training. We know that you're you're well into this. So, does that have any sort of impact on the rest of your career as a part-time consultant, full-time dad, part-time bum? No, the the the only thing I'd say is I've had more time recently to train, which is always a nice thing rather than, you know, spending 60 70 hours in front of a screen every week. Uh the flexibility of spending more time with my children and and and obviously choosing when I can work out and work out a bit more and stuff like that. So, I've been enjoying it. Um, and and yeah, just just uh enjoying the new dynamics of being uh, you know, part- timerr rather than full-time exec. Well, it's delightful to have you both here. So, uh, I'm looking forward to this one. Strong mind, strong body, strong ideas. We want to get to play testing, right? Remember Marcus Aurelius said that the body is the the h or houses the soul. So, strong body equals strong mind. So,

it all goes back to that. And I have to go to a little bit of a pun because I thought yes, let's take the heavy lifting out of testing. Oh, that's bad. I know it is bad. Uh, but we will dig into the latest report that you have over at Play Test Cloud. Love the title, by the way. This is great. I'll tell you, steal this strategy. Right. So, breaking down exactly that, that strategy. What are the key habits that successful game studios do and what makes them stand out? So I'll start with you Jack about what stands out for you in this report. Yeah. So the the things that stand out to me in steal this strategy is that we really validated a lot of the core assumptions that we've had about play testing and play test cloud over the last couple of years. The biggest ones are in and around that kind of multiple different types of testing. And then there's been something that we've known studios have tested for and publishers have tested for over the years, which is trying to find out if people are playing their game longer than the ordered session is. And we were

finally able to put some big data behind that and see that that core assumption was true. Um, I think the other thing that stood out to me in the report is going through 10 years of play testing data on any platform uh is going to be very challenging and it was a lot of work for a lot of people and it's absolutely fantastic that we've finally gotten to this stage and we're really excited to talk about it today. A lot of work indeed. I mean let's just park that for a moment and deep dive into that alone because um the methodology here I mean 10 years of data obviously a ton of work. share a little bit of that methodology that helped you basically confirm some of the most important points we need to make that aren't just common sense anymore. Now we can say that now we have a foundation we can build from. Yeah. Yeah. So on the methodology side like we've we constantly talk about how many studies that we run uh through the platform. So last year it was 5,400 uh the year before I think it was 4,200 something

like that. And we've had this kind of desire to look into these numbers more. And finally with a bit more bandwidth within our own research team as well as a lot of support from marketing we decided to really go deep into this and try to understand how a section of our clients are using the platform. Uh that section was the kind of the highly successful studios the ones who are doing really well in the market. Uh they're part of big acquisitions. They're performing really well uh just across games and across genres as well. I think another big part of it is at Play Test Cloud, we test everything from, you know, very casual games right up to the most uh in-depth 4X experiences and now on PC as well. And uh just kind of breaking all of that down over the course of a couple of weeks was um was really invigorating. And to see that what we've been telling clients through our own kind of gut feeling or maybe uh maybe vibes based uh stuff was all kind of coming out of this report and coming to light. It was

really, it was a really fantastic experience to go through and now we know I mean we sort of thought it but now we know that um testing is better than doing nothing. And the best studios well they test at multiple stages of the game development you know it's not just at the end to see oh do we have a commercial success so I mean that was very very intriguing. Yeah, and that's something that we've seen happen uh kind of to your point at the at the start of this like um you used to be able to start off with a mediocre game uh throw a bunch of UA money on it and hope at the other end when CPIs are much lower. I know we're talking specifically about mobile here. Um and that's just not true anymore. Uh you need to have a game experience through both a fun aspect or like a gameplay aspect and also a usability aspect. now, like if someone downloads a game and can't understand what's happening within the first 15 to 30 seconds, they're just going to delete it and go and move on to the next thing. So, um, we're starting to see a lot more

studios, whether they be the very successful studios or not, start to tap into that multiple different testing types. And uh I think that's going to be something that we only see more of in the future as we constantly see these bigger studios and publishers wanting to innovate and uh come up with new different testing methodologies to understand what their players and what their competitors players are doing. Well, think about it, Brian. We're going to have as many ways and approaches to testing perhaps as we have genres and subg genres of games. I mean, it's going to be you nearly probably customize your testing to your game. Who knows? I think it makes huge amount of sense and everything that I'm hearing here, Peggy, is just really feeding into so many of the people we have on the show, which is there's no oneizefits-all approach anymore. Every game has to be treated, you know, as as something unique. And it's that the processes that allow you to gauge every level of design, pre-production, you know, production, development all the way through. Um, and figure out what's working, what what you need to focus on, you know, what what you're missing

because we have the most discriminating and discerning audience in the world out there. And the cost for them moving on is practically nothing. So, if you want to get your players engaged, keep them engaged, and keep them coming back and build some kind of loyalty, you need to know at every stage that you're not leaving them unsure, uncertain. So, I'm absolutely loving this. And Jack, I'm in. I want to find out more. Awesome. Well, I'm I'm interested in that because John, that's uh that's something that you have also seen from your your own experience. How does this resonate with how you did testing or how you saw testing because it's about, you know, making those decisions early, you know, always killing your your darlings. Um, what have you seen? Yeah, look, the the industry's definitely gone down a qualitative rather than quantitive approach. Um, you know, I think I've been well documented saying this over the last 2 years that there's been this sort of like tectonic shift in the

mindset of a developer, especially around what they build, why they build it, and and the structure of building it. So, you know, as Jack said earlier on, things like a mediocre game with a high marketability and low CPI, you know, a year or a couple of years ago would be successful. um that's not the case anymore, right? Like users are looking for much bigger robust experiences. Things like metag game are now like the standard, right? Like you you can't launch a successful game without having like some form of in-depth meta that will a break up the experience so the user is not just getting hooked into the core loop, but b also has additional content and different engagement points to drive someone back time and time again. And if you look at Monopoly Go or or Royal Match, they have like racing games in in their in their experiences now, right? Cuz they're trying to tap into something that's completely different to the core loop. And there's a lot of these

different approaches. But but ultimately, what this means is that the prototype cycle of games has gone from like let's say for hyper casual it used to be like two weeks, right? Now, as it's evolved into hybrid casual, we're now seeing 3 months, which is the standard to get to an MVP, right? And that's that's obviously 6x 6x the time, 6x the production cost, 6x equally things like economy and everything else is much more complex. So, you need more people. It's not just small teams anymore. You've had to grow your team and your team is working even harder than it's ever done. And look, because of that and the additional production times, the cost of the projects go up, which means that you have less shots on goal per year. So, it's extremely vital that you test at multiple stages through the production because otherwise you could find yourself 6 months in and potentially a million dollars spent on something that you could have found out

after 2 months was not worth it or you would have found some critical information that would have pivoted the development. What you don't want to do is wait for six months and then realize, oh, the play test told you a load of information that you could have known after two months because then that's four months of money spent. That's like 300 uh 3/4 of a million dollars in this example. So yeah, it's just it's just so vital these days to be able to test at every stage throughout it. And there's also the other component which is developers are blinded by their own games, right? Like it, you know how many times I've sat down with some of my teams and gone like this is fantastic because we are like the upper echelon of people who know how the game should work, right? We are the people that know how games work. We're gaming professionals and things like balancing, right? You just because you've built the game, you think, "Oh, this ain't that hard." And then you do a play test and you the people are like nine out of 10 people saying, "Oh my god, this is so

difficult. What is going on here?" like and you're like, "Oh man, I had no idea." Right? So these these are all the important things that you know ensuring you know um a variety of play tests through each stage of the production will will not only help you directionally get the product into the best place, but it will also effectively save you lots of money and and because it's never been harder to launch a new game, you want to save money where you can, right? Oh yeah. So yeah. Yeah. It's big topic and and the report bears that out. I mean it says this is what successful studios do differently. So I did want to just walk through a couple of those habits those you know incredibly effective habits of successful studios to take us back. Um and one thing we they do and we've been talking about it all the time so far frequent play testing iteration. They iterate faster. They see better results. They don't treat play testing as a

oneanddone process. Now, we've been talking about it and John has been doing it, but Jack, what is the evidence for this and the lesson, the takeaway for our audience? Yeah. So, I think the it's it's not just that they're iterating faster, it's that they're iterating at all. And these successful studios are constantly trying to figure out every single part of their game. uh they might be testing the first-time user experience four or five times over the course of six or seven months and uh just trying to understand any little friction point that they can find within there to that makes the that makes the game harder for the player to use. And then once they get into that point where they've got a lot of content, they'll start to test um test longer uh parts of gameplay. So they'll start testing up to two hours, maybe up to a week uh worth of content as well. And they're not watching all of that video. they're going in there and they're seeking out specific parts of the gameplay to try to figure out is this working? Is it sticking? And is my product going to do well in the market? Um, we see an awful

lot of times we see people do uh when they go into either soft launch or open beta or worldwide launch, they'll also run a longitudinal study uh with us over the course of a couple of days to see qualitative insights into their quantitative data. So they're getting all of these metrics pouring into the data analytics team that's saying that hey at the third day 15 minutes in uh 20% of people are just not playing anymore. And if you rely on those you have no idea really why that's happening. You just know that it is happening. Uh when you add in the qualitative insights from a platform like play test cloud you're able to see exactly what happens at that point and you can intercept people and ask them about it as well. It's the it's a core part of what makes a lot of these big launches successful or it helps people understand why their big launch wasn't successful. I think that's been a huge thing that we've seen over the last over the last two years in particular. We've been part of some really big launches of games that have been in development for

a long time and they know what didn't work later on in the process because you can ask about that one feature that you introduced that you thought was so amazing and actually it's the one that's making them leave or you've done it and no one's actually engaging with it and you think it's some sort of like hero product or hero feature and actually it's got like really low engagement because of something you never thought about. Right. Yeah. that's hidden for it's hidden four little uh red exclamation marks deep in a in a menu somewhere. That tends to be the main one. So, so this is this is making a huge amount of sense and I think this is this is something that far too many developers out there um tend to fall into the trap they test their own work as you were saying John you know I I made it I played it it's really easy but also you have that ugly baby syndrome where no programmer wants to imagine that their work is anything less than perfect so we know that you've got some some really kind of good firsthand experience on how

strategic testing has saved time and money So can you give us a few examples or maybe just unpick that for us a little bit? Yeah, so I think um again if we go back a couple of years ago it was especially if you're focusing on like hyper casual rather than like midcore for example, right? There was there was people always did play test but they kind of did the play test at the point of like we're going to launch this or not, right? like they they developed the core loop and and there was a mindset of okay, we're going to do a play test now as a okay, this is the final kind of step and maybe we'll get a little bit of information, but the play test isn't going to influence us launching the game, right? Because we've already got some markability metrics and we're just going to get them the game to to market as quick as possible and and unless the the play test comes back with some sort of like critical bug that's you know we've never noticed before like uh then then it would have been impactful. Uh but now you know if we look at at games outside of hyper even in hybrid which obviously is the evolution of hyper and we look at casual and midcore

you know play testing throughout the production cycles is just so important right so we we have milestones you know from MVP up until hard launch right and each stage of that like so if we go from MVP then retention test then monetization test then soft launch hard launch you know each of The sprint cycles can be anywhere between two and four weeks. So those sprint cycles and in a milestone you could have multiple sprint cycles, right? There could be two to three months between milestone. So if you're doing sprint cycles of two to four weeks where you're adding additional content effectively, you're adding new things to the game that can either improve results or worsen them. And you as a developer more often than not because you've got an amazing product team, you've got economists, they're sitting down, they're jamming, you know, it's uh digital jazz and and then you know they they they talk themselves into this is going to be the best thing ever. And in actual fact,

it's not. Uh but they continue to to preach internally that it is because there is a bias or a biased nature to to what you do in developing your own games. and the ugly baby syndrome like you said Brian. So you know it's really really important that product managers and game designers in particular look at companies like Play Test Cloud as as really like a a massive tool in their tool set to be able to get unbiased information and and to potentially pivot the direction of what they're developing features or productwise. So, it's just so important that again throughout that those different milestones for those different sprint cycles like again if I'm doing a four-week sprint cycle for that's normally for a casual game it's four weeks for a hybrid game it's two weeks but I would probably do a play test every sprint cycle right because there's new stuff being added that you could get value from from an outside person looking at it um and equally if you think from the way the development works you might add something in one

sprint but then there's other sprints afterwards that will focus on more development of that particular feature. So if you don't do a play test at that initial point, you could then you know double down on it and overdevelop it when actually it has no impact. And uh yeah, the fact is regular play testing highlights all of these things and and way developers should look at it very simply is this will give you you know directional information that will influence the production which in turn will save you money from going down the wrong path. Right. How much how much money John have you uh can you share now that you're no longer at Quala you can maybe tell us you know Oh I can I can tell you I can tell you that. Okay. Uh, I can give an example. I'm not going to name the game, all right? But I'll give you, this is a real world example. So, there was a game, a puzzle game that we worked on and we spent 3 months going through to development and then we and

it had meta in it. It had uh 500 levels. Um, uh, but but we missed out something very critical, which was we focused on adding additional content, the levels in the meta. We did not do enough due diligence or work on the on boarding and the fatoule. And what we saw is we went to market after spending probably about a quarter of a million in in development um of this to get this game to sort of MVP and we went live and it had a 12% day one retention and we were like, "Oh my god, what is going on? This is terrible." Like we've just spent, you know, a couple million dollars on this. um why the game looks fantastic. All of the meta is there. we should see a really good retention. And then we did um then then we did we actually did a play test before the end and the play but but there's one thing about doing a play test and there's nothing about listening to it and and ensuring that

the information is passed and Jack can go into that you know later I guess about some of the pitfalls that developers fall in in terms of like doing play test and they're not actually actionable or or actioning the the information. But um yeah, what we found at the end of the uh the play test was the fatoule was way way way way too difficult and that we would have known that from week four and we could have corrected that and focused on the fatoule before launching and saved oursel a couple of months in in additional development that we didn't have to to to do or we would have re rep prioritized which in turn would have saved us six figures of some description. Wow. Okay. And that's not even the kind of most extreme example of that. Like the first time user experience and changing things around there isn't isn't as hard as the kind of compounding effects that you can have in

like a more midcore game or hardcore game that has more in-depth control schemes. So, with the the other end of those studios, with people who are making the the more complex game mechanics, uh we'll see them testing controls an awful lot earlier because if you mess that up at the start, you're you're goofed uh longer down the line because you're changing core game design elements far too late in the process. Um, I think another thing to maybe call out, John, on what you were talking about there was, let's say, like back in the hyper casual days or the way that some people still see play testing is actually more as a QA test. Uh, what they're looking for, what the developers or designers are looking for in there is, hey, like is there a bug when someone opens this menu as opposed to is this person enjoying the experience that they're having? And I think that's one of the one of the biggest misconceptions with uh games user research as a whole actually. This this isn't a QA function. Uh we're not here to just raise up uh random Jira tickets for people to work on over the course of two weeks. We're

here to find out why people are playing your game the way they're playing it. For a lot of our listeners, maybe that that's not something that they've thought about. So, can you maybe unpick some of the different types of testing, you know, play testing, functional, you know, and and just really give us some insight into the different types of testing that you guys see studios who are succeeding doing. Yeah. Yeah. Absolutely. So I think one of the biggest differences that we'll see with uh successful studios and studios who are um I don't know as Peggy I said uh running the race is that um successful studios will take take one type of research question for each play test as opposed to trying to ask like a hundred different things or come out with a 100 different outcomes uh in one test. So, a successful studio will come in and they'll say, "Hey, we're going to spend uh 30 minutes today figuring out the onboarding. That's all we want to know. If we find out something about monetization or level design or something, that's a lovely little bonus, but we want to have uh in a report at the end of the day, we want to have five

green things about the onboarding and five red things about the onboarding that we need to go and fix." Um, then they'll go into testing longer form uh stuff like I mentioned. So they might go out to two hours at that point. See what the returning player experience is like. See how the game is managing the player's frustration in certain levels as well. And then the kind of last the last methodology that people will tend to use then is that uh longitudinal type. So you're uh getting people to come in over multiple days to see exactly what they're doing. Uh you'll survey them at the end of every day and rely pretty heavily on that survey data too. One one of the final things to call out on that is the successful studios that are doing this don't necessarily have research teams. Uh we've seen some of the kind of most in-depth and most innovative research that we've done actually has come from product teams and game design teams uh who are willing to just take on the mantle and figure out exactly why players are doing what they're doing. Um, and I think one one of the things that uh that a friend of mine, John Hopson, called out in his book, The Secret Science to Games, is

there is nothing more powerful than the game designer seeing someone go through the experience uh that the game designer has developed uh incorrectly. Like that's that's what's going to make real meaningful difference uh to the game you make. Oh, listen. I have heard so many developers saying, "Yes, but you're not playing it properly. That's not what I designed it to, you know, to do. And and this is why, again, to go back, why programmers make terrible testers in some cases, is because they're not good at acting like a stupid user, you know, they know what they should be doing, not what, you know, somebody's going to do. So, it's it's a great point, Jack, and I am absolutely going to track that book down because I want to know more about I would like to think that game development is a science. It's got to have gone past the sort of the witchcraft and woo that it was back in the '90s. Oh, absolutely. And that's something uh we we can probably do another episode on that like talking about how it's gone

from how it's gone from, you know, being like games have gone from being this incredibly creative process that was really set out by visionaries in the kind of computer science world to something that still has a lot of that flavor but needs to have core kind of product research and development experience as well. Now, uh, again, like we spoke about earlier, you can't launch something mediocre or something that people can't use because it just it'll fall flat in its face. There's too many games out there. There's what, a thousand games going up on Steam every day. There's probably 10x that going up on the stores. Um, so yeah, you've got to have you've got to have something a lot more than just um just a creative vision there. Yeah. If we build it good enough, they will come. Yeah. It's it it wasn't true in the 2010s. sure as heck isn't true. Now, um, but one of the one of the biggest surprises for me, um, although it makes sense in retrospect, is that successful studios, and I'm going to come to you, John, don't just test their own games, they

test the competitors, they test the other games out there on the market that are in the same genre or the same, you know, type of game. Um, can you talk to us about this because this I I'll admit this is a new one on me. Yeah, look, again, I think it's something that's quite standard in more successful studios is is sort of reverse engineering the games, right? Like, so so you know what, example, Rolic bring Twisted Tangle to market, this rope mechanic that people have never seen before. Um, and you know, it it does exceptionally well, takes off, right? Then you saw how do you know you've found success is when you've got 10 copycats in the store all the you know and there's there's an age-old sort of uh thing in in gaming innovation or or which is like you've got pure innovation which is like something completely new then you've got a plus one innovation which is taking something that already exists and then

making an adjustment of innovation or adding something to it and then creating arguably a new experience. Right. And I think um a lot of the time, you know, when you get true innovation, then you get this plus one innovation, you know, spurs off the back of it. And a lot of studios get inspired, right, by by these new things. And we as a company would, you know, test games like this because, you know, it's something that we've never seen before and we were, you know, intrigued by it and we wanted to understand it. Equally in the last couple of years because of the rise of meta in games after you know hyper casual not having meta understanding meta and the different sort of things like do you want to have some sort of leaderboard or a treasure chest or one of these other comp a world builder travel mechanic like all of these different things that you have like some of this stuff is just we as a company just never thought about and and obviously we're playing other people's games and we get inspiration of it but we have no idea if if it's just a biased

opinion of we enjoy this, right? Or is it something that makes an impact? And so understanding your competitor's games in an unbiased fashion is extremely important for a multitude of reasons. You know, a few of that I've just just mentioned, but yeah, it's something that I think all medium to large studios are doing and I think it's something that the small small to medium studios probably don't think about. But yeah, having someone deconstruct like doing deconstructions of of competitor games, doing play tests of of competitors games, these is this is how you understand what's what's going on in the market and how you can take influence and incorporate some of that inspiration into your own games. So yes, it's a really really important part of thing which again I don't know if everyone outside the big boys does. Yeah, I'd like to tag on to that. Yeah, I'd like like to tag on to that one a little bit more. I think the um for a long time

people would have looked at big data platforms like Sensor Tower and said, "Okay, cool. These are the competitors that are doing well. Uh let's get two or three people in the studio to go and play that game uh over the course of a week to see what's happening." We've seen studios tag on players like real players to those experiences to understand a lot more about not how does this mechanic work, but what resonates well within it. Um, one of the games that, uh, we unfortunately don't work with, uh, but I want to call out because I've been playing a lot is, uh, Archer 2. And I, uh, I posted a couple of days ago, uh, is there anything Archer 2 can't do? They've had three different live events over the past, it's probably around a month now. Uh, the first one was when they implemented Monopoly Go into their game as a live event. Then they put in a fishing event. Yeah. Yeah. The dice rolling. Yeah. Yeah. Yeah. Yeah. Uh then they put in uh a fishing game which was very similar to what uh 10 square games uh used to make with um with fishing clash. And then uh right now they've got a kind

of digging miniame which has been basically I most of it's been ripped from uh royal match there I think. And uh this kind of like plus one innovation or finding out other ways to keep your players engaged and keep them in your game is keeping me in there for sure. and it's uh bringing people back for different reasons as opposed to just the core loop. Uh having these kind of different experiences to play. Um and yeah, it's been uh it's been really fascinating seeing other studios testing that type of thing. Uh whether it is those sort of mini games and figuring out what mini gamess are going to work well in the market or if it's more around the core loop of a newer game like uh it's in the report last year the uh the game that was tested the most the competitor game that was tested the most was Monopoly Go. People were looking at this thing and like how how does this work? What's resonating with players? And you can see that it works on sensor tower go you see the reports and billion dollars in two weeks or something insane like that. Uh but understanding the why about why people are check

yeah um but understanding the why about why people keep coming back to that uh is super important. I mean, what's coming through here is that you're turning testing into competitive advantage. You know, it's Intel. Um, but that requires a strategy and approach to understanding how to actually test competitor games. Is there anything that you're doing over there as well, Jack, where someone can come in say, you know, a studio and say, "Yeah, I want to test against competitors, but I just don't want to do like sort of look at a list at what's crushing it in sensor tower. I really want to understand and analyze what's working." I mean, can you say, "Yeah, then test your game against this or something because it is a strategy. You can't do this sort of without deep knowledge of the competitor landscape in the first place. So do you do some some sort of consulting there and say, "Hey, we think you should do this or look at that mechanic." Yeah. So

most of the studios that are approaching us about that have a very good idea of their own landscape. Um like uh I I say this a lot, mobile games come from business plans. And when you look at a a business plan for a mobile game, you're going to see, hey, this is the market. These are the target audiences that we're looking at for this. And here's some games in that landscape that we want to be taking players away from, right? Uh are there any new players out there anymore? Who knows? Uh so they would look at maybe let's say a game like Monopoly Go and say hey we want to steal players uh from Scoply. That is our core goal. So we need to understand uh why are they playing that game the way that they are? Um what's keeping them hooked in? And then how can I take them away? What are their motivations to continue playing and what would make them switch? On Play Test Cloud, you're able to talk to people who play Monopoly Go. whether it's people who have downloaded the game and played it twice or people who are really highly engaged within it. Uh so being able to talk to those players whether it's through uh through a play test or non-modderated test with a couple of survey questions at the tail end of it or interviewing

them or doing a what we call a moderated play test where you'll actually watch the person play in real time while you're on the phone to them. Uh and then you can extract so much information from them there in like two days. This doesn't take weeks. Uh I think that's something really important to call out here. If you want to get just a you know finger uh finger on the pulse of this you can you can get this span up in like less than 24 to 48 hours. Uh like results will start coming in for single session play tests after like after 20 hours I think is the kind of the average at the moment. I think it's such a a good distinction as well from what you said, Jack, about like the fact that you have access to different types of players. Like you you have access to players that have maybe played never played the game and have to download it the first time. You have access to people that have played the game for years. You have people that are gamers, you have people that are not gamers, you have male, you have female, you have all of the different age demographics. Like

because you're not just going to test with, you know, 20 year olds versus 50 year olds because, you know, maybe your your core demographic is 35 to 55 women, right? And if you've not got and I feel like, yeah, having that variety is something that you won't you can't get in a studio. Yeah. Like I I can't get that, right? I have to use an external partner and and play test cloud bring that variety of user to me and ensure that the right users are testing the right games and it's just so it's just even more impactful right like the information I get one other thing that I want to get to is because we're talking about gameplay testing gameplay but um Brian as we both know it's all going beyond games it's oh yes it's games are everywhere they're in everything. So, you can also be testing monetization models, ad experience as part of the gaming experience. Um, when games and brands get closer together, this can get very, very interesting. So, I'm

wondering, Jack, what can studios do or learn from testing and what you're doing in gameplay testing to improve these experiences? Do you see that coming? Oh, absolutely. Um we see we see a lot of studios uh a lot of the a lot of the bigger studios that have like big market insights teams uh they'll be testing ads very frequently through us uh either through like our concept testing tool or they'll actually be seeing how it works in the game as well. When it comes to testing maybe monetization experiences we could be looking at stuff like testing battle passes. Uh so that's a that's a very familiar one for us. Like how do people interface with this? Like can they use it? Uh I think a a lot of this comes down to can they use the thing. Um it's like kind of a core pro uh product research uh mentality that we have to have in there. Um but we see people test uh across the whole across the whole gambit of uh of different things there. Yeah. Look, at the end of the day, if the if the unit economics of the game don't work, then the game

doesn't succeed and you've just spent a year building something and a million dollars, $10 million, you know, on something that that just doesn't work. So, I do feel like as part of the testing process, it's not just the Fatoule or looking at the meta or or again the feature set. It's it's understanding the the buying behavior of the users because it will influence the percentage of of paying users as a as as a key metric, right, for like casual games. How many people out of a hundred are going to go to an AP within what time frame versus, you know, XY Z? And if you can do some some play testing and get some some input on how maybe to engage with payers earlier, Mhm. that's a massive win, right? It's the same as positive cash flow. The earlier you get the money in the better. So yeah, it's really important and it may be something that people don't actually think about when they're doing play testing, Peggy, if I'm honest. Right. Cuz again, say you're working in a hybrid casual area

where 50% of your revenue is from ads and and 75% of that is from institial. Then the no the nature of a monetization manager will be okay, let's do increase the frequency. Oh well, what we'll we'll monitor the retention to make sure that we're not blasting people and then loads of people churn. But in actual fact, you know, if you do a play test and 10 people say to you like, "This is the most aggressive ad frequency I've ever seen, but you need to tone me it down." If you're doing that in an early play test versus when you've got 100,000 down, right, that can that can make or or save you lots and lots of money. So, it's really important to factor that into as part of your playbook when you are doing correct play testing. This is just speaking to me. It's saying it's taking out the guesswork of every part of your of your game. You know, we're not putting a finger in the wind and going, "How is this game going to make money if we increase the frequency of ads? Will we turn people off?" You know, if we change ad providers, ad markets, you know, are we are we going to start seeing things? We had a guest on a few shows ago who was talking about quality

of ads. Um and and everything that you guys are saying about the move away from QA which is seen as a you know a final step before release into a more iterative I was can we say testing culture I think just it makes so much sense in the highly competitive markets today. Um, and just to touch on that h side of things, Jack. So, we've been talking mobile, but the report also mentions a move into PC and the sort of I used to say the mainstream market, but mobile's mainstream now. So, PC is very niche. Yeah, I don't I don't know if I don't know if if people will agree with you on the uh the niche end of PC, but um yeah, we uh moved into the PC market in November of last year after about a year of product development and research ourselves. Uh so we were really practicing what we preached uh with that. I think the big interest here maybe would be testing crossplatform

games. uh a lot of mobile studios are seeing that as the next key revenue driver for them whether that's through the web or through steam as well. The other thing that we've started to see people test is web stores. Uh so we'll see people test their web store uh on PC to uh skirt around uh the Apple and Google Play policies and uh understand is that experience usable? What do players think of that? Because if you can move like 2% of your paying users over to a web store, you're going to be in a much better position financially at the end of it. That is very intriguing because now we're seeing that there's going to be room for much broader testing strategies because we talk about web source and we'll have some guests additionally Brian who are who are really moving out in this direction and what they're seeing. I mean that is new territory that has a lot of testing. I mean, it's like, you know, this is it. I'm naked. This is my web store. This is me. Does this resonate? There's nowhere. There's nowhere to hide, basically.

Exactly. So, can't blame it on on boarding. You put yourself out there. This is the way you are. Um, and exactly on that point, John, I'm wondering from your perspective where you see play testing heading in the next years because we're talking about crossplatform, PC, web stores. I mean, yeah. You know, there's an app for that. There's a there's a test for that. Yeah. No, seriously. Like I said, if there's probably half a dozen plus reasons why you should play test now, there's going to be a dozen in the next coming 6 to 12 months. Wow. Um I think what Jack said are the big ones. Like crossplatform is becoming much more common and you know anyone who's played Genchin Impact, you go from your phone to your PS5. I did that for a long time. So, I think um you know we when when I was at Quali, it was one of our long-term goals was to find a game that was a crossplatform success and we had success on PC, we had success on mobile, we had some games that

converted very um converted in a porting way to Switch and did really well, but we never found up to to the point where we would have a crossplatform success. But crossplatform is going to be the future of gaming, right? in the next couple of years, this is where all of the big boys are going to go. So, giving it accessibility to play it in multiple ways and forms and on the go versus stationary, it's just so important, right? And it's something that only few developers have have have actually cracked to date, but it's going to be where the future is. And then the web store stuff. Completely agree, Jack. Like, let's be frank. And I I'll call Apple out and Google. It the duopoly is coming down. The walls are coming down. The DMAs coming from people. Fines are happening. And you're not getting the value. 30. There's no way that we should be paying 30% just to have our games on the Apple store now without promotion, everything that we used to get. And if you're not king or dream games that bring in billions a year, you're not going to get that premium infantry and that featuring at the top. And web

stores, you know, are going to be leading the way to solve the biggest problem in gaming, which is the unit economics no longer work. Profitability in studios is at an all-time low. We're seeing closures and layoffs because we cannot make it work with how it is currently. And that 30% that's that could be the difference between saving people and closing studios. So best believe if you're not doing the web store sort of strategy or looking into it as as someone who does have a strong AP breakdown then you're just going to get left behind or you're leaving money on the table. So it's those two things in particular the things I'm very passionate about at the moment. I can hear it. I agree John. I agree John. I can hear it. I just like from each of you a short sound bite in a nutshell. You know, we know play testing is what winners do. Now we know what they do differently. Um now those getting started with this strategy, they're getting excited about it. What's the

number one action item for them? I'll start with you, Jack. All right. So the number one action item for me would be to remember that successful studios are always innovating. Whether that's innovating in terms of game design or in the different types of testing methodologies that they do. The most successful studios always want more from themselves and from their players. And I think this report highlights a couple of different ways that you're going to be able to do that. I like the way that goes back into our love of endurance sports here. John, what about you? Your number one action item. Look guys, it's a no-brainer. Increase your frequency of testing and also the accessibility to PlayStations within the company. Make sure your product managers are using it. Make sure your designers are using it. you know, make sure your QA and your insights people are using it. There's no reason why it needs to be just pigeon hole as some secret amongst one segment of people. There's value for for everyone here. And like I said, there's six, you know, six to 10 different types of tests that you can do. Each one will have different value to different people. So, educate yourself on on the

playbook, right? And and the types of testing. Make sure you've got more accessibility in the studios with an array of different people. And then just increase the frequency of the tests because the more you test, the more results you'll get and the better decisions you'll make and and ultimately the more money you'll save by making the right ones. Can't beat that. Absolutely. Test more. Test often. Um there you go. The new that's already our mantra. I should hope so, too. So, as Peggy said, we're going to wrap this up, but I I have genuinely found it fascinating. So before we go, Jack, where can people find the reports? Uh, so people can find the report on the Play Test website. Uh, there will be a link posted there. Okay. And we'll share that here and we'll share it in the show notes as well. We absolutely will. And before we go, we ask every guest a couple of questions that brings it right back into the heart of things, which is video games. So Jack, John, I know we've had your answer before, so I'm going to come to you

second, but Jack, what is it that you're playing right now when you should be testing? You know, what do you play for fun rather than work? And what's your favorite game of all time? All right, so for fun, uh, right now I'm playing Archer Row 2 a lot. Uh, like I mentioned a little bit earlier, I don't know if there's anything it can't do. I was a big fan of the first iteration of it from Hobby and seeing, uh, seeing Archo grow into this kind of crazy, insane metagame thing. Uh, was fascinating. Turned out of it for a couple of years, and now with Archo 2, I'm just really enjoying all of the all of the different ways it lets me play a game. Um, my favorite game of all time. Uh, got the Pokemon Cup here. Uh, it's got to be any any one of the the three second generation uh, Game Boy games, which is uh, Gold, Silver, and Crystal. Uh, when they added in that nighttime mechanic, it just kind of it blew my whatever I was 10 year 10 years old. It blew my mind. I couldn't believe that they could make a game that seemed so similar to the previous iterations, but had this entire different thing with the new

catching mechanic of Pokemon at Night. And uh yeah, I really really loved that it by the way. I completely disag disagree with that. Pokemon Yellow of the first generation following the anime story is the best Pokemon. And I will just rubbish everything Jack just said. He doesn't know what he's talking about. He's not a true fan. Pokemon Yellow is the best one ever. We don't We don't have to agree on everything, John. We agree on a great place. So this is getting into the competitive testing. Yeah. Yeah. Okay. There are no wrong answers here. Yeah, John. There really aren't. Jack, thank you. And and John, how about yourself? Your your opinions about Pokemon notwithstanding. Uh the game Yeah. The game I'm playing most at the moment is Pokémon TCGP, which I've completely hooked on. I I've completed I've got 100% um sort of colle of the collectibles of of all of the different sets. I probably spent like 300 quid on collecting all the cards because like Jack, I come from a generation of being

obsessed with it when it came to to the Western world. Being 12, 13 and getting Pokémon cards for the first time was like a mind-blowing experience. Um, and my favorite game of all time is Final Fantasy 6. Yeah, you can't beat GTA London as well. Best one ever. Okay, I that that was my baby. That was really That's Yeah, that that that's another podcast. another podcast. But gentlemen, thank you so much. It's been an absolute joy. It's been a pleasure. You know, we said this to you before, John, and we made good on that promise. But Jack, it would be lovely to have you back again. So, it's one of these topics that I don't see getting any less important in the coming years. I told you we we're at the cusp of this, Brian. This is this is going to be massive. And it's great to understand what the winners are doing because then you can aim for that. And hey, I have to end on a pun. We had a winning team of guests to tell us about that. So, um, thanks again. And you know guys, our prows will be

crashing. This show is all about how to do your job better, how to make an amazing game, how to market it, and you have a say. So, if you have a story or know someone we need to shine a light on, then we would love to hear from you. We want to hear from you. We want to reflect the reality of the mobile games market in all its wonderful complexity and strangeness. So if you have any suggestions for us, if you have any feedback for us, you can always get in touch. You can email us at podcastpocketgamer.biz. You can find us on Twitter @pgbiz and you can reach out to us through the pocket gamer.biz website. If you're interested in listening to all of our podcasts, you can find them at pocketgamer.biz/mpodcast. And we would love to hear your thoughts on future shows and we've got you covered on all the major platforms. So subscribe to the audio podcast as Brian said, look for us on YouTube. If you want to read it, hey, you can do that too because we have a

companion post for you as well on the pocket.biz website. Tune in again for the next edition of the pocket gamer.biz podcast and we look forward to speaking to you in the near future. Until then, I'm Brian Baglo. I'm Peggy Ansalt. And that's a wrap. Until next week.