Playtest to Success with Christian Ress, CEO of PlaytestCloud

welcome welcome welcome everybody to the deconstructor of fun podcast today i have christian res ceo and co-founder of play test cloud a service that i've been using how how how long would it be i would say eight years probably eight years old together how long has has played this cloud been uh in business about eight years yeah yeah well that's pretty much because if i would have said 10 or 12 i would be talking about usertesting.com which i used before playtest cloud and the playtest cloud kind of came in as the de facto testing platform for games because usertesting.com was kind of like a generic testing platform so when this first came out and shout out to antihatra who's uh who's the um from uh from starbury games who's uh he's an advocate back then he was working at luda muga and uh and very much an advocate for playtest cloud in the finnish mafia so he distributed this uh with uh within the finnish mafia of

mobile gaming and um i think pretty much every company here uses it so um really great to have you on this podcast because now i'm putting a face to a service uh i've used for such a long time yeah thanks for having me uh hey mishka it's good to be here and yeah you know you mentioned auntie i think like a lot of this wouldn't have happened without him like he was wondering this is gonna make him think too much about himself like like this would have happened definitely without anything he certainly helped it was good to have him by our side yeah he was actually my first boss like he was my ticket to gaming so he hired me so um i'm always grateful to hunty but i just don't want his head to uh to grow too big because of all the all his recent and future success we'll be careful yeah how's it going christian yeah doing well uh you know we've had like you know berlin weather and like now it's switched like we're sunshine sunshine and sunshine and uh it's uh

wonderful actually because we've had so many grey months here we have a blizzard oh no yeah well good that we're indoors i guess yeah yeah well you know it's it's finland we we don't we don't proud ourselves with uh with the weather it's a defensive weather people that you focus on making great games yeah there's nothing to do outside that's true stay at home and play yeah suddenly yeah so do you want to just kick it in or do you want to like yeah let me know sure um where do you want to start uh why should people use play test cloud like you know why should developers and publishers um what's the benefit of it like i know i've i've been for me it's just like uh i don't know it's like it's like google sheets or a powerpoint it's just part of the tool part of the uh the tool i don't kind of even mention it because i feel

like it's almost automated but i remember when i when i moved to us and i was working at a corporation and um i've noticed that at large companies because you know finnish companies are pretty small in terms of uh size of popular size and size of population social sciences but in terms of uh in terms of how many people are working on a particular game or in a particular company they're quite small so so we kind of did our everything ourselves and that's why services like playtest cloud were de facto and like no-brainers but then going into a corporation they're like what do you mean like we don't do our own user tests like that's why we have the 20 people that are working in the user research department and we will bring in people that we can wash behind the glass and we can do the same thing behind the glass in houston and in in other uh you know middle american cities where we can understand a bit of better players i was like why like that cost 75 000 when we can do a test that costs like 60 bucks and do like five of them overnight without

without you know big issues anyways like yeah but this is me just telling my perspective but what's your perspective yeah i love it i think like the underlying thing about play testing is that we're all in the business of making games that want to give players a certain experience like either we want to have them enjoy something that's super fun we want to make a game that's super challenging but regardless of what direction the games go into there's always a vision and like a design for the experience player should have but how do you find out if your game is actually living up to that and how do you do that before the game is live because then it's too late like if you wanted to make this super scary horror game but you're launching it and people are like ah you know this is like i'm not like feeling it you just wasted two years worst case and uh that's really where we fit in so the way i see playtest cloud is that it gives you know game designers producers but

also user researchers tools to find out if the player experience is actually what they wanted to be and you mentioned like small companies big companies we actually see people on all ends of the spectrum using playtest cloud like we've got some shops like one person working on their indie game using play test cloud to make sure they're getting it right but we're also supporting like some of the biggest game studios out there who have these departments with like your 20 user researchers or something but they don't want to deal with having to schedule people coming to an office and i mean look at the last few years that has been difficult anyways but even before then you want to have user testing play testing not be something you have to worry about not be a barrier it should be easy if i need 20 people who love shooter games try my multiplayer game out tomorrow night

because that's when the new feature is done i shouldn't need to schedule this thing for four weeks i should be able to go and you are you can go to players cloud and just say hey get this done here's the build here's what i want to find out and then you get the videos back so why should people use play test cloud i think if you're focused on your players you want to give them a good experience ignore play to scout you get a play test you got to find out if what you're doing actually gets the job done but tools like ours make it super easy and we want to remove these barriers because everybody is you know so much needs to get done and like if play testing is a chore people don't want to do it we want to make it easy we want to make it fun to do fun to see people play your game yeah yeah so i was kind of summarizing the how i how i feel about it it's like easiness is definitely one of them it has to be easy like you have to be able like like with let's say

sensor tower you can easily look at how much game is making what are the downloads you don't have to do too much work but but another part another three things that i like about sensor about playtest cloud is as you mentioned the velocity so basically how fast it is to get the test done so you have to schedule it because that was the thing that was happening with the on-site tests as well as external um audience tests of course there they tend to go much deeper or they couldn't can we can go pretty deep with play test cloud we're going to talk about that as well but you have to schedule a time you have to do a lot of ground work before you can get the tests done with this one it's rapid it's like the overnight essentially the the third part which is really important is that you can do a lot of targeting so you should talk about shooter games that's an example but you can do anything basically you just look at what kind of games is your target audience playing and you can target based on that you can target based on on on you know the games they're playing the demographics the uh the countries the platform so you can do

all of that really quickly so you're making sure that you're getting your play your game in hands of the players you're intending to acquire at some point which is something that doesn't always happen when you're doing on-site tests and and also like with playtex cloud you also get some misses you're like this guy doesn't know how to play shooters like what is he talking about but at the same time it's so fast to get five ten of them that that you just you know kind of rapidly go through it and versus like having that one person come into the office and like okay that was a waste of time uh and the final one is you said videos and especially the format of the test is quite powerful so instead of sitting behind the glass which i've done too many times at a weird hours also this one comes in uh in the video format because it's recorded as the player has been playing it at the most convenient place where they are so you can see where the tabs are happening and most importantly you can share very easily the results you can just share the video with a bookmark so usually when you start play test play testing you're trying to find different

kinks and quirks and issues with with usability and being able to share that video and that especially like at minute 32 seconds you can see the player trying to search through the menu and being unable to close the menu and getting quite frustrated that is going directly to your i don't know ui engineer or whoever is in charge of it they change it a little bit and lo and behold that issue will never be arise and if you would be quickly launching your game you might experience low d1 but you won't know what the issue is uh but now you actually saw it and it was really easy to test that it's well worth of your money so easiness velocity targeting and format would be the uh the four things that i'm that i would raise it yeah i think what i would also like add to this is that it's all about like nobody's helped with running a single play test it's not about like doing this one-shot thing at the end of your development like hey let's do a quick round on usability or something like that yeah

sure you will find issues you can fix them and that can actually help you quite a bit um got a fun story to tell there maybe later about a company that got started with us like this because that's usually how it happens no just just tell it now well this is actually one of the first play tests we've ever done and i can't mention the company here but it was like a tv show ip they've worked on this game for a long time and they were just about to bring this into global launch they had skipped some steps they didn't want to like spoil too much about the game and they felt pretty sure about it so we i still don't know how this happened but we won them over like was just marvin and me my co-founder sitting literally in a basement working on playtest cloud but we got them to try playtest cloud and they ran a test and just ftue 15 30 minutes of gameplay and in the first you know how games work right yeah guided tutorials and all of

that people were not able to make it past the first step in that guide editorial they had to like do some interaction on the screen like drag a thing on a character to make him do an action and like 80 of the people failed at this and not like oh they had a trouble doing it and it took them 30 seconds or so no like we had people stuck on this for 15 minutes and like the average time it took people to get past this was like five minutes and the people at the company were shocked like all the testing they had done was internal friends and family and it makes sense because they've played this for so many times like they knew exactly how to drag this thing over to like the exact spot it needed to hit on the screen but real people out there couldn't have done this and like this was a high profile ip it would have crashed and burned when they launched this and sure they could have fixed it and like

three years later nobody would have thought about it anymore but the sheer fact that they were oblivious to that being a problem because they hadn't tested with actual people was uh i mean surprising for us but it also confirmed the value in making play testing so accessible but for them this was a huge win like they did a couple of iterations on like this part of the tutorial making it easier and easier and i think they still like have this in the game these days uh yeah even still an app store last time i checked like it was still there and like but you just like drag this thing anywhere close to the character and it's like boom yeah next step in the tutorial like they just wanted to remove any barrier to this because people would have not they would have closed that game i mean there's enough yeah yeah yeah and the only thing with that exactly and that's the thing like like analytics are really important but

especially when we're looking at the first steps and usability analytics doesn't really help like you just see that there's a problem exactly and then you can go you know chasing your tail of what the problem is doing a bunch of different changes and then this one you can you can kind of pinpoint pinpoint the issue yeah what if you like saying as uh play testing helps you understand why people play how they play and you know you mentioned analytics here i think especially later on in the development when you've got this they show you where problem is but you need other tools to understand what's happening and play testing is one that we've seen help a lot with that because it's quick and you can really pinpoint the source of a problem too it's like one thing that people don't like interact with this one thing or never choose that one character but why do they not do this and it might be something that happened like five days before in the gameplay where they didn't understand some crucial element

yeah but yeah i think back to like that story real quick for the next game they made like you can imagine that playstation became a much like more pronounced part of their development and uh that's also what we see the studios that are like highly successful and using playtest cloud do for them you know especially during the times when they're working on the ftve when you're working on that like session to session gameplay of the first few sessions they test extensively on playtest cloud like every few weeks you get a tail we've got some clients who are like on test two or 300 even i mean they've pivoted a lot with their games yeah but yeah like from the art team to the ui team to the game designers everybody just keeps testing all the time because they don't want to make assumptions so so what what's what's the phase of development where you should be testing because we've been talking about you know essentially gains in production kind of gaining you

know picking up speed to enter markets or testing the for two weeks um but what are the different phases and and is this something that you do when you're developing game or is this something that you do when the game is already developed because to i've never actually run the the longer play test cloud tests that last for i mean okay so let's take a step back through multiple different tests that that like how does this play just cloud work what kind of different tests and then let's talk they use cases maybe better when when when you kind of describe all the uh all the products in the portfolio yeah let me give you the actual pitch here just so we can lay it all out so status cloud is a platform for doing what we call unmoderated testing so it's all about putting your game into the hands of the players and letting them experience it just as the game is so there's nobody sitting next to them telling them what to do the way the platform works is that

you can choose from all sorts of different tests that are mainly based on how long players should play so we've got like very short first time user experience tests one session 15 minutes then you can do what we call multi-session tests where people play a couple of times but they have to take a break in between so something that's commonly used once you actually have a bit more to play but also to see to simulate that experience that people have when they you know they play a bit in the morning in the afternoon in the evening when they watch netflix something like that so very uh open-ended session if you will but you know what you're gonna get like i want one hour across four sessions something like that and then on the long end we've got what we call longitudinal studies where people play for set number of days let's say you want to get people playing for like 10 days um we can do that we'll give people like a day or two to take a

break in between so they can also you know take care of like whatever they need in their life on top of that we've got another couple of like aspects uh so you can do all of this with multiplayer games as well if you need people to play at the same time and your game is still in development you need to make sure all the testers are online at the same time so there's actually somebody like in the arena or whatever you have that they can play against so we can do this for like a short test we can do like we've done this recently like a 10 day longitudinal study people had to play four hours every day it was nuts holy and this was like 40 people multiplayer four hours every day and hopefully it turned out to be a really good one i can't mention the name but like sounds like axey infinity actually you have to grind it for money yeah well only so many games you cried that long um yeah

that's like that's literally played [Laughter] and but also like this is like all like very far like into the development of the game i mean you need a lot of content to like test that long we also have some specific tools to help you understand players earlier on in the development cycle so um we do concept testing which is pretty much showing players slide show with the videos you can use it for ad testing you can do narrative testing we see a lot of like hidden object or story based games refine the story this way we also have surveys so sometimes you need more than like 10 people to give you feedback because you're making decisions on things like naming the art theme of your game stuff like that but you still want that targeting that you mentioned earlier that we can do where you can find very specific players so we can do surveys with them up to hundreds or even more

participants and that's pretty much a portfolio so we've got like surveys concept testing single session multi-session testing and then longitudinal studies and also in this sort of order uh like we suggest using them in that sort of order but before i get into that let me talk a little bit about like the experience of actually using playtest cloud so when you work on a game you want to use play desktop oftentimes you think like okay i'm gonna need to put an sdk in for tracking or whatnot that's not the case so um we developed a platform in a way where you can just drop your build in whether it's on ios on android or if it's on the play store already you can also do that and then you get a recording of people playing this game and they're going to think out loud and if you haven't done play testing before thinking out loud means that the people

will just walk you through what they're doing they might be like oh you know i feel you don't know like how do i upgrade this weapon like i need to look at this that ah that's confusing you'll get feedback like this and uh together with that you see the screen recording you see where people touch the screen and that gives you a really holistic view of what these players are actually doing and how they are thinking about your game as they play to me like the key aspect of this is it helps you develop like some gut feeling for how people really behave in your games you get like all these like sensory inputs and uh as you watch these videos and as you iterate on your game you have a much better mental model of like what like whatever your target audience really is whether it's like uh yeah the shooter players hyper casual like kids uh from like germany or wherever you want to test from you'll watch these videos and you

develop that mental model you can then analyze that shared with your team and learn how you can get closer to whatever vision you have how does the uh like where do you get your players and how many do you have yeah we've got over a million now uh a million players the last two years were uh like a boon for growing uh like a database of people who want to play games i do have to say like you're you're really killing it with play to earn like this is it right we just need to add the coins just add a little nft and call yourself a metaverse and you're off to a billion dollar evaluation yeah we might have to think about that [Laughter] yeah so we've got this uh like north of a million uh players and maybe from the us uk and canada uh we're in the process of also opening it up to more countries and probably by the time this goes live uh our

announcement uh on like opening up uh testing in more countries than just these three will be live so in the beginning uh we're launching that in southeast asia germany some european countries um mainly based on where we have a lot of players um but we'll keep growing that if you ask where these teslas come from so eight years ago when you just started working with us we went to reddit and got people from there but then over the years our user acquisition looked pretty much like the user acquisition of any mobile game uh you know we went to facebook did a lot of targeting like looked for the sorts of players that we didn't have enough of yet and uh just grew that and at the stage we're at now with the size a lot of the growth is organic and so people get a little money in return for playing and of course the word on that spreads

regardless of like what profile we're talking and you know we do testing with kids as young as like ages three and four together with their parents all the way up to i think our oldest play test is 87. um so that's quite a spread of ages and experiences we've got there and yeah yeah no that's that's awesome um so so let's get back to the uh this sort of like original questions of like um when to use different tests and more like how if you can go through some of the examples of the companies that that have been using everything from from concept testing to the longitude surveys yeah totally uh i think our model is always test early test often and this is also something you hear a lot in user research and usability testing circles and we say that because you want to do the testing at a time where it can have an impact and like all

testing we do and we talk about is done because you want to change something in your game potentially or you want to find out that you don't need to change anything so you want to find out like what isn't working as early in the process as possible so you haven't invested like a year of time and creating art assets for style that just doesn't resonate with your target audience so how early you want to get started testing as soon as you have something to show to players i think it also helps starting so early um because you can involve your team a lot more one of the biggest barriers we actually see to starting to test early is game teams pushing back on like no no this isn't ready yet we can't possibly show like this prototype this like pre-production build or like even this story like outline to people yet we need to refine it more and that concern is of course understandable

uh because they want to refine it more they don't want to like find out that they know what's wrong with it they think but oftentimes when you put it in front of players you find out that your assumptions were wrong so let's talk specifics here um the very early testing is usually focused on discovering issues finding out if you know the type of game you want to build is actually good for the target audience that you have in mind um i'm going to talk about hidden object game here because object games like make it really easy to talk about concept testing and uh you know how games these days are like ruger's games of course peril journey and so on they're heavily story driven they appeal to a certain crowd and uh they live from this episodic nature where you can add more and more chapters to the story and people really become invested in how these games work

um when we work for studios that make games like this it's super interesting to see all the like testing that happens before they like do the first thing in unity or whatever yeah yeah they have like storyboarded out the entire game they have the narrative they have like uh mood boards with like here like the characters here's like the shipwreck here's this here's that and sometimes it goes as far as you know they make little mock-up videos of this to show the story to players and they ask them like okay um like who are the main characters in this what's the like why does this character like aspire to like uh find their lost lover because they test understanding you know it's all about like from showing this part of the story to our players do they get why they should care and do they care and then you start refining it and uh the cool

thing here is you can iterate super quickly like you put this into the hands of the producers the art team like the narrative director and they can like people use a powerpoint for this they put together a powerpoint put some questions on there and then put it in front of the players run the test in the evening in the morning they work on the next iteration and refine what needs to be refined to really make the story stick and we see this for other games too like some casual um like match three games that were all about cooking uh should have done this a bit earlier because when they did this like relatively late in development they found out that the style of art they chose on their food wasn't appealing to their audience and uh so this is interesting because um i've actually never done these type of tests with places cloud uh we've done cons like we've done those app store and it's like kind of like um

game description ones so where you write about the game and we tested different different type of settings in written format what i've done for the art style and the setting uh i've done a lot of tests on geek lab where basically you're just testing the marketability of the theme and not per se how they feel but that's actually a good idea to kind of do both uh to kind of hear a little bit of a qualitative feedback of why certain things don't work because we only see you know the ipms and the cpi of those early tests of what theme doesn't work what what works we don't really know why this you know cyberpunk thing didn't work was it because of cyberpunk or was it because of the style of the cyberpunk that was done and i think you could get more nuances by running the tests on both platforms yeah i think you need to do both like play testing cannot replace that like quantitative like can we like build a business out of like something like this but it can give you a lot of insight into

what resonates with players and why which will help you as especially if you're exploring like a direction of game that your team doesn't have a lot of experience with yet um on the outside like i have one other like fun example here and this was for resource management game thing like fallout shelter but in a different setting and um it was it had a very strong narrative component and what that team identified as one of the risks was that players will not identify with the characters that they put in there and they will not you know see them as what they really are and one example here was uh they had a guide in that game would like help like sort of explain how the game works and npc yeah yeah and this was um the hotel manager so war suit and uh like

big guy and like they wanted to have him look like he knows what he's doing people were like oh yeah i don't trust that guy like looks super shady and now the main character that guides europe players around and should help them learn the game is perceived as being like this shady like character which wasn't intentional at all and uh this was left field for them they didn't really expect that but they made some tweaks fixed that and uh the whole like onboarding and all of that flowed much better because people like didn't have that association seeing that character but i would say let's move on a bit here we talk enough about concept testing um the next stage really is you're beginning to build prototypes you're working on the mechanics you're trying to see like what works like what doesn't level design depending on what you're working on which is a fantastic time to play test so uh we're seeing a lot of games start

at the stage where you know you have like programmer art you have like some really simplistic things that uh try and figure out like okay these sort of controls do they work it's focused on the usability potentially but also level designs like will people understand like these boosters if we're talking match three games will people understand you know the meta game that we're putting together here does it make people want to keep going whatever it is that is like so risky that you prototype that first why not put it in the hands of players as well um like we often see reluctance with this like oh is it too early people will not know how to use it like yeah and they won't that's fine they will complain that oh yeah it looks like there's no art no music that it doesn't matter that's not what you're testing here like you know that what i want to know is are people able to like i don't know

navigate this map and move the camera with like this control scheme i devised or like does the weapon combining mechanic like get understood and then when you're testing this you have to laser focus on those issues you got to know what do i want to find out and i don't want to find out like is the game and the current stage appealing to players i want to find out do they get this mechanic does this sort of design like confuse everybody because it confused the entire team or like do players actually enjoy it so it's good way to work around these assumptions and you know when you make a game you have to make a lot of assumptions this is a quick way to test them and like get you something more data driven the studios that work with us that are like consistently like delivering games that you know are

fun commercially successful they start this early it's very aware that you know studios skip any sort of play testing and go straight to launching a game with a super refined first time user experience um and it's because it works like you need to see how people actually play even if you've done 10 games already especially then because you're not the player like you need to base this on like the actual uh yeah gameplay people are showing you what's your experience testing in this area like when you're like creating like new mechanics i mean i know what you guys are working on right now and yes i'm sure there are some things where it's like oh like will people get this like how do you manage that yeah i mean the earlier you get to testing the better to be honest like we as any other company actually of any studio that i've worked

at it's been the same thing it's like ah it's too early uh we know all these things but no you have to kind of you have to see players suffering playing your game and not being excited about it getting bad results that's really important it's uh it it puts a whole different you know speed into your into your team when they understand what they're doing is not actually that great and then it also gives a lot of hope because you're able to tweak quickly and get to good results and be like okay we're actually making the game better um so my my my experience is the same nobody wants to test when they test they're surprised and when they tweak and and improve the game based on the tests uh the tests improve and everybody's happy yeah and i think what helps is having like you need a champion in those teams who will advocate for putting this in front of players and you need to be very clear with the expectations of some of these early tests like stuff is not gonna work and

you might have to give players a bit more guidance that's all fine like it's still so valuable if you know what you're looking for and um you can also empower the the different like departments working on your game like art the game design narrative whatever you have for your game to do this testing on their own and it's gonna create this effect where like people see that okay they keep testing like they they keep talking about the stuff and then they're like oh you know we made this decision because people didn't understand this or like people really love this uh it's gonna make other people want to test as well and yeah yeah 100 uh what are the solar sort of experiences with these longitude tests like i've heard of them um i think i know a couple of companies that were doing them i believe super evil megacorp was running those tests uh but it's been a while since i've discussed with somebody who's been doing those those tests so can you talk a

little bit more about how they are being set up and and what are the results of those yeah totally so these later like longer studies where you actually have people playing for multiple days or multiple sessions are usually used more for diagnosing certain issues or optimizing the experience so i talked about like early stage testing being for discovering things and like making all these like decisions in the beginning like uh setting the game on the right tracks whereas longitudinal studies help you really solve retention issues or like issues that result from people not understanding certain aspects and ignoring parts of the game as a result um let me give you two examples here one is a game that like use these studies extensively during the development and it's an exercise game um like think like zombies run but with a very different take on it

and i think every single play test they have ever done with us like even from the early stages has always been a longitudinal study for like five or seven days um because for them it was extremely important to design this like session two session to want to gamepl back on like the next day and the day after to like go on a run or like a workout your game is set up for failure so they could have had the best first time user experience in that first session ever it wouldn't have helped them if they hadn't have visibility on the other parts so sometimes you have a game where it's so crucial that you're not just designing for those like first 30 minutes and day one but where you success is measured in like initial success like day 7 retention otherwise if people don't understand your game because it's super complex they will never come back

of course to some extent you have that in many games but they are somewhere it's super important but more commonly we see longitudinal studies being used because the game is in soft launch or it's launched already yeah and you know day three day five it's just not where it needs to be and usually teams don't start play testing when they first see an issue like this you're doing a b test you've got some assumptions for like what you can tweak to like get the numbers up but if all of that fails they usually want to understand why are people churning much more than like other titles we have on those days and can we somehow salvage this that's where play testing comes in um we let's say you have issues with day five your game is typically played like half an hour a day in multiple sessions or something like this you can set up a play test that matches this like typical play time put five 10 15 people in there

and we'll have them play they can't they can but like they usually don't churn during the play test that's not the goal of it but if you know you have seventy percent drop off by day two or day three we'll see that in the excitement players have to continue playing and um we ask people after every session after every day hey would you continue playing if this wasn't a test which can be quite useful to see okay an issue begins to happen on day two it might lead to something on day five and a game that we've done this for um again uh can't really name the game here but resource management a bit of city building and they had uh like an aspect where you were sort of like breeding certain animals together to like have new stats which was a crucial part of the game but a lot of the

players ignored that entirely and they could not figure out why they ran this longitudinal study they saw similar behaviors in the study where people yeah they got going with the game they felt like they understood it but they never went to this one part of the game but in some of the videos they understood why um players did not understand the benefits this breeding would give them for the like late game like if we're talking like day 30 day 90 and so on because they completely botch the introduction of that feature but they weren't aware of that and they fixed how this uh breeding mode was explained they changed the whole like unfolding of that and how they weaved it into the the story of the game and it resulted in a lot better retention people actually understood that game mode they tested it and uh when they relaunched like a version

with all of these changes in place to like how the feature was explained how sort of that like knowledge of like this mode was introduced to the players it did much better and uh for them this would have been the difference between like killing the game yeah being commercially successful so you know this was a long way of saying like if you're seeing something in the numbers that you don't understand it helps looking at what people are actually doing and thinking as they play this um yeah yeah that's an important part because you know i've been in situations where maybe the day one has been good but the day day two day three are kind of falling off the cliff and and you're trying to figure out should we change the future like what's going on because we've been doing those early testing and and that's where this longitudinal test is is very helpful because you know sometimes the analytics just doesn't show and you have a bunch of gut feelings you're doing a few updates and it's already a few months later and you haven't been able to improve so now you

need to get that qualitative information and for that one uh these tests are are very powerful um and how do you yeah go ahead yeah uh this reminded me of like a study we've done for a game uh quite recently and i love that setup because what they did is they paired this wasn't their first time around like they were a team that's very like experience with play testing and integrating it into their development process but they combined their soft launch with also running a two-week longitudinal study at the same time the goal that they had in mind with this is they knew they're gonna get a lot of data on you know what exactly is happening in the game during these two weeks but they expected that they're gonna have questions that the data cannot answer and they didn't want to you know lose any more time on like okay let's test a few things tweak it launch again like see what the numbers say so they ran a longitudinal study in parallel with

their soft launch i think we had like 20 people on playtest cloud site playing and of course you know for the soft launch they went pretty big to also get numbers on like all sorts of other things that you can't get from a play test and what they did two weeks in is then review the data and they also looked at the play test results and it helped them a ton because they could understand certain signals that they saw in there like this thing not being used or people like dying a lot here what not based on the play test data cause they could dive in and see like okay yeah this seems to be like we must judge the difficulty of this people from that audience are not faring well with like this game type but they prefer that one and it gave the team a lot more visibility into how people were actually playing the game in the real life and uh

yeah i thought that was a very smart way of combining your data gathering from like analytics but also running a play test and then speeding up how quickly you can get this game like launch ready perfect yeah yeah that's no that's a that's a really really powerful information there and because i know a lot of studios are only using plato's cloud for the early phases uh and not not really you know have not experienced like my me myself i for example haven't had the uh the uh the experience of using these large dual tests and kind of like relying that well we'll figure it out in soft launch we'll just use analytics and our gut feeling and we'll figure it out but uh but it is important so regarding to that like how do you how do you make sure the studio is doing analysis correctly when they're getting all this this feedback because i bet a lot of the clients you have don't have their own market researcher or sorry not marketing but user researchers yeah um and they're kind of watching these

videos themselves so how do you make sure that that the uh the analysis is correct yeah um i uh i asked this question to our team like frequently because you know yeah we work with people for whom this is the first time play testing and like what can we do to help them and one side of this is that uh you know played as cloud it's a tool and you have to learn how to use this tool i think that's important to acknowledge uh like none of us are super experienced like when we first start out in setting up good play tests but also in analyzing them and i think especially if you're starting out if like you're planning like your first play test if you think like okay i haven't done this before let me get started plan for some time um to actually learn the skill like you're gonna have some setbacks you will run a study you might see like okay are people

like did not get what i wanted them to do at all and it's not the fault of the game it's the fault of my test setup like maybe i chose uh the wrong way of testing maybe i chose the wrong target audience for this maybe the build was completely broken uh stuff like that happens and it's okay uh i think it's important to realize that uh play testing is something that you need to learn and that takes some time and if you look at like game design curriculums these days actually doing uh some uh education on play testing like we work with a couple of universities to also give the teams access to our tools for their like semester projects and so on but um so that's one part right plan for it to take a little while what you do to help them get started is these days we can provide a lot of advice so eight years ago it was just uh microphone and me getting started and

was very tech driven like the main value of playtest cloud was that hey here's a way to get a build in front of a player and that's still of course a key value but uh we've run thousands of play tests every year uh we see what works for pretty much any genre any sort of like challenge that you're having and we're so happy to share all of this like one decision that we've made early on in the company is like we'll never send you an email from a no reply email address like whatever email you get from us whether it's well the automated ones right uh yeah you hit reply you usually are talking to the people in our project management team or like uh sometimes even me and uh you know we've seen a lot we're happy to share advice on like setting up your study on analyzing it we've got a user researcher uh like in-house who you can just book a call with for free like if you're a

client of ours just like there's a cleanly you can go on their schedule an hour to like help him do whatever you need done so that's like the the general education that we do but we've also gone in to like studios of our clients done like trainings talks help them also help themselves uh like usually we know who the top users at these studios are but oftentimes other game teams don't know that so like we're also working a lot on building these connections within the studios and surfacing that knowledge and awareness of how play testing can be done um yeah i think and that's important like we're all learning this and it's a new field uh like yeah testing hasn't been around for mobile that long and i i think it's important that uh because of the video format i think it's really important to share those with the team because there's you know

especially at big studios they don't want to do that because they kind of want to focus everything on their own work but we're all here to make games and we're all here gamers so it's very interesting to see how people are playing your own game so i'm i'm in for kind of getting everybody to to look at these videos but at the same time you know you have to give ownership of fixing like for example the way we've approached it is a lot of times as as we do different fixes like we we give the material to the team or to a small group of people uh who are focusing on one two three you know few of these elements and they decide how they want to fix it based on the videos and they see so there's a lot of ownership given rather than than somebody else interpreting for them and saying like this is how you need to approach it so that's that's also kind of like distributes the uh the uh the ownership of play testing to to the team as itself and then very interesting to see the new video that comes out uh where the issue has been fixed and it's it's not there anymore so uh very very sort of tangible progress it's interesting to see how like this

has also changed over the years so when we started out uh with playtest cloud a few companies had any user researchers on staff it was like even like the big players back then like yeah they might have had like 2 000 employees but they had five user researchers if at all they were usually working on other things then like the gaming side um but uh what really has changed over the years is that uh the user researchers in the companies are also working to empower the game teams to do more research and this varies like some companies like have like a user research team at all use our research go through them whereas other companies take a more like wide approach and they want to empower everybody to run good studies and you know i talked about like let's get like artists narrative designers game designers level designers in on this so they can find out if

you know they're like hitting whatever like a goal they have with the experience they're crafting but it's of course important that they do it right you know everybody can live with statistics you can run a test that just confirms what you want to see but some of these bigger studios have grown user research into like an advisory uh function where teams can go to them and they will help them see hey okay you're running like a play test for like games that like this we're doing this with a company right now they're doing vr games but now they're switching over to mobile because they want to expand the market those teams don't really have experience doing play testing on mobile they've done some on vr but that's a very different beast they hired some user researchers who have worked on the mobile side but not with the sole purpose of running all the studies and like hogging that responsibility but teaching the teams how to do this

player-centric development using tools like playtest cloud to get the ground truth and i think that's a very powerful approach because play testing is everybody's business i think yes yeah if you're working on a game like you can't say like yeah that department is responsible for like the user experience like that doesn't make sense like even developers make decisions that will directly affect how the game is perceived so of course have them watch the videos have them run a test every now and then and empower people to do that i think that's also how you get to this point where people are wanting to play test better want to learn about it because they see how it benefits their own work how results of whatever it is whether you're designing that one screen or the entire meta game they will be better if you can get input from players and like if it's a skill that you possess

yeah yeah no no i agree i agree with that um so before i let you go like what are some of the mistakes uh when what that the developers do when they start play testing like what is the what is the stuff that you see because i think the expectations are quite high or low i don't know yeah i would say not testing is the biggest one um yeah i mean we talked a lot about that importance but you know still like we're experiencing like we're having like talks with studios that you wouldn't believe or like yeah play testing like we heard about it but like we're not doing that like and it's like why like and uh often times yeah it just battles you but then they get started or you sort of like force them to confront what they could learn with this and i think seeing is believing here so like if like somehow you find an advocate in there you get them to run a

play test and you see like oh my god like uh joe like 40 from seattle fails at this and you show that to the teams they're like okay hmm yeah maybe we should like do this a bit more often so let's say like most common mistake like not doing it or not or doing it way too late um i think we talked so much about like the risk management aspect here but we see that so often that the first uh encounter uh a studio has with play testing is because they have a severe issue like something is not working like retention is problematic or like just like they can't get a certain thing off the ground because people don't seem to get it and they tried everything else and this is sort of like the hail mary like okay let's see like can we learn anything if we put this in front of players and oftentimes they can and that changes their mindset then so i would say if i

can give any advice to people here like skip that part like just start testing and like you don't need to fail by being blind to like the player experience like there are tools out there that make it super easy i think if we want to talk a bit more specific i would also say we see this especially from like newer studios not testing with enough people so yeah you mentioned this earlier like you need to watch the videos on play test cloud and i would say that's probably one of the biggest time investments that you as a team and like as like a leader of the studio will make into play testing like you sit down you watch these videos to learn something from them and what we've seen is that some studios are very cautious and they think they're guarding their time by let's only do a test with like two or three people you know it's only like an hour of video material then versus like getting four or five hours of video material but i think that's like

it doesn't work out like that because you get these two videos and you're trying to get like every possible like bit of information out of that and uh you also have like if you only have two people you have two conflicting opinions yeah if if i run a test with 10 people i think about some survey questions which we can also help with like i can get these videos i'll look at the survey i'll read through the answers and i'm like okay people are having problems with like this and this they didn't understand what that character is doing and like nobody wants to keep playing huh okay now i know what to look at in the videos and i can go through these videos and like spot on like go to okay i know like they mentioned this mechanic i know when that gets introduced to that moment in the video i'll watch a few minutes here a few minutes there and allows you to get a lot more from the video so i would say like think about

play testing a bit differently where it's not like i need to absorb every second of like the player's experience but have a goal in mind go for a few more players and then use tools like the survey to narrow down what you really want to find out the result is you'll have much more reliable data when you go to your team like it's not like okay i'm married didn't understand that but it's like yeah we had these five people who were all in our like key audience that we want to reach not get this thing like folks it's serious we need to like rework this uh like i've been preaching for two weeks or something um so yeah i would say you know start testing early iterate like test again like uncover more and more things as you keep going because those are the things that we often see are lacking and they lead to you know games that fail games that don't live up to the potential that they have

i would i would also add that don't make the other play test play tests um exclusive in the sense that it's not only for a few people just allow others to to watch them i think it's really important for the team to know how what's going on because other otherwise they they only live by the gut feeling and they don't know you know the reality so really important to distribute that information um last thing what how is the platform evolving in the future because you've talked about all the tools that you have for now i know that you guys are working with soulston uh used to be known as 12 traits so they are offering packages with with uh with basically using your million players as a test group to to really understand deeply the audience but what other kind of stuff is coming out yeah uh it's a fantastic question so uh we're working on a couple of things right now uh i already mentioned earlier uh

testing in more countries it's probably the number one thing we've been asked for and right now plato's cloud mainly has players in the us uk and canada like because those markets are of course super in demand for testing but um there's no technical reason behind that uh i mean plato's cloud works internationally and actually like later this week we're like taking the first step there introducing our global player panel launching in over like 10 more countries a couple of regions and if you want to test in like southeast asia europe nordics those will be some of the launch regions we're also adding yeah like we'll have that we'll roll out more and more countries there as time goes by in the beginning it's focused on english speakers in those countries but we'll also open that up to like german studios being able to test with german players finnish studios with finnish players if that's what you

want to do exactly i think you can hold off on that market for a while i wouldn't make the launch time like arabian markets those kind of things yeah exactly south america is interesting for quite some genres too and that's also going to be one of the launch regions so basically just more tools to allow culturalization of your games yeah and yeah some games also just develop for those markets i mean yeah we're like in a global economy and we want to support those studios uh another part that we're working on is also making playtest more accessible to super small studios so we don't have this formally announced yet but we're gonna introduce some offers that are aimed at indie studios because so often even with the tools we have these days game development when you're like in the

seat stage if you're like only starting out you don't have a lot of money but you still want to make a good game and yes like we want to level the playing field there more we want to like put out some really attractive options for them both from the education point of view like how do you play test but also i mean from the financial side like make it really easy for folks to get started get the foot in the door and learn how to build games on that because at the end of the day they will be more successful with that and they can make better games we've seen this time and time again and uh it's also our story like we never took funding like we wanted to build something valuable that uh you know delivered value and we also want to support studios who want to do that who like want to build a game that like can crank the charts even with like small means so you know that's another thing we're working on and then there's like a lot of other things in the works like we

want to do like we don't support people watching the videos with like ai and automation to spot some of the cool things that are happening in the videos there's like work going on on that but we have a high threshold for when we want to launch these things because we want to make sure like these features are useful and they're pointing out important moments in the video and not just noise and yeah it's all about like making analysis easier making play testing easier and like unlocking these insights in a way that yeah you don't need to be a user researcher for which you don't need to be uh it's all about like if you care about games we wanna like make it even easier for you to like get these tests done well amazing stuff christian uh you your your service has been of tremendous help for hundreds of studios and continues to be and i'll continue to be a proponent and advocate for for your tool for the next eight years to come

and even more so uh thank you for playing this cloud i'll be honest like it's a it's a great tool um really invaluable for for for studios around the world thank you and it's been fun working with so many wonderful studios and people like yourself so thank you awesome thanks everybody for listening uh christian how can they get in in contact with playtest places cloud.com yeah plato's cloud.com hello at playtestclaw.com if you want to send an email but yeah go to our site you'll find uh many ways to get in touch with us uh we've also got a fun youtube channel where we talk about play testing have some introduction videos webinars and everything but yeah playtest.com and everything else will follow beautiful check it out and we're out