From Zero to Hero - Give Gamers What They Want with Rapid User Testing

our next talk is going to be with john lee from piku so he's going to uh just while uh our stage manager says manager sets him up there oh maybe maybe popping in just like to take this opportunity to remind you as joel and i were just talking talking about hi john i'll i'll bring you on in a moment there um just to remind you as joel and i were saying there um discord uh we've been running a discord for this event and uh it's uh it's a great place to mingle informally with the uh with other attendees but also our speakers as well so following up any of the tracks if you want to check in and ask any questions of any of the speakers you can dive over there into into discord the link is right there on the purple button in in meet to match so

if you go in there on your dashboard you'll see the the quick link into uh into discord there so that should get you started um uh the uh so this is gonna bring us up to our our next talk we're starting a couple of minutes early but that's great it gives us plenty of breathing room for questions uh john how are you doing good excuse me good how are you doing dave yeah all right thank you yeah i i kind of i'm sad that we're not actually in helsinki itself i love visiting the uh the finnish capital and it's always great to do a conference in person but you know this opens up to a whole new global audience i think you know we've had people here where are you uh where are you um zooming to us from today i'm zooming from the uh san francisco bay area in california amazing amazing uh so you're um it's your morning there thank you for getting up early to take yeah yeah happy to be here uh we uh we actually hosted a pocket game of connects in san francisco a couple of years back it was a fantastic city and a great place to hold a conference as well hopefully we'll be back at some point absolutely i i hope to be able to attend these in person so look you're here today to give us a

talk about um well it's called from hero to zero but you're going to be talking about uh rapid user testing i think is that right that's great perfect well if you're if you're ready to roll and you've got your presentation there uh i'm gonna suggest that uh that we get started what uh what i'll do is just remind everyone that if you have questions for john stick him in the q a box at the bottom of your zoom screen i'll keep my eye on those john don't feel you have to curate those live you just go ahead and uh and give us your pick food presentation and at the end i'll um i'll pop back in and we can look at those questions together if there are any okay that sounds great so look i'm going to fade into the background uh john over to you cool are you able to see the slides oh yes yeah yeah very good okay great cool well thank you everyone for having me here um today we'll be talking about giving gamers what they want with uh rapid user testing so first of all a little bit about myself um so i'm john i'm one of the co-founders of pikfu um i've been building software and playing video games my entire life but i have not

i do not build video games instead i prefer to build software and tools that help others build games and their software businesses um so i started my career a while back at microsoft doing software and product development actually for mobile devices long time ago before leaving to start a company with my co-founder justin and that was many years ago so enough about me um what are we talking about today so we're gonna be talking about two things uh one you know as i'm sure everyone already knows launching a game is hard and how can you know what users actually want and number two what is rapid user testing and how can it help your team's efforts so first point how do you know what gamers want so as developers today you have a seemingly infinite number of tools at your disposal

creating a game today is not like reading a game 10 years ago or five years ago or even one year ago right so let's take a look at the landscape here's a landscape of tools that are based on two axes one is the kind of data received from your tests the qualitative data as you can see column on the left and then the quad uh sorry the quantitative data as you can see on the left which is you know your your numbers your metrics your usage all of that stuff and the qualitative data the why on the column on the right um this that kind of data is gathered by conversations uh or videos uh but basically interaction with users off your game right either written or oral communication and then the other axis is who you're gathering that data from so across the top those are your target users those are the users that you want to reach that are your future that are your future users and then on the bottom row those are your current users the ones

who are actually playing your games so now we feel that game development is full of data but that data is kind of useless if it doesn't give you the understanding of what gamers want so let's say through a typical soft launch developers are using internal analytics and app store metrics to identify problems right that's quantitative data from your current users there's a ton of tools and a ton of ways to gather this now these kinds of problems during a soft launch could be anything from store issues like the initial download is too large screenshots that don't accurately communicate gameplay wording on descriptions that aren't enticing um to like anything like gameplay issues right game art doesn't align with your audience or like the future is poor the gameplay isn't sticky enough these are things that you can at least diagnose that you're going to have an issue with this kind of with this instrumentation and these kinds of tools now on the right side if you're fortunate enough to have the resources

to build a community those can be helpful um to sort of get that written feedback uh about from users of your game um and that plus you can comb the app store reviews for what what gamers are saying but they both don't don't help you understand why gamers weren't installing a game like it's hard to get feedback on a product if no one has a product to review now in the top left quadrant those are the quantitative data from your from your target users these are things that you are probably already looking at first right when you were when you first conceptualized the game you probably looked at app annie and did a bunch of other and looked at a bunch of other market research reports perhaps to see okay well these are the characteristics of the market that we want to go after this is the idea for the game and so on and you want to make that match so those tools are out there but that's more at the beginning uh that's more looking at the beginning of the development process

so that brings us to the final quadrant qualitative feedback from your target market and that's what pixu does in a fast affordable and now accessible way it helps pikfu helps you get insights from your target users even before you launch so let's take a look at some examples i want to go through a quick case study here of a customer who used pikfu to turn around a failed soft launch so this publisher who shall remain nameless they already had a string of hits under their belt and they're they were a top 10 publisher having they grossed over a billion dollars in app store revenue before launching this game so all this happened about a year ago and they've been working on this new game which is a character-based rpg um they've been working on this new game for over a year a full development cycle and they had just soft launched it in russia and where it you know thanks to the instrumentation they saw that it

massively underperformed in downloads didn't hit their metrics so the app store metrics showed that users weren't biting what now why aren't users downloading the game so here are three questions that any product manager would ask themselves to understand why users aren't downloading a game number one are we using the right screenshots and are they in the right order for in the app store number two is that app icon compelling enough um in the case of a character-based rpg are we using the right character in that app icon and then number three is our description working does it emphasize the most compelling features of our game to drive those downloads so we're going to dive in our publisher client used pikvu to help answer these three questions with three polls so we're going to dive in and see how

here's a screenshot of a sample pick foo poll now due to confidentiality i cannot share the exact polls that this uh this large publisher customer of ours did but what i could do is i could run very very similar polls with existing creatives out there so this weekend i went and i looked i took those polls and instead i i took creatives from a game that's near and dear to my heart brawl stars and i went and i ran recreated those same polls so those are the polls that we'll be showing here here's a sample pig food poll you can see it's a quick pull question across the top four options for people to choose from you can see the scoring there results underneath and then rows of responses to each option underneath we'll we'll dive in deeper here so the first test that the publisher ran was testing their screenshots trying to answer are we using the screenshots the right screenshots in the right order

so like i said earlier they took their screenshots that uh from that soft launch they put them up on pikfu and they asked what screenshot would make you most want to download and play a mobile 3v3 battle game well sorry i asked about the 3v3 battle game they asked about the character based rpg uh in this case we put up four screenshots they targeted 50 mobile gamers with app store spend and this entire poll completed with results in an hour and 12 minutes and cost about 200 so if you want to see the full results of this poll that i ran this past weekend you can scan that qr qr code and you can see the full results let's dive deeper into some of the comments that the panelists actually gave so these are actual comments from the poll that i ran this past weekend option c simply shows more information it has the side with all the options such as social news brawlers it shows the character looks and promotes

the ability to play with friends so in this case option c is the screenshot where it actually shows a carousel of characters that are playable in brawlstars this response was by a female age 35 to 44 with 30 to 60k of income with a graduate degree or another comment option d allows you to actually see the game option c b and a don't actually show you the game however c is better than b because you get to see your three battlers and it can give you an idea of being a 3v3 in order their preference for each one of the options so this last comment was by a male gamer age 35 to 44 over 100k in income with a graduate degree so the publisher uh analyzed these results and the action taken was they replaced the leading screenshot for the game and then reorder the others based on the rankings from the results of this poll

so the next test that they ran was about figuring out whether they were using the right app icon and specifically whether they were featuring the right character here trying to answer the question is it compelling does it showcase the right character so the sample poll that i ran to duplicate was i went and i grabbed three app icons three historical historically different app icons from brawl stars put them up here and ran the poll question which app icon makes you most want to download and play a mobile 3v3 battle game you see the three options there the audience was the same as the previous one 50 mobile gamers with app store spend in this case the poll completed in under an hour cost under 200 some interesting comments like i like the skull icons better than the smiley face in option b i think having an eye patch makes it look cooler i personally do as well both of the skulls are good however and much better than the generic smiley

that was from a male aged 25 to 34 30 to 60 km income with a bachelor's degree and then another comment the skull in option a is way more interesting and unique compared to the smiley face and it makes more sense for a fighting game i like the cute angry face far more compared to the eye patch that's from a female gamer age 35 to 44 100k plus an income with a bachelor's degree again you can see the full results of this poll and read all 50 comments if you scan that qr code so you know as an aside i think the current brawlstar icon is actually the smiley face in the middle which in this poll kind of lost handily so who knows so after reviewing their poll about the app icon what did our publisher do they changed the character they changed the character featured on the app icon uh based on the results so finally we get to descriptions um similar setup as before i grabbed the for this

past weekend i grabbed all the feature bullet points from the brawl stars listing in the app store and ran a similar poll to what our publisher did about descriptions answering the question trying to figure out the question is our description emphasizing the most compelling features so i i put up the four text blurbs asked that poll question targeted 50 mobile gamers with app store spend and in this case the poll completed in 51 minutes the total cost was about 200 and a notable comment here let's say option a rules this one because it makes clear that it's fun either by yourself or with others winner take all is a great line and he this respondent actually got into the details and quoted one of the one of the lines in that option a description you can see the other one here and see the full results by scanning the qr code so again action taken after analyzing the results our

publisher did change and reorder the descriptions based on the poll ranking results so these were the three these were the three polls that our publisher you our publisher customer ran to diagnose those three questions so to review the changes they updated the screenshot order they changed the app icon to a different character and they reordered the feature descriptions in the app stores in in their app store description so what happened after this after this the publisher did another soft launch uh in another secondary market where they met their target install metrics they saw that the downloads definitely increased hit their levels and that gave them the confidence to launch broadly into the us and that's what they did earlier this year so i'm happy to say that the game itself is doing well i think it has a 4.3 star rating it's going up and the publisher is continuing to iterate on all their creatives and other listings in the app store um and using

big foo so let's take a step back and see what's going on behind the scenes um pikfu gives you real user feedback in hours and not days so we think of pikfu as a type of rapid user testing and what is rapid user testing well traditionally the only way you would one of the few ways you would be able to get this qualitative feedback from your target audience is to do run some kind of focus group but traditional focus groups have generally been costly and time consuming and we think of pig food more as a quick digital focus group so by by rapid we mean that our surveys and our polls are simple single question polls that are completed in minutes and hours and not days or weeks and that start at a dollar per response and when we say the user in rapid user testing is not are not users who play or play test your game but there are panel

our panel of over 10 000 u.s consumers that are all under nda so they don't reveal anything that they uh that they provide feedback on they're all paid a small stipend per poll and we have demographic and uh psychographic and a lot of information about them so you can target by you can actually reach your target market and you can target by gaming specific demographics more on that later and they're all quality controlled by human feedback and machine learning algorithms now a little bit more about the testing you get a written explanation from every single respondent on your polls which you saw earlier in the examples you can test between one to eight options if you have two or more options the respondents will vote on those options and you can test images text video and audio all the all the polling is private and the results are private only the people you share it with are able to see it and so we think that this approach uh to

gathering qualitative feedback uh is is fast and it's easy and in this manner we're able to provide both quantitative and qualitative insight on uh your creatives so again in every poll all polls include written responses you can see a screenshot of all the different responses and um these are real people they're not ai you know people have asked if if you dive in and read the comments you can recognize that they are definitely humans um additionally all polls include demographic reporting of that entire panel of 50 to 500 people who have responded to your poll so you can see the breakdown within your panel your mini panel of age gender income mobile device usage any you can slice and dice any way you'd like and in terms of targeting we have the standard targeting demographic traits uh gender age ethnicity but we're also able to micro target on behavioral traits so for example what

kind of mobile device they use or whether or not they actually play mobile games and even how much they've spent in the app store along with favorite game genres and a whole bunch of other less gaming related but potentially related to your game traits so where does pixu actually help in the game development process we went through a case study about turning around a failed soft launch we have a number of publishers using pig food now and actually they use pig foo in every stage of game creation so in the pre-production stage um developers are using pig food to test game name and character concepts um you can do competitive analysis with piku i'll show you an example in a bit character development is a huge one as is various kinds of art design during during production we see publishers testing gameplay variations

and ui variations and at launch time and beyond piku is obviously used to test app icons and so on for app store optimization and even testing digital advertising or tv advertising before launching paying those big bucks to launch it out to your broader market so i'm going to go through a couple of different examples here um so you can use pikfoo to test your game's name you might want to put this out there to avoid various you know mistakes or reporter names that might not reflect so well on your game or on your company and ensure that your game's name is a draw and not a turn off so this is a actual poll the options are blurred out here but there's an actual poll from a game developer asking what name would you would make you want to download the game testing three different options in the in this case it's a splash image with the name with the name version across the front the audience was 500 respondents

to a general population the this poll because it's 500 people took about 13 hours cost about 600 dollars you can also test art themes with pikfu so i've heard developers say that testing is stifling to creative teams but what we found is that with big foo your creative teams are actually more free to explore different concepts and themes knowing that you're going to be able to then validate them quickly with your target market and actually see what resonates so in this case in this example it was four different art styles of a city themed puzzle game targeting 50 female mobile gamers age 18 to 34. took two hours to complete cost a little under three hundred dollars you can actually do gameplay testing because you can upload videos you can upload videos to pickfu so in this example we had a publisher testing a gameplay mechanic before committing to actually

producing them um so this was testing a battle formation animation so the question which battle formation do you like three options three different videos targeting 50 male mobile gamers aged 18 to 34. and this whole poll took two hours cost about 230 dollars and as you can see by the screenshot here option c was an overwhelming winner and i think that i would suspect that because they wanted to put this poll on pig foo the publisher was not actually that confident about which gameplay mechanic was going to be the best one or which formation was going to be the best one so this was really good validation for them and finally we come to my favorite type of pig food pool which is the competitive analysis um so with pig foo you're your competition doesn't need to know that you're comparing right but because you're able to take creatives your creatives others creatives and test them off market and with our panel with a

private panel you can understand how your game will be received even before launching during the conceptual phase or at any earlier phase so in this specific example the publisher was asking which idle rpg game would you rather play and why they tested two images from two different games i'm assuming one was theirs we don't know but we i'm assuming one was theirs and one was a competitor's targeting 50 mobile gamers and this poll completed in 28 minutes and cost 100 now the reason that you're able to see the screenshot here is that you have an option of making polls public in this case our customer did make this poll public so we we are able to show the screenshots but generally polls are private by default so those are some of the examples of using pikfoo there's a lot of other examples to show but you can find me offline or later for that so wrapping up in summary three things number one uh pickfu provides affordable

and actionable feedback in hours and on days and number two qualitative market feedback it's it's a valuable tool in your tool belt and it's underutilized and it can really help you stack the deck in your game's favor and then finally understanding your users is critical before you launch so if there's one main takeaway that's it um and with that i'm all done so thank you so much for your time um you can learn more by scanning the qr code or checking out that url and um if you are creating a game and you want to try pickfu you can find me or my co-founder justin on discord and we'll run a poll on your game so thank you very much thank you uh john absolutely great you know uh perfect and um i think we might have a few questions but i just want to say that was that was really interesting i was watching that kind of going now this is this is super useful and i've got a few questions i was going to ask you myself as well if that's okay we've got a couple of minutes so um i can see we've had kind of five questions

there we'll come to those in a sec um i'm grateful that you put up the q r code again at the end there because i was going to ask you if you could you could um have that again for photoscan so we'll leave that on the screen while we talk i i appreciate that and again i was going to um ask when i saw the the your earlier slides in there how it was that you had so much information on your panel like their educational level and that kind of thing and then you said that of course they're they're all part of your um the community that you've created so that they're all signed up under nda how how do you go about um getting those users those testers in those users do people apply to be part of the program um what's your what's your process for for selecting the uh the the people who take you take part in in the um in the polls absolutely so we work with a number of third-party panels on the back end too and we connect with a bunch of different panels and then we have our own screening questions and surveys and so on to allow people to come in through those third-party panels uh to join our panel and so when polls are published then we

choose the appropriate panel we put we put it out there and generally the panelists um panelists really enjoy doing these polls oh that makes sense that's good um and and also just from the the then the game developers perspective once they've um they've signed up to to to pick foo what's their experience so is it is it all sort of self-serve they once they've they've become uh they've subscribed to the the the service or whatever they and they have a game to test is it all kind of managed through a web front end and a person and a developer can log in what's the experience like of of getting something tested is it ready to be straightforward yeah so pikfu is completely self-service uh we've made the process simple enough where you can start a poll in one minute what we like to say is that you can start a poll in one minute you're going to get results in the next minute and then generally your poll we will be done within 30 minutes depending on your targeting so um yeah we don't we we do provide some um

we can help you with your pull but generally the vast majority of our customer just come on the site build the polls themselves launch them and receive the results yeah great it feels like a a you know a relatively straightforward and affordable way to get just to just to quiz a bunch of gamers about a very you know very specific things like you know like your your your your uh graphics your your icon that's the word absolutely yeah and and other facets as well that's great so i'm gonna i'm gonna dive in here we've got uh five questions and we're we're okay for time so i'm gonna put these to you if that's all right john looks like many of these are anonymous but um so one of our viewers asks uh could you please comment a bit on the difference of answers you see from certain user profiles um is it possible to foresee the preferences based on the profiles i guess this is about you guys spotting a pattern perhaps in the type of responses you get but um can you speak to that at all um i can't speak towards about patenting although that is interesting but i would say that um there are

absolutely different you can absolutely see differences between different types of audiences we see in fact we see publishers come on where they will run multiple polls targeting different audience segments because they want to see the difference in responses between let's say the general population um and then male 1 male mobile gamers or you know female mobile gamers um you can do that within the scope of one poll because we do allow filtering within your answers your set of answers by demographics by income and so on but you can get more responses by running separate polls so we definitely see um we definitely see customers running multiple polls two different audience segments and sometimes they're even using that to try to figure out does it make sense to actually target segment a versus segment b in terms of their marketing right like if they see that uh male mobile gamers of a certain age range are

actually responding better to their creatives maybe they're going to tweet maybe they're going to tweak how they're they're setting up their marketing messaging so that they can target that segment instead gotcha and in fact on that on that note the next question says do you find that users focus more on the on the visuals or on the texts and the the subtext there from this questionnaire is that they have a feeling that both people don't read anything no that's a complete yeah absolutely that's a completely fair question and so that's why the quality control side of our panel is actually uh matters a lot to us um we have um so we all the panelists do look at the options compare them and read and if they do not then any um any customer can flag or call to note any uh any responses that are dissatisfying or didn't follow you know didn't follow instructions and then we have algorithms on our back

end to scan all the responses and and so on and human raters as well to make sure that the respondents and the answers are on point and we will um i will just say that we will exclude with extreme prejudice panelists who do not uh who are not providing value right that's great so you're kind of making sure to curate that audience as well and make sure that they're going to give you the most value um uh now this is a really good question it's something that occurred to me actually and i didn't ask it because it was dropped into the chat here but um the question is is when do you believe is the best time to use this kind of rapid user testing um should it be during the design already to avoid loss of lots of time and work my own i was watching you uh your presentation thinking oh should a person be doing this the moment they have a concept for the game or should they be doing this just before release so i guess that you know that that's the question that uh has been preempted by someone in the audience there but what's your absolutely yeah that's a great question so what we found is that the earlier you use it in your process

the more um the more success a rapid user test like piku can contribute to your to your product or your game um because basically rapid user testing like pikfu is course correction so if you're if you think of you know you're creating a game and you're starting on a journey it's better to make sure you're heading in the right direction earlier rather than later in your path now if you can you can wait until you get to launch and you get to soft launch and realize that things haven't gone well and then try to course correct and pixel can be helpful for that but wouldn't it be better to know earlier in the process like hey i have a concept for a game targeting uh you know like ios like females who play ios puzzlers wouldn't you like to know that yeah this is a concept that actually resonates well with that target audience before you start committing to art to character sketches to design to programming all of that stuff right yeah so we think that we generally try to advise the earlier the better

in terms of this kind of testing oh that makes that makes perfect sense because i can see one of your examples was about um uh trying to work out perhaps what wasn't going as well as expected with a soft launch and i can totally see that kind of course correction if something's not right it's a kind of troubleshooting measure you can get feedback absolutely but but i i totally see that this is this is at the start of a project a few hundred dollars well spent just to make sure you're on the right direction before you you even kind of start because otherwise you've got to build you might have to be rebuild a whole ton of assets right if it turns out you're just going you know so uh well that makes a lot of sense um i'm gonna uh here we go let's let's dive into the next one there um i think you might have already asked answered this through the course of your presentation but i'll put it to you anyway because a a a an attendee asked it um how do you get these responses are the respondents paid and i think you said that they were but do you want to tell us a little bit more about that please yeah yeah so again we get the respondents by working with other third party panels and they are paid so they're paid a stipend for every single uh for every single

response that they give that's great and it means it means that they're they're um uh incentivized to take part and take it seriously absolutely exactly exactly that's kind of how that's what incentivizes the speed and uh how how rapidly we can get those responses great great great um so this is a final question this is kind of i guess maybe a little bit a little bit philosophical here but one of our anonymous attendees asks is there a risk that creativity might be overlooked to service user feedback i guess that's a question perhaps about the uh the kind of a developer's vision maybe versus uh mostly the focus group what do you make of that so i think that pikfu is a really good tool to get data points um about your direction but i think it actually still yeah i i alluded i alluded to this in a slide earlier with regarding our art concepts actually is that you can actually knowing that you have this safeguard and this this tool for validation you can

actually get more crazy with your art styles you can get more crazy with your wild concepts because you can sketch them out and you can test them out and maybe one of them will actually strike with your target audience that you would not have known about before testing or you might have thought before testing you might be self-censoring and saying hey we're targeting this segment but no this idea is never going to work or this art style is never going to work but now you can actually sketch it out and put it up there and within like a couple of minutes or hours you can know at least hey you have this validation like yeah that did work or you know like maybe it didn't work and here are the responses why and now at least i understand why something's working or not yeah absolutely and in fact that's a that's a really good good point because i was thinking that of course this is a business after all and and you know and and a person and a person's vision is one thing but actually if you're making a game to sell obviously you need to to appeal to your customers and that's great but you're absolutely right in fact actually as a way to kind of get creative and

then test those creativity and actually put a little bit of faith in the audience and see what they respond to what a great way to just you know see to try out some stuff yeah exactly exactly we see a lot of and we're seeing more and more sort of art concept tests on pikfu from our publisher gamers once they start realizing like actually we can test more things here you know yeah that's great well look john we're at the end of our time and it's uh we're just teaming up our next speaker so i will just take the opportunity to thank you very much for coming along and and giving us that great presentation round of applause from the audience you'll have to imagine that coming from uh forever a number of people hopefully uh sometime soon maybe next year we'll be able to come and see you in uh in person in uh in san francisco there or you'll join us over in london or something like that next year thank you um but yeah and in the meantime folks if you've been watching this uh this presentation you'd like to find out a little bit more about pick foo um john you're in meet to match i guess or in the discord perhaps and you put your your details are on the screen there so if i can email you yeah exactly so jonah picks who i'll be in discord all

weekend and in need to match great stuff well thank you very much for joining us and uh yeah well i appreciate it and everyone look out for look out for john and pig food thank you thank you very much you