Research Methods to Improve Live Game Retention, D1 to D100

[Music] all right good morning everyone thanks for coming it's very kind again a little bit of housekeeping before we begin please set your phones and noisy things to silent or airplane mode we'd ask that you please complete your post session surveys after this session ends we've left some time for questions and answers at the end of this particular session please keep those questions concise so that we can answer as many as we can and lastly if there's lots of questions or you want to chat to our lovely speakers at the end the post session area where you can chat to all of us is actually just outside this room out there in the corridor so we'll be there to have a quick chat if you'd like okay good morning hello my name is sebastian long i'm director of player research a games user research agency a player data consultancy and it's an honor to gather and introduce these four fantastic

talks on improving retention specifically how player data can make better games and game design choices leading to players sticking around in our games for longer we all want to make games that players love for a long time to do so to ensure we design a game that holds players attention for months we cannot leave game design to chance nor to intuition nor to uncertainty and the bigger our player bases get and the longer they play the more distant players real experiences get from our own assumptions or expectations player data has always been used to combat these assumptions the more we know of our players journey through our games the more deliberate we can be to achieve good design and keep them on that desired journey through user tests through

player interviews through surveys and polls in-game telemetry and a whole toolkit of data gathering research techniques applying player data to game development has never been more widespread than today in this era of live games and evolving games and connected devices where retention is measured more like in months than in minutes but getting rich insight from players our existing player base is really tough and researching potential retention before public launch is tougher still so in this session of microtalks we're going to take four case studies in using player data to explore retention toward building a game that players love to come back to again and again in each talk you'll be hearing from games user experience researchers so let's talk about that first games user experience researcher is a

professional game development role dedicated to gathering and analyzing player data to inform game design choices games ux researchers design and conduct research studies to generate player data data that is actionable that is rigorous that is on time and is persuasive design teams perhaps yourselves will roll that data into your decision making toward better design it's sometimes player data that inspires new game design ideas to be inspired by our players because we know them better their desires their motivations their spending it's sometimes data that continuously improves game design towards continuous improvement through iteration by better understanding our players interactions with our work in progress designs that we have games user experience researchers like us typically bring background in psychology cognitive science ux anthropology

and similar these backgrounds in research ensure that the data we generate is rigorous and can be trusted by the game development teams to affect all parts of game development as a user experience researcher myself i often find that player data means many things to many people so before we hear from these case studies let's talk about data player data is more than just analytics it's more than just surveys it's more than just app store comments in fact there are more than 30 formalized ways of getting data from players each of these information gathering techniques has strengths and weaknesses for example analytics is fantastic at telling us what players did in our game but it's often very poor at telling us why they did that eye tracking is good for telling us where players were looking at the screen but in isolation can't tell us whether

they understood what they saw the primary role of a games user experience researcher like us is to choose from these many research techniques and match them to the game development challenges you face and it's a universal truth that using just one of these only one leads to blind spots so it's not just neces not just ideal to use more than one but necessary to get good data to inform design in research terms each of these is called a method and when we combine and triangulate using multiple sources of data that's called mixed methods here we're talking about retention in this panel so we'll be focusing on these research approaches ones that are designed to gather data over long periods of time that is rich actionable and useful to you although we'll touch on other methods as well in each talk you'll be hearing advice

from these guys about why player data was needed which methods were used and the impact each of their studies had on their games to help bring these learnings back to your own teams i propose considering them through these four lenses firstly consider how these studies were made rigorous we captured sufficient data we reduced biases in that data secondly how they were made actionable ensuring they informed specific design choices thirdly how they were made to be on time these studies we're going to hear about how are they made to be done not just quickly but cheaply using resources we already had and critically at the right time during development lastly how they were made to be persuasive to ensure that the data they generated was efficiently actioned by the development team these four topics are the hallmarks of

good player data okay please allow me to introduce our first speaker alex cohost from glue thank you all for this opportunity it's a pleasure to be here and to share some insights that i along with the teams at glue and crowdstar learned in one of our projects last year focused on understanding the first time user experience and improving retention specifically in our game design home before diving into the details of this case study i wanted to share some background information for anyone who may not be familiar with this game design home is a free to play mobile game that gives its users opportunities to release their creative spirits through interior decorating and designing through partnerships with many top brands users are provided the opportunity to design spaces including but not limited to living rooms dining rooms and bedrooms using furniture and decor that is representative of real life

to provide more context for this talk it's necessary to share some information about the core loop within design home design homes core loop is all about room design challenges players are presented with five or so challenges every day and each challenge has its own description to prompt players with ideas about how to style a given room these challenges can be separated into two main categories those with design requirements and those without for example in the house hunters challenge that is shown on this slide the perfect painted lady players need to design a living room with eight required items and up to eight additional optional items and players were required to design this room with four eclectic styled items along with a blue sofa last year our research team worked hand in hand with the design home team on a project aimed specifically to improve early day retention targeting days two through seven typically in free-to-play mobile games

you'll see retention curves similar to this the one on this slide or there's high churn by day two but with a flattening tail with each day after that usually in free-to-play games retention has a cascading effect where the more users that are retained earlier on the more users that are retained later so using this chart as an example if you were to improve day 2 retention from 35 to 40 percent retention on days 3 through 7 should see improvements as well which is exactly what we in the design home team were trying to do so now that we knew know knew that we wanted to improve our early day retention how would we do that we we hypothesized that improvements the first time user experience would address this and so the team designed and created a new fatui which would focus on teaching players core interactions earlier in their experience for this project we relied on two primary methods the first is play testing where we watched how players

interacted with the first time user experience design noting any usability concerns along with noting what players did upon reaching the core loop after the onboarding process in addition to this the second method we relied on was a b testing to compare different versions of the first time user experience for play testing we partnered with our vendor playtest cloud which allowed us to recruit our target audience as well as to distribute builds and watch how players interacted with the different fertility designs we were able to recruit our target audience was just casual gamers that had no previous design home experience we asked that players to play the design home for approximately 20 minutes and we wanted to see how players would go through their first session in a completely natural way to assess how they interacted with the vatui as well to see if they are able to recall what they learned once these players reached the core loop because we wanted a very natural gameplay we provide no additional

context and no additional task outside of the onboarding process and what was already in the tutorial flow for this project it was broken out into three into four phases three particularly that were focused on play testing different iterations of the vatui as well as an a b testing phase when we tested the current design of the future we noted that we had two main issues the first issue was that players didn't fully understand the context for designing rooms and that players would place too much emphasis on certain keywords that were in the challenge descriptions in the example shown on this slide the description told players to design an eco-friendly living room during testing we witnessed several players become hyper focused on this eco-friendly keyword we noticed that these players would scroll and scroll through furniture endlessly looking for any sort of furniture item whether it was a couch a table or side table that was

eco-friendly even though within the design home game none of these furniture items were tagged as being eco-friendly the second issue that we witnessed was that players who moved on to other challenges did not notice that some of them had specific requirements players who did not notice the requirements would design some very beautiful rooms like the one in the slide but unfortunately would be met with a notice that told them requirements had not been met when they went to submit their designs not only was this a very frustrating experience for these players but what made matters worse was that these players had the potential to spend most if not all of their currency depending on what furniture they selected when designing the room with frustrations high limited resources and potentially nothing to do at that given moment players had a high potential to churn we hypothesize that these issues stem from the short and straightforward design of the original fatoui players

were introduced to the game and taught how to choose and place furniture and decor within a room but they are not taught specific aspects of the core loop including requirements and designing within constraints based on this hypothesis the team redesigned the tui to teach players additional fundamentals and interactions that they would encounter when reaching the core loop in the new version of the first time user experience the major change consisted of adding two new additional tutorial challenges these additional challenges were added to teach players key features and interactions including inventory management in the second challenge and understanding challenge requirements and how to use filters to optimally find furniture in the third challenge when the design home team was ready with the redesigned patoui we set certain criteria or success factors that players needed to demonstrate during play testing before we would integrate this new future design into the live game

the first success factor was that all players should notice requirements and they should notice this before they tried to submit a room design the second success factor was that we wanted to see players use filters to optimally find furniture for the designs within any given challenge when we tested the newly designed future 2.0 we encountered the same requirements issue that we saw when we originally tested the first design players were still not recognizing the challenges in the core loop had requirements and they still weren't using filters to optimally find furniture and decor so why were players not recalling that there were requirements within challenges we hypothesized that this issue was caused by the interaction design of the requirements and filters tutorial essentially this tutorial was guiding players to specifically just press a button they weren't really retaining information about filters or requirements which was supposed to be the mean the meaning behind that particular interaction

so the team iterated on the design of the specifically the third tutorial challenge and revised it to include a more granular approach to filtering where players will walk through the individual steps of seeing that the room has requirements opening the filters menu noticing the variety of filters available such as brand color style and even a classy requirements filter option to streamline their experience and then finally removing the filter once they'd already met the requirements after making these changes and re-testing the design all players within the play test demonstrated understanding of filters and requirements at this time the team had also addressed some other issues that are outside of the scope of this talk but it was at this point that we felt confident in the design and we wanted to release this to our player base so after making these changes and re-testing the design

so at this point the team set up an a b test with two different versions of the vitui and to mitigate risks traffic in the live game was split with 95 percent of our new users going to the old fertility design and five percent receiving the new fertility variation and if positive performance was recorded during the a b test we plan to increase traffic to the new for tui the data pm team on design home then tracked retention rates specifically looking at different cohorts and also implemented event tags to track how many users were each in each step of the fatu diving into the data what we saw when we measured day two retention was that we had significant lifts across all day two cohorts suggesting that the redesigned the fatoui created a better user experience that retained players when we evaluated retention at day seven we also saw signif we saw a slight

decrease from the lift scene in day two which makes sense as other factors such as economy design and whether players are progressing could be having more influence at this point but we still saw significant lifts in retention at day seven as well i mentioned previously that we implemented event tagging to do granular tracking and funnel tracking during future anytime you're testing a first-time user experience tracking each step of the funnel will uncover insights on which areas can be addressed that will also have the largest impacts to potentially retaining players in our case study during a b testing we uncovered two major charm churn points which happened to correlate with two usability issues that we saw during player testing unfortunately due to the time constraints of this talk i can't elaborate on these however the point i would like you all to take away is that by play testing the first time user experience design before implementing this in the live game we were able to

have insight into what was causing these turn points in the funnel the team had specific direction that they could take with data to further improve the first time user experience which would then incrementally improve retention so in summary by taking a test iterate and re-test approach with the first time user experience we we were able to prove early day retention both at day two and day seven by targeting specific areas within the first time user experience so thank you all and at this time i'd like to introduce our next speaker molly [Applause] hello everyone my name is molly sirota i'm the lead researcher at the sony san diego studio and i'm going to continue our talk on retention okay so specifically what i'm going to

talk about is leveraging a method known as a diary study to study retention and i just want to get a feel for the room how many people raise your hands if you're not a researcher okay it'll help me understand how to articulate what we did in this what we do at the san diego studio so this is going to be a high level view of how to run a diary study but then i'll get into the details so in the beginning you're going to recruit a handful of users it'll depend on your game and what your question is for us we typically recruit around 50 people a mix of spenders and non-spenders a mix of first-time users and experienced users and we send them a virtual diary which is pretty much a survey link but it's not a survey it's it's an open diary and users play your game then enter feedback in this virtual diary and then you read the feedback that's coming in and

then that loop continues they play your game they give feedback through this virtual diary you monitor data that's coming in and the benefit of a diary study is that it is leveraged to see how your game plays out or unfolds for the user over an extended period of time so depending on your question some people run diary studies for a few days or a week we have a tendency to run diary studies for three to four months and that's because i work on a baseball game mlb the show and what we're trying to do is capture our audience for as long of the baseball season as we can but there are other benefits to running a longer diary study for one you only spend money the first time you recruit people to this study and now you have a user base to constantly tap into any time questions come up in your studio about your game

so let me get into kind of the details behind a diary study for one your diary study is it gets at really the end benefit is that extended period of time but to outline it is you're encouraging your users to enter data on a daily-ish basis really it's just whenever they're playing their game your game and it's any time that they're playing your game in their own environment you're not bringing users into your studio or into your office to test they're playing in their own environment and giving you feedback when they're actually experiencing it the other thing with diary studies is that as a researcher you're giving them a suggestion of a topic to think about for the diary like game play or customization but other than that it's up to the user to then decide what

aspect of customization or gameplay they're going to discuss and the benefit of this is instead of using a survey where you're specifying the topic that users have to give feedback on you're now getting feedback on what's most important to the user in their experience and the strength of this is that sometimes when you're designing a survey you are selfishly putting in questions on topics that you want feedback on but you might not realize that there are other topics affecting the user experience that you didn't even know to ask a question on and that's another beauty of a diary study the other thing is that you are getting real-time feedback on the user's actual experience when you bring users onto on-site to your studio you are trying to simulate a gamer experience with a diary study they're allowed to play on their own time in their own environment so

you're getting feedback on their actual experience in their real world environment now as was previously mentioned with any research method there's strengths and weaknesses so this is going to be an outline of how you can strengthen the areas of a diary study that have some blind spots so one thing about a diary study is that it might not always address the topic that you were hoping people would discuss for example again i work on a baseball video game maybe there's some discussions that day in my office where people want to know is something going on with hitting are people struggling hit with hitting and then i look at the diary feedback and everybody's talking about pitching you have already recruited this panel of users you don't have to pay to re-recruit you don't have to wait to find the people they're already available to you so you can quickly send out a survey with close-ended questions

that get at any area in the diary study that was missed from hearing from people the other thing about diary studies is that it depends on the participant in terms of how articulate they are in the feedback that they give you so there are times where their information can be vague or just misunderstood and let's say you're reading diary feedback coming in and you can see that people are struggling with a certain aspect of gameplay but you can't exactly tell why again these people are already recruited they're already available to you hop on the phone you can call these people and ask more detail you can hop on skype and have them video stream gameplay if you need to see what they're doing if they're local you can bring them into the office at this point and get follow-up feedback on where the issue is actually occurring for them the other aspect of a diary studying and

really with any type of qualitative research is that you can't tell the frequency at which it is happening and that's because the sample sizes are smaller so again i said that we typically recruit 40 to 50 people for our diary studies but we have millions of players so it's not going to tell us at what frequency is this issue occurring for how many of our users is this issue occurring so if in your diaries you're seeing that people are struggling getting into a certain area of your in your game then reflect back on telemetry data so that you can see to what degree is this impacting everybody who's playing our game so now i'm going to get into specifically what we've done with mlb the show to study retention so mlb the show is a game that releases once a year but we do have live content that we release every few weeks in the form of baseball themed

events and the whole purpose of these baseball themed events is to offer new and different ways for users to play our games so that they continue to play our game what's shown up here is an example of one of the events that we put out to users and it encouraged users to build a baseball team made up of just baseball players that are of the age of 25 or under then they go online and play against users who have created similar teams so again the point of these events for mlb the show is to enhance retention so as a researcher i come in with a diary study to see is it working are we actually retaining people with these events and if not why not so let me talk you through what happened in the beginning of our study so the 25 and under event was the very first event

that we launched with the launch of our game and what happened in the diary feedback that we started getting is people were talking about the theme of creating a baseball team with baseball players who were of the age of 25 and under and they liked it great motivated just to check out the event we want that they liked the rewards they wanted the rewards great we're still working towards retention because we're capturing people's interest but from there on out we kind of went downhill users started to try and create a team of baseball players that were of the age of 25 or under and could not for the life of them figure out how old people were this information was you had to dig really dig on a player-by-player basis to find out how old they were and so it got to the point where users gave up we thought we were going to be getting data in the diary study on game play how

did this um theme or this particular experience play out for users did they keep playing to get more rewards were we keeping their attention and then we found out people couldn't even get in to play the event so what do we do with that type of information we create the next event off of everything that we learned from the previous event so what we did was one thing that happened with the first event is that it launched with the beginning launch of our game which means that our users base inventory was very small at that point even if they could figure out how old baseball players were they didn't have a lot of their own inventory to pull from which means we're suddenly pushing a lot of people to the marketplace which was not going over well with people so we changed the second event to work better with the inventory that users

already had available to them the second event had users build up a baseball team that had to do with right-handed hitters and left-handed pitchers this is something that the designers know is very easy for users to do at an early stage in our release of our game the other thing that they did with the data that came in from the diary study is they put out a patch to quickly address this aspect of having no idea which players to build your baseball team off of because they don't know who qualifies and who doesn't well now there's an icon that goes on any player within the baseball game that identifies them for the event going on the second that event closes that icon goes away and then comes back up on any player that then meets it matches the qualifications for the next event that opens up the other thing that designers did was they leveraged information that the user was going to need to build their

baseball team with aspects that were front and center on a baseball card whether a baseball player is left-handed or right-handed is on the front of a baseball card this was information that was going to be easy for the users to identify and what ended up happening is at this point we're seeing positive feedback coming in through the diary study when comparing when comparing how their experience went with the first event but we also look at telemetry data and we see that we've more than doubled the amount of people who are able to actually get into the event in the first place so in sum the thing with diary studies is that it allows you to study your users and your game play experience over an extended period of time so that even if your game is testing well when you bring them on site for a play test for an eight-hour play test a two-day play test a diary study allows you to see how

your game experience unfolds over time and to not only identify where people are dropping off but why people are dropping off of your game and for us it allowed us to quickly iterate so that we could minimize the amount of people that we were losing and therefore impact that our ability to retain users attention to retain users interests in our game and with that i will now introduce jen [Applause] everyone my name is jennifer torson and i am a user researcher over at scopely scopely is the publisher of free-to-play mobile games and we have a really diverse portfolio from casual word games to rpgs forex strategy

research is a really important part of the development process at scopely and we have a team of very serious and professional researchers working on all of our titles at scopely our research team works to deliver actionable insights at every stage of development from early concept to live operations because of this we believe in researching games holistically trying to understand how every facet of the game contributes to its overall long-term experience one of the ways we do this is by running longitudinal or long-term studies where we conduct research that looks at play experiences over a span of days or weeks rather than a single session while that may sound like it would take a huge amount of work the reality is long-term studies are easy to set up and run they only need an extra day or two of researcher time than a traditional usability test would and the benefits are huge giving us really deep insights into the game today i'm going to talk about two cases

where we use these tests for great results first we're going to look at star trek fleet command our mobile forex strategy game during that game's development we use longitudinal testing to understand player motivations and progression during the first week next we'll look at the walking dead road to survival where we tested post launch to redesign the way we taught players about characters during the tutorial to help increase retention down the line so longitudinal studies at scopely typically take place in three phases and use a variety of qualitative research methods the first phase is an on-site usability test these are pretty standard sessions that are widely used in industry we recruit about six people to come into our office and play the game one-on-one while a moderator sits beside them to observe and talk about their experience and this is where we can capture initial impressions about the game as well as any usability issues

that come up during the first hour of play our second phase of testing is a diary study so after they finish their usability session we send players home with the game installed on their personal phones and ask them to play every day for the course of a week or two weeks usually about 30 to 60 minutes a day we might let them do this as self-directed free play if we want them to just kind of test the overall experience or we might ask them to do specific tasks every day just kind of depending on what we're trying to look at players are then sent a survey via e via email every evening and ask to record their experiences from the day what they did how they felt about the game if they were ever confused things like that these are generally very broad open-ended questions just to get the player to give us as much detail as possible without us guiding their responses too much the final phase of testing is a follow-up interview where players come back to the studio to show us the progress they made in the game

we prepare for the interview by reading their diary entries every day and that really helps us to plan more in-depth and specific questions that really dig into the nuances of the players experience taken as a whole these three phases allow us to see the effects of various issues over time and to better suggest how to fix them for long-term results so the first case study is going to be from star trek fleet command again this is our mobile forex strategy game when we did this testing fleet command was still in development we had no idea at that point how players were going to be engaging with the game or what their first week was going to look like one of the features in the game is a mission system so a series of ongoing quests that we designed initially to help players learn about the game fundamentals and then later to give them reasons to keep coming back and getting into more difficult content our design team wanted to know if the system was working as intended players understand the mission system were they motivated by them

so we decided to use a longitudinal test to see how players were actually going to experience the system the first phase of the study the on-site usability testing was the first hour of play and we had really good results that first day we found some usability issues such as with iconography a few places where we were using jargon but overall players were able to understand and complete their missions so that first 45 minutes of play were really solid so then we shifted our attention to the next seven days experience a few days into the second phase we started to see some issues popping up that we hadn't anticipated in their nightly diary entries players started to talk about how they were frustrated with how difficult the missions were becoming and how they were left with nothing meaningful to do while they were trying to grind the materials or levels they needed to complete those quests and by the end of the week they were also reporting that the missions felt repetitive since they follow a pretty

standard formula kill 10 of this gather 20 of that and they really wanted something to mix it up and we knew that these could turn into really big problems over time players who feel like they can't progress or who are just bored are not going to stay around in your game so when players return for their follow-up interviews we really dug into the issues and what we might be able to do to help them and from these conversations we decided a good solution would be to branch out from the mission system and provide additional ways that players could engage with the game first in addition to daily missions we added daily goals as a way to give players easily achievable quests that could be repeated daily for good rewards giving players stuck at a difficulty wall in missions a meaningful way to get the materials they needed to progress next we updated events which we had planned as a late game feature for elder players to be available right from the beginning of the game these limited time events are always changing which helped

relieve the repetition that players were starting to feel this solution of adding more ways to play not only helped to solve the issues that we'd seen with missions but continued to benefit players long term through through rewarding multiple kinds of engagement it also gave our developers multiple ways to introduce new features into the existing loop after the game went live now the second case study is from the walking dead road to survival and this is our rpg after the game had been live for a while the game team decided that the early retention was not where they wanted it to be we looked at the analytics from players first weeks and realized that our players were not progressing through the story campaign as quickly as they should they were losing battles that they should have been easily winning our hypothesis was that these losses were causing player frustration in churn which was explaining our low retention we guessed that the most likely cause

for losing battles was using characters incorrectly so we decided to use a longitudinal study to see how players were actually using and understanding their characters during that first week phase one seemed to go really well during the on-site usability test um players went through the tutorial and a bit of free play and during this time they were able to articulate the key points that we were trying to teach about the characters that you can get new characters and that you can level them up we sent players home with the game and as they started to fill out their daily diary entries we realized that our tutorial had not been as successful as it seemed our players wrote that they were frustrated by losing battles and progressing slowly just like we'd seen with the analytics data and when they returned for their follow-up interviews we looked at their characters and our suspicions were totally confirmed they were using far weaker teams than they could have been through all of the phases of our testing players were able to understand and recall the major points of our

character tutorial winning new powerful characters and improving them through leveling up and evolving but from their interviews and their diaries what we realized is they didn't remember how to tell which were the powerful characters that they should be focusing on so the first problem we found was when players won their new more powerful character during the tutorial during the test players win dr stevens he is a very obscure character from the comics that none of them really remembered so it wasn't a particularly exciting moment to win him from gotcha it didn't make them hyped to keep winning new characters so we ditched him now players win carl he's a major character and a fan favorite and even better while dr stevens is a healer carl is a badass attacker whose special skill is super strong in early battles the lesson we are teaching is now much more memorable and clear if you pull for new characters you can get someone familiar who's going to be awesome

another problem came from the character that players learn to level up guy named judd now jed isn't even a character in the series we made him up so right off the bat not exciting for players to be using him worse he's also pretty weak in combat even after leveling him up when players returned they were able to remember how to level up their characters but they were also now using jed and all of their teams when they had much better characters available so they were taking away the very wrong conclusion that he was an example of a good character to use we solved this by having players learn to level up rick rick is the main character of the series a big fan favorite and one of the most powerful starting characters in the game he helped us to drive home the lesson that we wanted to teach good characters will be made better if you invest in them these changes not only helped our players to start winning those early

battles in the game increasing in retention but we also ended up doing a better job of teaching the value of characters and creating motivation to engage with gotcha and both of those are incredibly important for the long-term health and revenue of the game so now we've seen two examples of longitudinal research in action what both of these studies really show is that optimizing the early experience of the game has long-term consequences teaching players what to value early on lets them know what to invest in later and creating paths for players to follow encourages them to engage in the behaviors that you want to see and gives them a reason to keep coming back these studies also show that many important issues only reveal themselves over time looking at the game as a holistic ongoing experience was the only way to discover how players were really going to use and understand what we had built for them longitudinal studies are not only powerful as we've seen but they're also

a really easy method to add to your regular testing most of the play time is unmoderated players are just off doing their own thing at home so adding length to the study does not mean adding effort the surveys used for the diary portion should be short to encourage completion and don't require a lot of time to set up or analyze we tend to ask the same broad questions every day and for every game so surveys can even be reused between studies the usability sessions and the follow-up interviews are the most time-consuming aspects since they require in-person moderator time but when you consider that you are condensing a week or more worth of data into a couple of hours of moderator time per player the time is really a bargain at scopely we now use longitudinal studies on all of our games from futures and beyond i hope that sharing our studies today will help to convince you to do the same thank you

and i'm now honored to pass on the microphone to our last speaker sarah from glue [Applause] all right everybody who's ready to talk about days 100 and beyond my name's sarah i work at glue mobile we make mobile games great news a new culinary award show has just launched where accomplished chefs can earn extraordinary rewards even better news we're going to talk all about how we did ux research for this update first i'm going to give you a little bit of a background on what cooking dash is and what went into the 2.0 update and then i'll provide some solutions for testing unfinished features designed for experienced players that unlock late in the game i'm going to show you how you can empower developers to make decisions based on facts

i'll probably i'll provide ideas on how to keep their research fresh and timely when studying live events and finally we'll talk about how all of these things work together so that you will see adoption from your teams and you'll impact on your wonderful products okay so what's cooking dash it's a mobile free to play game where you must serve food to angry customers within a limited amount of time while using griddles with a propensity to burn things fun right cooking dash launched in june of 2015 and the update went out in november and about five months before that glue user experience research had the honor to work with the talented game designers producers and product managers on this update up until the 2.0 update cooking dash relied heavily on new restaurants as a retention driver the team wanted to rely

less on this content heavy rep approach and instead shift focus to a live event called trial of style all levels in this event are procedurally generated additionally many complex layers affect success in this live event and all of these things work together to motivate players to play in new ways by complex i mean the most positive sense of the term much like a delicious layer cake on the bottom layer here you see the trial of style but all sorts of things work to benefit you to get on on the leaderboards and to to do well in the levels such as prize wheels outfits a gauntlet set of episodes and even pets and then doing all these things earns you a set of golden saucers which in turn impact your ability to be successful in the trial of style so when the team came to ux research to

talk about what they wanted to learn they said we really need to know if players understand this concept of style and if they understand how to get more of it we said that sounds great those are great research questions we'd love to help you figure that out because style equals success and in the great leaderboard of life those without style perish certainly users are going to understand this metaphor so how did we work to understand if players were getting this relationship if they were themselves understanding style we started first with something that we're going to call today a moderated tutorial play test session then the team iterated and we moved on to some diary studies the team said to us hey what do we do there isn't a tutorial yet and these features unlock really late in the game we said no problem first you need to recruit some really experienced players

and then you create this moderated tutorial a moderated tutorial involves something like a script that you work hand in hand with the team to create and we do this like a play test we do this in our lab but it contains even more such as this example here we see the trial of style and the moderation would be you the researcher and you're basically becoming the tutorial so for example we let the play tester know that the trial of style has unlocked it's on their home screen and then we follow up with some questions these questions unpack things like player understanding of the new features relationship to the core loop and we observe the ability of the player to be successful and we listen to the player describe their success so what did we learn we asked the player to say hey um did you tell me what style is and here's what we found out they didn't

quite get it so they said oh it's an additional test of your skill but i don't really know how i got 91 super coins we said super coins oh no pims corey what are we going to do and they said don't worry we feel very validated after this study and we actually have so many ideas we feel so inspired and i've got ways to address this already brewing in my mind so after completing the moderated tutorial play test sessions the team went back and iterated based on the things we found and three months later we conducted a diary study a diary study for for this particular um for this update lasted 10 days we recruited 19 participants from cooking dash and we got a mixture of spenders and non-spenders over the 10 days participants took a survey and the daily surveys are basically the same thing every day on the last day they take a daily survey and they also take a wrap-up survey

now daily surveys keep it vague because we don't want to lead the player on we want to know about what they're uncovering organically in the game so we ask things like what was your intent to play today did you encounter any frustrations did you observe anything new what were your goals did you achieve them and how motivated were you to play today on a scale of one to seven you can plot this motivation over the duration of the study and you can understand the dips the highs and lows and the dips in their experience by looking at their free response questions here you can see that someone is trying to strategize in new ways so they are indeed contemplating success wrap-up surveys get to specifics we ask things like can you actually define the features and you can name the features and ask them to explain it what did you discover and when did you discover it along your journey

how challenging were each of these features for you how satisfied were you about each of these features you can show them icons or images from your game and have them to describe them or describe the function or describe what they think they look like and you can have them talk about relationships a great way to ask someone to explain to you if they understand something is to actually have them pretend like they're explaining it to their friend or to someone that doesn't isn't familiar with your game all right so we've talked about the 2.0 and we've provided some solutions but how do you help your team get to decisions based on facts creative teams spin it from observing the player engaging with their designs so for example cory and pims actually watch the sessions and here they started to realize that people were really confused about the bonuses and they the participants were thinking that the bonuses were actually rewards those players were really confused about

the bonuses that led to success in the trial of style they thought they were rewards how can we help players learn that all the cool stuff you design like outfits and pets also impact style corey says i already know how to fix this don't worry guys what's important is to talk to your team about consequences the fact is and hey we're trying to get to the facts is that players think style is just coins but we need them to know it's more than that how will this means understanding impact the entire player experience well if this is an entire meta update the entire system breaks down another way to help your team get to the facts is to layer data so we'll use that cake metaphor again you need to layer things people say with the things they do and in this case i'm talking about telemetry stuff you're pulling from inside of the game because you're going to be able to come to really strong conclusions and maybe you'll also make your dev team a little bit hungry let's take for example the price wheels

what we learned is that 20 26 of the items earned throughout the duration of the study were related to outfits we also learned that all of those things in red participants didn't quite understand the purpose of let's take for example outfits this participant says i don't understand how to get outfits only by spinning what are the benefits we also saw that no one even earned an outfit saucer over the duration of the study so so you'd think that people might be really confused about prize whales if half the stuff they're earning they don't really get but it turns out they really love them it was the most motivating feature in the game another thing they didn't quite understand was pets and when we asked how motivating was it to get a pet they said well i don't know i didn't see this feature i don't understand these i didn't get any of those so i don't know what they will do so together we we could come again to the team and discuss what we found and

and pimp said dang people like price wheels even though half the time they don't know what the purpose of what they are getting even is so with some tweaks we can make this feature amazing and corey said i know what to do so we've gotten to the facts but how can you keep your research fresh one thing you need to do is launch your launcher diaries before the live event goes live that way you can gauge satisfaction before and after the event for our study we used a net promoter score type question and we saw a 16 point increase overall so if you're keeping things certified fresh are you going to see adoption how did it go for us one of the things we saw and we i talked about it earlier was that people were really confusing style they thought it only related to the coins the team iterated and made this big s big purple s

in order to decrease this association this association that it's only coins the team reverted back to just the typical coins that it looks like in any episode and then any boost that relates to style also became purple they also fix the issue of confusing boosts for rewards by removing the progress bar here's what it looks like now if you were to play the 2.0 update and what about those pets originally the pets only had an aesthetic effect but after seeing what happened and seeing people's misunderstanding they decided to act add an actual gameplay benefit so the pet now auto serves food as you're playing the level and what about impact cooking dash has maintained steady retention for over three years and hey we even saw an increase in revenue after the 2.0 launch

so how do you test unfinished features designed for experienced players we recommend moderated tutorials and player diaries you can pow you can empower your teams to make decisions based on facts by helping them observe and get close to the your players engaging with their the experience of their game and you can layer data keep your insights fresh by launching before the event goes live and if you do all of these things i promise you will see adoption and you will be impactful thanks so much for listening to our talks today and thanks to all of our dev teams for letting us work on their wonderful products sub's going to lead us through questions [Applause] okay so there's a microphone in the middle of the room i think if anyone has any questions we've got a time for a couple maybe

hi uh thanks for the talk it sounds like a lot of your diary studies had very small sample sizes how do you go about recruiting to make sure it kind of represents your audience your users so yeah representative sample sizes sure so it depends on your project on what type of player is going to best represent what you need but you always want to get some portion so even if it's a smaller sample let's say i was recruiting 12 people for my diary study then i want and i need to see a mix of experienced users and new users i'll break that in half six and six still a small sample size right well data that's coming in from the diary study if there's a trend trend meaning at least two or three people which sounds really small but these are unrelated people they don't know each other if they're talking about the same topic in the diary study let's say they can't get

into a certain area of the game you want to balance any research method by acknowledging its weaknesses with another research method that is strong in that area so a diary method doesn't have a large sample so then you look at telemetry data to see how many people are actually getting into that area in the game and as a designer of this game is that a problem and how big of a problem do we think that is so you you don't need a large sample size to address what's going on in the game you need it to understand why you only need a small sample size to understand why something is happening okay regrettably i think that might be all we have time for but we're all going to be outside in the corridor out here so do come out on our system questions thanks very much