Ice Age Adventures: UX Diagnosis for a Live-Ops Game

[Music] really good afternoon to all of you I hope you're having a great day so far very excited to be the first-ever UX summit here at GDC 2017 brief introduction about me I'm ohm tonton currently working as director of UX at digit game studios Dublin its Ireland's biggest independent game studio prior to digit I've been working as a senior UX designer and researcher for Gameloft's mr. New Zealand and Australia studios and before that I was a creative director UX UI for education gaming startup tap to learn which was incubated right here in the heart of San Francisco I've been in the industry for over 12 years now and I've drawn many hats during my journey but I must say I finally found my niche in UX UX stands for players voice in the game development process it's something I'm very passionate about as a methodology and I think it has great many future applications so I think that that is enough introduction about me let's see what do we have for you today presenting

Ice Age adventures UX diagnosis for live operation games a case study this is my elevator pitch what we'll be looking at how to gather data and work with other departments within the company how to analyze the data you have collected and apply heuristic diagnosis pitching the solution to the stakeholder group and designing a solution that resonates with both business and user needs a small disclaimer a very special thanks to Gameloft as you might be well aware we are subject to a lot of NDA's and gaming industry the case study you about to see is a reconstruction of some of the some of the work I did on a published project however the data points and the figures that you will see in this presentation are not the real data they have been tweaked slightly for the benefit of this presentation so before we dive in let's have a look at what's the game is all about Oh speedy so another hides this

to rage so that was the fun part guys no I promise there are more videos so as you can see this game was based on a very strong IP Ice Age series which has a huge global fan following it was targeted at a casual player base and the game was all about rescuing animals building a shelter for them and the players could do that by playing popular casual games like match-three infinite runner and so on the target demographic for this game included IP fans young adults parents analog stay-at-home moms a lot of this data was generated to our active community pages we had a very strong community for this game and our Community Managers interacted with the most engaged players almost on daily

basis a few most arts about the game during its lifetime it racked up 40 million downloads worldwide we have super active player base of 5 million daily active users and it was one of the top performers in game loss portfolio now as a lot of my fellow UX designers would agree sometimes when we moved around projects we inherit suboptimal designs what exactly do I mean by that any game designers in the house so with all due respect guys sometimes the system you design though they are functional they are not the best example of usability and intuitiveness one such system which I inherited when I was moved on to the Ice Age team was the existing even system let me give you a bit more context when a game is newly launched it generates a lot of engagement but the fact is the longer the game stays live its novelty value starts wearing off and the players start getting bored of the same grinding mechanics so in order to refresh the

user experience as developers we have to keep adding new features and that is where the even system comes in by looking at user data we found that while the average session time was around 18 minutes which is not bad by industry standards for those players who took part in the even system this average went up to 3035 minutes practically double of what we were seeing and it is well known if you can have players spend quality time and effort in your game it's much easier to monetize them so the even system was designed to increase the average session length which could then have a positive impact on our monetization so far so good but hey what's the problem nowadays when we launched a feature most companies there's something called a launch KPI it's pretty much like what NASA does with its project right based on those launch code KPIs you determine whether a future launch is successful or failure now the projected KPI for this game was sorry for this feature was 45% participation rate of existing player

base however the ground reality was only 20 percent of the players converted which was of course disappointing now even before I start looking at data to understand where does the bottleneck lie it's important to start with certain assumptions we had to answer two critical questions here why why do we have such poor numbers and what is causing this lack of engagement is it the content system of the event is it not appealing enough but the content pipeline was designed based on upcoming ice age movies which was pretty new and of course before talking to our players constantly through our community channels asking them what kind of characters they would like to see what kind of adventures they would like to go so we can safely say that was not it because we had a lot of active player feedback on the content design then what was it is it the discovery and usability of the system our players trying to interact with the feature but it's so hard to use that they are getting frustrated and just churning or maybe the evens are too

hard to complete so the players might be signing up but the competition level is so high they're just not feeling like you know getting through it or getting frustrated in order to validate some of these hypotheses and rule out others I had to work very closely with other departments within the company for example data scientists who are looking at copious amount of player data cohort analysis and so on analytics to understand how players are interacting with this feature where are they spending time where they're dropping off business performance managers who designed the soft and hard currency gates of the game now just by looking at how much currencies or resources players are spending in a feature you can have pretty good idea of engagement because the more engage they are more resources they're going to sink and of course the Community Managers working with them we could design special poles in service to gather qualitative subjective data here's an example of the kind of data we were collecting and how we were analyzing it number of people clicking on the even Seiken number of

people hitting the join button number of people dropping off from base drill downs even completion rate and specialty design service and polls now just by looking at the analytic funnel we could see where exactly in the floor the drop was happening where the churn was happening this of course gives rise to the next natural question why is that churn happening now in order to understand that we applied a very popular UX methodology which is called heuristic diagnosis heuristics stands for set of special cardinal rules which when applied to a flow helped us understand how usable or intuitive that flow is you can find more information about it on the Nelson Norman group website I would recommend that so let's see what we uncover when we apply heuristic diagnosis to our existing even came flow as soon as the player comes back to the game he comes to screen number one right here now this information that a new event is available is presented as a notification in the bottom left corner of the screen right above the trophy icon to give you

a bit more context when a game is newly launched as a UX designer you only want the most important call to actions to be presented because you don't want to overwhelm the player or create a choice bias however as I mentioned earlier this is also true the longer the game stays live in order to keep refreshing the user experience we have to keep adding features and what does that imply for a game like ours when the player comes back he will see five to six notifications as you can see waiting for his attention these could be the daily reward systems your Alliance systems help request special offers and so on so it is very much possible that this information that a new event is available might be getting drowned in that noise now if the player clicks on the even icon somehow he come to screen number two here the even surprise entered in a laundry list format now by the very nature our evens for design to fall into different buckets like daily weekly monthly and so on but tell me as

a player can you make that out just by looking at the screen and that information is important because different players will engage with different even types based on their preferences I may be the kind of player who just likes to play daily events because that's the kind of time commitment I want to do but you can see that I cannot make that decision just by looking at the information on the screen if the player clicks on one on the even type he's about to screen number three where you can see some more information about the rewards some call to actions and access to leaderboards if you click on the leaderboard button you were brought to screen number four here player can see the leaderboard and his position relative to rest of the players in the game now let's take a step back and try to analyze why is it in the first place a player would like to take part in the event system well the two things what are the rewards what is it I as a player will get if I take part in the even system or sync my time in it and second what is the risk involved what are my chances of getting that

reward and that is what comes through by looking at the leaderboard now it is possible that at any given point of time we may have three to four even running in parallel let's say daily weekly monthly and a seasonal Halloween special do you really think as a player I would like to go through three drill downs and four clicks in order gather all the risk and reward information I need for each and every even type and we are talking about around 16 clicks and 12 drill downs that sounds like a lot of work right so probably the short answer is no and when we looked at the analytics that's where we found a lot of churn happening because people were losing their way going down these multiple drill downs it was a big pain point the next screen this is the worst possible screen by the way if we have saved for the last this is the screen the player sees when event is expired and he logs back into the game let's try to see what's going to the players mind then C

then they see the screen yeah there's more fun this wait so imagine you are player returning back to the game you see a nice congratulation message what are we what we are doing here is we are creating anticipation we are raising expectation imagine you are hungry you walk into a pizza parlor what's going through your mind the chef is plating up a slice of pizza for you and you salivate and you are expecting positive gratification any minute and that's what a pair is expecting I mean it's a bit exaggerated but you get the idea so she is expecting all sort of exotic rewards to fall her way anytime soon however let's see what's actually happening here you are coming back to the game you see all the smoke and mirror and then the final score is zero your rank is in thousands and you reward big fat none you don't deserve it go away so after creating anticipation and raising expectation and the player sees this Queen she's like what the

just happened I mean while she was expecting positive gratification and then she sees the screen which is utterly lacking in any kind of encouragement it seemed almost like a dirty trick you know almost like a slap in her face so after applying Hugh's dick diagnosis here's a list of issues we found out three drill downs and four clicks to gather all risk and reward information that player needs for his decision-making text walls which were repeated throughout all the drill rounds that's a suboptimal design busy layouts confusing CTS and even though the screen which lacks any kind of encouragement yeah that's a long list now you saw that how we collected qualitative and quantitative data which helped us understand where exactly the churn is happening we applied Husted diagnosis so we know why that churn is happening do you think now I have enough information to start designing a solution not yet because

even before I can think of start designing a solution this is one more step in between and that is getting a stakeholder buy-in the stakeholder group can vary in composition from company to company it can have your product leads dev Li it's creative directors and so on now it is important to understand that any update said you as in case of our game has been decide upon three to six months in advance so if you turn up at the last minute and say hey by the way we have important UX fix to push it's just not going to happen trust me and take my word on that what we need to do is we need to get a stakeholder buy-in so here I am presenting my analysis to the stakeholder group which is the product manager and the dev lead in my case and the product has a question for me well this makes sense but why do you think it's more important than 50 of the improvements that we already have lined up and the devs have their own angle whatever solution you come up with will it reuse existing widgets because if the

solution involves creating and writing more code that can inflate the sprint time thereby delaying the overall release schedule and that's going to be on my head I didn't want that of course now this is where you leverage working very closely with other stakeholders within the company based on the analysis the business performance managers projected there would be a 15% drop in revenue if we do not push for this fix and this helped turn the tide the product saw that is a strong case for UX revamp yeah the money really talks that got them sorted and the devs also understood that it's important but we have to be very smart and efficient about carrying out any changes now perhaps you can understand why it was important to get a stakeholder buy-in because I'm not free to design just any solution that might be so expensive to build that it will never see the light of day instead the solution I needed to design had to hit three distinct worlds business goals of increasing player participation rate and boosting ROI user

experience goals of reducing cognitive load and making the whole experience feel more rewarding and of course the development goal of working with strict budget constraints so now we have a green light the first thing I like to do afterwards in order to start on a solution is to look at the root cause of the problem even before I consider a wireframes or hierarchy issues the first thing I start with is the logic flows this is the existing even logic in our game is there a new event available if yes show player our new new notification on heart if player signs up maintain the new notification if not the showing remaining time now you may well recall when we were playing heuristic diagnosis I did point out for a game like ours which has been live for long any time the player comes back to the game he has 5 to 6 notifications waiting for his attention and we reduced it is possible that this information that a new event is available might be getting drowned or it may be the player is so used to the notification span he

just treats it as white noise and blanks it over either ways the discovery of this system is suffering so what did we do we added another step trigger or even sign up UI at launch now whenever a new event is available we trigger a sign-up UI what that does is it allows player to sign up for the even without going through any further drill rounds if he so wishes to what did we boost we added discovery instead of making player work to discover a critical piece of information that you want him to consume we have surfaced and brought forth that information for him the next flow I looked at was the existing even over logic did player make any progress after signing up if yes display a X reward value based on progression if not show unnerve value now a lot of my programmer friends would say what's wrong with this logic makes sense right if the player did not make any progress he would see another reward value but what I would

like to argue is well the logic is spot-on from a neuropsychological point of view we are dealing with a very casual player base a lot of teen and tween players and when we show them a reward screen their expecting positive gratification so what did we do we added another step display minimum participation reward now even if the player does not makes any progress after signing up we still give him a small participation reward what did we change compared to the previous flow we added empathy what this does is it makes us the developers look generous and shows players there is a positive incentive in taking part in the event system so the two key changes we achieved in a logic flows we boosted discovery by tearing up a sign up you at launch whenever new event is available and we added empathy by giving players a minimum participation reward the next step was of course to do some white boarding sessions some brainstorming along with your game designers producers and all

the other parties involved we created some rough wireframes once you were happy with those we created digital counterparts and after we've I traded enough on these we created a working prototype so let's see what did we change in the new flow in order to address all the issues that we saw during the heuristic diagnosis stage now when the player clicks on the event icon he's brought to this view you can see that each event is now its own bucket using a tabular system we rearrange the information hierarchy so it's easier to read from top to bottom and left to right and leaderboards is now a smart slide in slide out drawer what this does is this layout allows the player to gather all the risk and award information he needs within the very same view without switch of context and it reduces the number of drill downs from three for just one so the key changes we achieve and our new flow we rearrange information hierarchy so it's easier to digest we reduce the rounds from three to just one click leaderboard is now a smart slide instead of drawer

and reduce cognitive load because we got rid of all the drill downs altogether so we have a solution all which looks pretty good the next step as per UX methodology is to test it now you would ask why should we tested we have tested enough among ourselves right I would like to highlight the mindset with which the designers used to work earlier was about design for people and I say I am NOT the kind of designer who designs for people you might find that controversial right after all UX is all about people let me like expand on it a bit more when designers earlier used to say we are designing for people they are making all sort of assumptions yes this is what the players will like yes that is what the players would like or they might ask few people around them and that gave them the idea okay designed for people I know what they want but still they're working just on assumptions instead UX as a methodology

promotes designing with people and that is where focus group testing and usability testing come into play now normally you would go for a full-blown usability test but since we were pressed for time we did the next best thing and that was a focus group testing session didn't the company luckily in a studio like ours we had three different teams working on three different projects that meant we had a pool of players who did not know what's happening much in other teams project so we had a perfect pool of players who could give us unbiased reviews we tested a prototype with them we got some positive feedback and on basis of that this solution was being lighted the next step was of course to work with the UI designer on this project and the front-end engineers in order to make sure justice is done to our vision and to solving UX problems we may encounter while implementing it happy to say we managed to push this update out on time with this new fix and we were all very pleasantly surprised with what happened next

this new update not only managed to meet the launch KPI 45% but within a small window of time we saw the engagement rate go all the way from 20% to 80% which was a humongous 300% jump compared to the previous subject important thing here to remember is without redesigning or we hauling the entire event system or writing copious amount of new code just by applying the correct UX methodology in not only diagnosing the problem but also designing a solution which resonated with our player base we were able to have such a huge impact on our KPIs next there was a debrief session and we had some clear-cut takeaways do not create unnecessary choice biases do not over helm the player with too much information boost discovery if there is a critical piece of information you want players to see or interact with do not bury it do not make them work to discover it instead surface and bring it forth for

them show empathy and use reward screens to convert players this is interesting because after we added the minimum participation reward we saw that a huge chunk of players who were just early signing up but not making any progress saw a positive incentive in taking part and started interacting with the system and this partly led to the huge C hundred person jump you saw in the previous slide last but not least leverage data work very closely with other departments in the company because not only will they help you gather qualitative and quantitative data to diagnose a issue but when the time comes to make a push for the UX fix in front of the stakeholder group they'll be there backing you up helping you make a stronger case so with that ladies and gentlemen we've come to an end of a presentation I hope it was informative and the floor is now open for questions in case you have any

questions please come forward to the mic I am James Berg researcher with E a great presentation thank you you mentioned for this particular example you're under some pretty severe time constraints so I'm wondering what would you approach differently if you had all the time that you well maybe not all the time you wanted but a realistic amount of extra time added what would you have done differently to approach solving the UX problems but that's a very good question in fact yeah so there's a contrasting methodology that you have to apply if I had more time a reasonable amount of time let's say with UX built-in what we would do is ideally we would like to get players eyes on the prototype much earlier we would not say we have something we are okay with that we will do a proper thorough survey and afterwards once it's designed as I mentioned because we were pressed for time we did a focus group testing session so briefly I'll just expand on that focus group testing session done

with six to eight people within your company is not as informative as a usability test session in a usability session you can have actual real-world players you can invite them from outside and you can observe how they're playing the game what you get is a goldmine of information which is like much more reliable and much more solid to work with rather than you know coming up with something just for the sake of it both solutions are effective but I would always pray for having more time beforehand you'd mention that you're adding this after launch and that can be like really challenging to get something and then change an existing system did you have any kind of like resistance to how to adding things that were changing an existing system and did it was there anything you did to lubricate that process to make it easier for players to understand the change will you repeat that question it was there anything that

you had to do in designing the system to make it so that players who were already familiar with the old system were easily able to transition to the new system yeah so so again yeah you're right so what you normally do is you trigger something called a modular 4 to e right and if you notice this layout was not that far from the previous one it just that we cleaned it up we may much easier to switch between even types and the only bigger biggest addition was the smart slide in slide or drawer so for that we utilize something called a modular for 2e which is to go at first time the players come into the tutorial and since it was a much easier system to interact with it was a hit rate then all right where's that looks like we don't have any more questions thank you very much [Applause]

you