Moving the Needle: User Research at the End of the Production Cycle

Hello everyone. Hi. Hey. Well, welcome to my talk called moving the needle in user research towards being a production. Uh we're going to be covering today a case study in PVZ Garden Warfare. And just to get to know each other a little bit and do my part. Uh my name is Veronica Zamido. I'm a lead researcher at Electronic Arts. Which is one of the amazing aspect of working at EA is the great portfolio. So, I've been working on different franchises including Battlefield, NHL, FIFA, Madden, among other titles. So, keep it very exciting. My background is in psychology and HCI. Uh game studies is exactly where where I

fit. So, now that you know where I'm coming from, uh let's jump in right away to what it says. Bet that all of you are very familiar with this mantra, test early, test often. Which is fantastic, it's good, you should really do it. However, every project gets to the last stretch of production. You have that game, it's there, you are committed. It's looking good, but there are still some gaps in there. And those core game mechanics are good, but that final shape needs to be fleshed out a little bit more. And that's exactly what's going to happen. It's about moving that needle more and more and more. It's about asking, how we go from good to great.

There are specific challenges near the end of production and things to really really keep in mind is there is no time for radical changes. They're pretty much committed. So, the use of research that is going to happen at this stage is going to be primarily about fine-tuning that gameplay. For doing that, you need to deconstruct problem areas. And with that, I mean exactly to pass it like a fine-tooth comb through the game mechanics, through the different game elements. That deconstruction is is going to require sometime different methodologies and metrics are going to be your your best friend for this. And don't be afraid about changing focused. Right now, at the point number one says there's no time, you need to

actually prioritize and find those weak spots. So, a focus sometimes can pivot from one area to another one based on the player's feedback. To actually ex- exemplify a little bit more about this distinction between earlier stages is pre-pro towards leading production. The goals are going to be different. User research is going to be done throughout, but the needs are going to change. Early in the stages are going to be more about those core game mechanics and a user flow and usability, that sort of thing that you really care to wipe out of the board at that time, where later common things that you really want to do are related to progression, fine-tuning, and balancing. There are different techniques, so you'll be using different tools to

achieve this. One-on-one and think-aloud are great for earlier stages, where are later in the in the game development, you really want to go more questionnaires and telemetry, and the data is going to be different, so it has to be treated slightly different. What I want to really point out here is about the effective and behavioral data. With effective, I mean everything related to feelings, emotions, attitudes, things that are really hard to measure. And behavioral at this point of the development process, I do mean the in-game actions that players are doing. There is a plethora actually of different techniques. I just took those that are more relevant, that the ones that are the bread and butter that you really want to use, uh but there are more, and you will be cherry-picking at different stages depending on the study that you have at hand to use one

of this. I'm not going to go over all of it because that would be a talk by itself, but I really wanted to point out those in red, which are, as I mentioned, the bread and butter of things that you really want to use towards the end of production. Notice that those are also more on the quantitative side. Because at this stage, you want to be absolutely positive that you have a good sustain of number of people that are indicating that you're making that progress, and also quantifying this data is going to allow you to have those metrics to push that needle. The whole objective subjective is more towards if I ask you from one to five how fun was this game. And you give me a number of super fun it was five. And and that's the quantifying part but it's a tool subjective that you cannot

challenge that. So off we go with these techniques. Telemetry is not going to pop up right away at the end of production you should have been planning for that. The researchers and designers you have discussed earlier about which ones what telemetry data you really want to to track. So now that you have that framework of different tools that you really want to use and how you want to apply them let's go to Plants vs. Zombies Garden Warfare. And I want to play two very short clips. So Plants vs. Zombies Garden Warfare was bringing a known brand into the shooter genre. PvZ Garden Warfare took the traditional beloved tower defense game and turned it into a third-person shooter. The goal it was to

immerse that player into uh the world of Plants and Zombies this the fighting and the battle in Suburbia. And it was designed for fans of action shooter games. And also it was aspirational for kids even though that was not necessarily the primary audience accessibility was a key aspect of the design. So I'm going now to show you an app from the zombie side. This one is a clip from the game mode uh Gardens and Graveyards. So here the zombie trying to capture the garden transforming into the graveyard. And I'm going to follow up with this specific game mode as an example. So, it's pretty fun, very colorful. You have all the shooter mechanics but really stripped down to the very very essence. And boom.

So, PvZ was developed over 2 years. It started from February 2012 to February 2014. Key questions that we need to keep brushing by the ends were more related to character robustness, maps and faction balance, and progression. Progression from just unlocking your abilities to actually level up and having that long-term engagement. So, the part that I really want to focus on and share those examples are actually for the six last months of production. And I noticed that we were testing even until February 2014. So, even 2 weeks before it was launched, we were still testing and doing that fine-tuning because even though it wasn't necessarily in the it was already the produced, but it was possible to at least to even more changes and keep pushing that for the day one patch.

So, the user research goals that I want to focus on today that were key for this were assessing the map perception and usage, ensuring the balancing of factions, and invalidating the character's playability and experience. There were many many sessions done. This one shows you the the typical setup that I was used for those. From a design perspective of the study, it was a mixed method mixing this quantitative and qualitative data. The key tools that I told that now you know are instrumental like questionnaire was excellent for this part to collect all these effective and behavioral data. And there were many different sessions and for all those sessions were running between 24 and 48 people per study and sessions were ranging from two 4 hours to two full days.

Now here's an example was one of the maps Garden Ops Graveyard is a 24 players map. Bringing people in, making them play and collecting that data. So going specifically to the Gardens and Graveyards dimension is a is a mode with an expanding map. If you play Battlefield like a rush mode where you have those zombies will be in the attacker and plants will be defending. There will be a timer. If the timer if runs out and the zombies couldn't couldn't invade, zombies plants win. However, if the zombies transform the garden to the graveyard before the timer, another part of the map opens. This one is the map specifically for the coastal map that has five different sub maps. I mentioned to you that you want to define very clear metrics about what you want to do. And and these one were the

metrics that we found the most useful for measuring your maps and really what you want to change about them. So it were there in the the map size the amount of cover can you cover from the incoming damage, population how crowded or empty does it feel, the visual appeal as well as very important, responding if people were perceiving that they could go back to the action quickly enough or not and overall navigation, PVZ doesn't have a mini map, doesn't have a map. It's just off you go. So, that was also really important to tackle. To give you an example, there were um many questions where you saying this scale type of questions one to five. In this case with example about the the map size from way too a small to way too big and a clear definition about it was the

right spot that sweetness that you wanted to hear with the participants. It was some of the the most were coming really good in the maps. You see that everything is between 70 and 80% of people were actually uh thinking that it was very very good. There were some minor tweakings that needed to be done. But it was coming along. We did that the same exercise for all those different metrics. So, we knew exactly which one we are and that was exactly about the time to prioritize which one you're going to be touching because there's no time. You need to prioritize. Overall the important. So, from all those we did some minor tweakings for the amount of cover from the zombie mountain and also the responding locations in the castle map. But you really want to triangulate multiple data. And here we have uh

another angle from where to address if if the map actually working properly from the player's perspective. So, they were given all the different maps that they played and they had to rank it from most preferred to least preferred and from the most fun to the least fun. They actually those rankings were perfectly overlapping to each other, which is uh good. Sometimes they have people have different reasons other than fun to prefer something. And I'm actually Marina was coming out really good. All the comments were primarily positive. And we need to make some changes. We cannot cover all or sometimes not everything needs to be tweaked. And clearly the problem was with the with the pirate ships. Actually a third of participants ranked that one as the least fun and the least preferred.

As you notice on the comments, the themes that were emerging is that they were dying quite easily. Complementing data from I mentioned you want to tag for different angles and really really understand. Now we know from the quantitative side that where the problem is effective. Now we need to know exactly why. So here you have this aerial screenshot of the pirate ship map. The design intention of this map was to recreate a pirate battle. It was about creating this tension of boarding each other factions. The zombies started from the bottom of the screen and the plants have the garden on the top. So both factions will be reaching the the two ships and then the and board each other. It's like it sounded awesome at the time. It's like, yes. Uh however, it was a map that was highly disliked. We are shown that this screenshot to the

players. There were many other screenshots predefined for having a really good view of of the map and also for all the maps, but particularly for this one, which was the problematic. This is a heat map, but this is a heat map generated by the users clicking on the image saying which one were their least preferred locations. As you see, it was primarily related to those unintended deaths. The shoreline was not clear that you couldn't really walk into the water, but if you look at it, it looks inviting. Like maybe there's another angle to go around, but really it was pretty much instant death if you step into the water. Which a hit map which has death you have been different because there were many, many other deaths of people killing each other in other part of the maps. So, this one was for really good gathering of visually understanding

what part of the maps were not really working. And getting all their qualitative of the question comments about why it was not really working. So, this And in the middle as well, this boarding each other idea didn't work at all. It became more like walking off the plank kind of idea, which is very sudden. So, after each session we'll get together with the design team and analyze the data and what they did. We're immediately jumping on and doing some changes. So, introducing new elements on the map to clearly indicate that not this is not a path, this is decoration. You need to go to the other way. Uh the distance on the ships also tweak a little bit more so you're less likely that you will accidentally fall. Uh the character's engineer is love for everyone so that why I choose that screenshot. And and all those changes actually were really good. And uh we always also look at the critics when it comes out to see

what we missed, what we didn't, and what was actually picked up by them. So, that was actually uh working really well. Now, changing to the balancing of factions, having two roles of attacking and defenders is not necessarily the easiest way of measuring fairness of these two distinctions. A team deathmatch mode will be cleaner to see if they are these two factions are perfectly balanced, but the roles actually they do tend a lot because we knew that the roles were the characters had the same abilities on on both modes, but in this one was being more problematic and the perception was being different. So, it didn't have any if the telemetry that the outputs of damage were the same, people were seeing differently. And actually that was one of the common topics that were coming why that mode was what was the detrimental part of fun and it was that perceived unfairness even though it was

fair for the characters. So, also retouching it slightly some of the health and outputs and weapon for the specific mode to actually translate that. And actually that one picked up as well. So, we're like, "Good. Happy about that." The the the last part of the they wanted to share with you today was about the characters. And either eight eight core characters for perfection. And we went over a similar exercise of deconstructing these characters. And for that it was more about fun, strength, health, and abilities. There were also a lot of question about if you were a zombie, how were you perceiving plants? And if you were a plant, how were you perceiving zombies? Which was very insightful as well. Doing a similar exercise with uh ranking the characters, we can find that the Chomper, which was very unique, is a very distinctive character for for a shooter game where you can have

burrow and then he jumps from underneath and then he has a one-hit kill. So, it was pretty novel and surprising. I was ranking very well about contributing towards this fun. and then the the foot soldier and pea shooter more of the archetypical shooter characters was ranking very well, like excellent. But, so our problem is the sunflower, and sure the sunflower is a healer. If you look at the distribution of classes and preferences, healers this not the majority, of course. But, we also look into have different people, those who identify themselves as healers and those who actually are not healers. Um from the two different demographics, the sunflower was coming coming low. So, again, pivoting, let's just focus on the sunflower, and this one is the one that we need to focus on on help. There were

a lot of little tweaks primarily on on the back end of the sunflower. And here you have all the different abilities that he has. And notice that allowing us to get that metrics were very informal to actually see where actually increasing and moving that needle with this character. And and it was, it was coming well rounded and better. Similar exercise with all the other characters, but this one was the the one that needed the more urgency. After that exercise, you were like, "We nailed it. Characters, boom, done." Uh but actually not, because we got the basic all eight core characters. But, each character has actually there are six of each type. So, there are like six pea shooters, six foot soldier, six sunflowers, blah blah blah blah blah, and that's a lot, and the total sum is

48. When we were ranking all the characters, everyone was playing all the classes. That's why people were able to rank them. But imagine that a session where you have enough time to play with all 48 characters will be pretty much close to impossible. So, for that adjusting the methodology to make it fit. So, instead of having this ranking, and even the ranking, you really don't know how much of a gap there is with preferred top one to the last one. It's like double this middle this close to close. There was no difference between first rank and second rank. So, it constant sum type of questions allows you to have multiple people inter-subject for those you doing in the research side that you can compare them better. And there was a section for each type of character, and then we could also look at all the 48 characters at once, and

actually identify which one from the foot soldier family, which one we should consider making changes. And then from the whole 48 characters, which one should we focus to make those changes. In this example is for the for the foot for the foot soldier, actually the Arctic Trooper was was the most the one who did more damage. There was a lot of variability, but it was still kind of clear the reasoning why. And as a Arctic soldier, it throws ice, and you're supposed to freeze, and that wasn't very clear that the visual cues were not indicating that people were freezing their enemies. The the balance between freezing and how much output damage you can do, it was also not well balanced and being felt as as fair. Again, picking up it was pretty good that at the end, once all the players get it, it was this diversity that you could control it

was pretty So, we were happy about that. And now that I mentioned a lot of different specific metrics in this construction from the different game elements, we also have some metrics that are scoring overall. And those more game element metrics are awesome for the game designer, but also for management, producer, executive producer. They like to have these overarching metrics as well. So, I had to do a combination of both. And this one we're more looking at motivation, engagement, and enjoyment of the game. And also we can see how all those changes, how they were affecting the overall enjoyment. And comparing from September to December sessions, and it was constant across the multiple game modes and for how many hours people have been playing the game. So, that was a really good indication I I would really moving the needle. Yes, no. And here we have proof of how the impact of those changes

were happening. Now, the the second moving the needle part is uh we also track in a single metric about how well this game is performing. And this one to 10 actually resembles a lot to the Metacritic. And regardless if Metacritic is a fair assessment of a game or how successful it is, it's a really good parameter. And earlier on it was in a 6.8 average, so it would be a 68. Later on it was getting more towards the end of 7.3. And actually the Metacritic score ended up being 76. So, this is like kind of all those little pushes, all that fine-tuning that you want to get to something. And we were pretty excited that we were showing that improvement of how it would be translated to the public. Now, you ask me 76. Like are you happy with that?

Well, no. It's 76. You really want to at least break in the 80s. So, on the one hand, no, not really happy with that. On the other hand, we also did that discussion about what should if it was everything was going so well, what was missing? And what makes me happy at least because of why that 76 is that the core design intentions were there. All the critics picked up on those. And actually call it about PVZ being in the shooter world, refreshing the genre, making it more fun, more accessible, classic humor, and it was there. But things that backfired were about gradual content and game as a service. When it was launched, it wasn't all not all the game modes were there. Not all the DLC and maps that were actually coming for free down the road

uh were not really perceived by anyone. Not for the players because that was still being discussed. And not from the people who on day one saw the game and and played it. And there might have been a different strategy that has should have been in place to surface that within the game. So, actually that the part that backfired were more like uh other components that do change the perception, but actually those core game mechanics that were really fleshed out were really well. Uh fair enough. I'll take the 76. And even those have got lowered to 70, it was still within what the game was meant to be. So, wrapping all this up, main takeaways about the last iteration of production was that no radical changes. You are committed. You know that you have something good. So, make it better now. Make your priorities,

fine-tune the hell out of it. Deconstruct everything. There is no time to say if was meant something or the other. You really want to design that methodology. I share with you different examples of how those different type of questions can help you to understand from different angles people's answers and combine them. The repeated measures, the whole longitudinal study as well. And you really want larger samples. I even with 24, with I even not even going big data, but with 24 you still you still feel more confident and because you are repeating that through time, you're also reinforcing and know where are you going. And and focus on the weak spots. Uh don't fear pivoting from one one aspect to the other. As I show you in the maps, there were some problems on those core metrics, but actually the the big big problem came from another question and we focused primarily on

those. Combining those emotional responses and what people were having. And and that is it. So, thank you so much. And if you have a question, there are two three mics over there. I'm happy to answer any question that you may have. Thanks.