Horizon Zero Dawn: A QA Open World Case Study

My name is Anna. I'm the QA manager at Guerrilla Games. In games QA for about 12 years. And I'm here to tell you the story of how we tested Horizon Zero Dawn, didn't let too many bugs slip through, and managed to keep most of our sanity in the process. So, this is the summary of the topics I will be covering in the talk. Uh but first, I would like a show of hands. Could you please raise your hand if you are in QA, QA engineering, or you consider a career in QA or QA engineering? Yes, my people. Could you please raise your hand if you are in production? Yes, we love product production. Um and could you please raise your hand if you have played Horizon Zero Dawn?

So, just as a brief intro, who is Guerrilla Games? Uh there are only two relevant points um for this talk. One that we are first-party Sony studio. This will become relevant later in the talk, I'll promise. And that for yeah, the last 10 years before we um went into production with Horizon Zero Dawn, the Killzone series was pretty much our studio identity. Uh I don't know if any of you have played any Killzone games, but basically, they're series of first-person shooters set in this dystopian dark sci-fi future um where the player fights uh these iconic enemies, the Helghast with the red eyes, um in these like sort of like constrained and kind of linear spaces. So, we went from this to well, the open spaces and lush nature of Horizon Zero Dawn

with the robot dinosaurs. We kept the red eyes. So, as you can imagine, such a major shift in direction um meant quite a few challenges. So, I'm going to go over some of the challenges for the project itself uh which were very relevant for the challenge of testing it. So, this was our first open-world game. It was our first open um action RPG game. Um the RPG elements, open-world setting, non-linear component um the this introduced a lot of variables that had to be accounted for during both development and testing. And these components had a lot of interdependencies. We did not have tools to create an open-world game. So, we had to pretty much build them from scratch. Speaking of tools, the engine that we had used previously to build the the Killzone games did not support an open-world game. So, we had to completely refactor that and

retool it. And there were many things that we didn't know about the project when we started working on it. We didn't know exactly what the scope was or exactly what the gameplay loop was going to be um or even the things that we did know um that we wanted to the game, we didn't have implementation details about them. And yeah, we had never done anything like this before. Scary. So, specifically the challenges for QA. The scariest thing about the scope for QA was that all the game components interacted with each other and influenced each other. They're all tied together. So, we couldn't partition things any longer in like neat categories and test them in isolation. We have to like consider the whole thing. So, just to give you a very brief example of just how the these components interacted in unforeseen ways. Um

early in development, we were working on a vertical slice. And in this vertical slice, there was a main quest in which the player was supposed to capture um a camp. And yeah, kill the bandits, raise the flag, quest completed, yay. So, while we were testing this uh for a milestone, we're going to play test this with um actual users. At some in some runs, the main quest just wouldn't complete, and we couldn't figure out why. And we've tried everything we could. We had quest designers debugging. They were like, "The script is fine. Everything is fine. I don't understand what's going on." And then it took us about 2 days of like deep diving and investigating, and we figured out that there was a random encounter that had nothing to do with the quest, in which there was a wounded hunter, and you could just pass by and give them some herbs to heal them or not. And if you gave him the herbs, so if you were a nice person and wanted to help

him, um you couldn't complete the main quest anymore for some strange reason. There was no reason for these things to be connected, but they were. So, that was very scary for us. Also, our test team was quite small. Uh throughout the project, we had about eight testers internally. Um and we had help. Uh so, yeah. Um but still, it was a challenge. And because we couldn't apply the same processes that we had for Killzone um projects, we had to come up with something new that we hadn't tried before, and we didn't know if it it was going to work or not. So, it was quite difficult for us to anticipate and plan um what our test strategy was going to be. And additionally, nobody in the internal QA team uh had any experience testing open world games,

and our partners at Sony first-party QA didn't have either. So, that was going to be fun. So, what did we do? I'm just going to summarize here the components um and I'm going to go into detail uh for each of them later. But basically, we came up with a new team structure for our internal QA. Um we came up with a test strategy that was focused on what we thought was important and relevant. We used a lot of tools to help our efficiency in testing and to make our testing more informative for the developers. And we used a bunch of data to help us, you know, reorient our uh test strategy when needed. So, what it all comes down to is people, processes, tools. But let's talk about people for a bit. Our QA team,

when I started at Guerrilla, uh the QA team was three people in a windowless room uh whose job description was basically play the game, find the bugs. And some developers, when they heard the word QA, they would just roll their eyes. Uh we we've come a long way since then. Um so, this day, the QA team is an integral part of the development process, and developers now bring us fancy chocolates to say thank you for your work. They were delicious chocolates, by the way. So, what did we do with our QA team? We involved them really early. So, when pretty much the entire studio was still developing Killzone Shadow Fall, our previous project, um there was a small team that was prototyping Horizon Zero Dawn. It was seven or eight people. Uh and one of those people was a QA person who was assigned to the project and who was helping the prototyping team to test,

um with feedback, with research, and so on. We specialized our QA. Uh so, they gained deep knowledge of the areas that are assigned to them, and they used that to decide the test strategy for those areas. And that knowledge was immensely valuable, and it just improved our test strategy just tremendously. We were able to test more efficiently, and we were able to make our testing relevant because our specialized QA knew what was going on, knew what the weak points of an area were, what was being changed, what the developers were working on. Speaking of developers, our QA were embedded with the development teams. So, early on they started creating connections with those developers, with the producers, creating information loops, feedback loops. They were in daily stand-ups. Um they're continuously talking with the

developers and getting information from them, feedback, and guidance on how to test and what to test and when to test. Yep. Wrong button. Uh also, our QA were mostly autonomous. Uh so, they were the ones who decided the test strategy. Um they were getting their work either from their team, from their development team, based on daily stand-ups or just communication, uh their own experience or intuition. Um and that freed me up to focus instead of micromanaging people, uh, to focus on solving high-level puzzles that were related to our overall test strategy, how to improve quality in the game, and how to find the best cat picture to attach to a work email. It's It's a pretty difficult thing.

And with autonomy also comes ownership. And our QA had full ownership of the areas that they were assigned to, which means that they were fully invested in the project. And this is pretty important because QA professionals, we tend to see the worst in a project. We tend We tend to see all the words and all the bad stuff and what whatever is broken. Um, so that tends to give us a pretty negative outlook. So, when you feel that you have a positive impact on the game and that your contribution matters, um, that helps me to get that. People feel empowered. So, these are just, um, some of the ownership areas that we had throughout development, and every area had one, um, or two, uh, for quest because it's such a complex and large area, uh, QA people were assigned to it.

And QA has value beyond finding bugs. We used these people's expertise, uh, as gamers, their experience with the project, and their unique position, which is kind of halfway between players and developers. We used these to make the game better. Uh, as a brief example, at some point quite early in development, uh, we were working on the bow combat, like bow shooting, and there was something off about it. Just something we couldn't put our finger on it. The developers couldn't finger put their finger on it. It was just not what we wanted it to be. So, a few people in my team went into a deep dive mode and they spent several days analyzing every factor that contributed to the user experience. So, animation, sound, um, like the trails of the arrows, the

timing, the timing of the animations, just everything that was involved in this experience. And they came up with a pretty detailed analysis of how all these factors were impacting the user. Um, and the designers used this to tweak those factors and create a better experience. But, there were only eight of us and we couldn't test the entire game without it. Uh, we just Yeah, the eight of us. So, we brought in Sony's first-party QA. Um, we involved them early on, 13 months earlier than on the previous project. So, if you work in outsourced QA, don't despair. We developers do learn and this is a possible thing to involve you early. And from the beginning, we wanted to build a partnership with them rather than a client-service provider

um, relationship. So, we were very open and honest in our communication and we expected the same from them, which was a bit difficult at first because they're based in Liverpool in the UK and Guerrilla is based in Amsterdam in the Netherlands. And well, Dutch communication culture is blunt. British communication culture is the opposite of that. So, we had some challenges there, but we turned them blunt. We were also quite liberal with sharing information about the project, which made some producers nervous. Uh but I felt that this was a risk worth taking. So, it paid off. So, this was our team structure in Killzone: Shadow Fall. We have the internal QA. This doesn't work ever. So, we have the internal QA on the top

and the um first-party QA team on the bottom. Separate teams, single point of communication. Um So, there was like a lot of delay between like tasks being passed on, requests for information, that information coming back, and there was a lot of pressure on those single points of contact. So, if one of us got sick, then yeah, that was a problem. So, yeah, there were some potential bottlenecks there. And most of my time and my counterparts' time at um first-party QA was spent managing workloads and databases and tasks and bugs. So, that was not very fun. So, we went from that to what we call buddies. That's our technical term for it. Um we brought the teams together. We pretty much created one QA team. Uh we tried as much as possible to remove the barrier of location.

Um and we created direct lines of communications between the testers on each team, so that we didn't have to waste our time passing information back and forth, and communication could be more efficient. Yay. And these testers also shared their expertise with one another, especially from the internal QA team, who had all the information and all the knowledge about their assigned areas and who are talking to developers on a daily basis. They're talking to producers. They passed on the expectations of those developers and their own knowledge and expertise. They passed them on to our partners at first-party QA. And yeah, we removed the single points of failure. So, then the uh managers and and leads could focus on more important things rather than micromanaging. We also shared training. Um so yeah, we became one team. Go.

And uh yeah, removed the barrier of location. This means a few studio visits. Um so this is us, the two teams, just playing board games and going out for dinner. And in the lower left, that is the QA leadership in both teams doing um three escape rooms to in a day and expensing it as team building. We won all the escape rooms. So, we built a real partnership with first-party QA and we collaborated on pretty much everything including test strategy. So, when we started defining our test strategy, we didn't know exactly what it was going to be, but we did have some goals for it. It had to be adaptable because we didn't know what the game exactly was going to be, so we had to be able to adapt at the drop of the hat.

And there were quite a few hats being dropped. So, we were able we we needed to be able to affect major uh revisions of our strategy throughout the project as requirements were changing or new things were coming in. We had to be scalable because we had unexpected developments in in the project as you might expect. Uh for example, about um halfway through development we started doing a lot of playtesting. We hadn't planned for this. We didn't know that it was going to happen. So, this needed a lot of support from QA. So, we needed to be able to scale up and down uh depending on how the need for QA support was changing. So, we basically did this um by having um by constantly improving our training programs so that we could quickly bring on new people on the project without having to train them for 2 weeks first.

And and it had to be relevant to the new ways in which we were developing the game and the new workflows and pipelines that were involved. Um so, we we made it the testers' responsibility. Uh the strategy was guided by their knowledge and expertise. Uh it was risk-based. Uh and it was focused on the things that we felt were important, things that were changing all the time, or systems that we felt that if they failed, it would be a really bad experience for the user, such as the save load system. And because we had such a small team, we had to make it efficient. The relevant part is a component of that because we didn't test everything all the time. We didn't do massive regression tests. Um but also tools. Um we used a lot of tools to make our testing more efficient, more informative, and just faster.

And because we didn't have a lot of regression tests and we couldn't afford to have a lot of test cases and massive test plans and tons and tons of checklists. Um we focused on exploratory testing. So, 70% of all our testing across Horizon Zero Dawn and its expansion to Frozen Wilds was exploratory. And I would like to go into a bit of detail um with the adaptable part by way of an example. So, failed experiment number one. In the early days of defining our test strategy, we researched uh player types and personas. The premise was that we apply these personas to our test and we'll find magical new bugs that otherwise we wouldn't find. Uh so, we took the Bartle test and a few other models of player psychology and we

built nine personas out of them. I think everybody's familiar with the Bartle test. Yes? Um So, one of these personas was the hoarder. I think you all know the type, the player who picks up everything, never clears up their inventory. Um So, we thought, well, if this kind of player would have to pick up a quest item, then there would be issues. Uh if this player wouldn't be crafting something in their inventory when it was full, there would be issues. I think you're starting to see the problem. We already knew where the issues were likely to be. So, rather than spend four days or five days with a player assuming um with a tester assuming this persona and playing through the game like this, we could just make a task for stress test the inventory system that would be completed in four hours, which was a lot more efficient. And speaking of efficiency,

I love efficiency. It's one of my favorite things. So, I was mentioning before that we couldn't use a lot of I mean well we could but I thought that would be a better way than to use thousands of test cases, massive test plans, lots of checklists. So, instead we settled on a specific method of testing which is called session-based testing. Um and it's it's an explorer it's a subset of exploratory testing. What is this magical thing? Um we borrowed it from software testing. And we picked it because well, we couldn't test everything. We would get daily builds sometimes two or three builds a day and to test to 100% test everything on one build, it would take a tester about 2 weeks. So, that was not feasible. We had to focus our testing.

Uh it's a method that avoids tunnel vision. When you're going by checklist and you're just ticking off boxes, uh you're not really thinking about what you're doing. So, instead using this method, we forced testers to be more engaged with with the game when they were testing. It leverages the specialist testers that I mentioned at the beginning. Because they know their area so well, they know what is likely to fail, they know how to make it fail. Um and they also have constant information from the developers that tells them, "Oh, I changed this. I added this. Go and take a look." And in our experience, beep this is the most uh the the method most likely to find bugs. It's also very flexible. Doesn't need a lot of test documentation. So, this is just an example of one of our session-based uh tasks in Jira.

Um the part of top The part of the top is the charter. So, it is the question that we want answered through testing. In this case, we wanted to know if um the resistance to electric damage is correctly removed for the robots to listed in the charter. Uh, and underneath that are the results. This is a task that has been completed. Um, the tester aside um the results in there so that developers can can look at them. And at the bottom, there's a comment from the tester, which is a follow-up. Aside from a wrap-up and a summary of the results, which is pretty much I tested this with this methodology. I didn't find any bugs related to electrical resistance, but I noticed this, and I made a follow-up task to investigate it more. Because yeah, sometimes when you test, especially using this method,

you don't find bugs, but you just find more questions. And before I give you the context for these beautifully crafted pie charts, I'm going to say something that might sound shocking. When we started defining our uh test strategy, our goal was not to find all the bugs. Crazy, right? What good are we if we don't find all the bugs? We only wanted to find the important bugs. The crashes, the performance drops, the progression blockers, the bugs that frustrate the player, make them go on the internet, leave bad reviews for the game. So, then this is a direct comparison between the bugs that were logged against Killzone Shadow Fall, the previous project, and Horizon Zero Dawn, plus all their expansions and patches. And the orange and yellow slices are what we call A-class and B-class bugs,

which are the bugs that we wanted to find. The functionality bugs, the crashes, and so on. So, I think we managed to achieve uh our objective. Because in Horizon, we found a lot more of the important bugs, and we didn't focus so much on whether this piece of debris is floating 2 mm off the floor. But, QA is not just about bugs. We also do a did a whole bunch of other things to add quality and value to the project. We play tested the game, like in the very early days, um when it was too early for like formal play testing with actual users. The developers would go to QA. Hey, play this. Tell me what you think. Out of the those play tests, and also out of those feedback loops that we had established with the developers, came a

lot of qualitative feedback. And wasn't always taken on board or applied, but they were listening to us, and that's what's important. We did a lot of research, what we call comparative analysis or competitor research. Uh where we look at already released games and how they implement certain features or how they tackle different challenges, and think, "Okay, what can we learn from this?" Um are they doing it badly? Or are they doing it well? Um can we apply any of this to what we are developing? We also did uh QA reviews. This was a technique that was very useful for us. Uh where we got the entire QA team in one room, and for an hour, we would play a feature or a piece of content, and everybody would weigh in with their particular point of view. And this is where those experts, those specialized testers,

could really uh bring their contribution and just enrich the conversation. And it also brought awareness of what was happening with other areas of the game that you were not necessarily work working on, and just the state of the project as a whole. And after we released, um we put together what we called the secret team. So, this was a small strike team of testers, both internally and from first-party QA, who would just go on the internet, which is a very scary place after you release a game, and they would go on all the social media channels, on Reddit, on forums, on Twitch, on a whole bunch of websites, and they would monitor for any issues that were being reported by the players. Bugs, frustrations, things that they were confused about. And we worked really closely with the community team to communicate back to those players if

there was a workaround or to clear up their confusion, but we also investigated the issues that they were reporting. And we investigated almost 500 issues, and out of those, more than a quarter were fixed and released in patches. And that had an incredible positive impact, and we got a lot of positive feedback from the players for this support track. So, we talked about strategy and people. Um so, I would like now to dive into the tools and technology that we used uh to improve our testing and that our QA teams uh used during the project. This might become a little technical. Um So, to start with our in-engine tools. We have about 250 to 260 debug draw

modes and windows that are accessible in game. So, from a build of the game, you launch it, you can get Well, not you, but us. Um you can get access to to these debug views and debug windows. And they are very informative for both the testers uh and for the developers. So, what these debug views do, um they show either raw data or they show visualization of information that is not available to the end user. So, they exposed the inner workings of the game, which helped our testers better understand all these features and content. Um but, it was also very helpful for developers. When they got a bug with a video attached that had a number of debug views turned on that showed them exactly what was going on, they could go, "Oh, yeah, I see what's happening. I'm going to fix this." So, that was a huge time-saver for developers as well.

And with debug tools, um we were able to change what the game was doing, affect the game state, spawn a robot, god mode, and so on. Uh so, that sped up our testing and uh we were able to just finish our tasks faster. So, I'm just going to give a brief demo of how this works. Some examples of our debug views. So, this is a view of the nav mesh, uh which shows where entities, such as robots, um, might might walk, are allowed to walk. And this next view shows where they are likely to walk, based on their behavior, um, and any scripting that might have been added to that. And since we,

I mean, robots and machines were such a crucial part of the game, we had a lot of tools to help us test them. So, this is a visualization of their visual perception cone. Um, so, we would we could use this to test if the player was being correctly detected by entities. So, we had a lot of tools, uh, to test robots. Um, this next one is a visualization of the animation, um, frame. So, we could check for animation issues. We could trigger specific attacks, uh, which is useful when one robot can have up to 10, and then you don't want to wait until they naturally occur. Uh, you can just trigger them. And we also had tools that helped us direct the entities where we needed them

to be. Um, for example, the tester is just now ordering this robot to go to a specific spot. And we also had tools that were were very helpful when we're testing performance and, uh, visual. Like, we can change the time of day, um, and go with it back and forth, which, yeah, helps a lot. It's a great time saver. Speaking of time saver, this was a huge one for us. This is a collision viewer. So, you can tell at a glance where there is invisible collision um or where collision is missing or where there is mismatch between the collision hall and the visual model. So, if anybody has done collision testing, you know just how boring and time-consuming it is. So, this was a

great help to us. We also had a bunch of external tools. Um if anybody has been in the poster session that ran just before this talk, you might have seen the interactive bug map. If not, I'll go into a bit of detail um about it later. We also had the game editor uh Decima Ed um which allowed our testers to do some in-depth debugging of things like animation networks, scripts for quests, um and it was It saved our bacon quite a few times. We also collected a lot of data from the game, uh telemetry, game analytics, same uh same thing. Uh and this was a key part of localization QA. We have a very high number of dialogue lines that are spoken by NPCs and they are triggered contextually depending on

the weather, the location, uh which point in the main quest you are, what the main character is wearing. So, all these are triggers for which line the NPCs will will speak. And to have the localization testers trigger these in game, it would have been unfeasible. So, instead, we had them just play through the game naturally and we used data analytics, sorry, game analytics to track which of these lines were triggered um and to one to give us test coverage data how many of these lines have been triggered how many do we still have to go and also which lines never triggered. So then we were able to find a few bugs there because some of the some of the lines were not set up properly and they would never trigger. So I will show you a little bit now how

the interactive bug map works. This was a tool that was developed for us by the engineers at Sony first party QA. So this is the actual map as it appears in game. We have a Jira integration which allowed us to populate the map the map with bugs as they were located in the world. We could also preview any attachments like screenshots or videos or you could just click on the bug and go to Jira and see the details there. And on the left there is a pretty comprehensive filtering system that uses a lot of our Jira custom fields plus other parameters to really drill down into the bug distribution and that was very helpful for producers and also for artists who could just go into an area and see oh art bugs okay I'm just going to get them fixed them

and I'll be done. We can also get positioning data from testers and from players and filter it by PSN ID to track both test coverage and also the journey of our play testers to the world. So using this tool we were able to find um, some areas that were not being explored as much. So, yeah, the the journey of the of the players is here color coded. So, I mentioned that we were able to find this prob- some problems with this tool. One of these problems was that there were several areas in the game that were not visited by either players or testers. So, we wondered what was going on. We went and started digging. And we discovered that the detection range on the compass was too short. It

wasn't that there was no content in those areas. The players or and the testers were just not discovering it. So, they would enter the area, they would look on their compass, nothing there. So, they were like, "I'm not just going to spend my time here." Um, so, then we went in we fixed that problem and then we used again, uh, game uh, game data to validate that our our fix was working. And these are some examples of exploratory session based char- charters that we derived for that for that task. So, this tool led us to find issues that otherwise might have been missed. Now, I would like to talk a little bit about test automation. Um, again, this was covered in the posture sessions, but if you missed them, um, I will very briefly explain how it

works. So, the engineers at First Party QA developed framework um, for us. Um, we did a lot of exploration into automation testing because we're thinking, well, we have a small team, let's automate. But, because this was our first open world game, we had no pre-existing network uh framework for automation, and we figured that creating and maintaining scripts, especially complex scripts, if we want to automate the player moving through the main quest, one, that was extremely time-consuming, and it required resources that we didn't have. We didn't have people who could spend their entire time writing and maintaining these scripts. So, the solution we settled on um are AI-driven bots who just go through the world and perform simple actions.

So, yeah, the framework just gets information from the game uh about the world over the network and makes weighted decisions uh about which actions to carry out. If there are robots in the area, they will kill them. Um it will loot them. If there are no robots, it will just go for collectibles, pick them up. And we had 20 of these running 24/7, which greatly increased our coverage in the game world, and allowed us to find quite a lot of errors uh in either content setup or code um that we could fix. And also, these bots generated a lot a lot of game analytics data. And now, for the takeaway points. I'm hoping I'm really hoping that you learn something from it uh from this talk, and that you can

uh take home some some tips um and some lessons. So, let's start with the testing. The people. We found that trust was essential. And trust and communication and collaboration, these go hand in hand. It's a circle. You just cannot have one without the other. We started building trust between the QA team and the development team, the the production team um by communicating. So, yeah. Um communication, keep it personal whenever you can, face-to-face. Have your testers walk to the developers and talk to them. Get them know as people. If that's not possible, if your testing is overseas or distributed, Skype calls or conference calls or try to remove that barrier of location

and to put faces to names. Shorten and widen your communication lines. Have people talk to each other directly, chat rooms. Whatever works. Be transparent. And demand transparency um from the development team. Ask them, "What are you doing? Why are you doing this? How does this work?" Empower them. Give them Give them something to do that is theirs. It's their responsibility. They're accountable for it. They will feel empowered um and they will feel that they have ownership of it. Yeah, keep them learning. Keep them growing. Learn about the project. Learn about the game. Learn about how an animator is is doing their work. How a sound designer is doing their work. What does this mean for my work as

a tester? These points may seem super generic and obvious and everything, but remember that historically QA has been separate from the development team. And only recently that has started to change. And yeah, remember that QA can bring value beyond finding bugs. So, showcase that value to your developers and start getting them to come to you and ask for feedback, your opinion, "Can you help me with this?" In terms of test strategy, efficiency. Um this is part of both processes and tools. In terms of processes, focus on what's important. You don't have to test everything all

the time. Really, you don't. I know that some producers or other people like to do that because it feels safe. Um and yes, coverage, all the coverage. But especially with open world games, this is not possible. It's just it's just not feasible. So, instead, focus on what matters. What are your risks? What are areas that are being changed all the time? What are areas that have historically failed? Ask your developers, what are they worried about? What are your fragile systems? What are your weak weak spots? So, yeah, focus. Time box your testing. And use exploratory testing. It's very useful. It will leverage the the tester's experience, their knowledge about the project,

and it will just find bugs that you otherwise wouldn't find. And yeah, improve all the time. Look at what you're doing and ask yourself, "Why are we doing this? Why are we doing this this way? Is there a better way we could be doing? Is there a faster way? A cheaper way? What could our testers best spend their time on?" And speaking of testers, help them with tools. Support them. Tools can increase understanding about the project, can make things more informative for developers, and save time for them, which will make them love your testers. They can increase efficiency. And yes, data. Look at data, but look at it in context.

Track things, such as what kind of bugs you're finding. What are your testers doing in the world? How are they interacting with it? Is that useful? Not? If not, change it. And these these approaches are not perfect. Um I'm going to go through some potential downsides and how you can help mitigate them um regarding this specific approach. Expert testers means siloing. There's just no way around it. But you can mitigate it. One way we did it was with the QA reviews. So those reviews, getting the entire QA team in one room, getting them talking about what they're working on, and getting them to weigh in with their specific point of view, that really helped. It increased awareness of what was going on in other areas of the title, what other team members were working on.

And it it also increased collaboration because they were looking, "Oh, actually this feature that you just showed, it kind of touches on my area. So, maybe I can help you come up with some, you know, exploratory test cases for it." Tools are great and amazing, but if you don't use them correctly, they can cause more problems than they help. And we had this a few times where testers would become a bit too reliant on tools and they would use them as they were not meant to be used. Um and then we created bugs that were not really bugs, just incorrect use of the tools. So, to mitigate that, talk to the developers, uh get their perspective on how a tool is meant to be used. Um if that goal doesn't align with what you want to do, maybe they can change it or make a new tool that is better suited to your needs. And educate your testers, create documentation, training guides on

when to use the tools, how to use them, why. Um and just get them to understand what is the intention between behind using these tools. Recruitment is going to be a bit more difficult because finding the people who can become experts in an area, who are self-starters, who can work autonomously, who have the people skills to create those essential connections with the development team and the production team. Those people are difficult to find, and a lot of those skills are soft skills that are not necessarily apparent in an interview or on a resume. So, I don't know how to fix that. If you find a way, tell me. We're still working on it. And once you get the people, you have to train them and not just for the first 2 weeks

or the first month. You will need to keep them learning. Make sure that they have the most relevant information that they need to do their job. Um give them new areas to own. Um specialize them in new and interesting ways. Doesn't have to be areas of the game. It could be things like user research. There's a lot of overlap between that and QA or accessibility. Again, a lot of overlap. Anything that you can think of that, "Hey, this would really increase quality in our project." Your QA could help with that. So, empower them to do that. So, coming to the end of my talk. At the beginning, I mentioned that when we started, we had no idea of the scope of the project. Um looking in hindsight, now we have some numbers. So, it feels pretty good to look back on

what we've done and the results we have achieved uh despite the challenges. And if you take anything out of this talk, it's this. This is the summary of the entire thing. This is what we did. Open communication, collaboration, focus on efficiency, expertise, and just be flexible. And I think a testament of our success is just the very positive reception that we've had from players, from peers, from the press. I cannot count the number of times I have read or heard the phrase, "I platinum Horizon Zero Dawn and never noticed the bug." Mission accomplished.

So, we have some time for questions. I noticed you said you gave your yourselves a lot of autonomy, so I assume you could make separate decisions than the development team. And one of the things you mentioned that came up as a problem with that was like setting up certain play tests. Were there ever times that QA could deny certain um requests from development in that regard like say, "Hey, we want you to perf test this next week. Could you say, 'No, we have a focus that we're working on.'" Was that Was it that level of autonomy or was it separate than that? No, we um So, the question was um about uh play testing and if QA was able to say, "Oh, no, this is not ready for play testing." Am I understanding correctly? Yeah, and sort of could development guide you more than you wanted to. So, we understood um the importance of play testing. The The frequency of play testing was really crucial in iterating uh and improving the game. So, we supported that process as much as we

could. Um we had several metrics that we were tracking to figure out if something was ready for play testing or not like the number of critical bugs that were not solved um and just qualitative feedback from the specialist tester on that area who would say, "Well, we have all these progression blockers." Or "Actually, um we have all these problems that were not solved." But then instead of us saying, "No, this is not ready." We gave them the information of "You have all these problems. Fix them and then you'll be ready." So, then they knew exactly which parts to focus on, uh which bugs to fix, which bugs to prioritize, um and then they would make a new build. We would validate that, and then we were ready to go. I don't think we have ever had to postpone a play test because the quality of the build was not good enough. Thank you. Hi, I missed your poster session and now

I really care about it. Uh, where can I get more info? Or are you going to get it somewhere else? Um, well, the speaker for the poster session is sitting in the back, Gareth Tynan. So, you can talk to him. He has all the information and his team developed them. Uh, you described a system of AI bots roaming the world and just playing the game. Uh, besides the obvious crashes and performance problems, what other bugs did you find this way and how did you detect them? So, the bots, because they had, um, sorry, the question was, what kind of issues other than the crashes and the performance issues, uh, were uncovered by the AI driven bots? Um, they uncovered a lot of asserts and alerts which were caused by either issues in the code that were not severe enough to become crashes yet. Um, or content that was incorrectly set up and could cause problems later. So,

we uncovered a lot of those, um, issues and then our development team was able to look at them, analyze them, fix them in some cases where they were bad enough. Okay, thank you. Hello. Um, did you have a different test strategy between your internal test team and the Sony first-party QA team? And then the second question is, would you have done anything differently moving forward having a remote testing? Mm, good questions. So, did we have a different strategy between the internal QA team and the Sony first-party team? And would I do anything differently? Um, to answer your first question, for the most part, our strategies were the same. Um, we started with some divergences and some differences, but as the project progressed, we pretty much aligned those strategies to make

the the most of of our resources, to minimize like work duplication and people working on the same area just doing the same thing. That was a waste of time. Um, so with constant communication, it was a process. It took some time to get to the level where testers would just talk to each other and collaborate and figure out exactly what who needed to do what. Um, but yeah, towards the end of Horizon Zero Dawn and especially through development of The Frozen Wilds, we were pretty much aligned on everything. Okay. And would I do something different looking back? Um, I would get a slightly larger internal team when we are closer to milestones like alpha or beta because our workload around those times increased so much um, that we couldn't um, cover it with just the internal QA team.

And those were some tasks that had to be done by the QA team because they they required a lot of communication with development and and production. So then, yeah, I would get a slightly bigger, maybe like 10 people. I think that would be the magic number for the project that that we did. Cool. Thank you so much. Uh, this is a two-parter. Um, the first is you mentioned that your team was very specialized in terms of quest or world or enemies or different parts of the game. Uh, to what extent was your uh testers in um uh uh England also specialized. So, when we uh came up with the buddy system we uh the first-party QA uh testers we had a core of more senior testers who were very experienced and they were specialized in the same areas as our

internal QA team. So, for example, our quest internal testers had I think four or five counterparts at first-party QA. They had their own chat rooms and their own like communication channels and they were talking every day about tasks, like what needs what needs to be done, uh what's going on. Oh, this quest designer told me this, can you take a look? Uh oh oh, we have this review coming up. We need the build checked for this content. Uh and the second part is with people being so specialized, uh what sort of redundancy did you have in case, for example, the person on quests got sick or uh was on vacation? Um who was able to cover for them and how? So, in the beginning, we didn't have any redundancy and that was a problem. Um and then we started covering those holes just by having

uh specialized testers like double in other areas so that uh in an emergency, they could just take over, not completely replace that person, but you know, just tide things over. Uh and also when we had the first-party QA um buddies they were able to take on that load because they knew almost as much as an internal QA person knew about um about that area. Thank you very much. Hi. Uh thanks for your talk. Um I I still can't wrap my head around the fact that you shipped that massive of a game with eight dev dev QAs. That's Yeah, congratulations. I have a real question though. I guess one of the big challenges in is that the game changed a lot along along the way from what I understood from previous talks I attended like even 1 year before ship massive changes were made to it. How did you handle getting

the documentation to get test cases always up-to-date and stuff like that? Did you have a specific pipeline in place workflow? Oh, documentation, one of my favorite topics. And I think historically a very difficult task getting documentation out of developers. We had a lot of challenges there because our developers were very much focused on creating the content, getting the feature ready, shippable, everything, bug fixes, and not so much on documentation. So, the process that we came up with after a lot of trial and error and discussion was to make their job easier to to write documentation. So, we explained to to them depending case by case because obviously we need different documentation from a robot designer than we need from a quest designer.

So, we explained to them exactly what we needed in terms of like, "Okay, if I'm going to test your feature or your quest or your robot, these are the things that I need to know. These are the questions that I need answers to." So, it was very specific. So, instead of the developer feeling like, "Oh, now I have to write a 60-page document that details everything about the feature." It was just, Oh, I just have to answer these five questions or I have to fill in this two-page template that tells me exactly what QA needs to know. And that really helped. Creating the templates for them on the wiki on our internal wiki and telling them, "Okay, you just need to fill this in." That everything is is in place. You just need to fill in the information and you're done. So, that helped a lot. Also getting production on board. They understood the importance of having up-to-date documentation

and how that was helping us test the game. So, they were badgering their teams, "Remember to write documentation for QA." And we also made it part of milestones. So, part of our alpha criteria or beta criteria was documentation. So, yeah. Thank you. Hey. We are right on time. How big it was your external QA team? Oh, yeah. I forgot to mention that. At peak, they were about 60 of them. On average throughout the project, I would say about 20. Okay, thank you. All right. Thank you very much.