Playtesting 'God of War'

[Music] hey welcome to play testing god of war and uh to put it pretty simply play testing this game it wouldn't be the same without play testing play testing provided us and our game a huge advantage and hopefully you'll see that by the end of this presentation and so we hope going through this journey of play testing with you guys that we can either help inspire you or maybe even do something similar to us or even just learn about what it takes to play test a big game like god of war and so just to hone in the focus a little bit the play testing i'm talking about is like external play testing because there's a lot of different types of play testing that you can do you can have internal reviews you can do focus groups you can do market research but that's not what we're going to talk about the play testing i'm talking about is when you go in and you recruit people to come in and play your game uh when it's in development and then they give you feedback and so the people i'm with here today

is g trough and kevin kieger jeet's the gameplay director at santa monica studio and jeet was heavily involved and helped drive the play testing process from a creative leadership point of view kevin kiecker was a principal user researcher at sie and still is and he's right now focused on playstation exclusives uh his responsibilities where he planned our research strategy he guided our data collection and he helped us conduct analysis and reporting for all the god of war studies and me i'm my name is ed dirian i'm on the production team at santa monica studio and one of my primary responsibilities was to coordinate all of our external play tests and i worked as the bridge between our team and kevin's team as the user researchers and so when i'm finished with this presentation we'll have a little bit of time for q a in case you guys have any questions and note to everyone there is spoilers in this talk so i hope you weren't expecting no spoilers and i do spoil the last boss fight so getting that out of the way right now

and so before i get into the mechanics of what we were doing in our play testing process i'd like to give you some quick backstory and what kind of prompted us to rethink our play testing process to begin with and so it really all starts with this image this image represents god of war's new vision and it's not exactly what made it into the final game but it embodies the spirit of what creative director corey barlog wanted to achieve and the vision for this game was very substantially different than the other past god of wars it had a more intimate and grounded approach and it was centered on the relationship of kratos and his new son atreus and at the center of this vision brought this idea of character growth into the franchise and surrounding character growth you had the three game pillars three game pillars narrative combat and exploration and so it's an important point to make here that this game wasn't a remake or reboot it was a reimagining and it was a

continuation of kratos's story which means that at its core this was the same kratos that players have spent over a decade with and i'm pointing this out because whether or not people had played the previous games we knew that some people would come in with expectations once they were based on how these past games were and so the studio was faced with all these challenges with reimagining this game the first challenge being that it was just a brand new vision it had a new weapon it was in norse mythology it had a no-cut camera and it also had a boy i was afraid my voice would crack when i did that so i'm glad that worked out in addition since this was not a new ip we knew that there were going to be new and old player expectations ones from people who have played the previous games and ones who were new to the franchise and we also knew that this was just

going to be a difficult game to make because this was a really big big game with an extremely high quality bar this was the studio's first game on the ps4 which in and of itself required new technology and infrastructure and since this new vision was so vastly different than the previous games it meant that a lot of extra effort would have to be spent early on just designing and prototyping and so we all of these challenges basically made the early development for the game very difficult and all these things were kind of clashing in and colliding with each other all the time and as a result the early effort was spent on just kind of building these components up just to get it playable in the first place and so this journey from pre-production to alpha that we had was very long and difficult and there was this huge mountain of doubt and frustration boiling up on the team and it culminated with a couple pretty serious questions is this even god of war anymore and then more importantly is it any good

and so we carried this burden all the way to late 2016 which is when our alpha was and this is when we'd set out to play the game from start to finish for the first time but it was still alpha we knew that not every feature would be completely in but this would be where the first time that we as a team would see the whole game being played from start to finish and so what was the result well to put it frankly the game wasn't very fun and like many game alphas are we they're you know that's just kind of how it is but there was still a lot of friction and frustrating moments in our game it was very easy to tell that there were some things that were a lot further along than others and kind of an example at this time the cinematics and the key narrative aspects of the game hadn't been fully realized and so there was a lot of gaps and holes in the game that had yet to be filled and so this huge and complicated puzzle that we were trying to piece together just wasn't really fitting and we kind of needed to speed up because we were at alpha and there wasn't a lot of time left for us and so the state of the game at this

time was telling us that we needed to accelerate but not only accelerate doubt was starting to sink in about the game about the process and so we began to even question the mechanisms for how we were validating our work but we knew one thing what we were doing clearly wasn't enough so we had to rethink our approach so this is when we kind of took a look at our current play testing process because we were conducting these play tests and the studio had been doing play tests all the time all the way since the original god of war but we noticed that these play tests weren't being very effective there were some problems with them and so i'll dig into them real quick the first problem was that play tests weren't really telling us any new information sometimes it kind of felt like we were just looking for sand in a desert the feedback we had was pretty surface level but most of the time it explained to us in a different way what our problems already were people would say things like it's hard to control the camera or that they were

having a hard time to track the enemies and at the time we like we knew these were problems we weren't getting any feedback beyond that and our second problem was that we didn't have a global process because like i said the team was play testing but it was happening in isolation and nothing was really looking at the whole game together so we'd get feedback on levels we'd get feedback on combat but nothing was really looking at them together and as a result of this we were getting this conflicting feedback and this was very difficult to determine what the next steps forward would be well you know because i think this beast was different like this was a much different game than we were used to and it was a very different game a bigger game and this this new game had a new vision it had more complexity it had a deeper narrative and so play testing the game how we were doing it before just wasn't really working and so some just started quite questioning play tests as a whole

and this valley of doubt on the team grew even more why were we even doing it we just keep getting the same feedback it's a waste of time and so around this time our sister studio gorilla games had finished up wrapping a note a game of their own horizon zero dawn and they heard that we were looking at revamping our play testing process and they offered to give us some advice and so we jumped on a tall with a call with matthias and lambert to hear more about what what they had to say and they too said that they were having similar struggles with getting up game just as big as r stood up and they had similar experiences with the challenges associated with play testing such a game and they told us about working with kevin and how this partnership with the experience lab helped them create a play testing process that was very effective and so like we were experiencing they also had doubt and they questioned how much they should even rely or invest in play testing to begin with and ultimately what they learned and what they shared with us can be summed up in

this one sentence you have to go all in and if you fully commit it will be the difference and it will pay off and so this call with them inspired us we immediately contacted kevin's team and we got to work and at this time i would like to say that it would be a huge understatement if i didn't acknowledge how much of a direct impact kevin and his team had on the game but at the same time i know that not many studios don't have access to such a user or user research team and so i've tried my best to focus this presentation on the concepts which hopefully are universally relatable and so let's dig in and let's start out by outlining our play testing process and what we did so i'll start with some broad strokes of why we play tests what our testing goals were and then dive in deeper into the different types of play tests and how we analyzed our data and so i'll start by asking this question why are we even play testing in the first place well the first part is we're trying to validate how well we've realized our

vision and note i didn't say validate our vision the vision of the game was set we weren't looking for feedback to tell us that kratos should have a mustache instead of a beard we wanted to see how well we had realized our vision through our implementation of it and so we needed to make sure that we were getting actionable data that we could use because as i mentioned earlier we were struggling to get this information beyond what we already knew and finally we're doing all this in order to make the game more fun and maybe that's a bit obvious but i don't mean just going into a play test asking is the game fun for us it was about extracting and understanding this feedback in order to make meaningful steps towards making the game better and thinking about it this way was very important to us because it set the tone that these play tests weren't just to tell the dev team if they were right or if they're wrong these play tests were helping us get data and then it was up to the dev team to ultimately decide what to do with that and so we made sure that the dev team

was getting the data that they needed in order to help make them make the best decisions and so this whole process began with defining good play test goals and so we made sure that the goals were clear and measurable because before this new process we'd go in to test just asking is the ui intuitive and then just kind of stopping but we learned that if that asking the ui is intuitive is pretty a good starting point but we had to learn how to focus our feedback towards the crux of what we were actually looking for for example we'd ask more details about you know about the menu flow or how well people understood the tutorials and second we began to realize how much overall work that this would require and so we began to treat play testing as a game feature in its own right this meant that we had to support it with a strong production backbone because just like any other features it needs advocates and it needs people to fight for it and so we created a play test strike team this was led by creative director

corey barlog and our director of product development yumi yang because it was important to us that the leadership was right there in the middle of it weighing in and driving it forward and they were advocating for the process across the team the whole time because you will run into problems along the way and you'll need a mechanism for being able to adapt and react to these changes as they happen and so finally we needed to be able to find a way to measure success to see if these changes that we were making in the game were actually working and so we took it one play test at a time because just doing one test wasn't going to fix all of our problems and so we were using these play tests to make incremental steps forward and so we also closed out we made sure that we closed out every test with a debrief where the researchers would present us some top line summaries and some key takeaways and this gave us a good place to start targeting goals for the next test and so remember these things when you're defining your play test goals because it's important to start off play tests with a purpose

what are you trying to achieve and this was a good way of kind of helping us springboard that way of thinking and so our next hurdle was defining how often we should test to begin with and so this is the play test calendar that we originally proposed and i made this around when we started revamping our process around march 2017 and this shows the play test through november the blue tests are our usability tests and the tan tests are big gameplay tests which i'll go into those shortly and then like most calendars that producers make as soon as we started doing things it actually changed so it's not the exact calendar that we used but there's a few key takeaways from this that we think are pretty important to share the first is to reinforce to the team that there's always another play test it's important that the team is confident and then they feel like they can rely on play tests which means that consistency is key because if you keep canceling or moving your play tests it creates instability

and then the process just loses credibility and so we made sure up front that when we confirmed a play test we stood behind it and we did everything in our power to ensure that it actually happened and for us that meant that we had to be okay with taking risks and balancing new feature integrations with build stability because we were in development and that's just kind of part of it and so it was important for us to make sure that we the leadership was on the same page with this way of thinking and so this did lead to some heated back and forths about some last-minute check-ins and sometimes we broke very significant parts of the game there was a time where we broke atreus in one of these play tests and after you killed a draugr he would just yell draugr over and over again he'd be like trogger stronger seriously it was terrible but we had to push through this because in almost all cases it was better for us to play test the new things that might break stuff than to leave it out and then miss out on the feedback and so this mindset helped kick us into

high gear because every play test was really important and lastly it was important that we took incremental steps forward each test building upon the last this this meant that we used the different types of tests to our advantage so we'd look at big picture issues and we'd figure out where the pain points were and then we dive in deeper because for example you're not just going to nail the camera after one test we were fine-tuning it all the way to the end and so we had to make sure that we were constantly making that meaningful progress and so now let's take a look at the different types of tests we used and what types of feedback that they produced so we used primarily used two different types of tests we used unguided playthroughs and usability tests and both of these tests have very specific goals that they achieve and using them strategically helped us find and focus in on the core issues and i'll start by talking about unguided playthroughs

put simply unguided playthroughs try to find your pain points they look at the whole game and how well you've realized it and generally they'll give you an idea of where people are having the most trouble and where you need the most attention to spend and the goal of these tests is to have as little interaction with the play testers as possible much like how they would be playing the game at home this meant that we would only help them if they specifically asked us for it or if we knew that they were countering a bug or an issue and so these tests helped us identify and validate the holistic components of the game an example of these are like measuring the overall pacing or the narrative comprehension or the difficulty balancing just to name a few and looking at it this way gave us a nice mental image of where the top issues were in our game and that's where the usability test came in we would take these pain points that we were found in the unguided playthroughs

and then we'd dive in deeper and these usability tests were meant to isolate the root cause of where this friction or pain was coming from and it lets you investigate alternate solutions in a quick and negative way because in a full play test you'll get this noise that is created by other game systems which can lead to generalized feedback for example someone might say the combat isn't fun but then you use ability test and you find out that the real problem was in the camera sensitivity and so it was important for us that since it was such a big game that we had to have a proper mechanism to be able to filter out this noise and then hone in on our core issues and so these tests these usability tests were conducted in small one-on-one sessions where a user researcher would be paired up with a play tester and these play testers were encouraged to have an open dialogue with the user researcher we'd ask targeted questions about what they were thinking moment to moment and even though we were mainly looking for constructive feedback in these usability tests things that were wrong

with the game it was also helpful to hear feedback that was telling us that we were on the right track so here's a list of the features that we did some kind of usability testing on and as you can see many of these things are very core components of the game and that's very intentional and since we only had a limited amount of time left we needed to make sure that we were constantly moving these features in a right direction which leads me to this next question how do you extract meaningful data from this process which leads to positive results and for both types of types of these tests we learned that this depended on how we were asking for it and so we used two primary ways to our advantage feedback feedback that was volunteered to us or feedback that we explicitly asked for and so we referred to this volunteered feedback as undirected feedback we got our undirected feedback from either in-person interviews or we

conducted that we conducted or kevin or what kevin called the red button because at any point a playtester could essentially just pause the game and just type in a comment box about any epic frustrating or confusing moments that they were having and doing it this way this undirected feedback gave us the moment-to-moment feedback this was all unprompted and this often led to follow-up questions that we would have from the dev team to dive in deeper and the directed feedback that we got refers to things that we specifically asked for usually in the form of a survey and so we would always try to get this undirected feedback first because we didn't want to bias the players towards thinking that there were problems with these game systems that we were asking about in these questions and so here's some of our real data filtered by undirected and directed feedback and as you can imagine we got lots of data from these tests it was pretty daunting at times but kevin and his team proposed a

methodology of filtering this data using adding and subtracting so we'd ask these questions through the lens of whether something added meaning enhanced or subtracted meaning reduced from the game experience as a whole and we did this for both the directed and undirected feedback asking the question through this lens helped us not over complicate things and it gave us a nice sorting mechanism to be able to see the top issues and for us this kept the feedback manageable and we were able to move things into action very quickly and so here's a list of the things you just saw on the top undirected subtracts of one of our tests and this top example is that four people told us that the valkyries were too difficult yeah i know what a surprise and this this list this whole list came from the red button comments and the interviews that we were conducting and this was all unprompted and now looking at our directed feedback

you can see that people were struggling the most in this test with the enemy difficulty 45 percent of people were saying that and so overall looking at it this way and seeing enemy difficulty on top was prompting us to look deeper into the feedback we would see this and then so then we'd sometimes pull the play testers out and ask them further questions or we'd target you know more and more tests in the future to hone in on figuring out what that problem was and so overall organizing the sheets like this between directed and undirected adding and subtracting helped us build this consensus because they showed us the state of the entire game because if you're on the team and your feature was just one of those things on the list seeing everything kind of together in this way showed you the full perspective and this made it very clear where we needed to direct our focus and this was a huge win when we saw something go from heavily subtracting all the way up and then adding to the experience and that was awesome to see

and so as the needs of the game changed and evolved the data that we were wanting to know and learn also was changing with it and so we were constantly iterating and changing these data sheets as the game became more polished because it's important point to me that each department is going to be kind of looking for slightly different data level design might care more about seeing the breakdown by the level or the combat might want to see the breakdown by the player and so make sure that you spend a lot of time with your departments in order to understand what their needs are and what data that they need because it's hard to generalize some of this stuff i love this box water it's great that's a broad overview of what our process ended up looking like and to recap some of our takeaways make sure you have clear and measurable goals develop a good play testing cadence and stick to it

use the strengths of the different types of tests to your advantage and then make sure when you're getting the feedback that you're considering the whole team but a process is really only as good as the people who use it and part of our learning process was just doing it and so i'll give you some real examples that we went through which taught us some pretty valuable lessons and so starting this off with a little context when we're designing features our team like look likes looking at it in terms of fantasy and function the fantasy kind of refers to the intended player experience the function refers to the more mechanical things like the controls in the camera and so we when we apply this usage that we have on our team of fantasy and function we can see that it maps pretty well to the test that we were also conducting the unguided playthroughs were helping us validate this fantasy the usability tests were helping us validate the functional

and so i'll use a boat as an example i was able to dig up a pretty early brainstorming document of the boat describing the early vision and the boat was designed to allow the player to explore to collect to hear story and to experience combat in that order of priority and if you played this god of war which i hope all of you have it stayed pretty close except for the last one combat which didn't make it into the game so what happened the past god of wars had combat in everything literally everything even pegasus and god of war 2 which was a heavily scripted moment had combat i'm just going to let that finish because that's awesome well to put frankly we cut combat from the boat because it was overscoped the boat's core purpose was a way to explore and to hear more about the world and so when it came down to making this production decision the boat combat

being the last thing that that in order of priority that was going to be the first on the chopping block but that was a hard decision for the team to make god of war was combat and so after this decision was made there was absolutely doubt on the team because this kind of broke the classic formula but we continued on without boat combat and we got to alpha and the boat wasn't in a very good state the controls on the steering hadn't really had a proper tuning pass yet and the story and the narrative components just weren't even really in at all and the story in the narrative components referring to the the back and forth banter that kratos has with his son while on the boat and so we had ran some play tests recently and the general consensus was that people didn't want boat of war it was too slow it was too boring they didn't like it it was terrible and so the feedback that we were getting around alpha was constantly calling out the boat as a bad experience and so all of this negative feedback brought back up combat

see they hate it put back into combat and this was brought up many times in meetings even in the form of an ultimatum sometimes it would be like either you put the combat back on the boat or we'll just cut the boat all together well let's look at the feedback and summarizing it you can kind of end up with these two statements players were either saying there was too much boat referring to the fantasy or they said that it was too slow and difficult referring to the functional well like i said earlier the fantasy of the boat wasn't even in and so it makes sense that the playtesters were saying that the boat was boring and in this moment we kind of had to remind ourselves not to address this feedback that was talking about the boat's fantasy but we could address the feedback talking about the functional and so we asked ourselves what can we do right now to start improving this experience while we wait for this fantasy elements to come in well we have these usability tests where we can dive into this exact type of

thing and so we did and we made many great quality of life updates to the boat like you see here as a result of some of these usability tests and doing this helped us confirm and drive some of these things into the game and that was really cool to see and then finally once these narrative elements of the boat did come in we were able to get feedback on the big picture and then you were able to hear about kratos amazing storytelling so know any good stories to pass the time there was one that concerned a hair and a tortoise so what happens they wager on a race between them the hare is too confident of victory and foolish while the tortoise is steady and disciplined the tortoise wins you haven't told a lot of stories have you it's one of my favorite stories [Music] and also like kratos i have a pretty good story about the boat so let me paint you a picture real quick

we just come off a play test where we had an issue with the docks you just couldn't use the docks or the beaches so once you got on the boat we had to help the playtesters off the boat every single time they wanted to get off because we had to do this like debug flying it was terrible and so this whole test was a wash we couldn't get any feedback on the boat and so we started preparing for this next test because we were putting in these boat stories for the first time just like the one you saw and this was a big moment for the team because it'd be the first time that we'd be seeing people react to this intended experience that we had for the boat and so we're about two hours into this play test and the first player that got on the boat was rowing up next to a dock and then they just sat there they just sat next to the dock and there were people watching this back at the studio and they were like it was kind of heartbreaking it was like oh i guess it's broken again like you know what are we gonna do i guess we have to wait for the next play test

but then we realized when the next person came in the docks weren't broken they got off the boat just fine so what was this person doing and so we interviewed this person and they stopped the boat right next to the dock because they were waiting for kratos to finish his story and so for us that was kind of the moment that we realized that like this boat was fulfilling that intended experience and that people could enjoy this boat without having the combat and so what did we learn from this is that well we could have just added back combat at any time this feedback was sometimes even telling us to directly add in combat people would write in the surveys add boat combat but in these moments the play testers didn't really know what our final intended boat was supposed to look like because it wasn't fully implemented and so we we learned not to prematurely react to feedback especially since the design intent that we had for the boat wasn't fully realized

all right so for the next story i'll be talking about the final boss so we're about to go into a play test for the first time players would see the full ending of the game and we felt pretty good about it and to give this a little context kratos and atreus met freya pretty early in the game and a trace immediately became attached to her because atreus had recently just lost his mother so he began to see freya as a maternal figure throughout this game well in this scene freya's son balder is about to take her life yeah it's complicated but kratos steps in to save freya by having to kill balder so let's see what freya had to say about that no no no no my boy my dear sweet boy [Music]

freya he chose this i will bring down every agony every violation imaginable upon you i will parade your cold body from every corner of every realm and feed your soul to the vilest filth in hell that is my promise he saved your life he robbed me of everything [Music] everything [Music] yep after watching the scene playtesters were telling us directly in the feedback they didn't like how freya suddenly turned into a villain in the end they actually said that

and we're like what freya's not supposed to be a villain she had this impossible situation what was she supposed to do well we looked at the feedback and we noticed that there was this divide some playtesters were empathizing with freya and some didn't at all and they were even calling her a villain and so we kind of asked ourselves why would the play testers think that freya is a villain maybe the problem was in that moment they were identifying more with atreus not freya and so we thought why might that be and we thought that people identifying strongly with atreus might not be parents or they might not understand the impossible situation that freya was in and so we added a new part immediately after the cinematic i don't understand i know saving her was the right thing but she seemed all evil at the end not evil you killed her son lad her son

the death of a child is not something a parent gets over easily but he was gonna kill her she would have died to see him live only a parent can understand so you'd let me kill you if it meant you would live yes look there was no easy choice for anybody brother but i think we can all agree you did the right thing the world's a better place with freya in it just give her timelapse she'll come around and so in this new part we had to trace ask this question back to kratos as if he was one of the play testers who was calling for a villain this moment so you'd let me kill you to have kratos answer yes yes he would and that only a parent can truly understand what freyja was going through we noticed that after we added this bit at the game it worked we stopped seeing negative feedback talking about freya it was pretty pretty cool

it was great what do we learn from this well sometimes you might go into a play test and you think that you've got everything figured out and then suddenly you might get feedback that you don't expect at all it might even be the complete opposite of what you're intending it to be and you know everyone who plays games will have their own unique perspectives and as a result they could misinterpret something like this and so play testing showed us what it really meant to listen to hear people out because we could have just said oh well those players don't get it possibly missing out on a legitimate issue that maybe was isolating a big chunk of our players and it's an important point also here that gameplay testing isn't just about testing gameplay because we found that it also can be a pretty useful tool for helping measure narrative comprehension you don't know what this means yet

but i'll talk about our good friend the olive pizza guy and so what if somebody comes in and they play your game and they just absolutely hate it so much that they write about it and very painstakingly specific detail in all of the feedback that he gives well lucky for us this happened in one of our play tests and now before i read his comment i'd like to say that we're very open about sharing the data that we're getting from these play tests with the team and we'd send out the data every night after every test with a basic daily report kind of with the top findings and so now you saw it a second ago but let me read his comment to you and i quote the puzzle solving is a bore what's worse the amount of puzzle solving significantly outweighs the action and when the game is called god of war that's just not acceptable imagine you called pizza hut and you ordered a meat lover's pizza

and when your pizza arrived there was more olives on the pizza than meat that's how i feel about this section and just about in every other section of this game i went in expecting lots and lots of action lots of meat on my pizza but no i got lots of puzzle solving the pizza arrives stacked with olives who the hell wants more olives than meat on their meat lover's pizza nobody that's who now who wants kratos to spend the majority of his time as an imitation laura croft doing puzzles versus being a badass and killing everything that moves also nobody more action less stupid generic puzzles and oh that's another thing the puzzles are so boring and generic ring ring 1998's legend of zelda ocarina of time called [Laughter]

it once it's basic outdated 20 year old dungeon platforming puzzles back end quote and so naturally like i said the dev team read this comment and so halfway through the test the olive pizza guy was a celebrity on the team [Music] and it was very clear from his feedback that he was a fan of the pass god of war games because he kept referencing these games and all of his comments that he was making and so on one hand we kind of saw the thing that we feared there was a passionate god of war fan that just didn't like the new game but on the other hand the olive pizza guy's intensity clearly came from a place of love he was a self-proclaimed huge fan of kratos in the franchise

and when he wrote this comment he boldly and directly challenged our new vision for the game when you refer to the puzzles in the combat and so we learned a couple things from this the first being that not everyone would resonate with this new game we were making a reimagining of god of war one which had more puzzles and combat and that wasn't changing and not everyone would like it and that's okay and we also had plenty of evidence to support from our play tests that we were doing a pretty good job people loved the characters the setting and the narrative and we had made enough progress through iterating on these play tests that the problems weren't in the mechanics they were in the balance which leads me to the next learning is how to strategically look at the feedback because the olive pizza guy even told us that he hated puzzles in general so why would any of his feedback about our puzzles really help but he did like combat and so we looked at his combat feedback and in his own hilariously accurate but mildly anger-inducing way

the olive pizza guy had a lot of great points because he was trying to tell us how slow the combat pacing was during the beginning of the game and that we needed to re-tune the pace and that that our decision was that we needed to re-tune the pacing so that someone like him early could get enough of a combat challenge that they were looking for while still being able to engage with the puzzles even if he didn't really like them and so this was really valuable feedback to us because it helped call out a legitimate issue but i'm not done i received a text the day after we got this feedback and it said before he eats his normal lunch tomorrow you should buy this guy a pizza but put olives on it and so i did [Applause]

and we made sure it had a lot of olives and for the record just so you don't think we're complete jerks he was smiling under this picture i promise and we also bought him a meat lover's pizza also that he actually ate for lunch so and my final example is about our difficulty our game has many different difficulty settings you can even switch the difficulties around at any point so let's say you start by choosing the balanced experience or normal but then you thought the game wasn't enough of a challenge you could then bump it up to give me a challenge well the feature for switching this difficulties wasn't in yet so for a long time we were relying on the balanced experience to give us a baseline and so in one of these play tests we intentionally tuned everything way harder because we knew from the previous

test kind of what too easy looked like but we were looking for this sweet spot we needed to find out where the hard ceiling was and so we tuned it way up and we found out after that play test that that was too hard so we tuned it back well the next play test we're about to go into this this play test after the hard play test and we'd have the switching difficulty feature in and this made the combat designers very excited because this would be the first time that they'd see how people were utilizing this feature especially how they would react to the harder difficulties and so there was a player in this test that we were having and they were absolutely raving about the game everything was perfect the game was perfect the difficulty was perfect 10 out of 10. well our combat designers were very interested in this player because they were making all of these comments about how well and balanced they felt like the difficulty was and based on his feedback this player sounded like they were very

proficient in the game he had detailed analysis of fights he was talking about strategies in the arenas surely this player was a combat game expert well i've personally learned working on a game like god of war that combat designers treat the harder difficulties like the magnum opus like this is why they wake up in the morning for the hard difficulties and i don't think i've ever seen a more disappointed look in anybody's face when they realized after looking at the telemetry that this player was playing on easy give me a story so we learned a couple things from this the first is get critical features in front of play testers as soon as possible because our game was designed from the beginning to be able to switch difficulties through the middle of the game and this will this actually changed how people engaged with our game

and we saw a huge amount of engagement with people and people using it once we put it in and for a long time we were only testing balanced experience the the kind of middle normal difficulty and this made us it made it very difficult for the combat designers to action on any feedback we were getting that was saying it was too easy or too hard because we weren't really sure what the skill level actually was and the second takeaway from this is that we needed to look at feedback from all sides because we assumed from this player's feedback that they were playing on a harder difficulty because of how descriptive and analytical they were and so this reminded us to make sure that we cross checked all of our feedback with the videos the telemetry the difficulty settings and create this player profile to make sure that we painted the most accurate picture of these people that were playing as our game as possible and finally difficulty does not equal enjoyment and just because they were playing on easy doesn't mean that they still couldn't have a good time and enjoy the game

systems and still provide us valuable feedback on them and so if i can sum everything up and hopefully you kind of started to see the reoccurring pattern is that we weren't play testing to question our vision for the game we were using it to see how well we had realized it and this is an important mindset that we that we had to shift to because reading through this feedback can be frustrating but know that you can get valuable feedback that will help you make your game better through play testing you just have to be in the right mindset to receive this feedback and not everyone's gonna like the game and that's okay and so we told ourselves from the beginning that we'd go all in but it took time and it took practice to get into a rhythm and we saw this after every test that we got better and we got more efficient at it and we started seeing what data was valuable and what data wasn't and so now i'll close this out with some

final thoughts that hopefully you can take back to your teams to help improving your play test process now this is with the understanding that all teams are different and so while these work for us you'll have to be the one to determine if it will work for you the first thing that we think is team wide accessibility really helped we streamed the play test live to the entire dev4 which anyone could watch and this was valuable to us in a couple ways it was valuable to see the content being played from a dev perspective so you could reference back to the videos i didn't encourage people to ask a lot of questions which then they would relay to the researchers or anybody who was at the test at that for for the next day to look into more and the second was that it reinforced the idea that these play tests were for the team and that we encouraged people to make their own assessments on what was going on and to bring those ideas to their leads second it was reinforced through the leadership corey was highly involved in the play test process and he strongly believed in

it and you could almost always hear somebody on the dev forward talking about the feedback from the last play test or the upcoming play test and this all starts with the leadership being on board and all in on this process and third the relationship with qa is really critical because you will hit problems with builds right before a test and you might integrate something that breaks something down the line somewhere else and that's just a part of making games but what helped us a lot was that qa was in these triage meetings with us they were at the play tests and they were part of the process for preparing the builds because their perspective is very valuable and i can't stress enough the importance of that and fourth and last but not least reduce barriers for user research and for us just getting a user research team was a critical component to helping make the game better but it doesn't stop there the key was to ensure that we kept the researchers as up to date as possible whether that meant sending them updated build notes are there even several times

where during a play test we would call them up on the phone because the more context that they have about the game and the systems the better that they can help you and so if there's one thing that you take from this talk it's this just keep doing it because we had a very similar experience to what lambert and matthias described when we first jumped on a call with gorilla because there were times that it was very painful and we did have to move things around in the schedule a little bit to help accommodate but it also significantly impacted the game in a very tangible and positive way so that concludes the presentation part and we'll be sticking around for any questions we're also doing a lot of talks here at gdc in case you're wondering and we're hiring

all right question time hello oh sweet in addition to the external play tests with uh testers that you brought in did you have any internal play tests as well yeah so so prior to the external play tests we did do a lot of internal play tests within the team on a sort of uh grouped level so like combat for example would do their own internal play tests level design did some specific feature teams like the boat or like the axe we were doing intern some small internal play tests around that as well uh and then it's not that we completely stopped that but we made the sort of bigger leap and bigger transition into doing more you know longer term external play test given that the game got so big and it took literally sort of five days just to get through the whole game so you sort of used the internal play tests for rapid iteration and then the external was more for the fine-tuning of the features near the end i know i i'd

say probably that at a specific point like kind of how ed describes that alpha we sort of made the full transition to just doing external play tests thank you hi uh very good talk by the way i'm a bit curious about the um playtest strike team that you talked about that's led by uh corey barlog and yumi yang i was curious if you could go into uh why that was necessary what their roles were was it enforcing the consistency that you talked about was it kind of choosing what the topics were what topics were tested over like why did you think it was necessary to create them well i mean it kind of went into it a second it was it's important to have like you know production and the people kind of on top of everything like being that driving force because you know kind of our problem was that it was just kind of happening all over the place so it was it was kind of important to make sure that and even g you know being our gameplay director was it was important for him to be there as well because it was one of those like you got to have the big

players in there in order to help make sure that this these things happen right gotcha and i don't know can i happen on that for a second the other thing is it was a new process or they were undergoing a new process so having a strike team that could essentially fan out throughout the entire studio and connect the studio with the research group and make sure that everyone was heard was super important so having an organized and broad approach to that is really key and i think for us i guess like to ed's point i mean like i was one of the engineering leads on god of war and so for us on the strike team it was really important to have the other leadership as part of that process to decide because again like because it's such a big game everyone is running and and in a lot of ways everyone is kind of running with their piece and so you can have like everyone kind of going in a different direction to some degree and then play testing was kind of a way for us to unify and centralize like hey we all need to run in that direction right now and a big part of this i think is also just

making sure that like relationships are key in this process like even with user research but even internally in your own studio the leadership team has to have a very unified and strong sort of relationship with production with all the other devs and with user research because you really want to make sure again that like everyone has each other's back given that things are going to be really bumpy and everyone's going to want to look at like why that didn't work or you know what was the problem there so i think play the strike team helped us kind of get everybody on the same page making sure everyone knew these were the goals these were the areas and then we could leverage production to help us prioritize how we were going to get to the best possible build for the next play test thank you very very much hi thank you for being here it was a good talk so just when i'm play testing stuff if one person points out a problem i'm like okay but if a second person points out a problem that's when i'll consider it but um with you guys i'm assuming you guys probably get a lot more play testers so i was just

wondering how many people have to point out a problem before you consider it like an issue yeah it's not so much a matter of how many people to be honest it's a matter of kind of um understanding where that problem is coming from and how relevant it is and how important it is to the overall experience of the game so in our cases these the the larger group play tests were 20 people at a time okay um and uh as you can see from some of the data up there uh the fact that four people out of that 20 had an issue with the the valkyries was enough to be taking a look at that particular balance item but at the same time the olive pizza guy in a certain sense sort of single-handedly brought that issue to the fore it wasn't that we hadn't seen that before from other people but oftentimes you'll see a similar issue just spoken about in very different ways across a wide variety of people so it's a job of all of us to try to put those things together okay and one last thing in that data

there was um one point about kratos kicking the chain down and he goes down automatically and that was a problem and i've seen online that bothered a lot of other people so did you guys not consider that because it was one person or just not change it oh i'll take that one it was it was brought up uh and again it like to to kevin's point it's about looking at that sort of pain and then kind of prioritizing it across all of the other pain that we're also receiving and then seeing like okay is this going to significantly subtract the experience uh compared to everything else that we have to prioritize and get to and what does that impact have so i guess it is great to kind of assess that feedback make it very clear that we understand it and then then you know do the estimation of like what will it take to actually address it okay thank you great talk thank you about the freyja scene i might come across as a super villain but i would not have gone into

explanation myself like oh you know let's understand she lost someone um i would i would just leave it to the audience and and i want to know how much of your design and work rotates around the sensibilities and sensitivities of your audience and would you be comfortable looking back at trusting your audience to actually just go at it and just have it a talk at discord or just you know i don't know steam chat or something like that can i start that one yeah so that's one of the biggest reasons you're touching on one of the biggest reasons why we do have 20 people come in for a play test at a time it's the practical limit for us in the first place but why do we want a wide number of people it's not so much that we have then lots of numbers of people saying they have an issue but that we get a broad uh spectrum of the kinds of people who are going to play the game honestly and one thing we noticed just as a sort of a this doesn't actually answer your question but it's an observation that might kind of get at it

i'll have a follow-up effort um so one of the things we did was we decided to ask people a little questionnaire at the start about how much they knew about norse mythology so that we could find out you know are these people who are experts in knowing who freya is for example or who balder is for example and um we right away found out that people don't know a lot about norse mythology and it has a big impact on potentially how you're trying to communicate things and how people will perceive things so it's those elements of perception are really important i think and trying to gauge um and meet people halfway in cases where they don't necessarily have the right perspective or a full perspective to get the impact you want to try and drive so you you added that as a stretch to basically help people have empathy to situations like these um yeah i mean yeah to kind of follow up on what cameron was saying i think i think for us ultimately kind of you know you were touching on sort of the design process in relation to kind of having to

either explain or not explain but i think we're a we're very vision driven studio and so we fall back to sort of the creative director's vision on kind of what does he or she determine that we want to communicate or not communicate and when we posed this question to corey when he kind of looked at that feedback on like hey you know people are starting to perceive for you as a villain now you know it's really up to him and the narrative team to go like do we want to take this on do we feel like it's necessary to explain this point right and from his visions mindset uh you know he was like no i want to explain this and so we're going to double down on that so i think for us it becomes about sort of what is the vision we want to communicate and in that instance i think the way ed describes it it's we hadn't executed that yet we hadn't fully realized that part because for him it was super important that people empathize with freya in that moment so that's kind of how we find our way i guess

okay hi thank you very much for the game it's amazing first of all yeah and my question would be about how how did you get how did you make decisions based on all this feedback maybe sometimes arguable or contradicting that you got so whose role was that who kept for instance team focused on not making the combat when it was like so many ideas and opinions that we should so how did you make decisions like that i think that really was everyone's job and that's part of why it's so important to have that strike force why it's important to get everyone involved from my standpoint i mean a big part of my job is at least especially at the start of the process to try to help the team understand that we're working for them we're working with them in terms of research and we're not trying to change the game we're trying to make them realize they're help them realize their vision so basically that's what's driving a lot of the

prioritization of different issues that you see trying to weigh it against kind of where is the game right now what is it that the team cares about right now um what kind of impact are we seeing it have on players all of those things are going to go into the decision making on which things you tackle and which things you don't so it was like a group decision making it wasn't one person exactly yeah well i mean i'll give you a kind of a quick example we'd get a lot of data like kevin mentioned they'd help prioritize it give us a big debrief uh we take that information have a play test strike team meeting where we'd look at all the data corey would ask every each one of the leads to look at the data that directly impacted their team and come and talk about it in a very direct way of like why is this happening what do you think the next step should be confer with your teammates and then represent your team as a lead from that we would look at all of that and then assess what would the impact of this change actually have in the entire game and then ultimately it would be up to corey to decide if we're going to act

on it or not i see right okay thank you got it um great talk awesome game really appreciate it so we have access to a usability lab and research lab and so for kevin uh when do you think people should come to your lab is it like pre-alpha and i guess for you know kevin do you wish you guys started sooner versus where you're in alpha you're like we don't you know where's the fun so that's my first question and then the second question was a small thing was that on the chart for the unguided for the combat it was a negative four you pointed that out but it was also a positive four on the plus side so it was an even split on the eight tester so how do you kind of divide in the middle on that so okay first question um about when to start and yeah as early as possible but um you know don't do it for no reason of course it's expensive so you know spend your spend your dollars at the right places um but that said uh we have worked with people in the concept phase of and especially if you're coming up with a

brand new set of mechanics or you have things that you think are particularly like a vertical slice would be sometimes even doable before vertical slice even um for example i worked on like years of war with that team i shouldn't say anything about that particularly we did work um on uh elements of taking cover for example and other things that were really crucial and new to that game and a similar thing um happened here i'm guessing that in your case that it happened more as a matter of internal testing and internal processes but there's no reason why we couldn't have probably benefited from having consumers take a look at some of those things at the right time cool second question i forgot was it was the positive four and the minus four on combat for the unguided oh plus what happens if you get something that people like and hate very much yeah part of the strategy for asking those questions is that we expect that these things that we're asking about are so big and so important to the game that people can love them and hate

them at the same time so people aren't actually saying i don't like this or i do like this they're saying they're potentially saying things that there are things about it that they do like and things about it that they don't like and the key is just to figure out what those things are within that you know for example in the enemies let's say that they love the trolls and they hate the valkyries i don't think that's true actually but you see what i mean yeah yeah cool thanks thank you