Step Into Your Player's Shoes: Making the Most of Team Playtests on 'skate'

[Music] um hi everyone good morning um thanks for joining today I'm happy you can make it I'm happy I can make it because my flights got crazy yesterday and I got here at 2: a.m. um it's okay it [Applause] happens uh um so I guess before I start um I was asked to remind you to fill your evaluation forms um so you can scan your badge and he will send you a link uh so you can add that uh it helps us make sure that the talks are better and also some of us get into this really cool card deck uh so I evaluate your talks um all right so welcome everyone to step into your player shoes and how we made the most of team play test on Skate today I'll share with you a process that we applied to the internal

team play testing to make it more user centered and to gain more relevant feedback from the team so let me start by introducing myself hi I'm Andrea gonalves and I'm a senior re designer at Full Circle Electronic Arts I'm originally from Portugal uh so I might have a little bit of an accent um but I'm currently based in Vancouver Canada I have a background in both design and programming and I've been working on games for over seven years now I've worked in mobile PC and console games and I've also taught mentor and spoke at different schools and online platforms currently I'm working on Skate and I've been on the team for over three years now before we jump in I want to set some context to the talk first I want to share that we're still working on it skate is not out yet so in this talk I'm going to be focusing more on the process that we use and not so much on in-game footage so if you were hoping for a sneak peek I apologize

guys I will not be able to share anything that is not public yet um but you can see a lot of what we've been working on on our YouTube channel the next thing I wanted to talk about is maybe you are wondering why we're talking about team play testing at du Summit because it's not testing with users right uh so I wanted to start the talk uh by stating that teaml testing does not replace user testing and in fact in our team we've been testing with players from early on we have videos in our YouTube channel that you can see of wacky graybox world but teaml testing is another tool in your toolbox to ensure that you're going in the right direction with your experience and we have found in our team that there are ways that you can make this process be more useful in your ux journey so that you can leverage an existing source of feedback to increase awareness of ux so with that out of the way let's look at what we'll talk about today I'll

start by introducing team play testing and its strengths and weaknesses then I'll share our process which internally we call Focus team play testing to get the team to step into player shoes then I'll walk through how to put this process in practice in your own teams and at the end I'll share some learnings over using uh about using it over the past years so let's start to look by looking at template testing and why it's worth leveraging first let's talk about why do we test the reason we test as devs is that our goal is to make a good game for players and we have tons of tools to help us assess this there's a teaml testing there's heris evaluation and expert reviews there's quality assurance departments to make sure the game is working as we plan and there's external uh research uh such as us external usability tests with players and external free form tests with players

and today we're just looking at one in particularly over all of those tools so what is Team plate testing I'm sure a lot of us are familiar with it but let's define it just we're so we're all on the same page at its score teaml testing means you're testing with members of the team as opposed to people that are not your in your team like your players and it's often a common practice in studios and game teams it's also often recurring such is happening weekly or monthly and this team plate testing can be done with different size groups so sometimes it's just two to three people that are actively working on something but you can also be with the whole team and there's people that haven't seen any of what you're working on before in my experience in the teams that I've worked on we always had at least a team-wide play test that happened weekly template testing has one A really key strength and that's that it gives you quicker and lower iteration uh

quicker and low cost iteration Loops um this is for a few factors one is that it overall usually has less Logistics you don't have to recruit external participants and plan extra studies developers already have access to what we call Dev builds uh that are easier to create and access and it's not just a quicker to implement your design for the first time it's also quicker to act on feedback and Implement changes because of the things I mentioned like Logistics and access to the dev builds but also because the target quality that you need to make for the dev team might not have to be as high as what you would have to do for players but team play testing has a really key weakness and that's that you're not t you're not testing with your target audience developers are a very different audience and come with their own set of biases so what do we mean when we say that we're not testing with a target audience when we test the dev team

usually it means two key things one is that developers are too familiar with the game and are well developers and this means that they can see the game from a fresh perspective which is a cognitive bias known as curse of knowledge which makes it hard to look at something Through The Eyes of someone that doesn't have as much familiarity as you do but developers also usually use the game in a different way than players for their workflows where they are testing seeing what they are implementing and iterating on some work fls that the users will never use the other thing we usually mean is that developers and actually any player and any person has varying amounts of personal tastes and motivations they can value different uh features in games be excited by different things and different genres uh and that's not that doesn't always align with what we our audience is looking for and the game that we're trying to create this is not always this is not a

negative because it's awesome to get so many ideas and it's great to look at different genres but you need some parameters to be able to interpret all the feedback and suggestions external user testing usually solves this for us when we externally test we look for participants from the target audience and sometimes we look for new players so we can specifically get a new perspective and these tests can be sometimes free form or player play more organically or Focus where we look at a specific part of the game and get deeper inside but external testing is not always available since sometimes there's no bandwidth or budget and some smaller teams may not have access to these all the time so so far I covered the benefits for iteration that come from teaml testing but in our team we also saw so many benefits for the team team

itself uh full circle is a remote studio and this uh teaml test became a weekly touch point for us to come together with a shared goal of making a great game over the pandemic we had to adapt how we work and tools like Zoom a video conferencing app and mirror a digital whiteboard became part of our day-to-day as well as these tests and directly impacted how we were able to conduct them in this test we share our design and intentions for different parts of the game and this has helped build a shared Vision among the team it has also allowed us to get give a chance for everyone to give input on parts of the game that they don't always work on and lastly because we're working on a multiplayer game it actually became a fun activity where we can connect and have fun together as a team our studio is committed to diversity and inclusion we want to have different perspectives in the team and

we want to listen to them so this team play test became such an important practice to support this but with all these different perspectives um there also came a need to interpret the feedback so by now we saw that teaml testing can give you a quick and lowcost iteration Loop fresh eyes and different perspectives in the team and can help with Team alignment engagement and creating a Shar Vision but it has the big Downs side of developer biases so how could we leverage the pros of team play testing while minimizing its weakness although we can never remove all the biases at play what if we could try to find the closest to our players in our team this is what we try to do with our process that we call Focus team play testing we get the team to step into the player shoes we do this by providing them with scenarios to try that are rooted and player goals and we segment their feedback

based on relevant traits that they have in similarity with players and this has resulted in more meaningful feedback which for the context of this talk means that the feedback that we saw from the team became closer to what we were seeing from players and more team engagement and ux byy in so we saw uh more participation from the team and higher understanding of player paying points so now we cover team plate testing uh and talked about Focus te plate testing what does this look like so for us Focus team play testing has two key pieces one is a player lens that you use to assess the game from so that you can look at it from your players perspective and these include scenarios that the team will try and the user traits are you going to use to collect and weight feedback differently and the other piece are the logistics around planning and running the this play test the most effectively

to get relevant feedback let's talk about the player lens first the player lens captures who a feature or a game is for and how it's going to be used and it's composed of the scenarios and the user traits scenarios represent what players will actually do and how and is what the team will actually try in the tests so what does the scenario look like a scenario can be about performing a task in the game in context an example where we used this with was with our capture feature a feature that allows you to capture photos and videos in the game we saw that some players spend a lot of time in this feature making perfect photos and videos but a lot of players just want to quickly go and grab a photo of something cool they just saw so they can uh save it and share with others so when we asked the team to test we asked that they try um using these feature in that context of just seeing

something cool and wanting to quickly grab that so that has taken us from asking the team to do things like just take photos and use these two features such as camera angles to perform a trick that you're really proud of in the game then go to the tool take a fe a picture of it and save it and continue your gaml a scenario can be about using a feature for different goals we saw this with our building feature which is a feature that allows you to place objects in the world we saw some players use this feature to create skateable objects and this need a lot more Precision because people need to skate on them but other players just use it to reach higher areas in the world and explore so when we asked the team to try these tools we asked that they uh try it for these two different purposes and give feedback on how well it works for either a scenario can also be about placing the team in a c situation or or context that is similar to what players

will experience an example where we use this was testing our social experience in this case we place the team in breakout rooms of a certain size so we could get feedback on how it feels to play our game as a party and lastly a scenario can have constraints an example where we use this was also with our social experience and in this case we ask the team to not be on video conferencing and not use voice so we could get feedback on what it feels like to play the game if you're by yourself at home now that we looked at the scenarios let's look at user traits so the user traits represent different aspects of a player that are relevant to consider for a certain design an example of a trait can be the level of familiarity with a genre or feature we use this one a lot uh for two examples where we use this this was with our capture um tool for capturing photos

and videos that when we collect the feedback from the team we asked that they give their feedback in different areas of the uh board that we were using whether they were new or more experienced when using this zo another situation where we use this was with our uh gameplay controls when we tested the controls we asked the team to give their feedback in different areas whether they have low medium or high familiarity because we're asking the team to place themselves in what user trait they think is closer to them we gave them a proxy um as of um being how long you've been on the team is maps to your level of familiarity a user trait can be about a motivation to play the game or use a feature an example where we use this was with our building Tool uh the tool to place objects in the world we saw that some players want to make huge creations with many objects well other players

just want to modify something that is in the world quickly and even other players are not interested in using these tools particularly so when we asked the team to test these tools we asked that they give feedback in uh different color sticky notes depending on which motivation uh they most align on so that when we look at the feedback we can consider the player type that they relate to and lastly a user trait can be about a shared paino uh so we did this with our gameplay controls we saw there are a few few people in the team that had difficulty uh with the control complexity so we uh identified those members in the team and we actually did a team plate test just with with those team members to get more information on the pinpoints and then test uh improvements so now that we looked a little bit at the focus teaml testing process I'll show you how we've used this in the past to iterate through a feature and we miss mix and Max match testing approaches to get it from not

working to a desired State we did this for our capture feature uh the feature that is used to create photos and videos in game so we needed to do improvements to this feature we saw from externally testing with players that it was not working and it was expressed through U metrics that we were targeting and that were below what we wanted when looking at the actual feedback of what players were saying we saw that um the feature was too complex and complicated and in one of the tests U users were overwhelmed and almost no user actually ended up using it so we went back and we created a player lens to help us assess this problem for from and we saw with taking photos and and videos there users that want to do complex creations and spend a lot of time in the tool and users that just want to take a quick photo and continue playing and we we saw that um the issue was occurring for users that wanted to

quickly create so we decided to prioritize that and it kind of was supported by the research that we were seeing where players said I just want a quick way to take a photo we had another external user research test in the future for the whole game but we didn't have any bandwidth to do any further testing U Milestones along the way to make sure that this feature was improving so we looked at how we could do that with the team we started by doing internal one-on-one tests with the team we recruited participants quickly on slack through a survey where we asked motivation that they have to create and how new they were to the team then we conducted uh tests over zoom in a one-on-one scenario where we asked the team members to do H tasks that were related to quickly creating and after each session we would debrief and see how well they were able able to do which task and what were the

problems and things we needed to improve we did a total of two rounds of these one-onone tests e each of three participants each and at the end of each of those rounds we had a new version of the design um to test so after after the end of the internal test we had one version and at this stage we wanted to get wider perspectives so we went to the team wide play test at this stage in development our team white play test had about 50 people all at once in a sort of usability test at scale um these days these tests have like 100 people in them I know it can maybe feel a bit overwhelming by looking at the picture but actually when you're participating It Feels So energizing to feel like we're all playing the game at the same time and we're all providing your our opinions and we're making the game better together because this was not a one-on-one one test we had to find ways to deliver these tasks to the team at

scale we used miral the digital wideboard um tool to lay out the tasks uh that we had used in the internal oneone tests a simplified version of it so the team could do at their own pace and in the board we created two different sections for them to leave feedback as they go and one area if they were new new to the tool or more experien and after a test I had tons of feedback and and sticky notes to go through um but was really cool and I was able to find themes and see differences between what issues were affecting new uh participants versus more experienced participants and was able to prioritize and decide what I would action on so after this test we had yet another version of The Tool uh to to use and it was actually time to test with players again so we did an external test with players at this stage and we saw a lot of improvements after these tests we saw

that players were able to accomplish all the tasks that they were meant to do with this tool but we didn't quite hit all our Target targets and in particularly there's there was a usability Target uh that we didn't hit based on feedback we narrowed down the remaining issue uh to a learnability issue so there was um a part of the feature that players didn't understand right away but once they use it they found it intuitive so we decided to solve the in particularly with a tutorial just to walk the player through using it for the first time and we decided to test in the team wide play test so we could get a lot of uh perspectives we got positive results from this internal team play test but we also saw that we didn't have enough new people in the team to truly be confident uh of these results so we really needed to test with external players so we tested again externally and and we finally hit all our metrics

so we achieved our targets and there were no more critical issues with this feature at this stage in development so by leveraging the focus team play tests we got internal feedback that helped us improve the solution in a way that met player needs in the end and we saw the improvements happening within the team starting to map to the improvements of to our target audience more closely and this was available to us when extra user testing was not possible I've introduced the focus team play testing process and we saw um an example of it in practice now I'll cover an overview of what goes into planning and running one of these tests in the last section we looked at the player lens part of focus in play testing and now I'll cover the logistics which are tightly related with how to run these most effectively we'll cover

the overall process as well as some uh tips and tricks I want to share with you my hope is that um this will help you implement this with your teams maybe who knows next week for us uh running a focus in play testing has three important stages planning running and action and Key activities to do at each stage that relate to the process in the planning stage you want to create that player lens that you're going to use to assess the game from and this sets the user and the context of use so you're setting who it's for and how it will be used in our team usually this is done by U designer so game designer or experience designer we create these based on assumptions or research that we have in the planning phase you also want to Define this test format so how are you going to test to approximate the context of use that player will actually be in do you need to place the team in

certain constraints or contexts which tasks do you need to ask the team to perform and is it better as a team play test so you get tons of perspectives or a one-on-one scenario so you can get more deeper insights then it's time to run this test there's lots that go into running a team play testing but the most important thing in this process I think is the facilitation piece here here you want to deliver clear instructions of what we're testing and why we're testing it so people can be uh can participate and then you want to make people feel they can give you honest feedback and that their feedback and time is going to be valued the last step is action so depending on the size of your team you're going to have tons of feedback and you need to interpret it and Define the actual improvements you want to make and here is really helpful to contrast the feedback that you receive from the team based on those uh different player

types that you were seeking out after this you will have a list of improvements that you want to make um and you will Implement them and then you can use this process or a different process to validate that those improvements were actually effective I will cover just a bit more facilitation since I think it's an important part of getting the engagement uh from the team and good feedback the first piece is the delivering clear instructions and context you want to tell the team what we're going to try and why where it's at in this stag of development um and how it fits in the broader picture of the game this is going to empower them to give you feedback in relation to that and feel like they're part of the journey it's really important to create an enironment for honest feedback from the team and this comes from having the team feel like their feedback um that they are safe and that their feedback is

valued this comes from really encouraging everyone's opinions especially uh saying that if people feel like their opinion going goes against the majority is even more important uh and also providing inclusive ways for everyone to participate so Miro has been really uh great for us because uh people can deliver their feedback by writing instead of having to speak up in front of everyone and to show the impact you always want to look back with actions and results of what you do with the feedback so people can see the outcome of their participation another thing we found helpful is asking participants to tag their feedback with name with names um this has helped us follow up if we don't understand a piece of feedback or we need a little bit more information uh or also helps understand uh that team member across play test to have a better understanding of their player type and better supported and lastly and this is um

often a difficult one is to always direct feedback to your collection method um so um what what I mean here is that sorry to the collection method this can be really hard in the team wide play test where everyone is there all at once and someone may feel like something is so obvious they should just ask the group um what it is and people are going to hear their question and feel compelled to answer um this is hard to go around uh but this actually is the feedback that you want to get when someone feels like something in the feature is so obvious they should just know it but they don't and they feel like they need to ask the group so you want to intercept ask that they provide the feedback in your collection method and then find a way to unblock them so they can continue with the test so now that we saw the focus team play

testing and what it looks like and covered how you can plan and run one of these I'll share some learnings from using this process in the past few years and we'll wrap up um there was a lot to share um but I'll pick I picked a few ones that I wanted to share with you today one thing that we learned is that it's really important to reduce friction for the team at times we wanted to ask more questions give them more tasks or get more granularity on their segmentation but when we increase when we increased the steps and the complexity we saw the feedback decrease and the quality of the feedback decrease um so we always try to be mindful that the team is seeing something for the first time and they're playing the game which takes a lot of resources so we need to be mindful of their cognitive load we also saw that the effectiveness of uh template testing in this process can vary although applying Focus team play testing has been better in general

than what we don't apply it it's not ideal for everything and one case that we saw is when testing a tutorial or first-time user experience there's such a really heavy bias in the team of knowing too much about the game and being passionate and it's not realistic to have a constant in influx of new team members in the team to test with New Perspectives another thing we saw is just how powerful firsthand experience is so when the developers actually get to try the game as a game would as a player would actually do um they get a deeper understanding of the player pain points and this makes them it makes it them rally around fixing them so instead of being a jur of something that sounds like a minor issue they realize how frustrating and big of a deal it is so this has been particularly effective on showing how sometimes simple tasks are really hard for players to do and it's something we can forget because we use the game all the

time the last learning is uh that diversity is really key you need to have a diversity in the team to have access to all these different perspectives this will give you more chances to find people that share different relevant traits with players so how do you get your team to step into your player shoes you can do this by running more focused inam play test where you create a player lens where you provide scenarios for the team to try and feedback and get feedback based on relevant trades to the focus and you plan and run these tests in a way that um that creates a context that players will actually be in this will give you more relevant feedback and confidence that you're solving Real Player pain points because you set a focus you get feedback on that Focus because you give scenarios you get feedback on things that players will actually try to do and because you're segmenting you can see which issues affect different typ types of users this lets you wait the feedback to solve

problems players will actually face and it also gives you team engagement and you X Buy in because you provide a clear purpose and context the team can give you feedback in relation to that because their feedback is more relevant it's more likely that you're going to apply it and they will see the input the outcome of their participation and it's also a chance for anyone to give input and we heard many times from our team members mentioning that they this way they can feel that they contribute to other areas of the game they don't actively work on overall this process has been a tool to understand and empathize with the end player experience and help rally the team around fixing uh solving player needs so we looked at Team plate testing and how it's a useful source of feedback but can become more user- centered and we saw that Focus team play test help the team try the game closer to players perspectives and the focus team play tests lead to more relevant

feedback and higher team engagement so after today I hope you have the basics to invite your team to step into your player shoes by running more focused play tests to help improve your internal feedback and increase empathy for player pain points thank you for listening um I wanted to share before we go for questions that full circle is hiring so if you enjoy what you saw today and are interested with working with us you can follow the QR code uh or check out our website directly and with that said um I think we are out of time um but I we can take a few questions uh so we can take a few questions here and if it uh we have more questions we can do it in the wup room 2014 uh please go to the mics if you have questions oh sorry I'll put that back

yeah yeah um U first question there yes hi uh I have a quick question about uh so the with the way that you uh did the team play testing you had it all be happen at the same time through like a teams call or something would there be any change to the system if you're hosting this kind of play test as synchronously where everyone's kind of doing it on their own time over like say an afternoon or evening that they have a time to so if you were doing in the in person a display test uh or more so a asynchronously like people doing it basically on their own time when they have the chance to yeah um I think uh I would prepare the collection method the same way so providing all the tasks and where things are going uh where to give feedback and make sure that the instructions for how to test and how to deliver the feedback are very clear so people can do it on their own time um but I think I think the format Works uh well for us synchronously as well

awesome thank you thank you in a game like uh skate where creativity is such a big part of it how do you maintain a level of like abstract interpretation of the level design in internal play test where people are so familiar with the game World um um that's a good question I don't know if I have like a clear answer for that um but I would say the team is really diverse so you have we actually have people that are very aware of that and there's people that don't have the same familiarity so I think that's where the strength of diverse team comes because you have people that are super passionate and are going to give you like the fan feedback and you have people that are newer and can give you a perspective of someone that maybe is a potential um player um so I would say it comes from the diversity will give you perspectives from different people thank you yeah hello you guys are lucky that you have your own external play test team that's

normally supposed to do most of these play tests developers are busy every time they're play testing they're not developing the game that's why you have that team H what is the ratio of developers play testing versus your external team play testing and does that change as the game matures where you want more external versus more developer it's obviously going to be a balance of love the perspective yeah I I mean I would say probably in the beginning there's a little bit more internal but you should be testing with players as sus as you can but you do have access to your team a lot of the time um so maybe in the beginning there's a little bit more internal um and later you have access to players which is the best target audience so we still do internal because it gives you like when we're starting out features we can test them really quickly but we start testing with players much quicker because we have all the infrastructure ready um one thing that I wanted to mention about like having developers um testing the game and when they're doing it they're not developing is that um there's kind of like unseen gains of having developers test the game

and and I kind of mentioned with like the shared Vision um and and now they understand how the game is supposed to work and when they need to do their day-to-day decisions that you don't need like a full design spec to decide how to do one thing versus another because they participate in these sessions and then they understand where we're going they're able to make those decisions based on our best understanding of players so I think it actually like is you get a lot of benefits from the team culture that you create with these tests that sometimes can be like um forgotten because it's like oh the jurus are not moving uh we're not completing them fast enough yeah you hello uh first of all thank you for the wonderful talk a lot of actionable stuff um a question I had when you're doing focused play tests in a team wide setting uh how do you avoid the issue where if one person vocally focuses on one issue that the other the rest of the team basically starts

snowballing into recognizing that specific flaw and minimize the bias that people have regarding that yeah um there's definitely an issue that we had especially in the beginning when we started introducing this process um and what we try to do is set an expectation that uh first we want to collect things um by writing so people contribute as they go in writing and then we try at the end of the session create like a forum for discussion so people that prefer to speak vocally can do so um but of course in the beginning people are going to try try to talk if they're comfortable talking so you want to direct oh like we we don't want to like get into those issues or buyas the feedback but that's a great Point can you write it in the board so I have that and then we can discuss more at the end thank you thank you great talk um I got a couple questions first one is um how how early did you start doing Focus play testing with your team like uh in was it

in pre-production or were you already in full production um I cannot answer where we are in the process of the game um but we we started quite early um I think the first couple play tests um were more free form cuz we had like very Bare Bones um but I would say like half a year in uh we were doing Focus team play testing uh especially like once we started playing with players we kind of start seeing like maybe there's a bit a discrepancy here that we want to get solved okay uh second question is um are your weekly play tests always uh with your full team or did sometimes the one-on-one sessions replace those um in our team the week the team play test we have one weekly so it happens every week and because it's scheduled it reduces a lot of logistics of preparing and inviting everyone um and then different groups make their own like extra play tests um or one-on-one sessions when there's a need for one um and we also have um so we have our um more Focus

team play testing once a week but we also have a casual play testing um that is just for playing however you want awesome thank you yeah and one last question okay great uh so I have the question not about the pro set itself but more about the uh you know within a team how do you initiate this process cuz my assumption is this team testing process can be very or kind of time consuming to every discipline and say how do you convince the producers to say can how can we sped that much time and energy on our team to do such a thing like set up those scenario and they have weekly meetings group everyone together you know how how do we make that happen in the team yeah um so there's two scenarios here I think one one is a lot of teams already do internal team play testing so that's a

great opportunity to just uh try to sneak into that process and be like oh let's try this other way of running it once um the other part is kind of pitching for having um weekly team play tests and I think um I would suggest kind of starting to do it by yourself and show the value that that creates and then people will be more likely to invest in that um and then you can increase the amount of effort you put into it like maybe your first one is not focused in play testing is just play testing more formally and then you're like oh maybe we should add this um as a Next Step um I I think those are kind of the main things I can think about um but usually like if people already do internal team play testing is is easier to like try something new something else I wanted to say about the cost is that you don't have to create this player lens and format every single time once you create it for a feature is is really likely that you can just reuse it over play test so any changes you make you're

just trying okay we made these changes let's try it again from the player lens and and see if we're getting better um so the the cost decrease over time once it becomes something that you do often yeah thank you thank you all right that was it thanks [Music] everyone