It Takes Two to Tango: Integrating UX Research and Production at EA

hello uh welcome to my talk it titled it takes to to tango integrating ux research and production ATA so this presentation is about how we learn to dance together one dancer is production the other dancer is research when you start learning how to dance you're not good at it you step into toes on go to one side the other goes to the other side the two dancers are going to make mistakes like in dancing you're constantly moving like in development games too you're constantly producing iterating you're always moving there is no time to stop keep things coming and it takes practice to get good at it so have these two ansers

met user experience has been getting a lot of traction during the last couple of years within the game industry you could say that is one of the newest aspects Within game development for instance here we are GDC is its 29th iteration and this is the first time that is a ux summit that Al is very telling yet ux as a concept was coining the 90s by Norman it actually history of testing usability goes all the way back to the 40s so there has been a long history of measuring and assessing user experience in a variety of products particularly in games you have heard about play testing for a long time however such concept has been used in a variety of ways and generally without the framework of research and I'll Dave more in depth about this throughout the

presentation nowadays every major developer has a dedicated ux research staff I surveyed these numbers mid 2016 and I sort them my vertically notice that some of them are quite sizable it could be the equivalent for a whole company or to the development team in itself and more moreover as our understanding of player experience matures multiple disciplines are consolidating within larger departments and this is the case for Electronic Arts Riot and Ubisoft where games used research analytics market research and data science are part of an internal larg organization however during the Advent of ux research into game development

production might have been seen research As A disruption to process that they have already established while ux researcher were still adapting their methods and the vocabulary for game development context and that disruption took many shapes from production not seeing the need for research to production wanted but not knowing how to action on those findings mind you it was also for research to know how to convey those ux findings in a timely manner and how to communicate in such a way that were meaningful and actionable and this is part of the story I'm going to tell you today when you research are designed and not fully synchronized it leads to missing ux opportunities and opportunities doesn't mean that they have to be a major overhaul opportunities are also within prod redactions limitations of time budget

scope and technology and why I'm telling you all this well my name is Veronica Amo I'm the lead senior New Y researcher at Electronic Arts and over the last seven years I've had the opportunity of working an amazing games with of the AA portfolio and today I'm going to be sharing with you the story St on how we tackle ux research and production integration at EA spoiler alert this is how we are now this is what happens at the end of the story uh ux research adaa is a sizable team we are geographically distributed across eight locations worldwide and at e we have three prototypical tasks we have researchers recruiters and lab

technicians depending on the size of the company all those roles can be Carri out by different people or by a single person the latter happens more often in smaller companies or at early stages of ux research teams and that was actually my case back in the day when I started ux research Ada I had to do my own recruitment while designing the sessions and setting up the lab that was a lot of struggling so I imagine that each task take time away from the other one and it's not that it have to be done in sequential order so it's quite exhausting and it's actually not efficient nowadays IA we do have certain philosophies that actually Define and helps the way that we want to work so we aspire to work as one team across

multiple Studios and actually what's at the Forefront of our job is players so it's players first and all those things might sound great it's actually great but it wasn't with many bumps on the road and misalignments at first until we got there so how did we get there well first first and foremost we didn't get dead overnight it was a multi-year journey and sincerely is a path that will never end as the ux practice the industry and products evolve we will have to evolve as part of that and in this talk I'll share those examples those lessons learned that shape thriving and integrated the ux practice this is a journey with key mileston

and I'm going to go through those it starts rough and it starts with a wakeup call back in 2009 the NBA elite was being developed improving ux was a huge driver for that team they were high expectations they were introducing a brown new set of controls mechanics new to Evol it was meant to be a real boot of the NBA light franchise and consequently the game team was very self-aware of the quality of the ux data and how actionable it was at the time play testing was part of development process however it was led by designers and producers who selftaught themselves how to do usability and even though they have the best intentions in the world they could just smell that something could be better something could be done

differently H and that was the seat of starting from transforming play testing into ux research in 2010 we started initiatives for new approaches to evaluate NBA with players and bringing the edge of science for that I went Super Tech and I brought Advanced technique that it was the first time doing that for play testing so using ey tracking and Biometrics and Telemetry and all of those at once to assess visual attention emotional veilance tracking their in-game behaviors it was awesome the idea behind that was to tap into insights that surveys couldn't get and the effort was to give the most we could to NBA for understanding the

player experience there was a particular fascinating finding that players consistently look at the coach after making a basket and bear in mind that the coach was there in the background doing nothing no animation no voiceovers anything that could led to him so it was very unexpected and pointed out a Miss opportunity for positive reinforcement that Insight never appear with any other technique I was getting insights that were completely new on that approach and all this marvelous work was new so the process was a little bit slow and actually and those findings couldn't be done there was no automation it was very labor intensive so findings like those need to be left for discussion for the next installment of NBA time was passing by and the team needed

to focus on finishing the game an NBA elite was coming hot as it's common there was a demo schedule and release to the public and I'm going to show you next some Snippets from a video that went viral of a player playing the demo one thing I didn't like they make you playing that up and down view I really don't like that [ __ ] up and down view oh I can change it let me change that [ __ ] broadcast there we go I didn't even know you could change it I just noticed that you dig take care of the basketball unforced cre OPP for to and what the [ __ ] is going on hold up man something wrong with my goddamn buttons this [ __ ] is

crazy I didn't have it on a leite that's what it was all right let's go the response time to the buttons is slow as [ __ ] what's that about these controls is crazy you got to get used to these controls this is crazy man I ain't played live in so long what the [ __ ] yo y'all seeing this [ __ ] yo what the [ __ ] this [ __ ] just standing like Jesus in the middle of the Court oh my God man that's it for the demo man I can't take no more of this [ __ ] man it's hard a times man Deuces man deuces was a funny video

but there were obviously friction points in the UI there were friction points in the mechanic and this demo actually was a nail in the coughing NBA elite never ended up being released it was deemed that it didn't reach the desire quality that players would be disappointed and that the best decision was to not release it at all it was an extreme extremely hard decision and I canot stress enough how much of a shock it was for everyone it was a wakeup call as a video game company was very clear that quality process needed to be better and I could I wouldn't go as far as saying oh we know better ux research we have been the Silver Bullet for this there were other problems but definitely clearer and more itative ux checkpoints need to be

there with that the new ux research initiatives were a business case to bring it to the next level and there was work to be done we needed to improve the research practice from its foundations even though biometric techniques Pro to be very insightful we needed to change Focus from advancing methodologies to establish in ux process we needed quality research in an iterative process where production and research are fully synchronized that means having a plan knowing your steps we tackle this problem from three different angles strengthening the good research practice from a macro as well from a micro level of research and production integration regarding the basic good research practice there were imminent aspects to address such as quality data things like

ensuring woring of questions in service are clean and not leading these efforts were towards researcher skills and the goal was to have quality data so we can been certain that the research findings are good and that also leads to trust trust from production it's key they don't have to double check the data they don't feel they have to go themselves and trust is the Cornerstone for any relationship but ultimately doing research is to communicate those findings and like in Tango communication has to be clear and timely because you're constantly moving you don't want your partner to go that way and you that way or accidentally spping into each other toes so I'll show one example on this topic and you all have seen research data coming in this

form of prototypical question and a scale from 1 to five all player data aggregated and presented in a bar chart like this which is okay to know how good or bad things are however not it's not sufficient if you want to prioritize resources in production in cases when you have to look at multiple variables and do comparisons this is the case in Plants Versus Plants versus Zombies Garden Warfare where there were more than 40 characters that you have to choose from and Productions wanted to ensure that all of them were hitting a mark and if not focus on those first and that was exactly the question that we had which was great and looking to communicate findings we look at different ways we put like some tables so now you can really see the spread of

responses and even though the game team was happy with this you can start see that St to hard to read and when you had like 40 Columns of those like you cannot see through that clatter so that's why com intervals were becoming a part of integral practice of research and how to communicate findings confidence intervals are going to give you a way to repres and that variability from players responses which allows to spot meaningful differences it keep it simple yet with a richer inside so in that case let's say that the target score was 3.5 and the option in Orange is coming tight just hitting the mark could be better sure but it's coming at least 41 it whereas the blue option you can see was actually looking at a first glance stronger but actually it was inconsistent there

were polarized options it was a bigger risk and gaining prioritization from production was at Essence and that's how confidence intervals now are part of a basic research practice of doing research now at a macro level at EA we have a game development framework which is all the games developa have to follow I have a simplified way here which is pretty much all the different stages common in the game industry pre-production production and release so it's applicable to everyone in the room the value of stopping and getting intimately familiar with the dep process is that we can really understand from research where the efforts are at different stages what the biggest challenges are what are being asked by

Executives all those challenges is a clear information that needs to be common otherwise it leads to misconceptions about readiness priority and also vice versa how research activities are align to the best impact for those ux efforts so I add some of those prototypical tasks in the chart But ultimately that map is a map that framework is a map that ux and research need to navigate together so that's what we did we tried to find that common language so for ux research at EA in order to be part of the development process we need to articulate how we were bringing value to the table and we work on an expansion of that framework laying out all our prototypical task and Endeavors of those stages and from there we were able to work with production to far further tailor their questions on a specific

game characteristics and design intentions for example in Battlefield 1 a key design intention was being epic we make it explicit that we were supporting that feeling of epicness throughout the whole development process being the first person shooter in World War I meant that for instance weapons to feel old schooled yet authentic but not to the detriment of a slowing down players actions which we did there and another example was the introduction of large Vehicles like the Zeppelin we need to answer those questions Rel to the impact of introducing those Behemoth those Vehicles during a full match during our play testing sessions now we have a road map we know what

we're doing all that is key now we need to execute it so let's look at the micro level of how we work and I'm referring to this of what happens in each and single study and a default test can be divided into four big steps preparation execution analysis and Reporting it informs production action on those findings and then we go to the next iteration to the next test we were doing fairly good with production in the kickoff meetings everyone was attending uh it was going great during the execution of the sessions production was coming to the observation room they were there in the test now what the interesting part of the story here was between analysis and Reporting and we were experiencing a couple of Errors during that time at first we we were so eager to give them the best research ever that

one of the deliverables for the final report took two weeks from final session to hear report uh and that was detrimental we forgot that we kept moving that our partner was going over there dancing and by the time that they got the results they have made changes a lot of those findings were obsolete dying inside those application were gone something were salvages but it was very clear that we were not aligning with their pacing it was also generating anxiety from the day of the test to getting results they knew that we collected data and they wanted to know how we're doing what unnecessary settings for the dev team we went on the Other Extreme on the Spectrum and we Tred to be very very very very quick and try to deliver it within two days and and uh analysis was hurt it was more like a data dump no

proud of that work so we need to find that middle ground we need to do something to give good data good analysis in a timely manner and that's when we introduced what was a top liner in our process which is between just 24 hours after the last session production received top liner which contains a high level analysis of the test and then a final report two to four days later with a full scope and that helped greatly to our communication in providing information in a timely manner and keep moving forward it helped to the relationship as well as addressing their needs so now that we have the ux process laid out I want to focus on organizational structure for ux research

and with this I'm referring to where within the company should ux research live and organizational structures have implications on relationships the stability for the product and how you're going to be prioritizing content at EA we had a centralized ux research organization we engage with multiple game teams across all EA and as you know the EA portfolio is pretty large so in other words we have a single team which has a handful of researchers carrying out all the research activities for the whole organization this means that the researcher was a no part of the development team and that the research was supporting multiple franchises for instance uh someone will be working on FIFA as well as UFC and

NHL this type of organizational model was great for us at the time remember that wake up call we need that good sense of processes so these organizational mod was great for that why well there are like four key positive aspects for this uh for once it creates a really high strong Hub of experts that allows a lot of easy sharing of best practices reinforcing that quality we which was very important to mature our practice the range of projects was varied so it kept entertaining and refreshing which for returning Talent long term can be important the accumulation of knowledge across the different products allow us to have meta insights if we put all our is together all our Sports together we can answer bigger business questions for

this and ultimately it has an economical benefit too because Central teams can share those resources you don't need to build like two lab three Labs you can license products you can have your recruiter your lab technician that serves everyone so that was great for us and with a handful of people we need to cover a lot of game and that's where some challenges are we have to rationalize efforts we have one person with three titles three big titles well we cover them all but it's still one person and that leads to a constant priorization exercise of the portfolio which can be tricky because it has implication of what's important what's not important what we should tackle and you don't need a big portfolio I'm pretty sure that all you can relate to this with your within a single game you

have different features you have different modes something have to give attention something will not have attention and another shortcoming of the structure is the relationship with production which tends to fiz more at arm length the development team can perceive the researchers as external and the researcher feeling out of the loop from production sometimes the game changes a lot within a week [Music] so what we do at EA the research efforts and the improvements that we doing were paying off and we had great support it Ena to actually it's missing something but doesn't matter uh it was missing no they was animation you saw it there it is it was there so we were very lucky that we could actually increase the size of the team uh and we could actually now be

having people more dedicated to certain franchises and not dividing them so it was one person for FIFA one person for Battlefront and that was increasing fantastic for our bandwidth but getting more headcount for more researcher is not a light task as in any organizational for any role is something that just doesn't grow on trees think about uh evaluating what a decentralized model will look like and where people are independent from each other so however we concluded that the slower pacing of our practice didn't really didn't really Trump having that close close close relationship with the team at the time and something that was huge for us is that the lack of not having comparable results was a huge huge risk that we could not afford it was almost like calling for another wake up call so we wanted to keep the quality

we wanted to people to know how to dance properly not just shaking their body in some way not like this thanks Ian so by now we did have a better established a stronger relationship with the production we identify Champions within the game teams and we work with them to iron the process so things have improved a lot but what happened for us was keeping each other's in contact with our partner and one day one of the key point of contact from NHL moved to another project and that left a void into our synchronization and instead of waiting for the NHL team to come out with a backup feel for that position we step our game and we decided to propose to go

embedded with a researcher in other words that the researcher will be working now on the floor with that Dev team are being reintroduced as a team member rather than than a partner and uh inial team was very supportive of this approach and it was awesome it really helped us to improve to a new deep level of those relationships we have that extra time and dedication with the team now so we could actually now answer more questions there were more people from production asking more and more to the researcher and it was also an opportunity for the researcher to keep developing those skills so because we have that strong Foundation now we could refocus on those Advanced Technologies and bringing back to the team as in that case they were focusing a lot on UI and accessibility to Blind color players that we could go back to now keep

advancing the field the one risk of going hybrid is that now that researcher becoming an island and that's a delicated because it's an island from the other researchers and actually how to allow tailoring for certain practices but yet be in align with the overall process so we're working on that a Vari of different ways from our end and keeping that tightness of the researchers by allowing to have internal projects to work on things that I can connect it and to can actually can keep advancing our practice over and over now what I've told you so far is related to the grassroot efforts a lot of that change was from ux from the bottom up but really to change culture within a company it also needs to come from the

top down there needs to be management buying into ux and at Thea we were all doing our dancing steps but you truly dance when you're going with the music and the director of the orchestra is a key piece to keep us all together and dancing you need leaders on your organization to believe in ux and to support those efforts so in 2013 when Andrew Wilson became the CEO of EA as a new leader in the company he set a series of pillars for guiding EA and the most important ones was player first when I heard that I was likeo I was over the moon with that because he was really talking about that nothing is more important that our players that user Center design is the way that we work that is player first and nowadays there is no discussion how good a game

is without talking about players Insight with that was the whole journey of the AF of ux so far and I just wanted to recap on those shared learnings from today have a shared framework you need a good re research practice that is effective and communicate clearly but it sets a map to relate to each other for those steps at the macro and micro level so make sure that your findings are actionable And Timely all the time constant Improvement feel how your partner is doing and adjust accordingly leverage on opportunities find an organizational model that fits your needs and continue reviewing your deliverables and Tech as you go through and ultimately it has to be ux culture and it needs both dancers and the music

and to have those is go from the Grassroots but it also you need to find your Champions thank [Applause] you