BioWare Embedded QA: Your Support Class

[Music] hello everybody and thank you for coming to a bye we're embedded QA your support class my name is Barbara Clinic I'm a team lead at Bioware and I've been with Bioware for about six years fun fact it's my birthday today and I can't imagine a better way to ring in 30 than talking to you guys about a about what we do when when I started in QA I I had no idea that what buy or did I didn't know what embedded QA was it didn't really know much about QA but what we do it's special what a lot of Studios do by using a better QA it's the future and that's what I'm here to talk to you guys about about how we do things and if you are interested in doing a better QA or you already are doing embedded QA some things that we've

learned that maybe you guys can apply and this talk does run pretty long but both Garrett gear and I have a roundtable at 3:30 today in room one to one in North Hall a couple pieces of housekeeping please set your cell phones to silent and the team does send out a survey at the end of this presentation so please fill it out let them know what you thought of the presentation what you thought of me and what you thought about having QA talks at GDC in general thank you so embedded QA to understand embedded QA I need you to understand what I mean when I use the phrase blackbox testing so in blackbox testing which was the more traditional way of doing QA QA was given a copy of the game to test in isolation in that model the relationship was limited between Quality Assurance and development and they were seen as an external team not part of the

team itself blackbox testers would be isolated from developers working either in a large room or in a separate studio entirely blackbox testers didn't interact with developers directly and all test requests from developers would come in to a test lead and then be parsed out into an anonymous testing pool basically the testers didn't have access to the developer tools or source code you know there is value in blackbox testing even today but it is very limiting when it is the only QA model that is used on a project so embedded QA is essentially a peer-to-peer relationship between quality assurance and development developers it stems from a culture of communication and shared knowledge embedded QA are individuals who have access to and knowledge of the design tools and game code this allows us to build custom tools scripts and

levels to assist both us and developers and testing it allows us to understand systems thoroughly from the backend and not just from the final product that we are delivered it also grants us opportunities to extract data from the game code directly to drive verification testing QA worked directly with developers ideally in the same office if the space allows they are part of the scrum team they attend meetings with them have full access to their backlog and communicate with the developers they support on a regular if not daily basis this is a full working relationship where the expectations of what should be filed how and when is agreed upon by both development and QA this is basically what it looks like this is a very simplistic view in reality we have multiple team leads and a larger team overall an embedded QA team consists of

a blend of permanent highly specialized quality analysts and regular if temporary testers often the testers are specialized as well and are encouraged to form working relationships with the developers they directly support both are encouraged to document and share their knowledge with the rest of the QA team so the analysts will write test scripts for their ownership and they are autonomous units they tasks themselves and the testers that support them their work is driven by developer work as well as their own scheduled regression testing tying everything together we have people like me team leads team leads provide high-level deliverables and performance manage the team they also handle requests for coverage on things that span multiple disciplines such as a demo they drive convergence initiatives across multiple QA disciplines and projects and work to remove any roadblocks of the team a encounter so no

model is ever going to be perfect and the embedded model does offer many advantages but with those do come some additional things you need to keep in mind so discipline specialists our games have a lot of content and systems that can interact with each other in many different ways the best way we found to provide coverage for those was to split our QA analysts into various sub disciplines so you'd have your art analysts your narrative analysts gameplay and level design we do our best to mirror the way the development scrums are structured each area usually has at least one specialist in a couple testers dedicated to it our analysts become knowledge experts that build upon their knowledge of an area across multiple projects they get more proficient in that area with everyone this is an investment that pays off over time because they bring that expertise forward to the next project

they're working on which allows them to identify risks and potential concerns early which can result in savings when developing features so the catch is if you want to build someone to take on this additional responsibility you need their paid to reflect that this these are no longer entry-level positions as q ways not only perceived to be but specialization isn't limited to just our QA analysts becoming knowledge experts empowers individuals and we find that testers that are given an ownership well they take more pride in their work if you have something you own and are personally held accountable for you're more invested you're going to get attached and you're gonna perform better even if you are a temporary tester the return on this for us is that we have people who work harder who pay more attention to the details of their ownership and are more proactive when it comes to communicating about potential issues in their area so the cost of this is time

it takes time for both analysts and testers to become knowledge experts an analyst needs to set up the framework for onboarding new testers into their discipline and realistically it takes about a month for a tester to really become a knowledge expert in their area now the upside of this is of course that our analyst sense testers form closer relationships with the developers which results in better bugs as well as testing those relative and reactive the specialists feel comfortable asking questions of the developers meaning there's very little downtime looking for answers specialists because they know their disciplines so well will also be the first to know if an issue is the result of a feature currently being worked on or an actual bug they will also notice minor changes or subtle bugs because they're so intimately familiar with that specific piece of content that they own understanding a system they will also be better at identifying if an issue is the

result of a larger problem and thus systemic or if it is a one-off issue this often prevents us from filing multiple bugs that are caused by a single issue which would otherwise palete pollute and already busy bug database specialists also identified telemetry and testability requirements early so that they are considered part of the development of a feature and not a costly afterthought another advantage of this model is that QA have full access to the game and the code as well as the editor that is used to build it so with the great power comes great responsibility and we do take that power very seriously kyoya are not adding content they are not changing content that's going to ship the game instead they're building scripts or test levels that allow us to test content or systems in isolation both of these save our development a lot of man-hours over the course of a project as we find some of our test levels are used by developers to test

their content as it comes in we can verify content against the backend data directly from the editor itself let's say you need to check the log levels of an art asset well we have a script that will just extract that information put that into something where we can parse like Excel so we can just do a check like that which takes minutes instead of hours manually opening each asset or testing that asset in a test level using extraction methods like these we can ensure that our test scripts are also more accurate than they ever would if we had to rely on manual updating so what is the cost of this well you need your henna lists and your testers to be somewhat technical they don't need to have a computer science degree but they need to be comfortable working with a game editor even then you need to take the time to teach them how to use your specific editor everyone within QA a

Bioware is trained how to use the editor within their first few weeks if not the first week subjective feedback a subjective feedback is difficult because it is fundamentally opinion based and thus carries with it some form of bias traditionally QA found that their subjective feedback was dismissed with more merit being put on the results of external focused tests rather than internal QA the thing is internal QA are an opportunity for continuous focus testing that the developer can trigger on demand and in-house it can be used to complement regularly scheduled focus tests or use to dig deeper into existing focused test results as a company you are already paying for these QA so why not use them to their fullest potential for QA though the key here is being able to provide subjective feedback that is

actionable say this combat is too hard doesn't help anyone without knowing what the root causes a developer is likely to dismiss this as a one-off and they are absolutely right to do so developers are working on a compressed schedule and they don't have time to investigate one offs however by having a QA analyst analyze the cause they can provide better actionable feedback this combat is too hard because the enemies in this fight tend to do X Y or Z or this combat is too hard if you did this quest first and thus your underleveled this gives the developers a direction to focus on when they're addressing the issue being embedded also means that with the team also means that they have a better understanding of when and what kind of subjective feedback is appropriate at different phases of a project saying that a combat is too hard is useless if a balance pass hasn't been done on the creatures yet and the cost of this is

again time the analysts need to sit down with the developers to establish what the feedback loop is going to look like for both functional and subjective feedback what makes actionable feedback for gameplay is not going to be the same for level design nor are they going to stick to the same schedules finding and fixing a bug early is more cost effective . if it is a systemic issue finding and fixing it early prevents child bugs that pollute the database if it is a Content issue having knowledge experts within QA who understand the content and systems result in a much higher chance of those issues being found earlier this is why we have QA onboard a project as early as pre-production reviewing design documents reviewing first passes of writing QA providing feedback from the earliest steps providing subjective feedback about what isn't working early allows for course correction before the

feature is completed this results in savings of developer hours that are better spent developing new features or achieving stretch goals on existing ones our games tend to be narrative heavy and narrative is one of the hardest things to provide feedback on as it is highly subjective our senior narrative analyst has been working with the narrative team for years to establish a project agnostic framework between the writing team and QA in regards to how when and what we test when it comes to narrative content this was a project that he got buy-in from the narrative team directly and they reviewed and offered feedback on it finally finally signing off on the latest iteration this gave it the stamp of approval across the studio to be used as a standard approach for narrative testing so what you see on screen is a small sample of this framework it is a blueprint for how QA should be testing

narrative content right from their earliest planning documents to the final finished implementation this is important because it allows the narrative team to feel confident in the coverage they will be getting writing can rely that the format they will receive feedback in is appropriate for that specific phase of a project beyond that as part of our narrative QA framework our own created a guide for how to deconstruct feedback so that it is useful for the writers and this is especially critical when providing subjective feedback having QA who are familiar with your content and toolset unlock opportunities for the discovery of efficiencies for example Dragon Age Inquisition shipped with almost 600 unique codex entries these r entries that the players could unlock as they played through the game testing that these could be obtained legally was incredibly difficult there was no reliable documentation that

had been created and these were going through changes all the way up until we shipped so on the AI I manually looks through the editor which is not something you could do as a black box tester found the location of every entry and jot it down where was where was it located and what was required to actually unlock it this took hours and while for that moment you know two months before we ship it was accurate over the following weeks as content changed and moved around it was no longer accurate and so maintaining it was incredibly expensive so during DLC I got to work with one of our technical QA analysts brian has a talent for identifying opportunities not just to automate testing but to automate the acquisition of information when I approached him about my my codex woes he came back to me with a script that parsed the data directly from the editor

giving me a lick list of every codex name the exact location with the codex was placed and what was required to unlock it the first time he looked into how to create this script it took him about four hours after that any script he had to write that was similar in terms of it's looking for a specific type of data and then dumping it into Excel took him half an hour maybe and then to rerun this specific script takes five minutes tops so I could always have up-to-date information at the press of a button well this was huge I could pass this on to a tester and they could verify all of these were accessible in game because they knew exactly where to go and what to do but this is only possible if you have people who not only are they technical but they understand the project enough to know how to apply their knowledge to save you time even the analysts who do not know how to write scripts our position to identify

opportunities for scripting because they have an in-depth level of expertise when it comes to their own specific discipline all right so let's say that you've decided to try adopting an embedded QA model it's critical that given embedded QA relies on a partnership between development and QA there buyin is essential for making it work there is no universal formula I can prescribe it's going to work for every studio but I will instead provide some guidelines and it is up to you to adapt them to your team and your project first off identify the need if you are just starting off small what are the one or two things that are key or large features within a game let's say your game has lots characters that are constantly being iterated on but those characters play in handful levels that are static you'll probably want to focus on a gameplay

analyst over in a levels analyst first reach out to your production and team leads maybe there's a discipline that specifically feels they could benefit from dedicated QA what services or feedback with these teams like to receive that they may not feel they're currently getting are some developers or teams just more QA friendly than others and view QA bugs more objectively great start with them find your champions than development team and work with them second well now that you know what you're trying to address it is time to stand up the framework for what exactly these embedded QA will be doing will they be building test levels and scripts will they attend sprint planning how often will they sync up with it with their developers what will their communication channels look like what reports will they provide to their developers what tools and training will the embedded QA need in order to do their job well and who's going to

provide it ultimately what value will your specific developers gain by collaborating with QA directly run this framework past the discipline lead and production members get their thoughts on it iterate on that feedback and then present it back to them third as I mentioned before this is not going to be an entry-level position as QA often is you want people with experience either technical or show a solid ability to analyze games look don't hire people who are just looking to get their foot in the door as a stepping stone to another department if your HR is selling QA as a as a foot in the door tell them to stop that this is not how we're doing things anymore be selfish look for people who actually want to work in QA and pursue a career in it QA is not a parking lot for developers to be now

lots of our QA have moved on to other departments but every single one of them were exceptional QA long before they moved on to work in other departments they had they had a passion for QA just as much as they did for writing for sinan cinematic design for art those people you definitely want on your team but not the people who sit down in the interview and just say you know what I want to be an audio engineer and there's my foot in the door sorry let's go go look for an audio apprenticeship avoid people who are gonna tell anyone how to do their job if these people can't provide feedback without telling that developer how to fix that issue they're gonna burn bridges and they're gonna burn them fast and nothing is going to hurt your department more invest in the people that you hire give them training continuous training training doesn't stop after your first week and give them

support set up a training plan for ramping testers and analysts alike that is standardized and thorough this provides a baseline that you can rely on for your entire team iterate on that training keep developing it over time based on the feedback and changes that occur within the industry and your projects you want people who are independent and self-motivated people who ask questions and offer suggestions for improvements these people will help your department grow and not just sit there and run the test scripts that they were assigned you want people who will look at a test grip and say you know what I think that humping every wall is just inefficient we can script that those are the people you want it's no secret we work in a digital environment and you want people who have some form of technical knowledge maybe they have a computer science degree but maybe they just know how to reskin mods for for Skyrim that's fine those people are

technical enough for us we can teach you everything else what you don't want is people who struggle to find out the specs of their computer and what you really don't want is people who don't know how to Google how to do that all of this criteria and it is a lot applies for what we look for when we hire our testers as well as our analysts testers are never just a bum in a seat for us these are people we seek out we interview a lot of people and we choose them because they bring something special even if they're only with us temporarily create long-term growth opportunities for your permanent staff this is key for QA growing into a valid career choice if people see that this can be a long-term solution for them they are more likely to stay with you and remain invested and last but not least go slow learn from the challenges you face and iterate upon this not everything will go smoothly that's

that's fine reach out to other QA teams in your company in other companies attend roundtables and summits for QA share knowledge challenges and suggestions we can be a community to help each other grow but wait what about the traditional model am I suggesting that we abandon it entirely no we still need blackbox testers and we still use them we find they add the most value at the end of a project when everything is open for bugs they provide valuable fresh eyes feedback and they are not as susceptible to bug blindness either due to repetition or due to knowing the intended design they are ideal for regression passes on content that is no longer going through iterations or changing game testing becomes a numbers game here at the end of development during finally you want to test as much as possible before you ship and this is where the number of blackbox testers

that you can get really pays off blackbox testers are closer to an entry-level position and thus when it becomes about covering as much content as possible they are going to be a more cost effective solution the analysts and specialized testers assist the blackbox testers helping direct their testing and answer questions of the systems or the content so in conclusion QA needs to continue to grow and identify opportunities to be more than just the final link in the chain this is how we become a respected long-term career choice developers need to start speaking up and asking when they want more than just basic support we have much to gain by striving to work as a unified team developers we are your support class and we are here to provide buffs as needed thank you does anyone

have any questions we've got two mics over here hi I'm a producer on team that's trying to help out QA and we're in the process of potentially sort of transitioning to embedded QA where maybe 25% there and one of the roadblocks that we've specifically hit is that our QA derives a lot of value from being able to sit together and use each other as a resource and that's been one of the main roadblocks and being able to split them out into feature teams do you have any advice or insight on that we have the exact same problem our teams are huge so we found that using slack or any sort of group chat helped a lot a lot of it is also just culture building so we do a lot of team building events with the QA to get them they're still working together during those team-building events but during the rest of time they're sitting with our developers it's gonna take time some people are really

introverted and you have to work with them more and foster that relationship but yeah for the biggest thing that helped us because we work with external embedded QA as well have some kind of unified communication network and for us we've been using slack and we love it and it's helped us a lot does that answer your question yeah thank you hi I'm Andrew Barron I'm with Bohemia Interactive simulations over the last year we've done basically what you were describing at least in our futures group we moved QA into the scrum teams so we do teams of six scrum teams we have to QA in each team I would say that for us it was a massive success but there was definitely obstacles to doing that a lot of internal resistance to moving them out of the sort of black box role and into that role I was wondering what you found as the biggest obstacles in your rollout and how you overcame them so full confession we've done embedded QA for probably 15 years

organically so internally it's just the way things are at Bioware when we work with external teams to try and get them embedded a lot of it is just the testers are hesitant to come up to developers and work with developers that was our biggest roadblock was that they felt inferior in a way they would apologize that they had to go up to the developer like apologized 10 or 11 times which even for Canadian is excessive we had to work with those individuals and try and say you know what you are a peer and build up that mentality so that was our biggest challenge just break that that that wall and create that feeling like no we are all equals and there is no hierarchy basically I'd be really interested to know more specifics about what were some of the challenges that you faced but we are running low on time so I want to catch up with you later cuz I'm curious cool thank you see anyone else

haha wonderful you've been mentioning at the beginning that you were providing all the source code and all the tools and the training for your q ways but preventing them from doing any changes I've been kind of doing the exact opposite I mean I provide them everything they want but if they find a minor bug they're in collaboration with the developers and say hey I can fix this no need to raise the bug just fix it so I don't know I don't know what I have six people to so it's probably easy to manage but wondering what your um we have had people do that in the past and I mean that was the general rule of thumb the challenge with that is you're actually okay if I have my QA fix a bug they're actually creating twice as much work one the time they're taking to fix that bug they are not testing to someone else no needs to verify their work this is a discussion we've had going back and forth with the various teams and I'll be honest with you if it works for your team awesome go for it we've done in the past but it's it's

targeted and deliberate I would not recommend you know if you're just starting it to do it because I mean there's it's also that level of accountability that's a lot of pressure to put on a QA department and to be blunt I've seen other companies use their QA as in this case those artists they were basically using the MS junior artists but paying them QA wages which is unfair and honestly not the best use of reasons I don't think that's what you're doing but I've seen another company do that and it was you know it ended up collapsing on them in the end it's not great so okay do it but do deliberately anyone else well you guys have been a wonderful audience thank you very much [Applause]