Sounds as Intended: Quality Assurance in Game Audio

[Music] first thing I want to do is say a handful of thank you I'll make it quick um thank you Damen for mentoring this talk thank you so much um thank you to Brian Schmidt and games sound con for allowing me to give an earlier iteration of this talk at games sound con last year um and massive massive thank you to both the audio and QA teams on Diablo 4 for listening to the absolutely atrocious first version of this talk and giving me all the feedback I needed to make it good for all of you all right let's go so who am I why am I here talking to you about audio QA um I am a composer and sound designer in my own rate uh Amanda la music and sound I am currently the audio QA subject matter expert on Diablo 4 at blizard entertainment specifically blizzard Albany which used to be vicarious Visions uh prior to that I was the audio test lead for Tony Hawk

Pro Skater 1 and two at vicarious Visions I have an MFA and music composition from the Vermont College of Fine Arts I'm a cat lover and a yarn connoisseur you can see my my two cats there realia and overon they are vital to the game development process uh my co-workers get a little bit disappointed if they don't show up to our our morning meetings uh a couple things I didn't put on here I am also the chair of the Albany igda and I am also one of the organizing members of gwa Albany which is our QA Union at blizzard ALB what is QA this apparently has been a Hot Topic today uh so QA stands for quality assurance and it is what it sounds like we want to assure the quality oh no okay all right we want to assure the quality of the game what does that mean we want to make sure that everything that has been designed is working the way it's supposed to and isn't working

in any of the ways it's not supposed to right and how do we do that we gather information we ask questions we author documentation to guide testing we find and Report defects also known as bugs and then once bugs have been fixed we go back and verify those fixes to make sure that they haven't broken anything else um so this the image that I have here is uh a screenshot of the playlist from Tony Hawks Pro Skater 1 and two and just a little bit about this it works like an MP3 player on shuffle so things that we needed to test for it make sure that it's random make sure that no songs repeat until every song in the playlist has been played make sure that you can check and uncheck songs so that you can add and remove them from the playlist uh some other things we had to test the game had a t rating that means that we needed to make sure there was no language in the lyrics that was going to threaten that rating so that meant listening through making sure everything was properly bleeped properly censored right uh in order to do that we created an audio test spec a glossery of terms test pass a custom bug tracker for UI

including instructions for testing uh and so on I apologize my notes are on my phone because we couldn't make the computer work all right moving on common misconceptions about QA what is not QA there's a very common misconception that QA and play testing are the same thing this is not true uh while it is true that we organize and participate in play tests uh we being QA testers it is not far from the bulk of our job it's it's kind of more of a rare thing to be honest with you uh we are not breaking the game we are merely exposing the ways in which it is already broken uh we're also not fixing the game we we we are not fixing the game we're just telling you how it's broken and then it's up to y'all to figure out how to fix it uh it's not getting paid to play games all day a lot of the time we're not even really playing the game not in a natural way that that that a

human being would do it uh often we're actually testing in the engine or we're looking at documentation uh a lot of time I spend test planning uh very rarely am I actually in the game playing the way a player would we cannot find every bug there are going to be things that are only exposed in really unique circumstances there are going to be situations where you need thousands upon thousands of players in order to expose a bug and you just you're not going to have a QA team that big so there're going to be thing every game ships with bugs and not every bug that we find is going to get fixed and I have a couple little funny stories about that that's that's why there's lobsters on the screen right now so Diablo I resurrected I didn't actually really work on it I did a little bit of testing support but there was just the absolute funniest bug that someone wrote uh one of the testers had actually grown up in the Caribbean and knew the difference between a warm water and a cold water lobster and wrote a bug

that there were cold water lobsters being sold at a fishmonger in an area of the game that would have been warm water area it's the wrong kind of lobster that didn't get fixed didn't get fixed uh and then another one down here is something that also didn't get fixed in Tony Haw post skat 102 this is actually a bug I wrote myself and since this screenshot is just from Google uh you know that it didn't get fixed so if you look at that little uh con rete thing to the right of the screen there and you see how it's sort of turned to the concrete does not do that it doesn't work that way it'll bre it'll break it'll crack but it's not just going to get picked up like floor tiles do that concrete doesn't do that I know this because my dad does concrete for a living that didn't get fixed either as you can see so audio QA and by that I mean a QA tester or testers who are specifically assigned to work on audio testing why do you want that specialization allows for

expertise when you have someone who is specifically trained in your audio tools and in your engine you're going to get more and higher quality bug reports there's that doesn't mean that there's more bugs in your game that means that more of them are being exposed so that you can make more educated decisions about what you want to fix uh you'll get fewer dupes nmis wfs nmis meaning needs more information which is when you get a bug report that is incomplete and does not give you what you need to actually fix the bug and you have to send it back and wait for the tester to go and add more information to it wnf being a will not fix uh you don't want them writing bugs for things that are actually intended right a well-trained audio QA who understands your system and your design will also anticipate potential points of failure so one way that I did this when I was working on Tony Hawks Pro Skater one and two is that I had found an area very early in development I had found an area that when you skated on it the skate Loop did not play it just went

silent and I asked the audio team about it and this was pre- pandemic so one of the sound designers came to my desk and showed me in unreal the exact settings on that actor that were incorrect that were causing the loop not to play and since I then had that information I was able to figure out that that was probably going to happen in a lot of other places so because it was early enough in development that I had a lot of I had a lot of free time I went through every single level in the engine and I pulled up every single actor and I checked the Collision settings for all of them and I made a massive document per level with every actor that had incorrect Collision settings and I sent that off in bug reports and that got fixed so early that by the time the game shipped we really didn't have that problem anymore um that also falls into understanding where the bug needs to go because that was actually a level design bug it wasn't an audio bug and because that had been explained to me I was able to send it right to the level design team and the audio a team didn't have to waste time looking at it and

figuring it out that it wasn't up to them to fix so a well-trained audio QA tester will also provide additional information if they understand your middleware and your engine they will give you specific actor names specific file names specific wise events they will capture wise profiling sessions they will give you more information to make it easier and faster to fix your bugs uh audio QA can also gather information and perform tasks for you if they know what they're doing uh when I worked on Tony Hawks Pro Skater 1 and two I actually helped with some of the vo recording all of the exertion vo was recorded with the exception of Tony Hawk and Jack Black all of the other skaters exertions were recorded in House at vicarious visions and I was able to help set that up and and uh facilitate those sessions what are the relationships that your audio QA tester has with different members of your audio department well when it comes to design we need to know what your VIs and goals are we need to know what the theme of

the game is what is the narrative what is the mood so that when we hear a sound that isn't conveying that like Tony Hawk sounds a lot different than Diablo 4 right you don't want them to sound the same so we need to know what your vision and what your goals are when it comes to implementation we need to understand your systems and your structure how do these work how do these sounds work how are things set up and to be honest with you uh just the way that your middleware just the way that your project is set up and wise can inform our test passes a lot I've actually directly used the wise project for games to inform the test passes that I've been writing to check functionality when it comes to engineering we need to understand how the audio relates to other features how it's set up is the sound on the VFX and then when the VFX doesn't play the sound's not going to play and then we understand that that's not an audio bug it's a VFX bug so we're not wasting audio's time with that again when it comes to production we need to understand the timeline and the state that the work is in you don't want a

million UI bugs for missing audio when you haven't actually started to create your UI audio yet and when it comes to other QA testers other who are testing other features there is significant overlap in testing you really don't want to have your audio testers go through and retest features that have already been checked for functionality and art and so on just listening to the audio it it it's very redundant makes a lot more sense if you can coordinate that testing so that everything's getting done at the same time What do we do all day what does QA actually do so now we understand the goal of QA but what does the average day in a tester's life look like so often we'll start the day with either a smoke test or a build verification test the the terms are interchangeable really uh and what that means is we'll get a nightly build and we want to make sure that all the core functionality is available in that build we want to make sure you can get into the game want to make sure you can select a character want to make sure you can move around and so on and these

tests will vary based on the project but you want to make sure the core functionality is there so that the build is actually viable for other play tests that other teams may be doing and further testing if it's not you need to roll back to a previous build we'll do a lot of targeted ad hoc uh ad hoc just means unguided testing means you're not following a test pass you're just uh going through and playing the game sometimes the way a player would sometimes the way a player wouldn't uh you're trying to access areas that shouldn't have access to trying to perform abilities that you shouldn't be able to perform trying to use things that you shouldn't be able to use do things out of order see what happens when you do stuff that wasn't intended in the design right because often that you'll find things that are broken the first bug I ever found when I was working on Tony Haw Pro Skater one and two I actually was able to there was an issue I think with the velocity of the skate and I was able to get way higher than I should have but the bug I actually wrote was that I was able to jump up on top of a building and actually fall right through it cuz I had no Collision so that the first bug I ever

wrote uh we create and maintain documentation this is documentation relating to test but not always sometimes the documentation that we create can be very very helpful to other departments um I created a bespoke bug tracker uh for UI audio bugs on Tony Hawks Pro Skater 1 and2 at the request of one of our sound designers which was very helpful to me to keep track of that work being done but also helpful to him we can also perform Dev requested tests and tasks so if you've just finished work on the music for this part of the game and you want to make sure that it's all transitioning smoothly we can go and do that and we can go do it in ways that you wouldn't want the player to do it and see what happens to the music when we do that so you can get really intense testing on a specific feature uh we can also sometimes do tasks for you so like I said the vo recording thing uh I actually helped out with the batch material tagging for the skate system on Tony hawx Pro Skater 1 and two which saved a lot of time for the sound designers and test planning and

execution uh test planning is the art of actually figuring out what is to be tested how you're going to do it what the prerequisites are what the actual steps for executing the test are what the expected results are uh and execution is actually performing that test writing any bugs that come up and passing or failing the cases and generating reports on the state that the game is in we also perform confirmation and verification so once a bug is marked fixed the that's not the end of the story there's a couple things that we need to do for one we need to make sure the fix actually worked sometimes it doesn't and we also need to make sure that it didn't break anything else because that can also happen so Dev requests what can you ask your QA to do we can gather information for you we can we can do media capture uh we can tell you how to trigger certain events how to get to a certain part of the game how to put the the game in a specific State um I remember on

Tony Haw PR Skater 1 and 2 I was asked hey is there a way to set this up so that my character always spawns at the top of this building and I was able to figure out that you could move the the player start location if you were in the engine you could move it wherever you wanted it and just make sure not to check it in um we write bugs so you can ask us to write bugs if you find something please ask us to write bugs please don't do it yourself because if you haven't been trained in QA you probably don't know what is needed in that bug and what's going to happen is that's going to get sent back to QA and they're not going to know what you're talking about either uh so then they're going to come and pester you so you're much better off just reaching out directly to your QA and saying hey I found this weird thing can you go investigate it and write it up and we'll be happy to do that for you again you can ask us to test specific features or areas we're going to do that anyway but if you've just completed something and you want stress testing on it or you have priority that you want to make sure it gets done uh you can definitely reach out to us anytime time and let us know we can

answer questions for you so QA touches every part of the game right audio touches every part of the game every feature makes a sound that means that QA needs to understand how to test pretty much every feature of the game and to be honest with you we spend a lot of time testing things that aren't audio too so we often know how to how to start that Quest how to get to that location where that character is so if how that debug functionality works right uh so please reach out ask questions we can often get you that information really fast and even if we don't know we usually know who to ask and again helping with Dev tasks like the vo recording and the batch material tagging how do we make this work how do you get the most from your relationship with audio QA help us help you invest in your QA testers create documentation and maintain it invest the time to sit down with your audio QA point of contact and walk them through your wise project or your f mod project walk them through how things are set up

in the engine take the time to answer their questions give them tasks and testing that often helps we are capable of doing things on our own but having that guidance of knowing where you need our attention the most helps us as much as it helps you it helps us make sure that we are focused on what we need to be focused on maintain open communication answer questions ask questions uh invite your QA into your Slack panels invite them to all of the relevant meetings invite them to your Sprint planning and uh do triage bug triage with them invite them to some of your Social meetings create thorough up-to-date documentation create thorough upto-date documentation and maintain it please please please build strong relationships you know build friendships collaboration uh and advocate for your QA it's it's not really a secret that QA

in the industry isn't always treated very well and I think that the misconceptions around it contribute to that the idea that oh you're just you're just getting paid to play games all day no so please advocate for your QA if you don't have full-time QA please advocate for your company creating full-time positions if your QA is not paid well enough if your QA doesn't have benefits if your QA is not allowed to talk to developers um which tends to be more of a publishing QA thing than a developer QA thing so there is a difference between those two and I had it in my notes to tell you but since I don't have my notes I forgot to tell you that so publishing QA comes at a much later stage and doesn't have as much interaction whereas development QA is there from the very beginning and is in constant contact with the developers and contact and memes I think I kind of rushed through this because again I didn't have my notes so sorry uh does anyone have questions [Applause]

hi for the how to write a QA bug reports do you have any resources that could be helpful for proper language or proper orientation for those type of bug reports so they don't get bounced back and forth between Dev and QA so it's going to vary a lot based on your project and the type of information that is needed for your project um our bug reports for Tony Hawk different than our bug reports for Diablo 4 but the core things that you need to have you need to know what build you're on what branch you're on you need Repro steps and you need detailed Repro steps not just go to this place and do this thing but be be specific if you're using a debug tool indicate how to access that tool and how to do it give exact coordinates for where in the game you are uh be as precise as possible whenever you can uh you should also include your expected results of what you thought was going to happen when you encountered this bug as well as your actual results results of what actually happened keywords can also

help to make your bugs more searchable especially on a really large project that has thousands upon thousands of bugs uh media capture is very important if a screenshot is sufficient if it's just an art bug where a screenshot shows the problem in full that's great video is better and any other information that you might need the team that's going to work on it uh and so on awesome thank you yeah hey how's it going good how are you yeah good um I was just wondering if there's ever any um U I don't know if this is like technically bugs but um like kind of mix issues that might come into play like if I've made a um song and there's like certain amount of highend and then this sound designer has made some sound effects and they're all like really harsh as well and those two things are clashing if you would ever be like oh this is it is that a bug or is that a mix issue or who would kind of look at that yeah so if we encounter something and we're not sure if it's intended or not best practice is to get a capture of it a video of it send it to

the design team and say is this intended is this what you want right we're not here to judge your work we're not here to tell you we hate it we're not here to tell you that you're doing it wrong we want to make sure that the vision you're trying to convey is what is actually getting across so often we may run into something where we're not sure what the intended functionality is and the proper practice is to ask great thank you you're welcome thank you for your talk if if you were to hire an audio QA would you look for someone in audio with an interest in QA or someone in QA with an interest of audio or neither well I can tell you how I got hired which is that uh I had more of a background in audio and I was applying for a QA job and that was actually what got me hired for Tony Hawk because I happened to apply at exactly the right time that they were looking for an audio tester so the way that vicarious visions and blizzard hire is not to list a specific audio QA position it's it's just hiring generic QA

but I have helped hire candidates before I've helped interview candidates before and we do look for folks who have specific interests um because that means that they may have more knowledge that can be applicable to testing those areas and they'll have more interest in be paying more attention to those areas so I don't know that I've really answered your question but I'm also not in a hiring capacity so it's a little hard to say but uh I was primarily an audio person who was applying for a QA job if that helps it helps thank you you're welcome um thank you for your talk and I was just curious that if you have any standards for um listening environment for audio C in your team could you clarify a little bit what you mean like um some people can you know listen on headphones or speakers or surround systems and I was just um wondering if

um you guys have some kind of standardization in your um team so most testers will be listening on headphones um particularly if they're in like a big QA room they'll have to be on headphones right you're not going to hear anything but when it comes to whatever formats we're shipping in we want to make sure we're testing in so you will have some kind of a setup whether it's at the studio or like I have a surround setup that was sent to me that I need to finish assembling when I get back home to be able to test on surround so you do want to test in every format that you're shipping in but probably not every tester is going to have access to those formats okay thank you hi perhaps this question has already been um dealt with a little bit but um for those who already have some QA experience maybe outside of audio QA um I have work work experien in like software QA or perhaps they're a game tester QA but not for audio what would you suggest the skills they would want to have besides obviously audio knowledge to say work in what you're doing um and and what would what would

you recommend for them to know I think a knowledge of middleware and knowledge of game engine and again those doesn't have to be the exact middleware or engine that's being used like for example Diablo 4 is in a proprietary engine so there is no knowing that engine before you work on that project for an example but having a base knowledge of other engines it's the same thing is like knowing one Daw you can kind of you can kind of figure out the other ones from there right so you want to have some experience with engine you want to have some experience with middleware and you definitely want to have some experience in QA if you can uh I I I I don't think I can get much more specific than that because every project is so different and every team works so differently thank you you're welcome there it is hi um that was that was a great talking I was uh taking some uh rigorous crazy notes um you mentioned earlier about the uh the way the wise project is being set up um kind of help

can kind of help with that and I was wondering if you have any like uh best practices or suggestions when setting up a wise project as not the QA person but as like the um one of the main sound designers or one of the main people setting up the project are there any best practices when setting that kind of project up so it can be easily read and understood I think it'll be the same practices as uh you know if you're working with other sound designers wanting to set things up in a way that is clear and it's the same thing right so you're you're going to want to train your QA to understand how your project is set up so it's more that you want to educate them on how you've done things rather than trying to tailor what you're doing to them but the more clear and readable and the more sense it makes obviously the easier it'll be so it's more about taking the time to train your QA on how you've set up your project versus tailoring your project to your QA person person ah okay thank you you're

welcome hey Amanda thanks for sharing all the inside um as someone that fixes a lot of audio bugs uh let me tell you how valuable I think two services are that you that you provide one is regression testing to ask you folks to tell me exactly which code change or asset change made the bug appear is dramatically valuable because I can super surgically know that is the Delta that I can investigate and fix this the other thing is that I constantly ask QA is to reproduce the bug in the simplest form ever like if you are on the fifth mission in the third level after leveling like go into the test map do this console command there's your bug that is such a good valuable thing that you provide as well so thanks for that yes definitely that is a really good point sometimes you'll you'll run into a bug really deep into gameplay but but you don't need to get that deep into you don't simplify it right keep the steps as simple as possible make it as

reproducible as possible definitely awesome there any more questions I'm sorry the talk was so short I had other things I was going to talk about but I didn't have my notes thank [Music] you