Don't You Dare Say Showstopper: Bootstrapping Your Own QA Team

and get started so hi everybody my name is jesse vigil i am faculty at uh usc games i wear a couple other hats as well and i will talk about uh some of those sort of as we go but this is yes um partially about a qa team that we by necessity had to put together and also in general more about how to build a culture of assured quality as i like to call it so even though this is the education summit there might be other people in the room too and that's cool because this honestly can work for anybody who is small starting out thinking that maybe they can't afford full qa or don't have the resources to do it sort of properly and i say this because our games are increasingly public-facing and i'm not talking just about usc games although um i mean even just recently we were in the failing new york times

so uh it's just everybody in general uh the rock stars that uh we teach who are like 20 21 years old are increasingly not waiting even until they graduate before they are starting to take their place in the professional community and that means that to a degree like since more and more people from the outside world are putting hands on things we've realized that we have to start sort of changing to reflect uh this reality uh i spend a lot of my time and sort of my focus in a lot of the teaching that i do is i think basically on straddling the difference between finished and done which tends to work in both directions i end up with teams that say well i implemented my last feature so i guess i'm done uh or also the reverse who are the people who say i know that like i will never be finished art is never finished uh only only ever shipped so we've been making a lot of changes sort of as we go um even just

in our intermediate uh class we build in more time for dedicated qa in the advanced game project of which i am the production faculty there's now a dedicated qa phase and a proper beta that is not two nights before the show we've done away with a mid-winter sort of progress show in favor of things that are actually fully done because we are just brimming with so much uh stuff that needs and needs eyes on it that uh we're changing the focus and sean and i teach a polish and published class that is entirely about taking things that were made in other classes and spending another 15 weeks polishing them up and making them public facing but sort of the biggest thing that we started doing was usc games publishing which is the first digital press university press more or less we're a publisher on playstation we're a publisher on xbox we're a publisher and a few other people i can't say just yet uh but this is this is sort of this is

sort of the big time and we've announced our initial sort of slate but the debut title for the publishing label was going to be a student game that started as a jam and then had a lot of festival success one of bafta did more festivals and then pitched more or less to spend another 16 months as an advanced game project uh with the express purpose of becoming public facing and arriving on a console at the end and it went very well so well in fact that at the end of their show we very confidently announced the date that we had chosen and cleared with the platform which is july was until july 26th and then i realized that i was going to be one of the people who was responsible for that and i had like kind of a minor sort of freak out about it because consoles have a lot of rules uh like a lot of rules and this this this is a generic stock photo because we're not allowed to show photos of the binders that uh the consoles give but i'm guessing that there are a lot of people in this room

who have seen the real binders and know that this is almost not an exaggeration every console has different requirements they're each vague and confusing in their own ways and they kind of require a specialized skill set in order to be certified properly work the way that they're all supposed to that's the inherent promise behind a console large publishers maintain dedicated teams devoted to just getting through the technical requirements um in part because you know they can in a good business but also because you can incur severe financial penalties and we are talking about like low five figures every time you screw up after your free try which you know that's what publisher is somebody's going to get yelled at into a university it's like how much uh and i know all of these things because my first job in games was actually in qa so i started at the vendi universal games i guess younger and thinner that's actually me on half-life 2 during the only time we were allowed to work on half-life 2 which is when there was no one else in the building and i went i started in qa and i moved into requirements and eventually uh production so that was sort of always

like known when we were mentoring this project and we knew that if we were going to be serious about this chimbara was basically going to need a proper like team of like a crack team of commandos more or less to uh qa the hell out of this thing and make sure that it was going to get there but we didn't really have that and most small teams don't have that at the scale that is probably necessary especially when the the penalties for being wrong are pretty severe so how we decided to come about this was to solve the problem like producers and then execute it like educators we we knew by necessity this team was going to be made up mostly of passionate volunteers who at most we could throw some hourly student worker money at but for the most part they were going to be there because they wanted the experience because they wanted to support their friends because they were excited about doing this brand new thing with us so there's a lot of formal training normally to get to sort of requirements level

uh qa knowledge we didn't have that but on the other hand one thing we had that we didn't have even when i was being trained at vivendi was theoretically we're pretty good teachers and so we did what we do best is we built a crash course at a curriculum really quickly based on all the things that we know work about how we teach people and then try to teach them some of these things as quickly as possible but it's important to just sort of know that it doesn't really matter whether they're students or anything else as far as i'm concerned because one of the things that when i came to this qa everybody sort of starts as a passionate amateur for a long time and this has changed a little bit but it's still basically true that qa is full of people who love video games and want to learn more about how to make them and want to work in any capacity that they can so it's all sort of about onboarding more or less this is what our onboarding sort of looks like it was a casual meeting of friends more than anything super formal and uh the sort of early important soapbox lesson that i'm going to give

about this is that our entire industry tends to mock and downplay qa to some degree and i kind of hate this because it shapes attitudes about the importance of delivering a quality bug free experience when we treat qa like they're not part of the development team we train developers to think qa is not part of the development process so i tend to try to frame qa and i did with these uh with with this team as the last line of defense more or less the only people in a critical capacity who are going to be on the side of the game so while it sucks to get feedback that you don't want to hear sometimes it this is the last time that you're going to get it from people who are on your side and part of the team so i feel really strongly about this and i don't put up with that mockery sort of professionally and definitely not in my classroom and i'm happy to say that we had a really positive overall culture which formed the seeds of things to follow so in the crash course things we did we explained first of all

that there are different kinds of testing and while we had spent a lot of time as educators talking about the importance of play testing talking about usability testing sometimes they only ever heard the testing part of it so when we would whenever we talk to any of our students and say hey have you like really like tested this thing for bugs like oh yeah i tested it but they they just play tested it some more i even got an email from uh some producers just two days ago who say they were pl who say they were qa testing but it turns out they were play testing and they found no new bugs so we spent a lot of time just sort of very carefully uh laying out all the different kinds and what each one of those things sort of um what they do more or less in order to make it better and the best thing that we did was frame it out of the context of game development so this actually worked extremely well i told them all right think about it instead of like when somebody finds a bug a crime has been committed and your job is not to prosecute the guilty and your job is not even to

understand all of the whys you guys are the csi team more or less you arrive at the scene and your job is to gather as much information and evidence as humanly possible to turn over to the developers who will be able to do the wise be the detectives more or less and make the fix and then leave it up to production to prosecute the guilty um uh and that worked really really well one thing that i was trying not to do however was introduce a lot of complicated jargon more or less but then it ended up slipping out and when i brought in some industry mentors more or less old habits flared back up and pretty soon we were having our own conversation about well that's a dupe or can you repro that i should write a suite for that that's not a nab which is slang for not a bug but it turned out that using that slang was an unexpectedly really great thing first of all it professionalized them it made them feel qualified and vital members of the team just like any time you give any group of people a special secret code to talk in automatically you

feel cool and important um and with some kind of specialized skill set and so it felt more like being part of an elite team and less like doing menial grunt work and it also i think prepared them to interact professionally the students were on sony devnet they were on id at xbox asking questions of other seasoned developers and our partners directly and because they had those words i think it definitely helped make people feel a little bit more charitable towards like ushering along some all of us noobs more or less it also in hindsight i thought about what does it mean like why does jargon exist and part of it is because it's clear communication so there's never any misunderstanding about a vagary of a word more or less so because we didn't have a lot of time and we didn't have a lot of formality the communications between the qa team and the developers were very smooth another good thing that we did was i freaked out really early on and i called my old buddy from qa so in 2003 that's him on the

left and also there he is in 2016. he's tom doesn't look that different uh neither of these photos are at work either it's just like him at a restaurant and him in a kitchen we could never take any photos at work this is what we have but yeah so tom is now the head of konami qa and so i called him up and i said i we're going on console and tom said when i said july and he was no that's not not young or not uh and i said no we seriously are we told the world and so tom very graciously helped out and reminded me because i had not done a console in this generation my last one was ps3 so tom and i wrote basically a course reader but a manual based on what we remembered from our old vivendi training manuals what worked and what didn't and a lot of sort of practical knowledge that tom picked up when he trains new hires at konami the thing that we did was these manuals are also just as thick and dense as the ones that were responsible for servicing with the consoles and tom and i both had a moment where we were like oh yeah i didn't read that very carefully either so we stripped a lot of

that out and uh kept all the high level philosophy more or less and pulled a lot of the technical stuff onto sort of a case by case oh now we need to know that let's put that in there so that we don't overwhelm people like here's some examples of some of the stuff that were in the book we talked about how to play sub-optimally the goal is no longer to win the game or beat the level your goal is to find everything there is to know about the level to know it better be a cartographer have you seen every corner of the game have you fallen in every pit have you climbed in every wall i told a story about when i was on lord of the rings online and i walked into every tree in murkwood one week uh play counter to the designer's intentions if the designers want you to be able to unlock a door with a key then your job today qa is to get through that door without a key figure it out do it break the game that's your job we also built some sort of like rapid knowledge dump neo the matrix level uh just pour as much as we could into their heads power points these are some of the best slides this

by far is the most effective slide in that entire talk uh i stress over and over that developers are poor qa testers unless they have superhuman discipline because they know how their game is supposed to be played and they on some subconscious level always want it to work and will avoid things that their brain even subconsciously knows doesn't work um this is also true of their friends and peers who are often good at games and understand what a game is trying to do but i remind them that the majority of their end users are not those people and they are the ones who will find a problem they are the ones who will be unhappy and they are the ones that ultimately you are trying to service so it might be slightly hyperbolic but it really drove the point home and now that i have a toddler who grabbed my controller i actually found like two super cool bugs recently so maybe it's not even that hyperbolic we did go over some formal stuff so that we could run as much like a professional team as possible this also helped us when we were talking to mentors so we wrote a bug just the way

that you would professionally with all the pieces we did not invest at the time in a formal bug tracking system although i think that's sort of an improve on my part we learned i think the upper limits of google foo during this process but since we were a new publisher and we hadn't really decided on all of our techs uh just yet this was a a pretty good trick but one thing that we did was we built a lot of on-rails experiences for inexperienced qa people which are very similar to professional qa test suites but with uh with a few extra sort of interactive tricks do this did it work yes do this then did it work yes do this that did not work okay stop everything write a bug right now here's the form uh now you may click here to go back and keep going uh after a while we didn't have to do that too long before they just started using the forms on their own and everything was cool we also tried to make some games out of it when we did i invented a powerpoint game that's like a game show called name that severity where we all yell we look at bugs and then we all yell

what we how how severe we think they are on a scale of one or a to c or three uh like a minor graphical kind of issue i pulled things that i found personally or other like really great sort of things and we sort of talked through each of them and even like some of the edge cases too judgment calls and emphasizing the severity because it becomes a key part of triaging so bugs like this and we were all firewatch was new at the time we were all playing firewatch so a lot of our bugs were about firewatch um these sorts of things where yeah technically the game didn't crash but the rope is attached like basically perpetually at this point it definitely makes things complicated and later in the game you can't do anything so it's even though it doesn't seem like a crash that's as bad as a crash and this is important because coming to terms with triage keeping the notion that triage is coming early at the foreground kept everything constructive i think on some level we knew that we were going to let some things go at the end of the day but reminding ourselves that we were

going to be having those conversations early helped when it came to well how many hours did we spend fixing a c bug when there were 12 abugs still in the system how many hours does qa have left how can they best be deployed and make sure that they're using their time well also because sometimes again firewatch it was helpful to explain enhancements also which are we i i professionally we always use d for enhancement uh things that are working the way that the developers intend to work but maybe qa has a different opinion of a way things could work better sometimes inexperienced teams flood the bug system with feedback instead of bugs so we spent a while talking about the difference between feedback and bug hunting and making sure that those were in the the proper proportion and all of this worked out really well because i've been hinting at this a few times the most important thing we did was the culture the overall sort of environment that we

were building ended up being the sort of lasting legacy more than any of the hard and fast like wrote rules that we ended up teaching the relationship dynamics turned out to be the real factor in that team's like stunning effectiveness the qa students became better collaborators after the summer because of the sort of team-based nature of being in the trenches in that tiny room that didn't even have air conditioning sometimes and there was also no like substitute for practice so the more time that they spent with games they even started testing other games as hard as they were testing the one that they were being paid to test and it really sort of built them into people who thought critically but were able to communicate constructively and that ended up being a meta conversation that wasn't formalized or anything about but i'm already sort of seeing a vague sort of culture shift but the one thing that i do have to say that we were not able to streamline is i think the best on a university

level the best equivalent is sort of dedicated lab time but we learned that they were minimally effective um [Music] all of these lessons were not effective when we introduced them as part of like a sub class more or less only the dedicated time and large blocks of two to four hours to actually do the testing was where we started to see some real results i think there's something about just that amount of time and that closeness with the material that really starts to sort of cement some of the broader cultural messages other unexpected result to use a bug term we ended up with more mindful developers veterans of that summer crash course team have become more thoughtful and far thinking in their practice as developers they recognize bugs in their own work and know better practices for how to solve them they give better feedback to their fellow designers and they manage tasks and deadlines better and i will say that the chimbara developers became more thoughtful and professional for putting their game through a formal qa

process i think they have all become a lot more respectful of the process better taking feedback and more deliberate in some of the decisions that they make so then we shipped it and here we are at e3 uh which is the proudest i've been of any title i've ever taken to e3 and that is my 19 year old sony requirements lead with the lead designer in this photo it was a really happy day we still had another month of development um and fixes ahead of us when this photo was taken but um it was a struggle i'll say that it was not easy without about three vital students who really threw themselves into requirements and two dedicated faculty acting as producers we would not have made our release date and navigating all those procedures and all the rules and things would not have been possible without advisement from tom and some of our partners at sony and microsoft i'm going to say thank you guys so much just in case you're ever watching on the fault and it's worth mentioning that before

entering formal qa the developers had worked on this thing for about 18 months prior to us getting involved there's a lot of work into this this wasn't just us deciding oh i think it's time to put a game on a console in hindsight i think there was a point maybe late june early july where we would say why did we decide to go on a console this is the worst idea we've ever had the constraints ended up being super useful though and if i had it to do all over again i would do it exactly the same way even though there was a little bit of heartbreak along the way even just the differences between working on playstation and xbox which is coming highlighted some and i know there's at least one developer in the room because i just saw him lazy and sub-optimal engineering there was an over-reliance on plugins that finally had a real world consequence that became i think a really powerful lesson when it almost prevented submission from going through and it was nice to have an external keeper of standards we can say all these things but it was nice to have like big bad sony and microsoft say also

saying no it will absolutely not go on the console unless these things are adhered to that i think finally drove home some really important lessons at the end of the day also the entire team has like a really vital hirable experience now so i wouldn't trade that either if you are going to go down this road and undertake some of this insanity here are the best things that we learned so far make it part of your practice in every class even if it's just a mention like i said i feel it's important to talk about this as part of the process and that we're not done when we get right up to the point where we're fixing bugs we have in fact just sort of begun having a dedicated class or a summer workshop was a big leap forward that time together to focus on nothing but this paid off in space i don't know if we're going to have to do that every single time to be perfectly honest even if we get new people because the seeds of what we learned and how how we conduct ourselves and how we

think about things and help each other just as sort of a large you know developer more or less or a publisher across the entire university program uh have have already sort of been sewn and we're already started to see that the the student teams that are working on things for this year or for even the year after that are are internalizing a lot of these lessons and applying them themselves when they in turn become leaders and set up their own teams and their own cultures i think it works best with practical real software and goals rather than talking about this entirely in theory like i'm doing right now works a hundred percent better when we have foolishly told the world that something is going to happen on a specific date and also it's really really important to celebrate those successes i i've mentioned this at the beginning but making qa feel like part of the whole team and also a vital and special part of it made a huge difference i think in terms of those critical points since we couldn't necessarily even pay

overtime like a traditional publisher will dump on on qa to stay all night or do all those sorts of things really being part of the family and being part of the team and celebrating them when they did something right was what got them got the developers through at the end of the day giving them lots of scaffolding at the beginning is super good writing those guides giving examples writing sample test suites doing a couple of bug reports at the beginning for an inexperienced team so that they have something to follow goes a long way they'll pick it up pretty quickly and then this one anecdotes are key get them share them make them a core part of your teaching tom and i at the beginning were worried that we were telling too many war stories and just funny stories about some of the things that happened to us in qa but it turned out to be those are the lessons that people remembered and stuck with and did something with moving forward so if you don't have some of your own

but you probably do get a few more and build that into it the the sort of core teaching i think is that there's a big difference between hiring bug hunters and building sort of an overall culture we did not set out to do that but when we figured out what was happening i we leaned we leaned on the till pretty hard to get to that direction and it's been paying sort of lasting dividends uh chimbara in fact passed sony fqa on the first try with zero issues there's normally a tab that has a list for issues but that tab doesn't appear apparently if there are no issues which i'd never seen before we thought that there was a bug in that that we had we were going to get that report later and they were like no stop asking questions it's fine there's nothing wrong with it so that was a huge win tom and i high-fived over that one note the release on the tsn store july 26th exactly the day that we said we feel really proud of that but also the veterans on this year's teams are turning out significantly more stable and complete software at the

halfway mark than previous years i feel like it is partially due to example and partially due to training and there's just this like i said this culture assured quality experience for players that is starting to spread and we're sort of seeing who we're seeing this in the things that are stepped up to take that next spot uh this entire talk was about 14 years in the making crossing those uh photographs we're still learning things um i hope this was helpful um thanks a whole a whole bunch uh and i can take a question i think or two before i have to wander towards the uh the room or the place outside where i will keep talking if anybody's got anything you