Tools Tutorial Day: Tools to Reduce Open Bug Count at Media Molecule

Right. Hello. My name is Amy Phillips. I am a tools programmer at Media Molecule. And today I would like to tell you about the tools that we've written to keep the number of open bugs low. Um So. So firstly, why do you want less open bugs? Well, because if you are being stopped from working by bugs, you're not getting on with the fun bit of making games. And we all want to do the kind of creative bit. Um so if we can keep the number of bugs low, uh we can all get on with the cool bit. So, what we've done is connect a bunch of off-the-shelf tools together um by writing a bit of bespoke code, um but using other people's stuff as well, which means that we get the advantages of writing our own stuff, um because we're quite a small team, we don't have the time to write entirely our own stuff.

Uh little bit about Dreams. It is a user-generated content game. It's kind of like Little Big Planet, only much, much, much more so. Um so you can sculpt things. You can then stamp them down to make an environment. You can add a character to that environment, and then you can completely change the character, so you can like re-sculpt your character. You can animate your character. You can animate the environment as well. Um you can add in sound effects. Um you can create your own music. Um so it's a really cool creative sandbox. And everything that we're shipping with Dreams, so we're shipping like a game with Dreams, um everything that we're shipping we're making within Dreams. So it means that our code has to be really stable, otherwise the level designers, the artists, the audio people, can't do any work.

Uh so, Media Molecule, we've got about 50 employees, and in the tools department, there's me, I do 2 days a week. Uh there's Daniel Kidney, he's 5 days a week, and we've recently gained Alex Parker, um who also works full-time. So, we've been sort of 1.4 programmers, and we're now up to 2.4 programmers. Um so, we really want to get lots of bang for our buck in terms of these tools. And also, because Media Molecule is a very flat hierarchy, it means that any tool that we make has to improve people's lives significantly, otherwise they're not going to bother to use it. So, the tools I'm talking about today are in regular use in our studio, so they've proved that they are useful. So, I'm going to show you a video of Dreams. Uh this is our trailer video to give you a better idea of what we're making.

Am I dreaming? Or am I awake? Francis, do you You think there's zombies around here? The old fears they still haunt me. Hold on, Lon's twin, we're coming! They're on a train to nowhere. You are the driver, Lon. When I awake, will I face the worst parts of myself? I need to listen to that quiet voice calling me. That need to make something again. We'll make it. me and you.

Sorry, my poor laptop was really struggling to play that video.

Um right, so what I'm going to do is give you a quick um quick go around the life cycle of a bug, um telling you a little bit about the tools that we've got, and then after that, so you'll know kind of where they fit, and then after that I'll go into a bit more detail. So, if anything doesn't make any sense, then don't worry, it will probably be explained in more detail um later, and if not, there's questions at the end. Um oh, and also, you will be emailed an evaluation form. Please fill it in. Um I was supposed to say that at the beginning. So, we start off, a bug gets written, and it gets uh checked in. So, now it's gone orange because it's now starting to affect programmers. It's not got as far as the artists, the designers, and the audio people, um but it will be stopping programmers from working. So, our first line of defense is Autobot, which is our automated testing system. And I'll tell you a bit

more about how that works, um and what we've done to make the flow of bugs uh smooth and to help QA to put all the information into the bugs from Autobot. Um then I'm going to talk about our next line of defense, which is QA. So, when they find a bug, what have we done to help them to report that bug, make sure all the information is accurate, um and to attach information without having to faff around like copying uh screenshots from like one folder to another. We try and make everything one button click, so it's really simple. You don't get any mistakes, and you don't waste time. Um which then frees QA up to kind of break our game in new and inventive ways, um rather than just kind of going through the boilerplate stuff. Um so, if a bug gets past Autobot, gets past QA, um and then it gets into the build that gets deployed, um and starts

to stop our level designers from working. So, that's where it gets to that red state, and I'm going to tell you about the uh report a bug button that we've added to a Chrome extension, so that level designers can very easily report a bug, and we can make sure that the information they report is accurate. Uh then, when the bug gets assigned out by production, we've already attached all sorts of information to it to make production's life easier, and to make sure that the bug goes to the right person straight away. So, we don't have bugs kind of bouncing around and not being fixed because they haven't reached the right person. So, for example, in a bug we'll have the assert message, we'll have the programmer who added the assert, we'll have the call stack, um we'll have a link to recap which will have a core dump, it'll have the TTY, it'll have a video of the last 60 seconds before the game crashed. Um it'll have the Perforce revision of

the build that the bug was seen on, so you know kind of which build it should repro on. Um so, if we can get as much information in there as possible, it means that production's job is easier when farming the bugs out. Uh then, when the bug gets assigned out to a programmer or a level designer, um I'll talk about the tools that we've written to help them to repro the bug, including my favorite button in Jira, which is in one button press, it will replay the replay on your dev kit, and it will repro the bug for you. So, you don't have to follow through like step-by-step um instructions or anything. It's just one button press, and it repros on your dev kit. Uh then, when you've fixed the bug, um we've got some more tools to help you to check that you've uh fixed it, check that you haven't broken anything else whilst doing the fix. Um then, once you've checked in your fix,

more tools for QA to verify the the fix, and of course it goes back through our automated testing system to check that nothing else got broken on the way. Um and then, job done. You've deployed a new build, um everyone can now work again and create cool stuff. So. Starting off with Autobot, which is our automated testing system. It runs after every check-in. Uh so, you check your code in, Jenkins notices that there's new code, does a build, Autobot notices that there's a new build and runs a suite of tests on that build and it checks for asserts, crashes, hangs, uh checks for out of syncs, uh not a numbers, and some logic errors. So, it runs each test twice um to check for out of syncs. The first time round, it dumps out a bunch of sync points. Second time round, it compares with the

sync points. So, you know immediately if your deterministic game code is not actually deterministic anymore. Um So, it takes about between 10 and 20 minutes to run our smoke test. We found that keeping it under 20 minutes is the the sweet spot between making sure that you get enough coverage, but also making sure that you're not rolling too many revisions into each test. So, you can easily tell which check-in it was that broke something. And also making sure the programmers get immediate feedback on the stuff that they've checked in uh because you want your head still to be in the same space of the the code that you've just checked in um when you go back to fix something if there are issues with it. Uh so, once it runs the test Oh, yes, and it also runs some more in-depth tests overnight cuz we're not checking in as much overnight. Uh we've got some larger suites that test like sculpting

in depth. Um we've got some more detailed out of sync tests. So, if something is flagged as being out of sync in the smoke test, it gets added to the out of sync test, which then dumps out a lot more information about what it was that went out of sync. Um and because it's less busy overnight, we can just It doesn't matter about the whole 20-minute limit. We can just add in loads of tests. Um results are emailed out. So, the email goes to QA, it goes to the tools team, and it goes to anyone who checked something in that was included in that build. Um so you get to check uh whether you've broken something or whether your code has has all passed. Um So, here is the top of the email. Um you can see from the subject line that um I summarize what the differences were because programmers are not going to read all their emails. They're not even going to open all their emails. So, if

you've broken something, you need it to be really obvious from the subject line. Um so you get that. Assert plus one, there's an extra assert. And then when you open the email, there's a more wordy description of what it was you broke. So, in this case, there's a new assert message, but it broke 88 tests um which is like nearly half of the tests. So, you've clearly broken something really quite important and need to fix it. Um and then there's a link to the assert messages section where you can scroll down and get more information. Um so I'm just going to kind of nip through this. You probably can't see it, but I'll uh describe it to you. So, firstly, you've got a new picture. So, you can easily see what the new thing was. I.e. this is something you broke. Uh you get the call stack. You get the assert message. You get the programmer who added the assert. Um you get a list at the bottom of all of the replays that failed this test. Um and included there is the command

line that we you would use in Visual Studio to repro exactly what Autobot ran um to make it easy to repro and fix. And you also get a summary of the last five results. So, you can see color-coded when it broke. And you can also see a list of the changes that went in to that build, who checked them in. And if you click on the link, it takes you to a swarm page which tells you what files got checked in, and it shows you the diffs of those files. Uh so, in this case, it's run out of GPU long memory, um and you would have a look through those three revisions that got checked in and see who was using more of that memory, and then go and nag them to fix it. Um by the new button, uh occasionally there's also a Jira button cuz I look up in Jira to see if

there's an existing bug for this. Um and it looks up based on a cert message and based on call stack, but with the line numbers stripped out because they tend to change. Um and that's a clickable link. So, if there's a Jira bug, then you can click on it, go to the bug in Jira, read a bit more information, fix it, mark it fixed. So, that's the email. Um the results are also viewable via a web UI. So, the email is more useful for programmers. The web UI is more designed for QA to use. And they can filter the results, they can sort the results, they can see kind of what it was that failed about the results. Um but we've added some some interesting buttons to the web UI. So, there's the Jira button that you can see there, the orange and blue one. Um and that, as with the email, it goes off to Jira, looks to see if there

exists a bug. If there does, and the bug is assigned out to a programmer, so it's in the right state, it's going to get fixed hopefully soon, QA don't need to do anything about it, then that Jira button is blue. If the Jira button is orange, like in this one, um that means that the bug is not in the right state, So, either someone's marked it as fixed and it's not fixed or it hasn't been assigned out to a programmer. So, QA need to click on that button, go to Jira, um and make sure that the bug ends up in the right state so that it can then be fixed. If there is no bug in Jira um for the issue, then you get the red bug button. And that brings up the Jira create issue screen. And it pre-fills a whole load of information. Basically, all the information that we've got out of Autobot, we try and pre-fill it into the bug to save QA time and to save them copying and pasting stuff around. So, it

will attach the replay. It will mark which revision it was seen in. It'll put in the assert message. It will insert the programmer who added the assert because in Perforce, you can go and look up who added a particular line of code. Um so, that gives production a good idea of who to start with when they're assigning it out. Um It will also do a link to Recap. So, Recap is a Sony server where you can upload core dumps and you can upload your symbols and it will give you a web interface for browsing those core dumps and you can see the call stack and you can see the TTY and you can upload like the last 60 seconds of video. Um so, we upload all of our crashes and asserts to Recap um and then we put a link in the bug to Recap so that programmers can click on it, um can see very easily the call stack, um and can easily see what

went wrong. Um There is also the launch button. And the launch button takes the replay and launches it on your devkit, on your testkit, on whatever build you're running. So, it doesn't have to be like a a build running from Visual Studio. You could be running a package build and this will still launch that replay um on your build. So, you can have a look at the replay and check that it's doing what you think it's doing. So, one of our early problems with the automated testing system was that we were replaying a replay. And by the way, a replay is an initial state plus a bunch of inputs, which can then be rerun and you get the same thing. Um and inputs would be controller inputs. It'd be like any text that you've input, any inputs into the system.

Um so, you would uh if you were replaying a replay, you would expect to get the same end state as well. Um but, say for example, you had a replay where you were stamping down like five cubes and you were stamping them down using the trigger button, because that was the button at the time. Then, you recorded your replay, Autobot is running it, everything's happy. Then, a bit later, someone changes the button mappings. So, now stamp is not on the trigger anymore, it's on X. Um and your replay is still running, it's not asserting, it's not crashing. Autobot thinks that everything is good, but actually it's not testing what you thought it was testing. It's not stamping down the cubes anymore. So, our first attempt at trying to detect this was dumping out the game state as JSON at the end. Um and then we would do a diff. So, we'd have a control run where someone had watched the replay and said, "Yes, it's doing the right thing." And we would store the JSON game state after that.

And then, for every subsequent run on later builds, we would diff the JSON game state. And so, you would be able to see if it it changed. But, it turns out that that's quite brittle cuz people add things into the game state, they take them out, um things move around, and a diff of adjacent game state is not very useful to a QA person to know whether a replay is doing the right thing. So, our next attempt was screenshots. And you can see up on screen, um you get a screenshot from the end of the control run, and then you get a screenshot from the end of this run that's just run. And a human can very easily look at those two screenshots and see, yes, it did do the same thing. It's put down the same gadgets. Um everything's in the same place. Uh so, this replay has completed successfully. And we don't do like a binary compare of

those screenshots because they are actually different. Like, our sky rendering is not deterministic. Uh the UI for the palette has changed. The thermometer has changed slightly. Um but, to a human eye, you can see that it's done the right thing. And so, it means that QA can just scroll through the results and check by eye that it's doing the right thing, which saves a huge amount of time because we did have one poor guy, Paul Gedden, used to have to press the launch button and watch the replays on a regular basis um to check that they were doing the right thing. Um so, then that frees him up to again do more creative breaking of our game rather than the kind of boilerplate stuff. Um we recently added some performance stuff. Um so, Alex Parker, who joined us recently, has added in some performance tracking, but it's very much kind of work in progress. We're hoping that we'll be able to pick up when we um have spikes in the frame rate, for example, um and be able to see exactly

which revision those came in at. So, how does it all fit together from a tech perspective? We have a MySQL database, which we use to store um which tests we have and which tests run as part of which suite. So, I what replays get run if you run a smoke test, what replays get run if you run like a more in-depth sculpt, that kind of thing. So, that's in the database and we also store the results um of each run in the database. We use a Django web UI on top of that, which then means that QA can add in new replays, they can browse the results, um and it means they don't have to do database queries. It puts it all into a nice clickable web UI for QA. Um so, it means that QA have complete control and I don't have to do anything to maintain it, which is excellent.

So, to run the actual tests, we have some Python scripts. So, Django uses Python, so we wrote the rest in Python so that we could share code. Um we've got a script that watches for new builds um and puts in test requests. We've got another script that grabs the test request, um sends each replay off to a dev kit um with the relevant elf, um gets back the results, it parses the TTY, um it puts the results into the database. And then we have a third script that cleans up after everything else. So, this system generates a huge amount of data um and we don't want to make IT cry, so we have to delete the results after a day or two. Um we delete the screenshots um from the runs after a few days. We have to kind of go in there and clean

things up. Now, all of this used to be on one box. Um and that was when we had sort of five, six dev kits in the wall of dev kits, and it was all kind of ticking along okay until we got up to like nine, 10, 11 dev kits. Um and then it all went horribly wrong because we were using Orbis Control to launch our tests, which is a tool from Sony that takes your test and launches it on a dev kit. Um and that was using 5% of CPU for each test that it was running. So, when you've got like nine, 10 dev kits, that's 50% of your CPU gone. Network Neighborhood took another 30%. Um so, basically the tests were running and we were getting the results back and then we went to put them in the database and we didn't have enough CPU to do the database insert. Um so, we split it up into two boxes. We have um Perceptor, which does the database and the web UI, and then we've got Optimus Prime that runs the scripts. Um and launches them on the dev kits. Um

and gets the results. And that works quite nicely. And we've now got a wall of 27 dev kits. So, it's toasty warm in that area of the office. Um I was going to put a photo in, but apparently I'm not allowed. Uh so, I've talked about that. Ah, yes. Um so, we were using Orbis Control to launch our tests, and that was fine until we came across a really bad hang. Um and by a hang, I mean uh when the game is still running, it's just stuck in, for example, a while one loop on the main thread. So, it's running. Um it's not crashed, it's not asserted, but it's not actually getting anywhere. Um so, we were launching these tests. Um they were hanging, so that was then using up that dev kit. Um we didn't have a timeout because there's no timeout functionality in Orbis Control, and we'd only written an overall timeout. So,

uh you could hang all of the dev kits, and eventually the overall timeout would time out, but then most of your tests hadn't run, um and you didn't get very useful results. So, we replaced Orbis Control with our own PS4 launcher. Um there's a target manager API that we used from Sony, which you can use to launch builds on your dev kit. Um I wrote some timeout functionality in the launcher. Um and this also meant that we could upload our core dumps to recap, which gives us a lot more information about crashes um and hangs, because when uh PS4 launcher detected a hang, it would send a signal to the dev kit saying, "Core dump." And then, once you had the core dump, you could see where it had hung. So, you had a call stack for where your while one loop was that someone had put in the main thread. Um and we got some bonus features as well, in that you can change the

language, um you can change a pro dev kit to behave like a pro or behave like a normal PS4. Um so, that's kind of nice, too. Uh a little bit more about the replay system, which I kind of touched upon earlier. So, our game update is deterministic, which means that if you have an initial game state and a bunch of inputs, um you can rerun them, and you'll get the same result. Um there's a couple of drawbacks we've had. Uh so, the first is serialize revision. If you've got one of these replays, you want to be able to run it on later builds as well. You don't want to just be able to test a single build, which means that when you serialize in your game state, you need to know what you're going to expect to be reading. Um so, if I've added variables, I need to bump serialize revision, so that then uh I will know whether I'm supposed to be reading in that new variable or not. Um but, having this one kind of central serialize revision,

firstly, it's a point of contention for programmers because anyone who's adding or removing stuff from the game state needs to to bump it. And also, if you branch your code, it's a bit of a nightmare because it doesn't deal with uh being in a branch. You can't increment serialize revision in the branch if you're also going to be be incrementing it on trunk because then you've got two revisions that mean different things. Um so, what we do is when we branch to do a green build, we have a very short-lived branch, and if anyone needs to do a fix in the branch that bumps serialize revision, we pull a whole new branch, which is not ideal, but it kind of works for us. Um and the replay system is very important for us for re-proing bugs. So, that's Autobot. On to QA. So, for QA, um I've written a Chrome extension.

And if you've not come across a Chrome extension, it runs in the Chrome browser, and you install it into your Chrome browser. It's written in JavaScript, and it can access REST APIs, so it can query Jira, it can query Recap, um it could can query our game server. Um it gets information about what tabs you've got open in Chrome, and you can change the HTML within those tabs. So, if I detect a Jira tab, I can add extra buttons um into the Jira tab. Um and I did look at doing it as a Jira extension, but I backed quickly away from that, and the the the Chrome extension was much simpler. It can also access your cookies, so we use that to find out what your PSN login is, which then means that I can find your devkit via our server and talk to it. And it can display some notifications on your screen. So, we use the Chrome extension for QA

to add extra buttons to Jira. So, there's one button that attaches a screenshot in one click. So, you press the button um in your Jira web page, it goes off to your devkit, takes a screenshot, and attaches it to the bug that you're currently creating or looking at. So, in one button press, it's done all the work for you, and you don't have to be browsing around to find the screenshot, and it makes sure that you get the right screenshot, as well. We have the same functionality for starting and stopping replays. So, you can start a replay from uh the Jira page, you can repro the bug um on your devkit, and then you can stop the replay, and it will attach the replay to the bug that you're creating. And finally, there's a button for them that will fill in details of um for example, the Perforce revision that you're currently running, um and the game location where you are, so which dream you're in, um whereabouts are you within that dream.

Um and again, with the Perforce revision, because it's like uh five numbers, it's really handy just to be able to click a button and get it filled in without having to kind of try and remember those numbers, make sure they end up in the right order, cuz you tend to get quite a lot of mistakes. Oh, now I've gone the wrong way, sorry. So, on to the designers, the artists, and the audio people. So, we want to make sure that everyone is running the game within the same environment, and that is where dashboard comes in. Dashboard is a tool that we've written in-house, and firstly, when you boot it up, it checks that you've got all the dependencies installed. Um so, if you've like got target manager, remote viewer server, all the dependencies that we use from within the Sony tools, it checks you've got them installed, and it checks that you've got them installed at the right version. Um and if you don't, it gives you a button

that you can click, and it'll run the installer. So, that makes sure that everyone uh in the company has the right software installed, it's up to date, so we don't get any weird bugs because of that. Then, once you've done your dependencies, onto the builds tab, and that gives you a list of the the current builds that we have available on our server. And you can fill in build notes, so QA will fill in build notes, um and you can see kind of what's broken, um and there's a button that will update you to the latest green build. Cuz we want everyone to be on the same build and on latest code. So, we want to make it really easy for them to be on latest code. Um and finally, there's a button to launch that build on your dev kit. And that also checks that you've got the right SDK installed on your dev kit. Um cuz we've had weird bugs in the past, so if you don't have the right SDK, then it will go away, it will install the SDK on your dev kit for you, and then it will

run the build. Um quick note on asserts. So, we have fatal asserts, um and they're included in the build that we give to our level designers and artists. We decided this kind of near the beginning of the project because we wanted to make sure that if you add an assert in your code, um it really is a broken thing. And we want to make sure that people fix them as soon as possible. Um so, if you hit an assert, then it stops, and it's not recoverable, it's not skippable. You have to go and fix it. Um so, what does the Chrome extension do for designers? They have got a report bug button. Um which is the yellow button in the middle there. Um if they click on that button, it will open up a Jira web page in the create issues. Um it will pre-fill what revision they're running because that's not been particularly accurate in the

past. So, it goes off to their devkit, says what revision of Dreams are you running, and fills that in for them. It fills in the game location for them. And they've also got the same buttons that QA have got to attach screenshots and to attach replays. So, it makes it very easy um for a level designer who's having an issue uh to fill in a bug and then send that off to QA. And QA don't have to do very much extra work. They might add in like a repro rate or something. Um and then they assign it out to a programmer. So, very quickly all the relevant information um has been distributed to the programmer who can fix it. There's no Chinese whispers um and you know it's accurate. So, this uh is a picture of the the Chrome extension. And as you go down the screen, it's kind of checking that everything is working properly with the Chrome extension. So, if any of this goes red, then the artist knows that they need to take some

action. And it does explain to them what the action is that they need to take to fix it. So, if you get down to the bottom and you get a screenshot, it means it can talk to your devkit. Everything's working fine. So, we've now got a bug in Jira. We need to route it to the right person. And we've auto-filled lots and lots of information, which is really useful for production to farm the bug out. So, we've put in the assert message, we've put in the programmer who added the assert. And there is a bit of a gotcha here in that we have an assert in our game um for if it goes out of sync. And it's part of a macro, which means that one programmer added this assert. So, it means that any out of sync bugs, it says, "Yeah, Dave added this one."

So, give it to Dave. So, poor Dave ends up with a lot of out of sync bugs. Um and luckily Dave knows the code very well and can kind of farm them out to the right programmer. Um but at some point I need to go in and maybe change it so that it looks up the variable that goes out of sync and see who added that variable, for example, instead. That would be more useful information for tracking down down an out of sync. Um haven't got around to that yet. Um I can also, if anyone wants to talk to me about Perforce or Joy afterwards, I'm not going to go into it here. Um but I've had all sorts of fun and games trying to auth against Perforce. Um so, yes, you've got the assert message, you've got the programmer who added it, got the call stack, you've got the Perforce revision where it went wrong. Uh you've got a replay. Um so, you can reproduce it. Uh you've got a screenshot. Uh you've got a link to recap where you've got the core dump, you've got the TTY, you've got a video of the last 60 seconds of gameplay.

Um so, you should be able to figure out who to give the bug to. So then, the bug is assigned out to a programmer. And there are two more buttons in the Chrome extension. So, there is the launch replay button, which is my favorite button. And And got a video of it here. So, I start off with Dreams booted up um in a debugger on my devkit. Then, when I go to a bug in Jira, which has got a replay attached to it, I get a button at the top. And this button is added by the Chrome extension. And when I hit this button, it the Chrome extension downloads the

replay data from Jira. It sends it via our server onto my devkit. So, my devkit then starts replaying that replay data. And eventually, it hits the assert. And there we go. We've repro'd the assert in the debugger. So, I hit break. I can see what's asserted. I can fix the code. And then, once I fix the code, I can rerun the replay and check my fix. Cool. So, that's my favorite button. It's very cool. Um we also have another button to launch the game location. So, replays are very handy for programmers, but because you've stored the initial game state, they're less useful for content people um because any fixes to the content won't be reflected in the replay. So, we also have this idea of game location, which is like a dream ID and a location within that dream. Um so, launch game

location will launch um an artist into their dream at a particular uh point. So, for example, if QA was saying, "This tree over here is too green." they would attach the game location, and then the artist could launch into it and see the tree that they were talking about um and fix it. And then, when QA go to check the fix, they can launch the game location and see the new less green tree. So, once you've test uh once you've fixed the code, how do you test it? You can use the launch replay button again, and then check that the replay finishes rather than asserting um or crashing. Um you can also submit a build to Autobot. So, we have a script, and it doesn't take any command line parameters cuz I wanted to make it as simple as possible to run, so you can just double click on it. And it takes your local code, builds

a package, uploads that package to Autobot, and then kicks off a smoke test, which will then, when it completes, email you the results. So, if you've made a fix that touches like lots of bits of code, or we all have bits of code in our code base where you know that if you go into that and change like one one fix, you will break three other things. So, if you've been in one of those kind of tangly bits of code, you can kick off an Autobot run before you check in and just check that you're not going to break everything. Um and we also have a script to mark bugs as fixed. So, previously, when you checked in a fix, you would put in your Perforce check-in comment a reference to the bug that you had fixed, and a comment about like what it was that you had changed. Um and then you would go to Jira, and you would go to the bug, and you would

put in what Perforce revision you had just submitted, which again, bit error prone, and you would put in a comment about what you did. So, you're kind of putting the same information in in two places. So, we've just got a script that parses the Perforce check-in comments after every successful build um and if there is a comment of the form with the expletive exclamation mark and then a number, that means I fixed that bug number and it will go off to Jira, check that that bug is in the in progress state. If it is, then it adds a comment to the bug saying, "Amy fixed this bug um in Perforce revision 320. Um this is the check-in comment that she put into Perforce." And then it marks it as fixed for me. So, it it just means that when you're fixing bugs, it takes out a little bit of effort. And if we can take out lots and lots of little

bits of effort then everyone is working much more quickly. Um and we save time overall. Uh right, so you fix the bug, you've checked it in. Now QA need to verify your fix. And you can use the uh Chrome extension Jira button to launch the replay, so QA can launch the replay and check that it doesn't um reproduce. They can add a replay to Autobot. So, if it hadn't been picked up by um any of the Autobot test suites, um then they can think about adding a replay. And they would want to add a a new replay. They wouldn't want to use the replay for the bug because uh you want to keep them really, really short, and you want to make sure that the screenshot at the end shows everything that you have changed um as part of that replay. You want to make sure that the screenshot at the end shows you that the replay did what you thought it did.

Um I would also like to track stuff that gets fixed by Autobot um because if Autobot has um reported a bug and QA have kind of pre-filled that bug in. Um Autobot can also detect that that bug has been fixed and it can add a comment onto the Jira bug so that then when a programmer goes to fix it, they can see that it's already been fixed. Um but I haven't done that yet. Must do that. Maybe next week. Uh oh, no, I've gone the wrong way. Right. So, in summary, uh the tools we have, we have Autobot, which is our automated testing system, um which is a MySQL database, Django web UI on top, and a bunch of Python scripts. Uh we use Jira for our bug tracking, which has got awesome REST API. Um we use Recap for storing our core dumps and TTY and video. They have a REST API, which is improving rapidly. We use deterministic replays and they're

essential for the replaying of our bugs. Um our game server can forward a message um from the website onto a dev kit and then can send the result back to the website, um which is used by a bunch of these tools. And it's all linked together with the Chrome extension, which has kind of got fingers in many, many pies. Um so it's all about trying to make sure that we find the bugs early, um we attach all the information that we need to route them to the right person and to repro them, um and then we want to make it really simple to repro the bug and to mark it fixed. And it's just a bunch of little things trying to make sure that people have the information in front of them at the right time and trying to make sure

trying to make little improvements to the workflow. And all the little improvements kind of add up to a really nice smooth slick workflow and then suddenly you've got less bugs, which is awesome. So, does anyone have any questions? Hi. Um super cool talk. Uh I'm curious about the human validation of screenshots for the smoke test. You mentioned that testing the game stick was too error prone or had too much variation. So, does that mean that for every smoke test that's being run, there is a validation by a QA person for every test? Cuz that seems like a lot of stuff. Yeah, they don't check every one. Um so, I think it's about once a day they will pick a test and will just scroll through. Um and then if they are more likely to fail, I guess. Uh well, no, they would look at all of

the results from a smoke test. So, a smoke test is about 200 tests. Um so, they can very quickly scroll through and just kind of see by eye. Um and yeah, they wouldn't do it for every single one. Thanks. Hi there. I was wondering uh what you do to track which programmer added asserts. Is that some sort of metadata or is that just like the diff history that you look through? Uh there is a command. Um so, the P4 command line um I think it's P4 uh changes. Thank you. P4 changes. Um so, you can run that and you can pass in um like which revision you're interested in and which line, I think. Um Yes, so you can run that and it will tell you who submitted that change. That makes sense. Um so, that's used to route to the programmer. As a follow-up

question, what do you do when programmers leave? Either go on leave or leave the company when you're at those. Uh then you would have I guess the programmer would have handed their code over. So, production should know kind of who they have handed each code area over to. But yes, it would say in the bug this assert was added by Fred who has left. Um so, then production would know who Fred's bugs should go to. Ah, okay. Thanks. Thank you. Hi. Um at what Oh my god. At what point did um Media Molecule decide to have designated um tools programmers? Like how did you manage um like having the benefit of a designated tools programmer as opposed to them working just full-time on the project? Uh I guess as far back as I can remember

we've As you know, we haven't always had a a dedicated tools programmer. Um I can't remember when we hired our first tools programmer. I guess it would be when we got to about sort of 12-ish programmers, although I'm not sure I can remember. Um but actually it's been really, really useful because you write all these tools and then all the other programmers become more productive. And it's not just the programmers, the QA people become more productive. Um everyone gets more stuff done. Um and it's quite hard to kind of put aside time to invest in tools and infrastructure. But actually we found that it's been really useful and it has kind of paid back the time that we've invested. Cool. Thanks. Hey. Um I'm a QA manager and I'm kind of uh interested in like how did you um

implement the process uh at the very beginning uh together maybe together with QA? Like how did that that whole Yeah, implementation kind of work? So you mean how did we implement the process of like the bug flow around? Exactly. Uh and automation. I don't know, maybe it was a process you iterated on back from uh Little Big Planet or uh was this a completely new Uh it's definitely been very iterative. Um so I guess it started off with Daniel Kidney writing Autobot. Um so he kind of plugged together Autobot for us. Um and we've added lots to it and we've kind of connected it up to other things as we went along. But yes, very iteratively in that we kind of added that. Um dashboard was completely separate. We already had dashboard. Um And yeah, we just kind of added things in. Whenever we see a problem with the

workflow, we try and fix it. And then it kind of gradually gets better. But there wasn't some kind of grand plan at the beginning, this is what we're going to do. It just kind of happened. Right. Okay. Thank you. So the last two questions that's my question. I just want to a little clarify uh So this this whole summary of your current tool setup uh it started you basically you started arriving at uh during Media Molecule's earliest projects, is that true? Like the it it didn't just start with your current Dreams project, right? Uh I suppose so dashboard we had for Little Big Planet um and Tearaway. So we've had dashboard for a long time. Um Autobot is new for Dreams. Um we did have some automated testing on Little Big Planet um but it was very error-prone and it was mostly broken. Um, and and we didn't kind of give the resource to kind of

develop it and make it stable and make it useful. Um, I guess we were more concentrated on just kind of shipping the game um, rather than investing in tools. Whereas, like, this time round, we've had a bit more time. Um, so we've been able to write more more cool stuff. And the Chrome extension is completely new. Um, because I was messing around with Chrome extensions in my free free time. Like, I went along to a game jam and was writing a Chrome extension. Um, and then realized quite how powerful it was. The fact that you could, like, take this um, HTML webpage and just insert your own HTML. So, you could just add buttons wherever you wanted, which is really cool. Um, so yeah, that came about just kind of by lucky accident after a game jam. Um, So, yes. Most of it is new. Thank you. Um, oh, and I've just thought of another

tool that we have that I didn't actually mention, but it's really cool. So, we have a tool where you can record video and you can record, um, performance data um, in the form of telemetry and then you can display them together. So, you can uh, see where you've got performance spikes. You can click on the performance spikes in the web UI and it'll take you to the video of where the performance problem was. You can see what was on screen and what the game was doing at that point. Um, which I didn't talk about in this talk cuz it didn't fit. I didn't have time. But, it's very cool. Okay. Sorry for my accent, but, um, uh, you What was the the principal motivation to do your engine deterministic? It was for doing, uh, a replay or to do network and to do some love step and it was interesting Ah, so so you're saying why did we make it deterministic? Yeah, because the origin is not deterministic like most of

the engine. I know I even if I see the benefit of having deterministic replay, I'm not sure if I want to invest in making my engine deterministic only to have deterministic replay. Yes, fair point. It is a lot of work making your engine deterministic and keeping it deterministic. So we use Autobot to check for out of syncs, um but we do regularly find out of syncs and have to fix them. Um I guess we already wanted it to be deterministic um because if someone makes something in Dreams, um we want it to play the same on everybody's PS4. We don't want um people to be able to create levels that will run in multiple different ways cuz then they could create something where you press the button and the door opened and then if it wasn't deterministic, then some people might play it and they press the button and the door doesn't open. Um

and that would not be ideal for players. Um they want to be able to create stuff and know that everyone will see that stuff exactly the same. Um so that's one reason. Also multiplayer. Um it's very handy for multiplayer. Um and yes, for re-proing bugs. And also another quick question, you say that you for assign the bug first, you you you go to a to what to assert, but probably on your company I probably don't want to be the guy that implement that the array out of bound because the I will be annoyed by by uh by uh by everyone after? Have you tried uh over heuristic light like uh who do the last change in the in the in the call stack or whatever it is, even if it's not the assert, it's more on the top of the call stack. Have you tried that or not? Uh so, one thing we do do is

we go back to the email. Oh. So, in the email um you've got an idea of who checked in and broke it. Um so, you can see that there's three revisions there. It's kind of small, but there's three revisions, um and you know it was one of those three that broke it. Um so, you do get extra information there, but no, we've not tried anything other than looking at who added the assert. And yes, it's a good point. Yeah, if you fall off the end of an array, then I don't know who added that assert. Okay, thank you. Uh thank you for the talk. My question was you mentioned that you use Autobot to test for really scary changes. Um have you thought about making Autobot part of a pre-commit or like pre-checking process rather than post? And um I was just wondering if you had tried that out and how it went. Uh we haven't tried it. We did think about it briefly. Um but IT are very nervous about messing

with Perforce. So, they won't even let me add um a commit hook to kind of get the the check-in comment um at the moment because you just don't mess with Perforce. Great, thank you. Hey, um so, it sounds like uh devs can use Autobot for one-off changes before they submit. And do you automatically run Autobot when they submit and they get results? Is that every check-in? And then is that gauntlet? Like, it doesn't go to the main depot before it passes tests or Ah, so you're saying um how does it work in terms of when does Autobot run compared to when you check in? Mhm. And is anyone else protected from me submitting a change that breaks? If I just sync, do I get it right away or does it have to pass Autobot before it actually goes in? Uh so what happens is you check in the change um so it's immediately available and anyone can get latest and will get

your code. Um then uh Genkey puts the build together um of the latest code. Um then Autobot spots the new build and runs the tests and then emails the results. So yes, you do have like a 15-20 minute window where the code has not been tested. We don't really know what state it's in. So you might get latest and get completely broken code. So do you have solutions for that, tools to help with that? You know, it's a constant problem in the studio when people sync and it just doesn't build or it crashes or something. Um we don't. Uh what we do is the person who broke it has to fix it really quickly. That is our our current scheme. Yes, we could think about kind of um I know that some companies have schemes whereby you check into a branch and that branch only gets merged across if it passes a bunch of tests. So we could look at that uh

but we haven't yet. Good. Thank you. Hi. Um so if a member of your QA team uh comes up with some sort of a test case for something that should be covered by automation but isn't, code necessarily isn't broken yet but it should be covered via automation cuz maybe future code's going to break it. How do you have that QA team member um handle that? Like do they write it up as if it were a ticket? Do they contribute to the automation scripts themselves or just curious as to that process. So the tests that get run as uh in Autobot are completely controlled by QA. So, if a QA member came up with a new test, they would record a replay of that test. They would use the web UI to submit that replay to Autobot. Um, Autobot would run that replay once and check, um, that it completes. Then it would be added to the database as a new test, and then that QA member could go in to the web UI again and say, "I want that test to run as part of the smoke

test or as part of the sculpt suite." They get to choose which suites it goes in. And then, every time that suite runs, that test will then run. Thank you. Um, you said that keeping a game deterministic has been difficult for you. Um, could you give any examples of the main problems you see with determinism? And then, are there any tips you have for people that want to make their games deterministic? Uh, initialize your variables. That is, like, I'd say 50% of our out-of-syncs are uninitialized variables. Uh, don't use rand. So, use a random number generator that you seed and use, uh, deterministically if you want a deterministic random number. Um, Pointer values as keys. Ah, yeah, we don't we've not come across that one, but yes, pointer values as keys. Um, Sorry? Fixed timestamp. Fixed timestamp.

Over physics. Yes, I don't know in detail kind of how Anton has made our physics deterministic. So, Anton has kind of written our physics engine, and all I know is it that it simulates stuff as if it was made out of balls. Um, but that is about the limit of my knowledge, I'm afraid. Thanks. Hi. Um, um, my question is about uh integration of Autobot I guess with each game it might be sort of idiosyncratic how to get um the replay working. Um I mean, you can drop uh a user right into a certain location on a map in a game, but to re- recreate the exact game state I I imagine from game to game it's a little bit different even if you have atomic unit tests and asserts in there. I was just wondering if you have any recommendations for best practices uh for kind of writing that in an

abstract enough way that it would work across multiple games. I suppose you need a bunk a bunch of functionality for it to work. So, you need to be able to serialize out your game state in some way. Um you need to be able to then apply inputs on top of that. So, you would need a replay system. Um I'm sorry. I've kind of lost the gist of your question. Sorry. Yeah, I guess it was um I guess I'm just sort of uh wondering uh if there are any recommendations overall for writing like a replay system and writing your code uh abstract enough to do the things like you're saying like serialization of your game states. Uh I guess we've not really structured it particularly around the whole replay system our code. Although, the replay system did go in very early on. Um So,

I mean, we've written some code that makes it easy to serialize out your game state and to take into account serialize revision. So, when you add um a variable to the game state, we have macros that you use to add that variable or to remove that variable, which will automatically do uh uh will generate the code for you that, um, kind of checks against serialized revision. Um, so we do have that. Um, and then, yeah, I mean, the stuff for kind of saving out, the stuff for saving out the inputs just kind of serializes out the inputs. Um, and then the stuff for, uh, serializing out things like, um, text that we've input, um, we have a, a buffer for kind of extra messages that you can serialize into, um, but different subsystems serialize into that buffer in slightly different ways just because different people wrote it.

Um, so we do have quite a variety of code at Media Molecule depending on who wrote it. Quite often you can look at it and and find who wrote it just by looking at what the variables were named. Um, and kind of how it was put together, which is kind of handy. Thanks a lot. Cool. Thank you.