Automated Testing and Instant Replays in Retro City Rampage

[Music] uh so welcome everyone uh make sure to turn off your cell phones put them on silent and if you have a PlayStation vaita or a 3DS with Retro City Rampage DX get that ready start up the game but again turn off your uh your volume there you go so my talk is on using recorded button input and a deterministic engine to do automated testing instant replays and a million other things well a million it's an exaggeration so the recipe for this is simply two pieces so the recorded button input so you record the input that the button uh the button presses and the keys that they press and you have a deterministic engine which means that if you record the buttons here play them back tomorrow they play back exactly the same so this is about half an hour of

the game playing in 15 seconds and if I turned off vsync it would run in about five so that's an example of me using it for automated QA because theoretically you could run it after every single compile so any change you make to the code you can double check and make sure that uh everything is good and nothing's been broken it gives you access to QA 24/7 so I think we've all been in the case where we're pulling an all nighter to submit a milestone or submit some sort of deadline and it saves you from phoning the QA in the middle of the night and finally it allows Hands-On QA to be step two so you can kind of pre-filter things run the automated tests if they pass then you get the QA in there to double check things and it's also incredibly multiplatform so you can see that I ported to all these platforms and all of these platforms work with it

including the previously unannounced Ms Doss Port so it's running here on a Pentium 1 and it would run on a 386 if I still had one I also did another thing which I haven't talked about before so does anyone have a vaita or a 3DS right now okay if you do you can enter this code on the main menu so I started doing this uh around the time I finished the DX version and I started including these automated tests within the game so each platform can play through about 4 to 8 hours of itself um at super speed and it just allowed me to up to the point of submission to certification I can test the build and make sure that it's good so use case 3 was indeed submitting to certification and all of that last year I submitted 15 SKS which included nine console search submissions simultaneously uh without bringing in external QA

because it it really slashes the Redundant testing because you will find in cases with consoles depending on the regions they require a separate uh ROM a separate executable completely uh separately compiled but maybe the only difference is a product code or a network ID or something like that and so it really allows you to really really fully test one and then just automated test the other and then lightly test it um and also with resubmissions I don't know about you guys but on two occasions on two two different console manufacturers I've had to resubmit to certification to change two bytes of data so uh when you have your automated tests it really is just it's not that grueling oh my God I got to test everything again it's just change those two bytes run the automated test and you know as long as you realize it's an isolated change you should be good uh another Super valuable thing a lot of multiplayer games I know do this is catching gameplay bugs so every time the game is run it automatically starts recording and then when you exit or when an assert is hit it dumps that to a file

and so if a bug occurred you just stop the game flip this switch with the dash Auto playback command line argument and it starts playing itself it also saves a file so you can rename that file and say- Playback and then a file name you can say Dash skip to frame so it automatically throttles and goes right to the bug where it occurs by doing this you can trace down the exact frame where it occurs the entity in which it occurs and from there you can see in the example on the bottom there um I've narrowed it down further so I'm adding some printfs there to uh investigate further for the user end it also is pretty cool because it allows you to do user replays so in both cases the user can either save replays themselves to watch later or they can upload them to the leaderboards now the huge huge piece of value here is because all you're doing is storing button input it takes up very very little space we're talking 1 kilobyte 2 kilobyte 8 kilobytes um and so you can save that on on the leaderboards as attachment data that means you don't have to run your own

servers or anything so you can just sit back relax be like a super one-man Indie team and have this going where everyone once a high score is achieved it's uploaded to the leaderboards people will be like how the heck did he accomplish that crazy score they can watch the replay and see for themselves I also used it to control NPCs during some cutcenes and so pathfinding is great from moving from one point to another but if you want something way more Dynamic a cool car chase you can just record the button input for that vehicle uh and in this case three vehicles are running pre-recorded input for a sequence I also had the classic arcade attract mode so if you leave it on the main menu for 5 seconds it'll automatically start playing itself through various missions this was super great for shows like PA and E3 because the game is always just demoing itself even if someone's not playing it and in the Doss version uh it was super valuable to trim memory so once the game is locked it's not really changing you don't have a whole buch bunch of hands in there uh modifying

things the game is pretty stable you can really start to dig deep and so in this case with the Doss version I got it fitting in three three and a half Megs of ram which was pretty crazy uh and fun and uh that meant trimming everything uh trying to remove extra code that the compiler didn't automatically remove to compress the executable down and you name it and so one of the cool tricks here is for example um I have an array of pathfinding nodes well how many nodes do I really need for the pathf finding well I can find that out so you just add a bunch of stuff to track that you run through uh let's say 12 hours of BL repeated playthroughs of the game in a fraction of the time of course and then at the end you have this output of what was the maximum amount of path P pathf finding nodes that I used and then you can just trim it down and that was able to save a lot of memory so other examples of things that you could do with this would be speeding up trailer and video creation I actually use this a lot for trailers there are certain things where if you're playing

the game something cool happens uh you can play it back without the HUD or with different languages um and therefore you can save redundant recording you could also theoretically do asynchronous multiplayer I didn't do this but things like ghost races where you just keep track of uh um all the people's vehicles and play it back at any point you can improve remote QA and play testing which I used as well so even if someone's off site if they're playing your game they can send you the replay and it's like you're over their shoulder you can see exactly what happened and how and you can also generate things like analytics and heat maps you can record all the data from uh play testers or if you have a uh if the game is like three or four days at Pax and E3 you can record the hundreds or thousands of players the advantage here is that if you were to record just plain analytics from that uh you would be stuck if after you analyze things you realize oh I didn't think of something in advance I really wish I had analytics on this you can do that because you just had it stored as button input so you can

analyze it infinitely uh as much as you need so good news uh I whipped together this source file and so basically change one line of code and you should be able to try this out for yourself it's well under 300 lines of code well under 250 without comments uh and you just have at it so the recorded button input there are a few core components for so for example buttons or Keys whether it's pressed or not pressed values such as the analog stick Direction events so you want to keep track of if a mouse is clicked if a if you touch drag or whatnot and debug information which is optional but you definitely do want this during development and optionally if your game isn't fixed frame you're want you're going to want to use time Deltas in there as well RCR specifically I have a header so I store things like the control options the camera zoom the player outfit this allows me to get even more versatile because there are a whole

bunch of different um button configurations depending on the platform and so in certain cases some platforms have more buttons others have less and so in some cases one button will do one of two actions on one system but just one action on the other and in that case I can just keep track of which button configuration was this recorded in and play back with that uh and so it further allows you to play back on other systems the button streams are just one bit per frame whether it's pressed or released as I mentioned the reason I store them separately is because if one button changes you only have to update one button State you don't have to update the entire uh controller State analog stick as well the left and right sticks are separate and event streams so that's basically saying the event what happened was the most pressed was the most released was the did the most position move uh what frame did it occur on and from that point on you just do what you need to do debug information the big thing that I stored was a check

sum of the frame you can calculate that any way you want uh in my case I did certain things like entity positions and uh directions that they're facing and so on and that could be stored in as little as a bite because if the ENT if the uh check Su some changes it's very unlikely that the bite is going to match but the int would be different during development I stored this per frame so I can see instantly when it off went off the rails if it went off the rails and once I shipped to keep the file size further down I just store it once per second so in the rare case that it would go off the rails if someone is watching the replay in the commercial release it'll just say I'm sorry an error occurred uh and stopped playing I did some simple compress compression which worked great uh the first was just rly compress the button uh streams so that's just like how how long was it held down and then how long was it released and then I just zlib the entire file and that was fantastic the other thing that you got to keep in mind is you got to pair everything for consistency so to keep

things clean and simple as possible and consistent um anytime that you're recording something or playing back something those should be paired so if you're recording do this else do that uh the other thing is that you want to make it as low level as possible so you access the system API or whatever to figure out which buttons are actually pressed by the live user and then you're generally going to put that into an object and then the game is going to use that object well after you get it from the system and before that object is used by the game you want to inject or record uh the button input the more complicated part is building a deterministic engine but it's not quite as daunting as it could be so the most important thing is making sure everything is properly initialized and this is great because if you make sure everything's initialized you're just going to kill bugs before you found them and you're just going to have a more stable game so an example here is a static Boolean um if you were to do that

you got to be really careful random is another thing that can be very dangerous but it's easy as heck to uh take care of I recommend that you roll your own random function and then the bigger recommendation is that you use separate instances and the reason I say that is because if you were to use a separate random instance for your core game Loop and a separate one for ambient audio effects and a separate one for visual effects if you were to add more explosions or or change the ambient audio generally speaking uh your pre-recorded input uh for the gaml should be fine and so if you recorded a game session a year ago uh theoretically it shouldn't uh break it the other big thing is to not use system events or call backs and so if your driver or your API has some sort of event when a sound finishes playing don't use that because that's inconsistent when it occurs will vary so the trick is to use your own so figure out what the length of the

song is for example calculate that in game ticks and then every time your frame runs tick that once that tick reaches the end then you kick off your own call back middleware is something that's not the end of the world so many many AAA games are deterministic and ship with middleware many middleware is deterministic um and even if it isn't deterministic uh you can evaluate how determin like how much work would it be to make it determin istic um so it's it's again something that you just have to keep an eye on but don't fret about it floating point is is uh a bit of an issue but it's more if you're trying to be crossplatform so if you're just recording a replay on one console and playing it back on that same type of console even if it's your friends uh you'll be fine in 99.9% of the cases and even we're really lucky because some consoles are similar enough that record uh the replays recorded on PS3 PS4 or vaita

play back on each other so those systems are luckily just close enough the bigger problem will be games on PC because you do have more differences between even between Intel and AMD and drivers and other configurations that specifically is beyond the scope of the talk but if you simply Google fix Point determinism you'll get all the info you need about how to manually specify Precision um save and restore States between drivers and um you can take care of it but uh don't focus on portability unless you really need it so unless you realize I'm going to ship this on PC to end users and it's clear that this is not deterministic between wildly different configurations then you can look into it if it's simply for your internal testing don't generally speaking you don't even have to worry about it for visual reference this isn't really a this specifically wouldn't apply as much to AAA games but uh if you were doing a simpler game

something safer it's all about precision and accumulation so if you were to do just a a simple low Precision lookup table for your sign cosign you're fine uh in most cases the the less safe would be something like if the player is collecting coins and a coin is over here the player is over here once you're within the vicinity the coin starts homing in on the player every single frame you're calculating well what's the distance how much should I move that will guaranteed generate uh Precision errors and so it might hit the player on frame 300 on this system and 301 on the other system so that's something that you got to take care of another trick that I used was sandboxing so this means that the game is no longer 100% uh of the testing that you would like but 99 is still good what I mean by that is that I stub leaderboards I stub achievements I stub system utilities online state I fake them all and it it

allows you to just have a solid consistent piece of code that runs you're not going to be testing actually connecting to PSN because that could be inconsistent and there would be ways to try and handle that but that's just a lot more work so the simpler thing is to just leave that to Hands-On testing and just leave your automated tests to sandbox mode the other thing that I do is I have save games and I have the option to either save and load them from memory or save and load them from files the cool thing here is that if you have a retail game and your end user can play these replays uh then it won't affect their save games they kick off the automated tests it does everything from memory they quit the game restart it everything's as normal with their original save data now even if you took care of all of that you still will encounter some cases where your determinism is broken and sometimes the simplest solution is the

best and for me I did almost everything just by logging things so I log when I'm recording and I log when I'm playing back and then you grab a diff tool I like wind merge but anything will work wind merge is free that's why I like it um and then you could just compare the logs to figure out where things play back correctly or incorrectly and I think all of us have probably experienced at one point or another something where the debug works but the release doesn't uh or vice versa there's some sort of specific bug or different configurations and so this again helps you do that this is an example of the log and just showing you this is a very simple thing I just show a check sum of the frame the frame number and uh the exact function and line which I kicked off this uh poke and then if that doesn't work you can kick off of verbose mode and so in this case I dump more specific things like the entity position and facing and here you can see okay so this One Thing Happened on one and

didn't happen on the other and then from there things just went completely off the rails the other thing is just bulletproofing your code the cool thing is is that the more deterministic you make your game the less bugs you'll have the more bugs you'll catch and vice versa the more bugs you kill the more deterministic your game will be one huge thing I can give you as a tip is to try and keep your game Loop and your rendering independent uh don't intertwine them and this can be hard for for bigger pre-existing code bases but uh it's something that you should strive for because as you could see half an hour run in in 15 seconds could run in 15 seconds as I said if I turned off vsync now of course it's a simpler game uh but the short of it is even if you've got a a 3D AAA game if you're not running rendering every frame you're going to be able to run it a lot faster the other thing is just pull out all the stops run static an analyzers sanitizers and Fiddle around now this just just for fun you can create two different wildly

different game configurations just to see how robust it is does it work in a 64-bit thing with release optimized and different struct alignment different floating Point Precision I mean that's really pushing it you don't have to nail that but if you can then that's pretty cool and as you could see as well I I used it in so many different ways that the ways can build upon each other so the super simple thing would be in Windows for for example I created a batch file and so you play the game it records the input dumps it to a file I rename it to part one and then I load the game again it loads my save data play again exit save to and so then when you chain that you can just play through the entire game and you don't have to make sure that the initialization and de initialization is perfect because you're just killing the executable and reloading it if you want to do this on consoles though and you really want to make it robust then you want to have self-contained automated playthroughs so again this is the more advanced thing uh

it's really not needed I didn't actually implement this until after the original version shipped in 2012 so you don't necessarily need it but it is really cool and it opens more doors so this of course you just got to make sure that you properly init and de init everything um so once the game shuts down and goes back up to the top of the main Loop uh things are as if you just launched the executable and I recommend using the in-game save otherwise you're going to have to delete the save data before you run this depending on what your test is and as I kind of mentioned before is I just iterate through replay 1 2 3 4 5 once the uh it reaches the end and it's like oh file not found then you know you're done and it returns control to the user so then the user can play the game and see did it end up where it should have and if you had a build system or something like that you could actually spit out information or just analyze the screen or every which way to just completely you don't even have to look at it yourself

level replays uh where the person's been playing the game for a while and then they start playing a level and then it starts recording from there finishes recording and then they can play that back it's all about State uh the simplest thing is to just completely reset the level State whether you're playing or uh playing back or recording so it's just consistent and then again as I said you pair things so all of your Replay code should be the same as the recording code in the same spot entities are the exact same so if you wanted to control an entity for a cut scene or in some cases people have actually used this for racing game AI uh where you actually use button input to have the other AI players this uh I know of a game on PS1 that did this which is it's funny it doesn't sound like it would work but it can but again reset the entity State uh and keep things consistent rather than trying to just make sure that things are good backwards compatibility is another thing that I recommend and it's pretty cool that when the game shipped in 2012 and I recorded

some replays then that even after I've done the updated DX version and all of that stuff those old replays still run and it's actually not as daunting as you might think it doesn't make your code quite as messy as you might think it's simply a matter of saying well what version does this header say it is and if it is well set radius bug enabled for example and so the big thing I can say here is that I recommend that you try this even if your game isn't fully determined istic because if you're recording the input every time you play the game uh and a bug occurs and you kill it and then restart it with the replay let's say 50% of the time it goes off the rails and the bug doesn't happen well 50% of the time it does Repro the bug and working at other Studios I did have cases where there was some bug that happened on the PSP that happened about an hour into the game okay no it was it was it happened half an hour into the game but it only uh but we had to run it in debug mode

which ran it half the time so the play tester or the QA guy had to sit at my desk and play the game in slow-mo for an hour to try and Repro the bug because I sure as heck couldn't Repro it myself and after an hour he couldn't Repro it it was just a nightmare uh and it was a crash bug too so uh having something like this it really kills like 99% of those things the other thing is you just start simple like I said you just record it when the game starts stop recording when it finishes and then from then you can just build and build and build and so again uh download the sample code and uh check it out and uh thank you I'm I'm open for your questions so there's a mic over there and uh there and there so I have a question you mentioned uh you used the metrics what did you use

to generate the heat Maps um heat maps are something that I wish I did um I really really regret regret that uh a funny a funny thing that I learned with my game was that we pl we play tested it to death and it was getting all these Awards and it was amazing and then in the final crunch uh I was just like oh no there there's some gaps between these missions I need to add this oh it's too easy here I need to add a mission here and those missions that I didn't play test that I added at the very end were the ones that people riped on uh and so I I recommend everyone do the heat Maps do the play testing and and at the very end if you've got the choice between adding features or just polishing up what's there Polish up what's there so how did you manage to get um determinism between your dbook and and release versions of the game the big thing there was mainly uh initialization and so I found that almost every time when I found a case where something worked in release uh and didn't work in debug or vice versa it was simply

something wasn't initialized which was just a bug but your floating Point results would be different for your DB days indeed so in certain cases you can manually specify Precision you can do that on PC for sure uh sometimes the compiler options as well you could try and do that uh but yeah floating Point again could be uh trickier but a lot of AAA games do uh ship deterministic so it is possible but it's just uh it's something that you've got to pay attention to and just Google so setting the the floating Point Precision at conservative levels did that impa impact performance uh it it all depends on your game you know it's it's going to vary okay thank you hey Brian thanks for the talk oh thank you uh so Retro City Rampage is runs on a fixed update Loop yes do you have any recommendations if you're using just uh unfixed uh update Loops yeah so I would just just like we record buttons streams and analog stick

streams and all of that I would recommend that you just also keep track of the uh Delta time that was each frame and then play back with that Delta time fed in okay yeah thank you yeah any other questions what was your what was your favorite mission in a retr City Rampage it's a funny question because uh I guess there are two things here number one was generally speaking the cop out answer was whatever the latest Mission I had done was my favorite mission the other thing was um once I had been playing the game and developing it for years my favorite missions were the last ones I added which were like challenging to me and really awesome and then it turned out that they were too challenging to most people because they hadn't been playing the game for years um but uh uh yeah I guess I really like The Stomping missions the jailbreak Mission uh that one was pretty fun I if you were to implement Network networking play how would you have you

thought about how you would log and Replay that um networking is there are a lot of different ways to do that um but you know it's funny it's a lot of multiplayer games use this Auto automated testing because their game is deterministic because of multiplayer um but there are different ways you you aren't necessarily going to be doing doing your deterministic gameplay isn't necessarily the best way to do networking um sometimes you do want to do the the correction and stuff like that so it depends on the game but um it really is like I said lowlevel so you get the system input and then before the game uses it you inject in there to record and play back and it would be a similar thing like your packets you just like grab them at the top and like record and play them back yeah do you have a solution for asynchronous file operations or right uh that is that is an issue in that case I

would probably just sandbox it so that wouldn't be fully tested but then it would just uh do it synchronously when you're doing the auto playback and uh and whatnot any other questions does anyone want to see the MS DOS version it's reading from the floppy takes a second how much time do we have this might take a moment I can fast forward

okay we're close actually it would be sweet if we did have sound because then I could have the PC speaker audio so we'll see come in oh there we go it's funny it seems like we all threw away our 286s our 386 and everything people have collected Amigas and collected Commodore 64s no one cared about their old PC they're all in a garbage dump somewhere so it's actually surprisingly hard to track those things down so it's pretty close the uh the big thing was um I removed the 3D mini games and things like that

but uh every single Mission otherwise is in here it's it's been super fun uh I was done with porting and so I decided to take some time off and then during that time off I ended up doing this which isn't really time off but the the my justification was well this can't really be monetized very well so it's time off yeah so I'll uh I'll announce more about this at some point and uh but yeah it's pretty sweet it's wrapped up the only thing is that uh I've got a box of oh we're we're out okay well I'm going to add a controller support and then we're good all right thanks everyone