Succeed by Failing – A Game Dev’s Journey Into a Mobile Dev Ops Tool Chain - Mac Clark, Sauce Labs.

foreign [Music] [Applause] devs journey into a mobile Dev Ops tool chain so yeah please Mac take it away all right how's everyone doing today right on right on so um this is basically made for game devs um and uh so if you're not a Game Dev or

you're just not interested in this stuff you're not in the right place so anyways I'm Matt Clark this is uh sauce Labs uh I work for sauce labs this talk is called succeed by failing uh game devs Journey into mobile devops tool chain it's kind of a mouthful I like to start at the beginning um with a quote at the end of the day people won't remember what you said or did they will remember how you made them feel it's kind of an EQ statement which is like I want you to come away from this feeling like your time is valued I spent five years a little bit more making a game using Unreal Engine and three basically so it was a while ago and I want you to kind of take those lessons away and hopefully take some of those five years and the 20 minutes we have and you know gain some knowledge there this help helps you out moving that forward so who we are sauce Labs is a devops tool chain company we have real devices we have back Trace which I'm going to go into depth on we have emulators and simulators we don't do testing but basically we provide all the tools for you to test your games to make

sure that you can have less errors and crashes and this is a talk about how to fail often and right so we'll kind of go over the differences this is what I hope you take away from this talk our vision sauce Labs wants to deliver digital confidence in a world that relies on code saw slip strides to make it work for everyone everywhere and every time this is digital confidence so that's our little thing so I started out in banking I was a Professional Services engineer implementation and I did online stuff so how do you start from this sort of thing and go to full three-dimensional uh from 2D so I'm going to kind of go over that there's uh you're going to do some things right and you're going to do some things right or wrong hopefully you'll just do right and right but as far as making a game like right here you can see that I've done in blender the UV Maps

um and then I have a trophy room where like as you begin to catch fish in my game the trophies start to appear down the wall and you can choose them and there's like a little book and then I've included a couple level designs that I've done and I've shown you like a fish that you've caught I programmed the game I did the hype maps with USGS spatial data so they reflect real places and then I did all the art I did all the rigging all the animation all the programming all of the UI everything so we're kind of that's why I kind of want to take that solo part as a part of sauce Labs I get a 10 000 foot view of the largest companies in the world we're worth 15 of the top 20 studios in the world so it's been really exciting a couple of the ones that I can mention in our Roblox and half brick we have case studies in case you guys are interested I also sometimes forget to mention this as an indie or whatever you we have a free Dev tier so you can get 25 000 errors in the back trace for free and begin to Monitor and go through some of the lessons that I wish I had when I

created my game so you do some things right I feel like I did a lot of the event handling and pragmatic animations and the that stuff really right on that um this is right this is Kismet this is before blueprints I leveraged it to get through the game and then you start doing some things wrong um so here is basically the idea of catching a fish in the game so I come from the banking world so I used to do this in a database but I didn't have a database so I did this which you could imagine is problematic from a maintainability so in a point of view or even trying to sometimes troubleshoot it so what do I wish that I had um I wish that you know I'm up here as a part of this company representing the product because I believe in it so there is no right way some of the benefits of failure there's no right way you know in the end it's about succeeding and at what level does it look like but we can temper that a little bit right learn you can often learn that there are other

better approaches you grow as you fail you hopefully learn from those mistakes and you iterate on them and this is really how Innovation happens people aren't afraid to take risks everything stays the same everything stays stale all right so failing the wrong way these are the ones that I recommend you avoid there's no sense in making the mistakes over and over again right like it's just stupid to do that so if you fail to launch we've seen a lot of titles that like go through the process today a lot of the titles are one of the things I've learned more it's not about gut feeling it's a mixture of maybe there's a lot of analytics that goes into games now so there's a lot of data that's an analyzed in terms of where players start and stop but as you make a game if you can't technically build the game you have failed so I don't suggest that you start off with a game mechanic or start off with a concept that doesn't have that sort of traction failure to capitalize having a game drop below three stars or having bad feedback from your player base that will lose your players and then you will fail to capitalize on all that time that you spent and invested in

energy and then failure to engage if you don't Market your game you rely on viral marketing or other things then you're just really it's going to be very tough for you to succeed so you know it takes from a solo Dev perspective not only just the dev part you also have to have a business aspect where you engage in marketing and other things brand experience so I'm going to kind of dive in the idea of of what it means to fail and maybe not failing the right way which is to say if we're concerned about mobile basically at this level where you see three stars you're gonna have a 92 percent conversion rate with your game as it drops off jumping to a two-star game it drops like your your base your conversion by 306 percent in other words at 90 percent over here those conversions for three to for three to five star games you convert but once it drops below that that's 50 you start converting that that's more the way I think of it than this stat here but that means you have to put twice as much money for marketing all right so popular phrases when

sentiment I'm just sentiment I'm just going to like highlight this and make it bigger so what drives bad reviews every time please fix doesn't work keep crashing so these are the sort of fundamental things that will lose your player base and drop your ratings all right so the most frequent and impactful complaints I'm going to highlight this again functional error number one uh app crashing number three and network problem number four these are the things that we want to help you so solve so failing the wrong way poor experiences right if the customer has a band a bad brand experience 25 of those customers will never come back they're gone you'll never see them again so we want to help prevent that all right so how do you fail the right way well we're all software developers so we iterate you don't slow down continuous deployment continuous integration this has become the expectation in the in the industry shifting left and shifting right what does that mean in this continuous integration path that means that while

you're testing your game as much as you can up front while it's in production as well you have a fast solution that runs in the background to get those errors that are happening in production and crashes before they affect the rest of your player base and get them solved in real time and then regression testing don't make the same mistakes twice all right so I mean this is kind of my product piece here where I'm going to kind of just dive into what back Trace does a little bit so what we basically do is you can stop um think about us as like the mean time detection to an error and the resolution to a crash we can press that time cycle by upwards of 70 so instead of spending hundreds of hours looking through log files we basically uh have this ability to capture and prioritize and then resolve the issues so that below meantime the detection is the little stat the little mean time thing on the resolve part is the mean time resolution so all right this is the actual product we have an analytic shot where you can actually see

um over time how your releases are doing so in Crash free and error metrics at the top so you can see that these are good stats 98 error free users 94 with the air free application launches I've seen this go to 50 that means you have a problem and that means you need to solve it because you're going to lose players um failing the right way so what is it how do we how do we do this for you so how do we basically cut that mean time to detection resolution we have over here we have a fingerprint you can see that this error has been happened to 2180 times so instead of having to sort through that error 2180 times we fingerprint it with our heuristic engine a little bit of machine learning and we allow you to open up jira tickets we allow you to do slack we allow you to do any sort of integration that you want but basically we're built to scale and ingest errors in Mass every platform every console every operating system every Android every programming language we have an SDK or an integration for you so we can help you out and you can get a tire snapshot and we provide it in one portal for you to

basically do this so we're not only do we scale one of our other value propositions we help you organize and bring speed to what you do all right so failing the right way we slice and dice right so we can slice according to switch Android Mac OS so you can actually start getting into your errors so it's like sort of having like a 10 000 foot snapshot and now you want to take a magnifying glass to it zoom in on that error understand where it's coming from understanding what's important to you so you can do it according to release channel right you can just sort by production only the errors are in production and get rid of QA so that you can actually look at that release or you can sort by releases all right um now beyond that ability to prioritize and capture how do we resolve we basically take this information which comes to you as the developer this is the faulting thread this is the call stack this is the log files the screenshots we even do videos in this case we've had a source code repo setup we'll give you the line that your code is failing on for that particular error so this is the speed that I want to like

emphasize that once you guys have that problem in order to have that you know keep those four and five star ratings you want to basically bring this in right away give this developer and that's the kind of speed we're talking about with inside of a crash with inside of backtrace it'll be inside of our system within one second all right so this is just kind of more about the call stack decent multiplication and just a highlight of it we also do breadcrumbs like a little bit of information setting up to that um this is our attribute system oh that's good because I'm flying through this right now this is our attribute system and um we basically have an extensible metadata so anything that you want to bring into our system um so like if Unity comes in we have a special we're a VSP partner of unity so we bring in like 300 pieces of attributes automatically things like GPU shaders CPU Ram total everything that you'd want to know about what's going on in your player system it is anonymized we do have another tool that if you want

to add even more sort of Privacy Information to it that you can but this attributes extremely powerful it lets you basically bring stuff into the system and then bring it out if you want to export it to analytics or other things that will allow you to do that and we you can add stuff in so on the your level of the code on the client you can say something like there's a green car and a red car and a Finish Line and I want to get to know where those things are happening and when that's coming across this sort of Json serialization let you know that and you can add that to our system and then the other thing that you can do with the extensible data is then unlike any of our competitors we keep an object around for like 90 days so instead of waiting for that error or crash or that thing you're interested in investigating into the into the future and waiting for it to happen you can go back 90 days and see it into the past because we keep that information there for you it's extremely powerful all right so here's some Pro tips on my end these are things that I've learned

from my game that as I developed it as well so for the user interface as things have moved at least for me at that point between like HD to 4K I find vector graphics in all interfaces that I do to be superior to bitmap because you don't have problems with pixelation they have a lighter footprint they're a little bit harder to use SVG Graphics I did some work for Nike and I used the really concerned about brand and everything was SVG Graphics really crisp really clear very professional can be difficult to use so I use that for scale form for the 4K infinite Loops right if you have some sort of Loop in your game and it becomes infinite your players are stuck so at some point you want to basically find a way to eject them at some sort of maybe possibly keep account like past 10 then say nope time to break this Loop frame rates unless your game is like a competitive FPS shooter I don't suggest that you run at 120 140 frame rates for

me personally I put some hard-coded time not the best thing in there but I did some delays and I had a player who had a better machine than me for Designing it and he was running my game at like 220 FPS and was blowing it apart the second I capped the frame rate to 60 the game runs beautifully and smooth so really most players unless you're competitive FPS it's really not necessary to have that sort of thing and then keep it simple if you divide by pi you're not doing encryption I use really really simple math right so a computer can run like a billion calculations a second now so if you keep things to booleans zeros and ones choosing fals is switches you'll find that your game Just Books through the code and runs really really fast but by the time you decide to divide by something crazy and make your computer wait that's going to kill your game and kill your speed I've seen a lot of people be able to go in and create cameras um different corporations and they'll try to like measure frame rates per that I think just the idea of architecting your game simply it actually is very

powerful so all right so that's failed gracefully fail art fill fail often sorry I read a little piece of this and so I'm going through it so what does that mean what does fail often mean um this is the idea that you know as you moving through it you can iterate on what you're doing so one of the things in lessons that I've learned is Unreal Engine um for those of you that are familiar first of all how many people in the audience use Unreal Engine cool how many people use Unity awesome how many people use a custom engine okay so we support all those if you're an indie developer I suggest that you use an Unreal Engine or a Unity because you're getting so much out of the box you don't have to reinvent the wheel if we tend to see Enterprise clients the ones that use are very very large custom engines I noticed with Unreal Engine that basically As you move from like 4.15 to 4.2 or to five or any of those things one of the things that changed

most dramatically in order from you to be able to move your game from that to to level to level is programming you ran into whole different physics systems whole different animation systems so if you do your art first if you're and this is for small people right like small Dev small Indie teams you do your art first then you're gonna be able to move that faster through the process it will move to 4.2 to 4.27 without the friction that you have is moving your code base all right so once you have that idea of where you're at you're going to want to harden your version at some point once you start programming now's the point where you say okay we're no longer going to move to five it's going to be too problematic to go to those new systems like Niagara and we're just going to basically stay here and we're going to make the game we're committing to it and this is what's happening the render thread do not in any set of circumstances in my book as a small developer mess with inside of the render thread there's you could do all kinds of programming for these open like for unreal for these open engines but the

second you start messing with the render thread you're going to find out that it's a black box there's literally nothing coming back there's no errors you'll just have a game released and it'll start crashing and you won't know why so I highly suggest that you don't mess with the render thread now marketing I mentioned this before as in Indy or anybody really in the space you need to do one-to-one at least for every dollar that you spend of your time and energy that you know there's an opportunity out there for you in terms of what your opportunity cost might be you need to commit a dollar to marketing to that and just to touch on this is some of the really large players in the space that I see now they have data teams sitting there the guys that are people people sorry that are running like you know serious analytics data-driven analytics on where the players stop and what they can do one of the things that we do offer with inside of our RDC and mobile is we offer video recordings so if you don't have any information in your game that's basically capturing Privacy Information what you can do is watch these videos

the last three seconds of them find out why your players are leaving your game where you're leaving your game it's another way of doing that all right you can't fight what you can't see one of the beautiful things about doing uh with backtrace at a small level is you can start fighting trolls I hate trolls this is just my anti-troll comment all right so the future New Era of devops you have like traditional testing I mentioned the shift left mention the shift right I'm not going to stick too much on it continuing testing is becoming the norm you need tools both side back Trace can help you out in the future with sorry I can help you out on the shift left with games before they're released you can do things like run automation testing which is like Gauntlet for Unreal Engine or you all Unity or Unity Frameworks to test your game you can start to do like AI Bots we're starting to see people use machine learning to test their games ahead of time to push errors into the system and then on the shift right once it's in production your players basically are your testers all right I'm not going to touch too much around this but we basically offer

a full Suite from API testing to live testing automated tests to beta test we have Frameworks sauce labs.com um again backtrace has a free Dev tier and we do it all the way through the whole process CI cdc's become the norm which is continuous integration and continuous development so what does success look like so for me it's working on what you love like I'm from a solo Dev perspective maybe it's not the best thing to make a fishing game but it's like what I did it's actually a fly fishing game it's where I'm from in my state we love to fly fish that's something I do to get in touch with nature and something I do to get away from the computer and I recommend that when you do projects because you're going to stare at it over and over and over again that you do something you love because that way you're going to be able to follow through with the thousandth iteration that you have to touch it and then you can fail up right so I've seen there's a there's a famous uh GDC talk out there where a guy's talking about a card game that he built and you know the first one didn't sell

very well and then he's like oh okay and the second one didn't sell very well and he's like oh I'm that's that's horrible he took a job Contracting right but he continued to build his third one out there and then all of a sudden he noticed when he sold his third one his back catalog started selling so his whole thing and then emailed his fourth one and his back catalog started selling great so you don't really know what the success is going to look like you really just you can like basically you think you're failing but you're not and so my message to you for this whole thing is basically never give up enjoy the process have fun be creative and move that forward and that's it okay if you want to check out my game it's called fishing on the Fly you can see it my website Matt Clark I also built a robot with Unreal Engine 5. it's pretty Cutting Edge back in the day

[Music] [Applause]