Superstar Session: How to Spot Performance Problems that Gamers Hate & Even Major Studios Often Miss

[Music] thank you everyone for coming to this first talk I don't know about superstar but I will try and make it worthwhile and if I'm not specific enough in what I deal with feel free to ask me some questions at the end you know to get to the heart of the matter because really I want to talk about the vision that we have a game bench as well as some specific problems that people find and the vision means not just not like the tools or the services we sell today but what guides us which is a sort of philosophy that I'm hoping to impart if it's not specific enough just ask me questions or catch them at the end this is the company I work for game bench I've been there since the beginning 2014 2013

bloodier and yes so and we it's still the same core team that we were they're all together in 2013 and if anything the performance issues that we look at like framerate and battery consumption load times you know many different performance problems that affect mobile games if anything they've got worse over there are seven years we sort of thought they would get better but as games get more ambitious becomes more of an issue and also as a more exotic Hardware comes on the scene the emergence of Chinese hardware vendors for example just makes it harder still so yeah on the other hand though after seven years it feels like we begin to solve it to some degree at least we sort of know where we're going on how to solve this and that's what I want to talk about today so yes I'm going to talk about how to spot performance problems that gamers hate and that even major studios often miss including our clients I'm definitely not

going to name or shame people and I've hopefully blurred out game icons where there are problems but because you know I still think it's a miracle then anyone puts out mobile games frankly it's so so arduous and such a headache so I'm not here to criticize any studios it's just it's almost inevitable and it's really about how you deal with that so but I thought I would begin with something a bit more interactive anyone want to hazard it guess what that is such a small glimpse of an old monument that if you get it wrong it's more just like it's not a big deal get it wrong but say something not far away Nelson's column you know Trafalgar Square major attraction but I thought I would bring this that I can appropriate it for making a point and I'm gonna actually rebrand it as Nintendo's column

and because I mean I don't know about you but I have three switches at home we played a switch a lot in the office and I often think about what is the difference between the switch and the mobile phones and iPads that we play you know what is what is the difference between the games how come I'm playing which are three on on this and I'm playing north guard on this you know and I'm I'm only just now beginning to play a Call of Duty mobile on the iPad which is a you know which is this difference and you're making a switch game if you think about it when you're when you're planning your switch game or your port there are only three switches to think

about there's the regular switch switch light and the dr. Jinnah the switch right so God if only if it was that simple because things the width of the monument at the bottom that represents a close connection between the maker of the product and the consumer product it needs fewer device profiles just three that I can think openness I'm wrong another switch that means fewer bad surprises as you go through this process fuel pouring performance tests oh it works on this device does it work on this device does it work on this device is so it's so mining honestly after seven years the key things take us towards more deep and immersive games not to attack mobile games it's just that if you have a grand vision for a switch game it's going to be easier to

implement on switch okay what's this one yes this is the Great Pyramid of pyramid of Google I'm appropriating this culture artifact to to make my point because this to me feels like much more what it's like to make a mobile game right you your audiences again these guys here milling about at the bottom I mean thousands and thousands and thousands of devices in their pockets a game bench we we've we've got data on like 7000 different device models and you're probably just designing I mean we'd still encounter shooters that are just designing on iOS and then eventually they think well we should develop an Android version 2 and then eventually they think well we should hire some sort of outsourcing company so you you might start designing on just one or two devices you might

develop on like 10 devices that you happen to have in the office then you might outsource to a company to test on like 100 devices that they have in their locker and look at the results but when you launch it's more like 10,000 and that means just a world of pain rather than a graceful column complaints more drop-offs but ultimately and i reheat to say this because i'm not trying to attack or offend anyone but but less ambitious games because if you want to be super ambitious you're going to have huge commercial problems when you hit the the reality of of fragmentation on Android so how do you tackle this issue that is what drives us a game bench and we think we're getting closer to a solution how do you go from to a column of equal width from

beginning to end and I can only apologize to the ancients for this Photoshop but I couldn't resist doing it because what we've learned game bridge is that you can't solve the fragmentation problems if anything it's just more and more and more you just there is no solution to that or what you can do is even from your design phase when you're just playing around with ideas take that diversity of hard way into account and you can begin at the same width that you have at the bottom right it's just a it's it just seems like a a really gargantuan heart to make a really wide column which is what takes Photoshop isn't meant to represent you with me so far so happy you're okay so there's not really an answer is it it's just a really impossible seeming mission but I would call this ecosystem wide intelligence from the start so develop

and test before you launch is this an impossibly big goal we don't think so and this is how we've learned to deal with it if you're building something is huge as that huge monument of a big column or be broke of what wasn't here you need plentiful material you need to get that material effort effort and that is raw data and I'm going to look at some more days or in a second but that plentiful material is not enough you also need a clear quick way to build with that material and we call that actionable intelligence and we think if you can have those two things in your studio then you can cope you can have that ecosystem wide intelligence from the beginning let's get concrete now it's our counting down yeah that's okay good raw data where does this come from what is it look like this is a shot of a game bench web dashboard a session which has

come in and this is me playing rules of survival anyone played it it's like it's like the Call of Duty G and I just played the game I didn't have to think about testing or anything I'm just playing to win as I normally do sometimes actually and these this is data this is that plentiful material that we need lots of and you can see I've captured my frame rates and they're mounted jumpiness in my favor I've captured crucially the power consumption and this is quite ridiculous power consumption if you look at it so I don't know if you guys measure the power consumption of your of your games or how you do it but you know the best way to measure power consumption is in milliamps which is a unit of current and

you have a device like this which is in it which means that this game which burns 1.2 amps is going to last for probably less than 3 hours of gameplay which means if I play this game well I did play this game aren't they in this game I would see it deplete the battery by around 20% in half an hour so if I play for 15 minutes that's 10% loss of my battery and I don't know about you but that's and that's enough to stop me playing because battery is too precious so this sort of raw data it's hard to make sense of it in this format because it is raw but it does you can see it does have the knack they're nuggets of real performance problems in there we can also see that it's really heavily using the GPU we can see that it's using the CPU Saosin themselves for example red flag CP usage that goes over 12% it doesn't sound high 12 percent you might think well I've got like 88% to play with but it doesn't work that way because device has changed their behavior so here you

go here's a ton of raw data and the question is how do you get and not oh this is you see the next slide actually goes a bit deeper right these are all buttons that you can click on data it does include so this is a frame rate timeline here you can see I played for 24 minutes right so this is a big issue that affects pub G as well the maps can't generate quickly enough when you get in a car to escape the big encroaching bubble which can kill you in this game and you're trying to get away from it in your vehicle it's quite fury thing as a gay man lots of gamers are playing these sorts of games do complain so again this is just raw data but you can see it is capturing things that

matter to our to our gamers question now is how do you get it plentifully enough how do you get it easily enough that you can construct it into this huge monument and it's the sources that make all the difference one source of this is manual testing like I just did I played the game and I just captured all of this in the background and if you did this in your studio you might get tens of these sessions per day coming in from from the devices in your pockets for example you may also outsource your testing and get your your outsource company to to get this data to you and you'll get tens per day the next step up if you want to really get tons of these things to start making sense of them is to have an automated device form anyone here user an automated device banter to do basic testing yes very good and this drastically speeds up and takes the board amounts as well of testing sending

your game to a farm of say a hundred devices and with no human effort just getting them to chug through the game and send all of this data back that that device one might be driven by scripts that say do this go here shoot shoot or pull the trigger whatever a I different Studios have different ways of solving a device farm issue it's a really hot topic and one we're happy to be involved in a game image with some of our clients definitely helps you to get thousands more sessions per day next one up SDK if you can embed into your game a way to capture all of this data and send it back then well the sky's the limit really depends on how many people play your game because you could get millions per day if you have millions of gamers and that equals a ton of info the other

way to do this and which we've been building over that five years or so is to actually embed a game bench or a similar tool into the hardware so game enters in millions and millions of phones around the world already it's built in and the makers of those phones the manufacturers are receiving this sort of data every day billions of sessions per day are coming to manufacturers already so that you may not know how much the power draw of your game is or your framerate issues but they do because they're very worried about how your games perform on their devices and so they've they're getting this data in and there's no reason really why studios can't but the data on its own isn't that helpful you need a way if you if you imagine these billions of sessions or even hundreds coming in per day you have to make sense of it and more data is more problems if you don't have a simple way of making sense of for example is it good or bad and this is where we move from data into actionable intelligence

what we've realized by first we saw we started offering this data to two studios but what we quickly realize is they need a clear answer is this good or is this bad is this better than my competition or worse and so we just started making color-coded badges to really simplify the data so we know x is genre specific genre dependent but generally in a casual game if you're able to Road in under three seconds which many students we work with really aim to do you're going to get a blue badge if you take longer on a cash payment then eight seconds you risk getting a red badge slightly more lenient on high fidelity games where you're loading more assets but you can set thresholds yourself to some degree but without thresholds and and clear colors I suppose at the end none of that data really makes sense so here's a

three this is a real trial here it shows you how quickly you can see patterns in the data that you can use to get back to that more switch-like experience of understanding your users devices here has really long load times and so regardless of the device the gamer is playing on they're going to wait a long time for that thing to load so you could say okay that game has an issue maybe is worth addressing that issue but you can also see that this device the Galaxy Tab a budget tablet also is slow to load even on games which are faster on other devices so you could say well hold on maybe I need to tackle these things even on this device maybe I need to actually red flag this device so that I can understand any any device out

there which has the same chipset as this I'm gonna risk my load times being too long this is what I would call the beginning of actionable intelligence and it's the sort of data that we we now get and the ratings we now give to our clients once you have this horribly manipulated to turn it into this he's breaking all the way from beginning to end in your in your cycle it enables some better workflows what do these workflows look like number one as we just saw with the Galaxy Tab a device tearing grouping devices together by their specs to some degree so that you can predict performance if you just have the Tab A and you know it takes a long time to load but you could probably assume that others with the same chipsets or similar specs have the same that's a prediction that would allow you

to make different versions of your game for different devices you know if you if you detect that they have a low spec Galaxy Tab a make a version of a game that doesn't attempt to load so many big assets at the beginning for example crucially and this is the real change that I've seen over the last few years is immediate guidance during design and build if you imagine a developer or an artist sitting there thinking okay how many what draw distance can I set for my game how many assets can I put how many objects can I put in a scene what if one making that decision they had feedback saying well if you do this you're going to lose a thousand devices if you do this so you increase your draw distance you're going to impact 500 devices and you should probably make a different version of your game that doesn't increase draw distance what if you could make these decisions with this sort of instant guidance about the impact on your 10000 devices that are out there in the wild this is this makes a huge difference to the ambitiousness that you can have when building a mobile game because it gives you confidence your

game will actually run so you can get pass/fail resolved in seconds or minutes another big change is that now we have non technical and technical people sitting around together QA people developers are people product managers you know more commercially minded people in a studio getting round and agreeing KPIs based on on these sort of color-coded ratings and this is having a big impact on the quality of games too lastly 24/7 monitoring of performance in the wild what if you can spot problems that your gamers are having before they even find time to complain about it or post a bad review you can you can react much quicker and you can pinpoint the issue much quicker than relying on the old way which is put out the game and wait for the complaints so I've got time for some questions and I just want to mention we're upstairs middle of the first floor do we have a booth number 25 okay and we're here and

we love gaming we love talking about games and we want to help you to come and talk to us this was a vision of how we see solving fragmentation with this huge block of a distorted pyramid root we have an SDK for putting into your game to get data back from the wild during tests or on tour after release and we have labs where we do the whole thing for you if you're if you're resource-constrained or you want some guidance thank you but feel free to ask me a questions if you have any hey thank you I just have a few questions so first of all how much control you think they develop a house of a power consumption because I know that just the screen itself can actually take even more than they have the power

consumption yes the answer is a huge amount of control right on perhaps about half of the power consumption so if you see that example they put 1.2 amps probably about four to five hundred of those milliamps you can't you can't mess with their out of your control but the remaining seven hundred milliamps very much under your control and you can do things like reduce your network traffic or change the pattern of network traffic to reduce the burden on the CPU and reduce your power consumption and we've seen our clients reduce their power consumption of their games to stop those complaints about battery life coming in cruel things and I saw the chart that you had over time that's great how do you actually pinpoint every specific position or part of your game play two spots in your chart do you do you do something with your recordings or yeah it's like a video but it just takes we can't actually get video but this the one I showed you is just it captures one frame every second and so just like you get used to reading these charts after Wow

again and here is probably where I lost horribly at the end and you can click anywhere on this chart to see that to see that screenshot and see exactly what was happening very quickly you see patterns in games and you get to understand why this is dropping and what what shader what texture or what scene is causing the issue alright so you record like when it screenshot if we can increase that for some and we have in the past but yeah usually one per second there's a demo at the booth and you can see how it works in practice thank you I would like to better understand how you aggregate information above the FPS WT metric okay so we aggregate it do you mean across multiple sessions multi voices or just even within a session

within a session right so we we've learned that you can't just measure things like average frame rate it doesn't reveal very much and it doesn't really reveal problems the first thing we've learned is that you have to segment a session and you need intelligence actionable intelligence about what was happening in the game so for example there's no point measuring the frame rate of a menu screen or a loading screen they may been deliberately set to one at a low frame rate the first thing you have to do is identify what is game play and again mentioned we've we've found some ways to to measure that programmatic we should we have to talk about but you first identify camera you can see here we've got a game play one six a quick marks which shows that 22 minutes is when the game actually ended and we've got a median frame rate for that section of 59 FPS which is pretty pretty damn good right but we don't have perfect 60 FPS this Wii is a FPS stability which shows the proportion of session time that was spent within a narrow band around that amount so as this finger goes down yes

your average may have been good but it may be in a really choppy experience around that this also measures that in a different way what you really need to do is combine all of these metrics into one badge which is what we've done we look at multiple framerate metrics to a water badge so we won't award a badge to a game which has a really high frame rate but is really choppy thank you why suggest is that we really did that so I think if anyone else has any questions in image to cover and they move upstairs on the first floor so please do head up there everyone around the boss but just have the sheriff's from thank you thank you very much [Music]