Round Zero's game evaluation process: A date with real users

[Music] [Applause] [Music] all right so good morning Pocket gamers my name is John Rantala and I'm the publishing director of round 0 I haven't had my morning coffee yet so I'm trying to survive here but I'm glad to be able to pop open this track and this day so um round zero was founded last year and

we are the publishing level of finger soft and our mission is to be the best publisher for mobile free-to-play games and here's something about fingers are best known for the hill climb racing series so far the two games have gotten over 1 billion installs and they have over 15 million monthly active users and we actually use this huge pool of players to promote the games that we also publish so this talk is about the way that we do game evaluation that is how we test developers games with our players to see if they're hit or a miss next I'm going to tell you a bit about our original goals for the process and then a bit about the challenges we've faced and finally a word about the results we've gotten so our original vision that's still very much alive is flappy bird so they were taught in the company that if flappy bird would have

been offered to us for publishing well it would have passed our screening process it's very hard for us to guess what kind of games our players would like to play so we want to be able to ask our users as directly as possible instead of making the decision based on our own subjective likings or dislikes secondly we wanna allow developers to validate the game core the core gameplay and the core idea as fast and as early on as possible this is something we do ourselves so we test out prototypes with our players sometimes in small user acquisition pushes and see do they grow in matrix do they get a better day 1 retention etc sometimes we only push out marketing materials and see if they get a lot of clicks or not this kind of testing and from from the beginning we wanted to be able to allow developers to also test out their games and finally we are right now developing our own technology to

automate this process we want to make it as lightweight as possible for ourselves and for developers and we're launching or our own game evaluation tech next year so a word about how we actually identify hit games and how our process goes first of all we use game analytics tools so we ask developers to implement something like Unity analytics game analytics Fleury there's a lot of providers out there after that we cross promote the app in hillclimb pressings we Drive anything in between 100 to 1000 users over a one-week campaign this of course depends a lot on the marketing creatives and the store page how well do they converse and we also inform the developer if the marketing materials are not performing well so that they can actually do something about that as well and finally we look at data from both the game analyst tool and from the app performance and then we try to conclude whether our users liked the game or not

so that's the base for our rounds and the in the beginning we test always for engagement we look mostly at day 1 retention and see if our users like the game or not this is usually one round but if the results are a bit inconclusive the Raven returned she is almost there but not quite we might redo the round and after that if everything looks good we greenlight the game and then we test for monetization and finally we can move on to sign into contract so this process is pretty nice for the developers they get free installs live users that test their game app also they get to test out their creatives and the store page the conversion rates everything and this should allow them and us also to fail fast with the game and maybe developers can learn why their

game did not work with our players the challenges we faced so far first of all not all analyst tools are created equal it turns out this is not to say some are worse than others but they actually have a very different ways of tracking data some tools track even they want retention like completely different to it differently to another and a comparing data between these service providers has been minor headache secondly initially we thought that we wouldn't have the screen that much we thought that we could just let our players basically play the game a bit and then we could draw the conclusions but it turns out not all developers can communicate in English well and they are not also looking for a long-term publishing relationship that and that's basically what we're aiming for also well of course games might contain some inappropriate material or while

let's store policies or the games might not just technically work which makes our validation or evaluation efforts pretty much futile thirdly this is something we've been pondering on a lot when can you really validate the game core at earliest what kind of features must be in place do you have to have some kind of daily mission or daily gift to see the actual day 1 retention what's the effect and of course based on like different genres so the answers we've come up with are firstly now that we have tested over 100 games we have a lot of data on all sorts of different analytic services and we can compare between the same ones but we still like to say tell developers that we prefer certain analyst tools to be used so that we get like really reliable results secondly nowadays we take a quickly look the game's user experience just just right a bit in the beginning such as first time user experience and also we

asked a couple of questions from the developer team still keeping it really lightweight and finally we're developing different tracks for future complete games and for really early stage validation games the really early stage validation we call it the demo track and this is when basically we can test out the game as early as it can be played for two sessions or for two days so something between 10 to 20 minutes depending on how casual the game is is enough for us to test and then see some data and we only compare the data between other games that have gone through our early validation and the point is not to really reach any kind of goals yet but we do multiple test rounds and then try to see the change and/or that the game gets higher retention and a better session data better first session data every time so the results

so far we publish three games and they are very different from each other one is a puzzle game and the other is a mid core cartoon type game we tested eight games for monetization launch or soft launch and eighty percent pass rate for testing so we we try to still allow as many developers to actually test their games with us here's a quick look at some data from the games we've tested you can see most games fall in the action/puzzle or racing category and the overall day 1 retention average is somewhere around 20% and the three titles were published there pretty clearly over there over the hour 30% metric goal why only 30% and not the industry standard or 40% that's because first of all we use backfill traffic from low CPI countries and according to our expertise when we

move on to soft launch or global launch the game will get a higher retention of about 5 to 10% that you and of course we will also do consulting when we do the agreement and try to tune everything monetization and the whole experience of the game so that it will actually reach the 40 or 50 percent of day 1 retention and here's some more data total daily play time for active user so the amount of seconds the users actually play the game per day on average the overall average is somewhere around 500 seconds exception being puzzler simulation categories and we've actually found that the more time peoples of course spend in the game the higher the day 1 retention and the higher the retention is overall so it has a positive effect but that's

basically everything from me thank you very much and if you see anyone wearing this round 0 hoodie please and you have a really cool free-to-play game please grab us by our foodies or our hair or something and pull us aside and let's have a chat and now have some time for questions great so we're straight away we have a question over here we just get a mic to you because we were recording as we go along so the mic will come down to you in a second just waiting for that to get charged up just down here generally from here thank you for the talk so as far as they want retention as you mentioned different analytic tools have different way of reporting death so what are the differences of each one you take actually as the standard that works for you as far as the day 1 retention the

differences in reporting etc and then the way you look at it actually yeah that's actually a very good question the another tools that we like to tell developers that we like them to use our unity analogies or game analytics we think ok they have some difference in tracking sessions but the day 1 retention is calculated pretty much the same way but for example I think flurry has a bit different and overall rolling retention is it's a bit of a different thing and and it's not really comparable to these kind of for example they were a teaser from game analytics but yeah those are the two game analytics tools we have the most expertise from and and the most data but we're also actually our own game evaluation tech we'll also have kind of this analytics tool in place and that's also going to be something developers can implement with or just alone I mean with game analyst

or other analyst tools alongside with our own analyst tool if that answers our question yes I couldn't quite decide who is fastest I think they generally from hey thanks for the very insightful talk so a question on sort of the audience's you bring from a racing game to a variety of genres do you have sort of methods to normalize the differences in the genres and the audiences well sort of we are getting all the time more data on the retention for genre and we know of course that the racing games they do affect the players decision on ok now I'm playing he'll kind of racing so my mindset is on the racing games and then Blake that so actually the first title we published heels of steel it looks pretty much like Hill Climb is a 2d cartoon type of kind of a not a racing game it's an action game but it

bursts a lot of similarities and of course it got a very high level retention and very good matrix overall possible just because of this but that's also another thing that this is why our target is somewhere around 30% if it's almost there then we compare it to the different like different title from the genre and we try to kind of normalize it as well and we are very happy to always provide the second round if you're unsure about the results Thanks and the gentleman behind you yeah thanks for the talk I was thinking that when you end up in a deal with the developers team do the terms deeper case by case with different developers or do you have some similar terms between the cases well yes we do negotiate the deals every time separately but we have some starting

points on edge deal of course but mostly is for example we we do have different kinds of deals for hyper casual games in comparison to casual games and of course we're expecting something different from midcourt games but we are interested in testing all kinds of games and finding out something that works with our audience so yes we do negotiate it very much like case by case oh I've got a question I just want to check out yeah I'm kind of one things that I do work for a couple of investment funds we're trying to help identify games and stuff for that key part what we're interesting isn't also how people work in live ops and their understanding not necessarily to have a skill in house but have the understanding of the implication it has in the management of the game in terms of content delivery and ongoing production and that kind of ongoing commitment is that from any part of your kind of concerns or are you kind of providing that experience well a little bit of both so we actually want

developers to keep total creative control of our game of course if we're publishing a casual game or medical game we kind of expect it to have some kind of live ups and we do help with that but mostly it's up to the developer I'd say cool that's great anyway anyone else gonna go any other questions I've got one more if you if you haven't don't worry so looks like people she's being shy now so I other expert to me I think this whole area is fascinating a little bit to the question that didn't really from there and you know this whole idea of you know understanding a team and their capacity but I'm thinking more we all know that not everybody's first games are going to be successful yeah how much do you look at this as a as a fan of courting process as a process or sort of find and identify the talent if they might if they don't have a game that's gonna work this time round are you are

you looking to help them get to the next game or are you just looking to end that yes so excellent question that was actually like one of our original goals to allow developers to validate the core game and the idea and that's mostly just to help developers and also help with the industries and yes we try to build really good relationship with developers and that's why we want this process to be very lightweight so we don't really demand a lot when we go to testing just some communication and we already see if the team works well with us or not but anyways we're we were actually working a lot with for example universities and University of Applied Sciences so they have the student projects and we offer them the test rounds and it's all about kind of also educating them so that when in the next one to five years they make that hit game they will come to us and we will do the tests and we would already have a kind of solid good relationship I think that's really key isn't it because you know we've we need

to see these channels of ongoing talent and help validate ideas otherwise we're not going to see innovation and if we don't if we only looking at the things that succeed in the current market we'll never know what could succeed yeah yeah that's sure so you know again it's just trouble with as much as I am a massive if I like you guys we recommend that people have all this analytics in place the crux for me is that we actually need to be able to chest test new hypotheses as well no but it's tricky because that has comes a risk yeah market research is always kind of lagging behind what's current what's currently trending and actually that's also one thing that we're kind of aiming for to be able to find something that breaks it and something that this works exactly well thank you very much yeah thank you [Applause] [Music]