Liveops: Mapping Player Journeys (Andre Cohen Head of Data Science Tilting Point)

all right so mapping player journeys so a couple months ago joe asked me to do this and i was thinking wow where where to begin because if you if you google player journey and you try to find a wikipedia page on what this means you're not going to find it you will find you know podcasts i know joe has talked a lot in his podcast about it i've worked with it for a long time but it's kind of a nebulous concept right so that that's the that's a general idea a little bit about me just to see where i'm coming from with this stuff because this is liveops and i'm a data scientist so as much as i have uh something to say about you know how to

manage a game and operate a game that's live i'm coming from a very different angle right i spent seven years trying to become a professor i was really interested in the concept of you know teaching a computer to play a game and finish the game by itself i only picked pitfall from atari because the resolution of the screen is so small so it fit into the computers back then to analyze it then i discovered that you know to become a professor you have to be really good at fundraising and make getting money for your lab to to work on um as a student you don't really get the opportunity to raise money that's something you do when you have a startup so i started with a business partner you know making indie games in new york and we made this beauty uh just around the time that iphones came out a bit later premium already existed or like the concept of internet purchases and you know on his journey home

it's about a scarf trying to catch some acorns in a map as you can probably imagine from the graphics the game was not a success so we did not become angry birds or zeptolab or anything like that but it had all the components for understanding player journeys right and specifically when you have a game that has fixed features or fixed game play or fixed levels in this case it was some i don't know a couple dozen maps that we had and every little acorn was put in a very specific place by hand because we wanted to make a handcrafted game made in new york uh you start to really think about what can i optimize and that's where you know i started another journey and myself trying to figure out how do you optimize things with dynamic pricing and game optimization the journey started really quickly you know by like okay apple doesn't allow you to change the

num the value the price of an iap but we can change as much as we want the number of coins so that that was a couple years of tweaking around with algorithms and machine learning to to figure out how that works because as much as you know airlines have been doing it for years it's different in mobile then there was rewarded video ads you know why does everyone always get five gems for bad watched how do you optimize that because now you're talking about if all players qualify for this and finally the the latest incarnation is how do you how do you do pricing when you can't change pricing well you you selectively decide what to show people which i think is something joe kind of touched on in the previous stock if there is frequency and there is like what day of the week to show up an offer but who says everyone has to have the same offer right one example is you know chinese new year's may not be applicable in brazil it's not worthwhile there and so you can

change this and figure out how to optimize the game differently and then finally last year you know gondola was acquired by tiltingpoint where i had this completely new vision of of what is a player journey because for once i was now seeing you know what the ad revenue is and what the user acquisition data is like and how do you combine that into player behavior how do you get the complete picture from designing a creative for an ad how do you cut measure the click-through rate or the targeting of an ad what happens when those players come into your game and then how do you maintain those different kinds of players in the game and that's basically you know where i'm coming from now the thing with maps is in player journeys is that they're not real things right and i'm going to talk a lot about this in this presentation the where what you envision a map to be is very unlikely to be true and vice versa as well how many times

you've heard people say oh i'm from chicago but then you ask them where in chicago are you from well actually you know i live in foxhole it's a 45 minutes car drive without traffic sometimes that's not chicago but in their minds it is right because they have a different map of where they live the same thing i'm in in manhattan i'm in new york if you tell me you're in jersey city that is not the same but if you're in jersey city you may have a different vision of that map and and so you know in in doing the research on like what is maps and how do people deal with maps in general i found this quote you know of you know every map is someone's way of getting you to look at the world in his or her own way and working in in now tilting point with dozens of different departments which with its own view of a game and its performance i've come to realize that you know player journeys are everyone has their own way of seeing the game and understanding their player journey what i see player journey

comes from my background of data scientists but ad monetization has their own and and so on so a little outline for the talk first let's let's i think we should discuss the what the landscape of player journeys are like what are people doing what have they done what are some of the ways that people believe a player journey looks like then i'll share with you what my understanding of a player journey is and and also what the traps are because if every journey or every mapping there's always something that falls through the cracks right it's like uh you know maps of the world you know depending on the projection that you use the us may appear really large or greenland may appear huge maybe some countries become very very small and so there's this is the same thing with with games and finally i'll touch like about the the cool stuff that can be done with this with this kind of mapping that i'm about to propose that kind of gives you like a

sneak peek of like what's out there what can be done if you go past one data scientist as joe mentioned in the previous one you know you can probably do rfm with a pretty small team you know just getting the pipeline of the data right and you can you're off to the races i'm pretty sure up to common traps that applies the extensions is where you have to take it up a notch but i'll i'll share some of those things core game loops i mean this is by far the most old player journey system in the world probably since computers have existed there is the concept of thursday you're processing some input i'm going really you know back to the to the atari here you do you update the game state somehow based on that input and then you render it on a tv screen or a console or etc um and then you do it all over but what's important to like what this might sound trivial to start here but you have to realize that if anyone has made a game there's a big

part of the development process where you're actually already optimizing this flow and this is the journey before you launch the game this is a journey you're considering what are the inputs or the keyboard shortcuts that i want to have in my game that makes the player engage with the game in the first place right are you doing aswd or are we going to do arrow keys what about handicap accessibility do we want to you know ignore a certain group of people from playing the game is that or is there an incentive to add additional controls to the game to bring a new cohort of players into the game the same thing with rendering how often have you made a first person shooter and then you know oh man i think we're losing players on the lower end devices i think these are some like weird android phones i think we just got to lower the like increase the frame rate by any means like maybe reduce the number of shadows or lights same thing it's this is no different here and also obviously the whole talk now is going to be about updating the game

i'll leave the technical sides out of it so this is fundamental beginning journey the other one of like from the days that when i started working with games is already the early mobile funnel right that way you have start with the iphone visibility do people look at the app store and see my game and do they think it's interesting guess what there's been optimization of this first step already people were making so i've just found this on google by the way i don't know this poker game at all they clearly thought this logo was good but then somewhere back like 10 years ago when phones came out you put an advert an ad for another service on top of your logo because that might just increase your click rate at least that was the story open fade said people are willing to do anything and there's optimizations to be done there same thing with page views right how to make the screenshots more interesting then there's the install step you can't do much but you know people sometimes

don't click on the icon after they install the game then you have the tutorial stage is the tutorial concise enough in one of the games i worked on the tutorial had something like 12 pages of slides that you had to click through we lost about 50 of the players per slide yeah not a lot of people actually ever got to the actual game things you learn and then obviously there's retention and conversion and eventually if you try this hard enough eventually there's money at the end of this funnel and then when freemium is in full swing i would say like let's say five years ago we have this whole additional thing on top of the pyramid or the bottom of the base which is the ua version of it you know how do you make the ad interesting enough how do you optimize the click-through rate is it the right people you're acquiring and so on similarly and this is weird because when i was still in college i was interested in already games and the concept and amy joe kim had a has like probably

one of the more famous com presentations online about player journeys if you will or and i think she was more interested in the gamification concept but she also is studying this thing of you know how do players progress to the game and she focuses specifically on the stages of you know when you start a game you're on boarding and you're still very young and if you build habits you might become a pretty casual player of the game there's always a chance that you drop out or you churn and you never come back or you stagnate and you stay the same or you may become a master and you really like become a real enthusiast and you know go to blog posts and forums and comment about stuff and it's it's actually pretty close to what i've come up with for what i think player journeys are but the most interesting thing is that it starts at the lower left and it goes to the top right and it's the concept that you're going

from there's a starting place and there's a golden path that takes you to the ideal player right everyone wants a game full of enthusiasts that have mastered the game uh using her words and then the last one i promise of the player journey styles that there are in the in the world is the future model i think this is like the real liveops version of all of this that i've just said right it's this concept that which by the way curiously enough pop became popular in google i found interesting in 2012 which is just roughly around when amy joe kim is becoming very popular and with her gamification work and her research and consulting and ever since has only been increasing very steadily and again it's the for for the audience i'm sure this is familiar but it's you know the concept that you have a tutorial that you have to start and there's some milestones there might be multiple milestones eventually you get to the end of a tutorial that's kind of cool

but you got then you get the first experience of collecting rewards you might be able to start a new mission you have some mission milestones and you put essentially these little breadcrumbs through the entire game in order to track the progress of a player and check that they're on the journey that you've designed the game to be right so if you're going from new york to san francisco by car this is basically here the key stops and the key motels and gas stations you're going to have to hit in order to succeed on this crossing the analogy probably works better with oregon trail but it's the same thing and eventually they level up and eventually there's like this opening of the world right like once like in games when they're young like you have a very strong initial experience and eventually the game opens and you're free to do whatever you want and i i find it interesting how there's like always very different expectations of what the player is going to do

let me pause here right and i think for by now you can see okay there's many different ways tracking a player's progress or a player's journey there is and each one has its own benefit i'm sure everyone has seen one of these graphs before in their company they have seen some application of this you've probably also optimized one part or another of this like if you're in aso you're very familiar with icon optimization and how that affects the journey of the player the same way if you're in ua you also realize that the creative and the impression that the click-through rate has a direct impact on the the end of the journey and the end of the future so one of the things that that is interesting to realize is that at some point the map appears to be to us more real than the land right like you very quickly if you look at any of these diagrams i just showed you you're allowed to believe that this

is the path this is the journey and this is kind of what i thought was most daunting about this presentation of mapping player journeys is the concept of there's a single one right or that out how do i condense all possible things that happen in the game into down to one graph to show joe kim and his and and this you know virtual conference and so this is actually then i realized no i actually been doing this for a long time for more than seven years and this is my interpretation of player journeys and it comes from a different perspective it comes from the perspective of having worked with never a single game but have always been working with dozens of games at once how do you find a player journey or a system of map making that applies to all of them right because everyone has their own problem and they come up with their own visualization in my case right we were dealing with a normal company you know might have a normal developer might have a single game

and they have this player journey that associates with uh but i never had that i always had this like holy crap i am a single data scientist there is like tilting point is a good example there is 40 games in the portfolio each one with its own taxonomy of events the the different user flows different fatus different funnels different devices platforms how do you come up with all of that data into a single map of a journey and this was no different back in the day when i was doing gondola and dynamic pricing how do you do dynamic pricing if you don't understand the specifics of the gamer end right it's almost impossible to do it and so what i've come up with is a slightly different way of getting to know a game and understanding how players behave in that game that uh highlight some of the things that you can do in the game now this will sound extremely familiar

to joe kim's rfm and i swear to you we have not spoken in probably four weeks i didn't know he was going to present today but it's really interesting how it coincided and i've also never heard of rfm so the combination is fascinating and this is how i start when i get a new game on on my desk right i look at okay pick two dimensions days let's let's go look at days of retention and some like major level progression like level one and then if i look at okay there's like x number of players in this state if you will who they ended that day at being a level one what's interesting then is to look at okay how many of the players who were in this state in day one moved to the next state the following day day two but they were still in the same level basically they opened the game one day they closed it they opened the second day they closed it without making any progress it happens it's freemium and there's like some amount

that's why it's slightly lighter colored now there's a lot maybe that play a second day and they actually go to level two and they finish the day with level two and there's always like the over ambitious ones that we hope are the classic case you know maybe they finish the fatoui road map or plan and they are boom they through level three on day two and you can do the same thing again right there's other players who in day one get all the way to level two and the next day you can map out okay where do they go the next day and you keep doing this and and this is how i try to understand the player journey but this is zoomed in this is zoomed into a ridiculous level i'm going to talk like for some of the slides based on one of the tilting points games linguini's because i've known the game before even working at tilting point as uh for trying to understand it and so i've actually learned to learn about this game from an outsider and now having on the inside it's kind of fun to see

how it really works but for those who don't know right linguine's is a match three kind of game plus there's a word finding component like any official freemium game you've got a whole bunch of different you know missions and scoring points and a limited number of moves and you can permutate all these options to create levels and there's a whole bunch of boosters that you can buy and you might be running low of them there's a scoring system you know it has all the number of major kpis you could be interested in analyzing and if we do the thing i was just telling you guys about earlier right you might have days and you might have levels and boom now this is how players progress in linguines every day some players in d1 level one my drop pal let's face it most drop out right here but like any normal game that's free but you know this is how actually players go and this is not a fatoui model at this

point this is a player journey that captures every player in the game and immediately you you can start to see opportunities for optimizations that otherwise you wouldn't see because in a normal future model you might just be focusing on the direct diagonal i've always called the direct angle like the golden path in a game that that's where you want all players to go ideally this would be just one strong dark blue line and anything that varies from it is an opportunity for you to ask the question of what would it take to move like this slightly darker blue box up one notch like what could i have done like what what missed opportunities were there in this connection and this is where like to me machine learning comes in to figure out what was that what can i do if if i can give anything in the world to the for this to happen what can i cheat my way through this now if anyone is about to try to do this then with their data the very first graph you'll ever make and i made this graph early for

a couple months without success is that you plot the number of unique users in every one of these boxes in combinations and you get this and that's because in freemium the first day is 90 of your players most players never go past the the home screen and nothing looks interesting and the the trick of getting this to look more reasonable for understanding which again goes back to mapping and it's made for a very specific person data scientist in this case is that you have to essentially normalize it on a in this case per day basis so that you have a distribution of players so that every column becomes you know out of all players that were in day two where were they distributed across in this case levels and and i'm i'm talking a lot about player states player states is really important to to understand because this is a concept

that doesn't happen too often in gaming is first of all you can pick any two of these things on the left or any other ones in your game and you're going to get an interesting graph i promise you the second thing is it's very rare for game developers to actually record the full state of a player at every given second of time in the game i mean you don't have to do second but you every time there's something meaningful happen most often in events based systems like amplitude or anything else what you're looking at is just what happened on that event like oh they started a level great but it doesn't carry over all the other information about well to this point how many levels have they won how many times have they checked out the in-app the iep store how many boosters have they used and that's that's the pivotal part to making these graphs so a couple like quick troubleshooting notes here about these things because i'd probably spend way too much of my time in the last seven years doing this is obviously the hot spots are the most common one

one of the easiest ways to like just get past it to see if there's anything interesting in the in the journey that you're trying to map is to just remove a good chunk of it out and then rebalance the the colors to work out and all of a sudden you get better contrast the the funny thing you have to realize that when you do this is that you're probably looking at a fraction of the players now so my my you know dream of saying how what it would take to move the player from one box to the other that's great you just move seven players from here to here roi after one day's work in manhattan not that good so you have to keep that in mind the other one that i was talking about earlier right is player states never seen a game that when you get the data from the database you get the full player's date you have mostly missing data and you have to somehow triangulate what was the full players the full player state at that time to get a prettier picture and one that actually tells you what was

going on another one is dimensions and this is how actually i learned a lot about games and the way i start talking about games is not having a seven hour conversation with the game designer about the game where they dump me all the knowledge that they know of the game and the design so that i don't have to figure it out or read documentation instead i try to do it this way which is to just pick random different ways of measuring progress in a game such here a number of booster super boosters the player has used and the number of levels they have unlocked and then you get this line and it's like okay they've unlocked every level with no boosters used and this is like where you start to understand ah super boosters is something that only happens at a level 100 and right here i'm only visualizing up to 99. and that tells you and so i keep track of all of these findings and then when i go to a game designer you you have an idea okay let me debug some things here that i think i've learned about your game and let's see if we're right and this is how you also find bugs in

the data as well so a couple of quick things you know about applications and so on about this saturi obviously so in a different game from tilting point where we have levels and euros that you can unlock fatoui is very obvious uh you can't go to the next level without another hero and for the first three levels you get three heroes what is interesting though is the light blue colored stuff around the future and this is like to me this is like a great way how this kind of player journey helps you debug some of the things that are going on in the game sure this might be like five percent of the player base that's not going through the future model or plan game plan if you will but this is also the opportunity to make the game better right if you if you get those players back into the into the future track those are potentially big wins down the road when they convert and they become spending players and so a lot of optimization can be done

here into figuring out is the tutorial broken is there something is there a way to for the player to just do the wrong decisions do they not understand that they can add a hero to a level and so on another one that's especially interesting now with the ios 14 is attribution picking a different game that i found this is how the players develop over time in their player journey and these are organic players if you look at this one it's the same game same data but this is coming from facebook and that change is 18 and you can keep doing this and you and you keep looking at different ad networks and you keep finding that you can associate it might be by the targeting of course of those networks but you can definitely determine organic versus non-organic players from a combination of these jordan mappings

now how how can you use that for something else well that's hard to know with ios 14 but it tells you that there is a noticeable difference in journeys and how many people have different futures for instance based on attribution right that's not so much heard of and with this model i think it brings it up immediately to you and you're like okay now this is an opportunity to to see well clearly the facebook people probably spent money around here and they didn't here on the organic side so maybe we have to give them a little bump a little something to to move them up to wrap this up because i think i'm kind of running out of time is a couple tidbits for the future things that you can do up to here you can do with one data engineer and a data scientist for any reasonable size game here is this is the part where it gets a little more martial artsy what i was doing and and

joe mentions a couple ties is you and i think he mentioned it because he did multiple dimensions with rfm right every rfm is one a dimension and you can start looking at players and player journeys now in three dimensions not just in the two that i had what is the difference between how players develop over time based on the day of since install for instance or session or length of time playing the game versus levels unlocked heroes unlocked number of ieps purchased etc the more dimensions you add the more interesting these pictures get but also this is where mental capacity and brain capacity really like for humans kind of peaks and it's kind of hard to do and and so that that's a whole topic in itself the other one then you can use the the three-dimensional model the multi-dimensional model for and joe also mentioned is clustering uh once you get

multiple dimensions gondola started with k-means as well boom the beauty of doing k-means for for this kind of analysis of player journeys is that immediately you get much better chunks of player bases that you can do a b tests because if you're looking at any one of these boxes up here in the top right there's not that many players right i mean the first left lower left corner you have 100 of your player base but when you get up to the top you don't and it's very hard to say like i want to move this player base up a box how do i do it with an a b test you can't because there's not enough players most often but if you do the clustering you have a solid base and you have the constraints of that segment easily that you can plop into something like loom plum or another tool and as the last thing that joe also mentioned in his stuff which i thought was kind of funny is chains right what are what what can be

said about players that go through a very specific chain of clusters or a chain of in their journey where they stop in specific points the a lot of i think liveops does this already in a lot of games i've heard before and that is you know well offer something to a player at a price point which is you know the average of the last three plus maybe 20 see if they convert on a little higher package if they don't then knock it down one fundamentally it's the same thing as this but that's without the understanding of where the players are in their journey you're assuming that there's a direct diagonal and in this system you you don't do that in this system you're always aware that you're you're you might be focusing on a specific group of players but they are other types as well and you should be aware of them because that's also an opportunity so to kind of wrap it up player journeys it's moving past the one dimension and

they're not moving in one dimension so fatouis funnels uh player flows etc uh core game loops they're always focused on a single you know line of movement where they go from top down left right and i don't think that's quite so easy it's not like that at least not from the data science perspective that i have my perspective is that there's an infinite number of possibilities of movement and if you zoom out and you see all of them you you may see the one that you're thinking of the one that's not just a straight diagonal but if you look at the other ones actually there's just as many players to play around with and optimize and do something with like limited time offers different pricing especially different pricing because the people who are not in that optimal line are the ones that usually are struggling to get through the game because the game wasn't designed for those players and so it's a really good opportunity there there's an infinite number of journeys as i said in every game

there's not just one there is definitely something to be said about you know how do you use journeys to understand where the players come from pre-install right or or from like where is the source there is definitely different players and you can group these journeys separately and you can also separate the journey into multiple different journeys and right now we have the beauty of being able to actually have uh some test data to understand this in the future we won't and it's really exciting to see how that's going to play out and finally this kind of player journey opens up a box of ap testing possibilities that otherwise wouldn't be there in anything like a funnel or any fatoui system your your moments of optimization are either on the bar graph on the bar itself or on the transition between one bar and the next right like from starting tutorial to ending tutorial you can either fix the tutorial or somewhere there's a transitional screen that got screwed up maybe you had

an interstitial ad that threw people off in this player journey that i just showed and how i understand player journeys you realize that there is an infinite number of these little bars and boxes and lines of transitional periods and all of them are potentials for really big upsides so with that thanks in the first generation of mobile gamers came off consoles from titles like super mario warcraft and league of legends now as we enter the era of mobile 2.0 with three and a half billion devices globally gaming has mainstream easy to play wherever you are and something for everyone the categories have proliferated the best gaming companies in the world need deeper insights a framework to capture the market opportunity and now

introducing game iq a revolutionary analytics product that delivers a taxonomy of genres sub-genres and comprehensive feature analysis matched with app annie's legendary market data you have the insights to level up your performance designed exclusively for the rock stars of mobile and by the way i'm gonna i'm just gonna go ahead and say that based upon the audience we have here and i'm seeing some of the companies represented and the games represented i do think someone in our audience if you take the rfm concept and some of the the the stuff that i talked about in addition to what andre talked about somebody here is going to make a few million dollars i'm calling it right now but uh i agree it's fascinating that like amplitude doesn't have any any sort of you know multi-dimensional graphing uh

it's right right yeah and even humans someone out there to build that tool and so maybe we can actually answer this question that's more uh machine learning related but i do see one question about you know any recommendation on how to start building a machine learning system any specific service software tools etc is it amplitude i will say i don't think it's amplitude they don't have something there but andre how would you recommend people start getting just kind of get started with with the team yeah no i think you know the you you brought up i think joe a pretty solid foundation for it you know with the between firebase bigquery as your foundation it opens up a whole bunch of tools that i think are better ways to start because you have to experiment and the cost of experimenting if you dive too deep too fast is absurdly expensive and so you know things like using periscope amplitude even

connecting that with some machine learning framework or sas business out there like picon i think is one they allow you to get a really big jump ahead in experimenting with the data and visualizing it without having to program programming it gets very expensive and i would say like yeah you could use amazon or google machine learning services but those are not plug and play that that requires a lot more effort okay folks glad you could catch this presentation so good to have you join us this was actually part of a larger digital conference around best practices in live ops so click the link to basically get the agenda for all of the other talks that were part of this conference and links so that you can watch the other talks as part of that conference also if you haven't checked it out yet check out appiany's game iq product game iq is a great way to get market data on the mobile gaming market and app annie was the sponsor of this conference and finally if you want to be notified of future talks in virtual future virtual conference

events please make sure to click subscribe to the game makers youtube channel and click the bell notification so you can be notified of future events coming up we got a lot of cool stuff coming so stay tuned and great to have you with us catch you later