Building a F2P Liveops Playbook | ft. The Games Consultant Matt Emery

hey everybody welcome i am here with matt emery the mobile games consultant and i've got a lot of questions matt about actually building a live ops playbook and i know you've got a number of playbooks or at least a playbook on iop merchandising things like that we can we can talk about that but as i was actually kind of scrolling through your linkedin i was actually quite impressed by the number the number and like some of the companies that you've been consulting for so i thought we could kind of start with that like notice i'm looking here uh blizzard boss fight people fun glue d3 i mean you've got like so many different companies that you've done consulting for but maybe i could first start by asking you you know in terms of what is it that you current do as a consultant right now and how did you land such impressive clients as you have

oh thanks for thanks for uh the question thanks for having me joseph yeah i'm happy to be here um so i guess the quick the quick bio uh is that i got my start in 2003 in the industry and i guess that puts me at about 19 years which is kind of crazy to do the math but seven years ago i founded uh this small consultancy turbine games consulting and we focus on helping teams to you know rapidly improve the kpi of their existing live games of course in the free to play mobile space and you know in the last seven years i mean it's been a pretty long period of time in game industry terms and so you know across those seven years we worked with 60 to 70 studios um including plenty of smaller startups as well but also a few bigger names you'd recognize like the ones you mentioned you know glue uh sony wb blizzard big fish um but yeah happy to dig in and talk about live ops today and you know it's certainly uh starting out with a small consultancy you typically start with smaller names and you know

you you learn and grow and and prove that you can that you can move the needle and then gradually you know you migrate into some of those bigger names so it was it was really just a function of uh you know having seven years uh to to start to build the skill set and the and the knowledge base and also the team um that's why you know we have some of those bigger names on the client list as well all right matt well let's jump into it i mean i think live ops is kind of a broad topic and i know that you um this but this is kind of your your area your your kind of area of specialty and so maybe i could start kind of with a broader question like from a high level when you when it comes to live ops and let's say a client's asking you about a game i've got this game that we're operating we want to improve it i don't know how do you think about whether it's like you know what what to attack in terms of whether it's features content events sales like you know you're kind of coming in looking at a game what's your thought process sure no it's a great question yeah

you're right live ops is super broad that's part of what makes uh running a game's business so difficult and challenging is the million things you need to pay attention to and that can go wrong um i think when most people think of liveops though and and it's sometimes myself as well like the first thing that probably comes to mind are you know like the the calendar of events and sales uh things of that nature and that's definitely valid um i think at turbine we take a slightly different perspective which is that we think of liveops in terms of two distinct lanes of work right two distinct lanes of effort each serving distinct goals so in this model lane number one is work specifically focused on the retention of your elder players okay right so here you have you know your end game content adding new levels to the end of a match three game maybe you're adding new features and mechanics uh intended to keep things fresh and really just to prevent your elder players from ever getting bored so

pretty straightforward that's lane number one right don't let your elders get bored and leave lane number two is and that's really where we specialize at turbine is continuous kpi improvement and at its core you know it sounds like a buzzword but what it means is that you're driving user lifetime value up and you're driving cpi cost per install down for each monthly install cohort relative to the last what that means in practice is you know on the product side if if this month's installers come in and install your game right what they're encountering is actually a superior more efficient more well-oiled retention and monetization machine like a better product than what the previous month's installers arrive to and why is this important i think there's a prevailing myth and free to play that you know what if we're in free to play mobile games we must be making money hand over fist and we're greedy and we want to shake our players

down for every penny they have but the reality of course is more about survival honestly and what is an incredibly challenging and competitive ua environment you know cpi are very high and so acquiring users profitably is very difficult and on top of that uh cpi have this nagging tendency to creep up over time so even if your game can acquire users profitably this month come next month your kpi will need to be better than they were last month just to keep up with the rising cpi and you know there's a lot we could maybe later get into mechanics of why cpi go up over time but suffice to say you know the best performing companies we've worked with spend a lot of effort um dedicate a lot of resources to continuous kpi improvement and at turbine that's where we like to focus because that's where we found we can have the biggest impact and so there's kind of a few things i'd really like to try and unpack with you a little bit but maybe just starting with the the smaller

bit so you understood you've got your your kind of two-lane framework model um the first lane is the elder player maintenance or making sure that anything you're doing isn't affecting those guys and so and now shifting my questions to your main focus the second group where you're trying to do kpi optimization my guess first question i have is you mentioned you know you're trying to do a basis of evaluation on a monthly cohort basis and is there a reason why you pick a monthly time frame versus like a weekly time frame um i i chose a monthly time frame just i think to make it easier to have this conversation i think it's a little easier for people to think that way or maybe that's just me um when it comes to uh the work that we do you know we're focusing on um we focus on cohorts that are associated with certain split tests and client releases which tend to be roughly monthly but not exactly monthly okay but i just think in terms of illustrating

you know the reason for the concept yeah monthly is just like a nice you know a nice placeholder for people so situational but you know but i i get it okay yeah cool and then um when it comes to um and i'll get to the cpi question later but just also getting to like this stream of work um and so when you do like you kind of hone in on certain kpis and you're going to be trying to drive those kpis higher improve those kpis and from when when you're thinking when first when when you're you know when you measure those kpis um what goes through your head in terms of what to do so let's say you know let's say and i don't know exactly what you're looking at whether it's like you know d30 arpu or you know overall ltv or retention or things like that but like can you mention like maybe some of the metrics that you're looking at and then when one of those is low what goes through your thought process are you doing like ppid construction or are you

um i'm just trying to understand how you then think about okay this kpi we're trying to drivers or is is too low relative to industry comps or something like that and so then what what do you what what is your thought process or how do you think about attacking it whether it's from features content sales or any any other stuff that you think right sorry yeah no it's the right question i mean i think perhaps interestingly we we certainly look at a games kpi and look for problem areas that could give us ideas for optimizations but we don't focus a terrible amount on benchmarking okay a game against other games we actually we we haven't found that to be as useful as just looking at the game itself and looking for low hanging fruit looking for easy wins to improve any kpi that we think is movable with with reasonable effort so you know we yeah we don't really try to keep up with the joneses per se because a lot of games you know are very very different they may be acquiring from different user bases like there's so

many there's so many differences that make benchmarking a little bit less actionable to us than just focusing on how do we drive your game's kpi upward with the minimum effort so i can you know i can get into that approach a little bit more maybe i can go through our our process at a high level um so you know we take i guess what i would call it pretty intuitively an roi first approach okay right and in practice what this means is that we keep a backlog or you can imagine basically a spreadsheet with all of our initiative ideas for a particular game you know all of the things we think of that could be worth doing that could ultimately drive ltv which is what we're looking for if we're focusing product side and so we'll produce an impact estimate uh that rolls up to ltv for each potential idea or each potential initiative and then we have you know in this hypothetical spreadsheet another column where we enter our predicted effort on a one to ten scale and so then you can imagine this spreadsheet where you have a you know you have a list of initiatives you have

a column with all of your predicted ltv lifts you have your column with your predicted effort numbers from one to ten you divide the ltv column by the effort column that would give you a a relative roi score that you could use to sort the list you know in a very basic way so yeah pretty straightforward but there are several benefits to prioritizing that way right first and foremost um sorting by roi services low hanging fruit which is a term that you'll you'll hear me abuse over and over again because we you know that's what we love to do is find low hanging fruit and those will be of course modestly scoped initiatives things that would normally get swept under the rug or often neglected and passed over in favor of larger sexier initiatives and you know huge game features i'm sure people can relate to that another benefit to this approach we found is that you know when you build a discipline of forcing yourself to produce impact estimates you know ltv impact estimates and then you can obviously check your predictions against the actual results if you split test then

you start to build your prediction muscles that's probably the only way to build prediction muscles ultimately at least the only way i know how and then finally when you take this approach you know after working with working on enough games you're building a vault of past initiatives and their impacts that can be usefully applied to future games so at turbine you know we have a vault with hundreds and hundreds of split test results by this point from our our favorite initiatives that we like and so you know every new test we run it feeds our flywheel it makes us a little bit smarter and it helps us be better equipped to help our clients place the right bets on which initiatives can drive the maximum impact so you know we that that is kind of the process that we follow of course some of those initiatives that go in the list are informed by kind of a top to bottom audit of the product you know looking at their metrics and so on looking for weaknesses things that stand out that seem you know unusual that might reveal opportunities um but it's more holistic than that you know it's it's really just here's all the things we could possibly

work on and then let's sort them by roi and attack from the top down that's the simple the simple version got it okay yeah um so yeah it sounds sounds kind of like um a little bit like zynga expected outcomes or ice rice type model but sounds like you're really orienting around impact to ltv as your as your ultimate driver is that correct uh on the product side yes um similarly you know we do we do some work uh targeting cpi as well you know in like app store optimization and sometimes we help with uh user acquisition but most of the work we tend to do is product side uh and so yes on the product side we're you know we're targeting ltv that's that's the downstream metric we want to hit um we might get to it through retention or through engagement or through monetization but ultimately like the only point in touching any of those metrics ultimately is to get to ltd right yeah yeah in in terms of like so you mentioned um that you come up with a bunch of

initiatives that you try to determine your your impact or your your roi against and i know there's there are a number of different approaches in the industry in trying to come up with these kinds of initiatives like one could be more of a zynga based you know kind of market based model where you're looking at a bunch of different games bunch of different features from other games and then you're you're trying to see if you can adapt those back into into your game but for you when it comes to coming up with initiatives um whether it's just brainstorming stuff or just based upon your experience having worked on so many different games before and seeing you know seeing patterns or um what's your approach in terms of actually coming up with the specific initiatives that you would potentially you know put into your prioritization framework yeah it's it's pretty holistic right we pull from several sources um they'll probably sound pretty obvious but uh one of them like you mentioned that's super important is being familiar with other games um if you see a feature or a mechanic in the in a top grossing game then chances

are it's past the gauntlet of split testing you know it is uh it's there's some positive survival bias there so you can you can you can assume that something that you come across in a top piercing game is probably a mechanic that at least served its purpose um so that's a really really great source of ideas you know there are a lot of smart people out there working on a lot of games and so you know i i like to say that pretty much everything not everything but for all intents and purposes almost everything has been tried and and you can find it out there you can find the things that work for people um so other games is a great source you know we also have uh we have our own hypotheses our own pet features that we like uh solutions that we like and you know we we split test them across multiple games and see which ones work which ones don't so we have our own kind of vault of design concepts and wireframes and features that that we found to work some of those initially came from or were adapted from what we found in other games

but yeah we have that kind of second column which are things from our vault and our design backlog and then uh you know there certainly is room for boutique solutions to unique problems as well you know so it's uh it's not all bringing things from other games or from templates but for most for most design problems and most you know most optimization needs there are solutions out there that have already been tested right like free play games are there are differences but they're more the same than they are different in a lot of cases and so you know yeah a lot of it is just kind of pattern matching solutions to problems right and then in terms of like the implementation so you come up with an initiative and and you've kind of referenced split testing quite a quite a bit so it sounds like the way that your gauging impact would be through some type of multivariate test and um is that right just just just to clarify yeah essentially yeah we like to use the

term split testing instead of a b because we we typically have more than uh yeah more than an a and a b yeah um and and so when it comes to um you know split testing multivariate testing you know i think there are a couple of philosophies um it kind of feels like to me a lot of the folks who've jumped from zynga are more in the camp of not not actually running tests to stat sig right where they just kind of want to get a feel and because you know at zynga you're running so many tests so often and even at the scale that a lot of zynga games are at i i think it becomes you know um uh you uh you just need so many users uh to to test everything but um do you have a specific philosophy do you um are you kind of like one of these hardcore stat sig guys or do you just try to get a feel for how tests are kind of running and kind of run with that or what's your philosophy on that side

yeah yeah i mean i think anyone that runs a test looking for an outcome wants to you know wants to find a statistically significant result okay right there's that's always the goal um but i think there's this uh you know statsig is often treated like a binary yes or no uh result and and that that kind of it's it's less empowering to look at it in that way right there there in fact is a you know with p values uh there's a continuum of confidence right and so you know what we do is we we always want a statistically significant result um we often get them but we often get things that are close to statistically significant or like halfway between you know random noise and statsing to put it simpler you know we we calculate a uh we calculate a confidence percentage that we use for all of our split testing and so you know sometimes we'll get 80 you know 80 confidence that the effect is uh

is you know real versus um due to sampling error so you know we we take we just take a numerical approach with it and if we're ultimately replacing bets right we look at the data that we get um we you know we calculate all of our statsig in uh we do all of our testing in in a spreadsheet so we can look at the raw numbers and you know dive a little bit deeper into what's happening so we can make judgment calls ultimately though we're just getting the data that we get learning what we can and then trying to place the right bets on what is most likely to work versus not um rarely do you you ever get 100 confidence in a test but again it's all about velocity and making educated bets based on the data that you get so you know if you see 80 chance of confidence on a test then you know more often than not if we don't see any negative impacts to be concerned about we'll we'll try to roll that out and move on to the next test yeah maybe um another question i can

just go in like slightly tangential for for a second but um one question i have for you because i i you know i know your your work or you you mentioned that all of your work is on live games and you know i have a philosophy that new game development that practice is really different from um you know live a live operating game and i i was wondering if you could speak to um because it seems like you're probably you're operating in that environment for a reason but uh for you do you have similar beliefs in terms of those two practices being different and then what what would you say are the differences if if you do have that belief oh yeah i totally agree i mean i they couldn't be more different um yeah we we these days we don't work on new game development as much just because we we really like to have um you know real data and be able to move kpi and have the satisfaction of seeing things work um that's a little more difficult to do in new game developments i guess the way we see the world is new

game development and live operations they stand on opposite sides of a critical gate which is a credible demonstration of product market fit right um and in new game development uh i think a lot of people probably view new game development as like oh we're making a game but in free play mobile i think it's more effective to view it as a process of searching and testing many game concepts hoping to find like the right game concept worthy of promoting to live ops so it's more like a search you're looking for an unlikely alchemy of like two things that are individually hard to find but even harder to find together right which is a highly retentive game experience paired with um you know outstanding high converting ad creators that you bring in those high retention users at a highly competitive price and finding such a winning combo is is really difficult to do we don't focus on that i have a lot of respect for uh product leaders in the hyper casual space who probably bring like the most discipline and rigor to this process

because that's that's what they do every day um but regardless you know however you get there once you've demonstrated you can acquire users at a competitive price and also retain them long enough to hopefully support profitable long tail monetization then um at that point you can responsibly enter live operations where you're looking to scale this this hopefully profit machine and so that's you know that's when you get into liveops and then you're looking at you know if you're us you're looking at our two-lane model that we talked about before um and we just you know it we we enjoy the continuous kpi improvement um uh world just because it's very gratifying it's more templatizable um it's more scalable and it's just we just enjoy making numbers go up that's difficult to do from the the you know the new game development side is a little it's kind of using a different part of your brain it's a different process and and it's really hard next question i have is really around okay so many i don't know if you remember like a number of years ago there was

like this legal case because i guess some folks from zynga left zynga they joined platon and they apparently took the playbook the zynga playbook and and so and you know i i don't i i mean nope there was never any you know determination or at least publicly disclosed information about what does that mean the playbook mean i've kind of heard it was and i don't know if this is true or not but like the rumor was that it was just like a powerpoint with a bunch of best practices in terms of free-to-play um game game you know optimization and things like that and so you know for you and you've kind of developed a few playbooks around iep merchandising things like that but if you were to build your you know matt emery playbook that is worth millions of dollars as as you know as zynga sued platon for what would you put in that playbook or what what are some of the types of you know topics or best practices that you

would think about including in such a playbook sure yeah definitely a broad question um if you ever find that playbook uh dmv if anyone from zynga is out there who can send it to us please do i'd love to i'd love to get a walkthrough so you know if if if this is a developer publisher you know hypothetical like developer publisher studio playbook then it would need to cover all phases of the game lifecycle right like we talked about new game development and then also the different parts of live ops um and so you know again like the area we focus is just in that one you know primarily we focus in that one aspect of liveops which is continuous kpi improvement so i can go into i can go into our playbook at a high level um yeah by the way i think zynga's new playbook i think that that part was broken with you sorry bad joke sorry anyway go ahead

i got it i mean the industry is always changing you know playbook can uh can stand still um that's one of the things it's fun so um in our process uh i guess there's a core loop and a metaloop if you will um the core loop starts with an idea for a specific initiative and it ends with of course a live split test of that initiative in most cases and there are four major steps to get us from here to there so first step is to prioritize and choose which initiative to work on next and again we we take that roi first approach based on kpi estimates and um heavily informed by our you know historical vault of past tests and results and the goal here is to wherever possible pick the thing or always be working on the thing with the best possible ratio of impact to effort so once you've identified the initiative the next step is of course the design phase pretty straightforward you know we created a design dock for each initiative as well as a design for the

parameters of the split test because that's you know just as important and at this phase we also have a design library i mentioned before you know full of templates and wireframes and specs that we've used in the past then the next step is you know building the thing building the feature or initiative pretty self-explanatory and we have some process in our playbook here for how to you know reduce error uh in translating the designs into production code and art and then the last step is a big one which is split testing uh it's an area of particular focus in our playbook as there are uh you know anyone who runs blood tests can probably test this there's a million and one ways for them to go wrong and only a few ways for them to go right and so we built process and a lot of guard rails for making split testing you know as efficient and repeatable and fail-proof as possible so that's that's the core loop and over top of that we have a metastructure which is basically you know just a road map for how our teams collaborate together to turbo through as many of those core loops as possible over four to six months or more while also deploying and

realizing any gains from split testing along the way so that's you know from wide aperture that's that's what our playbook looks like and of course it continues to evolve all the time as as we do um and maybe like another question you know you've worked with so many clients before you've had you know a lot of experience working in a lot of different concepts and so um so when it comes to like kpi optimization or your playbook maybe you can talk about like i don't know uh some of the biggest mistakes or biggest learnings that you've seen from some of your client engagements or just sure you know generally if not if you can't speak to some of your client engagements yeah no i don't have to talk about specific clients here i mean we've made all these mistakes as well ourselves and learn from them and that's that's what the playbook helps with um so i guess the first big one will sound pretty obvious when i get to it is just not taking an roi first approach

if you don't estimate roi and prioritize accordingly there's again this tendency to neglect low-hanging fruit and instead just build big juicy features um you know big juicy features are great and they certainly have their place you need them but if you do neglect low-hanging fruit then you're just not getting the best bang for your buck and that puts you at a significant competitive disadvantage which you can't afford to have these days in this market um so you know don't neglect low hanging fruit that's that's the first mistake another surprisingly common mistake is and this one is is pretty relatable i think to a lot of people um i've been surprised how many product managers and designers don't play enough tough grossing games right they don't regularly play their competitors games as well as top raising games from other categories with the critical eye and as a result there's this tendency to reinvent the wheel or unnecessarily invent untested novel solutions to design problems that aren't novel problems

you know like i said there are a lot of mobile games out there a lot of smart people have worked on a lot of things and so almost any design problem you can think of uh you know for all intents and purposes you know like how do i get players to come back more days of the week or how do i increase average purchase price these are like red ocean design spaces where tons of things have been tried by a lot of smart people and mature you know heavily battle tested solutions have emerged to become best practice so you probably don't want or need to be in the business of showing up late to the party and just throwing a new dart at that dart board right like use your creativity elsewhere um i know i rail about that a little bit in soapbox a little bit in some of my articles but that's uh you know it's one of the ways we help our clients too is is just by having a broad perspective and having played and worked on a lot of games uh from different categories and so we frequently will cross-pollinate um or import mechanics from one category to another you know for example from casual to decor or from idol to arcade in and out of casino and so on

so yeah i definitely encourage pms and designers to play a lot of games uh if it's top grossing like you should be familiar with it it's just it pays such dividends to see what other smart people are releasing you know typically after split testing uh they're just it's just a fountain of good ideas that could be put to good use yeah i i'm actually surprised at how often that happens i remember you know some cases where people are thinking about this prototyping like yeah we want to make this prototype and i'm like you mean like this game right here and he's like you know it's understandable like i you know i can't i can't claim that i was never that way myself right like i you know i came from a design background and i you know like to think that like i'm smart i can come up with new ideas like this is what i'm paid to do um so it takes a little bit of an adjustment i mean maybe it's maybe it's uh coming from being a product manager and taking this roi first approach like you you sort of become more humble once you realize like

how big the industry is and how many smart people are also trying novel ideas um and so you know it it's definitely understandable why people want to reinvent the wheel sure and for maybe some of our audience uh like newer product managers or folks kind of getting into this if you do you have any recommendations in terms of like specific areas or best practices that you've seen drive big impacts that they should that you would recommend that they start looking at or starting to study so like um specific examples of of like uh high roi areas that we like to target um yeah i can give a few examples so one area we love to explore is um difficulty in econ tuning you know we found that aggressive tinkering or balancing here can have pretty profound results with only modest effort and that's what we love to find is you know these low hanging fruit type initiatives specifically you know we've seen ltv lifts as high as like 20 for

you know basic sources and sinks tuning similarly you know we've had ltv lifts of up to 10 for just difficulty tuning um a lot of teams have done some difficulty tuning but when we show up and and repeat it we tend to be a little more aggressive than people have a you know baseline tendency too because we like to take big swings in our split tests and see what happens um but regardless like you know 20 ltv lift 10 ltv lift these are the kinds of lifts that you would you would hope to get if you're lucky from you know a huge live ops feature that costs over 10 times more to develop so these are the kind of things that we like to look at you know anything that's a tuning test that can have a big impact is you know going to be very high roi if it works you just have to be careful about you know downstream impact another juicy target area for us is app store asset testing and you know this is something many teams are aware of and many teams do but probably not aggressively enough so testing new icons new screenshots google

play feature banners is a little bit less lesser known one um each of these of course requires like zero engineering because it's just art asset testing and development but when you find a new winner you know if you're testing on google play experiments or apple's new equivalent uh feature if you find a new winner each winner like each each type of asset can lift your install rate by two three four we've seen ten percent even higher um and so what we suggest is that at minimum teams test five icons five screenshots five feature banner concepts every month and basically never stop right the the roi here is so profound that when a winner is found it's like it's such a no-brainer that it's really hard to over invest in these areas and so yeah we really like we like aso testing and many teams obviously already focused there but again most don't do it enough not as much as we recommend and then a third area of focus uh that we write

about a fair amount on linkedin is uh ifp merchandising like you mentioned joseph and this is just another area where you know you're leading a bit harder on art than you are on engineering in most cases and yet you can still drive 10 to 40 or even higher lifts and ltv so you know anywhere where you can focus on you know lean into art resources and a little bit lighter on engineering resources uh that is very friendly too to uh you know the resource constraints of most teams and so that you know that any any wins you can find in those arenas are really really nice and so those are a few tips right and in that case you actually do have the actual matt emery iap merchandising playbook and i'll leave some links to your playbooks in the show notes but uh yeah um one thing you mentioned which i i thought i could maybe ask you a little bit more about is because you know at least in my experience and i'm wondering for your experience it's like this issue where you talked about aggressive economy tuning and one thing um at least

i've tended to see with especially newer pms was just like this kind of fear of driving too much change and like do you do you have any like guidelines so you know because like sometimes i'll see like you know whether whether it's a list of initiatives in in like expect outcomes framework or something like that more of a bunch of like you know blue versus you know blue versus green color yeah the color test do you have any like um i don't know like a specific guideline or you know kind of thought process behind how aggressive because i generally tend to be more on the aggressive side no no let's actually do something you know freaking crazy but i don't know what what's your philosophy around around just like the aggressiveness of like what you're testing or how much to test right like yeah yeah yeah i mean i think you're taking targeted shots right you're not just throwing caution to the wind and moving

things here and there like haphazardly right you have your in our model you have your list of hypotheses and so you're making surgical surgical tests but those tests should be pretty aggressive right um it's actually one of the other mistakes we see commonly when we work with teams is teams that have done split testing in the past and we like look at their past split tests to learn from them and and see where we can move things further there's definitely a tendency to be way too conservative in the changes that are being tested right like after i think this is just a product of experience right like after if you've had the experience of running hundreds and hundreds of split tests you'll learn pretty quickly that it takes pretty substantial changes to a screen or to a tuning value or a feature to drive a statistically meaningful change in user behavior right so don't think that moving tuning values up or down five percent is going to drive a statistically detectable input right instead try moving that value by 50 or 100 right right don't change a button color

on a single screen and expect a statistically meaningful result right you could try it you won't get a result and then gradually you know you will have more confidence to be more aggressive and then you'll start to see the changes you're looking for it's just you just have to try it i think and and learn to be a little more aggressive but you know instead of changing a button color i would say reskin the entire screen or multiple screens and make sure that the screens you're reskinning are like important high traffic screens that are critical to whatever kpi you're trying to push like an important point of sale screen or something um so if you make dramatic changes you'll get clearer and more actionable signal and less wishy-washy results from split testing and i think on the risk side you know the purpose of split testing is to allow yourself to fail to screw things up and to not tank your game right so if you have tests that seem risky and seem like they might piss a bunch of players off or you know tank your retention you can approach those tests in a very conservative way by just having them

target a very small slice of the audience and you can keep an eye on ratings just to make sure that you're not seeing any impact there um but yeah i mean if the point of this of the split testing structure is to allow yourself to be aggressive and uh learn and then decide not to rule something out if it sucks right well spending the time and getting no results at all is just frustrating in a waste of time yeah thanks for that matt um and so maybe i have one last question for you which is a final question you know do you have any final whether it's like an advice or recommendation or maybe biggest lessons that you've learned from liveops over your career yeah um so i think you know we learn a lot all the time we're always learning that's one of the joys of this job um one experience i had as a product manager that was particularly enlightening maybe maybe formative was was again running ua campaigns

um as a pm uh you know again i got to do that for a few clients and then for glue mobile on a few games there and that experience taught me you know two big lessons that i think kind of seep into all of you know my thought around uh my articles and you know our conversation today right which is one that profitable user acquisition is really really hard to do and two that cpi are always going up so you can't afford to stand still and expect to stay profitable right and it it might maybe it's we're taking a moment if we have time to dig into the mechanics a little bit yeah why cti go up sure um okay yeah it's not necessarily something that a lot of developers think about or have at the forefront of their consciousness and my sense is that you know on the development side i was in this boat um devs tend to assume that ua is just kind of happening happily in the background and it's probably going fine so i'm not hearing anything but the reality is on the front lines

campaigns like ua campaigns are usually just hanging on a knife's edge of profitability and not certain to pay back even after a year or so and of course cpi going up all the time uh makes things even that much harder and so why why does that happen right why do cpi creep upward uh for you know a particular game um first reason is what you could probably call like a maybe a golden cohort which people have heard of or or maybe a reverse selection effect which is that if you know if you run a bunch of ads this month um uh then the users who see your ads are and and the users who are most interested in installing your game probably will right and what does that leave behind a slightly smaller audience slightly smaller for your ads next month but who are also statistically speaking less interested in average right in other words every month you know you skim the cream off the top and the user's left over uh who have already seen your ads and chosen not to install yet are self-selecting right as harder

to get so that means every month you have to work a little bit harder to get those next installs work harder means practically that you have to deliver and pay for more ad impressions to get your next click and that means higher cpi so like your your target audience generally speaking gets lower quality and more dilute every month as you continue advertising so that's the first factor um second factor probably a little more obvious is that you know every month 500 new games hit the ios app store right so chances are a few of these at least are playing in your pond right there they're running ua targeting audiences that overlap yours and so basic supply and demand says your cpi is going to go up and then third your competitors hopefully are also doing continuous kpi improvement and by competitors i mean the people that are advertising to the same space the same users that you are trying to advertise to so if they're doing continuous kpi improvement then some of them are probably if you know if they're successful they'll be able to afford

higher cpi themselves relative to last month and that means that you too will need a bit higher and so your cpi go up so those three factors and probably others that i you know didn't mention or maybe maybe i'm not even aware of but those three factors um certainly conspire to put upward pressure on cpi and so because of that you know you can't afford to stand still um whether you like it or not and i don't mean to sound dire but you're locked in an ltv and cpi arms race with your competitors and so you need to continually up your game just to you know stay above the water line so these are things that really only became uh you know clear to me uh after running ua myself and so that's you know that's an experience i highly recommend any product manager try to get you know get your hands dirty running ua campaigns so that you can experience like this really experience the front half of you know the cpi half of the business all right matt well i really want to thank you for your time and insights and maybe one last thing is just if someone

who's listened to you today is interested in um getting in touch with you how can they do that uh like i said before linkedin is pretty much where i live so you can find me at matthew emory you can also email me directly at matt turbine.games great all right i'll also put a link to your linkedin profile in the show notes but thank you matt for your time and for our audience we will catch you next time thanks everybody thank you matt bye yeah cheers [Music] you