The Importance of Data-Driven Design in F2P

[Music] well that last debate was really exciting and I could like you said I could talk about this for days just pay2win versus play to win but a lot of Vladimir's points about gotcha based monetization and optimization I think is a good segue into what I'm going to talk about so good I'm Devin I'm the head of product management for the APEC region at EA and I'm going to talk to you today about the importance of data-driven design in free-to-play particularly in post-launch when we're talking about live operations and live feature development it's just critical that we use data as the roadmap to guide our decision-making so I'm gonna talk you know go over my journey a little bit first talk about a couple high-level concepts that are important to me some

success stories and what we learn from those and I think we will have time for some questions after is we have a drop out so I also won't have to rush through these slides I pack a lot in here so I think we may have some more time afterward to take questions and engage in some meaningful discussion which is great so my journey in free-to-play actually started on the M&A and private capital side raising money in capital for some of the early pioneers of the free-to-play industry back when they were at their very early stage after that I kind of meandered through the tech industry for a while even took some time to follow my passion as a pro gamer which wasn't terribly successful before landing at Kabam in the early days and was fortunate to be part of the web game days in the early sort of first mobile explosion stayed on in Kabam in what I called the second mobile explosion launching and scaling Marvel contest of Champions moving to Beijing to scale up a product team there and do basically the same thing in China before landing

at a ta where I'm based in thank you based in Shanghai now and my team focuses on the mid core pretty much the mid core segment of games with a bunch of different types of IP mostly PC online and mobile FIFA FIFA online and people mobile are two of our biggest titles but we've got a lot of other things going on to what alert has been a nice for us apex is something coming down the pipeline and PBZ has been something that has been a real surprise for us in China in terms of how strong the brand is and how long that game is been able six-six sustained I also lived through the whole battlefront fiasco and was able to help out the dice teams and sort of thinking about you know pay to win versus play to win and is it the right thing to do in a console business model versus a free-to-play business model I think it's a company we're still existentially sort of wrestling with that question of a pay to win versus play to win but you know over the course of about 12 years I've gone from someone who was really skeptical about the free-to-play

business model and really only like consoles and you know card games and RPGs to someone who's a true believer in the power of the business model and something that I think is gonna be even bigger going forward or talk to you first about a concept of the physics of free-to-play and I tend to think about you know physics because I think about you know dynamic systems that are governed by certain rules of motion and gravity and things like that and I tend to apply them to free-to-play and one thing I really like about free-to-play when I compare it to the console business model is you know how it behaves kind of as a dynamic system you know you've got acquisition and players coming in from paid and organic sources and viral sources you've got unfortunately players churning out and leaving the game and we're all just gonna focus on bringing them back in any way that we can but as players sort of move through that funnel to start paying and generating lifetime value that then you know feeds into the ROI calculation that funds future acquisition and it really does act as a system that's

always alive at the same time there are so many things happening in this ecosystem social connections are being made battles are being fought won and lost chats are happening different game modes are being played and this is creating such a treasure trove of data for us that the onus is really on and our players are really telling us basically what the gaps are and what to solve for especially particularly in post-launch when we want to you know launch new features in development and launch live ops and events and gotchas that are you know targeting our players that data is really showing us the path it's showing us what we need to solve for and the onus is on us and if we've got the right mindset and the right teams in place then we're really able to do crazy things in terms of growing and scaling our products the next concept I want to talk about is the concept of the North Star and this is kind of a popular catchphrase in tech right now find your North Star metric

and use it your North Star is basically the metric that really moves the needle for your games and your products so it's important to identify the North Star in order to make the right decisions but more than just a North Star I think it's important especially in gaming when you've got all this data in the game that's coming through and you're parsing through this it's important to break your North Star metrics into their constituent parts into their shards of the star so to speak so that you can understand what's really driving the North Star a lot of times for tech product especially in mobile retention is your North Star because these games are free-to-play you want to keep players in them as long as you can so you can grow LTV and it doesn't asymptote too soon and then LTV is another popular one which has a lot of different components to it right but breaking down these components and understanding the different things that drives them is really important in order for you to design features and design events that can then move the needle on these different things rather than just saying hey I want to increase retention or hey I want to increase LTV I'm gonna

I'm gonna launch an event okay well is that event the right one what component of LTV are you trying to drive is a repeat purchase rate it's a spin velocity the dark tau etcetera so these things are actually really important to understand early before you start to make those decisions let's walk through a quick example so let's say that you and your team know that you want to drive a 90 day LTV greater than equal to $5 right and you know from looking at your data that in order to do this you need to create a 7-day sort of repeat purchase a redeposit rate greater than or equal to 10% well once you've figured this out the onus is on you to start looking at your behavioral data and start to understand and dig deeper into what drives that right so maybe what you'll do is you'll think about all right let's map out all my players into you know on what day they actually repeat perks or if they ever did make a repeat purchase and let's map that against the different

game modes and the behaviors in the game of their playing you can do this pretty easily with something like heat map sitting you do this and you find out that hmm four players are actually repeat purchasing and this is dummy data by the way that they're spending an inordinate amount of time playing ranked synchronous PvP so that's curious but it's a good sort of guideline to say okay now the onus is on you to figure out how do we drive or increase the adoption of synchronous PvP ranked PvP in your game right and then also looking into why is that a driver right you could these can be qualitative data points or quantitative data points but it's it's not going to tell you it's not gonna point exactly to the feature that you want to design but it's going to help guide you in the right direction right there are several different ways that you can increase sink PvP adoption right you can create PvP leaderboards you'd create more PvP Center quests you could optimize your PvP onboarding to get more people to stick around in PvP

and get people to you know come back more increase the rewards of the value proposition of playing PvP versus other modes right but you know data is basically what you're going to use to get here and that's going to allow you to to point you in the right direction into into designing and sort of optimizing PvP in your game so enough about high-level concepts let's talk about some actual examples we talk first about FIFA Mobile which has been a real big success for us here in Asia the gameplay engine itself is based on our global gameplay engine but the game in Asia has we've worked the build quite considerably and created a sort of tailored roadmap for Asian players particularly in China where we've got premium placement on WeChat and are working with $0.10 on on you know running the game and publishing the game here so we've added a synchronous PvP mode whereas PvP was asynchronous in in the global build we've added other things like more sort of RPG progression

systems more sort of daily quests and retirement and other things and we're continuing to build this out over time so the build now in Asia is quite different and we're looking into expanding that now beyond China and talking about how to do that so it's been great for us and in terms of live operations live program to relate the lifeblood of revenue for us on on FIFA mobile now what is a live program a light program is basically an event that runs for a fairly long duration of time sometimes a week sometimes two or three weeks with sort of mini games and other quests and grinding sort of within that you know within that event within that program those those programs are themed a lot of time they're themed on real-world events or holidays or other themes we had for example a Copa America live program running we had team of the season Team of the Year World Cup etc but in addition to you know uniquely

engaging in the game and creating unique sort of goals there are also unique player cards that players can chase a lot of times that are higher power and we talked about Vladimir talked about selling power as well we sell OVR which is basically our universal measure of power and this is something we try to try to sell but we try not to linearly power creep it but also create more differentiation with team chemistry and other things create special editions of our players that we can chase in these in these live programs and the interesting thing is about this program or these programs in general is that we've actually been able to get app store featuring just based on program design itself which was kind of a new thing for us we always thought that you know if we approached Apple for featuring we would have to design a cool new feature but a lot of the programs are so unique in them you know in themselves and how players behave within the programs that Apple actually gave us repeat featuring just because of the programs or because of the you know sporting event that that the program mirrored so that was really cool for us

however it wasn't always it wasn't always that great for us right upon launching programs one of the things that we saw when we started launching programs in China was that players were consuming programs very quickly we had an issue with spend velocity we had an issue with average daily spend in our Tao and you know we wanted to dig into why this why this was happening right we looked at the data and we realized players were consuming pretty much all the content in our programs very fast the content consumption velocity was very fast players were content saturated and that's something that we looked at was percent of content saturation within three to five days and for a program that's supposed to be running for two weeks this is definitely not your design goal so one of the things we did was we tried to design our own custom metric around measuring okay you know how good are our programs at delivering sustained consumption rates over time and what is

the rate of change of consumption rate you know over a player's sort of engagement with the program and we created this metric which we which we just called program efficiency just a very simple name and you know what we saw early on was that and the size of the dot here is basically the duration of the program a lot of these programs were delivering low efficiency for us you know they were average length programs and we saw a lot of these players who would engage in the programs they would come out they would churn or they would you know do something else in the game and we decided to optimize this in a couple different ways the first thing we realized is that you know content is king right and you know we talk a lot about content treadmills being bad sometimes a lot of times we say content treadmill in a pejorative sense and it's actually not a bad thing and if you've got the right tools and tech to deliver more dense content it's actually can be a good thing for your game so one of the things that we did was invest right away in our back end

sort of automation and content development and we pivoted the dev team to focus a lot there we actually changed the composition of the dev team to be able to deliver more content at a faster rate so that's one thing that we did right away the other thing that we did is just in the design of our programs in the different nodes of the programs as players were progressing through them we made sure that we distributed Chace content evenly so that we players were able to engage and chase different things at different stages of the program so this was an easier change for us that pivoting the dev team but you know it did bear fruit for us over time well we saw even with programs that we ended up running for longer was that the efficiency of these programs was a lot higher so these higher efficiency programs helped us to achieve you know better results over time and drive revenue more consistently and you know as we've experimented and optimized this it really has been a great thing for our team and a great motivating factor it's one of the few foreign developed games that has been

able to sustain in the top 20 to 30 grossing in China the premium placement on WeChat and our relationship with with 10 Santa's has gotten better over time and you know it's next to you know a couple other games you know PES and MBA it is the top grossing sports game in China so this has been great for us and this is why we're thinking now of how do we take this more broadly into into other parts of the world we'll go back a little bit more in time and talk about something from my command days and this wasn't about identifying opportunity and really analyzing at a deep level what were the drivers of you know some of the success of our first gen mobile games so one of the great things about Kabam was Kabam really embraced the mindset of analytics and p.m. driven game operations and and using data to inform design so that was never a roadblock but we also were able to dig really deeply with the help of

data scientists and and bigger analytics teams into the drivers of longer-term trends and our LTV drivers project was one of the one thing that we did which actually impacted both our live operation strategy and how we serviced our players and also our entire development direction so in the first generation mobile titles we were able to generate a pretty high hit rate franchises that would gross 100 million or more we're relatively you know copious for us and we're wondering why because we looked at these games and we say okay these aren't that fun to play the graphics aren't great the strategy is very linear so what actually was is it just live operations and content or is it something else right I think content could have been the easy answer here but we wanted to look deeper than that we wanted to see if there was actually anything underlying so we took it upon ourselves to actually and this was a lot of manual work to create a large data set and taxonomy for us of

the different types of hard currency spent right this was event-driven our currency spent this was systems driven this was content driven or this was you know live ops tools driven etc and we did this over the span of 60 different client releases so the data set ended up being massive what we then did was we map that to the spend and transaction patterns of our players and tried to look back and see with the use of logistic regression and some other techniques whether they're you know what were the the biggest sort of drivers of long-term LTV growth and short-term LTV grow so the results that we found were actually quite surprising and one of the biggest conclusions was that LiveOps tools actually we're the biggest contributor to long-term LTV growth and even content itself and this was actually something that took a while for me to wrap my head around but when you think about it it actually makes sense right the ability to repackage merchandise content in different ways especially because all your users won't necessarily have all the content like

not everyone will have that content if it's released being able to actually surface and expose that content to different players in different ways is a great way to drive long-term LTV growth than simply just releasing something new so this was a pretty big conclusion for us and it actually informed a lot of our development on contest of Champions this was one of our earlier conclusions and when we were launching and preparing to launch Marvel the fuse box back-end was you know our live ops tool that we were going to use to service players and investing and becoming more robust in this tool actually paid dividends down the line we were able to build the few sports backend to serve analytics user acquisition live operations etc into one tool and it really had a positive effect on long-term LTB growth in Marvell the other sort of conclusion which actually came later once we you know sort of slotted Marvel into this study was the importance of seven-day players which we ended up calling regulars and

if you look at this graph over these these games right here what you'll see is that on the y-axis basically the propensity to churn and then the different colors are players who play for different days and you'll see seven days is almost invisible in the chart and that's because players who play for seven days very unlikely to churn and this was true across our portfolio this might actually be that surprising right of course players who are playing every day are not going to churn but the corollary to this which we found really interesting was actually the difference between six day and seven day players if you looked at combining this with average daily spend of a six day and seven day player a seven-day player spent more than twice as much on a daily basis than a six day player so we knew that these seven day players which we called regulars were very very important so this actually created some change for us because we were always thinking about pushing out new content and focusing on consumables over durables and you know this is why a lot of users ended up calling us cos scam right that was our nickname because our app store ratings were that high players complained a lot

they kept spending in the game which seems to be a pretty you know common trend but pivoting to focus on our regulars in creating seven-day behavior was something that we focused on more after this right creating a pivoting featured development to focus more on daily logging calendars and more retention mechanisms creating more rewards for 7-day play etc and this allowed us to you know continue driving LTV into more linear fashion versus asymptoting it and this is something that I've taken with me you know as we moved to EA and we call them committed players at EA but we strive to service them in a lot of the same way as we did at Kabam so wrapping up and I mentioned this before at the beginning when I said if you have the the teams and the mind set in place you can do really crazy things with data and it what it really has come down to for me is it really is a mindset right I've been fortunate to work at companies that have embraced data-driven decision-making I've also you know been at companies where there have been challenges with data-driven

decision-making and this you know kind of tension and dissonance between data and business and monetization versus creative right I've lived through the battlefront Fiasco where we almost had a knee-jerk reaction of saying okay nothing pay to win now only monetize off of cosmetics that's it we're like wait a second I mean maybe that makes sense for a business model where you're paying $60 up front but for free-to-play that's that's gonna screw us so we definitely shouldn't do that but really you know being able to invest in you know sort of rapid experimentation iterative design and you know informing this by behavioral data will allow the creative process to really elevate itself and will allow dev teams to scale in ways that they never thought they could right this is sometimes a challenge to create this mindset you know and I've lived through these challenges so I do have some thoughts on how to help create this but what I will say is that when you know PM's and designers and analysts and creatives are all working together in this way and all actually questioning

things and all have that sort of experiment hat on it really creates a cool environment on dev team so how do you create that and what are sort of my closing thoughts the first thing I would say is that sophisticated tools tech and data actually enable sophisticated mindsets and this is kind of counterintuitive if you have the tools and Tekin data in place you'll actually be able to come up with answers and conclusions to questions that people didn't even know to ask and that actually creates that sort of virtuous cycle on you know different roles in the dev team the second thing I would say is that build tracking in teams that enable you to sort of dig deeply into what's going on if you create this team and you create the ability to create answers to questions right away and you hire the best people you're going to be able to sort of come up with conclusions that are gonna surprise other people and it's gonna create this this mindset that I talked about before data is a great guide but it's not the end-all be-all I've been fortunate to work with less

data-driven teams and more data-driven teams I've worked on console mobile and PC at this point teams like Dyson Bioware who are definitely more about the creative process and I have immense respect for those teams and hopefully you know I think they've learned thanks some things from us too in terms of how to approach free-to-play business models and how to explore the future and lastly question is stress test your perception of reality as we get you know longer and longer in this industry we tend to sort of default to some of these things that you know these learnings and historical precedent in the past it's important to always keep that critical mindset switched on and to continually stress test your perception of reality if you've got the right people in place and the right data no matter what that reality ends up being you're gonna be able to find the solution so I would leave you guys with that and hopefully we have time for a couple questions as you know we've got I've got to move on Devin that was fantastic thank you so much I can't stress enough how important I mean particularly the LiveOps saying it just resonates for me Oh 20 years

running games the service and that is the one thing I think really resonates it's like the live ops matters more than anyone realizes yeah and the data is essential interested in working with companies that don't use it that's not a subject on that note thank you watching [Music]