Chrome Valley Customs level balancing talk, from Space Ape's Geoff Gilles

hi everyone uh my name is Joffrey um and I gave a talk at PGC on Monday and Simon H shared about that talk on LinkedIn and then quite a few of you asked for access to the presentation but we want to do even better so this is the presentation and this is me presenting the talk again and I've added like a few more slides um to explain a bit more so I'm Joffrey Gillis I'm a game lead at space sa games I've been at the company for almost 10 years uh anniversary in six weeks and I've worked on many many games uh be def games uh be Global launch games live games and then also games that uh never made it to to launch um the two two of my babies are one fast lane and then the second one bear uh you I've been part of the early death team on both games and then part those games to Global

launcher I'm here today to talk about Chrome Valley Customs this is our most recently Global launch game uh it's a puzzle game um and yeah Global launched end end of uh end of June uh if you've never played Chrome Valley Customs this is the pitch uh they're like many many car fans 260 million in the US and in the EU and and on mobile there are also many puzzle fans as you probably know Mash is a big genre um but yeah the the team the the early team working on chrom Val Customs I was not part of that team when they did the research about two years ago they realized that many of those puzzle fans uh are male players and that they didn't have a cool Mastery puzzle game um really targeted for them for that like M male demographic and this is where the

the appeal of cars that really big Market size and then the appeal of matre kind of like merged and then uh literally merg and then and then gave birth to Chrome Valley Customs those are UPS screenshots um and then and then yeah Chrome Valley is a match free puzzle and decorate game uh it's rooted in that car fantasy but really in the car fantasy it's about car restoration fantasy uh and then players get to play cool maty levels they're like uh garage themed and then and then interact with those rusted cars broken cars and then and then and then and then make making those cars um look beautiful uh the game was Global launch end of June so that's about eight months and uh and then since then uh we've been releasing about 200 levels every month and we've we've learned

Plenty when it comes to mhre campaign balancing and this is what the talk is about it's about like MRE campaign balancing if this is a subject uh you don't care about stop listening now uh but if you want to hear more about our journey and then and then how we build some core tools to balance this campaign then stick around and listen to this to this presentation um so the the challenge with matre and matre campaign balancing is um how can we balance effectively multiple thousands of levels uh my background is finance and uh when I was in finance I was doing uh also portfolio management and then if you if you think about a portfolio is trying to optimize the the risk and minimize minimize the maximize the return and minimize the risk um out of that whole portfolio thinking about M campaign it's it's the same trying to maximize a return and then minimize

um minimize the churn across the whole campaign and they're like hundreds thousands like we have 2,000 levels now uh of of individual levels and all of those levels they need to be um improved one by one but they also interact between each other and then and and then the challenge is like how can we optimize the whole uh what's unique about Match 3 is that the M like the match 3 content is that it is content that can and needs to be rebalanced constantly uh so yeah choose choose this book because the keywords are right uh we want to be active that's the main one and and then being active means means that will profit we profit more uh what what what makes masre quite unique is that the the the campaign is is linear so players are going going to play a level

once we never reuse a level um so once the level is played that player is never going to play the level so the level can be changed and the next player has never played that level which means it's a brand new experience so we have way more freedom when it comes to um changing the balance and that's again that's unique uh give give you like on the that's on one end on the other and if I think about beatstar on beatstar um the the beat Maps so the the noes layout um if if if if the music team would be changing those on a regular basis our players would really really hate that because the goal uh of of of beat stuff of like playing a song is actually to try to master the song and then and then and then that needs to be static it's like mastering that static piece of content um so yeah so the beat maps are super rarely rebalanced

um so yeah this this makes this makes I guess balancing uh more challenging but also like yeah more important when it comes to M Campaign which which I think makes this this this topic and then the tools we build even more important some basic terminology for you uh to to understand the rest of the presentation so the so the first one is churn uh you probably no churn but uh when we think about churn we we think about churn on a given level so what's what's again unique with mat is because the structure of the campaign is linear we can we can aggregate all of our kpis by campaign level and we do that uh whereas on most games the kpi would be aggregated either by day or by tenure so here we aggregate by by by level by campaign level and then the the churn is the percentage of players that

churn on a given level uh EGP is that spend Dimension uh it's it's an acronym that means endgame purchase it's a percentage of players that do that and most of the spend on a Mastery game is done on is done on buying extra moves because that's most of the value players are stuck in the level and then they want to beat the level but they don't want to replay the level 100 times and then and then they just buy extra moves uh and and when I was saying like we're trying to with this portfolio SL campaign to minimize the churn maximize EGP like those are like the the two big vectors um in terms of like kis and then uh two more two more like keywords one is APS uh so if you see APS anywhere it means attempts per success and it's it's the number of times a level has to be played to be succeeded so if the aps is five then on average the level has to be played five times to

be succeeded if the aps is eight then it's eight times and then having to play a level eight times of versus five times means that the level is harder um so it's it's it's it's aoxy for difficulty and then the curve the curve is just a sequence uh the difficulty sequence and you'll see some Curves in a minute um the curve the curve is is really important uh when it comes to M campaign balancing uh uh and yeah we went from nothing or barely barely nothing and a few hundred levels and since Global launch we've been adding 200 levels every every month um and and we're planning to do that for the foreseeable future and and we've also built some cool tools to balance all of that like we currently have 2,000 levels in one year we'll have 4,000 levels and those have to be like constantly like rebalanced and improved now uh so three tools um that advise anyone that's ever

working or will ever work on a master game to build is the first one is UN curve optimization the second one is level level reworks and the third one is to be able to do campaign AB testing um I'm going to talk about the first tool and the second tool and then this is a PGC presentation and it was supposed to be in 20 minutes so yeah they have time to to talk about the thir tool but if you have questions just just ask them online and I'll try to answer um so first tool uncurve optimization this is the Bedrock this is this is really important and it it really unlocks the the team um so like conceptually uh a campaign a master campaign is one linear because it's a sequence of levels and it's linear and players are playing from level five to level 10 and then so it's like incremental but like also two with we assume but now we have also like data to

back it up that there was like an optimal structure uh and and that structure that difficulty sequence uh is what we call the target APS so Target APS Target difficulty sequence uh means the same um and and there is an optimal one uh and what we want is we want because we have full control over that linear experience experience we we want all players to go through that optimal experience like the I like making music it's kind of like a song uh songs are linear and then they have like uh highs and lows and then it creates dynamism and and and it's ex it's exactly the same with a Mastery campaign we're trying to make a really good song uh by having the best SC so that's that's the that's the the Blue

Line there uh the actual curve doesn't have that exact exact shape but it's goes goes up and down um and and and the other dimension with match three is that the levels are really random so when players played the level even if we intended the level to be at a certain APS level when they played the level the aps based on that first balance might be different and it's often different and that's the live APS curve so level 500 played the actual APS is three uh our intend optimal curve would put it at five and then and then and then our job then is to try to tweak the level to get it to five now um what we did like with with with with this tool but like with all the tools we've been building with mhre to really uh one think about usability um

we've had this journey over the last seven months where some tools had poor usability and then if they were not used so have tools to be like easy to use and easy to read and then and then the second one um have them scale with the amount of data uh we have 2,000 levels now going to like I said 4,000 in a year 6,000 in two years um so uh the tool has been built uh in in a spreadsheet in a in a Google sheet and it's our supercharge sheet um this is the overview uh it's kind of like the super high level summary um and all the columns are information about the level and then all the rows are the levels themselves so if we would expand that spreadsheet would have like 2,000 rows and it has loads of information that's really cool to look at uh in the context

of uh doing this uncurve balancing like all the levels are constantly on a daily basis uh rated as either being too easy on target or too hard and and what we want to do is we want to tweak those levels to bring them back on target like ideally everything is on Target because we have proved that we found this optimal curve and then we want everyone to experience the optimal curve L um and what we did which I think is quite unique from what I've heard in the industry is like we pushed it a step further it's fairly easy to flag levels as being too easy too hard but like we push it further where we fully automated uh the process so um this is this is another view that's kind of like filters levels that are that need to be rebalanced because the live APS is not on target APS and and it's it automatically says or calculates the number of moves based on some rules we

have uh under the hood that need to be added or subtracted so if a level is too hard we're going to add some moves if the level is too easy we're going to remove some moves so in this case yeah too hard plus one move too easy minus one minus one minus one move and it's just a case of clicking a few buttons and then it's in the CMS of our game we push it to get we push it to to to live and then and then the game gets rebalanced um so yeah the the next slide that I like next yeah next slide that I wanted to um show is and it's connected to the to the previous one this is a view by level but we also have built some views by uh groups of levels um and and in this grouping it's about 20 levels and and then both views are really interesting to look at because like I

said this is a portfolio we're trying to optimize a whole and then to optimize a ho we need to have the best levels but we also need to optimize the interaction between levels um the second tool we we've built uh is so once we had that Foundation like the first tool is to be able to like rework levels and like reworking a level means and we means that we we assume and I always assume that a V1 is not perfect it can't be perfect like nothing is perfect V1 and and then and then what we need to do is to have the tools to identify levels that are performing badly to and then to be working those levels and then to prove with data with the before and after with that first version and that second version that the level has improved so in this image you can see this is is uh level V1 and then

in here it's the level V2 um so going back to that sheet going back to our principles of like really trying to automate everything uh we we use the churn and we use EGP and then we do some statistical analysis under the hood on all the levels uh and and we give each level uh a score in terms of both EGP and and like both and EGP so our level can either be bad okay or good and and then and then that that information gets summarized even further into like a priority system like um so looking at a like sorry I done example so level 344 it's uh Target APS 6 it it is an actual APS of 7.35 it is too hard and and and what we see is that the churn on that level is

0.5% and the EGP is 13.8% uh based on that analysis uh we estimate that the churn is bad so yeah I'm not going to go too much into detail but yeah it's bad um and and and EGP is okay so this becomes the signal to say okay level 344 if we wanted to improve that level we there are like some churn churn gains to to be had there uh what's been tricky if you build a similar pipeline is to track the before after we currently are not a testing every V2 V3 V4 uh we do before and after and and it's proven to be faster but also gives us enough of of a signal to say the level has improved um um and and then and then yeah that's been really tricky so you build something like this be be good friends with your on like data team uh and and

to show you like an example this is level 455 and level 455 uh is a target APS of eight used to be a live APS of 7.5 became an APS of 6.62 so APS went down with the rework because when we rework the level is being Chang uh the churn went from 0.21% to 0.13% so big drop in terms of churn and EGP went from 20.5% to 24. 47% so gain gaining EGP G gaining EGP did I say EGP before big big drop of turn increase of EGP and then again like applying that statistical like analysis uh before we reflect that the level was bad in term of of churn um for that APS now it's a good one and it was already like good in terms of EGP and it it's it St good has improved a little bit um you might have noticed that the aps has dropped slightly from the pre to post from that V1 to

that V2 and and and you might assume and you would assume correctly that this also would decrease the churn because there's correlation between making the level harder and then having play uh churn more so and this is this is completely right so some of some of the churn drop there can also be attributed to the aps decreasing by 0.9 uh 0.9 like absolute amount um and this is this is why going A Step Beyond you know just the absolute values but like really applying in this case some of the analysis we do really helps because allows us to with more confidence say that that that our level has been improved uh on that this is the level uh before the rework and then after the rework if you look at the at the level on the left the so the opening space it's a small Opening space but it's also surrounded

with many blockers that are like hard to destroy so the experience for that level is the play is going to match uh but uh the blockers are going to be hard to destroy so the player is going to be stuck into like small Opening space for longer uh and then the odds of sorry the odds of the player being stuck there increase and that creates like frustration so we had the signal which is like oh this level it's bad in terms of turn uh okay in terms of it was good on terms of AGP but bad in terms of sh that signal is used by a master game designer and they can now apply the I guess they like Master knowledge uh but we have also like more um more Deep dive like metrics uh that I'm not showing here to look at uh and really to assess the level and and then to change the level in in this instance they the person that changed this level uh understood everything that I've just said said and they changed the the

opening board and replaced those those blockers by blockers that are easy more more easily destroyed and that opens up the the top board like faster and then as the top board opens faster like more matches more Cascades uh and then the middle probably also opens up like faster because the uh the goal has been reduced from 50 to 402 for the cassette and then there is more progress in the level and that just makes like for a better experience which has reduced turn and then also increase EGP um and and then what's really cool is uh we have screenshots for all the level we rework and then and then we can look at them and and really try to build that that that understanding of how to make better like Mastery levels uh and then and then when when we do V2s we we we have this outcome Matrix so a a level that's been V2 can either be better compared to to V1 or it can be mix you know no impact or it can be

worse so some levels are being worse if the churn increased and the GP decreased that's that's worse and it happens it's it's rare but it happens and and we don't want to keep a V2 that that's made the campaign worse um we're trying to aim for the green zone um and I'll be showing results in a bit uh but yeah like some of the levels are in this diagonal where it's it's gray Zone it's like CH has decreased EGP has decreased should we keep the change should we not keep the change um when when this happens we do case by case so the results some of the results uh like one we're like saving countless hours of manual but high value repetitive work uh many words but but it what's crucial in there it's like it's high value and it's this is all that like being on curve uh having the live APS be on target APS it's high Val value we really need that uh but it's also

repetitive because it needs to be done three times a week um and and if it's manual it takes many hours it used to take our M team not a whole team but like part of the team uh like half a day multiple times a week just to do that and now it just takes a few minutes um which is really cool uh and and that frees UPS time and and and and and and brain brain power brain space to do more complex work more more enjoyable work uh so uh the Mastery team uh and the product team we've been like and and and and and the data team we've been working really closely to uh run some some pretty cool AB test um and and we've recently like found another optimal curve which has increased upu by 10% um just by changing the curve so I will take that um and then and then the third result that I wanted to call out is that 35% of the levels that we do rework um

fall into that green quad green green section like those three uh squares and we keep them uh and it has led to a total CH reduction of 1.23% uh which which has increased retention um but also while the turn was reduced EGP was increased and and and I guess you might say oh it's 1.23% understand is that is that much like one it's really hard to like reduce turns SL increase retention um if you've ever worked on a death game or a life game if you've tried it's really really really really hard um so it's like this is Meaningful this is just by Rew working levels uh it's it's practically free it's just content and so like we want to we want to have that free launch um and then two it's it's 35% of the levels we've we've worked so far and then and

then when I look back at the number of levels we've reworked like we've we've really started to um we worked them at a faster Pace as this tool was being built and it was being used and and then it was being improved so we working more and more levels and then I expect that percentage to go up as we really learn how to how to we work levels uh I had two tips that I didn't share during PGC but I could quickly share with you now because this is not bound by 20 minutes uh the first one is to separate your everyday metric from your Deep dive metrics so yeah like there's this temptation to have everything be at the same level when doing uh level analysis uh what we've done we use it more like a funnel so at the top of the funnel there is every day and um and then and then if if we has if we have signals that that that a level

is bad then we can use the Deep dive metrics to to really understand the level and then how it needs to be improved uh and the second one yeah you've seen that the sheet showed you just just a few like Snippets of the sheet it's a pretty big sheet I think it's the most complex sheet we have at spaceape uh and it's really powerful uh because building in with Google Sheets is can be really really powerful so it's a sheet again that's connected to our warehouse connected to our CMS does a lot of automation does a lot of data crunching and then and then and then saves us a lot of time but like brings a lot of like value and then and then everyone on the Mastery team is using that cheat practically every day um and it's something we hadn't achieved while trying to build those tools using maybe something that's like fancier like oh we're going to do a tableau online and then everyone is going to use a those Tableau online graphs no actually we tried and it was a lot

worse uh where we at is we have a document or what we want in the future we have this document where we have our mrey design best practices it's like the rules on how to make an amazing level and then we have the ab test knowledge uh we have the Deep dive knowledge like the our an etics team has done like amazing deep Dives and we've learned so much and then we have all of the V2 optimization learnings and what we want to do is like we want to create this like virtuous cycle where all of the data knowledge backs backs up best practices on how to make really good levels and then questions we have on how to make good levels can inform either like a targeted AB test or or a tip dive and really create this virtuous cycle so that our best practices uh get get more and more informed backed by data to to ultimately have this Bible on how to

make an amazing level first try uh that that has a really low amount of Jour and then and then maximizes EGP here so that's where we're going next and I think this was my last slide yes so I hope you've learned something uh if you want to connect I'm obviously on LinkedIn uh this is my LinkedIn uh I hope you appreciate this uh talk like we run we re-recorded talk uh and and yeah I hope to I hope to maybe talk to you one day thank you bye