Don't Change a Thing! The Challenges of Evolving Solitaire

[Music] this is don't change a thing the challenges of evolving solitire thank you for coming I'm super excited about what I have to present I hope that you will love it my name is Russell Caroll a little bit about me well I like to go by the name Russ and I like green I also like Nintendo I came up as a producer and designer so you'll hear a lot of kind of design background and producer background the the thing I like most the title I like most is Vision holder I like to think about projects from the beginning all the way to the end and try to encapsulate that and then I have a lot of awesome people I work with to make it all happen I'm old by GDC standards this is my 14th GDC so that I I guess that makes me old I have some gray hairs and I love fun this series of icons you see here are from games I worked on before I joined my current employer you will

notice a theme which is some planes and googly eyes I used to say if it doesn't say fun it needs to be redone it was just a quick phrase for me because I loved things that were happy and I also make like making games for everyone some people call these casual games games for everyone works for me and that's the main reason I work at my current employer Mobility wear an awesome place so enough about me about it solit soliter has had over 270 years of playtime today and every day we've got over 200 million downloads across Android and iOS the game is the number two installed game on iOS and if you remember when Pokémon go came out everyone was playing it and there was a company that did a quick look at the market to determine how much people were playing it versus everything else and they released the top five games played on iOS when they

did that we were happy and and not super surprised to see that soliter was actually the number three game played and the number four game played was our spider solitire so these games may not be quite as flashy as some of the ones you have heard about we have massive numbers of people playing our game which is a great opportunity and also present some challenges that we'll talk about today some more about it when I say the word Sol I often refer to five games we have a soliter suite of five different apps they are Tri Peaks spider free cell pyramid and Klondike Klondike is the one that everyone calls spider or sorry soliter except some people may call it patients those of you from the UK and other places there are 16 SKS we have those on Amazon apple and on Google I run two different teams doing the development for the game and that is across three velopment environments we

work in Native iOS we work in Native Android and then also in unity and to the question of are you still developing this game the answer is an emphatic yes we had 40 updates across the suite last year we're going to have more than that many this year some of those updates are small but we try I try really hard to have a feature in every single update so yes we're still developing a lot which gets into about my job when I tell people I make solitire I usually get the question of what what do you really do right soliter has already been developed it's been around for a long time what's your job well when I was getting hired I was told what I was going to be doing and that is ADD features to Solitaire add users to soliter and don't mess up the revenue yeah a little bit of fear there so this actually creates the first challenge in working with this large title and that is how do we make changes

that increase the revenue without making changes that decrease the revenue I'm going to start telling you about the Second Challenge and to do that I have to tell you about her this is Lisa it's a pretty good likeness this is Lisa I met Lisa when I was in a parking lot we were both picking up our sons not shared but we were picking up our sons and so we were waiting and we got to talking and is usually the case she asked what I do and I said well I make games and she said oh well I I don't play games so this is Lisa not a gamer and then she found out I made solitire and she was ecstatic she got so excited she's like I love solitire and she went on and on about how much she enjoys it specifically she loves a feature called daily challenges daily challenges is a feature where we have a new winnable deal every day and there's a calendar you can win the deal you check off the calendar for that day if you complete enough deals within a month

you get a big trophy if you complete more deals you get a bigger Trophy and if you complete all the deals in the month you get a really big trophy we call that gold all right pretty straightforward stuff she told me about how much she loved the game and then she said don't change a thing don't change anything now this is an irony right because if we hadn't changed anything the feature she loved wouldn't have existed we would have never made it to begin with and so this is really that second major challenge we deal with in our game and I'm sure many of you do in your games which is we have customers who resist change they they tell you they don't want change but we also need to make changes and so these create a tension so these are the two challenges players resist change companies they only want positive changes the question is how do you handle those challenges they're not unique to us so you can kind of ponder how you do it I'm going to go

through some of the effort that we do and that is a focused effort to first off protect the player and then to protect the revenue let's talk about protecting the player and the way I'm going to first talk about this is with in-game surveys I love in-game surveys I hope that you're using them I've heard a couple of people talk about them but no one who really was using them a lot so let me tell you what we did with them to do that I need to First tell you about a feature we put put into solitary prale last year called progression now this feature was actually a multitude of of smaller features all put together the first is that we have XP so every time you complete a deal we give you XP you can see there's an XP bar normal meter we also have daily goals each day there were three goals an easy a medium and a hard goal you get XP for completing the deal you get XP for completing a goal and if you complete all three goals in one day you get even more XP so there's a bonus that you're working towards

there are levels so you can level up you can be level 14 level 42 and lastly there are titles these are mostly just for fun but they provide a goalpost right that's kind of the ideas behind all of these is creating goalposts creating a progression for users that they can move through so every five levels you would potentially earn a title so let's go back to ingame surveys when we release this feature we wanted to know how customers felt about it right we want to protect the customer in order to do that we need to understand how's the customer feeling what do they say what do they think and in-game servers are a great way to do this they're unfiltered from a player where you go to a review and you might have to put your name on it and the consumer might be a little scared to let their friend know how they really feel about a game when we get a survey response back they are very honest they're brutally honest they will tell us exactly what they think and how much they hate us so this is good we get more information in fact six times more information so this means

that when we ran this particular survey we get six times as many people who responded to this survey as who wrote in reviews okay so more data much better it's also targeted we were able to write the questions to get the information we wanted so we didn't have to parse through all the reviews and say what are they talking about and what are they talking about we able to drive the users to specifically what we wanted to talk about and they're super easy to do the way we did this we actually use Survey Monkey so nothing hard right created a survey in Survey Monkey and we have a file up on a server that has a that URL in the file and it also has a couple of other pieces of information we just change that link and we change the other information and it puts the survey in the game shows up in a web view the players can respond right there that's it it's that easy because it's in Survey Monkey we can immediately go and analyze the data how did we do it it we had a five

question survey we started with four lyer scale questions if you're not familiar with the liyer scale I'm a big fan of it this scale you have a a statement and then the player responds by saying whether they strongly agree agree neither agree nor disagree disagree or strongly disagree with your statement we broke out those four questions into two parts two about progression and two about the UI which we had to change as we did progression I wouldn't recommend by the way doing those two things at the same time but we had to in this case so there it is the last was a comment box and this allowed users to say whatever they wanted about whatever they wanted and they did we ran the survey for about two weeks we got many many thousands of answers many many many thousands of data points and then we analyzed it this is actually a word cloud of the comment box uh data that we got it's not really useful but it's kind of pretty so threw it up there we looked at the trends and the opportunities that we identified

from what people had said and then we pulled the team together we had a meeting we talked about the problems we brainstormed ideas and then we put in together a plan of how we were going to address the feedback we'd spent many many months putting together this feature we now gotten back what the customers thought about it how they felt about it what they were saying about it very honestly and we went out on a two-e development plan it's not even two weeks we planned for one week of engineering time and a week of QA time all right so we Dove right in to try to improve things and I'll tell you how we did when we first released our progression feature based on the survey data 80% of our users liked to were positive neutral towards the information we had in there they liked it but 20% didn't like it that's one in five it's not a really good number so we took this one week period to try to fix that which is kind of crazy because that's not enough time

to do a whole lot but we did by just listening to what the players were saying by protecting the players by understanding what they wanted we were able to cut that number in half so that the next release and it took the weeks with the QA and then getting the data and whatnot was 90% of our players liked what we were doing that's a huge change other way we protect the players AB testing I'm sure all of you do this if you if you don't shame on you i' like to AB test I like to AB test features not whether or not the pony should be blue or red I like to identify should we have a pony at all I'm going to tell you about one test that we did which is on the easy read card front now I told you a lot of people have this game installed in fact almost almost 10% just a little over 9% of people who have an iPhone have our soliter game on it which is kind of crazy that means about 1 in 11 which means a number of you do and not just because some of my um co-workers are are here with us a number of you probably

have this game installed on your phone we get anecdotal information because we talk to everyone as we're out and about when we see them with their phones we talk to them and we see if they have our game and if they don't have our solitary game and they have another solit game we asked them well why not why don't you have our solitary game one of the pieces of information we kept getting back was it's hard to read your cards it's it's hard to read your cards that was a reason why some people weren't installing our game they're hard to read Now cards are the character of our game there are Mario or Super Mario Brothers that may seem strange to say but our players look at these cards all the time that's all they ever see that's all they interact with are these cards so changing the cards in solit super hard when I first joined Mobility Weare we talked about changing the cards to solit I had people that start shaking their heads and they're saying you cannot change the cards in Solitaire not that you should you can't do it cannot be

done well we want to listen to what our customers want from us we want to protect those players and we want to do that not just through listening but also seeing the actual data not just what they say but what do they do so we did a test and we did as a 50-50 test of new users Now new users of course because this means we're not exposing all of our existing users to it right it's safer already and we did it on Android which further shrinks down what we're doing I'm just going to run through the results really quick because it's kind of interesting there's a black line You'll see on this graph that black Line's a baseline anything above that line was an improvement below that line was not an improvement so this an interesting little piece of graph here the blue if you see is the phone for the phone having this new font increased our retention across the board day 1 day 7 day 14 day 28 pretty awesome for tablets which is the yellowish orange it actually didn't increase retention it decreased retention up until day 28

where it randomly went way up well that feels really uncomfortable so what we ended up doing is acting on what we felt really good about which was we knew on phone this font was making players stick around longer it was retaining users better so we made it the default on phone and made it available to all of our existing players as well let them know it was there so they could choose to use it so that talks about protecting our player this looks at what they say in surveys and what they do through AB testing I want to go back to the challenges I said that players resist change and companies they only want positive change so I want you to think about that for just a moment as a developer how does that make you feel if you're thinking as a developer I'm going to make changes but the players don't want them and if I do anything wrong my company's going to get angry at me how do you feel well the answer is

you feel afraid as a developer you are fearful and the response to being fearful is play it safe try not to do anything that will rock the boat I heard this when I joined the company I'm going to go through a few quotes here first one you have to be careful when you talk to management about changes I heard that right off the bat when I joined mobilityware which has been some years now we once changed the font by one pixel all notice it's underlined because all the players complained all millions and millions and millions and millions of our users they all complained when we change the font by one pixel these types of stories end up being culture they communicate to new employees how you interact with your games and what you can do at the company in this case what we were communicating was making the smallest most minute change was not a good idea don't make changes and lastly I still hear this one sometimes though I'm still working on it is you can't change anything on SOL so

you can hear the fear through these right you can hear that that there's some discomfort in trying to make changes I'm going to quote Ed katell here multiple times he wrote an amazing book called creativity Inc about managing creativity he's the president of Pixar if you're not familiar with him now president of Disney Animation he says in a fear-based failure of verse culture people will consciously or unconsciously avoid risk they will seek instead to repeat something safe that's been good enough in the past now if you're looking this and you're like Russ what's wrong with safe isn't that okay what's wrong with it really well here's the problem another quote from him copying what's come before is a guaranteed path to mediocrity in our industry there have been hundreds of games that have come out on mobile today there are thousands of games coming out every month you cannot afford to be mediocre you cannot

afford to play it safe and be just like all the other titles there are thousands of failing mediocre safe titles they're failing every day you've got to stand out you've got to be willing to make the risks that will help you stand out so when I look at the two challenges that I talked about earlier we have making changes versus players who resist change and then the second one was making Revenue increasing changes versus Revenue decreasing changes I actually think that second one's wrong the real other challenge is making changes at all versus a fear of failure so now we have three challenges we have protecting the player or sorry we have three ways think three ways to address these challenges we have protecting the player protecting the revenue and defending the developer but if you were listening to what I just said protecting the player and protecting the revenue comes at the expense of the developer it makes them fearful those two don't work together to create the third and if you put all

three together you have a mess so what if we protect the revenue and defend the developer you do that at the expense of the player the player leaves the money goes away both of those are fails so this is my belief in order for us to really grow we need to protect the player and defend the developer and I strongly believe if we do that we do protect and grow the revenue because we allow the developers to make to try new things and to find Opportunities to improve the products so I'm going to talk about this I'm going to talk about how we defend the developer and the first way is with guard rails guard rails are very familiar because a lot of the same things that are guard rails for our developers are helpful for our customers they protect our customers as well so AB tests they provide our developers a safe place to try things out and they also protect our customers from getting something might not

like multiple applications I mentioned we have five games that are soliter games we can test a new feature in a smaller title before we roll into a bigger one we're not unique in this King does this supercell does this it is a very common thing to do and then also we have multiple platforms if you put this all together we can put something into a smaller game on a smaller platform and then AB test it down to a small niche of the audience we still have a lot of users in that Niche but enough to test it but what it means is we can create a totally safe test environment our developers can feel totally comfortable trying something and if it doesn't work that's okay because we are not risking anything we are protecting them the result of this is really confidence and freedom but this is only the first step we've got to work with our developers as well in something we call Success Through failure I'm going to call this failing forward which is that we need to have a philosophy that if something if

we try something new and it doesn't work that's okay in fact that's what we need to do we need to try things that don't work to find the things that do I'm going to talk about a a crash bug that we had about three weeks after I joined the company we were trying to release a bill just before a company trip to Hawaii which is cool thing Mobility where it does we had a crash though in this build we'd already submitted to Apple it was sitting there just ready to go live but we were testing right up to the last minute because well we we needed to get all the testing done as we did we found that it was crashing consistently over and over so we tried to fix it right we tried to save it we have passionate developers they want to get it right so they tried to save it they made new builds and those builds crashed and more bills crashed and we thought well maybe we can just submit it to Apple and then when we get back from the trip we can then be able to go live but as we went on through the day we came to realize this feature was not well implemented we we did not do a good job putting this one

in how do you respond to that if you're a manager how do you respond to it if you're a company leader if you are a team developer team member how do you respond to it there's a lot of ways to respond to it I'm going to go back to Ed cill for a moment he says failure isn't a necessary evil in fact it isn't evil at all it is a necessary consequence of doing something new so what we did is we met as a team we talked about how to fix the problem and then we put a plan in place and we tried to fix it the next day one of the engineers came and he pulled me aside and he said hey I wanted to talk to you he said you know I've I've worked at a lot of different places and usually when we do something wrong we get yelled at we get screamed at and until like dude you totally messed this up and we already know that right but that's what usually happen he said it was really refreshing because you just said how do we fix it how do we make it right and I'm not going to say I'm perfect of this I am not but I believe in the principle and I keep trying to become better at it

which is that as we try new things we should expect that we are not going to get them right all the time and we should have confidence in our developers we should have confidence in each other as we fail forward what we do is instead of having people who are fearful of making mistakes which by the way leads to mistakes we have people who are focused on getting it right they're focused on trying new things and making things awesome to do this we have to create a culture we have to actually say it explicitly whenever I say explicitly my team Giggles there's multiple meanings to this word what I mean is we have to specifically state it we can't just assume that everyone's going to see us and they're going to just absorb that this is our philosophy because we act a specific way we have to specifically State this is what we want we want to try things and we want to move forward when we released that progression update I told you that we met as a team and looked over the

feedback the feedback we got some of it looked like this I hate the update hate it you absolutely ruined my enjoyment of playing I have a whole lot of these I could I could spend an hour just going through them and many of them are not fit for reading it's another one my old high schore records are gone very disappointing Noti L they put this in all caps I I assume it's because it's important to them assume so as we meet and we talk as a team about this type of thing our lead designer mentioned we need to have thick skins and soft hearts that is to say we're going to be receiving feedback from the customer directly we need to not take it personally which is hard because we're passionate we really care about what we're doing we care about those customers and that care feeds into the second part which is the soft Hearts we need to listen with our hearts try to understand what are the customers really saying right what are they really feeling how

do we help them through this we also need to remember that we win and lose together as a team this was a focus it's really easy when you see this type of feedback to go I didn't suggest that remember when I said not to put that in it's really easy at this point to bicker about well we should just take this out entirely right it's failed it's totally failed give up move on what we did is we took it as an opportunity we met we identified what people were saying and we responded directly to it here's a quick example in the top right hand corner you're going to see that there's a button it's a kind of a round long oval with a star in it and a number next to it this is where you get to the leaderboards after we made our update this button doesn't look like any of the other buttons in the game it was moved from the toolbar the bottom to the top it used to be just one big star now it's a star with a number next to it and our customers many of them couldn't find it they didn't know where it was they could not find their

high scores they could not get into the leader boards so this one's kind of an easy change we changed that to be just a big star button and then we put an overlay on it and it pointed up and said hey here's the button on the left side you'll notice that there is a gear now that's our settings button it's pretty obvious we moved at a Level under when we when we did the UI one level back and guess what some players couldn't find it we'd had discussions about whether or not we should do this ahead of time and we decided to go for it so when it comes back and it wasn't the right thing this is another moment where you can say I told you it was a bad idea right that's a fail in my mind because what we did is we wanted to try something it's a safe try let's try it let's see how the customers respond to it didn't work okay well how do we make it better and then we made it better and that's really the focus of everything I'm I've been talking about is Improvement little by little and we have improved since I've shared so many negative things I wanted to share some positive ones this is how the customers are responding to our feature the progression feature today I

just love it I love to do my daily goals brilliant game thank you and those are kisses I believe I I don't think they're trying to cross me out of existence but I I I wonder sometimes this one I actually really enjoy the goals I wish there were more which tells us what to do next and this last one now I have got used to the new free sale it is okay and I'm I'm really okay with this one and the main reason why is because as I look at it we got to 90% of our customers liking what we were doing 90% liking plus 10% being okay with what we're doing that's 100% good in summary we two major challenges that we we confront we have player resistance to changes we have developer fear of failure and just one big resolution which is do change things we do it in a couple of ways first off we use what players say

through surveys and we also look at emails and reviews of course and what they do through AB testing and all of our data analytics we use guard rails to make it safe for our developers so they can feel comfortable trying things out and then we encourage failures that's part of learning it should be part of learning that's how we try new things if you are not able to fail you're not trying new enough things you're not trying things that are going to be more unpacked enough for your company some cool stuff if you like this by the way creativity Inc I quoted it multiple times I had like more quotes that I cut out of this slide presentation there's so many good quotes in there absolutely love the book intuition intuition pumps and other tools for thinking this is dennet who is a philosopher there's a piece in there about how mistakes are the only way we learn which I think is a really fascinating concept and then make good art by Neil Gman which is a commencement speech that he gave you can just check that out online super easy if you'd like

to talk about this kind of stuff please be in touch with me my name is Russell Carol I uh work at Mobility Weare as mentioned and speaking of we are looking for people who want to change things so if you are passionate about customers and love trying new things also enjoy fun environments where it is sunny all the time and SoCal give us a ring and with that thanks for [Applause] coming I have two minutes so uh if there's any anyone who wants to jump on I think I have enough time for one question if there's anyone who wants to jump for dinner this is the time to run the question is how big is the solitire team so there are two solitire teams and the first team is about 14 people and the other team is maybe six seven those are the core client teams we also do a fair amount of server work and that is held through tools teams we also have ization work that is done through a separate team as well so if I start

totaling everything together it actually ends up being quite a large number of people many of those work on multiple Projects please quick question when you're talking about receiving feedback from your surveys do you get most of that feedback and ideas from like the agree strongly agree agree disagree statements or from the free form comments so the question is where do we get most of the the information back from the surveys is it from the the questions with the specific answers or the free form and we get it in both places the free form obviously gives us lots of Trends to look at specific data um the questions with the specific answers give me trends so when we did the first one versus the second one this is how I can tell the 80 versus the 90% right so I'm looking at both the data of what are people saying and how does that relate over time is usually where I get those um most of the information is going to be in the comments that we act on

all right I think that is all we have time for please fill out your evaluation forms let me know what I should change and uh thank you