Live Ops Production for Mobile Games PG Connects Digital #2

and now we're going to be talking about specifically for mobile games what is the sort of LiveOps production process and we've got a table from thinking age was coming up hey demo day and he's gonna take over from me and talk about my production for mobile games over to you sir hi there hi everybody I'm table Boyer and I'm here to talk to you about live ups production games so let me just switch to the appropriate background and start the presentation here and hopefully are gonna share my screen first this should be shared and presented so I'm actually going to take a little bit of a different tack that I normally do I've been doing live ops and live games for almost 10 years and I've done this presentation a presentation about live loss production almost ever

here since I started and I thought you know I've started taking over recently to become the head of production at the whole entire studio and that's giving me a bit of a different perspective so I'm actually talk about how do you create LiveOps production through psychological safety it's a little bit about me I'm the head of product at a thinking ape and I like I said been making games for 10 years in Friel a live environment and you know outside of a few small prototyping phases I've been almost exclusively actually working on live product and I've been without thinking it for seven years we have right now for products in the market kingdoms of war party in my dorm Keswick fire at casino X kingdoms at war was launched in 2009 and so it's one of the oldest games on the market and it's still around it still actually does quite well it has really really dedicated user base and we launched party in my dorm in 2010 and that game actually is actively growing

it's growing year over year in terms of revenue it's either basis it's growing and then we finally launched into the tech fire and about two years ago and that game is you know us taking everything that we learned in the last decade and applying it to a more modern game and we also have a casino set of casino games that that we launched a number of years ago and continue to maintain and so now we're working on a project called codenamed dolphin and I can't talk a little bit more about that until it's closer to lunch but we're hoping to launch it like the later this year and its really excited it's built on the kingdoms of heckfire and Trimble okay so psychological safety I played and ripped this off from the Google rework website which is a great resource for anybody looking to figure out how to manage more effective teams but the idea of psychological safety being you know something that's built on dependability structure and clarity meaning and impact and I just thought it was a really

useful framework to think about live ops and so I'm gonna try to use these concepts and apply them to how you can build this within your own organization so when we start talking about psychological safety how do you set up an environment where your team can feel safe to make mistakes and be vulnerable so in order to do that I think you really need to create an environment in your live ops game where mistakes can be corrected so we talked a lot about okay it's a different mentality to work in live ops once you launch the game that's just the beginning but what does that actually mean so from a practical standpoint you need to make a system where deployments are just easy and as risk-free as possible your technology stack needs to be dependable and people need to know that when they want something anybody can launch something it can be quick we even have our designers and artists able to fix their own mistakes and then deploy those mistakes into an environment where

it can be deployed and our designers actually do their own deployments and they commit their data through get so that we have a whole history they understand how to branch they understand how to push code and how to push data and that means that they can change anything that they may make a mistake on we can react to things really really quickly and you know last but not least oh something happened because somebody in chat can you guys still see my presentation everything okay yeah I can see our presentation okay we'll do a safety having a separate environment to create a safe playground okay cool I don't know what happened but something happened but will good coastal and so customer support and community team to escalate issues quickly in a culture that's responsive to their reports right so that just means that when in a live environment when things go wrong you need to take reports from your customer support team in your community and your players really seriously and be able to

respond quickly and I just want to like be clear that this is not an easy technology task like to build a scenario or a system where you have people authoring data and lots of different people authoring data and pushing it into a live environment in a risk-free setting that is really really difficult to build but it's the thing that you should be striving to have because when people have an environment where they can launch things and they can correct their own mistakes that's actually where you're gonna have the most success in in a live environment right and the next part is creating a safe playground so people should be working in an environment where they have task based maturity and this is actually more of a management concept but it applies really strongly to your live ops if you have somebody working in an area where they don't know what they're doing and they don't have support to figure it out they're not going to feel safe but I'm going to feel like they have a place where they can experiment where they can innovate and when they can try new things and so you want to put people in places where they have task based maturity and if they don't have tasks based maturity they should have mentorship and guidance from somebody who does you should be making tools that

work that are efficient and people should be encouraged to be explicit about what they don't know what I mean by that is if somebody is working in an environment we often tell people they could tell you make it that's terrible you don't actually want your your reports to be faking it till they make it you want them to be explicit and ask really good questions when they don't know something and that needs to be built into the culture because professional culture actually teaches people to opposite and that's not healthy for creating a safe playground for people to work and encourage people to do peer review this is standard for engineers to do code reviews and but it's not the standard for designers to do reviews at early stages of design or even you know artist is a little bit more common but you should actually be building that into your process that's how people review each other's work because of the consequences of making mistakes are low right so from a practical standpoint if you want people that feel safe you need to make it so that when they do make a mistake they can correct it right so you want to make sure that they can create it correct any mistakes that can be made they want that they feel safe and call

in the work that they're doing and finally if they do make a mistake be you know prepared to deal with it encourage people to write test plans for what happens if things go wrong Dupree mortems whenever a task is not routine meaning if people are working on you know pointing to date data that they've done a thousand times the processes are robust you're probably not important but not as important for them to let's say think about what happens if it goes wrong but if it's something new really encourage that I'm doing pre-mortem that's fine what could go wrong be explicit about performance in your one-on-ones right so don't make people feel like they make a mistake they might not get promoted or they might get fired those conversations happen as part of your management to happen in one-on-ones so remove those performance metrics and the performance conversations away from your live ops environment where people can should be encouraged to actually make mistakes because they know that they can correct them and that they have confidence in the work that they're doing and make people aware of what the cost-benefit analysis of is of a given task right so that they're aware of the stakes if you want somebody to work on

something and you don't think it's going to have a big chance of success but if it does it's gonna you know be it's like a big bet it's gonna be really risky but really rewarding then tell them that so that if it fails they don't feel like they screwed up they felt like hey we took a shot at something that had a you know low chances and get our best rather than feeling like they're like they screwed up so be really clear about that good stuff and you need to do that to create this psychological safety last part is like testing an oversight need to be really supportive right you need to actually give people time to test their work and to feel comfortable you know encourage other people to test their peers work as well it's okay for your high speed engineer to spend some time testing you know a design of a junior designer and give insight fact that's what you want because they're gonna have really good ideas you want that cross-pollination so give people time and encourage them to test each other's work you should train your managers and coaches to be asking good questions that are supportive and not micromanage this is always really really hard but investing in people's

leadership abilities and skills is going to pay dividends by creating that safe environment and finally teams need to have each other's back this comes from dependability so how do you dependable environment and LiveOps fix your tools right people need to trust tools and processes to work as intended if you have a broken tool fix it even broken process fix it you have broken data fix it you know you need to do this and you might not be working on the most exciting roadmap feature but in order for you to build a place where everyone's working on features and thinking about the next biggest thing they need to know that the routine work that they're doing the tools they're working the processes that they're using are all the right ones right encourage people to do start stop keep exercises very simple you get everybody in a room and say as a team or should we start doing well should we stop doing and what should we keep doing and have people vote on it and the insight that you'll get from that is phenomenal do post-mortems on launch this features processes right do them consistently and thoroughly think about things like quarterly reviews 360s and

if you're a lean of a department do skip levels with everybody in the department and collect feedback that will allow people to feel like as long as you take action from what they say they're gonna feel like they can trust and depend on the organization and then don't this is the most important thing for me I think that I see people do wrong is don't set unrealistic or arbitrary deadlines if you're working in a live ops environment quality is a really really important and dependability is really important if you tell people we need to launch this thing this quarter or else and then it launch it it doesn't launch and nothing happens they're not going to trust and depend on those deadlines being accurate they're gonna always blow up their estimates so be dependable and transparent yourself be responsiveness to live issues is priority this is part of that dependability thing is that you need to build on call processes and who's on call and when if you're working in a live environment that should be explicit and it should be fair it shouldn't be the same two people on call constantly have very clear escalation processes what do you expect for people to do when can they phone their manager when can

they phone the developer when they're not being responsive or the other person on call at what point do is it all hands on deck like those things should be explicit because they're going to happen your games going to go down you're gonna go try to launch something on a Black Friday and somebody's gonna forget to turn something on and everything is gonna break and you're going to need lots of people to figure it out so be explicit about that people should know build alerts into your tools make it so that you know if something is going wrong that the your system alerts you for us we get slack messages if there's leg in the game or if the server is not responding or and then we also get metric alerts right so if like you know a metric falls outside of a band have expected you know performance we get an alert about that and that allows us to focus on work and not have to be constantly checking metrics but if you do have to check metrics make dashboards that are very quick and easy for people to look at make people who launch things responsible for what they've launched our engineers should not be throwing things over the fence and saying QA we'll catch this they should be

responsible for the ultimate performance of that feature that launched that piece of code and they should follow it from start all the way to finish and make it some of that chain and responsibility is explicit and then encourage teamwork to solve issues and break down silos and so what that means is on our on call process it has someone from each discipline so if something goes wrong and is needed by the Steichen designers neither engineers needed that those people are available and they work together finally people need to know that they can trust their their leads to be dependable and support innovation right so again train your leads to give coaching and give radical can your radical panders when you care deeply about somebody and you challenge them directly right this is an environment if you can create that culture of radical candor where things get solved because people aren't afraid to have hard conversations and they should actually this is another management concept that you know creates dependability is leads should delegate the things they know best there's a tendency when you become a new manager to want to delegate all the things that you don't want to do but in fact you should delegate the things

that you're best at that's what you can train people that's what you can evaluate and that's where you're gonna have the highest leverage and go off and do the things you don't know really well until you have competency in it and then you can train and teach somebody how to do it but if you're delegating the things that you don't want to do or things you don't know how to do you're not gonna be able to effectively lead people and so people need to know that you actually can do the work that you're asking to do which is the next point you want to be dependable actually dog food your own work go to the front line to answer some support tickets go and use the tool that everybody says is broken yourself do a launch figure out what's not working and feel the pain that there's that they're feeling because then they'll know that you actually have done the work know that everybody understands that paint and they'll feel like if you're making a trade-off between building a tool or just doing the launch that you actually are feeling the pain that they're feeling and then act on those reports of inefficient tools and processes and make that part of your organization to do so and there's any of all other people to make changes around them if they see a process that's broken encourage people to speak up and fix it that's how you create a dependable environment finally you need structure

and clarity and so structure and clarity comes from goal-setting first of all I think so one thing that happens and I talk about this and when we talk about impact is that if you see an organization where maybe the CEO comes out and says hey I want to actually increase profit by 15% this quarter but the rest of the team doesn't believe that that's even possible because that's a number that the CEO pulled out of thin air and they're not going to feel like they're like their structure or clarity because they don't know how they're supposed to get to those goals and erodes their dependability and their owns their psychological safety so instead have your team's build the goals and if you need to improve profit then you should have the leadership team should go to the people that are working on actual features and say look we really need to improve our profit which means we need to like lower some of our costs and increase some revenue or one of those two things and have the team's come up with you know what's reasonable what's possible you can push them a little bit but if your CEO gets up in

front of the team and says hey I've been talking with all the teams and I think based on what you've all told me that we as a team can improve profitability by doing X Y & Z which is what you told me we could do I want to challenge you all to actually do a little bit better and maybe improve profitability by you know 17% because I know that that's ambitious but realistic and that's gonna be much more powerful than you know setting goals top-down so I really encourage teams to actually set their own goals and it's a really powerful team for teams to learn how to do this active you know setting a measurable time down and ambitious goals and then transmit those goals across the organization be transparent communicate them globally people should be checking into those goals consistently so that everyone on the team knows where every other team is on their goals we use a framework called okay our objective key result also blatantly ripped off from the Google rework look and we've adapted it to our own processes and then we use seven geese which is a tool for managing these kinds of one-on-one conversations and goals and allows us to check in to a call and

transmits it globally and then finally goals for teams should not actually be tied to their performance in fact when we talk about performance for an individual and we talk about psychological safety and structure and clarity people need to know what their clear responsibilities are so between people create working agreements that say hey as the engineer for this LiveOps process this is what we're going to do and this is who's going to do it and this is what the process looks like right those processes down sometimes you know we get into this tendency where we feel like oh well things are changing so fast and LiveOps I don't want to document any things it's wasted time but the actual act of writing the document is really powerful the act of actually running down and committing to what you're going to do forces you to think through the thing from a different lens there's a good razor for you to evaluate what's important finally to find the roles and responsibilities for each discipline what is each discipline going to contribute to every project and be clear about that and be really explicit about your expectations right there should never be any like couched hidden expectations about responsibility be

really clear have your managers be really clear have your entire leadership team be really clear about what they expect from each team each department and that kind of that goes back into creating meaning for people and people get meaning not just from things like hey we're working on something that you know maybe it's changing the world or it's like a social issue or but if you also get a lot of meaning just from knowing that they're growing professionally and in order to do that they need to have a concrete career path and they need to know that the tasks that they're working on are contributing to their career so everyone should know exactly what they need to do to get promoted and they should know the timeline for their promotion if you don't have a good sense of that across your organization then those people don't feel like they have meaning and they're not going to feel like they have structure and clarity and they're not going to feel like the organization is dependable or their peers are dependable when they get into a live office environment it can be really difficult for them to feel safe making these things and so what we've done is we've created really clear career tracks for

every discipline and we focused on impact skills what I mean by that is people know that they need to have an impact on the product but how they get that impact can you achieve through different skills and maybe you have somebody who is a designer that's really good at let's say user research and they can discover insight from players by talking to them and they build that into something and it improves your metrics versus somebody that maybe is really good at Dana mining and they don't mind some insights and they improve the product both of those things are you know have a level of responsibility and a scope that should allow people to know where they fit into your career track but they have different skills to get there so when you have your team developing you maybe don't have people that all look the same with the same skill set in fact we have a really diverse team that can achieve goals in a number of different ways I think you should celebrate that finally across teams calibrate your expectations right you want to give people concrete career paths and growth tasks they can't be worried about you know oh I wish I was on this other team where they're all getting promoted for doing the work that there were this really easy work and so

make sure that you're really clear about that it's really incredible to me how many times I talk to people and other organizations who don't have any idea what they need to do to get promoted I do this LinkedIn expert mentorship program where it pairs me up with people who want to get into product organizations and the number one question I get is how do I get promoted how do I move up in my career and the idea that their managers aren't having those conversations with them and it's an implied thing it's just bananas create safety for people so they know exactly what's expected of them don't have to worry about it they can focus on the work feedback is really important feedback internally feedback externally and feedback from the players and users of tools I'm getting mature on time here so I'm missing be really brief on this but encourage people to talk to the users of their work and encourage whatever channels you have for feedback to come in about the impact of that work and that will give people meaning when I see an engineer who you know we say oh look you improved in a performance by X amount like oh cool that feels good but

when I see a player say like you know I see floods of people come in saying oh my god they came so much faster I'm so happy they just light up they light up in a way that is completely different so make sure that people see that so that they can have meaning finally create external opportunities for people to send them to conferences give them you know encourage free to publish their work talk about their work so that they get you know meaning from feeling like they're an expert within their field and then the final thing that this all builds towards its impact you know so be clear about how your company success when you're setting goals how each team is going to contribute to that you build a recognition of the culture and calibrate your promotions celebrate your promotions across the entire company and if everybody if you do all of these things you know finally when you celebrate a promotion people are going to know exactly why that person got promoted they're gonna see the work that they did and they're gonna feel like everybody has impact on the team and again build company goals off of your team pool so that's kind of what it all culminates into is that you should be setting your expectations from the ground up from that live office environment comes everything else if people feel safe and they know that

they're gonna have to impact on that work and then the last but not least is that everyone should feel part of new ventures this one's really challenging but what we do is that we have incentives such that everybody high tide raises all boats if our company is successful everybody gets a share of the profit it's not tie if you're on the team making a ton of money you're not going to get a bigger bonus than the team that's on a new venture and so everybody feels like they're all in it together and you know that is more difficult but it's something that I think it's really important for this studio so again how do you build all of these things it's a lot of work but it basically comes down to in order for people to feel safe in a highly complex LiveOps environment you need to build all of that foundation before you can get to a point where people are willing to fail and it so easy everyone always says you know stuff like fail fast and fail hard and you know celebrate failure right well how do you do that you do that by building the right foundations that people feel safe to fail so I've gone over my time and

that is the end of the presentation I had to talk really fast and condense it but that was great I was fantastic and I think you know I think you nailed something there which we talked about we never kind of find a good way to express which is that you know there's a reason why you don't do crunch in life but you shouldn't crunch anyway but that's another subject but the reason why you can't do it in life is because you've got to release something next week and if you kill your employees they can't do it early now I think you've gone further maybe but actually you know this is about having a common endeavor and you've got to have that meaning that structure that that support I mean do you think do you think this is something that people are starting to understand well I hope that's a tough question I personally have been doing this for 10 years and I'm only starting to really understand the scope of it I've always focused on allowing people to just fix their mistakes and thinking like as long as you can fix your mistakes that's the most important thing but I didn't

realize how much things like career tracks contribute to people feeling safe because now if they make a mistake they know like I'm not I can still get promoted because I've you know I've been doing things that make an attempt and they make an attempt in the right way and I know what I need to do to get promoted I don't have to worry about that so much more if you ask somebody hey go do this tasks that really really sucks like I need you to QA this build I know you don't want to do it they know that like yeah I can do that because it's not taking away from my other work I know still that I'm gonna have a path to be promoted I know our path to career growth and I know how it contributes to team goals I mean it's I think it's fascinating because you know having started out in telcos I didn't have a career part in fact the fact that I'm that kind of person that just has to pick some things up and run with them now I could never be promoted right f3 they hired five people to do my work because I'm a stupid idiot they're just it's important and any way I can get promoted is to leave and I think it's a

tragedy I mean I'm not saying I wasn't good but it's a tragedy for people to be fit to feel like they're forced to quit rather than actually kind of run with the experience and actually kind of benefit and like you say all all ships should rise with that time and we have some tons of questions by the way the first one now I from which partly the talk this came in but it came in in the chat rather than the Q&A which okay ways that you prefer customers to inform you of issues with running of the game I how do you keep the feedback and make it active as much as possible yes so we're really lucky that our tech stack is built on a chat engine so we actually have in game chat where players can send us essentially help tickets that are a chat tool that we've built ourselves custom and we spend a lot of time and money in building that because we thought it was important but we also use discord discord is a great tool like if you don't have that in your game like put a

link to your discord start a discord get it to be an active place because people can actually send you can create a channel that's called bug reports in your discord you can have it alert you and alert you're devs that are on call but really it's whatever channel works for you you can use forums you can use Twitter use Facebook you can use all of them you know just depends on the size and complexity of your team but find something yeah there's a company called help shift that has live chat I think they send it to a server first before it gets sent to you so it's not like instantly live but it's pretty close but yeah like absence of that I would just use of discord it's a great tool it's um and actually one of the one of the earliest tricks I saw people doing in mobile was simply putting an email address in the front of the game because you know just that save so much hassle for getting the terrible reviews when you could have just say somebody talk to me yeah I'm with you dis good is a better much better way I'm not a huge friend it's good there's lots of things I know but it does the trick yeah got the right community manager well about it is you can set up notification so you

can put it on your phone and somebody can PM you and if your persons on call they actually get a notification on their file exactly I think those sort of things that just make life a lot easier I'm not a big fan of doing I don't do DevOps so I know how big a job that is or I'm a fan of is making sure the community teams now has a triage yeah exactly I think people don't understand the power of triaging because if you do it right then they only ever answer FAQ good not it needs to be kind of like stale but you shouldn't be expecting your team's to necessarily guys are talking their front ends and know everything about the game yeah we have autoresponders so with something that people ask a question too many times we have machine learning that figures it out and then we just write in a response like eff faq and actually help shift as it does that as well so they try to like make it so that most the questions get interpreted and answered by a FAQ before it hits you I'm going just right look at the help shift now so Top Chef yes so there's another one here which is from nee awake great or what which are the best ways

you prefer customers oh and this one was this met think the meaning is the particularly Juliet Dupree was talking about this kind of approach in general I think particularly meaning so this is something HR professionals like yourself constantly try to educate managers on this kind of approach what made it all pick for you I actually think this is all really new to me in a lot of ways I'm always kind of like a found different parts of it but what made it all kind of come together was I wrote that I worked and built and wrote the career tracks for my department and so I was revisiting unclear career tracks and I did interviews and skip levels with every single designer on the team and I and I was really explicit about asking them how do they feel about their growth in their growth path and what's working what's not working and I just discovered lots of misinformation that they had that they and then I found that their

managers were repeating that misinformation because they didn't have accurate information and so then I worked with my managers to come up with like a career track that allowed them to to grow people in a way that was that worked for them and used the right language and just was really explicit with it presented it to all of my designers and it was like I could just literally feel the anxiety lift out of the room and every skip level I had after that everybody was super happy and I realized people just need like the most most important thing that people actually need is that you know meaning in their work that makes them feel like they're both connected to the impact on the company but also that they are you know progressing in their careers and if they know that that stuff can just be like removed from the conversation like it's really explicit it just is so much easier for people to do the rest of the work right that's just such a hard thing to do and I thought it was so so I thought I thought all of our career tracks made sense to everybody really did it just come back and saying thank

you for that answer for her it's very it's helpful because you know this is a transformation transformative process absolute I think getting people to understand the kind of discover that's where exploration as well as you know you know basically looking what's available for me I think it was interesting I was in a number of different taco companies running games projects before I kind of join I went off and joined there's an nvidia and sony afterwards and then became a consultant but in those early mazing how those big organizations are very good at talking a big game very good talking about empowerment but very bad actually separating the responsibility from the authority so that i think is the biggest problem if you give someone responsibility but you give them no authority to act on it how can they have meaning yeah absolutely yeah and if you don't give them clear success metrics like this is what it means for you to be successful like it

the one that bothers me the most is the is the fail fast and don't be afraid to fail right that sounds like it's gonna be the soft skill like oh it's okay you feel everybody feels good about it that's not the truth right nobody feels good about failing so you have to create that underlying support I think this is what people like Gil Kerr have done so well in the way he's creating the whole culture of the business of supercell around this idea of celebrating that failure properly but he doesn't you know they follow through in it and I think finding finding ways to follow through and like you said making sure that it's clear that people's activity has meaning wherever they are in the organization yeah and I think that's something that's a magic treat to pull off absolutely you know and like I said earlier yeah I just don't think you can underestimate the difference in mindset that you have to have as a team when you're running a game that's gonna continue to run for a long period of time yeah and you know if you really want to see where the problem is your live ops team is go and do customer support for a week oh yes true

oh I'm CEO go do customer support a week and it will transform the organization I think any manager should and at least you know one day a month or something doing that I mean I'm down I say that I'm sure it broke sure I wouldn't end up doing it but I remember going up when I was a BT when we had this and why I played the online gaming service a dial-up modem long time ago had to go to the nor this part of Scotland and sit-in in a call center for a couple of days literally eye-opening it completely transformed my understanding what we did as a company let alone what we were supposed to do yeah yeah fascinating stuff on that note I'm gonna say thank you very much David that was fantastic really really like you know the whole approach you're taking and hopefully a lot of people learn from them anybody wants to talk and work I'm using to find just check me out on LinkedIn a unique name so I'm you see the frugal it's easy to find thanks Taylor

you