What is the 'gold standard' for live ops?

[Music] okay great yeah please take a seat I'm gonna I have the honor of moderating the panel and asking some questions but before we go into that I think it's best if each of you do an introduction explain a little bit also about what is your role in in the company what kind of company is it as we're talking about life ops I think for the audience it's always interesting to immediately also get a perspective like okay what are the games that are your you're running in as life ops how big is the life of steam these kind of things so maybe me how you can start hi my name is Mahesh vision quite complicated sorry about that not my fault I'm at ember we do a bunch of stuff and one of

our Studios handles LiveOps we are in the studio about 20 to 25 people and we usually run two to three LiveOps projects in parallel they can be both casual and mid core yeah this is the pleasure of pronouncing names yes my name is Sonia Angus so I work for xenos product lead losing is a large company so we have about 30 live games and depending on the team the live ops team actually Averroes as the team can well team actually has quite a lot of impact on the the size of the team plus as we are a large company so we have the benefit of also having central operations

basically different teams helping out with different campaign analytic segmentation like yeah benchmarks best practices so forth hello I also have a complicated name my name is Bora angiocath I am working at play Raven as a live producer before joining Craven I used to work at netmarble Games as a product manager so right now as a player even we are focusing on the Neva our Eve online game II were of a session so right now we are trying to build our live of structure and the themes okay great so maybe before we kind of look into gold standards which is kind of an interesting topic I didn't make up this topic by the way let's let's maybe

because I felt like this afternoon there were a lot of I think maybe because it's a relatively new term there are some people were talking about life services others were kind of yeah big teams with dev developers part of the whole thing then at the very beginning space I was talking about like ok our life ops is like for people that's that's what we do and there is no development because we basically cut all that out so maybe for each of you like just in a short one what is life ops for you and what is it not lie Bob's running game as a service without developing any new content or actually or even utilizing the development team so I guess our view on live ops is quite actually close to space space apes view on it ok so we

usually work with external clients that means that for us live ops means not taking over an existing project that the client doesn't want to invest the team anymore in it doesn't mean that it's at the end of its lifecycle it just means that for them it makes more sense to put certain team on a new project hand it over to us so we can you know increase their KPIs do you then still also address kind of development feature development and these kind of things or is that somehow somewhere done separately it depends on the project and the client needs in certain cases you only do like events or promotions in other cases or in other instances you have to add certain features or more content depending word what are the KPIs that needs to be improved okay what about you yeah for me

I said increasing the overall performance of the game and lifespan VDOT's now looking any other features basically this okay so when you start so at the beginning very beginning of a game what do you what do you think is kind of the most important thing to plan for so when you when you kind of start planning for life ULBs what is what is kind of the the one thing you know out of your experiences with with life ups what is this one thing where you feel like oh yeah we should have really really first time I did it wrong we should have really addressed that right from the beginning is it around tools or processes team what what is it um well I would say maybe at least two things one is that mmm too often development teams start to

focus on LiveOps very late how I see operations so different cycle of activations campaigns events whatever you are running in game or in the community so basically also the game so all of those should be considered from the beginning as it's it's so much easier to play sickly have the bigger picture like where are you taking the sort of Elda Elda game all the players when they have spent like hours and hours in the game and not to you know jump into the Challenge of we should actually do some sort of events when you actually see the players churning and I think the other thing which I sort of already mentioned was that too often many studios seems to think that LiveOps equals in-game events which is partially true but you could also do quite a bit

of interesting activations and campaigns like by customer support or sort of within the community which basically drives the players to the game so they drive the KPI as you want to improve but you don't actually even have to do any contents in the game so for us because again we're working with clients mostly the most important thing it's planning for the project takeover and syncing with the clients and giving the team and the client in time to to basically get in sync the second one is to get to know the technical challenges of the game there might be some things that you cannot plan for like things that need to be upgraded that will take two Sprint's to to do so instead of providing better

KPIs for a client you end up just upgrading tools and it's not something that you want to get surprised by yeah I think it's essential to know that once your aim in their lives so what's your you must have a realistic plan for it so what kind of activities I want to run in soft lunch at Global lunch or when the game shipped what you're going to do in the first three months so after thinking about this and you need to think about if you have the required tools or do you have any solid plans to move forward so you're in the middle of the eve game right so you're in the middle of kind of the strategy around that so yeah what are kind of the key things that you're currently looking at so right now we are in software so we have a chance to experiment in live obsessive so each day we see new challenges so and also when we run the game and we see what what kind of things were missing enough so we can start building these

before hands and before the global lunch so this also gives us a chance to communicate with the players easier so we listen to them what kind of things they need and we start start to build the game based on this okay so are you are you already kind of investing also heavily and tools and these kind of things yes meanwhile we are developing the game features we are also investing on them live of the features tools and missing my webs features as well so two things are going at the same time in the same direction okay so in terms of maybe as a rule of thumb because we're looking for gold standards I would say maybe best practices or some some experiences what would you say like is a good team size also four four four four life ups you know because we were hearing earlier like 50 people 40 people 35 people teams

sounded for me quite big then you know a presentation later it was actually broken down in like three parts three times ten but yeah what is your experience like good team size to kind of be able to kind of manage life ups and get also something done yeah actually I witness teams of tons of people or small teams it all depends how big you wanna run your operation and what kind of signal set you have for example I was looking for the Asian companies and I have hundreds of people working on my ropes they are running the games for at least several years for each a live event and they are making live cinematics animations all these kind of things so since this is a really big ocean and there are always something you want to experiment or go into more details but if you have necessary tools and signals that you can manage the game

it small amount of people to yeah I guess that's the it was exciting to follow the previous presentations as the size of the team actually varied like greatly I'm a big believer of automating things and utilizing well as you said like basically tie your feedback that's super super valuable but also of course like quantitative data so it's sort of a combo of that and that basically leads to tiny LiveOps teams but it's it's yeah talking about standards and talking about like best practices well it again as I think you said like it depends on the size of a company or a size of the studio or the team like I consider well I don't see actually live obsess as like a separate thing out there but some or someone who

is actually very heavily collaborating with me like product and analytics and marketing and customer support so it's it's sort of a amoeba-like live ops core is just a couple of people yeah I'm gonna back that claim usually it's five to seven people if you try to be as efficient as possible obviously you can go to 100 people but yeah I would rather be efficient and you know spend or throw just bodies at the problem yeah I think so too I mean if you have like a really small team because it is like a team effort somebody said earlier it's like the marathon that actually starts you and culture is so important so you have to kind of really you have to do it together right so and that usually only works really well with kind of a quite small team I would agree and kind of you

know 100 people teams or something like that that would suggest that you know in a half a year's time everybody's gonna be outsourcing life ops to some cheap markets or something like that I'm not so sure that that would actually really really work so but I think this is something that definitely kind of the the the MMO space has has done a lot with kind of big team so I was a long time ago CMO at Jagex and we had back then like I think I don't know exactly what I would say at least 150 200 people in in in kind of just community management and kind of all that on site which was felt extraordinary and then it's like okay oh we have all these bodies maybe we can kind of outsource all of that but yeah I think the the industry has also moved and it's more now about how can we automate certain things to kind of

actually really keep these teams efficient its small and kind of everybody knows everything a little bit etc you know I I kind of agree but is there anything else that we could kind of learn from the from the MMO space so to say because they've kind of they've been preaching kind of the living games for for a long long time and some of our games I feel in mobile at least are kind of moving towards something similar ish and be interested to see what you have to say cease you're working on a mobile version of them I know Morgan yeah so I think about the MMO games are and they are really community based and they are really hardcore players really dedicated players own it is good to observe how they manage to their community so that's why I'm while wearing soft lunch we always be in touch with the players we are doing as mating sessions these core sessions or trying to get as much as feedback we came from players so we can

maintain their expectations for the game so as yeah as I said many more games are always coming to driving so we need to keep it on focus on that part so I don't have huge six videos from the MMO area but I'm also sort of lining with like well maybe looking at the sort of perspective that what's basically lacking in in many mobile games and that's social and I think that's something that actually mmo's well quite naturally like leverage and also I used in in order to engage the players over a longer period of time and you're also creating different types of raids or events or what-have-you so I think that that's something that well soon as focusing on casual and mid core mobile

games and I would say that even with with our muscles like the social ideas are quite weak so that could be something to be more heavily pushed to live ops type of activations whatever they are yeah I think that mmo's are by definition they are very social like truly social experiences you play with your friends you interact with friends on a daily basis and I would be really interested to see if us as an innocence in the mobile industry we can do something similar with with games on a long term so how can we turn you know this this thing about social that we are so proud of in in mobile how can you turn that into like real social interactions I don't have an answer to that if you're wondering yeah but it is certainly I think the social aspect is certainly something it's kind of the

glue very often of a game I mean if you if you if you think of life ups that is something for kind of the long-term it is much more about kind of in my opinion at least really like bringing something fresh and new also for somebody who has played the game for a year or two years or something like that it is about that retention and I would even argue that those players no matter how you categorize them about this game there very very cool so to say I mean they they know everything about that so I think yeah I think in mobile it's may sometimes be a bit more challenging to kind of get that social glue also preferably inside the game and not somewhere on like I don't know a Facebook or reddit or wherever but I think that can actually also make quite a quite a difference right do you work with kind of games where you have like real where you have a lot of chat and

these kind of things or is it like the social interactions very shallow yeah for our for our games we have chats also you can have some social features that you can attack players together reinforce them help them so without the community you can't actually reach a certain level so you need to always be a part of community help each other's yeah yeah so we have very strong like social features in like games like seeing a poker with friends and and such but yeah still I don't think they leverage the aspect of social well enough in in like different live operations yeah we usually do community management but we usually do it through outside channels so channels that are outside the game so

I would love to see to find a way to have better interaction inside the game that goes beyond just using chat again I don't know how to do that but obviously it's it's a good challenge to think of yeah so yeah I heard somebody earlier today say yeah and then we hand it off to live ops with it where I felt like okay this is maybe a bit degrading you know in a way because for some that's when the journey actually really starts and I think there were enough examples actually where you could see that the journey actually then starts but in terms of like kpi's what do you kind of look at is this more as a is it you look at the specific game KPIs to kind of continue to improve that are there also like company KPIs that are important for a kind of size of life ops and these kind of things what are

what our golden standards they're okay actually this is one of the questions where they actually might be a go lose most often like yeah it's it's sort of depends but well I would claim that majority follows engagements and retention it's quite yeah sort of basic basic stuff and then just you know digging deeper in terms of what are actually doing like if you are doing a specific event what are you going to trigger like what type of actions whatever targets do you set and by the way that's something that you should always define if you don't from now on you will so you know figure out the hypothesis and actually targets when you start the different campaigns or event it makes it so much easier to evaluate

if it's if you are on the right track or not and you can always improve yeah as Tonya mentioned we usually focus on engagement and intention metrics but I believe in the long long term these kind of metrics also affect business key pairs as well because our idea is to make our players more retains more loyal more happy so in the end we get more loyal users so we can monetize them maybe in feature so it also depends on what kind of campaigns you are running for example let's say that you are running and events to bring inactive players back to the game so you want them to join back to the game as for example they didn't log in for 15 days and you are making campaign to retarget them from with user acquisition so in that since the matrix we follow is retention matrix also some of the array related

marketing metrics yeah I was asking this because I know I mean Nick didn't really have time to go into that but because Spacek for example they basically look at it they try to improve the the revenue per life whoops person there is a really good track in the GDC volt also about lean life ops and there they specifically say like okay they try to continuously increase the revenues per life ups person so you can obviously also increase the team but then equally you kind of have to also increase the overall revenues but that obviously helps you focus very very much on efficiency efficiency efficiency to try to kind of squeeze that to to to the max beyond kind of obviously monitoring games KPIs and these kind of things and

looking at kind of the health of the game and where you can kind of improve and tweak things do you do a lot of a be testing in in life ops maybe you can share some experiences there some some best practices so yeah how much do you do that well again as a large studio we have own tools to do maybe testing and also what's basically I think crucial part of it is is segmentation so we segment players not only based on consumption or whether they are like lapsing or active or whatnot but actually focusing more on motivations and different types of like behaviors so that is also a key part of sort of narrowing down like what are we actually who are we actually talking about as

easily when you are looking at that massive like a big number of our user base it does not take you that far if you just look at you know a user base as a whole so or even like for consumption like how much they pay if they if they do but for us like segmentation and a/b testing goes like this so they are two separate services but they basically are in love but yeah we do well again it's it's a bit cert to always say that yeah it depends a bit but yeah it actually depends a bit because it depends on for example for my product it depends on me I basically think like what would actually make sense for my game so but yeah I'm a big believer of testing different type of split testing so I I do eat quite extensively also for even

so because it's people are quite conservative so specially if they have spent quite a bit of time in a game so they basically expect certain things and if we happen to change the certain things yeah we want to make sure that you know we don't piss off the players we recently started making a B test for Eve more office tension but if I can give you some examples from my previous works who used to it's an important part of the life of increased efficiency so we used to use a B test for almost all kinds of in-game activities for example offering a package timing of it and the color of the button or discount rate and what kind of image we want to use or content of the bundles so if you are a be testing all of these think Reese the efficiency to the maximum yeah obviously

I be testing ABC testing it's it's like the basic of LiveOps we do a bunch of other stuff we look at player progression for example see were player where players are are stuck and what and how would that impact the future of the game for example if you have a level where I don't know we you have 40% of people churning let's say and you say you you decide to make the level much much easier you need to take into account effect okay how will that impact future monetization of the players that were already in the game or only the Atlit at that level but they were monetizing better because the level was more difficult so it's it's a bunch of stuff that you can look at and it's it's part of what makes live ops quite fun to work with because you you look you work

with numbers and you can see quite quite easily the impact and the forecasting of each action that you take okay so what would you tell the audience what was kind of what is one of the biggest mistakes actually to make in in in life ops where do you think it sometimes can go awfully wrong not knowing your audience not understanding the weaknesses of the game not paying attention to data I think that's like a capital mistake for me okay yeah there are so many possibilities - well I think actually the biggest is that you know you just start too late so it should be like part of the full or total experience not something you glue on top of when you see that players start to

you like get you know done with all the existing content you have prepared I would say that not knowing your audience and not knowing your product you need to be really expert with your product know its strengths weaknesses so you can work with it better yeah maybe last question and then we're going to open up to the to the floor do you see any kind of trends or what are you expecting like in the future I feel like there has been it has become a little bit more popular let's put it that way to kind of it almost has become a little bit of a buzzword even though it's still a bit blurry for many but you know what do you see is like the key trend or where do you expect are we going to go with with life ups I think it actually became more data driven than it used to

be and also it became providing more personalized experiences to the players along with that I think that brands and big companies also started to realize the importance of games so I'm expecting to see more promotions in game it's connected to live ups with other brands and companies they want to reach different audiences so we can see more stuff connected to live ups from different brands very good comment I agree also well I believe in social I believe in local now that we have data qualitative and quantitative we can actually provide local events or experiences or our campaigns whatever that is and automation so there is lots of things to do in that area things I was gonna say automation but yes I think

that but automation is what I think that's one of the biggest things that we can get way more efficient at so again we're gonna see you know smaller and smaller teams or teams that are more efficient in adding actual experiences to to live whoops projects and put a lot of the day-to-day stuff to the routine stuff on automation ok ok so yeah before we do Pecha Kucha i think it's in the other one in ten minutes so some questions you're gonna get a microphone and there's also a question yeah thank you very much for now and then some questions all right I think you were much something you mentioned earlier in the metrics that you were that were key to calculate its engagement for you guys at your company

is how do you define engage it and how do you make it a specific metric because everyone uses engagement as a collective of retention number of session but what is it exactly for you guys yeah so should I start with it depends on what are we looking at yeah so we are looking at like the stickiness over time we are looking at like session metrics so again it a bit a bit depends on what are we actually looking at but it's it's it's quite basic things regarding like engagement yeah for me for example before making any kind of live events or live campaigns we try to understand what would be the impact and then we started defining the metrics we want to measure for example let's say that we want to have a huge PvP event and we would like to see our results in increase in session times more battles more

engagement among among other players so yeah it depends on what you run you create it depends on questions and hello you mention from jiva I would like to ask if you have experience or you see potential in life of support of things outside the game for example real life events or the support of forums or other communities so if you tried some of those or you see potential in the future for those thank you yeah I think I already mentioned that yes I believe in and we actually have very good results from events that are not actually happening within the game but they are like activations that just trigger players to play more or play

certain level or certain you know with a certain goal so we have nothing but very positive experiences so II thought that goes more towards the like customer support type of or VIP type of activations when you have big enough and you can do also like more targeted activations outside the game for specific groups of players yeah we call it community management and yeah who you you need to do it you need to get people involved as I said before it's part of making the game a bit more social and making people feel part of the community it also drives engagement as well yeah I believe it's an important thing community events for example when we first announced the game it was in the

Yvonne fest in Vegas so we had a really good comments from the players and they first see the game in the event so it also kinds of bond to players to the game and crazy solid community okay well I think that's a wrap okay thank you very much thank you Bora and Sonja and me hi and yeah [Applause] [Music]