Panel: Live Ops Disaster Stories: What We Learnt

[Music] [Music] and next up we are going to be going into what the topic i've been looking forward to most and it's not because my business partner is hosting it and i can heckle from the sidelines but it's actually because i have so many of these myself that i've experienced um we're talking about liveops disaster story so ella head on up and and i'll if i can get the panel to come up and join you and i think you'll be uh in you'll be introducing people um hopefully that mic there should work uh testing testing uh is that mike working or do i need to have this yeah yours oscar then you can't heckle me right

um so i'm ella uh i'm oscar's business partner for my sins uh i'm i'm also the ceo of fundamentally games uh as if you've been around today you've probably heard we're a living games publisher so we basically do all the things that publishers do plus run live ops for games hence why i'm here um this panel idea actually came from a digital panel that we did at pocketgamer a few months ago and the request from the panelists was that they wanted to be able to tell the disaster stories so we are going to do a typical panel i'm going to ask some questions but actually what i've asked that is for each of the panelists who have prepared a couple of disaster stories the goal being probably a few laughs but also uh hopefully some learnings that you can all take away and either try and avoid the mistakes yourselves or maybe at least you know be aware of them so um before we go into that i would like everyone to introduce themselves particularly related to your role with liveops so let's start this end sorry i'm mario i'm an executive producer at jagex developing a new mmorpg uh i've been developing mmorpgs

for 20 years i started with everquest in 2001 2002 and i've been now worked on 10 to 15 different mmos both as a developer and a publisher uh so hi i'm sarah um you've definitely i've done my experience great so i started in um gaming around 2006 also in mmos actually doing very early live ops uh which we'll talk about a bit hopefully in a moment um yeah it's mostly in pc free-to-play um and subscription games so i specialize in sort of like taking free to play games from subs to uh to free to play uh as a business model but more interestingly i've developed an interest in social gaming and uh i've been actually working in social game spaces or playing in social game spaces since

1991 back on mushes and mutts hello yes so i lost my voice so i'm sorry so i really gotta whisper into the mic but i'm fully burling background from activision blizzard warner games etc most recently with robot cash so my live ops experience has to do with being an online retail store on the blockchain and we've got a few of those so uh slightly different angle great sounds good so before we dive in um one question because everybody asked this it's a different thing what does liveops mean to you all right okay for me live operations is any product that i'm operating as a continuous service where i keep the development team and a separate team supporting and updating and operating a game uh past the point of retail release to customers so for me liveops is all the effort that goes around maintaining a service

so on top of that a little bit mine's also it's very player centric it's about keeping players engaged in the game and retaining players um and lengthening the time of uh that those players want to spend with you playing your game yeah so pretty much ditto all the above plus um anything that happens after the launch of a game or a product or a platform in the games uh back in the old days you'd launch something and you just pray and walk away and go on your next project that's all changed whether it's free to play whether it's mmos whether it's platform selling games so it's all of that keeping it going i remember we used to have a break when we launched the game remember back and now it's like 24 7. okay right so we're going to go into the disasters so um i've asked as i said each of the panelists to prepare a few disaster stories talking about the context of it why it happened hopefully what they learned from it to avoid it again which we'll see if that that's true um who wants to go first go on try you guys can hear me sorry you guys missed my introduction as well i'm sorry

sorry no right i'm so sorry not noise and so do you want to introduce yourself before we carry on sure yeah hello my name is arjun and i work as a producer uh you guys have already trumped me with your experience uh i've only been in the industry for around five plus years now and i worked for studios like zynga a gaming startup called moonfrog labs and right now i'm working for sega hard light and i've mainly worked on live operations so far basically all these companies have these titles which are forever franchises that have been running for a really long time for their loyal like fan base uh and i've been working on those um and another point just to add on what liveops means to me i think what we do in live operations is give players a reason to continuously keep falling in love with the game that they they are already

playing and i think that's a very powerful factor for me we give players new reasons to fall in love with the game that they've been playing for a while right all right go for it sarah as i said that's much more beautifully put than i said i might steal some of that yeah i'm gonna steal that one as well um so uh to contradict something that oscar said earlier actually i i totally agree with no knee-jerk reactions but there is a point where um sometimes you need to look at things very fast so the example i've got which still keeps me awake at night is a land rush launch it's a launch of a new game be a relaunch albeit a major feature launch and being able to get those players into the game because you'd be surprised sometimes you can't so i've worked on so first sort of free-to-play uh conversion i did we had to spin up extra hardware on the fly um we had players queueing and you know i walked away without going oh my god i'm never going to do this ever again and

several years later find myself only being able to get 10 percent of my players into game which is something you really don't want to do so it could be anything from making sure you've got the open connections in the game on the tech anything that is a barrier of getting those players in the game you really want that not to be there when you're doing a major sort of event when you put a lot of marketing money in or you've done a lot of promotion um any barriers that might assist test it stress it stress test it work through it because that is the point you those players will come into the game only the hardcore will keep trying the rest will leave and they won't come back and yeah that has caused disasters for me before so that's a sad story sorry it's okay who wants to go next uh i'll go next so i think one of the big points you you brought up which is interesting which is that one of the main keys about being a disaster is how badly it affects the players right because you have disasters that go unnoticed but cost you a lot of money and then players cost you money and players so again mine is going to be

coming from the platform perspective so what's unique in our platform is we allow players to resell their games so think of it steam on the blockchain so you have a game you resell it the problem is the way it works is you you basically have your game in your stash and you press a button that says i'm ready to resell it's not automatic it's if there's somebody who's gonna willing to buy it so you get into a queue so of course the first thing that came up during our user testing is wait a second you've got sales all the time i don't want to resell at three dollars i bought it for 49 euros i want to make sure and so we created this feature where you can press a button that says okay do not sell my game during a sales event of course what didn't work so we launched our platform and we've got you know tens of thousands of users and um everybody's trying out the sales thing because it's cool i get to get rid of my games i get to get store credit to buy new games and what happens is all their games get sold during the sale event and so the problem is then we're in a position where they're coming to us

saying hey i want my because they get 25 cut of whatever it should have been it should have been 30 euros instead it was five euros so you do the you do the math 25 percent of 30 versus five big difference so our initial reaction was well sorry we messed up it's a feature so you're uh you're gonna get only that because we still you know we're paying the publisher a certain amount of money and they bought it for that price so everything together that's how it works and of course the player the consumer is king and queen so we had to eat it and we had to get him the exchange so that was one of those disasters that initially was starting to i was seeing and ramp up i'm like alright this is going to be a pr disaster let's nip it in the bud but that's the thing is so what is the lesson learned um when you're doing something new and innovative over test it you know and there you go okay who wants to go next oh yeah yeah i can look right so being a producer one of the main battles always for me is prioritization um and working

in live ops uh there's always a constant battle of what takes higher priority uh like stuff that needs to be tackled two months down the line stuff that's gonna become like a really massive problem for us two months down the line versus what's an actual issue right now and when you're working for smaller teams uh this battle becomes even worse because you have only limited number of people to tackle these problems this was an incident that happened when i was working for my gaming startup in india and we were working on casino games that would target the subcontinent market and cricket is really massive over there and we had this really big cricket tournament coming up in couple of months and these kind of sporting events when they're live your kpis hurt really bad because players lose focus they shift to them um and we wanted a feature to tackle that uh we kept de-prioritizing it for a really long time because there was always like a lower fruit that we could

tackle and always kept getting pushed down the line uh suddenly we find out it's only a week away everyone's alarm bells are ringing really loud we need to tackle this problem right now um the event is supposed to start on monday uh we're sitting on wednesday right now two days of actual weekdays left to tackle this uh we come up with a spec team things of like taking this challenge up and we design a very basic feature that would let players log into the game and bet some currency on the ongoing sporting event it was basically a simple feature that would remind them of our existence uh so that they don't forget about us once the tournament's over we start working on it thursday morning come to the office sunday evening is when we finally leave the office after doing the release

everyone's just messed up after like endless cycles of development uh me as a producer i'm doing localizations on this feature so we released the game uh it's gone live the match starts on monday evening and we are like keenly looking at the metrics of the feature and surprise surprise there's barely anyone interacting with the future and that's when it hit us like the key part of this feature was uh push notifications that would actually remind players that they need to come back to this feature and we never hooked up push notifications for them and again alarm builds team gets triggered we all come back to the office work another couple of days integrate push notifications feature finally goes live again uh and this time thankfully no mess-ups were made uh but we like built a decent term feature that served its purpose

and that's when my studio took notice of this thing called planning and having a decent road map for the year which has these key events marked uh so that you don't forget about them and you start planning way way in advance so yeah fun times okay yeah right and yours i'll do it quick save a little time when i was first a producer so i started as a game designer and my first job as a producer like mario we're promoting you you're going to move and we want you to work on star wars you know so you're going to move to texas work on star wars galaxies that's our other mmo besides everquest that needs a lot of attention and you're going to be in charge of a new part kind of a new project we're doing with lucasarts it's called the new game enhancements and i was like oh okay that sounds cool like that sounds like fun what's that gonna be like so to keep this quick because this is usually like i need a beer and like you know a couple hours to discuss this but

i was young i was in my 20s i was the youngest producer there i was quite scared i was in a lot of meetings with lucas executives and sony executives talking about the future of the game and essentially what they had decided was at the time star wars galaxies which was the first mmo that had been done in star wars universe only had about 300 000 paying users right so for lucas that was a failure that was not acceptable it wasn't uh i think you know the marketing person i spoke to about it said mario i couldn't you know defecate in a box put it on the shelf write star wars on it and it will sell two to three million units so you only have 300 000 that is uh that's a problem for us the lead designer of star wars galaxies on his own uh unsanctioned just kind of for fun was working on a combat prototype which was a twitch combat prototype uh so if any of you know the game it was turn base

uh sandbox game uh so it didn't have any of the features of games like world of warcraft whatever our lead designer was ref koster you know so he was the one who was really directing this kind of open world sandbox feeling of the game and what happened just bad coincidence was our other lead designer who took over after ref got promoted dan was doing this kind of little experiment in his office and the producer from lucas just happened to walk by he got a demo they saw the game you know using twitch combat and so the the commandment then came down from the vice president of lucasarts which is you have nine months we've already bought tv commercial time we need the game completely converted to this style of combat and uh we want the whole game so this is not originally it was a new tutorial um and what they told us was you know we asked you guys to develop star wars we wanted han solo and luke skywalker you gave us uncle owen and aunt peru and that's not what we want for the future of the franchise so that started a nine-month death march

essentially so seven days a week in the office dinners every night there of course all paid by sony and stuff um and uh in the end we released something that um i had to give my first ever interview to the new york times about how my first trip ever to new york i was like 26 to explain why i had destroyed this beloved mmo that people had put hundreds of hours into and what was our thinking and of course back then you know this is 12 13 years ago i couldn't say luke is what you know i can't blame we had definitely there were people internal to sony that um you know agreed with him right it wasn't just a one person saying hey you're gonna do it this way but for them they knew i think at that point they were gonna make a new star wars mmo they'd already negotiated and signed the contracts and so the idea was make this one work or kill it right and um so that's where we learned you know basically they ran it like a retail product so they bought tv time and everything that sets a hard date that means there's no room to test there's no

room for qa there's no room for anything you just have to put it out there and hope it works and that was my first disaster experience with an mmorpg wow okay anyone got one that can beat that [Laughter] well let's go have we got more oh okay i've got your microphone so you can't sorry so my first job at ea i'm never worked in publishing only worked as a developer so i knew nothing and they're like you know you're there with all the publishing people they're quite

ruthless right they're like hey if this game doesn't sell this much we're killing that studio or you know they're they're running like an american business basically which is one thing i sort of respected about ea was they're very honest they would tell people here's 50 million go make fifa or whatever you have to sell this much it has to come out on this date and it's all your responsibility whereas when i worked at more like sony or ubisoft it was more like they managed the budgets you know they would they would kind of hey here's you know a million a month or whatever to make your game and we'll tell you when it's done so it felt it felt like you were more in control at a place like ea so at ea i was there in 2008 and 2010 was the world cup so they had done a deal with fifa to release on the page of fifa.com the home page of fifa.com which gets 1 billion hits a day during the world cup a free-to-play version of fifa that they were co-developing with a company in south korea called neowiz but it had to be ready to go day and date with the world cup which is in 2010 i think that was a south korean world cup

and so i was in my first meeting i was again i was in my 20s you know i'm in this meeting with like you know the head of fifa the vice president of this and that ea has so many vps and i was a director you know and they were like so mario you're really cool you helped us with development we appreciate that you're not like just a mean publishing guy so we're going to tell you something we're probably not going to make it in time for the world cup because we're a bit behind on the schedule but you know keep that between us right like kind of think and in my mind you know i'm thinking okay do i my boss is the head of all online in europe and north america and i'm like do i tell my boss do i rat them out do i keep do i like keep it inside what do i so then i of course i was scared so i told my boss i really trusted him he told and ben went to the co it went to the ceo of ea at the time was john ricatillo and then you know basically the fifa team even to this day so the head of the fifa team at that time was a guy named andrew wilson who is now the ceo of ea

[Laughter] so i think maybe even to this day i'm sure he'd laugh about it now but back then you know he was like yeah you're never coming to another meeting again we're never talking to you about fifa and they had to reduce the scope of the game to make it in time for the world cup so they only release they're supposed to do multiple countries and i think they only did the us or a couple of countries to make it they did get it out on the date i believe so i was i was left out of the loop a little bit after that well let's see can anyone beat that one as well i can't beat either one of those but i got a funny still brings back memories we were at warner brothers just knew back in the games and we were doing matrix now we had a client we had a partner for matrix the console in the pc game but we decided to do matrix mmo internally with the studio that we bought and that was i mean the studio had never done a multiplayer mmo game and certainly hadn't done it with a movie date because at the time the movie dates were sacred if you missed the movie date you weren't launching now fortunately they're splitting them up

so i mean i can't tell you we're like a month out from movie launch and i'm spending a lot more time on the console on the pc we're working with atari was a partner at the time and so we come in and we start looking at the demo for the game and i go this is single player i'm like yeah but just imagine now all we got to do we got a great map and it was literally like three textures and two colors i go guys you could be working 24 hours a day go back in time this thing's not shipping so all the executives who knew nothing about games outside of the games group right were all coming in goes when is it coming is it a week before are we doing like a review because we got this whole like you know we're doing this whole like review of the film two weeks before this premiere i'd like to play the game at the premiere and so it was my job to tell them the really really bad news which there is nothing coming out except the console game this mmo world with a team that's never done it ain't happening and the only thing that bailed us out we have to get sony to come in because they're experts and we had to give them dc online so the whole story ended up that we've eventually launched it but the only reason why they even took it

over because we realized that we just even after the film we didn't have the wherewithal to do it sony obviously knew this stuff and they said well there is something we like in your portfolio so we'll take the dc online and that's how that story evolved so bad for my career but funny story [Laughter] and uh arjun what about you any other yeah but not as such massive scale stories like uh this is more of like day-to-day game development so this was back when i was working on formal to web when it used to be pretty massive um and we have this concept in live operations called the on-call team and everyone generally hates being on the on-call team on-call team is this uh rotational team uh which has a poc from every department and that team has to be available 24 7 because it's a live title and issues can come up at any given point so

we were like this is around midnight um and we started getting a couple of reports from our cs agents that players are not able to load their game in a particular geo and generally like game not loading is like the reddest alert that you can ever get players are blocked so we were all young super ego and we came online we were like today we will solve this issue uh without looking properly into it we were like oh okay so this particular language is not loading for players our qa came online did a quick test and obviously it wasn't working so we fixed a hacky fix and within like 15-20 minutes of us getting the issue we fixed it released it back in the day of web it was super easy to send out a release and we were done within like half an hour of first report we were already done fixing the issue and we

gave each other virtual high fives and we were like yeah good job see you guys tomorrow in the office we were about to log off and suddenly our inboxes exploded with reports of no one's able to load their game anymore regardless of whatever language they're playing the game in uh what had happened was we managed to fix that one language but with the hacky fix that we sent in we ruined everything else and now with so many reports coming in more people started coming online we reverted the fix that we did pushed out a proper fix which thankfully fixed all languages you could play the game regardless of whatever jio you were in um problem solved next morning everyone's back in the office and that's when we asked the question that we should have asked last night what was the actual impact of the initial issue and it turns out there were a grand total and i remember this number till date there were a grand total of 11

players using that language and the reports the reports that we were getting uh was from this one player who was not able to load his game and he had multiple accounts so he started spamming and reporting issue from his multiple accounts and yeah okay that's that's a good one i think yeah okay sarah are you actually it reminds me of a story um in a similar sort of vein but um we um many years ago so when i started co-masters online which sadly is no longer with us we were working on a game um and we got some reports in from sort of like custom support players i can't remember which basically talking about you know there's an exploit in the game which always makes you exploit in the game and it turned out players were intercepting traffic between the client and the server and pushing money into the game and effectively breaking the economy so much as yours like we were in the hours

you know the messages came in i remember messages coming in about 11 o'clock at night till three in the morning we were there banning players stopping accounts until we could sort of write rain control back over the economy but we did and we survived but yeah that was a long night that was a long night has anyone got any questions oh there's some uh one back there um thank you for this chat this is really awesome if you don't laugh you'll cry right um i just had a little fun question um what do you guys what's your opinion on the current um kind of epic failure of the battlefield 2042 launch that's happening since november launched with a bunch of bugs and stuff

and it seems like they're losing their audience so i just wondered if you guys had any opinion on that you looked a bit uh like you might have an opinion i just think it's sad because you know call of duty had an off year then as well you know where they kind of recycled basically older stuff it helped halo which i guess was good um because halo hadn't come out in quite a while i think honestly it's a shame you can see it's it's an in-between game between you know the next battlefield right the next big ea would do this right we would have mainline battlefield and then you would have kind of like a smaller project that was a subset multiplayer project but normally those smaller projects were like bug free so i guess they felt the pressure to compete with activision and maybe take advantage and that was where they fell down so for me i think it's really sad because it can really affect that team that team has tight targets has a certain amount of money they're expected to make so for me i'm just sad and uh hoping that uh they address

it in live and fix it and they've already i guess basically admitted to it and and come out and said they're gonna deal with it so hopefully that turns it around for them and and the game keeps running anyone want to add anything so i think that because i've been in that position many times it's the it's the divergent agendas within the organization that result in that 90 of the time the only organization that it says back in the day not today that kind of got around that because i remember they had a saying was blizzard before they became activision blizzard they were like it ships when it's ready but very few people do that and ea just like activision was just like is at the end of the day especially when you're a sequel when you're a first timer yeah let's get it right let's bake it there's always going to be somebody in a position of decision making who's going to say guys if it's at least 70 bug free ship it i need the numbers i need to please wall street so it's business and so unfortunately it shouldn't happen with a with a title that size but it does so you've got to get how do you prevent that i don't really know except

that you get somebody powerful enough within the organization who's got a creative say to say guys this is our baby don't touch this do your numbers on the other games and blizzard for a long time got away with it anyone else i can add something so i think one of the things is uh after cyberpunk up and i've seen this in mobile development now uh developers feel more comfortable pushing back uh to execs now saying that the feature or the game's not ready to be shipped uh and i've seen it in some of the organizations that i've worked now where we as developers have also understood that it's okay to talk to players and tell them that we'll take couple more months but like ship a better product to you and that kind of a conversation has started happening more freely after like the last two years and what's happened with multiple titles so i think that's a good take away out of the current

situation at least for mobile i just add something as well i think it's also it's it's a bit of it is sad sad for a number of reasons because this is the advantage we have with live ops is that the game doesn't have to be ready on day one so why not say i'll reduce the scope and i'll release content in you know in a cadence that goes after that so you can actually build yourself into a position where you think it has to be all singing all dancing on day one but as long as you've got good core gameplay and you've got some really key features to the game you can make really big announcements and really you know get extra engagement with players if you plan when to release some of those things make sure there's not too many bugs instead i i actually got a great example where it worked in the other direction so i warned we had the arkham batman and that after arkham one huge i mean it did well but the pressure was enormous and then somebody said hey we've got this lord of the rings platform or ip i want to do shadows of mordor and it got green lit but it was so under the radar it came out super game super polished

nobody was saying that's going to make my quarter and that those are the best kind of games right and now of course the next one more pressure any other questions it's on okay great i'm actually curious because the question about battlefield 2042 is interesting because right now the steam numbers are showing that battlefield 5 the previous title is showing the combined stats of every other battlefield title um so i'm a little curious because it resonates a lot with the live ops story too do you ever think that the uh the frequent cadence of games being released right now could be maybe you know revisited with uh older games essentially being reinvigorated by having a live ops team come right back to the the the older titles and just breathe new life into them what do you guys think of that so i totally agree and you can look at some of these like games that come out like among us is a good example which been around for a long time and existed

with just fairly you know not minimal features that sounds wrong but you know a good solid core gameplay and now adding in maps and cosmetics and things has sort of invigorated the game so yeah i definitely think there's there's value there players love the games and there is a value of nostalgia as well so i think yeah totally yeah i would say i think it's not about whether it would work i think it would work i think the question is it's like the iphone for instance can we just upgrade and keep iphone 11s and keep them going or do i want to sell iphone 12s right and if you're a retail publisher and you're spending conservatively 50 to 100 million on a game which is what they do then the first 90 days at 60 usd um is important and it's and it's it's you know still where 50 to 60 of their revenue comes from right so i think specifically with the case of battlefield you did see a big bump not just from steam game pass because ea

tied everything into game pass right so a lot of their older titles are up there but is that going to make more money than putting 100 million in battlefield 6 and selling that for 70 on ps5 and xbox series x probably not right and then again having a year of fixing bugs exactly yeah i think what's interesting is if you look more on the indie side because they don't have the money to spend 20 million in the next game they've got these old games that are still alive and they've got the community the problem is they don't have the same constraints and the same objectives and you know what wall street has so to your point everybody you know the publishers they want to sell the new iphone right but then there's another thing you've got to worry about is when is it the right time even if it's not about i got to make my numbers i got to please wall street when is that shifting just right when have you reached a peak sort of when that okay this game was great i still got my 400 000 users but is it time to upgrade it and i could do something better so i think it's all about we're in a new age now where most games are going to have longevity so get

to know your community community community see where it goes manage it it's pr to the nth degree and then then you figure out wall street i think there's also another thing as which comes down to have you got the infrastructure in place to do it it's not easy to shoehorn live ops infrastructure into a game already exists so there's a balance of you know is it worth the investment or not as well any other questions oh okay thanks um my question is like we're talking about games of releasing on a yearly basis i on the flip side of that when do you think a live ops game should come to an end say for example world warcraft is what 15 years old now and it's still using the same engine and all the technology from back then when do you think it's time to sunset and then release a new game and what are your thoughts on that i've worked on games they've been going for 15 odd years so i could probably argue maybe never no player numbers and definitely engagement and looking what's happening and you know revenue you need

to be able to keep running keep the money coming out in to keep the doors open effectively um yeah i think so i mean my funny stories i worked on a lot of the rings online how long does it take to get to mount doom sorry apparently forever so we never got there certainly in the time we had it but yeah i think that that's probably my answer i mean i guess the honest answer is when it doesn't make enough money to be supported right which is the truth blizzard since they did wow classic i mean you're talking about probably a billion plus a year right so if you have even if the game's 50 years old if it makes a billion dollars a year i'm sure that you keep it running right keep it running until it runs out of cash i work you know on on runescape at jagex it's a 20 year old game you guys probably the numbers are public a company house who does 120 million a year right so will they shut that game down definitely not they'll probably consider making new

ones but you have to be careful about where the market is today cannibalization all this kind of stuff right you also run the risk of losing your audience as well so yeah that's a huge risk it's like as much fear about the money is i don't want them going to a to a competitor so if i'm going to do a launch a runescape 2.0 am i going to keep at least most of them yeah so this is from context of farmville one like the og farmville uh we had to sunset it eventually but it was a very difficult call because even though like the player numbers kept reducing the core set of players and some of them had been there from the day one launch uh it's very difficult to disappoint those players because that's basically the audience that's gonna stay like for whatever farming franchise that you release uh these players were playing farmer one they were playing family too and we were in a similar conversation where it had eventually become costly to keep

the game running but it was also a call of do we want to disappoint the set of players because if we disappoint them this they're gonna stop playing any other zynga game from this point onwards uh so what we ended up doing was there is an aspect of automation that you can attach uh so that some amount of events uh regular cadence can keep running on its own without with like minimal uh maintenance but we had to eventually sunset it as well but yeah okay we're gonna have to wrap up there so thank you very much all four of you i i learned quite a lot i think um the best thing i take away is that planning and priorities and decision making are key which is great because i basically go on oscar about that every day and i feel validated now so [Laughter] well thank you so much for all the panelists and thank you to everyone for listening i'm sure if you've got any other questions you can try and grab one of us at the end um and uh yeah thank you everybody thank you

thank you [Music] [Music] you