How FIFA Delivers Live Content to Users Fast

How's everybody's GDC going so far? I'm having awesome time. First time, so ridiculous. Fun amount of buzz, too. Um so, I am here from Electronic Arts Burnaby campus, uh where we develop FIFA and basically talk about how we deliver content to users really fast. Try to see if we can get rid of that buzz, but um So, let's go, yeah. So, live content is better. Um it doesn't matter if you're going to monetize your game or not. Doesn't matter if you've got plans right now to monetize or plans down the future to monetize or no plans whatsoever. Creating live content basically allows you to create an engagement to the users. You you create compelling content that actually draws the user in and they like want to play your game. What is What do I mean by live content?

I'm not talking about game patches. Game patches are things that will go out and make code changes and whatnot. I'm possibly DLCs, but to me that's not really live. That's basically in chunks. You're still having to go through the certification flow on uh console whatnot. What really I'm talking about is in-game downloadable content that's released on a regular continual basis. Basically, it creates a compelling reason for the user to constantly come back. Examples in the industry, so when I was looking, I was mainly looking for console examples and there really aren't a lot of console examples for live content. There's tons of stuff for PC. There's a ton of stuff for mobile. But for console, Call of Duty, again DLC, basically map packs, weapons. Uh Forza, you can do car packs and also add-ons. Um Harmonix is probably one of the best ones for actually doing live content cuz they're constantly releasing new uh

audio packs and from different band members and basically song packs and whatnot. So, all of this creates a reason for again the user to kind of come back to the game. So, how do we do it? Or how do we as an industry want to do it? Um basically, first thing is your game has to support it. Your game has to have a framework could be a framework to actually be able to bring that live content and display it to the user. The live content has to be special. It can't be just mundane little add-ons here and there. It has to be something that the user actually wants to play. Something the user wants to engage with. Content delivery is not equal to the game or server updates. Basically, do not couple your content delivery system to any sort of game updates or server updates. If you do, you will slow your pace down and you will not create a reason for the user to come back on a regular basis, cuz they'll be waiting weeks, months for

that content to appear. Use tools, not engineers. Leave the engineers to solve the complex problems that we all have to face. Use tools. They can be set up to validate. They can be used to like, you know, clean the data and make sure it's correct. So, FIFA live content. I'm going to quickly just show a video now just so you guys have an idea. Some people probably know what we do on FIFA, others might not. So, just going to Ooh. Let's try that again. Football. Football? Possible he's open. Opinions. Ridicule. More opinions. Blah, blah, blah. Gentlemen, PUT YOUR GAME WHERE YOUR MOUTH IS. BUILD YOUR ultimate team. Put him with him, him, him, and him. GO SHOPPING.

DRIBBLE. PASS. SHOOT. SCORE. Rule it. Lose it. Win big or win something. Make dreams come true. Okay, not that dream. This dream. Sometimes be like Other times be all Show the world you have the ultimate team. Build your ultimate team today. Put your GAME WHERE YOUR MOUTH IS. SO, THAT'S BASICALLY WHAT I'M talking about when I'm talking about FIFA Ultimate Team. It's essentially a items-based game where you basically purchase packs, build a team, and uh compete against other people to get coins, and that that loop. Um you're essentially wanting people to be engaged in that loop, and to basically constantly be buying it. Wow.

It's extended. I'm just going to try something here. Just give me a second. Ah, there we go. Um so, what kind of content do we actually update? So, in a year, we're basically putting out 1,200 in-form players. Those are basically players that have uh increased their stats, have been in form for that week or that month or year. 2000 plus player transfers, 100 plus tournaments, 40

seasons. And seasons basically are made up of different divisions and you move up in the divisions as you compete and get points and wins. 100 plus pack offer promotions. And those are all tied in with like the challenges, 150 different challenges, 300 plus in games of the week. 52 different match day update drops. So basically transfers and other database drops on match day. And then that is all kind of messaged to the users. 1000, approximately 1000 messaging banners and 21,000 lines of localized text. So we're producing a lot of content in a year. And we're pushing that out to users on a continued basis. Pretty quick as well. So player updates, we actually have team of the week which is players who have played really well that week. Typically team of the week the producers will choose which players from the week. They'll also give the community a chance to actually pick players as well. And typically one or two of the community picks will actually go into that team of

the week for the for that week. So the players will basically have different versions of their items. Much like baseball cards will have different versions for like really good plays and what not. Team of the year in season. Again, at the end of the season at the end of the year we also put out a whole batch of stuff. So team of the season, we release 18 to 36 new version of per week. And then for the team of the year we're releasing 11 players over the course of a week. So every week, every day we're releasing one or two players. Just creates really a lot a lot of excitement in the game. Um high basically high profile players and what not. Man of the match, they did if someone did really well, they'll basically get selected. Special versions are interesting. Basically, it's uh versions of players that and special items. Versions of players that are like career uh highs. Like a career high Messi if he's basically made a special uh

scores or whatnot. We also have special items. Uh special items are interesting because they're actually versions of the player that are not released to the market, but are granted directly to players that actually play our game. So, professional players will show up, and you can match up against them, and you will see that special version in the game and be like, you're actually matching up against that particular professional footballer. Um and then FUT squad updates. An example is Ronaldo here. Started out as a gold version. Basically, had two team of the weeks released as well. Um ooh. We had two team of the weeks released. We have a team of the year as well. So, that was uh mid-year. Um these are from '14. And then, we've got a man of the match. So, he had a spectacular game. He basically just killed it. So, we had an extra man of the match. And at the end, we actually had a World Cup. So, during World Cup, we're releasing special content. The interesting thing about the World Cup one is that if you got that, it actually live updated. So, if he played really well, our stats changed on

the actual item live. So, we were constantly updating that item. And then, basically, you're getting a lot of people the community buying into it. So, this is an example of team of the year Messi. Uh team of the year Ronaldo Ronaldo basically getting released. Someone was saying, "Yes, yes, just another version of Ronaldo. Never be able to afford." Uh we're able to buy. Um team of the year Ronaldo. But don't know what chemistry style to use. So, people are communicating and talking in the forums as to like how to actually use that particular item. The day when Ronaldo scores a hat trick is one of the most horrible days for Sweden, but for Ronaldo, it was fun it was Tuesday. And then also like haircut upgrade. This was for our World Cup. If you notice, he's got a different haircut style. So, yeah, we updated the hairstyle in the item and whatnot, too. So, um basically what that what we're trying to do is create massive community engagement. We basically have the forums, Twitter, we have third-party

sites that actually link into our database and show various items and whatnot as well. There's a lot of forums on that and people will actually engage and like really get that excitement going for the actual week and they'll have a lot of communication. Thousands and thousands of posts on those websites, our own forum will have like tens of thousands of posts and if you ever want to like if an I if a pack promotion is happening and you go into Twitter and go to @EASPORTSFIFA, it's just a constant stream. Like it's just crazy how much uh communi- communication there is. Speaking of pack promotions, we don't discount, ever. So, what we do is instead of discounting, we actually create special versions of packs that have a higher probability of getting rare players, that have um more value. Um so, we don't ever we don't want to get into that whole we're discounting and then people only purchase at discount times. We're really trying to just drive that economy and

making sure that people purchase and see value in what they're purchasing and it's not the Walmart sort of I'm only going to buy it on blue light special sort of days. You need to tell users what's new. Messaging is key. If the users don't know what is actually being released and what's new and exciting, they're not going to engage cuz they have no idea. And we, because we are a football game, we tie back to real world events. For example, we've got transfers, Chinese New Year, um Team of the Year when basically at the end of the year we're releasing stuff, and the best of MLS. So, at the end of the MLS season we'll be releasing release items as well. So, basically trying to tie it in with real world stuff, things that are like you know, particularly important to users or to gamers. One of the additional modes we have is challenges. These are actually really cool. Essentially, they are replay the game

um shortly after the game has been played. So, we basically will take a game that's played, we'll set up a situation where you may be 20 minutes into the first, two points down, and your job is to score three and change history. And it would be some major match that's happened that week. This basically involves artwork, localization, tons of change like tons of data coming in from our data teams to basically set up that match. We can actually set this This particular one was set up in 45 minutes. So, 45 minutes after the actual game ended, we had it up, people were able to replay it. We had Chelsea actually got a deal on with the games the with the clubs, whoever won, I think it was Chelsea that actually won. Actually was. Whoever won basically tweeted out to their supporters to play the match. Um and then we were tweeting out within like minutes of the game ending that it was going to be up. So, again, drives a lot of people into the game.

So, 45 minutes, EA promoted at half time, and winning team promoted the challenge as well. And we actually because of that, because of the promotions and the quick turnaround time, we actually saw an 8% increase in users for challenges. It was the first time we'd done a challenge in that quick of a time and gotten that much buy-in from both the club and also engaging at half time. So, it was kind of interesting. Messaging. So, basically, again, our key thing messaging, telling the user what's new, what's important, why they should actually be engaged. Um top left-hand panel there, or I guess top right for you guys, uh is the team of the week. That's basically where we message. It actually has So, that's the area. It's prominently displayed right in the default for the team. We actually have message panels throughout the game, including right in the main menu. So, as soon as you launch FIFA, there's messaging. And then, we also have deep linking and

actioning. So, if it gives the player a chance to engage right away. Again, driving people into that content. And we're actually um cuz we're a global team, we have to support a global game played in pretty much every country in the world, we have to support a fair number of languages. So, this is an example of one of our uh one of our messages, score five with Messi. This is for a challenge. Um we're basically supporting 18 to 23 languages across 11 SKUs. So, 18 languages on console and PC, 23 languages on mobile, and that accounts for 11 different platforms that we're actually uh messaging on, and we can get that accomplished within 1 hour. That's fully localized, all the art. And if you look, for example, the Arabic, it's completely different. You have to line up stuff special. Um some other languages, Cyrillic, for example, the

line up really is not always fun. Basically, we're able to get that in less than an hour and it cross to everyone. We also have the ability to target users um and basically say, you know, we can target users themselves. Um and then, what we also found is it actually is a huge deal to actually um create messages that are regionally specific or regionally relevant. We're finding that, for example, we had three times more engagement on this particular message compared to just a generic message or no message. And we're It's weird. Having no message and having a generic message didn't have a huge difference. People are still able to get They're They're regularly coming back and looking at the the challenges and seeing what's up. But, if you actually did a regionally specific message, this one This is a Spanish Spanish Mexican. And we actually had a special match that was played in Mexico. You're actually looking at huge like

three times the amount of engagement because now again, it's relevant to the users that are there. And they want to engage in that. So, this is basically a quick overview of like how our stuff works. And the reason why content delivery needs to be dis- disconnected from or decoupled from game and server updates. So, normal developer goes into the CM, as we all know. That'll get pushed out to a test server. Essentially, you get Let me see if I can get this. Here we go. So, this is basically our test cycle. Um, you go QA testing on the machine. We have legacy testing cuz we're actually supporting multiple versions, multiple years, and across multiple platforms. So, that'll get cycled through. Once it's done, it'll get pushed to the stage server. Goes cycle through there, goes to the production server, cycles through there, and then goes live essentially. That can take anywhere from a few days to a few weeks depending on how much code changes there are and how much testing is

required. Obviously, if you're trying to tie to real-world events, a week of waiting for the server changes to go through is you can't You can't You can't react to anything. So, we actually have content production using tools that go directly on stage environment. A lot of validation, a lot of code play, making sure everything's correct. That gets tested by internal QA team. Um that can be done within like minutes, and then it's basically pushed out to the production team, retested live. And it just it we can basically cycle and change stuff on a continual basis with some changes going live in seconds, other changes taking hours or minutes. So. Get let allows us the flexibility to really react to whatever happens. Um there was a transfer last year, and I'm sorry I can't remember the name of the player, but it was a sudden transfer. He announced it, and like nobody knew he was going to actually transfer. We were able to make the change in the game,

push out a new version of the player within minutes of the actual announcement that was the press release. So. So, decouple this from server deploys, fast content iteration in minutes, not hours or days. That's hugely important. The main thing that lets us do that is tools. Now, we've had a little bit of a weird history with tools. Um we first started, we basically did client applications. It's a C# app that ran on a client server or class or yeah, a producer's machine or QA machine, local machine. Uh direct database access, a lot of APIs weren't set up, didn't have that ability, so the tools are directly interacting with the database. Some content was managed through SQL scripts and data files. And then we were coupled to the schema. If schema changes happen, you're basically reworking the tool. Um it's difficult to manage permissions mainly because we didn't have a centralized permissions management system for this. It could be built out,

but it just adds complexity. And again, you can bootstrap your application and have it check for updates. We didn't do that, unfortunately, so it was difficult to manage versions. And you know, you you would basically have the QA teams, the production teams, everybody using the tools, and then you'd push out an update to bit for a bug fix. And, you have to track down everybody to figure out who's got the old version, who doesn't. And, it required a local setup. You need to set up Perforce and everything on the actual uh the user's machine. And, some content was still coupled to the place because we're using SQL scripts. So, this was our dark days. Um we weren't pushing out a lot of content. We just didn't have the ability to push out live content very quickly. This is probably similar to what a lot of companies are at right now, even, um and are kind of working through that. What we do now is totally different. We actually have a single sign-on portal. Basically, we only support Chrome cuz it just makes life easier than having to

support multiple browsers. But, single sign-on portal, producers can sign in. We have it's basically Angular front end, C# back end. Really nicely uh nice to develop in, and we can iterate really quickly. Um we're using REST APIs only on the back end servers. We do not couple to our database ever. All content uses a tool now. There is not a single piece of content that that goes live that uses scripts. It all goes through the tool. It's all validated through the tool, and producers and QA actually uses it during development as well. So, now you're decoupled from the database schema. We have a centralized permission service, so we can control exactly who has access to what tool and what environment. Um and it can be read access or read-write access, so. And, it's always up to date. It's a website. You go to the website. If we push a new update out, just a website. So, only need a browser. No more setup. The you know, you don't have to try to convince a producer or a manager to try

to set up the .NET 4.5 framework on their machine in order to run it and also possibly Perforce and other things which they kind of sometimes run away from. Or you end up trying to support by you end up supporting them by actually setting up for them. So again, coming back to this, so tons of updates, everything through tools. By using the tools, we're actually able to do that all with three basically three content producers and one artist. So a team of four basically puts out all of that content across 11 SKUs in a year. And we're actually increasing the number of the amount of content we're producing on a regular basis. These numbers from last year. We've actually increased that this year. We're actually producing even more and more updates. And that's all optimization on the tools. Creating better flows. So, here's some data. We're publicly traded company, so I

can't share tons of data, but I will share some. Um Plants vs. Zombies DAU. So, Plants vs. Zombies is an example. It's an EA game that is actually developed at Burnaby as well. They don't have a content plan. They basically just release updates. Um you know, they have in in uh game packs uh with stickers and whatnot, but really they don't release new content on a regular basis. So, what you're basically looking at here is the regular spike, drop, spike, little bit drops. There's another content update, big content update, drop, another content update, then a giant content update, drop. This is for the full year. Now, compare that to FIFA. So, the very bottom line is actually PVZ the DAU. FIFA 13 was the yellow line. As you can see, we kind of have the same sort of pattern as PV as PVZ for FUT 13. Um it's basically because we weren't producing a lot of content. A lot of the spikes are on actual patches and DLCs or patches.

If you look though at 14, beginning starts out low and then we actually start wrapping up our content plan, our content schedule, and it's kind of it is peaky, but it peaks and kind of stay steady, peak, stay steady. We're actually now in an interesting place. Every year we're increasing our numbers. Um this is weird. I think we just had a lot of issues with our servers, but we've gotten stabilized now and it's basically steady. Um what we're finding is throughout the entire year we're not having that slow decline to nothing. Um we get a decline at the end just over Uh I've lost my mouse. We get a decline at the end here on pretty much every year, but that's an anticipation for next year's game. So that's usually at like the middle of summer and it just kind of dies starts dying down cuz everyone wants to buy the new one and not spend money in the old one. Um one other thing is in our game your stuff doesn't transfer over to the next year. You're going to

basically new stuff every year. And our users love it. Oop. Um so this is basically example of our stickiness. Now we've got an increase in DAU, which basically means if you don't have good users coming back and you're basically just having people who aren't really playing the game, our stickiness would have dropped. The nice thing is our stickiness is actually almost exactly the same as last year. Even with the giant increase in number of users, we're actually getting good decent players in the game, engaging, and staying in the game. So what's the benefit to the gamers? Compelling always updated con- experience. They're constantly wanting to come back. Added variety. Basically keeps the game fresh. It gives the user a reason to increase engagement. What's the benefits to for the developer and the publisher?

You basically have increased gamer engagement. What does that mean? Increased acquisition, increased satisfaction in the game. People are happy to play it cuz they're constantly getting new things and constantly getting excited. Means that they're willing to stay in the game longer. It also means you've got an increased monetization potential. If you keep the players in the game and they're happy, they're more willing to actually spend money if you choose to monetize. So, takeaways. Compelling content to gamers fast. You need compelling content. You want to get it to gamers as quickly as possible. Means you can also iterate on content if you need to. You want to separate the content from patches and deployments because if you tie it to your patches and deployments, you cannot react. You will slow yourself down. Use tools, not engineers. And this is massively important. Engineers should be solving complex problems, not deploying data. Data should be deployed through tools,

and producers can create their own tools, and they can play, and they can actually have fun and create stuff that gamers will be actually be compelling will will find compelling, and they can iterate on it a lot faster. And make the content special. Give the gamer a reason to actually come back. So. That's us now. Happy. So, again, my name's Novi Dillon. This is my contact information. Um please fill out the evaluation forms. It'll basically help me and everyone else know how uh I did. Hopefully I did well. Um yeah. And if you guys have any questions, um just come up to the mic and I can try to answer them as best as I can.

I've got a question. Yeah, have you ever had to roll back one of your updates that you pushed out? So the question was if have we ever had to roll back one of the updates we pushed out? Yes, we have made mistakes. Um we last year, we actually released a piece of content that basically killed FIFA. You couldn't even log in. It just like done. Um so we had to really quickly make that change and roll back that change. The nice thing is because we have the tools, we can quickly disable it and get everybody back in the game and then figure out why on a different environment. Have Have you played with any kind of multivariate testing? Multivariate testing? Yeah, so like AB testing? Yes. So our messaging system is actually set up to do that and we're actually expanding that hopefully this year. Um so we do do that. Um unfortunately, uh the rest of the content isn't yet set up to actually do multivariate testing. It is something

we're planning on doing. Um but yeah. It's a It's a technical problem on the back end. Everybody wants to do it. It's just a matter of finding time to get it done. Uh have you ever had your your community base uh decompile your archives and actually get information that's that's patched ahead of your actual release? And so what's the impact that you guys Yeah, so luckily we haven't. So the question was have we ever had our community basically decompile our our archives and our patches? Um luckily we haven't. We've actually been really lucky on that. Most of our games are not like we have the PC title. Um but it's got fairly good protection around it so we don't actually we actually still hasn't been hacked and it was released in September, which is kind of amazing. Um, and then the rest of them are very closed platforms. So, getting being able to decompile your Xbox game is going to be nearly impossible. Um, so we luckily haven't had that problem.

Hi. Uh, with the content delivery that you're doing so quickly, how do you manage that with the leagues? Do you have any restrictions? Do they have any rules? What's their process for you? Yeah, so we actually have, um, So, let me just Was it just managing content basically making sure that we don't have content coming out earlier than it should? Or just, you know, marks, players, anything. Just the sort of approval process and how much lead time do you need to give them? Like if, for instance, they say the Premier League and Yeah. another team doesn't want you to do North London Derby because they're like, "Well, what about my team?" Um, So, that kind of thing. So, um, we have a very, very good licensing agreement where we have a lot of leeway and we kind of can make those choices ourselves. We can basically choose to say we're going to put out North London Derby and there's really no choice. Um, we've got really good lawyers. So, um, but ultimately we, you know, we want to support all the leagues and all the teams and kind of give them

good, you know, visibility in our game cuz it's just good for everyone. Um, yeah. It's a very impressive system, I got to say. But, one thing regarding is that three content producers, one um, artist. What about the localization? You said 23 languages in under an hour. How do you do that? You have the localized team on site as well or So, we have a localized team that's distributed across the world and it's actually shared by all our EA teams, not just us. Um, so we have a system where we can drop it in and it gets sent out to everyone and basically gets localized pretty quick, like within within half an hour to an hour on regular times. Now, we have generic text as well if we need to get out stuff really quickly that we already have pre-generated where we can drop in team names and certain things. Um if we have some more leeway like we know that something's going to happen in a week, we can actually go and get some really specific like very detailed information. That takes a little longer. So, it kind

of we trying because we're trying to iterate really quickly, we're keeping those descriptions down a little bit smaller so that they can get iterated really quickly, too. I would love to actually have that automated um with like an actual automated system, but I think one of the other talks where actually the very first talk on Monday at 10:00 and I cannot remember the gentleman's name, but he basically said that uh right now the machine learning and basically machine translation isn't good enough um especially with context. And that's where you get real real problems is the context. Cool. Awesome. Thanks, guys. Um have a great GDC. Thank you very much.