Beyond NPCs: AI for Game Operations

Um so I last spoke here in 2014. Um back then I was a a research student. Um my PhD was in AI for games and many of you may remember me as the the shouty guy uh talking about how we could do things a slightly different way. Um and I disappeared into corporate America at the end of 2014 and uh started I actually moved here which was strange and joined Zynga and I've worked my way through the ranks and become director of central and strategic analytics as Steve mentioned. And that's a big lofty title uh that doesn't really mean much. Um but what it sort of means is that I provide kind of technical oversight over a majority of our kind of AI and machine learning and data science across Zynga uh as well as leading a team of of practitioners that are a centralized resource. So you might have heard of Zynga. Um back in the old days we were um very

well known for FarmVille. That was kind of 2009. That's the thing that kind of kicked everything off for us. Uh and since then we've diversified our portfolio. We're now uh running games like Zynga Poker. That's now celebrated its 10-year anniversary. And then we acquired studios like Natural Motion uh and we now have uh CSR 2, Dawn of Titans, games like that uh in the mid-core space running out of the UK. Words With Friends obviously remains our kind of staple title. Uh everybody's played Words With Friends on the uh uh Everybody has played Words With Friends on the planet, I believe. Um and more recently we've been doing some acquisitions uh picking up studios like Gram Games in uh Istanbul with Merge Dragons uh and the Small Giant crew in Helsinki uh for Empires and Puzzles. So that's the kind of games that we make and where we are, but let me give you some numbers on that. After over 10 years, we've got actually 50 plus games live today across our entire portfolio. And that's across mobile. We're mobile

first these days, but we've still got our our kind of presence on Facebook, on Facebook's instant messenger platform, and and iMessage as well. We have across our own our whole portfolio, we have about 85 million players per month. And that's driving in 2018 $907 million in revenue. So, I've pulled those numbers from our most recent financial report, so I don't think anybody can dispute them. And certainly corporate didn't when they reviewed these slides, so it's probably right. Um So, let's talk a little bit more about what central and strategic actually means. So, like I say, it's a it's a small group of data scientists and data analysts who sit centrally, not embedded with our stakeholders. We have a game analytics crew who is all embedded really supporting in a service kind of role, the the actual day-to-day of our games. So, these are the people who are giving the insights to the product managers, actually kind of delivering a lot of of

insight and knowledge into what's going on inside that game. We don't do that. We sit centrally, and as a central team, we provide support to the people that don't have coverage. And that means like corp dev for example. Like when we go on one of these acquisition sprees, well, there's no point having a team dedicated to that because it's kind of infrequent work, so it falls on my team. And then on the strategic side, we're kind of focused on things that cross division or cross company, things that are going to touch multiple teams across the entire kind of global operations of Zynga. And one of the core things that we're kind of centered on, mostly because of my personal biases and kind of that's how I'm steering the team, is a focus on kind of novel uses of AI and machine learning techniques. So, what do we mean when we say AI? This is going to get defined repeatedly and repeatedly, but in the context of kind of big corporate business, uh in general, we would like our AI systems to

be one or more, ideally more, of three things. Assistive to humans, better than humans, or faster than humans. And if you can tick at least one of those boxes, it's probably a good candidate for for some cool AI Um and the idea is that we can drive operational efficiency. Um we want to move humans up the value chain. There's not really a whole lot of value to humans doing mundane tasks that an algorithm could do. What we want to do is put the humans in the driving seat, kind of driving the algorithm, and actually kind of providing more impact than doing grudge drudge kind of work. So, who here knows anything about mobile advertising? Cool, then you're going to enjoy the next slide. Um cuz I got dropped onto a project around mobile advertising. And it was the first thing that I kind of took on as as part of this new responsibility, and I said, "Cool." Uh at that time, I knew that our games

were supported by ads, and there was some magic that made ads appear in games, and that was all I knew. So, let me give you a crash course in mobile advertising because you will need it later. So, imagine you have a game, and you want to show some advertising in that game. Uh we use what's called mediation. And the reason that we do that is that we're going to try to find the a good price to sell this piece of advertising at. And we use what's called a a waterfall strategy. There are other strategies out there like unified auction. Um but our waterfall strategy means that we contact a mediator, and we say, "Okay, well, can you talk to ad network one, and can you try to sell this piece of advertising for $20. Um, and they will contact their advertisers and kind of all of this is happening in real time. Um, so the price here is in a cost per thousand, so CPM. Uh, so it sounds like a big number for one piece of advertising, but divided by

a thousand. Um, and they say maybe they say yes, in which case we're done and we can move on and it's all good. Maybe they say no, in which case we go, "Okay, cool. Uh, what if we went to ad network two and got $20 there?" And maybe they say yes, and maybe they say no. So then we might go to ad network three and $15, and now we're getting not desperate yet, but we're going to get desperate as we go further down here. Um, ad network one, maybe we drop the price and we say, "Okay, well, we you couldn't do 20, but can you do 13?" So each one of every time we reach out to the ad network, they're running an auction on their side, and the the dollar value that we're giving here is the reserve price. So imagine that they are running an eBay auction with a reserve price at that point. That's kind of giving you a sense of of what's happening here, and why it's this cascading waterfall. Um, so at the end of it, we maybe drop down to $5 at network two. So 18 months ago, we asked the question,

"Okay, cool. How do these configurations get generated?" And the I the answer we got back was that there are people who sit down and figure out what they think the price point should be, and which order the ad network should appear in, and what goes where, and they do that all manually, and we said, "Well, that." Um, so 18 months ago, we said, "Okay, we can optimize this process, so let's actually start a a project around optimizing this." And we created this thing called Zoe, uh, the Zynga ads optimization engine. Um, it turns out that if you work at a big company that has a Z at the front of the name, most of your acronyms end up starting with Z. Um the other one is ville. Something ville, contract ville, manager ville. Just an insight into corporate America there. Um this is a really interesting problem though cuz at the end of the day um we actually partner with 40 different ad networks. Um in fact it might even be closer to 50.

Uh each network can appear multiple times at multiple different price points. This is one of the gnarlyest optimization problems and the most fun optimization problems I've ever had to sit down with. Um cuz you've effectively got this kind of indefinitely long list of knobs that you can kind of put on the thing. You've got a box full of different knobs and different settings and like you can just play around with it. It's fantastic. Um In an ideal world you would start out at the highest price point that you think you can get to and you would go okay, down the list. Dollar, dollar, dollar, dollar, dollar, 99 cents, 99 and you would just brute force it, right? That's kind of the optimal strategy to maximize revenue. The problem with that, and I kind of mentioned it earlier, is that people are waiting for that ad. This is all happening in real time. So the more times that we actually go out and the more we get it wrong, the worse the user experience is because they're sat and they're waiting uh and you've just got a blank screen there and that's not good. And it turns out that if people are

given a blank screen when you're supposed to give them an ad, they will eventually just piss off and never open your game again. Uh and that's not what we're going for for our user experience. Um so let me introduce you to how Zoe works from a from a block kind of point of view. So imagine that you have uh data to ingest because you're tracking everything and you have a whole bunch of sales information about what's actually being uh done. That reporting is generally on a 24-hour aggregate basis. So we get uh performance for the last 24 hours. We ingest it. Uh it's data so we have to clean it because it's trashy in places. Uh, and then we do a whole bunch of aggregation type stuff. I'm going to gloss over the next box. Box, we do clever stuff. Um, and then there's a post-processing phase. And then what happens next is it gets shipped out to the ad networks. Now, that sounds easy, right? Um,

turns out that the the state of ad tech is woefully behind the times on the publisher side. I joined this project and I thought, "Okay, cool. We just fire off some API calls, right?" No. Um, turns out that most of the ad networks have got to a point now where they have web portals. So, you open up a web browser and you log into their portal and you set things up and you've got to set it up in two different places cuz you've got to set it up on the mediator side and on the network side. Um, that's not great, particularly when you're doing that at like 30-plus networks and you're reliant on these vendors not changing their website because you're trying to do it using like Selenium or something like that. So, that's suboptimal. It turns out that there's a whole bunch of ad networks that are even further behind the times and you actually email a human being and they will make the change in their system for you. That's actually easier for us because we can generate an email automatically. We can't like figure out

how to Well, we could figure out how to log into these websites, but we decided not to. So, we introduced some uh drudge work for some humans and we actually spit out a text file and say, "Okay, cool. Go put this into a website, please." Um, and they do that. And then we go and we go and do some sales for the day and we show some ads and it's all good. Uh, and then we get reporting for the last 24 hours and close the loop there. All right, so that's how Zoe works at a very kind of 30,000 ft view. I kind of arm waved the clever stuff and I'm not going to get away with that in this room, so uh let's talk about what clever stuff means. Um we The reason that I like that block level diagram is that it really highlights the extent to which this is a modular approach. It means that we can sub different strategies in uh and we've actually developed three or four different solvers to try and uh approach this problem. Uh we've been doing a whole bunch of experimentation with um algorithms with update candidates. Our

initial kind of uh view was that um the frequency of update actually made a difference. We we kind of we took a parallel to the stock market and said, "Okay, well, if we can get to a place where we're doing effectively high frequency trading, that feels like a win." Turns out it doesn't work like that. Um we also played around with orderings. So, in that example that I gave you, everything was in strictly decreasing CPM uh order. We played around with like trying to do it in fill rate order, trying to do it in a whole bunch of other strategies, and it turns out we were being just too Um so, the algorithms that we tried uh was predicting the price point based on market trends. Um so, we called that system the gradient surf, and you know, as as the kind of stock prices uh so, the the the parallel that I draw is stock price. So, imagine that each line in the waterfall is a single stock ticker, and it's kind of trading at different prices over different times. If you can predict where it's going to go, the theory was

that you can set your price point appropriately and and off to the races. Um that didn't seem to work well. And here's a piece of advice for you. If you've got something that's fairly experimental, don't put it on your biggest game. Uh so, I got a lot of angry emails from Words With Friends, um and we turned that one off cuz it wasn't doing well, and we tested it somewhere else, and actually it turns out it's actually not great. Um we tried linear programming with bin packing. Uh in fact, I think it's binary programming rather than linear programming. But the idea is that you can kind of put out the uh all the waterfall lines and the potential prices and do a whole bunch of like linear programming around that for which one should be turned on and off. And that gets you in a world where you can bin pack around how many requests you're going to get in the next day. And it turns out that because requests is like softly connected to the AU, it's not directly correlated. But if you know, if you can if you've got a good sense of how many people are going to play your

game on a Thursday, you've got a good sense of how many ads you're going to need to serve on a Thursday. And at that point, that puts you in a world where you can say, "Okay, well, if I'm trying to fill a bin a backpack that's this big, how do I do that?" And then we tried some simulation-based approaches and a a whole bunch of stuff around kind of doing things that are a little bit more kind of roll-out oriented. And one of the reasons that I'm not going to get deep into which one of these is great, the first one definitely is not great, um is that I don't think we've solved this to optimality yet. And like I don't want to say, "Hey, do it this way." Um also, legal don't want me to say, "Hey, do it this way in case we're onto something." Um but the point is that this is a a really good opportunity to use some cool machine learning to actually optimize something that's around your business that's not actually inside of the game, but actually drives outcomes. So, let's talk about those outcomes. Um first thing that I have to kind of caveat is that this is so hard to figure out what's

going on. Um every one of these line items in the waterfall, like I say, it acts independently. They they kind of are treated as their own commodity within the stock market. And it turns out that there's no real strong way to AB test that. Cuz as soon as you try and AB test it, A and B start doing different things, and that is really awful. Um So, we've spent a lot of time kind of just staring at charts, trying to understand what the hell is going on, trying to find uh apples-to-apples comparisons in terms of historical performance. So, what we know is that we know that we can mess it up. Um that algorithm I mentioned earlier really proved that we were actually connected to revenue, which is intuitively it has to be true, but it's good to get the validation. Um when you just torpedo everything, you have to find a silver lining somewhere. Um but what we found is that

you can't actually really tell a difference between the the operation of the humans and the and the algorithm. Like, we're getting comparable results. You can't kind of look at them and go, "Oh, well, this is obviously the one that's underperforming." And what we think is that we think we're driving upside, and the reason that we say that is that when we do find something that looks historically like it's an apples-to-apples comparison, and we drop Zoe on one of the apples, it starts to diverge. And that's really kind of a good indicator. It's not conclusive, but it's a a pretty good leading indicator that we're onto something. And like I say, the the this frequency thing, intuitively you should be able to make the changes faster, react to the market faster, and and get there. Turns out, we think we're causing so much turbulence in the market every time we change around our waterfall, that uh it actually is worse than just kind of holding back. So, we've not rolled this out to the entire portfolio yet, but uh we're starting to really strategize about what that's going to look like and how we can

do that. So, some of the things that we're looking at are kind of like predicting upside. So, if we can say that there's uh an opportunity on words today, then we can actually direct the the ad operations people to kind of focus their attention there, take the guidance from Zoe. Whereas, if we see tomorrow that it's somewhere else, we can actually be reactive to that. So, cool. You all now are smarter about mobile advertising a little bit, I hope, and benefited from about 18 months of me staring at stuff going, "What?" Um so, let's talk about something else. It's kind of a related problem, uh but user acquisition. People show up in mobile games because of reasons. And some of those reasons are that we paid for advertising on various channels. Um how would you go about optimizing your budget allocation? At a meta level, the way that you do it is that you focus on this idea of return on ad spent. Basically, this is a a kind of measure of how quickly you are going

to get the money that you spend on the ad back. And that varies on different channels, and it's be that's kind of because uh the different channels have obviously different audiences, different types of people around. So, maybe you might decide that you're going to spend a whole bunch of money on Facebook and a very little money on Google AdWords today. Um the trick to all of this is maximizing the aggregate return on ad spend. If you can do that, if you can do a whole bunch of optimization at a meta level, then you can get to a world where you can actually start to do that. But then we go can go a step further and a step deeper and say, "Okay, well, what about targeted acquisition? What about specifically for this person, how much would you bid for this player?" Two things going on there. One is the real-time nature of that because like you have to be real quick. Uh you have to have a system that runs in in like sort of millisecond order of speed. Um but you also need to know how how much you value somebody who

looks like that. You know, and in order to get to that, you need to have a good sense of whether they you know, has somebody who meets these kind of criteria that the that the advertiser is kind of presenting to us. Have they enjoyed the game? Have they paid for the game? Have they like turned out immediately, in which case maybe we're not going to spend that much on them. So, this kind of is why analytics and AI is completely intertwined, particularly, you know, at at companies that have established kind of data groups. This is always going to be true, I think. But the insights that we kind of drive based on the data that we're gathering, so you know, you at the base level you are gathering that telemetry. You're figuring out what's going on in the game. Then at the next level you're driving the insights and you're driving that kind of understanding of how people are interacting with the game, what's going on. So, this is kind of the very service-oriented view that I mentioned earlier in the game analytics group. But then you can take those insights and

start building decision systems around them. And those decision systems can actually be driving some really cool for your business. So, one of the things that often comes up when we start talking about data science and and kind of this kind of idea is the propensity modeling. So, propensity modeling is basically this idea of given historical data and what happened next. I'm going to drive a model to kind of tell me, okay, based on this I understand what the kind of causes to factors in this were. And then we can start rolling it forward and predicting the future. So, you have to So, let's give a concrete example of that. So, so like likely to lapse, likely to not come back to the game. Well, I'm going to look at a a snapshot of time in the past. I'm going to label all that data based on whether people came back or not. And then I'm going to feed it into some sort of analysis or or modeling system. Could be uh a random forest, it could be

a regression system. Um but the trick is training a classifier on that labeled data set and then run it against the recent data. And that's going to allow you to understand who's coming back, who's not coming back. Then you can take that kind of idea forward and start doing some interesting things with that. You can start surfacing offers to people who are not having a great time. Give them the opportunity to kind of, you know, get more out of the game at a cheaper price point. Now, this frankly is very low-hanging fruit. Um it's a a very obvious way. It's a very product management-centric way of approaching the problem. Um but it's effective. You know, you are you are kind of customizing the game experience and kind of giving people an incentive to hang around a little bit. Um One of the dirty secrets about that though is that if people were likely to lapse already, it's cuz they kind of weren't enjoying the game. And giving them a sale,

okay, maybe you get them to hang around a little bit. Maybe they but it's still the same game, right? What if you could go a step further? So, take those kind of understandings about propensity scoring and use them to build out some pretty deep uh holistic models and start to actually manage the experience of the player to match likes and dislikes. Um My fundamental theory is that people who enjoy a game are motivated to support the developer. And if you're not enjoying the game, it doesn't really matter. You know, sales are great, everything's great, but fundamentally, that person isn't a good match to that game. So, incentivize people to and optimize for fun experiences. We can do that by modulating our live operations. We can actually tweak things in real time and say, "Okay, well, you want this kind of experience. Therefore,

maybe we're going to alter the difficulty because we know that in an evening when you're going home, you've just been sat at work all day and you actually just want to grind through a few levels and not actually have a challenge. Whereas in the morning, we know that you as a player want more difficulty. You want to actually experience something that challenges you because you're getting psyched up for your day. If you can take everything from the data science side and match it with this kind of idea of matching people's uh matching people's desires, ultimately, personalizing the game to be what they're looking for. It's not actually that much harder than showing them the sale. You need the same indicators, right? You need to know what's going on with them. So, I've said a whole bunch of stuff. I'm starting to run out of time. So, I will throw you a final disclaimer. Zynga scale makes some of this easier to justify. Um

1% lift on our ad revenue means millions of dollars. Uh that makes it a real no-brainer to chase some of this stuff. Um but if you start thinking about things in this way, you're going to find the opportunities for your business. You know, and and your mileage is going to vary. Like, this is why I'm really loath to tell you exactly how our ad stuff works cuz I think yours is probably different. And it's probably you know, it's probably not going to work exactly the same way because it is so related to like the relationship between the networks and the and the vendor and blah blah blah blah blah. The point being, your mileage will vary, but but start thinking about this stuff because AI is a lot more than what we've historically talked about in this room, and I'm going to be yelling and swearing about that at 5:30. Uh so, please come along to the cheering tantrums uh session. Um but I want to encourage everybody in this room to think strategically about how AI can support your businesses. Um those opportunities really do exist

everywhere. You know, I've talked about advertising, I've talked about user acquisition, I've talked about personalization, but you know, what does AI for community management look like? Well, you know, think about it. There are people in this room who have opinions about that. Um what does it look like for even like stationery cupboards? How many pads of paper have you gone through today versus or in the last 6 months and versus how many are you going to order? There's probably a model in there that will optimize stuff and actually help you kind of use your budgets. That's a really daft example. Don't do that one. Um We've really only scratched the surface of what we can do. So, I want to challenge everybody in this room to go forth, open your eyes, and actually use your skills, your AI skills, everywhere you possibly can. Uh and if you would like to do so at Zynga, analytics.zynga.com. Um you can reach out to me uh [email protected]. You can find me on Twitter. I'm @luked. Uh please reach out. Tell me just how wrong I am about

all this stuff. Um but this is you know, this is a fundamental philosophy for me. You know, at the end of the day, the obsession with MPCs is cool. It's kind of nice. It generates really interesting things, but it doesn't generate business results. And that's kind of what I want to see us do more because we can. That's it. Any questions from anyone? Everybody's piecing out really fast, so I Hello. Thank you for your talk. I have one question uh about uh this persona- personalizing players' experience, right? Sounds really great. Uh so, I was wondering uh does it like if you make the game more fun for the player means like maybe you're giving him more or her more rewards or

something like that. Um, and doesn't mean that it will basically uh, decrease chances that the player will buy something. I mean, like do you have anything on that? Okay. So that the the question is um, I guess it came through the mic so it's probably okay but I'll repeat it anyway cuz I was told to. Um, the question is like if you personalize the the experience do you end up in a situation where you end up giving people more things and as a result kind of cannibalize your sales. Um, the answer is maybe. Um, you know, it's definitely something to keep an eye out for, right? Um, but at the same time if you're doing your personalization in a smart way, probably not, you know. I don't want to say yes or no to that question. It's definitely a factor. It's definitely something to be aware of but I think there's ways you can mitigate it. Um, and part of that is like you're providing a fun experience for people. And okay, people it's changing the value proposition maybe. In which case maybe change your

monetization strategy. Maybe that person to you is more valuable watching ads and they're going to stay in the game longer because you've given them something to keep them around so they've got more opportunity to be incentivized a different way. Okay. Thank you. Does your scale give you the ability to drive back into the marketplace and say give us a better interface and we'll give you preferential treatment so that we can actually do this more the way we want to or are you not allowed to comment on that exact thing? You know what? I am so the the question uh, to repeat the question is uh, does Zynga get to push back on vendors? Um, I would say we have good relationships with a lot of vendors. Um, in fact uh, you know, we're in a position where I wouldn't frame it as pushing back. We are able to contribute to people's road maps. But also like without legal in the room, I don't really want to give a definitive

answer. So, you know, we have good relationships with various vendors and different, you know, our scale does give us some some opportunities, but we're not, you know, if you're talking about us talking to Google as a as a business, like Google's significantly larger than we are. So, it depends. In other words, you may see improvement in the future. Yeah, can I ask one more? Okay, let's now we're out of time. So, let's give Luca final round of applause.