A/B Testing for Game Design Iteration: A Bayesian Approach

I think I get the nod. Okay, last talk. Um, I'm hopefully not going to make it too hard for you guys, but uh I have a little bit of mathematical content to wake you all up on a Friday afternoon. First off, I'll just introduce myself. Uh, so I've been in the game industry for quite a while. Started long time ago doing Commodore 64 game development. Then was the co-founder and CTO of Havok, uh the physics special effects company. And then a core, which was a Lua virtual machine virtual machine company which got sold to to Havok. And then now Swerve. And uh Swerve is a company that focuses really on app and game developers developing mobile titles. Uh and we provide a series of, you know, platform tools and technologies for those guys from analytics to AB testing to to messaging and targeting and and all that sort of cool stuff. If you're interested, come and talk to me afterwards. That's all I'm going to say about Swerve. So part of what we do in Swerve is AB testing. And uh we've been involved in

that for the last three years. And so we have evolved our architecture quite considerably, you know, over those three years. And I thought I'd give you some some information on how we've evolved that, some of the background to the design of the current system that we have, and some of the learnings as a result of running tests in the wild, you know, with millions of users and just uh you know, how that's worked out for us. I also wanted to sort of do a little bit of soap boxing comparing perhaps uh the sort of more classical experimental design approach, a frequentist approach to say the the more current uh Bayesian approach, although Bayes is arguably an older technology. Uh so I'm not trying to say, you know, either is, you know, perfect, but I think certainly the evolving opinion is that Bayesian approach is a much more suitable approach for the sorts of things that we do in this industry. And I'll try and make that case and do some degree of comparing, you know, and contrasting the two different approaches. So first off, just in case, and I'm sure I don't need to do this terribly uh terribly much, but I will just introduce

the concept of AB testing and the why of AB testing. Well, we all know now it's it's all about rapid iteration. You know, it's about getting to soft launch in our titles as early as we can, getting them in front of an audience, sort of using that information that we get from how they interact with the game to essentially get to a final launch, and then you're constantly iterating that game service. And the idea is you're looking at how users interact with the service, the things that are working, the things that are not working, the things that are monetizing well, things that aren't monetizing well, and you use that data to hopefully upgrade and adapt the service to user behavior. And that's that's the idea. In order to do that, you need a framework that allows you to make changes very rapidly inside the game. And you need a a sort of an underpinning data science that gives you the confidence to be able to make decisions in an appropriate way, decisions that are informed by data and informed by you know, real actionable results. Ultimately, what we're trying to do is balance an equation, right? Now, I'm sort of glossing over obviously there are lots of different types of developers and lots of different types

of business models, and this probably is a talk that focuses a little bit more on the assumption of a free-to-play style style model, but I think a lot of these ideas translate to any any sort of model. But fundamentally, you're there's a cost to you in developing your game. There's a cost to you in serving the game to your users. And that cost needs to be balanced by the things that the users gain value out of inside of your game, the things they pay for. So, either they pay up front as a premium title, or they pay after the fact with you know, in-game items or or or upgrades or whatever. But fundamentally, you know, there's a lot of different activities that you're going to get involved in that hopefully, you know, accrue to a lifetime value for your users that balances the equation of you know, the spend that you have to get the user into the game in the first place. So, I'm going to look at some of the the AB testing techniques that we can use on the right-hand side of this sort of equation. We're not going to be interested at all at this stage in how we get users into the app, but it's more what do we do after they arrive in their first session. How How can we interact with them and how can we learn from

their behavior? And obviously this is the equation we're trying to balance. We We want to an ROI. We want to make money. We want to a profit. Uh and fundamentally we want to make sure that we're making more money than we're spending. So ROI is a an interesting thing and and lifetime value is perhaps an even more intangible thing to get your your head around. You know, arguably lifetime value is an infinite series. You know, you you wait for long enough and people will spend more and more in your game and eventually they'll they'll stop playing. So over what period of time do you want to measure this? And what sort of time frame do you have in order to make changes inside of the game? So you know, just looking very quickly at some data from games on our service, this is a a classic profile of you know, popular games with with high volumes of users. And this is showing the spend curve. This is where spending happens inside the game based on you know, from day one install. How How early in the game are people spending and how much revenue comes in in those early stages? So then turns out the vast majority of the revenue, vast majority of people's

engagement with your title tends to be in the very early stages of their of their play. So usually you know, 50% of revenue comes in the first 30 days. The other 50 is spread over maybe a year or longer. So it's all about having a very quick reaction time, being able to fire out tests and test things and look at what users are doing and react it very quickly in order to sort of maximize your ROI. So nearly finished my AB testing soapbox. You need to understand what users are doing, which is about analytics and getting data in. You need a an ability to test hypotheses against your users. You want to be able to try things out and see what works and see what doesn't work. And essentially use a data-driven approach to making those decisions. Uh and finally need a way to sort of act on the the data that you're seeing. You need to push out content or lock in changes or you know, adapt the game service very very rapidly. And that sort of promotes the idea of having a data-driven layer on your game, which is something in in the gaming industry we know a lot about, you know, in in sort of separating tools from runtimes. It's

exactly the same concept can be used for for AB testing and for live update of the game service. So, the mechanics of an AB test are very simple. At any given point in time, you have an hypothesis you want to test. You split your users into separate user groups. You show them different variations of the game, different content, different price points, different whatever. You look at the data coming back in and you assess the validity of that data. You try to determine is this actionable or not. You use statistics to do that. And that's actually where we're going to focus the the talk today on. And then ultimately you want to choose some winning variation or treatment and push that out then to all your users. And this cycle continues ad nauseam. You know, on the successful game developers who are really adopting AB testing are running multiple tests in parallel against different segments of users. And you know, having to deal with some of the complexities of that, but essentially it's all all the time testing and all the time updating. So, from a platform perspective, the mechanics are quite simple, you know, at a high level. We have a game which is

being served typically with a a back-end server storing game state and and user user save play and all that sort of stuff. And so on the left-hand side you have your BI stack, your analytics stack, your your testing platform. And as the game is uploading or as the game is being started, users are being bucketed into A or B groups and they're getting custom content. So, the A group gets one version of the game and the B group gets the second version of the game. Now, I'm not going to get into the details of serving that, but think of that as a content management system where the content that you get streamed down to the game depends on what user group you're in or what behavioral target group you're in. And that content might be just different imagery, it might be data controlling the flow of a a UI, or it could be price point information, or or whatever. Users then play the game, interact with it, hopefully enjoy that game. And as you're doing that, you're streaming event data back to your your back end service. And that event data is essentially tagged by is the user in population A or population B. And you use that to generate pretty graphs and,

you know, boot up Tableau and and do a lot of analysis. And ultimately, based on that data, you make a choice, and you lock in perhaps variation A or variation B. So, you have this sort of feedback loop where content is all the time being deployed with logic to the game. The game players are playing away, and that data is coming back, promoting us to to make different choices about content. And this is sort of a constantly adapting and shifting system. And that sort of operates nearly in parallel to your regular game development cycle, where you're pushing out new, you know, app updates every 2 weeks or every 4 weeks or whatever it is. These sorts of iterations can happen in in hours or even minutes. And if you automate some of this, it can happen in seconds. So, what what would you want to test in your game? Well, anything, of course. Anything that's available to be tested can be tested. And the types of things that we see people testing, typically, the low-hanging fruits are things like messages, you know, different layouts of of UI, different message content, different calls to action, all that sort of thing.

Uh tutorial flow is another classic one. So, looking at building into your game sort of a data-driven view of the tutorial, being able to dynamically reconfigure the tutorial on the fly. And essentially thinking of the tutorial like a state machine that you can sort of play with as users are interacting with it. So, this is an example of an actual tutorial. And there's a choke point where users are falling off at, say, the seventh or eighth stage in the tutorial. And that's a a good motivator to go in there and try and AB test different versions of whatever it is you're presenting to the user there, cuz it's clearly not working so well. Then you can get into the sort of the economy of the game. And there's a whole ton of this sort of stuff that goes on in the background. And and you know, the simple things to think about are are changing price. But typically, you don't just change price cuz that's a bit too disruptive to your to your users. You'll typically go in and look at marketing activities. And if you've got an in-game store, you can look at promotions and discounts and you know, trying out different discounts at different points in time. You could try out different exchange rates. A classic uh technique in economy management in games is as users progress

through the game, you sort of induce an inflationary effect into the game's economy. Things get more expensive. And sort of we sort of know this intuitively, but how more expensive and how quickly do that is a sort of like a game balancing and economy balancing thing. And you can use AB testing to to test that out and and to test if an exchange rates. Uh something that's used quite a lot is your sort of price sets and uh the pricing that's available to users. Now, you're not changing the value of anything here. You're not saying something that was $2 is now $4. You're just essentially changing the inventory that's available in the store. Uh in this case, it's you know, uh coin bundles or whatever. And if you look at these price sets just a little bit closely, you'll see that as you go from top to bottom, we're sort of increasing the value of some of these things in in fairly subtle ways. In fact, the users are arguably getting better value because we've increased the price point from you know, $1 to $2, but we've more than doubled the number of gems they're getting. So, it's not an unfair thing. But all we're doing is we're sort of changing the choices that

users have inside of the store, which is exactly the same as a retail manager laying out their shelves. You know, deciding what things to place on on key places on the shelf. And we've exactly the same problem of scarcity in games. You have scarcity of of screen space. So, what typically happens here is testing has motivated the different price points. And as users go through the game and perhaps become a paying user, you start shifting around the inventory that's available for them to purchase. Again, to try and you know, increase the monetization inside the game. By basically getting out of the way of the user, typically people who pay a lot in games and like paying a lot in games don't pay at the lower price points. They prefer to pay at the higher price points. So, you know, this is a way to to allow them to do that. Here's some examples of that in action. So, this is a a sort of a complex graph, but I'll try and step through it. On the bottom, we have the value of the very first purchase a user makes out of one of those sort of price sets, one of those gem bundles. And there's multiple different price points they can purchase at. So, let's say the $2 to $5, $15,

$30, so you know, a classic uh stratification of pricing inside of a game or a free-to-play game. And you can see that there are users who purchase first right over there on the on the $70 line. And some of these guys then continue to purchase at the $70 level. So, that's what this sort of second faded line means. Each user there first purchased at 70 and then their total purchase so far is 140. And you can see the steps further up as user can continue to purchase. So, these guys up here are our VIPs. These are the guys who are spending lots of money in the game. You know, clearly a high disposable income. But some of the things to notice here are AB testing smears. So, what we've done here is we've looked at different price points and we've looked to see can we shift say a price point that was $28 to $29 to $30 and look at the price elasticity, the sensitivity of the users to those changes. Now again, you're you're shifting, you're also giving them better value. You're adding, you know, to the coin bundles that they get, but you're just looking at their sensitivity to the specific price point. And that's,

you know, classic AB testing used to try and make very small changes in the economy, very small changes in the inventory, which can have big impacts on the actual underlying revenue itself. Here's something else that you want to test, the sort of marketing activities inside of your applications. Something that people do a lot in games, obviously, is ask the users, "Can I send you a push notification?" Or, "Would you like to register on Facebook?" Or, "Would you like to tweet the thing you just experienced or whatever it is. So these are sort of ways to increase the virality of your game. Now to do that the last thing you want to do is just ask them right when they install because everyone says no. Because there's no benefit to the user at that point. They haven't learned anything about the game. So a classic thing to test for is let's wait. Let's create a a sort of a marketing strategy that says we'll wait for a little while wait until the user has played the game a couple of times let's say three sessions. And then we look at a key point inside the game that we know people have shown high engagement after. Let's say there's a a video reward sequence at the end of

completing a level. So right at the end of the level you play a pre-roll video and then you ask the user would you like to share your experience on Facebook. And at that point you bring up the Facebook registration page. So there's lots of choices that have been made there and the danger is to assume that you have an intuitive you know a good opinion as to you know the best place to to make that call to action happen inside the app. The best thing to do is to have lots of opinions and to try them all out against your users and see which one works best. So for example how many sessions do I wait? 1 2 3 4 5 try them all out. Test different groups of users and see what your sort of conversion rate is for the Facebook call to action in this example. So this is an example of testing the marketing logic inside of your game. So the when and the where. So what you end up with is data. And this is sort of the the classic result you'll get from running a multi-treatment AB test. That's a sort of a bit of a a contradiction in terms I agree but AB no one is actually doing AB testing in

games they're all doing sort of multivariate testing or multi-treatment testing or constant optimization but we just call it an AB test just for simplicity of communication. So here you can see an AB test running with I think it's four variants. And you see over time the pattern of behavior is changing our our confidence in the winning variant has changed dramatically. Certainly from the start you can see there's a huge amount of oscillation, and then things slowly settle down, but there's still an inherent amount of noise. And this is the challenge you face. So, if you're running these sorts of tests, you have to have an underpinning mathematical model to valid- you know, validate the choices you're making. You can't just look on Tuesday and say conversion rate A is higher, I'll go with conversion rate A. You need to, you know, know what's happening underneath that. So, in this case here, for example, you know, at the start of the test, looked like one of these variants, you know, the yellow one was winning. But very quickly, the blue one started to take over. And at some point, you're going to make a decision to lock in a result. Otherwise, you're going to run the test for too long and, you know, lose out on possible extra revenue or extra audience

engagement. And I suppose the question is, A, how how do you produce this data? How do you produce these probabilities or these confidence levels? And B, when do you decide to lock in a result? Uh so, I'm making an assumption here that we're not looking at a multi-arm bandit style scenario. We're just looking at something that is managed by a product manager to look at data and then lock in a result, which is sort of a a classic AB test. So, I'm going to run through how how we do that. And ultimately, we want to look at, you know, conversion rates over time. Uh and we want to see clear separations between them before we make our our judgment. So, I'm going to use a very simple and rather silly example, and apologies to all the copyright owners that I'm clearly uh walking over at the moment. Um So, Candy Candy Crush Bolt is a obviously a fictional title. And what we're going to do is we're going to launch this title, and we're going to try out some different variations of it and see impacts on a a conversion rate. And the conversion rate we look at is day one retention. Uh the least understood and most overused uh KPI in the game industry,

but that's a topic for another talk. So, let's imagine our fundamental conversion rate for day one is 30%, which means 30% of users come back on the second day into our game. And I'm I'm pretty happy with that, but I'd like to push that up a little bit. So, what I do is I start playing around with the game play. I've no idea how you actually play this game. I'm actually quite intrigued and might try and code this up someday. Um, I suspect it'll slow down the speed of a cannonball style, you know, infinite runner, but anyway. Imagine we've made some game play change and this is obviously the the change I've made here has absolutely no bearing on the math. Uh, and there's an impact obviously on the game. There's an impact on the number of users who come back the second day. And in this case, we were expecting a 30-day a 30% day one retention and in fact we saw no users come back the next day. So whatever we did with the game, we've completely destroyed it. Now the question we ask ourselves at this point is after 50 users we we've seen none of them return on day two. So that's a really bad result and I guess

the panic attack starts to to settle in. The really the question we're asking ourselves is how bad is this and is it enough information now for me to say, well, I should stop that. I should move and make a change to the game and I revert out the change I've just done. That's a sort of an interesting question. I'm going to sort of ask that question both from a frequentist perspective and a Bayesian perspective. So first off, the null hypothesis view of the world. This is the way we think about it. So the null hypothesis states, well, in the absence of anything else, we assume that the change you've made had no impact. And underlying my assumption is that I have a model for how users are coming back to my game. 30% of them return and I have some idea of what variability I have in that 30% sort of from day to day. So that gives us a spread and this is usually modeled in some sort of, you know, normal distribution or binomial or or whatever. So my null hypothesis is that there's no difference. So having observed the users, I assume from day one that there's no difference between that and my 30% use case. What I'm trying to do is I'm trying to disprove that. I'm trying to say, well, actually based on the data, the null hypothesis clearly

doesn't hold anymore. Now if I saw a retention rate say maybe of 35%, I don't have enough information at the moment to dispute the null hypothesis and I'd probably accept it. But if I saw a a a a a conversion rate that was off the charts, I'd probably start thinking about their that null hypothesis is no longer true, and so I reject it. So in our case, we've got a 0% conversion rate, which is obviously very bad, and it's off the off the charts. So we would reject the null hypothesis, and this is our frequentist result. And that's actually the result of a classic AB test. But what does that result tell us? So essentially it means that the 30% conversion rate that we had previously is unlikely to be the new conversion rate that we're seeing now. That's actually all it really tells us. Doesn't say anything else about how unlikely that is. It just says it's unlikely to be. Now we capture that information in what's called a P value or confidence value. And the confidence value is just based on, you know, normal distributions and standard deviations and all that good stuff. And what we say is, if the

value we see is sort of within 95% of the main center of the bell curve, we accept it. If it's outside of that, we reject it. And in our case, it's certainly outside of that. Now what the P value actually captures is the probability of observing a result which is a as extreme a result assuming the null hypothesis is true. So that's a bit of a tongue twister, and it's quite hard to get your your head around, but it's it's essentially saying the probability of the observations you see in the data given the model. So we sort of assume the model is true, and we look at the data and we try to disprove the model. That's how uh sort of null hypothesis testing works. And what you end up with is sort of a truth table like this where okay, maybe the hypothesis is true and we accept it, which is a good result. But maybe the hypothesis is true. There's no difference, but we reject it. We see an outlier, and we say, "Actually, I think I made a a good positive change to my game here, and I accept that as being something I'll make a change on." But in fact, that's a false positive, and that's where the that's the root of evil, you know, for uh null hypothesis testing in

uh in the AB world. So I'm going to sort of talk about these false positives. A P value is a number between zero and one and usually what we say is if it's less than 0.05, we accept that there obviously has been a big change. It's not just due to random noise and we make the change and we lock it into our game if it's a positive change. A P value of less than 0.05 simply says the retention rates are different. If you sample this thing 90 you know 95 times out of 100, that retention rate will actually be a different retention rate. And if you dig into it and express exactly what that means, it says that the the data that you've seen supports a rejection of the null hypothesis. And that's essentially saying the probability of seeing a result as extreme as this is less than 5%. And that's again really tricky to even communicate you know to your CFO or to your head of design or anything like that. So that sort of leads into an argument that P values are difficult for us to intuit about. They're sort of unwieldy

to work with and it'd be nice to have something else which is just a probability. Is one better than the other? Give me that probability. And that turns out is exactly what a Bayesian approach gives you. And so you don't have to sort of wrangle your head around you know normal distributions and null hypotheses and rejections and and non-rejections. The other problem with this approach is peeking. So the classic thing people do is set up these AB tests and they just constantly observe the result and once the result gets below 0.05 they say I'm done. And that's a really bad thing to do it turns out. What you should be doing is sort of looking at the power of your test, deciding in advance how big a change do I want to be able to detect. So let's say I want to be able to detect a 5% change in retention. I need to have you know observed that retention change over time. I need to know how that distribution looks like so I need to have some idea of the variance of the standard deviation of the retention stat and that gives us a number of participants and typically that number of participants will be say 10,000 or 100,000 or it's hugely dependent on the size of change you want to you want to detect. If you want to detect a really small change, you know a

shift of retention from 30 to 31, you probably need to observe a very large number of users to essentially compensate for the noise that's in your system. So, what does this mean? This means people typically ignore this and say, well, I've computed all this and I see I need 100,000 users. I start my test and I immediately start looking at the graphs and the P values. And as soon as I see something hitting a 0.05, I say, "Stop, I'm good to go. I've got the result I want to see." And what that means is it's saying, "I'm taking the very first time that the noise has pushed my P score below 0.05," which is a really bad thing to do. What that does is it increases the chances of you getting a false positive, and it's actually quite bad. Let's imagine during the course of your test, you were prepared to accept a 5% false positive rate. So, you're running a test and you expect that one in 20 times I'll I'll accept the result and it'll be wrong. If you look at that test 10 times in succession before it's actually reached conclusion, you're actually increasing the likelihood of you seeing a false positive by a factor of five,

which means you need a essentially a factor of five more observations to to compensate for that. So, people generally speaking don't realize that. And there's a good blog post by Evan Miller that I guess anybody who knows about AB testing has come across before, but it sort of I've just pirated his data here and it's a a really good exposition of some of the problems that exist. So, the second problem or the third problem, should I say, is the family-wise error. And this is, you know, multiple uh if you have multiple treatments, again, you know, the chances of you seeing a false positive if you have more than one treatment increases every time you add a new treatment. So, for example, if you have two treatments, uh your false positive rate for exactly the same test is now up to sort of 9.75%. So, again, it's nearly doubled. And people tend not to correct for that in their AB tests. Okay. That's the bad news, sort of. Uh now, the good news, the Bayesian view. The Bayesian view takes a completely different approach, and it basically says, "We have an expected retention rate. It's 30%." That's what we saw before. So, that we we've acknowledged that our system without any changes has a 30% retention rate. I'm going to look at the

data and based on the data, I'm going to update my view on what the likely retention rate actually is. So, I have a model of what it was before. I have data coming in and I use that to create a model of what it might be now. And in this example with 50 users all not having been retained, what actually happens is we get now a new expected retention rate, which is 15%. And if you look at these shapes of these curves, these curves give us a sort of a our belief in that retention rate. And the first one had a a 30% with a fairly large spread. The second one is centered at 15% but a tighter sort of spread, which means that we're actually a bit more confident about the 15% number than we were about the 30% number now because we've observed some results. But the nice thing about this is it still admits the fact that we might be wrong and it's something that's immediately usable. It looks like now the conversion rate is is 15% but I'm prepared to accept more data. And when you're dealing with these sorts of systems, runs of failed conversions happen all the time

and you have to be very careful about that. So, I'm going to go through and talk about this and real examples from games and I'm going to show you how you build up this Bayesian approach to to AB testing. But the difference between the frequentist view and the Bayesian view is essentially the Bayesian view gives you the probability of the model given the data. If you remember the frequentist one was the probability of the data given the model. So, the model in the frequentist view is sort of locked in. Here we're sort of saying as we see new data, we'll update this model. We'll sort of learn what the new shape of our probability distributions are and use that to to make a an assessment. So, trying to put some trying to get a little bit less abstract about that. The canonical example is the the coin flipping. So, I'm going to stick with that. You know a coin, it has a head and it has a tail just in case you didn't know. And when you flip it, one of those comes up and with a 50% probability it's heads or tails. All right, we got that bit. So, let's say we run an experiment for a period of time and look at the number of flips. And And just look at this, you can see immediately there are runs. So,

we see series where you get, you know, seven heads in a row or five tails in a row, and that's completely expected. People get very confused about random systems sometimes and think that's That mustn't be random anymore. But ultimately, we get a series of heads and tails. And you look at the sort of the trend of those heads or tails over time. You look at the distribution of the or let's say the the percentage of heads and see how that works. And what we see is, you know, early on it's all over the shop, but eventually, after enough observations, it settles down. So, it's a settling down to an expected value, which, you know, anybody who has a stats background knows this stuff pretty well. But the point here is, early stages, even something as simple as a coin exhibits a whole lot of noise. And if you're sort of modeling that and trying to produce data out of that, you have to be very aware that you need to, you know, run a lot of data before you see anything that's actionable, that gives you any real value for the the long-run average of the the coin. So, just a bit of terminology just to get things straight. So, I'm going to talk about the probability of an event X.

We can talk about the probability, the joint probability of event X and Y, so X and Y. And then we're also going to look at the probabilities of X given Y. So, X, assuming Y has happened, what's the probability then of X happening? So, these three things together are going to be used in in the Bayesian way for us to essentially trying to figure out what's the probability of the model being true given the data that we've seen. So, let's look at our heads and tails model. There's a a nice sort of function that captures uh the the probability of tails or heads in this case. So, we'll assume heads represents one and tails is zero. And the probability of uh heads given some parameter of our coin is given by this Bernoulli distribution, which uh many of you might have seen before. This theta thing is the fairness parameter. So, if it's 0.5, it says it's equally likely to be heads and tails. If it's 0.2, it's more likely to be tails. And if it's 0.8, it's more likely to be heads. So, it's a nice parameter that gives us a control. Think of that now as your conversion rate, your day one retention. You know, theta is your day one retention probability. Uh and in this case, we're assuming our day one retention is 50%. So, every time

we flip a coin, every time we play a game, the user comes back the next day or not with a 50% probability. So, for a fair coin, that's 50% or 0.5. And if you sort of push 0.5 through that uh Bernoulli distribution, you'll see that the probability of heads is 0.5 and the probability of tails is 0.5. So, there's nothing particularly exciting about that. So, that's just the probability for a single throw, but we're more interested in lots of users and the distribution of those guys. So, you build up a Bernoulli Sorry, not a a binomial distribution by essentially running the Bernoulli process multiple times. So, flipping multiple coins and then looking at the proportion of those flips that are heads or tails. Uh and in this case, you just have to account for the combinatorial with the nCx. So, just graphing that, here is 20 coins or sorry, 20 tosses. And here's the distribution of the expected number of heads. And as you'd expect, we'd expect the number of heads to be around 10 if it's a fair coin. But we also allow for the fact that there are possibilities it might be 15 or possibilities it might be five, but with

significantly lower probability. So, this captures our sort of view of, you know, how likely it is for a different number of heads to arise. Now, for an unfair coin, it shifts. So, this is an unfair coin where it's more likely to come up tails as opposed to heads. If it was heads, it'd be shifting to the the right-hand side. And here, the theta parameter is 0.2. Okay. What we have here is a model of the probability of number of heads. We can actually turn that on its head, and no pun intended, and think of that as a likelihood. Imagine we we observed a series of flips, heads and tails. So, there are 20 of those, and we had this data set. And the question we wanted to ask instead is, what's the likelihood of theta? What's the likelihood of theta be or the the being fair? I see 50% heads, 50% tails, that means it's very likely theta is 0.5. But if I saw 10 heads and two tails, the likelihood of theta being 0.5 is quite low. So, we can use exactly the same uh sort of function to capture likelihood of a particular value of theta. And here's some graphs of that. So, let's say you did uh one or two

throws and you saw one head. One head means there's also one tail. So, we get this sort of fairly smooth distribution to say, "Well, there's nothing really to be gained or we don't we haven't learned very much. All we've learned is that there is at least one head and one tail on this coin. So, the probability of uh tails being zero is uh is zero." As we flip more and more times, we start gaining confidence about what the actual this model parameter is, this theta, this fairness value. And again, think of this fairness value as your day one retention or your conversion rate for an ad or the likelihood of someone to to purchase an item in a store. It's all the same sort of process, uh give or take. So, this captures our view of how likely a value of theta is given data that we've we've observed, which is going to be uh useful in a second. So, X is going to be our observations, the number of heads. Theta that we've just talked about is a model parameter. Our our model is essentially the binomial and the parameter is essentially this fairness value that

we've talked about. So, it's a a value from zero to one. What we've looked at so far in this sort of likelihood is the probability of a given set of data given theta. So, the probability of X given theta. And that's what uh that previous distribution gave us. That's actually, if you recall, very similar to what the frequentist view is, the probability of the data given the model, right? What we actually want to do is to determine the probability of the model itself. We want to say, "Is it true that the conversion rate is 30%?" We don't actually want to know, "Is it likely that we saw a conversion rate of 28% if the conversion rate was actually 30%. So, that's the wrong question to be asking. We really want to say what's the conversion rate and what's the the the probability that our model is correct that the conversion rate is 30%. So, that's what we'd like to get. Now, looking at those two, you know, P X given theta and P uh theta given X, it looks like they're they're fairly similar. But, be very careful, these are not the same thing. They don't interact. Uh P P of X given theta is not P of theta

given X. And a nice way to intuit about that is just to put something real in there. So, this is a obviously uh a different version of LaTeX being used to produce these diagrams. So, the probability of it being cloudy given that it's raining, think about that. If it's raining, it's very likely to be cloudy. So, you'd imagine that probability is high, right? It's not equal to the probability that it's raining given it's cloudy. And again, think about that. If it's cloudy, the the likelihood of it raining is actually a good deal less. Maybe 20% of the time uh that it's cloudy it actually rains or maybe less uh or maybe more in San Francisco. Um So, these two things are clearly not the same. And what we want to do is try and find out the right-hand side given the left-hand side. So, we'll just do a little bit of math. And we end up with this rule here. I'm not going to go through the math, but uh folks who are interested, it's just the conjoint probability expressed in terms of the the conditional probabilities. So, this thing here is basically saying the probability of Y given X is the same

as the probability of X given Y times the probability of Y divided by the probability of X. And if we expand out the probability of X into something a little bit larger, we've actually done something very interesting. We've managed to figure out a way to relate the sort of frequentist view of the world with the Bayesian view of the world. We now can relate the probability of the model given the data to something which is about the probability of the data given the model. And that's exactly what we want to be able to do. And Bayes' rule is this. This is what Bayesian statistics and Bayesian inference is essentially all about. So, there's two different forms, discrete and continuous. Uh you know, we're working pretty much with the discrete form here. And if you start taking this and applying it back to our model of the coin, what does that look like? Well, first off, the probability of Y given X is what we're looking for. It's the probability that our retention rate is 30%, or the probability that the coin is fair given the data that we've observed so far. Given the fact we've seen no uh users out of 50 users

returning, what's the probability that the retention rate is still 30%? That's the question we're asking. And we relate it to what we know, which is a model of how coin flipping happens. So, this in this case it's going to be our binomial, and that's our our likelihood. We also have a probability of the model itself. So, this is a one of the really interesting parts about Bayesian. We say, "What's the probability of theta?" Now, theta, if you think about it, is the fairness. So, this is a way of you expressing to the system how much you believe that the coin is fair. And you might not believe it's fair at all. You might have no information, in which case you say, "I don't know anything about P of theta." Or you might say, "Well, actually I'm very sure it's a fair coin. I'll say P of theta is 0.5." and plug that in. And this down here, for want of a better term, is just a normalizing term. We're dealing with probability uh distributions, so they need to sum to one. So, this underneath it is just the integral of the thing that's on the top. So, we don't need to get too concerned about that. So, if you just break that out, and these are the classic terms people use in Bayesian. So, that the probability of

the model given the data, the thing we're looking for is the posterior. The probability the data given the model is the binomial thing, and and this is the likelihood. This is something we we sort of choose. We choose a model that maps to the process we're trying to test for. The probability of the model itself, the model parameter, is our prior. And that's the thing that we use. That's our industry knowledge comes into play here. And we can say, "P of theta should be something because I understand that my retention rate should be 30%." And I plug that in. So, it's a really good way of bringing your industry experience to to bear. And the last thing is just a normalizing factor. So, this captures this sort of prior captures our belief. Captures our belief in the model. How strongly do we think that our prior experience is right? And there's a number of ways of doing that. Um and you know, we might say for our coin, well, we think it's pretty fair. It's likely that theta is 0.5. But, we're going to allow for the fact that well, we hadn't seen it being manufactured. So, we're not completely sure. You know, we haven't measured it. We haven't weighed it. We haven't looked to see if it's a a dual-sided coin. I mean, a a dual-headed

coin. Or we might decide, actually, I've seen, you know, someone play with this. A magician has been using it on stage, and it's clearly, you know, been doctored. So, I'll model up by a distribution which is different, which is skewed maybe to the right-hand side. It's more likely, I believe, that this coin is going to come up heads. And that's you can encode that straightforwardly into the prior. It's just a normal distribution. It just needs to sum to one. And that allows you to capture your your your understanding. So, what we do in Bayesian is we plug all that together, and we try to iterate based on data towards an understanding of how good that model is. So, we start off with essentially uh an initial guess. We have a prior. We say, we believe the coin is fair, and we create a distribution based on that. We plug it in. We know what our likelihood is because it's the binomial in this case. We observe data, coin tosses. So, we look at 10 tosses and see how many heads or tails there are. We plug that into our likelihood function. Multiply all that together, and we get a new, you know, result at the left-hand side, which is the posterior. And this new result is essentially a new

distribution. It says, based on everything we've seen, our belief in the theta's and our likelihoods, this is our best guess now as to how fair the coin is. And it's still a a distribution of probabilities. And what we can do now is plug that back in. We say, all right, we'll use that as a starting point for the next iteration. And then we get a lovely sort of iterative loop where we can continually refine and plug in the data that we see, plug in the new data that we get, and evolve over time this probability that the coin is fair or the conversion rate is actually 30%. So, selecting priors is tricky. Now, the great thing today is with computers and, you know, numerical methods and, you know, PyMC and lots of great mathematical libraries out there, we sort of don't have to worry about some of this stuff. But, typically in the past people used what's known as a conjugate prior to make all this simple. So, just very quickly I'll describe that. This is our likelihood function. When you plug it into the previous Bayes, you get something horrible like this. So, obviously this is a scary function. Lots of things can go wrong here, but the most important thing is

we've got a prior here and here. So, we're going to multiply our binomial by a prior. We hope that that will come out to something that's easy to work with. And then underneath of it we've got this integral with a prior. Again, a bit messy to deal with. So, is there some way we can sort of stack the odds in our favor and make life easy? Well, you can by using what's known as a conjugate prior. So, we would like that multiplying the prior by this binomial thing gives us something that looks basically like a binomial again. So, that when we multiply again, it looks again like a binomial and we can just deal with this algebraically rather than numerically. Similarly, we'd like that integral just to be tractable. Ideally, it's, you know, analytical so I can actually just run a formula and get a number at the end. And that turns out to be fairly straightforward for lots of different distributions. So, that's what a conjugate prior is. It's something that just works nicely in this sort of formula. And the conjugate prior we use for a binomial is something called a beta beta function, beta distribution should I say. And that's if you look at the shape of it, just look at how it's written, it's sort of similar to that binomial. A binomial is in fact

a specialization of a beta function. So, I'm not going to get into this. This is the, you know, the normalizing term, but just to say it's a factorial at the end of the day. This is it's an integer value. It's really easy to work with. So, a big scary integral turns out to be just a couple of multiplies on a computer, so it's very straightforward to code out. And it turns out this beta function, previously we had uh the probability of a head or tail, we can actually create a prior that maps to the number of heads and tails you've seen before. Literally by plugging in for this first parameter A, the number of heads plus one, and the second parameter B, the number of tails plus one. So, if you flip a coin and see 10 heads and five tails, stick in 11 and six for the A and B parameters, and that is a beta prior that represents exactly your knowledge about that coin to date, and you use that to kickstart the system. It's really simple. So, here is just examples of that beta distribution. And uh this is sort of an interesting one. This is uh I have no information yet. I've never flipped a coin. In fact,

I don't even know if there's a head or a tail, so I can't say anything about it. So, essentially it's a an uninformed prior. It's just uniform. We know nothing. After say flipping a coin, you might find it came up two tails. So, what this means is I now have evidence that tails exist, but I've no evidence that heads actually exists. So, I can't say the probability of heads being zero is is uh is is non-zero. So, we get this sort of strange slope distribution. As soon as I see a head, I start getting the classic sort of bell curve shaped thing. And different shapes of A's and B's give you things that look like exactly like the binomials we saw earlier, and that's true cuz there's always a beta to match a binomial. So, plug it in. Again, it looks a bit scary, but we have our binomial and we have our beta. We multiply those things together, and we get something that looks like this. That also sort of seems like it's relatively scary, but it's actually really simple. All that's happened is we've changed the values of A and B in our beta function, essentially. And that turns out that

Bayesian update step in uh full Bayesian inference engine using binomials, uh which powers certainly the the Swirl baby testing system, involves just adding one number to another. That's fundamentally what you're doing, and then some stuff afterwards to to interpret the results. So, I'll skip over that. So, putting it together, we first off decide on a prior which captures our belief about how strong that the coin is biased. And I just is a function with a certain shape. We run our experiment and observe some data for heads and tails. Based on that, we update our view of what the posterior distribution is our model is. And using that, we can sort of feed it back into the system and keep going. And essentially at any point in time we have a a new belief about how well our model represents the coin or represents the conversion rate. And at some point you stop and say I've done enough. Some conversion has has been achieved or I see a result that I'm I'm happy with. It's got to be above a certain threshold. And we'll talk a little bit about that now. So, this is what it looks like when you

run that. It's an iteration process. And all that's really happening is the shape the shape of the beta is changing representing how confident we now are in this model. And at any point in time we can stop and say this is what we think our conversion rate is. It's looking at this is about 30% with a spread of, you know, .1. Now, if we plug that in to an actual experiment and start running it. So, we start off with an a uniform prior. That is, we we don't have any prior knowledge at all about this coin. And let's say the coin is actually fair. It's it's theta parameter is 0.5. And we start flipping it and seeing what happens. So, we get two heads, two tails, 10 heads, 11 tails, 31 heads, 29 tails. And as we go, we can see what's happening is we're increasing our certainty that this coin looks like it's fair. The the dotted line there represents the 0.5 line. We can see that even though it's still even though it's it's centered at 0.5, we're still allowing for the fact that we might not have seen all the examples of the coin's behavior. And so, we're allowing for the fact that it could be 0.6, 0.7 or something lower. But as we

see more data, that tightness starts to happen. And essentially we get more and more confident about the spread of the actual true uh distribution. So, we're pretty sure this coin is fair cuz it looks like this one is very close to 0.5 at its peak. Let's say we have prior knowledge that the coin is fair, right? So, we're actually starting off with something like this. This is our prior, and we've drawn this curve ourselves. Say previously, we saw 50 heads and 50 tails, and now we start the experiment with essentially the same sort of data, run it a few times, and you can see even though we're sort of getting the same data coming in, it's not exactly the same, but it's pretty close. Generally, the shape of the curve hasn't changed at all. What's really happened is nothing has come in to uh upset our belief that this coin is actually fair. In fact, all we've done is we've improved our certainty. It's become sort of tighter. Let's look at the opposite case, where we start off with a biased coin, and we start observing what what's uh the true case is. So, we start off with no prior. We've no previous knowledge at all. We've We haven't seen the coin being made, and we start observing the

result. And in this case, you can see that over time, we build up a profile that I uh this coin is definitely unfair. It's sort of off to the right hand side. Certainly, that the mass of our probability distribution is not centered around 0.5. So, we'd say that's, you know, that's probably an unfair coin given what we've seen. Now, what happens if we start out with a a preconceived bias that we think this coin is fair? What does that mean? Well, actually, that changes things, and this is maybe the negative side of priors. So, if we're pretty certain the coin is fair, and then we start observing it, and then it comes up like uh a very unfair coin. We see maybe 20 heads and four tails. All it does is really shift it slightly because there's a weight of evidence already expressed in the prior. And that's where you working with the prior is really important. You need to be very clear about how much uh you know, confidence or how much certainty you put into that prior because it it'll impede progress in your experiment if you're really sure. But, that's a good thing because it means that you're saying, "I'm really sure the conversion rate is 30%. I want to see a heck of a lot of

evidence before I'm prepared to accept that it's not. So, in this case we're not sure of anything. So, when do you reject? Well, quite simply we reject based on, you know, how far into the center of the bell curve or into the center of the the mass is the observation that we have. And in this case we can use we can look at the credible interval. And this is just looking at the probability distribution, figuring out where 95% of the mass is. This is where it's sort of similar to null hypothesis testing and then checking to see if the 0.5 value is inside of that. And in this case it is. So, we would express this as saying, you know, we're 95% confident that this coin is fair. Similarly with the previous one with the uniform prior on the biased coin, you know, we're pretty confident we're greater than 95% confident that this coin is unfair in this case. So, it's outside that credible interval. So, the prior captures our belief and captures our experience. And it's a really creative way mathematically to get business knowledge and intuition into a system like this, which the null

hypothesis approach has no real easy way to do. Strong beliefs means that we need lots of evidence to conflict our belief, which is good. Means I really need to be convinced before I'm prepared to accept that this coin isn't fair. It also provides inertia and allows convergence a lot more quickly if indeed your your belief is true. And with enough samples though that you know, the importance of the prior diminishes. So, that's a good thing as well. And it acts like a really nice filter at the start of your test. So, the noise at the start of your test essentially is reduced significantly by a decent choice of prior and that makes the experience for product manager or an analyst looking at the AB tests and makes it a lot easier for them. So, let's look at running a test. So, we've got our Candy Crush Bolt thing again. We've got our variation A and B. We start observing results for both A and B and looking at the conversion rates. What we have now is sort of two of those coin flipping tests. And we we accept the fact that there's probably two conversion rates here. We we just want to know which of those is better.

So, with multiple variants, things get a little trickier. We're no longer dealing with one distribution. We're actually dealing with a two-dimensional distribution. Uh and in order to actually evaluate the results, we start getting into a little bit more hairy stuff. And that's where the computers start shining. So, we can start using sort of quadrature rules and other things like Markov chain Monte Carlo. But, here's what you're actually doing. You're getting surfaces for a two-variant test or two-treatment test. If you have a three-treatment test, it's now a volume. And four-treatment you're into four dimensions. And it gets harder to to think about or to visualize. But, the sort of 2D or 2 and 1/2 D version is fairly straightforward. So, this is our conversion expectation for A and a conversion expectation for B, the two different versions of the game. And in both cases, they're sort of centered or close to that 30%. Uh which I should have skewed on my And this is what we're actually trying to evaluate. We're We're trying to say, is A better than B? And A is better than B if essentially the area under the curve where our theta A is greater than theta B, you know, is is larger. That's essentially what we're

trying to evaluate, which is this thing here. So, we just look at the area where, you know, theta A is greater than theta B. We integrate under the curve. We compute the volume of that shape. And we say, is it, you know, greater on the right-hand side or on the left-hand side? And if it's greater on the right-hand side, then A is better. If it's greater on the left-hand side, B is better. And the volume gives us our degree of certainty. And it's a really cool way of saying, I'm 75% uh certain that A is better than B. It gives us a real number to work with, a probability uh to work with, which is why it's so useful. What prior might we use for our game? So, in this case, we have a prior which is, say, 30%. But, we need to figure out a a spread. How certain am I about the 30% conversion rate for for day one retention? Well, maybe I'm not very certain at all. But, what I can do is I can look at the history of the game. So, maybe I've run the game for a month. And I actually have data to say what sort of conversion rates uh what sort of retention rates was was I seeing. And here's an an actual game that's you know available now with a a conversion rate around about 30% for day one and that's the spread. So what we can do is we can just very simply fit a beta to that in

this case was a beta with an A of 42 and a B of 94. Really simple to do with you know Python and or. And then you feed that into your system. You start running your Bayesian analysis. You iterate with all the data that you see, you iterate with the coin tosses people do or do not convert and you observe the results and you can observe the results all the time and make a choice at any point in time. You look at this and nothing's really settled down. So we can see that one variant looks like it's doing better than the other. It was oscillating a bit at the start. This used the uniform prior so early oscillation is classic there. Um but we're probably saying at this point that we're 80% sure that the top variant is better. It's probably not a good point to stop. You really want to let this run for a a little bit longer. It's only been running for a few days. So here's some examples of real tests that I've run. Um this is sort of how we read out the result of a test. This was a a control and four variants or a four treatment test. And in this case the the mathematicians among you should be revolting because

there's no such thing as 100% certainty in any of these things but we round up from 99.99 to 100 so forgive us that at least. It just looks nicer on the dashboard. So what we're reading out here is the probability of any one of those treatments being the outright winner or the probability that they beat the control. In this case here's how that test ran for a while. So again it was a uniform prior in this case an uninformed prior should I say. So lots of noise at the start but it eventually settled down into a very you know stable result. All the other treatments were practically 0% and we were very certain that one of these treatments was by far and away the winner. So that became actionable very quickly. And here's how the conversion rate looked over time. So again at the start this is the actual you know is it 30% or is it 31% for for day one conversion. In this case it was measuring something else. But that credible interval that I talked about, we're actually tracking that over time as well. And that's a good thing to do cuz it gives you an idea of the separation of the different treatments and how confident you are. What you really want to see is a full separation of your 95% credible intervals to be really sure that these

are different conversion rates. But again, depends on how you're setting up the experiment. Here's what it looks like when it's not so successful or not so is a bad term. This is a a successful test, but the result wasn't a success. So that that the change we made didn't actually have the right impact that we wanted to see. So this is what it looked like. The control very very quickly drowned out the the treatment. And after a couple of observations, and you can see there are literally the spikiness of this graph sort of represents the the number of observations and it very quickly, you know, drowned out the treatment. So we we killed that test really quickly. And this was the observed uh conversion rate uh of the posterior. So again, you can see that initially we've so little data that the probability distributions of both the control and the variant are overlapping. You know, our beta distributions are essentially all over are overlapping each other. And you can see here this is just a a visualization of the shape of the beta distributions that are tightening over time as we become more and more certain about the true conversion rate of each of our population groups.

Now we're making some big assumptions here. First off that users are independent. This isn't such a great assumption to be making for example in multiplayer games. If you're running a test where, you know, there can be an interaction between users, then you have to be very careful about the design of your test. It also seems that users convert very quickly. In fact, the model assumes users convert instantaneously. So that's not great when say you're looking at a a day 30 retention test. So you have to sort of frame that slightly differently, but I don't have time to to get into that here. Another thing we assume is that probability the probability of conversion is independent of time. That's also untrue. You know, sometimes people might purchase things more frequently during lunchtime and less likely to purchase, you know, at 4:00 in the afternoon. So, there is a sort of a a relationship between time and conversion rate. So, you have to make sure you counter for that in some of that the modeling. Really, a lot of the complexity of Bayesian, you know, AB testing really comes in the later stage stuff. The implementation of an a Bayesian AB test is really straightforward.

This is what it looks like if you try to run an AB test without any corrections on a conversion that happens long time into the game. Let's say, you know, after a 7 or 8 days of play. Uh in this case, actually, it's not. This is uh probably about 3 or 4 hours into play now that I'm looking at the data. But, you can see that the conversion rate hasn't settled down. It's sort of starting to become asymptotic to something. So, when you're running tests and if you see something like this in your own graphs, wait until it flattens out so that you can see, all right, my population has actually reached a stable conversion rate and I have a decent uh estimator. So, some benefits and features of Bayesian. It's continuously observable and this is genius. Unlike the null hypothesis stuff where you have to set up the test in advance and being fairly rigorous about that, Bayesian just look at it and you get a new probability every day, hour, minute, even second depending on how you implement the the infrastructure. So, there's no need to think about population sizes in advance. You just let your users flow over your test and you look at the results and you build up whatever new piece of data that you have a new model that represents what we believe about the the game.

We can incorporate prior knowledge using the the prior, which is really cool. The ability to bring in your own business knowledge or your own experience from previous tests or experience about what you know about games and bring that to bear and and sort of factor that into the mathematical you know, implementation is really cool. And the result at any point in time is a true probability. It's a value from zero to one that you can sort of intuit about and doesn't have any strange special casing about null hypotheses and false positives or well, clearly kind of false positives, but it doesn't have any of that strangeness of a P value. And as well as that, it gives you a magnitude of the the difference between population groups, the magnitude of the effect you're seeing. Again, with null hypothesis, all you get to know is is it or is it not likely to be due to that distribution. So, it's just a binary choice, whereas in Bayesian, you get a magnitude and the probability, and that all makes for a much better assessment of the results of the test. And frankly, it's a really great framework for building lots of different analyses inside your game. So, you know, get on to your your analysts, get on to

your data scientists in your companies, and get them to build a Bayesian framework because you can start to you know, look at lots of other different things and not just simply AB tests. So, some useful links. That's just for those of you who want to download the paper or the the the the the talk later. I'd particularly recommend that first link. If you haven't heard about uh Cam Davidson's uh the probabilistic programming uh thing in Python. It's it's a book written in IPython, so it's fully interactive. It's the most genius thing I've come across in the last year. So, if you just Google for Bayesian hacking, I think it is, or Bayesian hackers, you'll come across it. Absolutely brilliant book. Uh can't recommend it enough. It's not finished, so uh it there's a lot of errata to come, but it's it's great. Uh other than that, I'd just like to thank you for your time, and uh ask if there's any questions. Thank you. So, we have one question there. Uh we

might use the Actually, there's one over there at the microphone, so I'm going to go there first, right? Oh, sorry. You want the previous slide. Yes. Uh hi, thanks. So, um one thing that uh I think most people who've worked with MCMC know is that it is quite computationally expensive, and I'm wondering whether you have successfully scaled some of these approaches to tens of thousands, hundreds of thousands, millions of users, or whether you do some kind of sampling, or use closed-form kind of solutions, or how how it is that you run this at scale? Yeah, good good question. In in fact, uh if you're re-running this or you're computing the the results every new user or every new observation, clearly that's an issue. So, we batch that up for a start. So, we look at series of observations and then we sort of re-compute at regular intervals. To be honest, most of the tests we run are actually straightforward. We don't even have to resort to the MCMC stuff, and we can use this analytic closed-form solutions that exist out there for computing uh the area under a

multi-dimensional uh sort of beta functions. So, there's a there's a very good Java library whose name I sort of forget at the moment, but if you ping me afterwards by email, I'll send it to you. And that's a a great library that we use internally, you know, for computing these integrals. Okay. I think it's you. So, I'm not trying to um defend frequentist statistics, but one nice thing about the Neyman-Pearson approach is that it at least gives you a way to think about inferential risk for type one, type two error with respect to sample size. And I'm wondering in in your setting, are the sample sizes typically so big that you don't really have to to worry about it too much so that the data just flows in, and at some point your credible intervals are obvious, and then you're done? Yeah, I I think you're you're absolutely right. Um I I'm not sure this is probably an untrue statement, but um we haven't needed to really think about that too much because sample sizes are definitely in the hundreds of thousands of users and beyond, in which case you know, you

don't really have to think too too much about this, which is the the good news. Certainly, if you were doing a test which had, you know, only 50 users, then you know, you'd obviously have to be very careful, you know, you'd have to be careful about degrees of freedom and all that sort of stuff, but generally, by switching to a Bayesian approach and by just relying on the fact that there's lots of observations, we haven't had to get too deep into that stuff. But I think that you're absolutely right. There's a I think there's a a deeper set of knowledge on the sort of frequentist approach out there. The experimental design research in in sort of frequentist is rich and vast. Uh what I sort of like about Bayesian is just simple to be honest from my perspective. So, I was able to understand a little bit better. Cheers. Thank you. First, thank you. It was a very concise talk on Bayesianism and as I'm sure you know, there's always a really big divide against frequentism and and Bayesianism especially in a situation I have been dealing with lots of basically stats 101 or someone who has an undergrad with math that really, you know, yell at sample size or bring up these things about Bayesianism. And what kind of experiences have you had

working with companies or clients of saying, "Hey, this stuff really does work." or what advice do you have for basically converting people to the benefits of Bayesianism? That I can only go really from our own experience because largely when we talk with companies, we divorce them from the complexities of how we implement this. So, we in some cases we'll talk directly to their data scientists and and and get into it. But in my experience, getting to grips with and creating a very robust sort of frequentist approach to this, we found quite tricky. And we went a good deal down the road of doing that, you know, looking at correction factors and done it and all those sort of things. Um but eventually decided to make the switch based on some advice that that we got a couple of years ago. And since then things have just gone so much easier. I think horses for courses. I think in the high-tech in the computer world which I come from, computer science world, I think there's a natural affinity to Bayesian because it you know, it it

unlocks itself once you have the capability of running these calculations at scale where you can divorce yourself from having to figure out really complex, you know, conjugate priors and can just rely on what are also tricky things that MCMC methods and things like that. It just sort of fits our model view of the world I think a lot better. So, you know, I wouldn't personally get into the the The, know, the A versus B comparison of those two schools, we have just found Bayesian that's it suits our our team better. That's all. Thank you. Yeah. Uh if you start taking advantage of uh some of the more complex methods you touched on like uh doing real values instead of discrete. Um so let's say you're optimizing for revenue, you're looking at ARPU or ARPDAU, and you also want to do multivariate testing. So you're kind of pushing it to um the most of what you talked on. Do any new uh challenges emerge there? Uh tractability of the computation or um needing to switch to different methods? Um

I I guess so far, maybe we haven't gone far enough into this to to hit up against some of those tractability issues. We found the majority of the use cases that we've come across can be actually nearly trivially solved uh with Bayesian. Now, you do have to implement hierarchical methods, you have to get into things that are beyond binomial and negative binomial and Brownian and all that sort of stuff. But fundamentally, once you choose the right model and once you choose the, you know, the the right Bayesian approach, it all sort of falls out. We we haven't come across an issue for us in terms of computational tractability yet. Now, I think the direction that we find ourselves going in is away from the sort of more curated AB test approach to the more automated optimization and multi-armed bandits and starting to think more on sort of that direction. And I think that's very interesting where you can just sort of automate across a, you know, a parameter space of different options, but and and that's that's probably where we're going to spend most of our time. Thank you. Cheers. Um in terms of a a company looking to buy a system like this, can you talk about

like the return on investment? Like say a company wasn't doing this sort of approach and then a year later was adopting it, what sort of effect would you expect to see in their revenue and business model? Uh that's entirely a function of to what degree testing becomes incorporated into the culture of the company. So, I think, you know, there's no again, there's no silver bullet. A piece of technology or a cool platform doesn't actually deliver the ROI in and of itself. It sort of has to nearly infiltrate itself into the the way you develop the game and the way you develop the economy and the way you manage the store or whatever. And, you know, I think that's still a an ongoing story. Like, I I can't tell you any specific examples of customers working with us because we sort of measure ROI nearly on a on a on a AB test by AB test basis. But, each of those has value, you know, and it an individual test could be worth 100 bucks, could be worth 10,000 bucks, depending on what needle you you shift. You know, I I think some companies have defined their success on the basis of

always being testing. Uh and so, I think it's there's huge value to embedding that in your culture. There's obviously a there's a a spectrum. You don't want it all just to be data-driven and testing. Uh there's still also about design and and intuition. Okay, so a different question. You You talked a little bit about how do you tell if a change is good or not good? Um you know, a game is you got a huge surface area of things you could change. Yeah. Um if you change everything every day, it's a mess. Yeah, you can't do that. Yeah. so really an interesting question is is like, I have a whole bunch of ideas that my game designers want to do. Which one is the one that I should try first? That's sort of a a very interesting one. It's not quite a philosophical question, but really a kind of optimization problem. You know, if if in the limit you could express every parameter of the game to an AB or an optimization system, a whole point you you hit hit a sort of a diminishing law of return that, you know, I just get too complex and you're changing too many things. I have no idea the answer to that question. Our advice to any company working with

AB testing, certainly to begin with, is keep it simple. You know, talk to your your your CFO, talk to your the guy who owns the P&L, and ask him, you know, what do you want? Uh what's the business goal? and then work back from that into what things inside the game can impact on that business goal most quickly. And generally speaking, that's something to do with, you know, the economy or the monetization mechanisms inside the game, and things that directly impact on early stage sort of engagement and retention. And that's usually where we start out. And then over time, you start building more and more tests. Uh, and I think we're still early, actually, certainly for mobile AB testing and this sort of technique in general. You know, in the limit, you think about an auto-adapting game based on what users are doing, you know, some sort of crazy machine learning system that adapts itself. But, uh, I think that's that's a limit we'll never want to reach, I suspect. Okay, listen, thanks very much for your time, and enjoy your beers.