The Attention Budget Rule: why tutorials should teach maximum 3 things at a time

You want the curve to have some dynamism to it, right? You want cuz it's not You don't want it to be like maxing it the entire time. You don't want the game to be just nothing but explosions or something so the players like max segment the entire
Why? Why?
Okay, [laughter]
come on. Cuz you over the player.
It's rolling [music] with those coins.
We're the two and a half gamers. [music] The midnight crew talking UA adverts and game design too. Mat feel shaku bringing the insight. We're rocking [music] those vibes till the early daylight master eyes on the prize. Tracking data through the cyerspace skies. Felix stacks colors like a wizard in disguise. Jack [music] craft realms lift us to the high. Two and a half gamers talking smack. Slow hockey sick got your back. Ads are beautiful. They light the way. Click it fast. Don't delay. Uh-huh. [music] Uh-huh. Uh-huh. Oh.

Uh-huh. [music] Uh-huh. And that's it. Hello everybody. Welcome to our ultra rare special episode.
Super rare. Okay. Super rare episode because it's uh it's a full house with five people on the podcast. So, you know, it's going to be very interesting. My name is Matthew Lanch.
I'm Felix Broberg.
And I'm
And we are your hosts. And we have two special guests, Katie and John. Hello. Welcome to the podcast. Can you introduce yourselves and the company you work with at the moment? Ladies first.
Hi everyone. I'm Katie Madding. I'm CEO of Enhanced AI. Prior to that, I was at Ajest for over a decade. Uh CPO there through the Apple 11 acquisition. So been in this ecosystem for a while now um and now loving still being in the

gaming industry with Mhance. I'll hand it over to John.
Hi. So my name is John Hopson. I have been a game engine researcher for over 20 years since the original Xbox. Most notably I worked on Halo, Destiny, World of Warcraft. Let's see. I' I'm now forgetting names now for all the games, but I most notably founded the research teams at Bungie and Blizzard. So, and I'm very excited to be here at Mhance making uh new technology for testing games.
Nice. So, for for today podcast, we we're kind of brainstorming like what to do and how to kind of talk about the Mhance AI. By the way, if I may ask, did you have like a brand rebranding
recently? Okay.
Because our prior name was pretty rough. [laughter]
Okay. Okay. Okay. Um, so it used to be sin smitter if anyone is confused and still doing a play on words, but now it's enhanced but with an because we focus on emotion intelligence. But we're pretty we're pretty excited about it.

Our new brand's pretty snazzy.
Yes. Very very very cool. And for today, so we are doing um a play test and creative report. So when we're brainstorming the like the kind of topic and how to talk about different things, I thought why not to actually take a look at one one game that I work with and I I think it would be very beneficial for everybody to look at how you kind of do the play testing, how you do also reporting of creatives. We do a little bit of other case study at the end and we obviously going to ask questions about the reports and the methodology and everything. So John, I think if you can, we would love to see with with our listeners that uh and obviously subscribers on YouTube how the report looks like and what you did pretty much for school girls.
So uh to get into to start from a little bit about sort of how we do research. So we're taking a group of participants, we're having them play the game while have being recorded in their face and their gameplay. We're feeding the facial

analysis and the gameplay recordings into our AI to identify segments of gameplay and to identify the player reactions to what's going on. We feed them into our AI to identify loca where players are reacting like where they're paying attention, where they're frustrated, where they're excited, and then combining that all into sort of a 360 picture of the player experience and boiling down hours and hours of gameplay into just the moments that matter to the players and therefore the moments that should matter to us as game developers. To as people play, we're recording their face. We can see all sorts of we have all sorts of graphs of their re facial reaction and their attention. In particular, our methodology has a high precision. So an RO C A of 89. Now I know that's a big number, but I will let Katie explain what that is because I know
W R O C I have no idea what you just said precisely.
You're speaking alphabet letters.

Okay. What is that?
Simply put, it's kind of like a lie detector test or if you've heard of R squared, a similar thing, but basically for a binary output. And so what we're trying to do is say is a user engaged or not. And that's very simply what we're trying to do. And so basically this number is let's just say out of 100 different players means we get it right 89 times, right? And so if you were to flip a coin, that would be for reference 0.5. You're just getting lucky, right? Like your model doesn't actually know. But for us, having this high of a score is actually the academic um determinance of excellence, right? So it basically means for us, our model genuinely can tell if a user is engaged or zoned out. Basically, they're bored, we've lost them.
So like statistical significance basically.
Exactly. It's it's the version of that for us saying are they engaged or not.

Why can't you why can't you say that like [laughter]
evidently we need to update our marketing slides
for the people who are nerdy about this stuff. They really want to know
exactly what method we use to calculate it. So we have it in there I'm looking at sometimes man.
Sure of course because Mr. PhD of psychology yeah of course but you're not the only person with PhD psychology right? This is true. I did get his PhD in psychology before I turned to evil and used it for making video games. So, so we're recording what's going on. We're looking at moments of high and low engagement. And engagement to us is the combination of they care about what's going on, right? They're making a facial expression. They're smiling. They're frowning. They're angry. They're happy. I'm less worried about what their particular emotion is than that they care because if they care, then I care. We're also pay attention to whether or not they are paying attention. So, are they concentrating? Are they blinking a

lot and looking away? Or are they focused on the screen? Because we want that combination of they care about what's going on and they're paying attention to it. I don't care if they're happy because their per their friend off screen is making a joke to them. Like, that's not important. What I care about is are they paying attention to what's going on in the game? And does their emotion say something about what's that moment? What if you have like a very professional poker player and with zero emotions but very high concentration?
So we are normalizing everything for each individual. So some people are super expression expressive our classic Americans we find that actually when we run Slavic guys through here like Eastern European guys they are just very serious. They're not they're not going to let me have anything.
But we're normalizing it. So what might be a big smile for me they would never do but they they'll do a little twitch and we'll normalize it so we recognize that for them that was a big expression.
Okay that's very nice.
As we go through let me Yes. Okay. So

for this particular study we looked at the first 15 minutes of Skull Girls that that initial first-time user experience that is just absolutely critical to get to the rest of the game. Like you can build a 10-hour game, a 100 hour game, a permanent hobby game, but if people don't make it through the first initial experience, they'll never get there and see all the other wonderful stuff that the developers have built. So we were focusing on just the first 60 minutes. Uh we brought in seven people, so small, relatively small sample and in the sort of 25 to 45 both men. Yes. Okay. So to talk about a couple of the key things that we found, the core game of course did great. Like we all knew this like Skull Girls is a wonderful game. We know that this is something that this game is a has a lot going for it. Has a wonderful art style, has great music, sound design, character design, wonderful stuff going on. However, and people had a great experience once they got into it, but that inertial experience sort of got into the way. In the initial experience, people did a lot of button mashing and had a lot of tutorial messages that sort of slowed

down their initial sort of onboarding to the game. And so, they didn't quite get into the groove as soon as we would like if if this was my game. What's really killing us in this case is that no one reads text, but we were stopping the combat to show text. So, the people are getting into their fight. They're bashing back and forth. There's great animations. There's sound design. There's things going on that are really exciting. And then we pause and tell them about an optional blocking me mechanism that they don't even need cuz it's an early easy fight. And so, players get frustrated having their wonderful uh violence interfered with by education. Wait, I said that wrong, didn't I? Okay, so in that same vein, like the tutorial taught a lot of things. Like it taught blocking, it taught throws, it taught dashing, it taught a lot of mechanics that are good things. Like fighting games are very deep games. There's a lot of like rock paper scissors relationships going on. There's lots of things that you need to know to figure out the timing and what works better than what. But this game was trying to teach all of that upfront

so early that players didn't know why they cared about it. and it was more than they needed to win those initial fights. So, it felt sort of useless. We wasted their time making them stop and read when that wasn't necessarily what they wanted.
Do you have any data that would kind of suggest how many mechanics or anything that uh the player learned during the tutorial? Like if it's 1 2 3 4 5 and if it's different between Asia and the West as well potentially. So it's less about the number of tuto of mechanics we're trying to teach and and how many we're presenting sort of at a time. So it's like there needs to be for everything we teach, we have to give them a moment where they're open to paying attention to what we're teaching and where they can immediately use it and see the impact. Because that's the thing. It's like just telling them, "Hey, by the way, you can do this combo to throw someone." That information doesn't have any value to them unless they can immediately do it. Throw someone and be

like, "Oh, okay. I get it. Cuz that's the real point of that teaching is to give them that moment of like, oh, I see why this matters. And so it's about spacing those out so that they learn those sequentially and have time to sort of digest it and sort of recognize it. But in general, like no more than three things would be my suggestion is like no more than no one's going to remember more than sort of three things at a time.
Yeah. We have this saying that's called attention budget basically that like you fill it up and then you need to wait until it's kind of you know you master what you learned and then it's empty again where you get bored and then you throw new stuff at people and then they again can learn and fill up the budget.
And I would also say like in the same vein
teaching something once isn't enough. It's like one of the things we learned on Halo whenever we popped up a tool tip to teach you something we would repeat it periodically until we were sure you'd gotten it. So, like we popped up a little message saying, "Hey, press this button to pick up a weapon and they did it." Okay, great. We then snooze that

message for a while and then say 5 minutes. If they haven't picked up a weapon in 5 minutes, okay, we start bring it up again. Okay, you do it again. Okay, now we snooze it for 15 minutes. And if you go for a time period without picking anything up, okay, sounds like you need another reminder. Because so many games teach something once and they expect you to remember it four hours later when it actually matters in that boss fight or whatever. and people's memories just don't work like that.
Yeah. I wanted to also bring up the the attention budget again as I said like west versus east that's quite quite the difference as we always look at the the games coming from the east
and per genre like especially if you're you're you're playing a paradox game. You're playing some super deep war game that requires you to remember 5 million stats like those guys have all the attention budget. They will happily like they're there for those crunchy numbers. This is a fighting game. That's not what people are playing this for. And so they have much lower budget for remembering uh a wall of text. Let's see. And I want to show you real quick what this looks

like for the player. Okay. So here's a player playing in here's a little bit of what the tutorial experience looks like.
Let's see if you can share the screen again.
Yeah. Let me see if I can share this and it's sort of visible. So this is a bit of our Okay. So, this is a bit of our sort of internal interface for sort of going through uh the play test results. So, look at these folks. Okay, we're going through the tutorial. We're fighting. We're fighting. We're fighting. Stop. Okay, now we're teaching something. Fighting. Fighting. Now, we're stopping to teach something. Fight. Where's my fight? No. Okay, now I get to fight again. Nope. Now I'm stopping to learn something. Fight. Fight. Fight. Stop to learn something. This isn't the experience people are here for. They're here to fight things. And of course, because this is a tutorial fight and we're interrupting, we're interrupting. Because this is a tutorial fight, it's an easy fight. So none of these extra special moves are actually necessary to win this fight. It's so it creates a situation where players feel simultaneously bombarded with a ton of

information and feeling like the game is too easy cuz they didn't actually need any of this to win these tutorial fights. Go through pop back to our presentation. So we're teaching a ton. It's all a little too early. As I said, the prompts just kill the tension. Any sort of momentum you're you're building up in the fight is like bang bam bam bam and pause to have somebody pop up and talk to you. And this wasn't a case where every tutorial prompt in the game was bad. It was just the ones that happened during combat. Like, as we'll talk about later, people loved the meta and the meta teaching worked great. It's just that those moments when they were in the middle of something, right? They were having a fight. If this fight's important, then you shouldn't be able to pause it to stop and teach me something. The next issue I want to bring up, and this is a very common one that we see a lot, is that at the end of a level, we saw a lot of passive experiences. So, uh, to show you some examples, actually, this this is a good example here.

So, go through again. We're having a fight. We're having a fight. This is great. Everything's super exciting. where see this nice big experience peak as things are doing great and we win this and we win. There we go. So, okay, now let's count them. Okay, so we show the winning animation. We show a victory screen. We show a reward screen. Takes time. We show a loading screen. And now we're finally back to Oh, and now we interrupted for a nice passive cut scene with the player talking at you. With the AI talking at you talking, talking. Okay. And now we're finally back to gameplay. And that's sort of the experience that players had throughout this where they're just like, you've just had this fight, you're all ramped up, you're winning, this is a victory moment, and now you're just sitting on your hands watching the game play itself. And all those things are fine by themselves, right? That like a victory screen is great. You have to celebrate the players wins. A reward screen is great. You have to you have to tell them what they won. and loading

screens are a little necessary and whatever, but we tend to stack those up at the end of a of a level or a mission or whatever and it creates a combined experience that is not what any of us intended. Like nobody on the team sat down and said, "What we really want to do is these five boring things back to back. It just that's just the way things come together." And this is why sort of real player play testing is important is that their reaction to things is going to be different than ours. And when all these things come together in the way they do in the real game, they have an impact on the player that is much more profound than any of those individual elements would have been by themselves. Like any of those things by themselves would have been fine, but showing five of them back to back is kind of boring.
What are what were the like high and low engagement moments in in this? Can you can you kind of just summarize and then maybe pinpoint the those high engagement moments afterwards as well because I

know like the combos and specials and all of the all of the the skill trees and stuff which is definitely there and it's really important for for the players. That's it also amazing. So, if we had to sort of sum it up, uh, some of the things that ended up being sort of super exciting, the combos and special moves were one of the highest engagement things cuz like they're big flashy animations. They tend to happen at key moments by definition because they have big impacts on the fights. The art style was super popular. The team management and all the metagame stuff was super popular. Choosing your meta relic. The story dialogue was great in sort of small doses. Big dumps of story dialogue were not that popular, but in small doses, the voice of the game really came through. And a lot of those early game choices were very uh powerful. Things that tended to be the low engagement moments we saw pretty much universally uh overload in the tutorial during the

first few minutes. Like all that information we were dumping on people, the infight combat pumps we've mentioned, the end of text, end of level passive chains of experience. Our text heavy loading screens, the button mashing was everyone realized that button mashing is bad. Even the people who love button mashing realize it's bad. So they know that they're not supposed to be able to win this way. And so it it doesn't really feel good even to them as they win. And finally, that initial dialogue dump was also a pretty much a a pretty roadblock up front. So that's sort of the quick overview of the highle and lowle. Let me zip back here. So, we had our those change of past experiences that were a problem. Those early fights taught a lot of bad lessons because they're so easy and you can button mash through them. So, a lot of people came away from that going like, "Oh, this is not that deep a game." Which is not the lesson we want to teach at all. But that's sort of our quick overview of top issues. Is there any of those you would like me to deep dive on

more?
I have a I have a question on the curve. Basically [snorts] the slide there that that was done there as you said
for instance the victory screen and a few loading screens and stuff on top of it cause like this kind of low engagement and like low say we would usually call it churn in gaming sense like it it causes usually churn like this is boring and people leave. So would that mean that in order to get some let's say like a proper curve would mean that it should never drop to some point and you should kind of aim for like like this kind of a stable curve that would be like a good tutorial in this sense.
So you want the curve to have some dynamism to it, right? You want cuz it's not you don't want to be like maxing it the entire time. You don't want the game to be just nothing but explosions or something so the players are like maximum excitement the entire
Why? Why?
Okay, [laughter]
come on. Cuz you over burned the player.
Yeah.
Burns out.
Other than you, most people don't want to just watch explosions. But you have to have some dynamism. You have to have

some high moments and some low moments and give people a chance to sort of relax in between those high and low moments. So having a little downtime at the end of a fight makes sense. That's a good thing. It's just that having like five moments of downtime chained up at the end of that fight and having an extended period of low reaction is a bad thing.
And maybe if we want to pull up some of the slides, we can have it while we're talking. And I think
uh that one. Yeah.
Is this the one? Yaka.
Yeah,
I'm sharing I'm sharing. But this is the one I think so. Yeah. Yeah.
So I think another really good example here is like that neutral middleground area that you see here. [clears throat] So this kind of neutral state, it's a good place to be and you're not bored, right? Like you're still active within the game. But when you drop below that to the passive state is then where we're saying you are now disengaged, right? It's been one too many additional screens. And I think another key difference here is you mentioned churn, right? Which is something obviously on

the live ops analytics side you're going to be tracking. I think so often, especially in these early tutorials, the decision from a lot of that analytic side of the houses, we need to make it easier. It's too tough or we entered in, you know, something that this level was just too difficult too early on. When in often we actually see that these are peak engaging moments, right? Like the users are really, really dialed in. They have heavy concentration. They're emotionally engaged. But the issue just goes back to what John's been pointing at, which is that you have to train and then immediately give them the agency to exercise those new things they've learned, right? And so a lot of times there can be this kind of big difference of, oh, we need to make it easier. We need to make it simpler. That way they can actually get through to level three. But in actuality, you're then causing the users to kind of be like the players to say, this is kind of a boring game. Want to do that. You can make them difficult and challenging at the get- go. And that might even be the hook that keeps them coming back. But you just

need to make sure that all of those different mechanics are explained and then give the player immediately that agency to use them. And you don't want to overload them with a million things and then not let them actually try them out. Does that make sense? It's like so basically the curve should look like the flow state curve, you know, like everybody kind of go goes after pretty much like not too difficult, not too easy, very hard to get precise on the sweet spot.
Or or even just sort of having some going back and forth
oscillation. Yeah,
flow is a big word and a very like has a very specific meaning, so I'm not going to promise I can detect flow. But we've done some puzzle games where they have this really interesting pattern of reaction where like they're intensely focused while they try to figure out their sequence of moves. And then while they're executing those moves, they relax. You can watch their attention drift as they execute like those four moves. And then they focus really hard and pick out their next pattern of moves. And like that's a great experience for them, right? They're getting both. They're getting some relaxation time, but they're also

getting some intense concentration time. And sort of that trap pulsing back and forth. I think is one of the key aspects of flow is that you're not just perfectly in the in the same mood the whole time, but you're sort of moving back and forth inside that band.
And how many users do you actually need to watch to get a statistically significant like pool of users to actually come to these conclusions?
So, if we're going to statistical significance, we need to get up into the like 25 30 user range. And that's sort of tricky and takes a while. In this study, we did a smaller sample of like seven people. And this is more in the usability scale of like it's not 53% of users had this problem. It's some users had this problem. Here's why. Here's something to think about. And so this is a fast light version of the study to get people sort of okay, here's what matters, but we're not going to give you exact sort of benchmark data

versus on the creative side of the house. That's where we're getting a 100 players plus to make sure that we can have that statistical significance because it's a very different you're watching 30 seconds of the exact same content kind of deal.
I mean you did this mini case study.
I mean obviously this is interesting because we were always talking about
Yeah. Yeah. A lot of a lot of match three stuff. So
can you walk us through this because you know we want to know we seen this creative millions of times even today. So looking at this case,
so these are obuscated versions of the creatives, but they are very similar to what we saw. I now want to I want to play sheep swipe now though.
That actually looks like something that might be, but so these are offuscated, but they're very similar to the original visuals. And three of these were from a competitor and three of them were our clients versions. And so we were trying to understand why when these are all very similar and this is a well-known

sort of genre of ad why are some of them working and some of them not working when I as a normal human being really they all bund together and thankfully our method was able to notice some moments where the performance was different and was like okay this sort of sort of tension worked in that ad but not this one and you can start to build up a pattern of what's working and what's not. I mean here.
Yep.
Okay.
See, so yeah. So in the I can't get over the names. Okay. So in a cat match 2, we had a very sort of flat initial experience. Whereas when you look at the sheep swipe one, we saw much more sort of dynamic experience where there was sort of moment of tension, moment of knot, and it we had some some texture to the experience and not just sort of this flat, okay, it all feels the same the whole way through. So having some tension up front made a big difference. be able to show some initial like and you don't want to like kill the sheep in the first four seconds, but you need to at least sort of threaten the sheep.

Yeah,
that that's good.
Yeah, that's what our Chinese friends always tells us. It needs to have near-death experience experience all the time. All the time.
No, not all the time. You need you need it up front definitely to get that like initial hook and get people paying attention. But then you need to give them a little relaxation, then put them in tip in stress again and back and forth. This goes back to what uh Jakob was talking about where it's like
okay,
if you just show like one maxed experience the whole time, it's probably not going to stick. It's going to all just feel like one giant roar of noise. But we we want to be like, okay, here's a moment of stress. Here's a moment of relaxation, stress, relaxation,
and make it sort of give the player some some texture to work with. I mean, we talk about this also in in Forex and where you have the gates mechanics and you need to make that decision very quickly. It's also tension between like do you make the good decision or bad decision and then you die pretty much

because you make a bit
Yeah, the the failure moments are interesting that that they work so well at getting people especially going into that call of action because the player's like I I could I could have done that better. I could have fixed that. And so they have that again they have that moment of like they could see themselves playing the game and see themselves succeeding and that leads to that gives them some momentum going into clicking into the app store into the download instead of just being like one of our problem some of our problem creatives had popped up the call to action during a moment that wasn't very high engagement and therefore it just kind of sort of eh whatever next ad keep
okay so what what would be then the like for instance here in the recommended changes output it would be like a better CTR or M you know the metrics which one would be
I have no idea what you're trying to ask
like a pretty much making a creative better what would you aim for like a like the difference between like this one is again the one that you I remember you had that one where you know spinning blades without it and then spinning blades with the threats you remember

oh yeah because there was the the needed experience and it was the needed experience really
so so the difference was what better oras lower CPI that was the overall
it was better os and lower CPI pretty much because there was there was near near dead experience in one but there wasn't in the second iteration there was a guy who was actually chasing me as as a main hero but there was even like bigger tension pretty much during the whole creative and that was way better in terms of the the performance which is pretty much kind of here but it's really interesting that like this let's say here like that
oscillation
that's chart yeah the sheep swiped the chart right there.
But isn't that Isn't that the thing where the king pretty much like escapes one danger and then moves to another danger? I think that that's that's that's the one where where it's just not that one screen. It's just like I'm going to die and like you know. So, but he he changes like every four or five seconds between the screens.
Don't try to guess what to what the original creatives were. I can't tell you like we were sworn to see.

I know. I know. I'm not trying to say I just see too much of them.
Yeah, we see too much of them. And I will say one thing I will say about the the recommendations that you saw there. I mean we make recommendations. People want them. I am not nearly the expert on these kind of ads as you all are. And so most of our suggestions are there to get the conversations going, right? They're there to be like to get to give them so we're not just showing up with a bunch of problems like this part sucked. It's like, "Okay, here's something you could do." And they the best reaction that the client can give me is, "Yeah, no, no, that's not going to work, but we could do this other thing." And the conversation starts rolling and they they and I want them to shoot down my suggestions cuz they're better at this than I am, but I want them focused on it and sort of engaging with the data like that. So, the recommendations are just sort of a gateway to the conversation and not supposed to be the magic answer
for me. Like I mean the recommended changes are basically like operationalized what I said like our Chinese friends told us which which they learned throughout the years you know

like no research no nothing you just like you throw thousands of creators and see which sticks and then they pretty much build this kind of a knowledge of intuition of like this is what works and we see it all the way on the Chinese creative knowhow in the forex market which is exactly that like immediate thread and experience pretty much tension all the way then peaks and stuff around that
you can see it also in here and it's
Yeah. Yeah, of course.
From other Turkish friends pretty much [laughter]
also from our Turkish friends.
Same.
Definitely works.
And but
and the and the beauty of it, it works for every genre. Like you don't really need even to be in the same genre as these guys because for instance, everybody is now using like some kind of variation of Kingshot creatives because that that that's how it works basically as for the current market. But yeah, those are very very good like I would say operationalized findings basically. One thing I would say is that even with the right recommendation and the right sort of those those good principles that have been tested, how you implement them matters. And so your version of putting

the king in peril early on uh sorry, putting the sheep in peril early on may work while mine doesn't. And there are subtle differences that so it's not it may not be the concept that's the problem, but the execution. Making games and making ads is hard. And it's easy to have sort of the right idea but not be able to quite push it through to reality.
Yeah. Because we always get this like, oh, we did AI ads, but then then we saw those AI ads. I was like, guys, like this is this sucks. I mean, this [laughter] is terrible. So, these are there are AI ads and AI ads. They're also like save the character or save whoever. And like seven different variations and six of them absolutely not going to work, but that one is going to work because there's all these small things here and there that you need to kind of know to make it work. And in that same vein, like that's one of the things I'm hoping we can do for our clients is when you look at performance data out in the world, you get a pretty binary signal, right? It's like this worked or it didn't. They clicked or they didn't. But you may have tried 15

cool things in that ad. And being able to say, use the emotional reactions to be able to say, "Okay, these four of those 15 things worked really well. Let's double down on those." instead of throwing out the whole concept because some parts let it down the larger uh creative.
And I think that that's always the like the tagging war that everybody does as well trying to figure out, okay, it was the blue coat for sure and that's [laughter] the the piece of evidence you have. But I think that that's where it's nice, especially when we look back is we can actually see those moments that truly are the hook moments, the engagement moments. So then you can build and repeat those different hooks in a thousand different ways.
So we have this we we talked about the playlist. Can we can we share the report in the show notes for for the listeners so they can they can check it out as well or they should reach out to you guys or both? Ideally both.
Absolutely.
That would be awesome. Anything else you want to you want to share with the the

audience or we didn't touch.
Does your data ever differ from surveys?
Oh yes.
Yes. So, it is absolutely true that sometimes people will say one thing in their survey. They'll say one thing out loud, but their behavior is different from what they did in the game. This is always true. My favorite version of this ever was on Destiny. We had a reviewer say the game, I' I've played this for 300 hours and there's nothing to do. I'm like, okay, dude.
I love those Steam reviews.
There's there's something wrong with you. But their experience is clearly what their takeaway from their experience isn't the same as what they did felt in the moment because they clearly in the moment were having fun and keep and want to keep playing. But when they summarized it later, it felt boring. And I think those differences aren't a case of which signal is right cuz both signals are true, right? What they're saying is what they feel. What they did while they were playing is real. But those differences are opportunities. Like it's like, okay, how did we fail to communicate there? Why did this person play for 300 hours and

still feel like there wasn't enough content? Like what was going on there? And those are the things that you can drill in on and in my experience lead to really interesting ideas and really interesting ways to make how do we how do we like square that circle? How do we get both of these things to work tend to drive a lot of really exciting creative decisions? Yeah, for me like this is always like a community management basically because
somebody plays for 300 hours and then it's kind of annoyed about it like Yeah, that's that's called the dedicated players.
Yeah. Like there was something to do.
It's always like oh what what what do you like and then it's like oh I don't like this but then still I don't like this offer but then still like buys 10 times. It's the same thing. Do you like art style? Like do you like cartoon or or realistic? I don't know but plays both and loves both or just yeah these type of things whatever
yeah that boils down to the fact that you shouldn't let your players design your game because they don't know what they want sometimes or most of the times

you want yeah I think to that note we have the other example of like the pimple popping where we for a client had this like pimple popping ad that was hideous
but like literally the engagement was just off performance. Yeah,
every time it happened and then in the surveys, right, it was like, "Ew, disgusting, gross. I hate it." And it makes sense, right? Because like you don't want to admit that you like popping pimples. No one's going to talk about that when they go out to dinner.
Yeah.
But I think that it goes a little bit to like, okay, well, clearly this is an engaging moment, but there's a little bit of social bias and like this is not an acceptable thing for me to be into. So that can happen as well.
Yeah. But I love it. No, like this the same thing with like you know nearly every match three game other than like we have the saving situation, we have freezing family situation where there's like a freezing family that you need to save. It's literally like every second match three game. Yeah, [laughter]

I even yeah emerge as well because yeah, oh my god, everybody's against it but then then people are like oh my god I need some giant companies in the market literally built on this creative that that's why they work
and ideally we want to understand like okay what's working about that so you can design something that works equally well but is less obviously derivative and embarrassing right it's like how do we get special there you go
what is the socially acceptable version of popping yeah Like I don't know.
Squeezing something.
Yes. Squeezing lemons. Okay. Nice. So, um All right. So, we'll share the report in the show notes. We will share your contact details as well. So, guys, please feel free to reach out to Katie and and John and us or meet you at MAU. The whole team is going to be there. We are going to be there, right? Or
Isabelle. Isabelle is there.
Okay. Perfect. Isabelle is there. We are there with Yakoup. So, please feel free to to meet meet with us and thank you very much for coming and see you next

time, guys. Right.
Thank you.
Cheers. Bye. Bye. Bye. Oh.