The Battle for Best User Rated Game on Metacritic | Design Club

[music]
Good time zone internet, hello and welcome to Design Club, the only game design show this week that pivoted our topic so that we can do this week's topic next week. And this topic, which we're going to say we would have done next week this week. I am your humble host Will, joined as always by my pal Jeff. Say hi to the people, Jeff. Hi to the people, Jeff. Hey everybody, I am Jeffrey Zac, and the studio director of Extra Credits and Extra History, and more importantly for this, long-time game designer. Been here to talk game design, game strategy, game culture with my buddy, Voice of the People Will, our Twitch overlord. Hello, hello. So, uh one of your hats, Jeff, uh that's going to be incredibly relevant for this conversation This one? No, not that one. Okay. Although, maybe eventually. Um uh is that you also are a stats nerd.

Uh A little bit. Yeah, just a little bit. Just just a scosh. Um you're a person who enjoys the intricacies of tracking data and seeing bigger patterns in the industry and uh the like. So, when I mentioned Mhm. that there is shenanigans afoot at Metacritic, Jeff went Let's say more. So, here we are. We're going to We're going to talk about um messing with Metacritic. We're going to talk about why that's weird and why I I suspect people are kind of at least partially engaged with it. Um and we're going to We're going to have a little talk about that. So, for for everybody who's never purchased a game on Steam, actually, I don't know. Jeff, how many of you By By show of hands, how many of you were unaware until I said it that

Metacritic scores are also shown on Steam? Cuz like they're they're not or can be shown on screen. They don't have to be, but um they're not generally viewed as as important as the uh as the like Steam review scores, right? But like if I go to Fallout 4 um and I go to view page and I scroll down. Yeah, if you go on the right-hand side of the Steam page and you scroll down, there is a box on some, not all, that list the Metacritic score. I can even tell you why it's not all, but keep going. Sure. Um and so that combined with Steam's user reviews is sort of one of the ways that people get influenced to buy games. Um yeah. Most people haven't paid attention, knew but didn't care. Yeah. Um Oh, social There we go. Social stream.
Yeah. Yeah, and I think for for some people it

doesn't cross your mind and for some people it's a big deal, right? And it really depends on how much you want the the numeric value of Metacritic versus the vibe check that is um Steam reviews. Steam reviews. Steam reviews are not are not numeric. They are up or down and you're getting a percentage, which is really a vibes indicator, especially because Steam reviews are incredibly volatile and subject to the whims of the community when it comes to things like political decisions by the developers or things outside their control like [clears throat] DDoS attacks or price hikes or disappointing updates or DLC announcements or whatever, that score is very much in flux and usually represents the current um sentiment of the community. Yes. That just gave I'm adding this to our notes. Um

What? Actually, I want to ask you and the community right now because we have we have good chats. We do. In everybody's opinion, what is the purpose of a game's review score? That's a really great question, Jeff. We actually didn't have that on our list and I actually that is what we should be starting with.
Yeah. Yeah, yeah, yeah, yeah, yeah. Okay. So so I have kind of a the well budding idea.
purpose of a something called a review score associated with a game, a book, a movie, a product, you know, on Amazon, whatever it is? General purpose from the name of what you would expect from it or and then we can get to the other things, too.
And this is going to be I think personal to some extent, right? Because I don't know that we're looking for an authoritative definition if we're looking for an authoritative
Oh, I got an authoritative definition. Right. But you have it. We're not asking them to magically [clears throat] figure out what it is, right? Um But even just mentally. Right. Okay. So

so this is this is good. Is this game worth my money? That's a good it's an indicator, right?
Not just game, like I'm thinking I'm even broadening it out.
right? Is is this purchase Right. Worth my money. Worth my money. Okay, cool.
I think that is a fantastic Mhm. actually working definition. Sure. Um I also like uh the cynical it's something to put in an advertisement. Yep.
Right? Uh we see it in movie trailers all the time, right? You see like The screen. Right, yeah. So-and-so gives this three and a half stars, right? Yeah. Um
[snorts]
Uh rough indicator of the overall quality. Um Oh yeah, do people like this game? That's a very Yeah, do people like it? Is it Well, and and that's one of the places where it gets a little sticky now, isn't it? We were just talking about how I don't like the art style for either League of Legends or World of

Warcraft. I don't. I don't like them. I wouldn't review them badly, but I don't like them, right? And so there's there's some amount of um fuzziness on that topic, right? Um A shorthand of the game's value for the price asked. Yep. Yep. Yep, yep, yep. Throw in one that we're going to talk about later, too. Yeah, very very low scores can also have their own appeal. Yeah, bad movies can be fun. I I think of all of the movies I watched in college, I reference Trolls 2 the most, which is a god-awful movie that I cannot recommend. But you can watch it if you wanted to. And that's right, yeah. Copper Chef's got a point. It should be vibes, but I can't trust any score nowadays. And we'll talk about that here in a little bit as well. But I get the general sense that it is a qualitative indicator.

So so so sort of the summation of what we're getting from chat are there's a qualitative indicator of um how likely the product is to fulfill its promise. Whatever it's promise is. You will enjoy this 2 and 1/2 hour movie. You will enjoy this 30-hour video game. You will enjoy eating at this restaurant. You will enjoy this new can of different-flavored Coke. Coke with green pepper flavor. You will enjoy it because it is well reviewed. Enjoy is an interesting term, too. Yeah. Because I want to move for a moment. Something I want to talk to you later, I'm going to bring up now. Sure. Which is reviews and this is going to help our Metacritic thing. Can we I'm going to divide them into two groupings because this is a little bit of a trick question. There are let's call them for the moment

critic reviews. Reviews done by people who let's just say have some training or expertise about the subject matter. Sure. And then they'll all those call them for the moment community reviews. Ones where anybody can go in and leave a review. Sure. And in my mind these are very different things. Okay. And necessarily so because for example, let's take Transformers movies. You can pick any of them. On a Metacritic score, I remember looking at this at one point and maybe a little old and outdated but it doesn't really matter. The average Metacritic score for a Transformer movie was I think a 32 of 100. Sure. Bad. Low.
But these were some of the best highest grossing movies of the year every year. Because from a critical standpoint, they were objectively bad movies. But audiences still enjoyed them.

Because what the critics were reviewing was different than what the audience was going for seeing. Right. It is very different. Mhm. If you If you sent a Michelin reviewer, someone who is uh experienced and has a lot of experience with reviewing fine dining and you ask them their opinion Mhm. of a fast food joint, right?
Yeah. Their um No, Anthony Bourdain famously liked In-N-Out. Sure. Sure, absolutely. But we can pick up we can pick like, you know, and when I say fast food chain, I don't mean the good one that you like, I mean the terrible one that you only go to when it's the only place open in your town and you're out way too late. You know the one I'm talking about and it's different for everyone and it's different for every place, but you know the one, right? And so its purpose will be different for the consumer versus the critic, right? In the In exactly that same way. Actually, I might even go further. Instead of Michelin star, say you take

somebody who's a wine reviewer. Sure. And then you take somebody like me who, because I don't have a great sense of smell, you know, I don't know if I can tell the difference between a 10 and a 30 and a 50-dollar bottle of wine. Sure. I'm happy. It's like it's good enough. Sure. Like that person reviewing the wine and I have radically different experiences interacting with a bottle of wine. Sure. And because they're trained professional at this and I'm not and have a slight impairment. Sure. And so, you know, while I'm like, "Oh, hey, this is great." They may be like, "No, it's not." And their training makes them like it less because they can appreciate the finer things. There's a whole range of reviewers. Mhm. And people who are trained were expecting to have different styles of review than a place where you can go and just tens of thousands of people can give it an up

or down vote. Right. The other one is trained reviewers know how to use a scale. On a 10-point scale
Yes. a trained reviewer can say, "No, this is a six." or this is a seven or this is a 6.5 or a three or a nine. And they they don't dole out the nines all the time because they know they need to be rating this on a scale and if 10 of the games they rated were all nines, that means either they were on a really weird streak of reviewing particularly good games or that they couldn't discern the difference, which kind of defeats the point of them being a professional Right. reviewer. Audiences come in and are most likely to give something like a one or two star or a nine or 10 star. Right. Audiences very rarely can define that, "Oh, this is a five or a six." Most people and most people who give reviews come in because they either really love something or really hate

something. So, they come in and they're like, "Yeah, this was great." Like, I've seen the way I review Audible titles. Like, if I enjoyed it, it's a whole bunch of five stars. If I didn't, it's not. Yeah. Like, I'm not coming in doing a professional review. Sure.
And so, by its very nature, professional reviews are far more granular and audience views aren't. There is a
that they snowball a way more than professional reviews do. And one of the things about that that's also very worth noting and will come up pretty strongly in this is that reviewers usually don't look at other reviews before they review the game. Right. They don't want to have an anchoring effect.
They don't have an anchoring effect. Whereas crowds um do. They just They do, right? If you
you're going on to the Steam game to review Fallout 4, Yeah. you're going to see the reviews as you're getting into the where I leave a review.

Yeah. Like, it's part of the It's part of your interface flow. Yeah. The Yes, you have to look at the current review score in order to leave a review. You're totally correct. And this isn't saying reviewers don't have bias. Nobody doesn't have bias, but theoretically a professional at at a thing is more able to recognize and adjust to their bias because that's their job.
Right. Where somebody coming in, you know, if I'm going to go rate a bottle of wine, my set of biases might be I look at it like, "Wow, that $50 bottle of wine doesn't seem like it's five times better than this $10 one." Right. My bi- And you know, there's a whole range of things in there. So, part of talking about reviews is acknowledging that A, you know, we aren't machines, but B, there are different people leaving reviews, and therefore, categorically, you kind of need to separate them into here's your professional reviewers or your trained reviewers for whatever the subject matter is. And you know, if you're a good wine reviewer, that

doesn't mean you know anything about games. Like your subject matter expertise is limited. But there's basically, theoretically, for trained reviewers and there's community reviews. And these end up with very different results. I also think that a big facet of this that's worth addressing is that they, um, because trained reviewers are usually, though not always, doing this professionally or semi-professionally, they tend to be more complete. Yeah. Right? Versus we'll take, um, we'll use a very simple example. We'll We'll look at the Steam reviews, right? What was the last, like, if you if you go into Steam reviews, you will find some that elucidate on their reasoning, right? This is why I don't like this game. These are the things that I found wanting in it. This is the, um, this is the, uh, these are the things that I love. These are the things I thought were awesome. And then you were you were for a while very likely to get crude ASCII art in some of the reviews, and you still might for some games, and

those those are not helpful, right? So, it it becomes a sentiment meter rather than a critique, rather than an analysis, right? It's still useful as a sentiment tracker.
Oh, yeah. And these are all useful. And this I wanted to kind of separate them out as we're getting into the conversation. Yeah. By By way, um from uh debatable awesome who said, "You know, it's a remarkable literary which reading other people's opinions shapes your own, especially subconsciously." We actually, back when I ran a studio named Eedar, which was a video game research studio did an industry study on this that really shocked a lot of people. And the really short version was this was right after Plants vs. Zombies came out. And we went to a college campus. And this is basically how our study went. Was we grabbed hundreds of college kids and did a quick pre-check to make sure they hadn't ever seen or played the game before.

And since it just came out, this wasn't as big a thing as if now. And Plants vs. Zombies is unarguably a pretty decent game. Yeah. Like it's it is a good solid game. But what we did was while they were waiting to take the waiting to come get paid money, I think it was $10 at the time to come play the game for a little while. Um while they were waiting for it, there was a little sheet with reviews of the game they were about to play. And we faked these. There were some that basically gave it a good score. Review scores, some that gave it a high review score, and some that gave it a low review score. And so actually the medium ones actually, I think was the control. It was what the game's actual was. Which was pretty high. And then we took the ones and like bumped it up and bumped it down. Sure. And what happened was the kids basically read that and then go got to play the game um for a little while. And then they got offered to basically get paid the money they'd been offered or keep a copy of the game instead.

All right, which was you know, it was and Plants vs. Zombies wasn't that expensive. So Yeah. the people who you know, and we put the baseline on, read the real review, played the game then. Sure. The people who read the higher review scores and the number's a little fuzzy in my mind cuz it's been over a decade, but I think we're 30% more likely than the people who read the low review score to keep a copy of the game instead of the $10. And so this was, remember, they read a review and then played the game. They got to play this game for half an hour by themselves. And so that one bit of information, whether the game was rated lower or high, made more than a 30% difference on the people who basically decided to pay real money for the game, i.e. they didn't get given $10, they got given a copy of the game versus not. That is a remarkable difference for one piece of information followed by you then get to go make your own opinion

Right.
playing this game by yourself.
Right. You are
Like that's a staggering amount of difference Yeah.
It wasn't like they just just decided after the review. They then got to go play it for half an hour. Yeah. So Anyway, sorry. Fascinating study but you were saying something.
Our preconceptions from reading reviews Yeah.
are strong. Absolutely. I'm actually kind of curious if I can um figure out a way to get my Steam DB add-on to hide all review scores. Mhm.
Kind of interested in that um for when I go uh looking for games cuz every week I run a review show on um a roguelike game uh called The Roguelike Review, it's on Tuesdays here on Twitch. And uh I'm always I'm always fascinated by that. Um Anyway, so throwing it in. We And one of the reasons review scores are worth talking about is because review scores influence people's behavior so strongly.
Yes. Even though we don't really realize

I mean people realize it they but they don't realize how strong this is. Right. But it's also a little bit different in different places and it's partially based on how much you trust the reviewer. Yes.
Which we can also get to.
Yes, absolutely. So, that's a review score from a critic, right? And singular critic and singular review score and we'll say that the crowd is a a reviewer, right? That gives us a sentiment analysis of some kind. Um and then enter Metacritic. Metacritic? Metacritic is an aggregator of reviews. It takes uh reviews from many professional reviewers and uh sort of aggregates them and gives a final score. So, for instance, uh Fallout Yeah, they have two scores. They are aggregated critic one and they are community voted one, which is similar to Steam ones. Correct. And the um the aggregated critic one is normally, though not always, the one that people uh ascribe value to or ascribe meaning

to. For instance, um if you if you're on Steam uh and you look at Fallout 4, it gets an 84 and that's the critic score uh for Fallout 4. Um Although interesting fascinating, different versions also get different scores and the Steam version links to the PlayStation 4 version, which has an 87 instead of 84, but that's a moot point. I was just confused for a moment. Interesting.
Yeah, very weird. Actually, that isn't uncommon and a lot of it is because despite the fact that it's the same game, there are minor differences. Like how well did they adapt to, for example, the PlayStation's scheme versus say your desktop one. And some have better frame rate on consoles than on PC. It's actually not uncommon at all for there to be a different review score based on what the platform is. Like there's a lot of games that scored say high 80s, but on their Wii version

were like mid 70s because the performance wasn't as good or the things they had to cut to make it run on the Wii impacted the quality of the game. Mhm. And both
So, that is actually that is a Metacritic thing is that they look at the um they'll give an overall, but they'll also give it on a per platform basis. Yep. And then uh worth noting that there are two different scales. The number out of 100 on Metacritic is the critic score uh and the number out of 10, usually with a fraction, uh is the uh user score. So, in this case, Fallout 4 for the PlayStation 4 has an 87 on the the meta score and a 7.0 on the user score. Um and you can you can, you know, do your you can add your own reviews and what have you on Metacritic. But, its function is largely to serve as this aggregator, right? This this place that condenses um many reviews into a number, right? Into

a singular number and That's its job. Yep.
Here is one number per four game. Yep. Yeah, what is my job? You aggregate data. Aw, right? Which is and that's useful, right? Because if you don't want to read all of the reviews from, I don't know, Mhm.
[clears throat]
PlayStation Universe and IGN and Digital Chumps and Gaming Nexus and Slant Magazine and Telegraph and Digital Spy and PSN Stores and GamesRadar Plus and everyone else who reviewed Fallout 4, you can just look at the number and get an idea. Now, By the way, well, that was a really interesting list you read and I'm going to get to that Uh-huh. in a moment, too. Cuz one of the things when I was running my game research studio is we, after doing some research, realized how much how important the review scores were and actually made our own internal version of Metacritic. And because of that, I know a lot about how Metacritic works because I think I designed a slightly better one, but I'm

biased. I am biased, but it was also, you know, it was internally used and it We know it was pretty good because we ran data models against it and looked at our results and we're like, "Oh, wow, that works well." Yeah. But I have deconstructed the Metacritic stuff and I also have great respect for them because it was interesting enough doing it just for games and they do it for games, movies, books, and a whole variety of other things. They basically cover media. Yes. Oh, music. Yep. But um let's talk a little bit about how they do it because part of that is what makes it interesting. So we're going to get to the user or the community stuff in a minute. What I'm going to start off with is and is the profess- let's go on professional critics. And Will, tell me if any of this is interesting and/or surprising. Sure. Because there's a lot more to it than you would think. Sure. Like the base

version would be like, "Oh, cool." You know, go slurp in a whole bunch of stuff from the internet and average it. Right. Like that would be the base thought on how you do it. Mhm. So thing number one, what sites do you take? Right. Metacritic is very specific on a per industry basis about what sites they consider to be professional critic sites. Because if you look on the internet right now, there are people on blogs leaving review scores. There's community scores on somewhere like Steam, which could theoretically count as a score, all the way up to like IGN and GamesRadar, but also over to things like um Reader's Digest or CNET or Playboy or like any number of other things. And so the first thing Metacritic did was on a per industry basis decide what critics are worth covering. Sure. And then they had to be consistent on that. Like if they decide

Reader's Digest is an acceptable place, that means they then have to collect every review Reader's Digest does. It isn't a one-off thing.
Correct.
we are now setting up the intake data from this one place indefinitely.
Because if you don't, right? Once you've decided that you're accepting data from a source, you must accept all data from that source because anytime you don't, it's a bias, right? It introduces um another factor in the review score, right? So you have to take all of the data that they're offering.
Mhm. Essentially. Yeah. So number one, you know, it's an aggregation of professional review sites, but A, it's curated in that they don't accept everybody and they have some criteria to decide and they never disclose exactly what that is. But you know, from that list we'll read you, all of those are sites you listen to you're like, oh yeah, those all sound like professional review sites. Yep. And you know, none of them are Bob's blog, you know, Bob's game video game

blog 123 Sure. kind of thing. So and they keep track of that. One of the other things they do and one of the reason that Steam only cover has some reviews is they have a minimum number of reviews that they have to get before they consider this to be something they can report on. Meaning that if there's a bucket, you know, or a slop bucket, what what do I mean? Um bargain bin listing of games. Yes. Uh actually, let me rephrase it. IGN cannot cover every game that comes out. Yes. Both it's too much and honestly, most of their readers wouldn't care Yes. about a whole bunch of games. So IGN only covers a certain amount of games or certain style of game. Yes. The same like Reader's Digest probably only covers a couple of games per year. Where, um, Wired covers more games.

Like, different groups review different amounts. Because of this, there are plenty of games that might only get two or three review scores from their entire list of all these places Metacritic tracks. So, until you pass X threshold, they won't publish a review because they're worried that if there's only two or three sites in there, they're going to be publishing something that is disproportionately different from what they would normally publish because they don't have a large enough sample size of data. Right. They recognize that their role as an aggregator is to take a, um, differing, uh, or to take a number of viewpoints. They don't necessarily have to differ, but to take a number of viewpoints and to report on the average of them or the aggregate of them, not to report that only one reviewer played Hello Kitty Island Adventure and do they think it's the best game ever and they've given up all other games ever and so Metacritic's going to give it a 100, right? They're they're trying to avoid that phenomena where Or maybe one of the sites that they've said, you know, is worthwhile is, you know, we love strategy games.

And so, they might only review strategy games and they might only review good strategy games. Which would mean if they happen to review, you know, something that goes in and they're biased towards liking them. Anyway, there's a whole bunch of reasons for this. And so, there is a minimum amount, um, They also have to do something, they have to transform scores. Because, okay, they're rated on a 100-point scale. Right. If IGN gives something an eight, is that an 80? Probably. Mhm. Like, for groups, they if they were doing a 10-point scale, it would absolutely be an eight. But if IGN can only either do an eight or a nine, Right.
you start having to ask questions. How about when you get to letter grades, Will? Yeah. Like what's a B-?
For example, is a C a 70? Right. Is a D a 60? Right. Because most people would think a D is an awful game, but a 60 review score isn't equivalently

as low Right. as a D would be for some things. Right. Especially because of the um there is a cultural bias that differs depending on where you are and how you took to schooling regarding letter grades specifically and their worth, right? So there's there's a whole bunch of stuff that has to go into transforming a letter grade into back into a number, right? Transforming a number into a letter grade makes sense. Doing it the other way, much more difficult. Right. And it's an interesting thing because for example, I've seen IGN give games in scores of like a 30 before. Sure. Like for games that were particularly bad. An F or even a D Right. Like if they give if it was a letter grade, IGN might have decided a 30 is like a D minus. Sure. But if you do a straight transform, that's more like a 60. So you

get Basically, if you're doing something like this, you have to figure out what your scale is for transforming letter grades into a 100 point scale and 10 point scale into 100 point scale and 5 point scale into 100 point scale in a way that is consistent across a whole bunch of other things. And so not that this is rocket science, but just calling out there's a lot more to the way you have to do the transformative math than people initially think of until you actually start thinking about how to make this system yourself because you want to do something like this. Correct. And the other thing is Metacritic weighs different review sites differently. Sure. Because if it puts a formula in, it might decide that is should Wired's review be considered as heavily in the scale as say IGN's? Or should Reader's Digest? Or like

And so there's also some sort of waiting that they do on some of the sites that they use. And all of this goes together and also knowing that every site doesn't review every game. Right. And so what Metacritic's job is is to give you that single score, but they also have to look at how many people review it, does it hit the minimum number, how they do the transforms, how they weight each reviewing group as to its authoritativeness, and all of this is what they traumatically transform in their back end into the one number, which is also a living number, because at any point more sites might come in and post a review or even change their review. And so all of this goes into the one score that Metacritic gives you, and I know this because I reverse engineered it to make our own score. Yeah. We just weighted a little differently. Yeah.
was still very close. Okay.
So Metacritic's great service is it can look at all of these sites and come back and say, "Cool, this game is an 82."

Yeah. Uh And that's and the work that goes into
Sorry, I I I thought a little bit about that.
And the work that goes into that and the confidence they have in their methodology is why we refer to that critic score as the Metacritic score, right? Because that is that is the their product. Yeah. Right? That is the thing that they are uh putting all of the work into and saying, "Yes, we have this thing. We have confidence in this number that we're going to publish, and the official Metacritic score for this game is an 84." And that is why they are the industry standard at least in video games for game review scores.
Right. But they also do user reviews. Right. Which is way less exciting. Yeah.
[clears throat]
But and user reviews, you also like there's a whole bunch of that you can't or shouldn't do for user reviews. You're not weighing the you're measuring the individual user as to their

authoritativeness. Yeah. And since the user's reviewing it on Metacritic, they don't have to do transforms between, you know, eight out of 10 or this is a B minus or whatever is on the scale. So, this is purely a numbers vote. Yeah. I don't know. If you get a thousand people to come in and vote, they take those thousand, they add up the numbers, they average them out, poof, there's the review score. Right. Uh What I don't know, like does Steam let you edit a review? I believe it does. Or it has at some point and I believe that it still does. Mhm. I think Metacritic does as well. I'm not sure. Like it's interesting. If you don't let people do that, you avoid things like weird review bombing and sentiment, but if you do, that lets people play more of the game and be like, "Oh, actually by the end this was pretty good. I'm happy." Yeah. Um Yeah. If Or not. Yeah. So, the ups and the downs again to all of that is it's Metacritic for the what

they measure is is only the critics' reviews. It is the meta meta review of the critics. Where the user score is whatever people want to vote and however many people decided they wanted to actually go vote on it. Yeah. Now, the truly fascinating thing is that um How do I put this? It feels at least to me that the user reviews on Metacritic inherit some amount of the um Uh what's the word I want? gravitas gravitas, yeah, of the of the critic reviews because those are user reviews on Metacritic. They're not user reviews anywhere, they're user reviews on Metacritic and because if Metacritic is trusted to be a place where reviews are aggregated, I feel like uh consumers value that user review

differently than say a Steam review score, even though they're functionally collected identically. Well, I also feel like the people who would go to Metacritic to review a game are more involved in the reviews, like they're more interested in that and it's an extra step as opposed to Steam where, you know, you open up your game from its launcher and you can be like, "Oh, I can slap a review on this right here." Yeah. You're adding the the extra step of difficulty discourages people from doing it idly. Right, by adding by adding just ever so small a barrier, you're greatly increasing the desire that someone needs to have in order to go and leave a review. Yeah, like for example, I bet that the Audible, excuse me, the audiobook reviews on Metacritic are probably better than the ones on Audible because on Audible when my book when my book finishes, a little thing pops up and I'm like, "Yeah, you know, five star, five star, five star, done." Or if I went and took that extra step,

I'd be thinking more about it. Right, the the product review that's um while it is temporally most adjacent to it, it's also not a thing you put a lot of effort into versus I have finished this audiobook and I now have thoughts I wish to share with the world. Well, part of it too is I know part of it is self-serving which defeats it a little bit. I know that what I rate highly, when Audible does its recommendations, it will rate things similar to that. Also true.
So, I don't have a big incentive to be like, yeah, I you know, it's either kind of I enjoyed it or I didn't.
Right. It's more of an up or down vote. Do I want more things like this or not?
Sure. And because I know that, I rate based on that system. You know, if it's Dungeon Crawler Carl, it's going all five stars.
Right. Or if it's a Guy Gavriel Kay, but like there's certain things or John Scalzi book, like it's just getting the stars slapped on so I get to know when the next thing comes out. Yep. Uh

Okay.
So, so that's kind of the Metacritic stuff. And because of that and what Will was talking about for rating, in addition to highly rated critic games, highly rated user games on Metacritic are often looked at a little differently than a highly rated game on say Steam or the PlayStation Store. Right. But,
[laughter]
sometimes things get weird. And it's worth pointing out not like this is sort of an unavoidable part of having an open review system. Yes. Um which is that some weirdness is a foot at Metacritic and is the this this weirdness is actually the genesis of our conversation. So,
Indeed. last year Claire Obscure Expedition 33 came out and it to universal acclaim. Loved by critics, loved by players, loved by um award panels, right? So many awards, all

the places, all the different award shows, record-breaking numbers, loved, streamed, everything. Uh and it in fact set a new high target on Metacritic for um for the user score, which is a 9.5. Previously, the highest had been a 9.3. Wow. That's really hard cuz there's always somebody who wants to come in and zero or one star a thing just to be, honestly, a dick on the internet. Yes, that is very true. Yes, and and some people enjoy that greatly. They enjoy being contrarian, we'll say. They enjoy They enjoy yucking other people's yums. They enjoy feeding in MOBAs, right? They're They're always
channeling their inner 5-year-old. Yeah, right, for sure. And you know, that's a whole different topic, but that applies to review scores, too. Um so

Jeff, I'm going to give you a picture of this, by the way. I I I not handy, but I tell you what I'm going to do. I'm going to read you the list of the top user scores on Metacritic, okay?
Okay. This like top 10 or something?
Yeah, top 10. You're going to start from 10 to go to one? Uh yeah, we'll start from 10 to go to no. Number 10, the Xbox version of Silent Hill 2. Okay. That makes sense, well received. Number nine, Chrono Trigger. Number eight, The Legend of Zelda, I believe this is Ocarina of Time. Is this all time, by the way, or
This is all No, this is all time, excuse me. Number eight is actually Majora's Mask. It's rated higher than Ocarina of Time. That's very strange to me. Uh number seven is Heroes of Might and Magic 3. Okay. Number six is The Witcher And then, by the way, the rest of these are tied for second, so they're kind of arbitrary, but Okay. Tied for second, we have The Witcher 3: Wild Hunt Blood and Wine. Okay. Metal Gear Solid 3.

Okay. Metal Gear Solid. Okay. Silent Hill 2 for the PlayStation 2. Mhm. And Disney's Cory in the House for the Nintendo DS.
[snorts]
One of these things, Will, is not like the other. Right.
[laughter]
And that's where we get to the weirdness, right? Which it And I think it's actually doing something really important, um which is drawing attention to this problem. Um By the way, for those of you who came in from our thumbnail on YouTube, I initially had the cover of Cory on the House on as part of that thumbnail,
Yeah. but it is such an off-putting game cover that eventually I just removed it and put the Metacritic logo there instead, Yeah. because I thought it would tank our views by putting the actual game cover.
It's such a bad game cover. It's not a good game cover. It's not a good game. It has uh hold on. It has So, the Metascore is available after four critical reviews. There are only two critical reviews for it. One of them is IGN and it gets a 30 and one of them is TheGameBoyz and it gets a 40.

Mhm. Critics do not like this game.
Right. And getting only two reviews means that it was considered such shovelware slop that no other site deemed it worth the the couple of hours for one of their reviewers to go and play it Right. and put a thing on it because they thought so few people would look it up and the ad revenue like it's pretty simple. If they think enough people will read it to recoup the ad revenue, Yeah. like somebody gets assigned to go review it. Yeah. So, um the the user score is a 9.3 Mhm. after 7,339 user ratings. For uh comparison, Claire Obscure has 23,947 user ratings.

Um and we'll choose something older. We'll we'll take um This is Oh, this is Chrono Trigger for the DS, by the way. Um that's based on 1,000 ratings. So, Chrono Trigger has a 9.2 after 1,000 ratings, Claire Obscure has a 9.5 after close to 25,000 ratings, and Cory in the House is catching up at just over 7,000. Yeah, which is an absurd number of reviews considering that only two sites and that makes you think for a moment, "Hmm, why does this have so many reviews and why are they so high?" And of course the answer is it's the internet. Right, it's the internet and it's memes.
It's a meme stock. And also, it is a good reminder. Like Although, who organized it? Do you know? Reddit. I'm I'm a I'm I'm going to I'm going to blindly say Reddit, but I'm pretty confident it was Reddit. Reddit tends to do this. This is a Reddit-esque thing.

Um but I do think it is an interesting highlight, right? In the same way that the meme stock stock comparison is apt in that it draws attention to a real thing, right? Like GameStop getting shorted, right? The the original OG GameStop meme stock stuff, not the all of the everything afterwards that has become real weird. Real weird. Excellent Folding Ideas video on that. Um but the Number go up. And number go up. Um the original like, "Hey, there's a problem in the system and this is serving to highlight it." This kind of feels like that. Not that there's anything wrong with Claire Obscure or anything particularly great about Cory in the House. There isn't. But it is serving to highlight and remind us that this is very cookable. This whole system is while we rely on it and use it as a way to um Or at least the the user-facing side is cookable in a way that radically different than the

critic-facing side. Yes. Because theoretically, a popular group on Reddit could be like, "Hey, let's all go do this one thing today. It'll take everybody 5 minutes. And let's like And that will always be the downside of community aggregated things is that if a community or excuse me, of just aggregated things that anybody can do is if a community decides to disproportionately throw their weight, say in this case pumping it up or in another case review bombing. Right. Like it's a thing It's hard to gate that. Right. It's um uh John Oliver's audience influencing the vote for national bird for I don't even remember which country. It might have been New Zealand. Uh John Oliver got in on that and was like, "Hey, I think this should be your national bird." And then his audience screwed with the voting, right? Or Yeah, John Oliver's like, "Look, somebody put their national bird group thing up for a vote."

Right. Yeah.
"Wouldn't it be funny if" Right. Or the um or the uh uh what what am I thinking of? Uh Boaty McBoatface, right? Or any any time Several towns have done this in the US. I don't know if places outside the US do it, but it's really funny what they do. People have like contests to like name snow plows. Mhm. That I don't know why that specifically, but I've seen it multiple times around the US. People will like name snow plows and they're all memes because it's a thing that people get to suggest ideas for, right? So, there this does serve to like highlight that, right? Um Mhm. And so, this is this is a this is a great question, right? Should regular user reviewers start to have higher powered reviews? Oops. Oh. I'm That wasn't what I clicked on. Oh, okay.
It's you're locked out cuz it's how long we need to leave it up so people can read it. Give it a second. You got You got to give it a second. Um I'll I'll

get that one after. Um, no, it's good. I'm Yeah, uh we have people talking about their their calendars arriving, which is which is excellent. Yeah. Mom, why is Batman our national bird? Correct. Correct. So, there isn't a way to stop this and keep I don't think there isn't a way to stop this phenomena from occurring elsewhere. Um, and by the way, we've chosen what is possibly the most low stakes version of this. It happens the other way all the time, right?
higher stakes, which we'll come back to shortly.
Yeah, for much higher stakes, um, and for much more serious reasons than Reddit got bored and thinks Cory in the House is the best DS game ever and should therefore be the best user rated game ever. That appears to be the goal from everything I've seen, both when I looked at it for breakfast and when I looked at it for this, that appears to be the goal. Right? Um, And like, it is theoretically a solvable

problem, but the steps you would have to take to solve it
Right. would in some cases defeat the purpose of general easy community reviews. Correct, because the um, this is very much on the pro- on the access of security and accessibility, right? That is uh, security and accessibility is a um, continuum, right? The The more secure you are, the less accessible you are. The more accessible you are, the less secure you are. Because you could, for example, take um, more of a Wikipedia style approach, which is you have to have an account, you have to be signed in, we track what you do, and if we've caught you messing with things, you will be banned and all of your past work will be deleted. Right. Kind of stuff. In other words, like for Metacritic, they could say, you know, we take this super seriously, you have to be signed in, you have to do this, you know, maybe require a driver's license put in so we know you're a real person, and if we ever detect you taking part in what seems to be a meme push, you know, your account will be deleted

along with all the like there are ways you could theoretically do it, but that would make some of this so not fun to participate in for something that Metacritic actually gets a lot of value out of. They're like, "Look, we spent all this effort on one thing, we spent this tiny bit of effort for this one other score we get to show as well. You know, we can like Yeah. And then I think Steam Dash has a has a great point that seems had some attempts at solutions by taking all data, but not aggregating all data. Steam a while ago introduced several features where you could choose to exclude from the overall score periods of extreme review volatility, which is generally shorthand for we detected a review bomb because 10,000 people left negative reviews in this span of 20 minutes. Right? So, we're going to not include that data in the overall aggregate because it doesn't we don't think it accurately represents the picture. And in some cases kind of

going back to what is the purpose of a review score? Like if the purpose of a review score is to say, "Is this game worth my money?" That has nothing to do with did they use AI slop in it? Or did they make a political statement that I particularly don't like? Or is there a female lead protagonist and I don't think women should ever be portrayed positively in game? Like there's all sorts of stuff that have nothing to do with how good is the game, but because people can't comment on that, they use the game's review score as a proxy for sentiment. And theoretically review score and sentiment are very different things. Like in a ideal world, you'd have a review score and a sentiment score. And people could say, "Great game, but bad sentiment." Like we don't think this is good company or good day, you know, whatever it is. But because there isn't a separate thing, and a lot of cases the purpose of a review score, is this game worth my money? Is this a good game? Gets

subverted to reflect on something else. And that kind of goes with review bombing or other things as well. Yeah, um and it is quite interesting because uh there is a and and I'm sure I'm about to get a whole bunch of people's in the comments and in chat saying, "But I do that." Who don't drill into the actual text of the reviews. Because that's not what's top level on any of this, right?
[clears throat]
There's a There's a bar on the Steam page that says overwhelmingly positive or overwhelmingly negative. And if you scroll down, there's just the Metacritic score. It doesn't even tell you how many critics contributed to that review score. It just says the Metacritic score for this game is 70. Right? And the And the the overall analysis of
[snorts]
the game on Steam is overwhelmingly negative. And if you don't drill into the why, because the why is not compulsory either to be given or to be received by someone

looking at the review score, you just end up with this confused conflation of the quality of the product and the sentiment around the product. And those are two, right? And this is something you don't get as much with like the Cory in the House one, but you get absolutely with review bombing. Yes. Because you're really Again, a theoretical review score, is this game worth my money? Is radically different than, you know, as I saw in the comments, you know, is this dev a bigot? Is so-and-so and so-and-so? And it goes back to what Click and Again with Dark Wolf said, which is May I click on it again? Yeah. You know, is the art good, not the artist. Yeah. We're not caring about the people that made it or anything around it. It is a review of the game. So, interesting question. Uh and I didn't think we were going to get into the philosophy of like review scores, but I like this. Um Well, hey Will, we're here. We might as well do it.
Right. I would argue that for a large portion of the population, that is a distinction

without a difference. That for many people, they have come to view their the worth of an object or a product or a service as being inherently tied to the um to the overall picture that that participates in, rather than just like a singular thing. So, we'll take a nice We'll take You know what, Jeff? We're just going to light it and throw it in, right? We're going to take a nice controversial example, right?
Sure. The Harry Potter game, Hogwarts Legacy. Oh, okay. That was one of the ones I would kind of call to throw it in.
Yeah, right. So, the I think there are some people who would argue strongly that it is impossible to separate the question of is this game worth my money, Mhm. the [clears throat] purpose of review score, from these are the

purposes that the money which I am paying for this game is going towards, right? And so, there is that part of it that as consumers become more sophisticated, as we have more and more connected, involved consumer bases for things like video games in particular, because we're all online, and that's probably not great. And when you're done watching this, you should probably go touch some grass. I know I need to. But, like the um those two things are not separable for for a large number of people in cases like that, right? So, it becomes difficult.
because like that game we're talking about, the Harry Potter one, you have Rowling who's the author who, you know, well past the series, I think, started making some statements that a lot of people are like, wait, what? But also the team that made the game was incredibly diverse. And like skewed heavily into LGBTQ. Right. And so it gets even more nuanced on that is do

I support the work of those people even though I don't like the person who initially made the IP? Like it's a it's a complicated matter. Right. Yeah, there is there is a huge um and again, all of that all of those questions which uh someone might consider and ask and take into a nuanced approach to should I buy this product? Should I recommend this product to others? Is it reflected in a review score at all? Even the Metacritic score. Even the well-considered um aggregate of the Metacritic score still boils an entire multi-page review of a game on IGN down to a number and then plugs that number into a formula and then says this game gets an X. Right? This game gets a 90. This game gets a 70. That game gets a 30. Right? And so it it does this is like I think we're coming up against the limitation of this which is even if it's not the purpose, we still have to admit that like

all the review does, especially on Steam, is tell us whether or not people are particularly happy with that game and that developer right now at the moment that we're loading the Steam page.
And some of this is tricky, too. Like take the book Ender's Game which is unarguably an amazing book and in some cases a life-changing book for a lot of people. Mhm. You know, can you hold that or should you tell people not to read it because 20 years later the author came out and made, again, some statements that a lot of people wouldn't agree with. Right. Like and this is a hard thing and it's a, you know, per person choice. And like, you know, for me over a lot of time of thinking about it, I decided that I wasn't going to let what creators did later rob me of my ability to enjoy something in life.
A thing that was foundational to you at a point in your life.
Right. Right. Yeah. Right. And that's
not buy recent stuff they've done. Correct. But again, it's it's a hard line I mean, it is a hard thing to figure out in there. And this is why I'm

also advocates of not putting of leaving review scores as review scores and finding other ways to talk about sentiment.
Right. There's a reason I don't do a review score on Rogue View. I spend between an hour and 4 hours playing a game and talking about what I like and don't like. Right? Sandman being another Yeah. Sandman, yeah. Yeah. Sandman
multiple pieces of Sandman art that I own that are way older than Yeah. any recent happenings. My complete is behind my green screen, right? And so, um it is this is sort of like, in my opinion, the biggest shortcoming of this and sort of it's really I I want to if if this gets to the people who did it, who started this, my hat off to you team who who believes in Cory in the House. Salutes to you on your upward journey. May you make it someday. And for also for allowing us to be reminded that like

there was more going on here than just oh, this game's good cuz it's got a good score, right? Not that Cory in the House is bad. I haven't played it, but by all accounts it's bad. We will not comment positively or negatively on games we haven't played because that kind of defeats the whole purpose. But, um yeah, like review scores, if you're looking at their actual purpose, which is say how good is the game, that's where I trust the critical version of it and why I think the community ones will always be Ascendant.
lacking or suspect Yeah. because people are coming in again. There's a couple of reasons, you know, they rate either really high or really low. They don't have the nuance and they're coming in with a lot of I'm going to use baggage in the not in a pejorative way, but in a, you know, the baggage that they have interacting with Biases. that IP, that platform, that style of game and everything else around them on how they their experience is with it.

Right. And I think that the um I also think it's very worth noting that it at this point that like it is worth acknowledging that um some that a lack of understanding of the facts regarding some games particular as we get into remakes and remasters is also a problem with review scores, right? Like seeing reviews for like this game feels like it's taking an awful lot from this other game Mhm. when the game that the review is left on is a remake of one of the original games in a genre and the game that they're comparing it to and suggesting it took things from is actually like 10 to 15 years younger than the game that we're experiencing or And and possibly it don't matter your like Right. And like
game came out 20 years ago. Yeah. They get released. People like, "Hey, but this game 10 years ago they seem to lift it a lot from it." You're like, "No. That game was lifted from this game

Right.
which is now, you know, remastered up to the Yeah. Yeah. I I What was it? Um Oh, it was Dune. Oh my god, it was the new Dune and someone went "Hey, this is a lot like Star Wars." AND I WENT "NO. THAT'S NOT HOW THIS WORKS. That No. Stop. No. Back up. No. Backwards. [laughter] Not how this works. Right? Like Yeah. Yeah, Shakespeare in the original Klingon. Yes, correct. Exactly. So, I do think that that's another
Shakespeare until you've read it in its original Klingon. Right. And I think that there's that's that's like a good example of part of this, too, that does cloud it, right? As we have a less informed or more Like if I'm an expert in roguelike games and I spend 300 hours on a roguelike game, and someone else has never played a roguelike game before and plays a roguelike game for 10 minutes, unless you drill down and look

at how long we've played those games, Steam review Steam counts both of those reviews equally. Right. Cuz I'm not a professional critic. Right, in Steam's eyes. Right? And so Well, you haven't registered and gone through the process of setting yourself up in a Right.
Like if we were publishing a set of reviews under Extra Credits
Right. you might get picked up as a professional critic for that. Right. And and and and I do want to let's I do want to do this, right? [clears throat] I There is absolutely value in user reviews.
Oh, yeah. It is in understanding what that value is and how to glean the most appropriate value for your purposes out of that data, rather than taking it face value. Oh, this game has a This game has a Metacritic score of 9.3. It must be a hidden gem among DS games. I should go and spend hundreds of dollars on a copy of this, right? Like

Will, you know what I actually think would be really interesting on Steam? Yeah. What if they took What if they rearranged it so when you saw right at the top where the reviews are right now Mhm. if there was two scores there, one said, you know, Metacritic 83, and then it said like Steam community sentiment, you know, overwhelmingly positive. They don't call it a review score, they just rename it to sentiment. Because at that point, you're getting kind of the best of both. You're getting here's what the critics think, and here is the here is both the recent and overall sentiment Mhm. of what our audience is. Yeah. Uh by the way, I looked I thought I was joking. I was not. There are copies of Cory in the House currently going for between uh $150 and $500 on eBay right now.
[laughter]
Mhm. Amazing. So, I want to bring this back a little bit. So, we were talking about like reviews and other things. Metacritic, because of the way it does

things, also has a huge role in the game industry because we've seen that you know, it's probably the most it's for I'd say it is the most trusted review site in the entire game industry because people know there is variance. You know, maybe that editor at IGN doesn't like your particular kind of game, so they're rating it a little lower, but you know, GamesRadar does. So, like it averages out a little bit of the rough edges that people just bring to it. But also, because of the world we live in, and because of how big games are nowadays, there's also a lot of teams that will get hired to make a game. Or and in a publisher-developer relationship, publishers often contractually want to set a bar that says, you know, we're going to be paying you $23 million to develop this game. But part of what you're guaranteeing is

it's going to be a good game. And how do you rate it's a good game? Because in a contract, you don't want it, you know, external producer Bob says, yeah, good enough, because you know, you get in your bonus your team's bonus of like $2 million to pass out among the 130 people who worked on it for 3 years or whatever. Like contractual milestones. If it's just relying on your external producer to be like, "Yeah, good enough." That external producer who's part of the publisher might have invested interest in not saying that to save his publishing studio millions of dollars. Yeah. Or other things. And so, over time as people started looking or for a way to make a contract more just cut and dry where people can understand I did it or didn't, they're started getting put in for some studios on development, you know, must get at least a, you know, 75 or higher on Metacritic or 80 or, you know, but they would put in some number where X amount of time in, so like 3 months in

this needs to have a review score of at least X on Metacritic to trigger this extra payout clause. And this was the publisher basically looking to ensure good game, but also the developer looking to ensure isn't decided by external producer Bob who might have an interest in not getting us paid out.
Right. And so, and those are always always always set on the professional reviewer side, not on the sentiment analysis side of the audience. Right, cuz it it turns out it's really easy to cook sentiment analysis, right? You can You can do an awful lot if you're like, "Hey, if we get X number of positive reviews on Steam, we'll give out a free thing in game or whatever." For that for the 2 weeks after that our next, you know, all of our DLC will be free. Right. There There's plenty of ways that you can cook that. Yeah. Yeah. And And I think that that is the I would say one of the more valuable um portions of Metacritic is that it is this external mathematical arbiter,

right? That it is not It's not doing Oh, whoops. Keep going a sec. Sorry. a lot of bias. It's It has some, right? Like Metacritic has some bias in what it what it ranks and weights and what it includes it doesn't include. But that is a known and continuous factor, right? Rather than saying like Metacritic doesn't like hidden object games for instance, right? Um you you it's it is a mathematical arbiter rather than a um rather than an a singular independent reviewer. Uh and yes, I yes, Gladys, absolutely. 100% I'm I'm reasonably confident Steam does have a rule about non-Steam reviews like that. Um I'm suggesting that uh it might be if they were using the user review section of Metacritic, then it wouldn't be encouraging them on Steam, it'd be encouraging them on Metacritic and that would be hard to undo or track.

Right? Um any any time there's the possibility of allowing any kind of external uh influence on it if it's not in something like Steam where the review is directly tied to the purchase location, uh it's very hard to track. Um yeah. It it's an interesting thing. Like all of this is just It It goes so much deeper than most people think about. And again, I only know this because I spent years running a group that used our own version of Metacritic as one of our data points when referencing all of this stuff because in the same way as we talked about Plants vs. Zombies and good well-reviewed games equivalently sell better. Yep. You know, this is a very true thing. And review score is still a good shorthand for how well was the game put together. It's the elevator pitch. Right? In a lot of ways, right? I am I am a big um

opponent of using the of overusing the elevator pitch. In the same way that I'm a big fan of a big opponent of overusing the review score, right? And that is a great place to begin a conversation or to in shorthand express to someone who is using similar shorthand and idea. And then if those things aren't true, you need to use it to begin the rest of the conversation or say more about it, right? Because again, we're we're hammering out a multi-page document down to number Yep. Right. And one of the other interesting things is for years and actually, I want to kind of swing into a new arc with this if you're okay.
Yeah, that's fine. Um game reviewers were notoriously onerous with publishers and um the publisher would be like, I want this and the reviewer is like, well, I'm doing my review. Like the games media despite the whole

um gamersgate thing is actually surprisingly good and independent. Or at least was for a long time and I'm going to get to a little bit of the shift at basically flipping the the bird to EA and 2K and all of the other groups and saying no, game reviews aren't for sale. Right.
Like this was their professional livelihood and for people who went into become game reviewers in the same way I feel like movie reviewers and a whole bunch of other stuff they didn't go in thinking this was a high-paid job or you know, it was going to make them Ferrari money or other things. It was I love games. I love reading about them. I love playing them and I love writing about them and being a critic. Yeah. And one of the problems we've seen is that publishers are are getting less and less interested and for good reason on their side on leaving it up, you know,

all of this audience, you know, first impression anchoring effect up to any chance, you know, to to chance. Studios, publishers, um, will often pick the first people who get to see their game, only supply limited numbers of, um, review sites with copies of the game, and then will pick the ones they think are most likely to be, um, and predisposed to liking the kind of game they're doing.
Yeah. And, um, they try to manipulate it as much as they can. Like, again, I understand that. Like, if I had a publisher working for me for a game I'm devving, I would absolutely want them to be sending out early review copies to I to people who we think would treat it the best, because, hey, as a developer, this is the game I've just spent years working on. Of course, I want the people who are going to like it to see it first.
Right. Like, that may not be the best thing for the audience, Right. but, um, with the rise more and more of

influencers and with the consolidation among gaming media, Yes. you know, we now have less sites doing reviews and more people also going to watch somebody say play the game on Twitch instead of getting a reviews for or go look up somebody on TikTok or Insta doing, you know, a quick story about it, or other things, you know, that in that person you like on YouTube who does shorts, the funny game guy, like, "Oh, I'm going to go see what he thought about Clara Obscura." And, you know, that may be the thing they check instead of the Metacritic feed or others. So, we're seeing a shift in some cases in ways that people engage with Cuz before Metacritic, there was also the you either read a magazine, but it was also the gathering of the playground or the water cooler, it's like, "Hey, I hear you picked up the new, you know, Mario game. Is it any good?" Right. And if your buddy's like, "Yeah, that's really good." You're like, "Oh, cool, it's really good." Like, all of this is kind of a proxy for who you go to ask before spending your money. Yes. But we've seen some kind of

consolidation. We've seen publishers showing up at less trade shows, doing more digital events where they control the whole thing and a developer or a reviewer doesn't actually get a hands-on copy of the game possibly until it's released. And like all sorts of things that have been making it harder and harder for review group sites to actually do their jobs both internally from consolidation and in some cases lack of funding, but also from publishers being less willing to engage and less trade shows like E3 and other things that give them chances to engage with it.
Right. Nintendo didn't send out any review copies of the Switch to as I recall. Um they they held back on those because they didn't feel like they needed the review scores. Yeah.
Which is kind of an interesting concept, right? Because from a like as someone who wants to advocate for consumers in this I absolutely want there to be review scores so people know whether they're spending $500 in the middle of maybe not the best economic times on a fancy brick

or not. Yeah. Right? And it's tough because um we don't have I think we we have seen that people are willing to spend money based on that sentiment uh score rather than a reviewer score more and more and more frequently. Yeah. Or there different people have different things that they're looking for. Yes.
On that. Some people might want that objective meta score and others might say, "Hey, I tend to like the games that, you know, Steve the streamer does. So I'm just going to go check out what Steve says and if Steve the streamer likes it cool." Yeah. And that is a tricky thing. But one of the things we did notice and I I don't know if this is still true cuz I don't run the data studio anymore. But game reviewers were very very close to what game consumers were on their taste. So, you didn't get the Transformers thing where, you know, the movie, you know, the

critics were giving Transformers a 30, but the audience it was the best-selling, you know, one of the best-selling movies of the year. Right. That never happened in the games. Right. There were I mean, I don't know for the past couple years, but for the two decades before that, there was never a game that was saying the 30s, 40s, or 50s of review score that was a like breakout hit. Right. Ever. Like it didn't happen. Like there were occasionally ones that might have snuck in on like a 70. But usually to be a breakout hit, it was at least an 80 review score and for a while a 90. Like there was a direct correlation between review score number and amount of revenue earned from game sales Yeah.
to the point where it was almost we could prove causation. He And that's And I think that there's um a bunch of
that was just cuz the reviewers lined up with what game audiences liked. Right. And I think that part of that, too, is

that the um the industry is I'm going to say flatter than like movies. Right? So, so so here's here's here's my argument for this. If I want to watch winners, which just broke every record for uh the which broke the record for most nominations, by the way. I saw that this morning. It was like 16 or something?
16 out of 24 categories winners is nominated for. Um I hope it wins them all. Uh the um I have to go to a specific venue to watch that. Right? I have to go to a specific venue for movies to watch that. Um but if I want to watch a singular person who might know play or or uh make an artistic endeavor in film-esque fashion, YouTube video, I go

to a different place. I go to YouTube. But in games, if I want to buy the latest and greatest I don't know, okay, latest, it's probably not greatest, Black Ops game, right? With all of its controversy, I go to Steam. And if I want to buy the solo dev teeny tiny project that I know the developer and want to support them and think they made a cool thing, I also go to Steam. The industry is much less striated in terms of the consumer side than the [clears throat] than a lot of other industries are, I think. And I do think that that that weighs into what you're saying. Which
Okay. I didn't even thought about that. I mean, my thought had always been we have a 100-plus years of film critique. Sure. People go to film school, they get taught by film professors. There are things you get like what you might think makes up, you know, the best bones of a movie Sure. may not be what the mainstream audience wants to go watch and have a good time.

Like there's a difference between In the same way, it's that wine thing. Yeah. That $50, $100 bottle of wine is enjoyed more by some people than, you know, kind of the general mainstream group at $10 who couldn't tell the difference between all of them. Yeah. Like there's kind of that level in film. But for games, fun games are fun games. Like there aren't a lot of art house games. There are game of that style. Yeah.
That um kind of Well, I guess And part of it too is people know the kinds of games they like. Yes. If I'm like, Will, here's an amazing art house game, you might I like, "Yeah, I don't like art house games in the same way you don't like racing games or you know, whatever it is." But if you go play it, that reviewer who reviewed it will most likely line up with your feels on it. Yeah. And then the people who review games are big game players for and enjoy their big sprawling mainstream fun games. Is there also

something to the idea that there is a skill involved or an acquisition of skill involved in playing that many games because it is an act of experience? Maybe. It may also be I I surmise part of it, right? The idea that um it's not just that you've played all of these games is that you've played you've gotten good enough at all of these games to play and um and beat it. There's also um there's different time scale scope. Yes, also very true. Very, very true, right? An hour and a half movie I could sleep through. Right? But I can't do that to a bad game. Yeah. I'm still in the
difference in the inter- interactive versus passive experience. Like there's a whole bunch in it. And and I think that
are Yeah, it's My point is only that like games have all of these unique factors that lead to us having this really strong correlation between what reviewers are saying and what the public is generally saying with

exceptions for the sentiments more. Here it is. Well, I've got it. You ready? We're keeping on the food analysis. Okay. Movie reviewers are like people who review wine. Mhm. Game reviewers are like people who review burgers. Okay, I dig. I think Yeah, cuz like burgers are more accessible, right?
Yeah. They're more accessible and it's one of those things that like if you're like, "I'm in the mood for a burger." The person reviewing burgers probably going to do you well. Like there's certain criteria of burgers that you're like, "Yeah, this is going to be a good burger." And you know, there's variants, of course, but And that's different than I guess the depth that you find in wine or something. And it's not a perfect analysis, but I'm going with it.
I mean, I could talk to you about burgers, but that's for a different time. Um No, but like I bet if you and I Okay, my dad has gone all around our very big hometown

and is like he and his friends have spent 15 or 20 years the city of San Diego rating burger places. And he has a list of what he considers to be the best burger places in all of San Diego from him and his pals have been doing this for decades.
Yeah. Yeah, yeah, yeah, yeah, yeah. I bet anybody who likes burgers, my dad could give his top 10 list to and they would be happy at all 10 of those places. Yeah. Yeah, yeah, they might not order them the same, but they will agree that they are all Yeah, superb examples of what a burger shop can do.
Right. I mean a burger shop, a place that serves burgers because some of them are like notoriously like this one dive bar, it's like the burgers are amazing.
[laughter]
It's not a burger shop, it's a dive bar, but they heard about good burgers so they went to check it out.
Right. Yeah, yeah, yeah. No, I take I take that. I take that. Oh, chat, I think [laughter] I think that's I think that's a good analysis to end on. I like this. Um thank you so much for joining us today as we talk about Metacritic. Did we Did we miss

something? Are you one of the people who's leaving positive reviews for Corinth House? Do you also claim that it is a life-changing majestic experience for the Nintendo DS that caused you to experience the universe in an entirely new way? Leave a comment if if you did. Um we'll be back next week. We're going to be uh hopefully checking out a game uh next week. And uh if you want to do something
the name of the game or no? No. No, they get to find out. They got to show up tonight. Okay, we're going to be checking out a game that's in early access Yeah. two of the developers Yeah. on the game specifically looking to watch Will and I play a friend slot game and hear our and the community's feedback Yeah. about the game because they still have time to tweak, tune, and otherwise continue to evolve their game.
Yeah, so come with your come with your feedback hats on next week. If you're watching this live, hang out. Matt and I are going to jump into Hytale and give you our first thoughts on that with some special guests here on Twitch.

But I believe that will
that why we need to Hytale it out of here to get ready for Oh, man. We almost made it, chat. I think we almost made it an entire episode without a Jeff pun and there at the very end he had to spike it. Oh, I was so close. I almost got us out of here. I'm sorry, chat. I failed you. All right, everybody. Say bye to the people, Jeff. Bye to the people, Jeff. See you next week, everybody.
[music] [music]