The Kuleshov Effect - Improving Character Interactions in Games - Extra Credits

1918, Russia. A darkened cinema plays a short film. It's only a minute long. It shows a man and a bowl of food, a man and a dead child, and a man and a beautiful woman. The audience stumbles out into the night after the film raving about the acting, the subtle emotion portrayed. But was that emotion in the film or was it something else the audience brought to the experience? That film demonstrated the Kuleshov effect. But what is the Kuleshov effect? Well, let me tell you a little something funny about that film. Every clip of the man was just stock footage. They were the exact same clip. In cutting between the man and the food, the man and the child, and the man and the woman, the filmmaker Kuleshov showed something. That we ascribe, even retroactively, emotions to human beings on the screen based on juxtaposition, context, and our own emotions. Hitchcock later hilariously demonstrated this by putting

together a short montage of him staring out into the distance and smiling. In the first clip, he shows him doing this and then splices in footage of a mother playing with a child. And as he put it, this makes you feel like he's a kindly old man. In the second one, instead of the mother playing with her child, he splices in a woman lounging in a swimsuit. Now when he smiles, you feel, as he so elegantly put it, that he's a dirty old man. Don't believe me? Watch the link of him demonstrating the Kuleshov effect. It's in the description below. Oh boy, when Hitchcock smiles that second time, he is without a doubt a dirty old man. Now there is a bit of conflicting research on how impactful the Kuleshov effect is, but there's been enough study recently to say it's a real thing. So I'm starting to think the variance in impact might actually be based on how well you use it. But enough film talk. You're not here for that. And if you were, you totally know where to go. Wink. Let's say you know how to use the cool shove effect well. How is it

relevant to games? Well, as much as I hate to say it, in the majority of games, acting is notoriously not great. And even with all the facial mocap that we've developed, there's still only a few companies that really nail emotive facial expressions. So, it's safe to say games need all the help they can get in that department. And we've spent a huge amount of time and money trying to solve it technically. But perhaps part of the answer is psychological rather than technical. Okay, thought experiment time. Imagine a Bethesda or BioWare style RPG where you go up to NPCs and have conversations. Almost universally, these conversations happen with almost no camera work. No matter what choices you select, we normally just see two people, or monsters, or aliens standing almost static chatting with one another. Now, depending on how much your game's engine is loading and unloading assets for those scenes, it might be possible that when you make a dialogue choice, you cut away or pan pan off from the two characters talking to something else nearby. For example, imagine a conversation you're having with an NPC

who might be a thief or a traitor. Cut to a shot of their horse right after you've asked them about whatever crime you're investigating, and now all of a sudden, you're getting the sense that they're thinking of making a break for it, which in turn makes us read their facial expressions for the rest of the conversation as much more suspicious. And the most interesting thing about the cool shove effect, as far as games are concerned, might just be that it's most effective on faces that, without other context, we'd read as neutral, which, honestly, is a lot of in-game faces. Often for lesser characters that we're not going to do full mocap performances for, we just have an idle animation going during these sequences, which means they're a great blank slate for us to project on. All it would take is some conscious planning of the environment they're located in. Granted, this means that we'd probably have to lock down our dialogue a bit earlier than often happens in many projects, but that's probably a good thing. The larger question though is can we get our players to feel as though our characters are expressing more emotion simply

through context. The cool show effect is specifically about using film style cuts to give implicit meaning to facial expressions, but does it have to be just cuts? Well, let's look back at the three things we said were key to achieving this effect. Juxtaposition, context, and our own emotions. First up, juxtaposition. What's happening right alongside whatever character expression we're trying to interpret. In games, we have a lot of stuff. There's sound, writing, mechanics, and the systems themselves. Each of these elements can be used to provide context for what's going on. Ever watch a really impactful moment in a video game with the sound off? A lot of the time it comes off as melodramatic, laughable, or just totally unbelievable. And this is because the sound often gives us the context we need for our brains to fill in whatever we'd otherwise see as essentially stilted acting. Not to mention, sound also gives us an emotional base to project onto these characters. A sad Nobuo Uematsu

track goes a long way to make a collection of pixels seem to be expressing emotion because it puts us in the right mental state to project those emotions onto those characters. Good writing of course also helps us believe a scene as well. But perhaps the most interesting from a game design perspective is systems and mechanics. This might be a small part of why the Paragon Renegade or light side dark side choices don't really feel impactful. As soon as we slap that mechanic on there, we're being told the emotional context for a line rather than being allowed to project our own context onto whatever the character say. It also exposes the mechanical and game like nature of what we're playing, providing an emotional context of this is just a game, which is never what you want your players to have as a base while their brains are supposed to be interpreting a meaningful scene. By contrast, it might also be part of why "blank will remember this" works so much better. It retroactively contextualized your actions to seem as though they were full of meaning in the experience. And it helped solidify the

character you were speaking to as a real person whose actions mattered to you. Finding the mechanics and systems that provide this sort of emotional context is one of the challenges of the next generation of games. Because if the cool shove effect teaches us anything, it's that creating believable characters doesn't solely come from the performance of the actor, but part of it comes from the audience itself. Therefore, no matter how high fidelity we get our NPCs to be, no matter how well mocapped, we can't look to technology alone to provide emotional performances. Rather, to get rich meaningful performances we want in our games, we also need to consider the element of the players themselves. And even what, in a strange subconscious way, the players bring to those performances they see. Because if we ever want our games to feel really real, we need to create as much psychological fidelity as we do technical. See you next week, everybody.