The Schema is (Still) Mightier than the Sword: How Cognition Predicts Player Spatial Coding Systems

[Music] good afternoon my name is vanessa hemovich and i am an associate professor of psychology at digipen institute of technology and my research and the classes that i teach focus on cognitive science ux ui and a lot of ways that psychology dovetails with game design and i'm really looking forward to giving this talk today and the role of schemas as a driving force as we'll see with perception information processing and decision making so for today i'm just going to sort of outline kind of where we're headed and the the idea of how is cognition relevant to game design is something that we're going to look at here and i wish i had three hours to do a master class but unfortunately i only have about 20 25 minutes so i'm just going to sort of boil this down and schemas have a very very complicated relationship with human behavior and they are easily formed

easily activated and never really turn off and today we're going to explore the nature and the relevance of schemas to player behavior and how and why they function to drive player perceptions and how they guide decision making and things like player choice so what is a schema this is a good place for us to start and one example of this is obviously what you see here this nice cute little puppy dog and in research ischema is defined as an organized mental structure or representation of stimuli and schemas really do work to streamline our information processing and categorize objects to help them relate to one another general categorizations like this can more or less be considered universal this is a picture of a dog not of an airplane and schemas tend to reside in our long-term memory that said one more complicated aspect of schemas is their subjective nature so if i had everyone in the audience close their eyes and imagine the concept of a dog we would see a lot of variety in this room i might picture a german shepherd and you might think of snoopy or something like that so in

terms of looking at schemas in action what we can see here is that when we first learn often when we're young what a dog is a schema is formed and this is called assimilation and this is the process of fitting information that contains the features we associate that defines this stimulus so for example a dog typically has two ears a tail four legs and tends to bark and when we are out about and about in the real world we see these stimuli and we carry forward what we've assimilated in long-term memory with us is this a dog does it match the known criteria in this case the answer would be yes so over time what we do is we learn that life can get complicated and schemas evolve and develop into more specific and hopefully more accurate mental representations and this is referred to as accommodation and it allows mental frameworks to adapt and progressively change from simple concepts to more complicated representational concepts like this so we learned that schemas or dogs in this case contain a bit of

variety and our mental frameworks adapt and by the way we are doing this our entire lives this is not just something that children go through as they learn and grow devs experience invoke and change their schemas so do artists and programmers players do as well really nobody is immune so in cognitive science what we do is we sort of work through schemas and activate them in different ways and one is through what's called conceptually driven schemas and this is known as top-down processing so here we see homer deliberately invoking his mental framework of bacon which of course is delicious and i hope you're all not thinking about bacon and can instead stay with me here as i continue to talk about schemas and not talk about bacon and strategy is another good example of this as well so with top-down processing in a game let's say like rocket league what happens is what's called endogenous thinking and players learn strategy quickly and they learn best practices and procedures to use these strategies use this knowledge to inform how to play the game

for example you should not really in a game like rocket league send more than one teammate into the initial strike you'll want to also maybe think about dividing the field and only watch the risk in terms of chasing the ball don't bunch up and keep the ball centered maybe if you're going for a more aggressive strategy personally i find that the best offense is a good defense and if you disagree with me i look forward to meeting you in the arena schemas are also activated sometimes purely from a sensory level or when the environment delivers a stimulus sort of at our feet and this is more of a what's called data-driven type of processing and here homer is sort of minding his own business and he happens upon the cola machine he might not have been thinking about this attitude object before but he definitely is now and it may even change or alter his behavior and for those in this room who have maybe experience playing fortnite or other battle royale games what tends to happen sometimes the environment is so intense that player

strategy just goes right out the window i'm going to drop into this zone no there's 30 other players that are here now what do i do now where do i go and here the environment is telling the player what's going on and what they might get into is a little bit different than what they might have expected and they need to prepare themselves for what's coming so sometimes players have to adapt their strategy based on how the game is going and schemas are what shoulder this burden for us schemas allow us to shift between top-down processing and bottom-up processing and meet the demands of the game so mentally there's an app for that and it's called the schema so in other games and i'm using destiny here as an example we oftentimes refer to on strategy based schemas for planning and the more that you are aware of these schemas the better you're going to play but that doesn't mean that players won't also be interested if something does sort of pop up in front of them like a golden treasure chest or a loot crate and there are things that will inform the player that they're close

at the environmental level it could be a sound cue that they're getting close to something that might be interesting or worth checking out so one question that i always pose to my students is is it possible for schemas in terms of top down and bottom up processing to coexist and truly be simultaneous and if we think about it the answer is yes it is possible knowing strategy and the schemas around that is a good thing being open to the demands of the environment and sort of knowing that we should react to what the game throws at us is also a good thing so you may start with one strategy the game throws something else at you and schemas allow us to adapt and shift between these different types of combinations of mental processing so games are not only able to do this they are notorious for doing this in the case of what we have here robo recall there are plenty of game examples where devs absolutely throw a wrench in the plans of the best laid strategies that

players have the player understands where to go the checkpoint is here the game has informed them it could be a narrator a narration it could be an npc and they already know where they have to go unfortunately enemy agents arrive on the scene they're probably blocking the path and the player is forced to sort of accommodate and use that environmental input in a way moving forward and this is schema accommodation i have a plan but the game world did something else so this is one thing that i wanted to kind of highlight with this presentation another one is how do we predict what players are going to attend to people are complicated and so are there perceptions but the good news is that science has repeatedly shown us that we can to a large extent predict certain aspects of human behavior and one way is to invoke schemas that will guide information processing through what's called conceptual fluency and in terms of understanding player perceptions and predicting those perceptions conceptual fluency is one of those things

that's all about invoking schema consistency and designing game elements that rely sometimes pretty heavily on how to navigate the game world and wrestle with stimuli and adapt their strategies so players have a long history as you can see with certain icons for schemas and we build game icons around those perceptions that can and should be easily recognized and when we do recognize them what happens is it aligns with what we know and what we expect and it's automatic information processing is easy and it feels good and really this is now sort of getting into the domain of ux versus ui which i don't have time to talk about today but i wanted to just sort of introduce this concept here because we are going to circle back to it so let's consider an example of how let's say a typical player would interact with something like a game menu and the new game is generally listed at the top of the menu with exit at the bottom and that's a schema that i think every single person in this room probably has and when those puzzle pieces click together

with what i know and what i experience this generates conceptual fluency here is another example from the witcher series and you can see again we have a relative match that new game is listed at the top and exit or quit is at the bottom here is yet another example where things are aligning and in all three instances the mental rule is in place and we have a schema match information processing is easy everything is as expected and cognitive load is relatively light however what sometimes happens is schemas don't always match to their environment here suddenly there is a mismatch with this rendering and easy and automatic thinking is momentarily disrupted and as the player is asked to accommodate to this new arrangement that violates some of their prior expectancies and cognitively we have to work just a little bit harder just a little bit this creates what psychologists call conceptual disfluency and conceptual disfluency leads to something called attentional blink so when schemas are violated mentally we often have to pause orange juice and chocolate chip cookies

is not in my wheelhouse my students say go on you should just try it i'm not having it i'm not going to do it it's too much for me i don't have time to tell you what's wrong with this combination but it's not a good one for me and when the disfluency is too strong what happens is we reject it here's another example of my students they send me emails and they say just go for it and again i'm not having it it's feel free to do as you see fit i don't want to hear about it so with conceptual disfluency is another example again from my students this is something called cage of thrones which is just a google click away and most of the major characters from game of thrones are photoshopped with the head of nicholas cage on instead and to each their own the thing about cage of thrones that's interesting is that nicolas cage doesn't really belong in game of thrones at all and it creates a very profound attentional blink and this what's especially interesting is that these characters are so iconic

that if you put your hand sort of over the head of this character you could probably identify who it is and audiences don't even need to see their faces this hairstyle alone is enough of a cue to invoke a mental representation of a certain character and we see nicholas nicholas cage's face instead and by the way to be clear cage of thrones is an equal opportunity photoshop endeavor and no woman man or child who has a leading role in this series is particularly safe i just happen to pick two characters here that are that are popular and there are plenty more and it's perfectly fine to laugh when there is conceptual disfluency but the point to remember is that there's a schema violation and we do recover from them and they can be interesting but what happens that i hope that you would focus on is that there's a shift from automatic processing to a more controlled type of cognitive processing so getting back to games as an example i'm sorry with mario is that some of you may recall in mario odyssey there was a bit of a a

slight wardrobe change i'm not sure if i can say the word nipple gate but i'm saying it chest goombas was also another reference point for this poor mario i mean mario deserves to have nipples doesn't he he's he's he's an iconic character and this look generally had one of two reactions and players did respond in a certain way that sort of violated their expectancies of the traditional norm of this franchise and attached attentional blink shifts our thinking oftentimes to a more deliberate and systematic type of thinking and controlled analytical evaluation oftentimes will require consideration we can move from automatic processing to something else now science also shows us that we can predict human attention so it should be no surprise that there are ways to study knowledge formation through schemas and it again is our schemas that really do wield the power for us to do this

through something called cognitive mapping and these are schemas that players create often very quickly and revolve around their preferences or perceptions across spatial environments they guide and inform player decision making and information processing and they can be automatic sometimes they do require a little bit more deliberate or controlled thinking and in the case here of titanfall this is a bit of an older image now but still relevant players are using their schemas to navigate their game spaces to help them understand streamline thinking where they are where they need to go and what might be in their way so for a game like this i understand that i can jump or parkour my way from point a to point b and through this exact route that's how i'm going to get there if we wanted to give somebody in this building directions to the movie theater you would go down the escalator we wouldn't invoke route mapping you take a left outside the door you cross the street and i'm

totally making that up i don't know my way around this city at all but you get the general idea that we could give somebody directions through this type of schema processing now another way that we see players sort of wrestle with information is through a topographical map maybe in the mind's eye even abstractly about where they are in terms of distance and landmarks and territories so if i'm at ground zero in the case of this game fallout how far am i from diamond city and players ask themselves these questions internally all the time how long will it take me to get there and game maps help players to solidify these working schemas about their environment so through survey mapping sometimes it really is advantageous to or even critically necessary to hand players that topographical map just please be aware that the research is starting to show that some players have preferences and they prefer to learn through route mapping while others have a tendency to sort of lean more towards survey mapping in terms of understanding their environment it's a lot like some people are more kinesthetic learners some are more audio and visual and the data is starting to show that this also seems to be the case for processing information at the game level

through these types of maps now in either case there's a clear benefit to encourage players to map their environment and generate strong spatial frameworks around their schemas so in the case of overwatch where is the choke point is that going to be the best place to put my team how many can fit over there are we too exposed where's the best vantage point and cognitive maps really do align with environmental data and like something like google maps they update regularly based on player experience time and knowledge so we are accommodating our schemas pretty much our entire life in games there have been a few examples where player mapping is not really supported this is one reason why something like mapstalgia came along and i encourage you all to check out this website it's fascinating this is where players reconstruct mental maps of games that they've played entirely from memory and i think some people cheat a little bit here and there but the level of detail is pretty impressive and it also demonstrates the longevity of these mapping schemas that we have and especially when they're tied to

strong emotion in this case nostalgia the the ability to draw upon that prior knowledge is fairly strong and it's also easy i particularly encourage you to check out the resident evil evil map that's in maps algebra because it is absolutely fantastic and the level of detail is really really good so going back to examples where games sometimes don't support this mapping procedure for me personally when i'm playing dungeon master it can be a little bit more difficult to do because the player could create a mapping schema but it's not really supported there the walls all look the same the corridors left and right are very similar and mapping schemes can form but it's not easy today we seem a little bit more able and willing to help players with their mental map schemes early on in fact now our players expect it and they expect it to be done well and this isn't a terrible idea but we have to be very careful with our mapping not to push it too quickly with players we need to in the case of route mapping give them

enough time to get their hands dirty in the environment and explore because that's how something like route mapping occurs and if we just throw them instant teleportation or the epic ride too early that can be potential missed opportunities for them to really sort of get down and dirty with the game environment and assimilate these schemas about space and ability to travel and how long of a distance is between point a and point b now one of the last topics that i just have barely enough time to highlight here in the interest of time is again the predictive utility of schemas for knowledge formation so as an example here i have six chairs and this is a concept known as category judgments we engage in stimulus recognition when comparing what is known with what we encounter and we make these category judgments all the time all day this is one reason why our brains get so tired and here is a very simple stimulus with the example of a chair so i think just about everyone in this room can agree that there are six chairs on the screen

this is not some psychological trick sometimes my students think that i'm trying to pull one over on them but we have a schema for office chair and rocking chair we also have preferences my personal preference is the chair on the bottom row in the middle i love those chairs they're so ergonomic and they're great i'm thinking you can probably guess which chair is the favorite of my students and i will give you a hint it's in the top row and on the far left and that is one repeatedly that they enjoy sitting in now we might be surprised to find that top right top left chair in a game like the stanley parable well it's the stanley parable so maybe that's a bad example but we have these models for recognition based on the schemas that we find appropriate for them and these category judgments can really be useful in terms of knowledge prediction so again getting back to games one way that these oh okay so one way that these manifest is

through what are called template or feature matching schemas and in minecraft players develop a very very efficient working model or mental rule of thumb that you need exactly three stone blocks and two wood sticks and with template or feature matching typically it's a cookie cutter recipe that only a certain combination of objects will get you to sort of the end game here for what you need to do in the game skyrim is another example of this with the golden cloth where the task for players to complete is based on a template and the game hints at what will match and if there is a match the content unlocks and again only this combination will do it or number or correct order of things in games like riven a personal favorite of mine or even monster hunter or any game that sort of throws a wanted poster at players invokes a very specific cognitive schema based on distinct features and what players will do and we can predict this pretty robustly is they will compare the

schema in their mind with what is sort of thrown at them in the game is it a match yes or no and if the answer is no we can predict to a fairly strong degree that they will ignore it or even outright avoid that stimulus and these schemas are activated so so quickly it's almost impossible to see and we can't study it through brain imaging but the point is is that these schemas really do propel behavior forward or backwards i don't want to approach that thing or maybe in fact i do now these types of sort of ways that we work through different environmental stimuli uh-oh there we go can also manifest through what are called recognition by component schemas so schemas help us find to find the basic building blocks of the things that we encounter in the everyday world and i'm all kinds of confident that if i didn't have the reference points on the far left that you could sort of disentangle that messy stimulus or stimuli that are listed on the right

and the world is messy game environments are messy they're challenging they're complicated and through these schemas we have the ability to sort of break things down to their their base levels and and repair them in our mind's eye and it's a little bit more cognitively taxing but we can do it and it is in fact quite possible to do so so with that in mind we are able to when we encounter things in a game see let's say for example these two models here and these two objects we can understand through sort of our prior knowledge these are the basic building blocks of the halo rings or a a train that is not feeling so good in fallout in fact a good horror game in limbo is a nice example here we'll often take advantage of players schemas especially in this way and we don't have to show our entire hand and that's the beauty of using these schemas to really inform game design because a partial queue is enough and schemas will fill in the box for us they

will take over and particularly in reference to monsters players don't even need to see the entire object in order to understand the danger that's involved as you see here devs artists and designers often allude to this and they're invoking this type of recognition by component schema without really knowing what to call it and we can do this with sound we can do this with light it doesn't have to be a purely visual stimulus we can use these to sort of invoke activation thresholds and imply through a sound cue that gets stronger you're getting very close to the loot crate and we don't have to show them the entire thing a a change in light will sort of guide players one way you're nearing the end of the maze but you're not quite there yet and while schemas are very good for encouraging certain player behaviors they're also very very efficient for other things as well and one of them is to help them establish characters or even entire storylines so one way that this manifests excuse me is through what are called configurable models and through these

schema models as you can see in this case of assassin's creed as long as the deviation isn't too far and i'm thinking now of poor mario again the idea of the mental model in in the case of assassin's creed with this protagonist is that sort of every game every every new game in the series there's a little bit of a change but it's relatively the same there's always the the cape and cowl and there's some type of gauntlet and there's usually boots involved so these characters are iconic and we accept that they change over time as the series changes over time but batman can never have a red cape like superman and neither one of them is really allowed to ever wear overalls unless superman is back at home in smallville i guess that wouldn't be a total deal breaker but when deviations steer us too far from the established prototype players oftentimes will say no that's enough it's it's orange juice and chocolate chip cookies i'm not going to do it and it is rejected as a concept in the

case of elite dangerous clearly what we have here are configurable models that help to communicate to the player that there are different achievements and they still look simple enough to the achievement prototype but clearly these are different and the overall concept is still easily understood and not too much of the challenge mentally a more complicated example is what you see here with overwatch and other games as well these configurable models may need a little bit more explanation but at the end of the day the achievement schema model that players have assimilated it still sort of is with them in their backpack and it maintains fairly good consistency over time as the achievements increase the design changes a little bit but players are still able to accommodate these changes and really sort of go with it again in the in the interest of time there's a lot of other things that i could say but i sort of need to wrap this up here i hope that i've been able to share just a few ideas about how schemas do manifest themselves in games and i'm going to leave you with a few key points

here one is that schemas are very very important to help us understand what it is that players are doing and predict how they are going to work with the information that we throw at them onboarding attentional focus this is obviously a huge domain to explore but i hope that you would consider them from the perception of schemas because schemas really do play a very important role and it is okay to violate those schemas in fact it's pretty fun to do it and twist players expectations and again i'm thinking of the stanley parable it's just on my mind today but we have to be very careful with this because what can happen is an over reliance on different types of thinking and they are more systemic deliberate and more controlled types of thinking sometimes we want players to be on autopilot because that's a really fun way to throw a wrench in their plans and if we are violating their schemas too often or too hard that can really cause a problem a flow is an automatic cognitive process

and schemas are very much within that realm of flow and help players to achieve flow there's a few golden rules with flow that some of you may know we need clear and immediate feedback minimal distractions and so on but in terms of schemas i'll just present sort of one final point here if you want to virtually guarantee the absence of flow in your game it's really easy to do all you need to do is place your players under heavy cognitive load because what happens is that is demanding that the player activate schemas to sort of keep up and change and accommodate and the point that schemas help players the most is when they are trying to reduce cognitive load and if we invoke them too hard the task becomes too difficult and we want players to be boosted in their ability to sort of work with new content too much stop and think will also get in the way of that process and automatic schema activation is absolutely one of the keys to flow experience also seems to matter and we can think about chess where controlled thinking and strategy is super important

and sometimes players have to work at that level but as they become good it becomes an automatic process but at the end of the day loss of immersion does happen when players stop and they have to ask themselves what am i doing and where am i going and controlled cognitive processing is not bad but when we are interrupted when there is uncertainty when we don't know what's going on there's a change of shifting from controlled sorry from automatic to more controlled types of thinking control thinking is not bad it just is not the optimal condition in which to invoke flow so i hope this talk has been informative i am really glad to see psychology manifesting its way into games and games design kind of a bigger level i think this summit is fantastic and if you aren't on board with psychology it's fun and it's not scary and you should join us we have cookies we probably don't have orange juice but there's plenty of things that we can still explore so i think we have just a

few minutes if there are any questions i'm happy to address those we're going to get kicked out in a minute or two if there are no questions thank you uh yeah first of all that was an amazing talk uh second of all uh something popped into my head when you were talking about route mapping versus survey mapping i wonder what your thoughts were on how the legend of zelda breath of the wild treats its map because it seems like there's a lot of both route mapping and survey mapping in that and it does let you play around a little bit and get your hands dirty there are a lot of things to say about that particular game one is that it is an example that attempts to sort of hit the mark for i i'll just say both player types that tend to really enjoy route mapping and then others that prefer survey mapping and they're kind of throwing everything in the kitchen sink at the player and letting the player decide which i think is fantastic

i don't know how effortful that thought process was in designing that concept but it seems to be working but for some players it doesn't really work at all so how on i say this to my students all the time how on earth do i make a game that's going to make everybody happy and they seem to be hitting the mark enough and it seems to be working enough i look forward to the day when we think of the different schema preferences from the prototype stage from from the first stage that partially answers your question thanks and i'm happy to talk after by the way hi speaking of schema subversions like in stanley parable are you aware of any sort of research or work being done into seeing how games can work to help people challenge or rethink the schema that they have outside of the time that they play games you know about the world or themselves

or society beyond the game environment pro-social gaming right there are a lot of pro-social games right now being created and pro-social research utilizing games i don't think we're quite there yet but we're getting very very close and if you would like to become a cognitive psychologist and help us with this endeavor that would be fantastic because these are the questions that are now being started to kind of revolve around the community and i certainly don't claim to know all of the research but the pro-social side of gaming seems to be very high level right now which is great and you're talking about something that is distilled down we will get there we're just not quite there yet okay thank you hi um when you create are creating a new system and say you find that it's sufficiently distant from existing mental schemas that it causes a lot of this dissonance and all these problems uh do you generally find assuming you can't stack it directly on top of those schemas that's better to get close and have those exceptions be a

little bit outside the norm or to go a little further out and have the person have to create new models for that entire thing that's a wonderful question dark souls comes to mind just off the top of my head your dad your dad your dad your dad you're dead some people responded very well to that others needed a bit more time to adjust sure stanley parable we've completely violated expectancies i think the important follow-up to that is how we design after the fact because sometimes we just go for the punch of this is totally not what you expected and then what i think that's an important consideration to take into account i love a good schema violation it's better to maybe start a little bit smaller but sometimes devs will come out swinging and it can kind of go one of two ways know your audience is important and um good ux research will also help you get there i will say thank you thank you hi um

i'm too short so i'm worried that this is too similar to the last question but i was wondering is there a way to overwrite existing schemas without causing schema discomfort or at least a minimum amount of schema discomfort and if so what how would you go about that i would like to think that hopefully i converted one or two people in this audience that psychology is really cool and applies to games so if i've won even one person over i'm happy that's a fairly easy example to follow up on your question reinforcement is really valuable because if the player is encouraged to think a different way it's a little bit harder for us cognitively to do it places a little more cognitive load on us but if the rewards are there and i don't mean a carrot on a stick it could be an intrinsic motivator it could be group based that's a really efficient way to sort of encourage players to keep a more open mind and it's it's a that's a very large question but uh reinforcers come to mind as kind of a

big helping hand with that thank you it's a great question hi um so first of all thank you and i have two questions which i think will have the same answer but uh one is that was a lot of information really fast and my hand hurts from trying to take notes so where can we go to learn more and also i'm a consultant for companies that don't usually think about games and so it part of what we do is go out and try and find evidence that we can use to back up our designs from ethnographic research to other examples in the industry and so this is why they might have the same answers like what would you recommend looking at to help justify using some of these ideas well i do know or i would hope that because this is being recorded this talk should be made available in the vault that would be my hope the slides will also be made available as well if anybody can't wait i'm happy to email you the powerpoint slides at any point so that's certainly an option as well and the second part of your question was

um where what are well actually you kind of just answered that as well but the first part was actually where could we go to learn more like dig more into this oh well ux is gathering momentum ui research psychology sort of these two fields are colliding which is great we have the traditional academics who study schemas that are just now starting to sort of transition into applying this more to games so you're a little bit we collectively are a little bit ahead of the curve here which is fantastic but in terms of a book i i will direct you to celia hodan's wonderful book on supposing we've already ordered that that is a very very good resource we need a million more books like it one just on schemas would be a very large volume so the research is getting there the academics are coming but right now a good google search is probably your your best bet always you can feel free to email me i'm happy to sort of give specific resources for more

specific types of questions and i can sort of wade through that research i do that a lot for people who getting get in touch with me again great thank you all right i think we're going to get kicked out of here so feel free to come talk to me and thank you for coming you