Brain-Computer Interfaces: One Possible Future for How We Play

[Music] on Friday last day of GDC my name is Mike Amarinder I'm an experimental psychologist at valve I'll be speaking to you guys today about brain computer interfaces and why I think there might be one possible future for how we play so what is it experimental psychologists do it foul basically this so maybe not so I'm not on the clinical side I I'm on the research side of things I got this picture because it gets a laugh makes people you know happy at the start of a presentation what I've been giving it I use this picture at the start of basically every talk I've given for the last five or six years right and it reliably does get a laugh and so I want you to think about that for a moment what kinds of things could we do as gameplay designers if we get elicit reliable reactions or if we could determine we were eliciting reliable reactions to various things our games were doing think about the notion of reliably knowing that somebody is

experiencing a particular sensation and how that might change how you design games and play games so that this is the question of motivated the talk how you interact with technology in 10 years all right what is the future of computing what is the future of gaming what comes to mind when you think about consuming entertainment in the future right is it mobile is it VR is it a our right as it screens everywhere is it something like Minority Report now you know we're in the vision track this is we're supposed to be speculative to be to be forward-thinking and so what I'm about to cover is one possible direction that things could go so here's a quick overview of the talk we'll go quickly through current generation interfaces and then I'll talk about three ways that I think BC is could significant impact us as game designers what's that about ways to improve play testing we'll talk about a gap adaptive gameplay and then we'll talk about what I guess I'm referring to is true BCI but actually replacing current interfaces with interfaces that interpret your thoughts

and translate them into something useful and then we'll end with a discussion of the challenges that the roadblocks the things that might get in a way so let's define BC is this is my definition but you'll see similar you know definitions kind of wherever you look but basically it's a communication pathway that translates neuronal signals so things that are happening up here into actionable input for some external system right could be a game could be a screen could be whatever I also want to make the broader point that well I'm going to describe or most of the talk will be about measuring neurological activity truly what I'm talking about is acquiring physiological data from a player right and asking the question is what changes when we can incorporate information about a player's internal state what they're feeling what they're sensing what they're experiencing what they're thinking and incorporate that into our game designs so again while we're gonna focus on information acquired from the brain there are other modalities methodologies that will give us knowledge about you know the uh player's internal state and

we're going to cover them briefly another point to keep in mind is that in one respect every interface is a brain-computer interface right we're always translating our thoughts into a system response it's just mediated by a piece of hardware right so what differs is the medium through you know with how we process the information that is an intent and turn into something usable and so we might be able to accrue a lot of the potential benefits I'm going to describe today by not measuring great signals directly and what what really matters is having a reliable way of according recording information about a player's cognition or emotion or experience and so I'll describe other ways we could do this briefly but really we're going to focus about brain computer interface stuff today because I think in the long run this will give us the most bang for our buck so I'd like to kind of walk you guys through an analogy if you think about having a conversation with somebody you know there are both verbal and nonverbal parts of the conversation right the verbal the words that you say and the nonverbal right like the the change in somebody's tone of voice right if they're facing you or not their facial

expressions right and when you get verbal and nonverbal information you put them together you get a complete conversation and so the the I guess the the point I would like to make is that right now with games we have traditional inputs but we might be missing the nonverbal part of the conversation right there might be other data that a player could be providing to us as game designers that we're not acquiring and so what happens when we add in physiological data is it possible that we could make new kinds of games I'm gonna spend the next few minutes trying to convince you guys that yes it is and I think this would be a useful analogy or useful framework to keep in mind for the rest of the talk so let's talk about current generation interfaces you know broadly speaking you're transforming some intent I want to jump I want to aim to the left I want to move forward I want to reload can you combine it with you know current generation control mouse the keyboard and so forth and it turns into an on screen action and so let's briefly cover what we can do with current generation technology all right mice and keyboard this is actually my my keyboard my mouse with

the mouse you get a one-to-one mapping of hand or wrist location to allocation on screen you can shift directions you can you have button presses as well keyboards transform keyboard presses and on-screen actions you can navigate with Wasley you're limited in the number of things you can do by the number of keys but also by player memory right you know keyboards could give you you know with combinations of keys you know thousands and maybe even tens of thousands of possible inputs humans can't remember that many not to play a game so memory is actually a fundamental limitation here so how many possible combinations do you think you can remember off the top of your head when you're playing a game 10 50 100 there is a limitation and so a question to think about is what if you didn't have to remember everything what if you could just think about what you wanted to do and it happened like would that change how you play games so the gamepad I work for valve since the steam controller there also lots of other great game threads out there again

we have buttons that you know mimic kind of what we get from keyboards and we have analog sticks and touch pads to mimic what we get from a mouse analog sticks instead of being one to one mapping give you a velocity vector right you get a direction that our displacement and you have D pads for more binary actions we also have guest role controls right things that let you make more naturalistic movements we boxing you can actually box so you get a wider array of possible movements and more naturalistic which is kind of intuitive to like I guess in kind of a will be a useful memory cue it's also petite you know moving your hand and your wrist as not as not as tiring is moving your entire arm a quick summary of what we just discussed is you know getting remapping intent on screen output we're doing a reasonable latency you have to think a thought and translate it into a finger movement or hand movement or a wrist movement we have restricted bandwidth we can only do so much at once how many different cues can you press at once

we're starting to add a widening of Raving for patterns and we're are getting more naturalistic movements okay and these are great right like they they work for current generation games and that's fine but the the I guess the question I'd like to ask you guys is what happens if we didn't have to use them how would you imagine an idle interface like think about ways to improve current interfaces or what new kinds of interfaces might be what are better ways of interacting with games with pieces of technology right are we at optimal is this it or can we do better what is your endgame for an interface what is your dream interface like what are ways things could be improved so if you couldn't come up anything and like the eight seconds I gave you to think of stuff I made my own list so you know interfaces could respond quicker I could allow a much broader space of possible input commands we could have complex patterns of input we could have chains think about you know doing thing oh I want to undo the last thing I did and rotate this thing that I did and whatnot

we could also maybe get additional axis of information from the player and so I want to drill into this one and this is going to be the focus of the talk right with BCI is we can make progress on all of the I guess the top four ways we can improve and improve or potentially improve interfaces but what I'm really curious about or what I'm really fascinated by is what happens we get additional data from the player we're not getting with current generation interfaces so what could we do if we had more or different kinds of information from the player right this is the question this is the question that kind of motivates this research what about all the data were missing from the players experience right we know what players are trying to do in game for the most part you know if they're jumping or reloading or selecting a character right or trying to attack but we don't know how they're experiencing it so are they happy or are they sad are they engaged are they detached are they challenged are they

bored are they frustrated you're curious are they exploring or solving puzzles are goofing off or playing seriously are they learning something new are they learning something old are they distracted are they remembering something are they speaking to a friend of the getting angry are they getting happier all right so do you actually need any of this you know maybe not but I would like to find out I guess like like I pollicis I'm going to make in this talk is that we would get an improved player experience and a qualitatively different player experience if we had access to internal states and emotions and cognitions and decisions and so I hope the end of this talk you can kind of see at least why I believe this and maybe you might feel similarly okay so let's get to the the good stuff here's a picture of me wearing my suite open BCI EEG headset it's open source company there there are other EG headsets that look a little bit different but i kind of like this one because it always makes people ask me what i'm doing when they see me wearing it so again broadly speaking a BCI or at

least as I'm using it here right is a way of transforming what happens up here there's something usable for a system out here alright so we want to interpret physiological data and turn it into something useful and so well again I am referring directly to measuring neurological signals right again truly anyway we can reliably detect emotions or decisions or intense or thoughts you know could fall kind of like under the space or under the I guess well could be used for what I'm about to start describing so this is a cool looking picture of a neuron you guys might remember neurons from your eighth grade biology textbook neurons are nerve cells they are the atomic units of the brain I promise this is the only I think biology slide in the talk yeah so neurons are the atomic units of the brain if you put enough of them together and you organize them in various functional and hierarchical ways you get a brain right and so they made you communicate by firing or not firing they send electrical signals down various pathways and processing and the cluster together in various brain

regions and seem to person you know provide distinct substance of functionality although not always about a hundred billion in the brain and connections between neurons is called a synapse they're about a thousand trillion or 1 quadrillion sent out to connections in a human brain and when neurons fire and this is the important part they produce every single aspect of conscious and unconscious experience every thought you have every sensation you have everything you perceive every feeling you have they're a consequence of neurons or bundles of neurons or patterns of neurons firing together or not firing so in some respects were actually already inside the matrix like what we see an experience and feel right is constructed by neurons firing up here right and when you have you know we have a thought neurons fire in a particular way when you have a different thought neurons fire in a different way there is a neurological correlate our origin for every single thing that happens in our

brain but our reality is constructed by these patterns of firings and so if we could reliably measure them then maybe we could start doing useful things with them so just to drill into that a little bit deeper what are we actually measuring patterns of activity in the brain whether they're temporal so happening over time or spatial where they are happening in the brain and we're also measuring responses to external stimuli what I see in here and feel and touch or imagined when I'm thinking right all of these will produce you know all these will induce neurons to fire so what we're actually recording right is you know firing rates of neurons or firing rates of systems of neurons or blood flow to areas where neurons fire we simply want to understand like locations of increased levels of activity in the brain and hopefully what they are and if we can take these patterns of activity and describe them in terms of something a player is experiencing that's the trick we need to say okay this pattern of electrical impulses means happiness or I

mean sadness or it means I wish I had reloaded a fraction of a second earlier or I see somebody doing something cool in a game and I wish I knew how to do that so how do we measure this the sweet BCI headset or EEG headset there's some lower left it's an EEG I'm gonna everything that has an acronym I'm going to try and say although I think I'm going to mispronounce them so you guys can laugh at me if I do you GSR lecture and sefa encephalo grams they measure electron tivity in the gray in the brain the one in the middle is f nears its functional near-infrared spectroscopy it's measures light scattering off blood flow and then you have fMRI functional magnetic resonance imaging also measures bless or essentially oxygenated blood flow and these are ways of detecting essentially just neurons firing I lead to other physical signals we could measure heart rate galvanic skin response essentially is a change in the

pH composition of your sweat that's correlate of a physiological arousal pretty reliable pretty useful eye tracking we can see we're looking in a display get an index of attention we have facial expressions which indicate a motion we have muscle tension which is an EMG electromyography sensor that can also kind of give you insight into players of motion and then you have posture which could give you guys your social cues and perhaps measures of engagement as well all of these might be useful and all of these might get us a long way towards kind of like the things my budgets about to describe but we are gonna stick with with BCI's and I guess that's the last term uh give you guys that caveat so I've also been talking primarily about the central nervous system which refers to what's happening up here in your brain we also have a peripheral nervous system right we have nerve cells that go to our fingers go to our fingertips and reach down to our toes people have different estimations of how long it takes a signal to travel from your brain to your finger one number to

use could be 100 milliseconds what if you could intercept the signal right here at the wrist and say yeah we know that you means you want to turn your your hand this way move the mouse this way what if we could shave off ten milliseconds or twenty milliseconds or even 30 milliseconds off of your reaction time you think people who play games competitively would want that what if we could predict your movement before you made it would that be interesting there are people working on this right now seems possible another interesting thing to think about with BC is as I've been talking mainly about the reed side of the equation what about the right side of the equation what about inducing so it so the reed side is interpreting brain signals what about inducing them or creating them this already happens with perception and cognition right when games show you something they're inducing patterns of neuronal neuronal activity when they make you think something you're reducing patterns of neural activity you can also do that directly um people are investigating this now this next slide might be the most important distinction

to think about when we're discussing BC is you can create interfaces that communicate non-invasively outside the brain and you create interfaces that communicate invasively from inside the brain or from underneath the skull from inside the skull lots of things we can do we can do not invasively EEG is electroencephalogram MPGs Magneto and cephalo gram we can also go invasively with electro credit Congre fee or a stereo electroencephalograms a lot of the things we're gonna discuss again we can do non-invasively but some of the things i'm gonna describe we might only be able to do invasively how many of us are willing to have somebody drill a hole in our head to drive a benefit maybe not too many but in the future who knows we're gonna discuss why people might be willing to do that so here's a EG data this is you know the extra data we can turn in to still be useful this

is from the very first recording I ever made so you know my brain waves you can see that electrode number eight was not seated correctly so we'll never truly know what I was thinking at this point in time but the trick here is to turn this into something useful Sibley divisive data recorder over time one way and I want up too much analysis slides in here but I do want to kind of give you guys some insight and so what you can do is there's like essentially that the firing rates of electrical activity in various parts of the brain and you can categorize them delta waves are slow and beta waves are quick you can see where you get more or less of them in various parts of the brain and so you can take these patterns of activity and again if you reliably measure them you can turn them into something that says yeah a player is feeling this or is thinking this or is too intending this there are lots of signal processing techniques to do this machine learning and deep learning techniques are very useful in this space because we're dealing with noisy data and a lot of work needs to be done but the visit this is one way that people kind of try and make sense of what's happening so with me C is what can we

actually measure today and measure non-invasively these are just you know headlines from journal articles here's an incomplete list we could measure someone's learning if they're learning something and their ability or faculty to learn we can measure surprise or novelty committer if they're excited or relaxed good measure effect which is just what psychologists refer to as kind of the direction of emotion whether it's positive or negative looking rather whether they're attend you know what they're attending to something if they're engaged if they're bored we could measure responses to in games Timmy lie and we're not always getting these signals reliably but we're starting to figure out how all right so just think about what she would want to know about your players there is a long list of things that we can get right now with current generous and technology current generation analysis and current generation experimentation so I guess I'm going to make the case that there is

increasing BCI interest these days and the reasons for that would be you know we're getting better at imaging you know detecting data also its stimulating which is the red side of the equation and also the analysis like deep learning is very helpful here also we're figuring out ways to aid people with disabilities people with sensory impairments or motor impairments or cognitive impairments we're given the ability to yeah how people see again and can't see and hear again who can't hear and manipulate things again if they've lost a limb or if they're paralyzed there's also fears about general AI people have those concerns and they'd like to figure out how humans can compete and so they're looking at BCI's and people have a general interest in augmenting human capability can we push ourselves past what we we are now it gets a little hairy but again it is driving interest there's also one more factor that was kind of important because that we're gonna measure brain signals you need a way to get people to wear a helmet so only we we us as game designers had a way of

doing that well you know not everybody plays VR but one advantage that the VR and AR potentially have as well is you're getting consistent contact or at least semi consistent contact with the source of brain activity so you might be able to do interesting things if you could commits people to wear a helmet that had eg sensors all right so let's get to the interesting stuff that was an overview I wanted to get everybody on the same page to see people kind of understood what I was talking about well I'm gonna talk about for the remainder of the talk I'm gonna talk about I guess three separate areas of interest how BCI's could could you know change the way that we design games and play games and our players play games so we'll start with play testing improvements then move on to adaptive gameplay and then we'll talk about the end game actually replacing interfaces and you know play this thing adaptive gameplay can be done now the information placements will what will be in the future but in some cases are actually starting now so we'll begin with a play

testing this is an example of a test subject playing Team Fortress so play testing is how it's how valve makes our games we gather data from people who are not us and then iterate and improve the game as a consequence I imagine it's a lot lots of people make games predicting is great here are a few traditional approaches direct observation you're watching somebody play the game you can ask you know people questions afterwards or sometimes during you can give people surveys game metrics or how we test on the player base at large look at look at kind of some measurable human behavior and also doing usability to focus specific aspects of what's happening plan testing is great there are downsides if you ask Google questions they rationalize they invent they can fab you late and conflate and they forget things they also only measure what we record right any behavior we're not recording we don't know if it's happening and it can be hard to know why sometimes as a psychologist I'm happy to walk you guys to all the ways that people

understand why they do what they do but that's a separate talk we also lack moment-to-moment insight about what a player is experiencing they can tell you in the moment but that changes the experience they can't tell you everything and large sample sizes are impractical right we can bring in 30 people a week maybe a hundred people a week what if we could play tests on two 300,000 people a week or 3 million when we become better game designers so let's walk through six specific ways that BCI's could be interesting how they might help us get moments moment insight to what our players experiencing how we might get more objective data and do it at scale the new kinds of data we could get how we could measure things over time and then how it becomes I guess a source of way of giving us more confidence as we could let it let it kind of work with other methodologies we currently have so moments move an insight to me this is the the thing that

I guess most excites me I guess at the moment right because we can do this right now and get data we haven't had before we get a real-time understanding of a player's emotional state moment by moment second by second we could start freaking out responses to any individual components of gameplay so think about that for a second you can understand how a player reacts to a specific enemy or a chat message or a bullet that's fired or if they die or if they kill a character they choose a character or if they see a piece of an environment the environment they like we're gonna lemonade gross responses like oh I like this yeah the game was you know 5 out of 10 or was great or wasn't we could determine the components of a player's experience that lead to the overall impression in essence we get to ask a question at every moment of the playthrough and we get an answer without interfering with the experience I think that's spectacular so this is what I'm most optimistic about now you know I'm presenting a rosy you know a rosy

picture right like work needs to be done right but with current technology that this is possible so get more objective data right internal sensations are not interpreted by the player right they're not conflated or created Naumann we have one memory you remember memory problems as I said right we know if someone was feeling challenged instead of just hoping that they were apply to our question honestly right we could truly knows if a player was feeling you know we're doing game balance right and we asked them to play dist was that hard some people don't like being honest we would know think about this instead of trying to induce a particular emotion in gameplay like is a player afraid are they excited are they surprised what if we could actually medal right like we haven't we're designing within ten we're trying to induce particular emotions and sensations we could actually know if a player was experiencing them we also do bus at scale we derive so much

information from bringing people in and play testing involve but what if we could do that on everybody we do get them but we have much more information about how people were playing and consuming our games at the moment play this thing on the population means you look at you know changes and measurable player behavior like twenty percent of players are playing this map what not but if we could get the advantages of direct observation and of internal state off to everybody we'd be much better game designers right if BC is or any form of physical measurement were prevalent and all our players had access to them we had access to that data we could always be play testing on everyone and be asking every question we care about maybe not every question but a lot of the important ones we're also getting new kinds of data right data we've never had before that we're no longer limited by the behaviors

we can explicitly measure we can start trying to infer rationales behind behaviors looking for positive responses or negative responses and whether a player is more or less engaged we can get more accurate measures of overall sentiments and we're getting much more granular measures of individual sentiment and more granular measures of how players are responding to individual gameplay components here are a few examples we get predict when someone is about to quit both the session and maybe forever you better understand game balance we could see a form responses correlate to overall player sentiment alright this isn't something I've been I guess fascinated by for a very long time right is what we're seeing people write on forums and in emails is that a loud minority or a loud majority we could better understand if the data were getting from forums and for people writing in actually correlates to how people are experiencing the game out in the real world we could better

understand impressions of a new weapon a character of cosmetic I map a game mode can figure out what parts of an update somebody enjoyed and how they feel what the game's interface and figure out which players were toxic which players are good teammates we can find out and was satisfying home as a game play and the least well parts of the environment do they hate and how hopeful the tutorial ends up being weaned also compare responses over time we could look at sentiments pre and post update and assess sentiment we could estimate a game's lifecycle you can do this with surveys and other methodologies but effort is required what if we knew how someone felt on their first session how they felt on their 10th how they fought down their hundredth and how they felt on the session that was the last session of your game they ever played very useful to understand how a player was experiencing your game over time and how their feelings about your game change

over time and it becomes another useful tool in the toolbox we can combine you know data about someone's internal state with every other data source we have gives us more ways to forget the why what is occurring so we become smarter game designers this is me trying to relax I wasn't relaxing I was trying to relax this is data from me seeing the unusual weapon I made in tf2 if there's unfamiliar unusual ZAR weapons in tf2 that have particle effects I made I like seeing it it makes me happy this is me getting a kill on tf2 so the question to ask is where else do I have a similar response to this right is getting a kill the Sam is seeing my unusual there actually are some similarities here we'll go into don't get into the weeds on that and I know it looks a little bit different but both of those are positive reactions and so the

trick here is saying okay we think this is a positive reaction where else do I have similar responses in game if we could do that good start saying yeah these are the things that I like about Team Fortress 2 and maybe these are the things I don't so that lets play testing let's go on to step 2 this is step 2 and our BC a timeline the second major way the games can be impacted what if we could reliably measure various turn all states what kinds of gameplay could we begin to create this is a slide from our well yeah I guess a great slide about left4dead we're just gonna came out a long time ago but wasn't one of our first attempts trying to design games and were adaptive so the middle line here is essentially an estimated arousal level for the player right you could see it has some some peaks and valleys and the goal was to generate changes not arousal level by increasing the number of zombies that were present the desired population of

zombies is on top the the colored zombies are boss monster or the boss monsters are special infected and the bottom was the actual zombies that were created so the goal here was to create you know a series of peaks and valleys for the player right was that you know is that the right pattern of arousal why who knows but what if we could reliably measure that and figure out what the right pattern of arousal was and have a game adapt to what a player was experiencing how would that change how you make games what if all you knew was a player was experiencing positive and negative emotion how would this change game design could we start placing reactions in context just think about think about everything you can do you just need four players for them positively or negatively towards the game or even better a specific aspect of the game and we'll go through some examples and obviously you know we don't want to eliminate all negative examples positive things are positive by comparison to negative things but we

could definitely adjust the balance we could figure out ways to mitigate the impact of negative occurrences and heighten the impact of positive ones so the thing about having adaptive enemies what kinds of enemies do you like playing against in game what kinds of enemies are challenging what kinds are boring if we knew the answer to these questions we could have the game give you more of a challenging kind and less of the boring kind instead of like difficulty levels we become a thing of the past right you don't have easy eto casual novice advanced hard hardcore nightmare whatever we have dynamic difficulty adjustment right the difficulty that's designed for you to give you the challenge that you're craving at that point in time sometimes you might want things harder sometimes you might want them easier we'd be able to figure that out what is it like to only be able to kill an enemy while you're relaxed maybe it's nothing maybe it's a gimmick maybe

it's something different a new kind of gameplay so as we walk through these examples it may be the kinds of changes I'm discussing they won't quite work with conventional game designs and that's fine and so let's start to think about new kinds of gameplay experiences that might be better suited to make use of this data is their adaptive enemies its thing about adaptive teammates what kinds of teammates do you like playing with or hate playing with well parts of the game do you enjoy doing we can pair you up with other people who like doing the things that you don't like to do and form work better and more cohesive teams if we could identify toxic players or your feeling that somebody's a toxic player we could automatically mute them or keep you separated what if you knew who your teammates were feeling and what if they knew how you were feeling again I'm asking questions not because I necessarily necessarily have the answers but I think that BCI technology makes them seem like interesting questions to ask what about adaptive rewards you know examples of a few Awards in counter-strike we have new items badges

friend requests social rewards XP we could figure out what kinds of rewards you like and the ones you don't we could start tailoring rewards to the individual player so we would avoid the things you didn't like and give you more other things you did right the female wouldn't care whom is that games in some sense become with us static and obviously games are not static but on one axis they are right they could be a lot more adaptive they could learn more about you and they could respond individually what you're doing and how you're experiencing it what about avatars that mimic your mood what if in-game characters reacted to it what if you had conversations and interactions the more closely mimic reality I just imagined my list of conversations alright we're not gonna solve a Turing test tomorrow well we could change how you interact with NPCs with other players they could see you are feeling frustrated or if you were happy or if you're sad just like true

thing about it that you know games right now we do our best interactions to get a whole lot more realistic and engaging here's an image from dota just take a look at it what aspects of would you like or dislike which heroes are characters in the game do you like or dislike how do you feel about the trees are the flames the health bars the combat the strategy the tactics all of a sudden we start becoming able to assess how you're responding to various elements in-game and maybe we could start having the game change things we could make small changes we could make big changes how would you even make medium-sized changes so this is the question how does game play change what you know what the player is experiencing like what aspects of the game could we change if we knew how the player was responding if we knew if a player was frustrated or engaged or feeling

challenged or immersed what if they were in flow the game could respond to the player as a consequence of knowing what they were doing we could give you cues when you're frustrated could give you challenge when you're bored we could figure out what keeps you intrinsically motivated or better understand what we're failing we could figure out where extrinsic motivation is unnecessary or unneeded when you figure out what leads to flow or immersion and when you're in it and when you're about to become in it we get improve accessibility to figure out when players needed assistance we can enlarge text change audio a huge change gameplay reduce complexity would you figure out what a player needed and respond accordingly here's a screenshot from Valve's best-known game Alien Swarm came out a while ago it's a top-down shooter awesome game a lot of fun to play this is an experiment we did a while ago

where we use physiological signals physiological arousal as an input to the game we gave you four minutes to KO 100 enemies and the timer would take down quicker if you got more rest if you got more excited which would then cause you to get even more excited and then it would take down even quicker and so you get a negative feedback loop and so you know it's one design that may or may not actually end up be suitable for consumption but you can start thinking about novel kinds of gameplay experiences when physiological data becomes an input a direct input to the game imagine a spy game where you actually have to be a spy and four people with your voice your words your behavior and your thoughts what about a true role-playing game we actually had to play as the character Europe you know we are currently pretending to play as okay maybe these are gimmicks right that's fine maybe there's something different something cool could we teach players

how to play our games more effectively how great are we designing tutorials right now maybe we're really really good we could probably be better we could figure out when a player was learning or in an estate that was amenable to learning right let me figure out why they learned help to learn you better understand one of the components of skill in a game and teach them to other players and conversely could we teach games how to play our players more effectively I could spend hours on the slide but I won't we have limited time remaining but think about you games game dynamics they're no longer static if we had access to this data you know we start discovering what the player enjoys or doesn't we're making fewer assumptions about the players intento preference and creating a broader range of experiences for the player right just as players respond to the game games can start to respond to

the player if these are the kinds of gameplay experiences you like we can start to give you more of them less of the ones you don't and games will start to learn how players want to play their games and as a consequence gameplay becomes adaptive and personalized we start modifying aspects of the game the difficulty that you know individual components of the gameplay the rewards the teammates the weapons you get which is the visuals to sound the interfaces all of us can adapt to what you're feeling and experiencing all right so gameplay is no longer one size fits all it's tailored to the player it's no longer a game for everyone it's a game for you step 3 interface replacements the true BCI the end game so we're gonna start getting speculative or at least more speculative than we've been so far so let's go back to that question how will you interact with technology in ten years right well

when you say be static people sometimes they think no you're gonna sing a thing and it happens removing the middleman right now we have human intent translated via physical interface induces a response in the system so what happens if we remove the physical interface we have human intent it just directly leads to a system response if we remove the middleman what becomes possible if we get reliably understand what you were thinking or feeling and have a system respond how would that change things right this may be decades away but people are doing things now that imply that it might not be when this is one direction human intent leading to a system response we have to go back the other way too the system has to respond to the human so it can do so via you know current display technology or I want to remove

the middleman from that direction too we need to figure out how way for the system to send a response back to the user directly without mediation so how would we do that well we can already kind of do that you know what do you see and perceive you know which you which you taste and feel and hear what you think all of these lead to neurons firing but what if there was another way there is I've kind of alluded to this before you can make neurons fire directly you can communicate with neurons by generating an electrical current in the brain neurons will fire so you can make them fire all on your own or all on their own this is one way of doing that this is a TMS machine stands for transcranial magnetic stimulation it uses a magnetic field induces electrical current in the brain and this often used in research to increase or suppress a neural activity in various parts of the brain and they used it in for therapy or treatments insomnia anxiety depression PTSD whatnot

if you're curious what it looks like the CTMS machine in action yeah yeah exactly exactly how it goes all right so what's possible when your movie interface entirely we're gonna go back to the diagram you may have seen you remember from earlier in the talk five ways that BCI is could change you know the way we interact neural prosthetics understanding restricted forms of cognitive intent for new ways to augment perception or cognition and then the true endgame being able to enter the matrix so neuro+ that these refer two ways these are sensory or motor replacements all right vision think about mission it's simply it's a contract of neurons firing right light passes through the retina it's transmitted to various aspects of the visual cortex these neurons fire in particular ways and we get the perception of sight what we're seeing right now if your retina fails we could

bypass the right now and still give you site if you make neurons fire in ways that you can understand and interpret it would be the same thing with cochlear implants how people with auditory impairments here again thing but people who are paralyzed well if we can stimulate neurons in the motor cortex we don't need signals from the peripheral nervous system people are doing this right now trying out paralyzed people walk and blind people see what if we understood we were mentally what you were thinking we could replace a mouse or a keyboard or gamepad they're probably people in this room who are working on that at this moment being able to type with your thoughts navigate through an environment with your thoughts make a selection with your thoughts what about augmented perception could we make you see an infrared like the predator can we increase contrast sensitivity have you seen with a higher resolution I give you access to other

spatial information like echolocation good make you navigate like a bat could we add a sense could we improve your sense of touch lets you hear our different frequencies taste and smell new things you've never tasted and smelled before people are looking at this - what about augmenting cognition cognition is tricky there's a lot we don't know what if we could focus attention just make you better at attending to what you wanted to attend to what if we could stimulate appropriate areas of the brain and prevent you from being distracted what if we could recruit neurons for the processing tasks or increase your short-term memory capacity make you have more in memory at once that would be incredibly powerful what if we could improve memory retrieval or the resolution of long-term memory if we could help you learn quicker or more effectively or speed up your reaction time to various stimuli you help you interface directly with machines if you treat insomnia be as

some sort of BCI interface are you augmenting cognition to me the answer is yes that's happening now and then there's the matrix the endgame right calling it like an augmentation of experiences on you know it's under selling it but if neurons are truly creating every sensation that we have well if you could stimulate neurons directly you can put people inside the matrix this is again you know it's decades and maybe even centuries away but the path forward the path to that I guess reality people can see it and they're actively working on it so who talked all about well yeah this is me contemplating challenges now with my BCI headset the EEG headset yeah so we talked all about you know the possible things ybc eyes might be interesting so let's talk about the problems the things we have to do to get there as a psychologist I know I shouldn't put negative slides at the end of the talk and want you guys to go on

our positive note and I do have a few possum slides at the end but I absolutely want to be realistic about the challenges of lying beside implementation all right like I'm saying a lot of optimistic things up here and that's fine right I got trying to like you know encourage kind of like discourse around it but there are a lot of hard things we have to do I guess to enable these technologies so neuroscience has a long way to go neuroscience is in its infancy we understand some things sensory stuff is great motor stuff is great a relatively great higher-level cognition is tough and how we get there it's hard to tell there's so much we don't understand like how do you think I thought how do you remember something that happened last week how do you solve a problem how do you play a game how do you feel loved some what you don't understand and so much we need to figure out to get to where we want to go there is a common

analogy that people who are doing BCI research like to use which is that recording neurological activity from outside the brain or outside the skull as like standing outside a football stadium with a year against the concrete and listening to the cheers and trying to forget what's happening mad people to tell when something important is happening loud cheer if it's positive or negative depending on which stadium you're at but data is incredibly noisy its noisy inside the brain and it's way noisier if you have to pass through the scalp I'm a skull and hair data is noisy so neurological data is hard to parse we need to figure out how to do a better job you know of analyzing it and processing it and going hand-in-hand with noisy data right it's bringing out the right tools to analyze and process it we're the most effective ways to take a look at it there's always going to be lots of variability right both in an individual look what I'm pulling now is not gonna be the same way I feel like a second from now or how I felt three seconds ago right and then obviously across populations we are going to think differently and have different patterns of activity so we're looking for

something that is stable some pattern that is reliable and coherent what might be tough to figure out so we could just look for you know spike trains Peaks patterns of activation whatever but you just still large neural networks of the problem that's one approach people are taking but it's definitely a hard problem we need to solve and it goes hand in hand with a better understanding of neuroscience let's go back to the stadium analogy right if we have noisy data well one way of attacking it is to get a lot of large number of samples right it helped us it will help us get around the noise and analysis issues but we definitely need it so you can build a pretty useful dataset if you had lots of people giving you data and if you knew what they were seeing and hearing like if AG sensors were built into a VR headset and everybody was playing the same game that might help you make progress just a thought Imaging technology is the same slide from earlier has limitations right

obviously we can't get people to carry an fMRI machine around with them but maybe EEG headsets or FD r--'s or new new technologies people are working on can get there but what if we need invasive approaches are you willing to let somebody do this so we can do a lot of interesting things you can improve playtesting we can have adaptive experiences would be some rudin or rudimentary movement controls all of us would non-invasive stuff if we want to go invasive people need to see a music like benefit music used to be crazy shining a laser into your eye to improve your vision now it's commonplace well the same process you know with same transformation take place for the basic technologies like people implant electrode arrays in your brain maybe maybe not so William plants get to the lasik stage there's also another reason I showed you this picture right this is the third time I've shown a picture of Gabe right are you laughing as much as the first

time the answer is no right what do we nice to have that kind of data from our players wouldn't I become a better speaker finally how you guys were reacting the third time you saw this picture that's opposed to the first time so the next time I give this talk I'll do a better job there also issues with the hardware costs and dissemination building these things building imaging technology getting it out to people right since a technology cost money there needs to be a benefit to customers right what if we're unwilling to share this data with customers with us sorry we developers right this could be personal stuff we need able to convince them at various utility in helping us make better games and creating new kinds of you and play experiences that might be tough to overcome some of the technology is cheap at least element on the cost side of things but some of my not be so there need to be benefits customers can't just be theory when you make

something that actually works we're also losing tactile feedback right there's something comforting about holding a mouse well I guess a mouse or a gamepad or a keyboard right if I miss hit a button on a gamepad from trying to you know press B to jump or sometimes a I get pretty good sits and feedback what happens if I think a thought I think jump and nothing happens is it fault for thinking incorrectly or the system's fault for not understanding what I was trying to do so this is not the largest problem we're facing but it is an issue to consider I must last one it is maybe the biggest right this is Don Quixote you know attacking the windmills attacking an enemy that does not exist what if PCI's are not a necessity what if players don't see benefit what if we as developers can come up with ways to take advantage of having insight into internal state like I do believe I

can see clearly how this would happen like I've laid out you know variety of ways I thought we could you know get better data from play testing and design gameplay experiences that don't exist but absolutely could be wrong right this is theory I'm describing things in the abstract and it's easy to make things in the abstract sound cool so I want to now just possibility that well I am optimistic right like we don't know yet right like we actually have to create something and see again you know we have some data but suggest it's there but like let's acknowledge the possibility that it might not be so we're at the end of the talk I'll summarize where we are why we think we see eyes are interesting moment to an insight into new experience or sorry into experience if they do it at scale we'll get all kinds of awesome new data and we get to design games that are adaptive that's probably covers a lot I probably should have made that like use a larger font for that point

but hopefully have made the cases too I think that's interesting and the same for replacements for trishul interfaces right the end game new kinds of experiences become possible so one can all this happen well we actually start doing the work well we can start making progress I'm played a sting and adapted gameplay now adaptive games might take a few years to get going but we can start the process now then interface replacements well again people are working on things now but to be very very far away so I don't know exactly what all of this will take place but what I do know is that in the future I very much hope this is how we play so thank you very much [Applause] so we have a few few minutes for questions so I'll answer any now if you have them and if not I'm happy to go to one of the wrap-up rooms and chat as well so so is valve doing anything

dealing with the collection of data from a typical brain structures so because one the challenge is going to be that people that either have abnormal brains and don't know it try and use these devices and don't get the same responses as others yeah so it's so at the moment you know we just have to collect data from anyone who is we can get data from and they start figuring out you know which aspects of it are reliable and if there is something different about some individual person so that will likely come up as a consequence it's not something we're looking at directly at the moment and also that's gonna become important later on when you're providing feedback to the player so for each game you may have to do a calibration of the actual individual person yes yeah so absolutely like what this actually looks like in practice right ideally we get to a point where players could play the game you know without any changes in rn4 you know a couple hours or whatnot and we would get enough data to start making you know testing hypotheses about what they might you know like or don't like or if they're feeling engaged or whatnot so maybe we'll get a lot better at it

can turn it into something that it's five minutes or half an hour but yeah at the moment I guess the way I see it is we probably gather a fair bit of data at the beginning great sure Nicole's our phones ear design and yeah interesting talk I also am really interested in the opportunity with VR and other sort of headsets just so we get more measurements up there I'm just curious in the on the point especially about adaptive gameplay why muck around with the skull when you know you're looking at and emotions are so important to you know to balancing if you've got even if you have frustration and you can measure it which we which we did you know 15 years ago the you know your your frustration curve some players play to get super cuphead frustrated yes in order to feel the feeling of Fiero yes I want because you can't push a button and win yes so just saying you know whether they're being positive or negative we don't want to pull if you pull all the negative out then there is no actually gameplay it's flat they want to feel

that emotion and so then going back to be CIS and EGS you're really only measuring those systems the data is noisy agree but they also can only measure you know the first quarter inch of activity which is all neocortex where emotion happens is in the deep brain the hypothalamus the amygdala and all that it's wires neurons but it's also pipes oxytocin serotonin and all of these chemicals so I was curious like why why what do you why do you why are you so optimistic about you know BCI's that are measuring from the external you know they're non-invasive when we could just measure facial expressions like on your earlier side we used poliquin's facial action coding for the four keys to fun which is our model on emotion and how game play creates it yes so maybe that we can use you know webcams looking at somebody's face and get almost everything we need I think the reason to think about BCI is is that we actually can get a little bit more I think then we can get I think we can do a better job with emotions even if we can't get

deep into the brain although hopefully changes in imaging technology will kind of address that as well the purpose of the talk is not necessarily to say they you know BCI's or what I was describing with the EEG s are definitely the way to go i tracking and a webcam and you know a GSR sensor might get you everything you need but if we are gonna think long term and we are trying to think long term trying to understand what's possible and what we could get that with more granular than what can will give us it seems like a geez I guess they give us a lot of things and potentially a lot more so we're of emotions not just you know three or four yeah we're absolutely not wedded to it it's boy again you know I had the one slide about physical signals I mean I'm speaking about BCI cuz we think in the long run that's where things will end up but you know maybe over the next five years it's just webcams and we get a lot from that or you can also measure the micro-expressions you could actually measure the muscle contractions on the face yes that's what the EMG and whatnot and if you have a VR headset you can have an EMG sensors yes curiosity surprise anger contempt a lot of stuff

that's happening on muscles yeah thank you I think's great so I do EEG and VR research just as part of my graduate program yep one of the problems that we have with just having VR be well eg be a good control interface for VR is to reliably classify just different trials and let's say like an event related potentials paradigm you have to average together a whole lot of trial yep and while that brings user frustration down with regards to like trying to select like different objects it then takes that control time up yep so is like valve looking at any like solutions for that to help bring the not VR bring EEG and VR together like in in the consumer space a little bit yes so so one hope we have right which is optimistic is that you know if we had a hundred thousand samples of the same person doing the task five times as opposed to one person

doing a task 500 or 5,000 times that we could overcome you know that the issue that you're describing right that maybe you know may not be able to meet it be the case but the hope is it by looking at how people reliably reacting to a variety of things and over time and the large enough sample sizes that we could start getting a picture and so we would start with yeah so I think in the long run we'd like to be able to eliminate that at the moment in the beginning yeah we realize that's a problem we'll have to overcome it and the hope is that with enough samples we will but that might be naive alright alright thank you sure hi I was wondering you talked a lot about the adaptiveness of changing gameplay based on the the actual physical emotional response yeah but I was wondering how far along as a technology in terms of actually you know controlling characters within a game such as you know could telling someone to go forward and jump like how far along is that so it's right in the beginning so like not and basically people have trained you know people using egi sets to like play Tetris for

example right you know it's it takes training and takes work but so it is possible at the moment but like actually like doing like you know complex navigation to a space that's you know we're not there yet but and you know maybe that we eventually reached you know kind of like they come up against the fundamental limitation of what we do outside the skull and then it's like well did you forget a way to either be stopping the research or we forgot a way to provide a benefit where people are willing to get implants ok yep I guess that's all that time yeah I will I guess we can go to like one of the wrap-up rooms over there so happy to chat with you [Applause]