(P)Layer Character: Designing Immersive Narratives for Location-Based AR Layers

[Music] all right hello my name is robert morgan i'm a game writer and i'm a narrative designer and since last year i'm also a digital dramaturge whatever that is i always start with a question and this year the question is how often has this happened to you you're about six years old and you're happily playing an imagination game with a fellow player pew pew you shoot your opponent with an imaginary gun fair and square except no they tell you no you missed fair enough this is a collaborative imaginative space so this time you take careful aim announcing i take careful aim and you shoot bing they tell you that force field deflected it i'm seeing a lot of nods you're not playing it right no you're not playing it right i'm going home yeah i had an older brother so this came up

a lot now did their imaginary equipment include a force field a minute ago they didn't mention it but then it wouldn't have made tactical sense to mention it would it war is deception and today's six-year-olds have to grow up so fast now in principle digital games are supposed to resolve this issue right we can all play in a space with stable agreed-upon rules and an algorithm somewhere determines whether each bullet reached its target except of course it rarely works like that in practice what i want to talk about today is what happens between these two systems of play because in the last few years i've been working somewhere between these two spaces i've been making augmented reality layers of digital narrative and gameplay that are laid over the real world in real venues and public places with a mix of digital interactions and interactions with real world puzzles and actors

it's a state of play that's part arbitrary system and part consensus collaborative play working out how to make narrative compelling in this space has been a huge challenge but i believe it's a challenge we have to meet because ar technology is already adding extra dimensions of gameplay and story to real places and to people's normal lives because reality is cheap and plentiful the graphics are great and it rarely glitches out reality is also notoriously poorly designed with major balance issues our players are increasingly accustomed to shifting between realities today i want to talk about how to make stories that can shift realities with them because there's no doubt that whether it's internet of places or hyper local content or context sensitive ar or reality browsers which is the latest buzzword i'm trying to get going pretty soon the devices we carry or wear

will be able to direct us not just from our driveway down the freeway to our destination but to a specific platform and a specific train a specific carriage a specific seat to a specific product on a specific self and shelf in a store and that's to say nothing of the promise of augmented reality to allow us to tag content to the real world the appeal is obvious the time is coming where we'll have to deal with the fact that we're increasingly able to customize our own level of reality it's a paradigm shift that i think will be at least as significant as the original smartphone revolution and my ambition is to start figuring out how to use these new paradigms to tell stories that happen to take place in the real world there's quite a lot of echo can everybody hear me okay yeah okay great so first off a bit about me i started off my career as an assistant editor at playstation i ended up a lead writer and voice director on uh wonder book book of spells and its

sequel book of potions which i wrote in collaboration with jk rowling it was an augmented reality title that came with a real paper and cardboard book in the box and you showed the book to your playstation camera while sitting in your living room and the first lesson that i learned about ar the most important lesson really the ones that stayed with me ever since is that not only is the player and the protagonist kind of the same person when they're sitting in their living room with a magic spell book looking at themselves on a camera feed and watching themselves play the game but it's also that you have to start asking players certain kinds of questions in order to get them engaged more than they might otherwise be getting engaged in a flat screen or traditional 2d game while players were first getting to grips with the book we had a voice prompt that prompted them to start interacting which asked us question very simple question are you a wizard or not i actually

thought it was a bit much when i wrote it first i thought that players might find it hostile but to our surprise in user testing kids were shouting at the screen yes yes i am a wizard yeah i want to participate it's a question everybody knows the answer to because it's in the books harry gets asked this very question at the end of book one but more importantly a player sitting in their living room answering this question is answering inside the fiction it turns an idle moment an awkward kind of self-conscious moment into a moment of self-definition for the player and they define themselves inside the fiction it's a question that helps make space for them to put on their wizard hat and robe in order to participate and in many ways i've been asking this same question ever since now i have either three jobs or i have one job in three parts depending on who you ask i write games specializing in vr scripts most recently a fisherman's tale which came out from innerspace i consult on narrative and ar and vr for various people and for the last three years i've been creative

director of play lines and we make location-based interactive augmented reality theater or depending on who i'm talking to mixed reality digital performance or xr enhanced live content whether it's ar or mr or xr or my new favorite merged reality my goal has been to explore this hybrid medium to experiment with ar's promise of delivering digital layers of adventure that players explore by navigating what for simplicity's sake i'm now calling the hard layer that is actual reality our goal is to build a track record of making large-scale narrative-led ar using cheap minimum viable technology in this case digital audio and real-world actors and i want to do it without waiting for the visual technology the visors and the holograms without waiting until they're good enough to create complete and convincing digital fictions in the real world i want to build a track record now so

i'm certainly not going to slow down to wait for the definitions of ar or the rxr whatever else to catch up what we found is that in real world ar adventures where 90 percent of what the player experiences is the real world digital narrative works best when you deliver just enough just season it over the players experience just enough to recontextualize the real world around them to make players relate to what they're seeing and where they're standing and where they're going in a more adventurous way in ar you cannot tell the player you are in the dragon's cave it just doesn't work but instead we can give the player story that takes place inside them we can tell the player you're being watched or just act normal as you'll know if you've ever said this to someone at a party you can completely change someone's relationship to the world around them just with seasoning them with a little

bit of story this is mixed reality and as you'll know if you've ever walked down the street imagined it to be going pew pew or used a stick as a sword or imagined enemies in the bushes or pokemon at the bus stop activating a little bit of imagination goes a long way now most important slide in the talk what does all this actually mean i have a video of our most recent installation which was our future storytelling festival in new york it is a collaboration with a uk grime artist called harry shotter this is a hip-hop experience that turned any venue into a story-driven nightclub experience content warning because of the nature of the content there is swearing and although it is an anti-drug piece it has a reference to drugs in it let's find out if the video works please hmm let's see hey sound

how you been [Music] having shot as i'm seeing in what 20 minutes excuse me mate what room is the drummond base never mind that you look like you're out your [ __ ] face no mate just drink him my mate is sober well if i catch you with any drugs it's game over now let me tell you about yourself you absolute weight smile hold on how are you gonna lie to my mate get the same old lucas from the estate hold on sorry lad sorry ellie hey come back come back just let me talk to her let me talk to her as for you player i see you've made your decision goodbye hey man what's happening you don't look right talk to me man you still shook from the fat are you playing a joke you want to talk and spliff that boy don't look right hey come look at this [Music]

so that was an installation of future storytelling we were nominated with finalists for the innovation and storytelling prize it was an interactive layer of music and story spread throughout a historic venue you explored rooms characters in the audio lair would come and talk to you tell you where to go next and the character that you chose to follow the room that you chose to go to which storylines you chose to pursue by physically navigating the venue determined what happened in the story and led to one of five different endings and as you saw there are interactions with real world actors who exist both in the digital layer as a character and in the hard layer so in that case players received through the digital layer a password which they then gave verbally to a bouncer to be allowed into a vip area now at its basic level players arrive at a venue download an app plug in some headphones and start exploring we see the venue with bluetooth ibeacons which we use to track the player much more accurately than gps and so we can start treating where the player goes as a story decision made inside the game layer we have a game engine running in

the back end and we can track those decisions and make sure the consequences of your actions follow you through the story this is one of our narrative design diagrams once we're able to deliver story directly to the player like this we can start building up complex layers of interactive fiction which just happen to take place in the real world but in these experiences the players the site the interface between fiction and real once they're on board the narrative the player gets to decide how much is fiction and how much is real and the story is there to help them along because some players need more help than others particularly if they're a bit self-conscious if they struggle with the doofus factor the doofus factor niantic themselves have identified the pokemon go potentially suffers from the doofus factor when your app has a critical mass this can actually be a good thing because seeing other people on the street being dorky being visibly surrounded by our fellow dorks is a really great feeling welcome to gdc by the way how's it going everyone

it helps us all bring us out of our dorky shells but because in play lines we use audio we're speaking directly to one player at a time and we often install our fiction layers within busy festivals or public spaces to add a dimension of story or adventure to create a population of engaged exploring users who see the same space in a more adventurous light but if we leave those players to get over the doofus barrier themselves without any help at best we're limiting the number of players who'll get the most out of the experience and at worst we're excluding players who want to be part of it but who aren't primed to do kinds of the kinds of semi-imaginative play or role-playing out on the street in conventional play in conventional design we think of self-consciousness as the antithesis of immersion the kid who does this who keeps destabilizing the reality of collaborative play this kid is somehow worse than the kid who always has a deflector shield it's part of why punch drunk theater who

created sleep no more anonymizes their player audiences by putting masks on them it makes players feel more explore environments without social cost but in our installations we found that achieving immersion in ar is not the same thing as chasing down and eliminating the player's self-consciousness because we don't want to overwrite the player's identity even if we could and the players self-consciousness yes their shyness is part of their identity it's a mistake to pretend the player isn't there because in ar the most important reality you're augmenting is the player themselves because and if you see me speak before you won't be surprised to learn but i think this is all about embodiment in my vr work i argue that what's exciting about vr is not just who you're seeing but who you're being the player is radically embodied in a vr scene in a way that they just aren't in any other format and so you have an opportunity not just to give the player something cool to look at

but to show that the simulation looks back because embodiment comes with a lot of powerful effects including self-consciousness in vr if you pretend the player isn't really present if they're just no more than an invisible flying camera then the player doesn't have any emotional or physical stake in the scene they literally have no skin in the game but in ar the players really standing in front of you and they really are wondering if you're going to embarrass them or heckle them their pre their feet probably hurt and their phone is running out of battery and they're worried about brexit embodiment the fact that they really are there that's what's keeping them feeling like it's all real maybe even a bit too real and if you exclude all that from your fiction it's a huge flashing sign to them that it isn't real that they don't have to emotionally engage this doesn't mean that players don't want to take part players want to be part of your fictions that's why it's called willing suspension of disbelief

but it's a collaboration and you have to meet them halfway so how do we design interactions which allow players to merge the two realities they're dealing with interactions that bring the player more into the fiction no matter what they're feeling when they're standing in the real world worried about the doofus factor well the easiest example comes from working with kids because kids are accustomed to moving between shifting between multiple realities and changing the rules but they're also generally not too good at playing a specific character that you give them and following the rules of that character they'll blow right through all of that and expose exactly how much space your fiction leaves for them and their real feelings especially boredom especially self-consciousness this is the dream finder anyone familiar with this character this is a costume character at epcot he was part of a ride that was all about imagination the character was originated personal heroes ron schneider and he wrote a book about the early days of the character in the 80s in the golden era of

the autograph book kids were coming up and instead of interacting as a character instead of talking about imagination giving him a chance to work a little bit of character magic they just wanted an autograph and then to go because by and large kids want a safe social interaction where they know they're not going to screw up especially when confronted with somebody who's not only cool but kind of intimidating the character has a bubble of fiction around them the characters trying to invite the kids in the kids were bouncing straight off because they wanted a safe interaction behind the autograph book and so one day ron started asking kids what are you why are you smaller than all of these taller people what's a child explain to me what a child is it's one of my favorite pieces of interaction design ever oh it's banging kids love answering questions that they know the answer to especially if an adult doesn't know the answer the dream finder takes the risk he looks

silly he's already done all of the clowning so in comedy terms the kid gets to be the straight man through this question the performer expands and opens the bubble of fiction with an easy point of access for the kid and it actually uses the player's own self-awareness to fuel their immersion because it asks them to account for themselves to explain their presence to use their self-consciousness in a way that is now suddenly in fiction in a way which encompasses the emotions that normally would be disruptive to immersion and instead it makes those feelings part of the fiction that they're participating in it's another way of asking a question which the player already knows the answer to they have a gut reaction answer ready to go and that answer brings them further into the fiction this is what i mean when i say that in ar narratives in the real world we have to deal with the player and the protagonist as the same person it doesn't just mean telling the player who they should be role-playing as

giving them a persona we have to meet them halfway their feelings including their feelings of embarrassment or foot hurting are hard for them to ignore and we ignore them at our peril when we move ar and gameplay and narrative out into the real world onto the real street we can't demand total suspension of disbelief from our players before we let them play the game because in arlen 90 of what the player experiences is the real world much of that experience is their real selves there's no point trying to tell the player you're not playing it right because when the player is the protagonist what you're really saying is you're not right you're saying you're not in the right kind of body to play this you're excluding them you're denying them in ar we can't try to overwrite who the player is any more than we can try to tell you that you're standing in a dragon's cave when you're not in ar role playing you can either deny

reality to overwrite it or you can augment it more importantly you can either deny the player's reality or you can augment it you can't do both augmentation is not overwriting it's addition it's cooperation because asking players to ignore what's going on with them is asking them to ignore parts of their reality that limits the amount of reality available to us as storytellers to augment and we're missing an opportunity to surprise the player by inviting and responding to feelings they think they're hiding feelings which normally don't have any place in fiction the same feelings that give a ring of authenticity to really good characters like self-consciousness suddenly they're allow you're allowing them to be their own protagonist it's just like when the player says i have a deflector shield bing there's nothing inherently wrong with that bit of imagination suddenly introducing a deflector shield oh i guess we're in a sci-fi setting after all

it's world building the players imaginatively contributing to the play but the player came up with the shield because they didn't want to get shot they're all played in order to get something they want and i think that can be a powerful thing that shows us how we can use what the player wants to help them turn themselves into protagonists as long as we make room in the story for what they're feeling give them a way for their emotional journey to be part of their hero's journey that doesn't always mean giving the player what they want and in fact we've had success with embarrassing the player whether that's by giving them a hostile rebuffing encounter from an actor or dramatizing a sexual encounter in the toilets during a busy festival not this one don't worry that was fun though but the fiction made space for those feelings so we're starting to figure out how players relate to digital narratives in real environments but because of the doofus factor that relationship to the narrative

can all too easily be disrupted by the social context around them by other people but if you do it right the presence of other people around the player can also enhance their experience while you're trying to get players to feel adventurous in real world spaces festivals public places you also need to help many of them get past the worry the chance however small the fear shared by all nerds really by anyone who plays imagination games the fear that they will be heckled by jocks so we had to work out how to make self-consciousness a virtue in our ar layers we had to find new ways of doing it like giving our players the opportunity to look cool in front of other people one of the ways we do this is to create fomo fear of missing out there's nothing that creates fomo like a velvet rope so you saw in the video our bouncer character wearing sunglasses this is a character who not only provides a gate in our gameplay once the player has found the

appropriate password in the digital layer they can verbally give the password in the hard layer and get access to the vip area but that vip area is not accessible to anybody who isn't running our experience so in the middle of a busy festival we created a purely exclusive environment that everyone who doesn't have one of our bright green play lines branded pairs of headphones on looks and goes those people know something that i don't how do i get in there and the only answer is that you have to run one of our experiences most of immersive theater is fueled by fomo fear of missing out because a larger possibility space than a single user can see prompts replay and a sense of reality hero moments where the player gets a unique visible place in the immersive theater story can increase engagement but sometimes it comes at the cost of immersion for other players because people begin to game your story to treat it as a vending machine for story moments and so after our players get through the red rope they enter a dance area it's

kind of like a silent disco it's full of other players we move them from the end of their solo unique player directed highly consequential journey in audio where we've been speaking directly to them and we suddenly move them out into a public space where everybody is listening to the same music channel even if they're surrounded by different digital characters because we like moving from the private to the public because it brings the players out of their shell even more and gives them a community experience of the adventure that they've just had now in his book about working and designing themed entertainment ron schneider the dream finder he describes the history of arguably the first theme park coney island in new york first there was steeple chase park it was a collection of rides and attractions with no particular theming like a traditional fairground then there was luna park the rides told a story like the great chicago fire or journey to the moon but they were all jumbled together with no unifying theme in ron's words guests were slammed from

one reality to the next and finally there came dreamland an architectural setting inspired by arabian knights and although it's kind of a cacophony of appropriation it was a unified setting for stories and adventures where the player didn't leave the setting as they moved from activity to activity and as ar allows us to start bringing gameplay into the real world whether that's static single room hologram experiences site-specific installations like play lines work or location agnostic experiences like pokemon go i think we need to figure out how to create stories that the player wants to help tell open worlds of story content that can guide players between locations between activities whether that's their grocery run or a series of rides or a series of escape rooms between different levels of reality without them ever leaving the story because it's a story that they're telling themselves now if you're a business owner does the

proliferation of ar layers does the possibility of tagging your local environment mean you'll lose control of your spaces well in the digital layer yes will people will people be able to graffiti over your commercial frontage in some ways at some levels digitally augmenting things in a way that you might never be aware of yes the idea of trying to stop happening will soon be as old-fashioned as the idea of trying to stop people taking photos in a museum you can try but you're competing with humanity's collective capacity for mischief and more importantly you're missing a chance to understand what it is that keeps bringing people back to galleries even though they've already seen a photo of the mona lisa because places and taking people places and giving them experiences in places really matters if you own a cultural space you might be about to find you have little control of the digital

utterances which are starting to be inscribed on that space but there's also never been a more exciting time for spaces to become multi-context with different visitors in different audiences able to use the same space as a gig a narrative a larp a scavenger hunt a canvas all seamlessly and simultaneously does there need to be some control yes because no one has the right to walk down your street and scroll a digital swastika on your house whether they do it in a layer or not but i think it would be a mistake to try to confine the tools to try to confine the ability to inscribe on the public digital layer to official channels to leave it in the hands of the self-appointed because that would mean essentially handing over monopolistic control of digital public space to the same monolithic structures that we handed over control of public public space in the real layer to the

highest bidder advertising dominates our world and pretty soon digital layers that might contain advertising will be available on every street corner but is that all that's going to be available the promise of ar ultimately is the ability to customize how we see the world but what we need to avoid is allowing this technology just to give more power to the powerful because there are already plenty of people who can pick and choose and customize their reality ultimately the promise of ar is for us to be able to leave each other notes in the margins of reality but that makes it more important than ever to make those tools available to the marginalized which is why i believe that learning to tell stories in this medium the hardest medium of all in my opinion reality is important because story is something that unites us despite the fundamental separation caused by our inability to share exactly how we each

see the world narrative is the universal language that we developed so we could compare notes about our disparate experiences of reality and as soon as we start working with reality as our medium we have a responsibility to reality because some people's realities are harsher than others in real life everyone is their own protagonist the games in fiction generally define and specify who protagonists are and can be we tend to see the same kinds of protagonists coming up all the time but not everyone gets to be the kind of protagonist we typically see in games in games some people get to be more protagonists than others but in in ar if you exclude the kind of person the player is from the story then you're missing out on the opportunity to play with them and that's the only way to construct meaning in ar brick by brick emotion by emotion back and forth in ar the player is already embodied in your setting they're standing there waiting to be

part of the story and if you don't let them if you keep insisting who they should be if you keep telling them that they can only play it your way then you might find yourself six years old again playing an imagination game taking careful aim and firing your imaginary gun and finding there's no one there and that's where i want to leave it thank you very much uh anybody want to stick out yeah hi hello hello um so i own an escape room and i'm really sorry go right into the mic can you hear me now yes okay um i own an escape room and i'm interested in integrating some ar cool to my rooms do you feel like that there's a space that's too small a physical space that's too small for you i think it's a good question i think that's like asking so the question was is there a space that's too small using escape rooms as an example

i think that's like asking if there is such a thing as too small a story and i think it's about this job of story is to effectively fill whatever the dimensions of the space is with the right amount of narrative so if it's like that hemingway short story about the shoes that's fine too you know um so no i don't think so and i think that ar is all about adding an extra dimension whether that's threat or affection or warmth in principle definitely not and it's an area that i'm interested in as well and i think there's a lot of potential there but i also think that one of the potential areas where ar could contribute is actually extending the dimensions of the space digitally so that the puzzle and the story might begin before you even get to the escape room and follow you home to the train station questions i'm probably going to be doing a breakout after this if we don't get to

everybody i don't know where the breakout room is i'm sure i'll find out uh but just stick around afterwards hi hi uh you talked about ron schneider and kind of the like theme parks and themed entertainment as immersive entertainment and storytelling and i was wondering in your experience with ar how much does the uh the design of the actual physical environment that you're in affect the story experience and how much of like i look for you as a writer for ar how much of it goes into uh like when creating experience how much goes into designing the actual environment and and changing that yeah and how much of it is like in whatever digital layer you're working with so the question is how much do we design the spaces and what impact does that have on the digital layer of storytelling actually in practice most of the jobs that we've done so far as part of play lines have been with relatively minimal set dressing because we do it within existing spaces

so we'll take a museum or a cultural center or a park and the idea is to not try to alter too much because i'm interested in recontextualizing that environment so we use what our co-founder dustin freeman describes as reverse level design where in in conventional level design you have the beats you have the story you want to tell and you build the environment we start from the environment and we work out the story from that place based on the opportunities that we see is there a natural fork in the environment for a decision is there a cool landmark or a nice little nook we can tuck players into to have an intimate experience so a lot of what we currently do is reflecting on that but that's also partly because those types of projects don't have the budget for a lot of on-site set dressing and also because they tend to be while creating an additional digital layer while the site is also in normal use so we can't have too much of a physical footprint i would love to set dress full environments because i wanted i want to do shakespeare in this format you know

um but i think that will come as more spaces become interested in putting this sort of thing on but it's again this ties up with the idea of doing it within an escape room where the more you can control the environment the more that you can reflect that back to the player and get the player doing something specific but that it doesn't have to work that way if the story fits the environment correctly that makes sense yeah thank you thank you hi um sort of similar question but you mentioned some physical limitations people might feel while they're playing for example their phone battery dying or their feet being tired do you factor any of those into the narrative that you're designing yes so if you're playing pokemon go and you see a pokemon on the other side of a chain-link fence and you can't get to it or it's up a flight of stairs and you happen not to be able to get up those stairs the game is still functioning as designed that's not

strictly a bug that's a bug of reality that has been ported into the game and so the limitations that people experience whatever those are trying to make sense of them and allow people to still access the story is really important because otherwise you're just replicating the unfairnesses of reality so i mean just that's it's a rather extreme example but uh we've designed in storylines with parallel routes to make the the same route both accessible by the stairs and accessible by the elevator and had you know slightly different content in the elevator or an easter egg or something so accessibility wise we try to stick with what the venue can do and to enhance and add richness and and make that part of the experience i mean the issue of uh the battery starting to beep i can foresee a time when we might procedurally ping in a piece of audio that tells players that their you know power pack is dying okay um it's an interesting one that's that's

worth further thought i suspect it's a good question though i'm afraid i think that's all the time that we have thank you very much fill out your speaker evaluations have a great time and i'll find out when the break where the breakout is thank you