Integrating Live Actors into At-Home VR Experiences

[Music] hi i'm samantha um julian tanya and tara and three of us are from uh tender claws tara's from pie hole and we're going to be talking about integrating live actors into a non-location-based experience so how many of you know a little bit about tender claws or i've heard about us cool awesome so our studio kind of works across a range of different disciplines and media our two most recent projects are the ar app tendar and vr for virtual virtual reality but we've also worked across mobile apps installation visual arts and even sports with tumbleweed and leaf blowers and one of the things we're going to be talking about today in terms of the xr community is there's a lot of legitimate and well-founded buzz about the parallels crossovers and intersections between theater live performance and immersive

digital media um the term virtual reality actually comes from about the 1920s and 1930s when arto was writing this book uh about some of his essays and his um experience with theater uh called theater and stubble and i've highlighted the term virtual reality so you can see this is of course translated from french which the meaning is slightly different there so when people talk about including live actors it's often considering a really promising and interesting space but is often kind of dismissed as being a little bit difficult for you know obvious reasons for possibly production or scale but it's such a fruitful and interesting territory that we think it's worth pushing it and pushing on the constraints and finding unique ways and opportunities to give home access to live performers so when we were creating this project these are some of the design considerations that we were originally wrestling with and as artists who have history working with theater and vr and who have spent the last year in the trenches working on

this kind of non-location-based at-home experience we wanted to give this talk to spark further discussion about how performers could be integrated outside location-based entertainment or lbe so here are some of the things that we're trying as a jumping-off point for bringing live actors into the home and these are the structure of some of the things we're going to talk about today and who's going to be talking about them so to start i'm going to show a little bit of teaser of the under presents [Music] distinguished guests are you looking for a little

entertainment you've come to the right place the venue outside of time and space may i present the under [Applause] sweet and so for some context on the underpresents there's actually these two intersecting worlds so the game begins in the under um which is this kind of vaudeville stage that exists outside time and space it's inhabited by these quirky lost voyages and souls that wander off course and became lost in time and then they all kind of perform for you under the guise of this mysterious um mc or the pro sorry the proprietor of the stage but it's important to say that there's actually a story within a story so even though we'll be talking mostly about the under and some of the live elements that are there there's actually a long-form content of a another narrative that takes place on a ship which is a the last few months of an ill-fated trip

from a research vessel where players witness a survival narrative that unfolds between characters while discovering that they may or may not have the power to influence the crew's fate and change how time operates on the ship so tara's going to quickly briefly briefly run you through some of the context of the ways we're using live actors so we can talk about this um yes so two of the main ways that we are talking about that we're using live actors in the experience one is doing these live stage acts as hosts and so we create these different types of acts like a jazzercise routine or a game show and that takes place on the stage of the under which is highlighted on the left side and different characters can slot into those roles so you can have different characters who do the game show host depending on which actor is plugging into that act and over on the right this is an example of a one-on-one live encounter and those

happen and those can happen in multiple locations but they tend to take place in more intimate spaces and create a more intimate experience for the player so the actors uh and this would would inhabit these different avatars and those are highlighted up there and they interact with the players who have a more anonymous look to them to their design and beyond those two specific live actor roles we also have the actors inhabiting npc characters that you see in the more narrative uh the survival narrative part of the experience that samantha was just talking about so in other words you would hear you would see pre-recorded scenes with those characters but we also have live actors um at key moments possessing those npc characters um npcs and uh and making having these live moments with them um

and so beyond those three roles we work with actors to develop characterization voice um and story and structure of these moments and develop movement with them so each chara each actor is able to actually have their own characters that kind of go with their play to their strengths and abilities and then they kind of tailor each pre-structured experience or encounter to their own to their own thing their own vibe so yeah so as a lead designer uh narrative designer rather i'm going to be talking about scripting and how do you create scripts for kind of a more improv live active live characters and acts the scripts have to simultaneously be structured but also modular and variable so that they can feel responsive and some of the inspirations for the project that led to our thoughts on scripting are these three

medias anybody recognize any of them besides journey yeah cool so we take a lot of our inspiration from not just games but from different media as well so this book called the invention of morale is largely about a time loop that features echoes of the past it's ambiguous to the reader if the characters inside it are kind of living or recording but it's key that the narrator because the narrator believes they are alive it transforms the narrator from a spectator into a participant and that ambiguity heightens the mystery and the desire to explore the island for this for this uh narrator and sleep no more which is a pretty well-known you know immersive theater piece it also operates on a loop if a guest is really persistent with a character they may be rewarded by the character breaking their loop to engage in the guest with an intimate one-on-one encounter this act of signaling out the guest creates a heightened moment where

the player is seen and the process of being seen is really important to enhance the feel of liveness and then finally in terms of multiplayer interaction a game that i think was really seminal for a lot of us um especially as a tenor clauses journey we're thinking about you know how when we first played journey there was a very formative moment when we realized that what we thought was our ai guide was actually another living person that we were able to recognize that through their um motions and intent so at that like moment was something super powerful that i'm going to kind of get to later we also decided to in terms of the multiplayer experience limit some of the ways that players can communicate with each other purposely in order to encourage emergent forms of play and new ways of communication and gesture between players so what all the references have in common is this idea of a pivot point between

liveness and non-liveness and how they come into contact to generate a space where there's an extraordinary awareness of the other it's sort of an uncanny space and i realize that you know the uncanny valley there's connotations within that and it shares some of the features of what i'm talking about but instead of eliciting a cold eerie feeling of coming against another kind of humanoid creature it's i'm choosing to use the uncanny as a term to describe the space of discomfort and wonder that derives from comforting the ontology of another being as simultaneously human and non-human and the power that mystery generates in that moment to hold our attention and to bring in players into like focusing on specific areas of the game so the main impact of the uncanny is a way to focus and draw attention to key story moments um it also adds magic and anticipation to pre-recorded content that is much less resource intensive

because there's a sense that anything could happen at any time and in many ways in the structure of our game it can and does but this can be more or less true in certain areas of the under especially there's something that i'd like to show which kind of for me exemplifies the moment when a outside uh player or user is in is brushing up against this definition of like movement that is both living and not living simultaneously and how that kind of instincts wonder and attraction and this is this beautiful scored unintentional performance piece [Music] so um at the living mannequin the purpose of that is to kind of bring this liveness into the space to attract passerbys and have them one you know wander into the world of the store into the world of the setup or refocus

attention to specific objects and um in terms of i think punctuating a lot of what we're going to be doing in the game with these moments that are one-on-one i'm thinking that some of them can start in a way that there's a kind of discrepancy and mystery is this you know other person is this other character this other you know npc or performer living or none um in order to actually write for that there's ways to work around the technical and design narrative constraints the resources um that you need to keep character consistency and maintain character voice without needing the same actor on staff 24 7. so basically what i'm saying is that there are certain npcs and this is one particular moment of how we use live actors this isn't like all the different ways where they can be inhabited and possessed by a live actor and it's not immediately clear to the

player that that is what's happening however the constraint is because we have the because we have the character and we have the voice actor this is an established character people know the voice of this character so how do you create dialogue that any actor can join in and use the voice of the character to interact with the player um and a lot of like the kind of i think nuance about writing these this dialogue in this way is uh really interesting um so some one way that we could approach this and that we're working on is this idea of sound boards um what we'll get into later is there's actually it's very much how our actors engage with the audience is very much like puppetry so there's certain like interfaces they can manipulate and work with that you know we're trying to make you know more like kind of natural and seamless um and one of these interfaces is a way of triggering sound clips but it's not just it's i think it's not just like any writing for sound clips will work it has to be a very

um a combination between how the actor knows to trigger the sound clips at the right time and what feels right for their performance and writing that feels it's seamless and interruptable so in order to kind of script narrative encounters one of the things i'm ways i'm thinking about this is creating narrative backbone there's a guiding principle there needs to be enough anchoring like points within the encounter to give context to why the player is there why the actor is what their minimum roles are so that there can actually be room left for the player to enter these roles and enter into play with the actor one of the things we're thinking about is how to focus these encounters on a meditative activity or an activity that can be shared because that also opens up rooms a room for play and ways that the player can be responsive to the actor and the actor can know how to trigger certain bins or certain areas to feel like very responsive and one-on-one uh

sound clips with the what depending on what the player is doing and as players are engaging with the actor in very much wit and very much um it's kind of like they're performing along with the actor they're really filling in the blanks of what is happening and depending on how the the players engaging with them actors can kind of decide how much dialogue they want to trigger um whether they want to do a lot of the the interaction through gestures whether certain points they want to like trigger exact elements of the story to punctuate what is happening and it's up to like um i worked with ben vegas and pai hall to develop a lot of these story boards and it's up to us as writers to kind of create basically text and framework for the actor to use as improv tools from their toolbox to kind of grab from right so i can kind of give a very specific example this is like an early draft of one

particular npc and the context is that there there's on an island skipping stones and it's a very like short meditative action but there's a narrative backbone which is mostly on the left that is digestible small breakable chunks that can be triggered maybe in sequence but it doesn't have to all be triggered um and it can also be um kind of yeah triggered at the actor's decision uh it's important when thinking about writing this sort of way that like each of these kind of have to stand on their own because you don't know much how much time is going to be passed between the next element that's triggered so they have to be complete thoughts and they have to all kind of relate to the tone and the atmosphere that the actor is trying to convey so on the right um there's also a way to and these are just small samples really you need i think a larger set but a way to create bins

where the actor knows that okay if i create if they um if this person throws a rock um the rock mechanic is really like elaborate so it's actually really like skipping stones it's not the best throw maybe they want to create a one-on-one a one-to-one response to that and they know that there's a bin that they can trigger to do so and that the tone of that bin will be short responses that non-repeatable that um are that can feel kind of a little bit of natural in the moment and how you actually have to write for that is really complicated because you have to vary sentence length you have to like make sure that the actor knows that it's not just going to be once a statement and one's a question and they won't know how to like respond to that in the moment they know there has to be a uniformity to like the particular bins they're triggering and finally on the left because the throne the stone throwing is rather realistic we want to create a

sort of um invite the physicality of the actor and the player into the scene and if the actor decides it's relevant they can actually train and teach the player to kind of throw the stone and ways that they might engage or give tips that feel responsive to the action that the player is doing at that moment and then one of the things i think uh on the right you can see that's writing for these spaces of how the and i know tar is going to talk about this but ways that the players may respond to the actors it's a little bit of kind of fortune telling um both trial and error and fortune telling where you think okay what are the range of like player possibilities and ways that they can engage with the actor what are things they might do and then you have to have like a back-up plan for like the actor to you know is someone teleports away what do they say

you know what are some of the ways they can engage with that um and you know do they and like we are open you know we want the player to be able to leave and like you know feel that's correct so we're not trying to bring them back but we still have to think about like how the actor responds to exact different ways players can behave and of course there's it's live you know so there's so many things we can't account for and that's part of the magic um so now julian is going to talk about interfaces and infrastructure hi everyone when you're integrating live reformers into an immersive vr experience you have to decide what you're optimizing for for the under presents we want to create intimate moments between players and actors and we want to maximize the number of meaningful interactions where a player feels truly special we don't think that we could accomplish this by having a single shared space with hundreds of players it just wouldn't facilitate those kinds of

interactions instead we're aiming to have lots of shared spaces that each have 10 to 20 players that leads us to these questions how do players and live performers join these shared spaces and when they join a space which space do they end up in how do multiplayer interactions work within these spaces and what tools do the live performers have to facilitate unique magical moments for the players hi i'm julian and as a systems developer at tender class it's been one of my roles to try to come up with good solutions and implementations for these questions first how do players and live performers join shared networked spaces remember here's the player on the left and for them the answer is really simple similar to journey we want to give them very seamless experience where they don't need to think about what how the networking works they just enter

the space behind the scenes their client makes a request to our backend servers and that causes the backend servers to verify the player's identity retrieve the information about past encounters that that player has had and then we retrieve information from our database that provides information about what shared space they should end up in and then we send that information back to the player client which can then join that space the situation is a little bit more complicated for actors because we want to give them agency in deciding where they could have the most impact but we also don't want to overwhelm them with options and so we created the simple interface that lets them choose to enter a shared space or room with players who meet certain criteria like never having seen a live actor before or never having seen a specific type of stage act actors can also jump into a space to help out another actor or they can enter a random space if they're feeling lucky

and they want to try improving for players without really knowing much about what they're getting into the situation behind the scenes is actually pretty similar to the flow for players the actor's device asks our backend servers hey what room choices do i have the servers then verify the actor's identity and they actually make sure that they're authorized to wield actor powers we don't really want hackers or regular players to have access to all of an actor's special almost god-like abilities and then the servers fetch a curated list of categorized spaces that the client can display for the actors once an actor player has joined a network space how do we support interactions within that space from an interaction perspective players have a limited but versatile toolset for communication like snapping and multi-hand grabbing which facilitates sweet interactions like this one

behind the scenes we decided to use a hosted real-time networking solution photon a lot of you have probably heard of it and it lets us synchronize a small amount of shared state per space and per player like the positions and rotations of each of a player's body parts we also use remote procedure calls sometimes for a short-lived one-off events that don't really need to be strictly synchronized when a new player joins the existing space so for example if a player snaps that's a very short term event and we don't need that to be synchronized when a new player comes in we don't care that the snap happened in the past our custom servers are reserved for more higher level decision making so matching players into shared spaces and also monitoring because we really want to know if we need more live actors on staff or if we currently have too many live actors for the amount of players who are currently playing

that also provides a persistence layer so that we can store information about players past interactions with live performers and we can use that to match them into spaces where we think that they'll have the best experience finally what tools does a live performer have to create magical moments that make players feel special actors in the underpresence can actually speak with players and players can't and actors can use their standard improv talents to great effect but they also have very powerful abilities that are unique to the underpresents they can shape the virtual environment for example by conjuring objects out of nothing or even completely changing the layout of the virtual stage these are mostly one-off abilities and there are a lot of them so one of our big challenges has been in creating an interface that could support this flexibility without getting in the way of the actors actually acting in some of the early prototypes we had actors would be super engrossed in the interfaces and staring at their hands and that really detracted from the experience of their

acting in our current iteration we're using a radial menu system that allows an actor to select what kind of ability that they want to use and then trigger it on a button press and they don't have to use look at their hands throughout any of this process which creates a very fluid experience however this is an expert interface and it requires training to use the benefit is that it leads to seamless performances that are interwoven with unexpected delightful moments actors need to be able to refer back to players based on what players have done in the past in some of our early prototypes our live performers had trouble referring to players because they do look very similar and they can teleport around which can make things very confusing for actors that's why we added these colorful layers labels above players heads and you might have noticed those in some of the earlier images these are only visible to the live performers and they may seem like a small addition but it made a world of

difference working with pai hall has been great because they've helped us figure out how to bring immersive performances to vr and that's something that tara will talk about now hi i'm tara again um and i'm part of pie hole i'm a theater director and um pie hole is a live arts collective um with five members um and we make live theater and experimental theater that takes place in a lot of different formats and spaces and environments so we started out actually as a puppetry doing a lot of puppetry and so you see that on the left that's a show that had the audience sitting in one place and looking at a stage um and on the right you see an installation um those are audience members in an installation so we we play around with a lot of different formats sometimes with the audience seated sometimes with them roaming and sometimes with the performance basically taking place among the

audience but in all of the different cases of our work we're trying to foster agency among the audience and kind of rewarding a curious viewer so just a little bit about the puppetry um that is sort of like one side of our background as a company we do a lot of what's known as physical theater which is it has a strong visual component to it puppetry mask work and basically work that's very physical and um an awareness of the body and creating kind of efficient storytelling through the body which has proven to be very relevant for performing in an animated vr setting and we also do a lot of have done a lot of ensemble driven created work in theater and so what that means is that our members are not just performers they are also they're generative performers so they're also writers and they have experience improvising and they also

have experience kind of developing characters and the writing for characters in a larger narrative context which is also something we've drawn on in the process of working on nvr so and that kind of work is often referred to as devised theater so devise theater and physical theater are both backgrounds that have proven to be very helpful for this process so the first thing i'm going to talk about is just developing content for live encounters kind of picking up from where samantha had left off and i'm mostly talking here not about the sound boards that she spent a little bit more time with but the unscripted unscripted i'll put that in quotes because a lot of the encounters that we've been developing they have a set structure like a chess checklist that the actor needs to accomplish so it has an arc to it but they're improvising around that structure

and so to create these moments first of all we've we samantha has provided a kind of bible for the under that has the under mythology or backstory in it so that any performer coming into this process can actually know what world the world is that they're creating it and it kind of gives them what they need to know to be able to actually um improvise because if they don't know that backstory and art don't have a fluidity with it then they kind of don't have any guidelines to develop a character and then when we actually coming up with these live encounters the first thing that we really need to ask ourselves is what makes something feel live and um and throughout the process we keep questioning this um we may have some guesses about this this seems like it might feel live and then you actually try it out and and you realize that it's really obvious that the player doesn't realize it's live so that's been something that we continue that we've had to continue to troubleshoot throughout the process but

one starting point that we found to be very helpful was to ask the question can this happen without a live actor and if the answer is yes then it probably isn't taking advantage of the liveness of the actor so that's that's been a really useful starting point of this can only happen if you have a live actor doing it um and uh and then once we go from there and we start to develop the actual ideas then we identify those key structural points that i was mentioning that will allow the actor to riff off of um while knowing what they need to accomplish and then each each actor can develop their own voice and their own characterization so that different characters can inhabit different uh live encounters or or live stage acts so some of the challenges that we found working in vr that are different from theater mostly

one of them is just the experience of being in this kind of awe-inspiring animated vr setting can be pretty overwhelming so both for the performer who may not have much experience in vr going in there uh it can just being in there can be such an overwhelming experience that it takes them a little time to find their footing to be able to have that kind of fluidity and flexibility to do all the things they need to do in there and it also can be overwhelming in a way for um for a player going in there who just sees this space that's expensive and they just you know want to go and see like this is a view from the stage um and so the audiences kind of would be where those stars and the star and square square are on the floor and then there's those stairs and the archway in the back that kind of invites you to go see what's out there um so there's a lot to kind of compete with um for the performer uh in in kind of addressing the audience

and another challenge if you're coming from theater is uh just the audience behaves differently than they would in a vr experience than they would in a theater experience so here we see the audience throwing onions at the stage and a theater audience is just typically more reverent to the to the performers and that's you know there's a lot of different kinds of theater things that can be called theater and some that where people are a lot more respectful and maybe some where they're they're more talkative you know there's a whole range but just generally speaking in vr or i think probably because of the gaming gaming tendencies and all of that it's like it's not like you're just being a jerk by throwing things that's actually a way to test the limits of the space it's like a legitimate way to encounter this space especially if you're you're being told that there's live elements but like you want to test that um so just understanding that the players coming in are not necessarily coming in there just to see you the actor do their

thing there they have um yeah so they're testing the limits and it's a different kind of it's a different way of kind of commanding the space so one of the things we so we can draw from some theater techniques to think about how to command the space as a performer and that includes physical and vocal techniques like i was saying the physical theater background is helpful for that and just things like having using your body in specific ways to kind of create suspense or command control through contrasting rhythm and same thing with the voice using different volume and different rhythm to kind of pregnant pauses etc to kind of draw people's attention in and break up the the rhythms that you're playing around with um and those actually have been very helpful especially for the live stage acts and then for the one-on-one encounters um also thinking about the use of just volume to create a more

intimate experience and draw people in versus when you're doing a live stage act and you want to reach the person who's running away or whatever um and then and then we can also turn to theater live performance formats like drag cabaret burlesque or and comedians live comedy um and how they do crowd work and um handle an audience so this is taylor mack who's like a superstar um drag performer and his advice is he he talks about how you know in a drag show you know you never know who you're going to be encountering you might have an audience member who just goes really rogue or is antagonistic or something and so he always says you know if that person is being a rogue if you just sit there and try to like ignore that then they're going to win their spotlight's always going to be brighter than yours but if you and if you try to kind of combat it and you know kind of take on that antagonistic relationship with them again their

spotlight's going to be brighter than yours and they're going to win so what he says to do and i'm totally paraphrasing but it's like you go and you stand inside of their spotlight and you kind of scoop them into what you're doing and you make them part of your act and then you win and that's actually a principle that the the performers have really been drawing from when they're doing the um especially the live stage act but actually in the live one-on-one encounters too you're trying to have this intimate moment and this particular player just wants to like swoop around and you know not get paying attention to you so you have to know you can't just like keep insisting that they do you kind of have to navigate that a little more artfully um and then so and then on top of those theater techniques we've been working with tender claws to kind of establish a power dynamic that helps the the actor command more control considering all those challenges i was mentioning so the technical interface and the kind of super powers that the the performers have like the ability to teleport a

player or make objects appear all of those things and the the fact that the live actors have the ability to speak and the players don't also helps them kind of have a little bit of an edge on the players in terms of commanding space and another key element here i mean i'm talking about these improvised encounters and so i think a lot of people when they hear about improvisation they're like oh you don't need to rehearse right but there's many reasons why actually improvising these acts requires even more rehearsal because you need to not only be comfortable with the interface as julian was describing but you also need to be able to have such a fluidity with the structure and with the world and mythology of the under and this character that you're developing that you can change course at any moment um if you don't have that rehearsal time then you'll just be trying to fit your script and you won't be able to have the fluidity to be responsive to all the changes happening around you it's different every time

and then the other key thing that we have discovered through this process is creating a balance between what i've been talking about so far are ways for the actor to kind of have control and command and draw attention but then the other side of that coin is to also let go and to allow the so you want to be able to acknowledge the players and make them feel seen and respond to what they're doing but you also don't want to insist that they pay attention to you the whole time so because like we've all been saying like just discovery and exploring the space are all really legitimate ways of inhabiting this experience so you know we want to be able to allow people to explore and run away from you if that's what they want to do but still acknowledging it so that it's clear that we see you and i'm a live person who sees what you're doing and then there's other fun ways of kind of teaching the players how to behave and that kind of allow you to be either

reverent or subversive so for example the live stage hosts are able to place the players in a cage um and so that's kind of like a punishment like for being bad you know if you throw something at the stage the live stage act can the host can say if you do that again i'm going to put you in the cage and of course then the player will most likely do it again and then they go in the cage so it's like a punishment but it's also kind of a reward because you get to see this other view of the space and so it's just ways of kind of and that's a really good example i think of putting the spotlight on the person and scooping them in so you're punishing them but you're it's just another layer of the experience um and then so at sundance we did what in theater we refer to as previews which is just the first time you encounter an audience and so this gave us a chance to really explore different types of audience behavior

so one of the things that we notice is that sometimes yes you have these people who just want to throw things the whole time and um sometimes the host would react to that um by also throwing things and now we're throwing things that's what we do now um but then you also would have these audiences who were really interested in just hugging everyone and just so that's why i have this huggers versus throwers and this is just all these different ways that the players have found to interact with each other and sometimes you have a group that's actually shy and if you don't have one player in there that's more outgoing to kind of teach the other players how to be outgoing then you might need the live actor to help encourage those kinds of interactions and then uh and then again another thing we discovered was the tendency for sometimes you would notice that a player is trying to say something verbally and that's not a power they have so then the actors would adjust what they were saying in the subsequent performances so that they wouldn't

they would take it it would take more advantage of the verbal tools that the players actually do have like snapping or they can nod or pick things up and hand things to each other and not ask questions or prom give them prompts that would make them think they're supposed to give you some kind of essay in response or monologue or whatever um and then uh and once again and i think i've already mentioned this but doing the previews also really showed gave us that a full experience of when you know you think you have this seven minute live act that has all these points but because of the other circumstances of what the players have experienced and what they haven't experienced you might just have to cut it short halfway through and completely change course and say okay well that's it for this show and then you you you whisk off turn into another character so that you can then be there to have a live encounter with some other with some other player so just having that kind of versatility and flexibility

becomes really key in in this setting um so then now i'm going to turn it over to tanya who's going to talk about production and the logistics of everything making all this happen thank you i'm tanya also i'm the producer for tender class and uh logistically how does it happen uh i'm briefly going to talk about organizing multi-disciplinary teams developing workflows for acting virtual spaces uh the glorious human factor and i'm going to close with why is this not a location-based experience so organizing multi-disciplinary teams one fun thing that we haven't mentioned here is that tender class is based in los angeles while pie hall is based in new york city so this hit on itself made it interesting we had to be really careful when coordinating all the communications

but also not only the communication but the way to translate the terms you know the terms that developers use versus the terms that the actors were using or even like things that samantha would be creating uh and we had to agree in naming conventions um because our teams are not caught with the same scissors so within tender class we all have different backgrounds we come from fine arts from film from programming from puppeteering theater and we all wear many hats all the hats sometimes this is not a hat um the combination of all these disciplines and the collaboration with piho helped us uh helped everybody in these teams brought their their perspectives we were able to get different vantage points and different situations it was very beneficial to have somebody attack the same problem from a different angle our

first organizational stone that you already heard about was the bible that samantha created and thanks to that we were able to allow for different departments to continue to add to this world a good example of this was understanding the layer the layout of the stage and how these would affect the live actors interactions with the players so for example our art director carter looked for a bunch of reference on stages and then jeff from piho would draw this elaborated sketch informing us of the actual layout of the theater and we had these discussions on ideas of how these spaces could be useful how could we use them to create moments with the live hosts and moments with the actors doing the one-on-one encounters and then our creative director danny would start working you know with the team on potential interact potential

interactions until we finally had our glorious stage um all these crowdsource information also helped make decisions to allow both the player and the live actors to have some agency for example a decision was made to not allow the players on stage unless the live host invites them and this way the live host can still control their space and have the ability to use this power to make the player feel special so then developing workflows this has been an ongoing process and will help us when we actually scale this project because we're going to have to train actors and the reality is we've realized that we need to train them in vr in the virtual world thankfully the untethered nature of the headsets that we've been developing in has allow us to actually rehearse in the headset in the virtual world where different people can log in

some of the actors will be training tara will also be there giving notes samantha will be there giving notes it's been a very interesting experience but because of what we soon had to consider the human factor we started having a lot of interesting findings one which tara already mentioned which is like when the actors come into this world settling into the vr space and into the character's physicality is something that varies from person from person to person embodying characters means puppeteering different sized bodies uh and the movements sometimes have to be like really exaggerated and you can only understand how to move your character in the space in vr for example one of our characters the cook oops sorry i'm gonna play with spazzo there you go uh the cook is a very large character and everybody who embodies it

you know they have to like do like this very very large gestures which we realize it becomes really really physical um so some of the learnings from our previews which uh tara talked briefly about essentially again we had to consider the human factor one interesting thing uh was how do we do actor shifts how you know how long are these people gonna be acting uh and we look to sag to the screen actor guild and equity which is a theater union but neither really reflected the work that these actors were doing sag actors can be on set for 12 hours but you know at the same time they have a lot of breaks every time that they're setting the camera changing scenes you know these actors are resting and in equity they're meant to have 10-minute breaks every hour and 20 minutes but all of

these really doesn't mean anything for the way our actors were working hello and welcome to thunder down at the under my name is jimmy string bean salami ohio and i'm here to show you a good time if you know how to play by the rules i know how to get a show to you um sorry even during our rehearsals uh one of the findings was understanding the physical and the mental demand for these actors um the live hoes have to be engaging loud they have to be trying to attract the attention of the players and at the same time interact with the rest of the audience actors doing one-on-ones or possessing npcs have to be in content state of alert and in character while they're in this virtual world waiting for the opportunity to start a new adventure with somebody but the reality is you start losing track of the time when you're in the space and the passage of time is really different so we settle

when we're doing the previews on two hour shifts for actors it will be like 30 minutes log in a 5 minute break 30 minutes again a larger 10 minute break at the hour and then again another hour uh and then we change actors and this seemed to work really good um so the last thing i'm going to talk about is why one of the why would we choose to do this uh non-location base if you know our work tender class creates long-form narrative projects and we're always looking for ways to encourage exploration and create the desire to come back and find out more about the world and the narrative in the underpresence we want to create a space where people want to come back to to be able to scale the experience we will have a loop of pre-recorded acts so that when players come into the world there's always something exciting happening the owner will run without dependency of the live host on stage however the part of the magic will be to find

yourself interacting with a live host or being swept away by an actor for a more intimate moment but you won't really know when that is gonna happen um finally our takeaways uh the way we've been trying to bring live actors into these spaces create modular scripts optimize responsiveness designing narrative to maximize actual resources build infrastructure to support objectives make players feel seen then set them free and consider the human factor when training actors to perform in virtual spaces so that's all for today we're going to open it up for questions if anybody wants to ask us anything thank you very much i don't know this is how i ask question i find i think it's really interesting when you talk about

trying to encourage players to understand that they're dealing with a real actor and um from a design from a game design a narrative design perspective can you guys elaborate on that at all as far as like um what tools are you giving or how are you directing the player in ways that give them utility to discover that they're dealing with a live actor um part of that is actually in the on the difficulty of real life the difficulty of something that's a more ai um and more of an npc to be exactly responsive in particular ways that you know is not possible by the presence and the movement and like that kind of the change of the discovery of how that happens and in there's actually not all of the one-on-one encounters are already pre-recorded voice a lot of them are live voice so that actually you know can contribute

significantly um and being able to call very one to one on to what the player has done and make references to things they've already done previously is one of the ways we're doing it right like so if if a player picked up an object and then dropped it and then picked it up again then to like really oh you who dropped the martini glass and can't keep you know can't keep it in your hand maybe you've had too many already you know like just finding ways to be really specific to that player and what they've done helps them there's no way that like it's pro program like there's no way that that player's actions have been done consistently enough for that to have been pre-programmed or something this is probably a question that would be answered by actually playing the game but i'm just trying to get clarity on the balance between the sound board and the live dialogue if people have the sound board and the reason for that was to have a consistency of voice with the

pre-recorded what you do with the voice when it's live and how do you balance that can can you help yeah i'd say like 80 of all the live encounters are not sound boards it's a very specific case for like very specific npcs that the player already has relationships with in the game um so when you get to the sound board it's a thing where it may not you you may not realize it can be played by different actors so specific characters have sound boards and other characters are created by each actor in the moment okay got it cool thank you um yeah so you mentioned that players can talk i was wondering if that was a decision from the get-go for if you had experimented with letting the players talk um yeah i think we can all talk to this that was a decision from the get-go for many reasons one of it is we were

inspired by journey in our decision to you know the kind of response of this of like the emergent types of play and gesture and behavior that comes from that um and we're also we wanted to give like power to the actors as well but yes yeah it allows the act it allows the actor to guide the experience a little more fully um because as soon as as soon as the players are able to speak then it's like a whole other narrative emerges and they bring in their own real-life history and um it there's so many narrative components to this experience and it's such a unique world with its own rules that having having the players bring in a whole his personal history can really it makes it a lot harder to guide them through the narrative and also from a strictly technical perspective it's a lot easier to have bandwidth for a single person talking at a time than every player on stage at the same time um and you know you all know how bad skype calls are try imagining that in vr

one other thing too like i think a lot of people at least in the theater world like when you think about interactive stuff or where you're being put on the spot i i think a lot of us feel like ugh don't make me talk so i think what we found in um the preview period with this was like a lot of people actually found it really freeing to not have to say things and to be able to like it kind of open them up to take more risks and to participate more because then they didn't have to like worry about saying something clever or something like that so and i know that that's a taste thing but um i've i personally found that freeing the same basic question so just as a quick follow-up and do you see yourselves ever exploring that space where players are a speaker do you think it's just like the the challenges are too high and it's not really something you would um i think that something that danny who's our creative director also mentions too is the decision for the

players to not be able to speak is also coming from giving them different ways to explore the world um and that is something that we are actively trying to do so you know like they have these things like the snapping or uh holding hands and it's really like a way to create ways for them to explore i think there's plenty i mean we're not you know we're trying to do a very different thing than alt space in terms of our artistic intentions so yeah okay um uh well so so fantastic and i i love um i love not being able to speak and sleep no more i mean i think that's uh such a wonderful part of that experience so don't don't let the players right i don't think um my question is i guess mostly for tara but maybe for everybody in particular like how did you approach um movement for the performers you know in terms of well just their performance and

and like veering on dance yeah um i mean i think it really felt for me connected to our work in puppetry and mask but thinking about efficiency of gesture big enough gestures that can read and um things can get really muddy i think what we found when we were doing motion capture for the pre-recorded stuff it's amazing how quickly how many little gestures the the the setup picks up and so we realized that we just kind of had to pare things down in a way because if it got too muddy um or if it if it got if yeah if it wasn't a fish if the gestures weren't efficient enough it would start to get muddy and blurry and it would be harder to kind of it wasn't as legible basically um yeah uh and then with the live encounter interface it's a little bit different than the motion capture because you don't have like sensors on the body so um

yeah so that and then there's also the like interface you have to interact with and all of that um but it's a lot more in the upper body and a lot less in the in the lower body i'll say part of one of the aspects of the scalability is we're using quests that are dispersed over like hubs in various places so it's the it's not like motion capture it's using just the quests for the movement yeah so that's why we don't really have their legs in the picture so much i mean they can walk there's creative decisions we make around that yeah okay thanks this is also a question uh more for tara um being so involved uh purely with live performance how did you find tender claws and how did you realize that your future was with this vr tech um like how did we how did i meet tender claws yeah sure well we've known each other for a long time and we've actually been working in parallel with one another for over a decade and so we've actually always had

a shared sensibility and we've also all we're both very interdisciplinary companies with members with all different backgrounds it's just that we're often bringing people from different disciplines into theater and they're doing this more in interactive media of different types so it's kind of funny that this is our first like full-on collaboration um but it was exciting for us to kind of do what we were always doing with other artists bringing them into our home and to like actually bring our stuff to another discipline um yeah and we pitched and created this project explicitly to work with them thank you three more minutes um how do you plan on maintaining a project like this like obviously you have to have live actors in there and then supporting players from different time zones across the world possibly and then like the longevity of it

like how do you how do you plan on tackling that yeah there's a that's a complicated question with a lot of different stages um i think we can all speak a little to it um we're gonna treat these like runs of theater uh we're gonna have essentially at the beginning like a four month live service and we'll see how that turns out uh we're saying the the whole game is meant to be can run on its own without the live actors um but depending on how starts playing things start playing out we'll have like different runs or not if it gets super popular then we can pay a lot of more actors but we are you know considering for both things like the live service and what happens when that's not there i think tonya already mentioned this but the idea right now is not to have live actors available 24 7. so as a player you would go in not knowing whether you'll get a live actor or not a live encounter or not so that's part of the experience i think right right now we're

testing on amer north american time zones so probably from 10 a.m to like x amount like 9 a.m um and we can use data from vvr to kind of you know work with like scheduling but we do want there will be multiple more than one actor at the same time yeah yeah and some of our monitoring interfaces are built so that we can see what usage is at specific times and that will help us adjust as time goes on cool thanks uh i'm sorry that we have very little time left so this is going to be the last question we can um uh meet up in the wrap-up room in uh room 204 after the talk i just had a question about the um sort of direct improvisational acting versus the sound box stuff is at one point you had an actor directly speaking and other times you're they're choosing what to say is there a balance between those um how do you balance that

and what there must be a lot that's lost in the sound box versus versus direct dialogue if that is happening at all yeah so direct dialogue is 80 and it's mostly other characters the sound boards are very specific npcs that um because there's about four hour content of just single player game and multiplayer experience that you don't see that's not involved um well that the actors come in and out of um so there's particular npcs that you meet and you get to know over time and because their identity is established that's one of the reasons that we kind of have this like a sound board but it's not um it's really the delicate and the graceful way of writing that sound board that doesn't make it feel canned which is important so it's a separate category of actor that will be sort of hired to operate the sound board or um it's just a different role that you're playing it's a different role people will be we've trained people on different methods yeah and your intention is that that as you scale up you'll just you'll hire

more and more actors to fill those speak direct dialogue speaking roles um yeah yes we right now we plan to have a couple of hubs where we'll have actors there working uh doing shifts um because we're using the quest we can just give headsets to some actors like for example pai hall and have them log in from their homes uh julian is working on a system where we can start identifying the times where there's more people we can send alerts being like oh my god there's a lot of players right now we need like people in and the actors who have been giving headsets can just log in and it's important to mention that we actually have an elaborate behind the scenes system of prioritizing people who have not yet seen a live actor in different ways so that they're you know sorted in specific ways so cool thanks guys thank you i think that's uh that's all our time thank you so much