Lessons From Theatre That Can Transform XR Experiences

[Music] hi i'm james horn and i'm creative director at pixel toys limited we're a vr and mobile game developer based in the center of the uk in leafy lemington spa a creative hub for the uk games industry and also home to code masters playground games and ubisoft amongst others as a game studio we're known most recently for games workshop games such as warhammer 40 000 freeblade age of sigmar realm war and most recently warhammer 40 000 battle sister which is available now on oculus quest and rift in addition we've also made drop dead a zombie narrative shooter that originated on gear vr and has found great success on quest over the last few years now none of this of course sounds even remotely linked to theater so let me give you some of that side too i trained in theater and drama at aberystwyth university in deepest wales an absolutely beautiful place which also

has a history of pursuing creative innovative theater whilst there i worked with some fantastic pioneering creatives such as mike pearson of brief gough richard downing of human zoo and dr david raby of lurking truth what they have in common as theater makers and what i saw in them was the breadth of theater and how it can break through from the more stifling or traditional forms that can often be seen in the higher profile theater that we see today in london or on broadway my work there concluded with the founding of word theatre a small experimental theater company that produced a series of devised pieces over a number of years across west wales our largest performance was word a performative journey through the story of the bible soundtracked by big beat music liberal use of angle grinders and a lot of scaffolding poles that selled out to uh to play to sell out audiences across wales and very much achieve what

we set out for it to do it was about innovative and new ways to express old ideas using what at the time was pretty high technology so what i'm going to do today is i'm going to take you through my journey with theater and games jumping into xr as we go through and give you some of my thoughts on how i believe the two interrelate the synergies that are there and most importantly how they can inform each other in their creative processes to do this i've grouped my thoughts into four main headlines that i'll explore in more detail as we go on those headlines are theater is spatial theater is interpersonal theater is refined and theater is okay so let's fast forward career-wise i worked for a time as an actor and i did various bit parts in tv shows and bits of theater but eventually ended up changing careers and heading into video games

as mentioned previously lemington spa is a hub for games development so i joined code masters working as an environment artist on dirt later on moving to freestyle games where we made the amazing dj hero as part of activision before ultimately ending up at pixel toys this is where my eyes were opened to the possibilities of theater and games coming together and it came via a samsung phone strapped into a plastic holder i joined the company just as they were beginning a project on gear vr it was creating a zombie shooter now i actually find zombie shooters a little bit lazy and cliched but what struck me as i put on that headset for the first time even in a fairly rough prototype form with prototype code was the sense of space immersion yes but also that depth that sense of architecture the relationship of you the protagonist the audience

to the content and this is a key tenet of theater and recognition of this is vital for creating compelling vr experiences so what do i mean by this well peter brook the acclaimed theater director wrote in the opening lines of his seminal book the empty space in 1968 i can take any empty space and call it a bare stage the heart of theater the most basic unit is a space for performance whether this is the amphitheaters of ancient greece the commedia del arte performing in the the piazzas of tuscany shakespeare's great wooden o the globe in london through the proscenium of the victorian age and then those paradigms being shattered in the 20th century by modernism and experimental theater things like theater in the round site-specific work

but one thing they all have in common is a physical defined space a spatial relationship between the audience and the performers so why does this matter for xr and i'll use that term now rather than vr because this is true for augmented reality mixed reality spatial computing whatever other terms you like to use they exist within a space in the case of vr it's a digital space that cuts out the real world and shuts it out creates a new world for augmented reality it's a digital space that sits on top of and exists in relation to our real space in the case of spatial computing it's a digital space that interacts with our real world space and this is a significant difference to flat screen or pancake formats as they're often called which are not i the user i'm kept at a fixed distance

through the window and my presence is not relevant to the content this is actually an ideal moment to bring another small part of my journey with theater and digital technology out through 2019 into 2020 i had the pleasure of being one of six digital fellows working with the royal shakespeare company and magic leap plus some fantastic partners in i2 media at goldsmiths and the university of portsmouth we were researching how digital technology could transform theater and performance going into the future and the relationship of audiences to digital technology over the course of the year i investigated the potential of magic leap working with the other fellows to create an interactive dining experience based on a midsummer night's dream and featuring live actors projection mapping and the magic leap we're preparing to present this work which is titled where once we at south

by south west in march 2020 and i think i probably don't need to sum up ultimately what happened to those performances as sadly global events overtook things but i'll share some elements and learnings from this as we go along any piece of theatre considers the space that it will take place in and defines it for its own purposes the needs of the drama are reflected in this a stage designer working with the director will spend a great deal of time during the design phase of theater considering the movement and the relationships between the audience and the performers proximity can heighten threat or intimacy space and isolation can reinforce a sense of loneliness a busy ever moving stage can illustrate a create chaotic drama this is very important for how we create rxr experiences too we need to design them for space

an example of this was from my experience was a devised theater piece i worked on some years ago called jenae directed by judith aston as part of the show based on the life of the anarchist poet jean gene who we can see in the center there we wanted to create the feel of a descent into madness the audience were positioned around the room and an increasing number of actors moved around the space surrounding them moving even between them the action music and sounds were getting less and less focused on in on single actors and individual performances and became more and more chaotic around the audience members as the madness built certainly wasn't a comfortable experience for the audience it was certainly memorable and striking and this is very important for how we create our xr experiences too we need to divide design them for space for xr then we need an awareness of

space and spatial relationships this is both the actual physical space that the the experienced takes place within uh in the case of augmented reality for example we need to author an experience that is flexible to use a space yet it's also defined and controlled how do we restrict where the user can place the experience and then how can we take advantage of this we need to consider those usage cases where will the experience likely be what are the commonalities where in the world will this be is it is it local or is it global a living room in australia is a very different place from one in the uk in terms of space likewise the us and ultimately what are the spatial relationships that are core to your audience when we began developing the dream magic leap rsc project in 2019 we came to realize that this ambiguity about space

was a huge opportunity but also a huge challenge in the case of magic leap the headset can see and can scan the world this is absolutely amazing but is but is it actually aware of what the world is and thus can it respond to it the engineers at magic leap working hard to add this functionality into the sdk this is object semantics and that is ultimately is the code understanding the meaning of something which is actually a step further than computer vision so when i scan a chair or a table the system knows not only that that shape is likely a table or a chair but what a table or chair represents or is useful and then how does our authored content relate to that for more information there are some really great videos from magic leap out there including a very comprehensive breakdown of lumen os perception

and david molyneux which is well worth checking out so what do i mean by this it's talking about spatial awareness and performance and why would this be useful this again circles back to theater and acting an actor performing within a space is aware of and understands the space in which they act by means of what are commonly referred to as physical actions and we've got a great quote here from from constantin stanislavski the great theorist and actor in every physical action unless it is purely mechanical there is concealed some inner action some feelings this is how the two levels of life in a part are created the inner the things going on within and the outer the things we're presenting they are intertwined a common purpose brings them together and reinforces the unbreakable bond

that actor performs within a space they respond to that space this could be the simple action of leaning on the back of the chair whilst talking it could be the subconscious coming through as you use a prop to stem and define the dramatic action in an xr experience that activity when related to a real world space can be incredibly compelling at a very basic level uh playing dr grauds on magic leap for one of the first times in my home and when facing the enemies it took me a moment to realize that the enemies were both behind my sofa using it as cover and also shooting around some of the objects in my room these characters were in my space and actually at that moment the experience becomes much more profound this content is not being projected on my world but rather is within my world and within my reality it then becomes a form of theater

anthony arto was a visionary playwright and theorist in and in 1938 actually coined the phrase virtual reality for the first time uh in this quote he talks about how um how alchemists create gold and that's within uh within the theater when characters object images come together in the theater it becomes a virtual reality and that's where the really amazing stuff is the theater performance is a virtual reality the xr experience is a virtual reality we create immersive realities for our audiences so these same spatial relationships are hugely important when developing a virtual reality game experience of course recently we launched warhammer 40 000 battle system on oculus quest as

mentioned before and one thing within that warhammer world is the sense of scale and space that we wanted to create we wanted to have soaring cathedrals we wanted to have that sense of scale that the brand carries with it these huge brutalist concrete structures over size people tanks weaponry it's really a striking thing when you first experience a space marine spatially in relation to you in vr games workshop he made the franchise even questioned whether the scale was correct they stood beside this eight and a half foot tall marine for the first time in vr so in this sense we we designed our environments and our gameplay to maximize the theater of our vr experience now the core narrative within our game is between the player another battle sister called sister rosalina and a primary space marine called julius they're played by the fantastic victoria and obie that you can

see here within the game we have a motion captured narrative and i was really keen for those moments that you have where you where you're telling a story between these three characters that you would have that spatial intimacy that you get when you're in a theater this is particularly interesting with a space marine who's eight and a half foot tall the amazing thing about telling a story in this way is that there is a sense of presence that you as an audience member get when you're standing close to these actors who are responding to you looking at you and each other in order to do this we had to treat the motion capture shoot like a theater experience i directed the shoot from the point of view of the virtual audience member so we get that connection those little looks between characters that intimacy this for me was one of the reasons why i was so excited about virtual reality we are telling stories from and showing

actors performances in this very personal way we configure the space the the way we block out that action in exactly the same way as we would in the theater block like theater we're trying to capture the nuance of performance and ultimately we're configuring the space to do that and this is where we cross over from the theater of theater is spatial into theater is interpersonal because it's about space but it's also about people and the relationship between people and that contact now obviously i know there are lots of fantastic virtual and augmented reality works which have no figurative or human elements within them but there is definitely something special about meeting human beings within a virtual space you can see the popularity of things like vr chat and the under presence performance of the tempest which brought

live theater into virtual reality likewise the rsc's dream project earlier this year that streamed live mocap to huge audiences perhaps this is magnified by the situation we've been in over the last year and a half under lockdowns but just meeting people digitally or virtually still has a power something that's also important to take from this interpersonal theater is the contact that takes place between an audience and an actor if you watch a recording of a live piece of theater on tv it's often somehow a drier less exciting experience particularly a film before an empty house add an audience and that energy rises be there in person and it's an entirely different experience partly this is because of the shared experience you have with other audience members that unique moment in time

shared with a select few but also because the audience completes the experience shown here are images from king john fantastic uh performances that played at the swan theater in stratford in 2019 2020 the excitement of the audience in this show was so important to two really key scenes within the play there was a boxing match that ranged across the stage the pugilists were even falling into the crowd at times as the as it as the performance almost became a sporting event the messy match taking place then again later there's a wedding that descends into a real food fight where the audience roars with laughter whilst trying to avoid being hit by flying cakes and hot dogs the actors play to the crowd they catch their eyes they respond to their responses they feed on the energy of an audience

and this is what we're missing by not having live vents at the moment and this can be the same with our xr experiences in battle sister we had our mocap but we intentionally broke our mocap to introduce elements of this into our dramatic action we know the player is wearing a sixth off helmet oculus quest oculus rift and we'll be meeting when we'll be moving around the scene as our actors talk so what we did was we layered in responses through predominantly through look at and they make a huge difference um to to the feeling of immersion within that scene we have responsive gestures where characters will hand the the player items at certain parts within the storyline and it feels great to reach out and take that from a character who's who's reaching out to where you are we'd love to take this further obviously layering in more responsive systems putting in alternative takes and captures for responses to different ways

that the main the main player can can respond and what i'd say is it's really great to consider this kind of thing for impactful moments you don't have to put it everywhere but for those moments where you need that contact those those key parts the drama it's worth doing however i would say it is quite expensive to develop adding any of those systems in and trying to do them in a way that's subtle and looks natural is quite hard it's definitely a cross-discipline effort you need your coders you need your narrative designers and you definitely need your animators in there as well all to try and get that subtlety and to make sure that those systems are finely tuned to where the player is looking and ultimately there are diminishing returns as well so i'd say keep those in fact impactful moments but the more responsiveness we can get to play a position uh the better now whilst working on the magic leap rsc fellowship year

i did a series of experiments using motion capture to convey a piece of shakespeare the duke of clarence's speech from richard iii i found that the sort of filmic motion capture performance didn't really work when placed into an xr environment in this case the magic leap partially this is the hardware the light field technology is fairly early in the magic league one which means that the subtle nuances are captured are lost likewise a limited field of view on the headset meant that full-size humans didn't really fit very well in the frame and this led to the experience becoming more of a traditional tabletop look shrinking down the performance so it can be viewed in full but what this required was first was for us to for the actor to play to the crowd giving a much bigger performance that you can see here on the on the lovely unity robot um the rsc's own seven ages of man also shows this in a very refined way

so if you're designing your your experience theatrically with that awareness of space and the power of those interpersonal connections you can also make the user work a bit to see that so what i mean by that is framing the action so the player is encouraged to look around interact and interrogate the performance to engage if we look at the scene from part way through battle system we can see the player's view of malachi he's our resident inquisitor a traditional cutscene would frame the actors so we would catch all of their performances in one or two shots depending on the camera angle there can be a tendency for us to do this in xr2 to frame all of that action right in front of where where the player or the user is looking however in exile we can stage the action around the player so as the action unfolds they look left they look right they engage with the action this is certainly a more engrossing

experience and can lead to an involvement that is ultimately more memorable on a practical note i'd say that if you want to do this um you need to rehearse with your actors and what i'd say is we always book up we book in our shoots for motion capture and then we kind of work back from there book in more rehearsal time with your actors and get them as involved as possible you need to build a company of actors and help them to understand what the game is going to be help them to understand the nuances of their individual characters get them their script as far in advance as possible share it with them share all the details you can concept art and you'll have some actors who are much more engaged with the piece that you're creating making them aware of where their audience is going to be is also crucial one thing we found when we were working with um with victoria and obi is that

working towards a virtual reality camera is much harder because the player can be wherever they are so we had to really target down where we think where we want the player to be listen to your actors as well theater productions are collaborations and not dictatorships and act as a professionals and have really good insights into how a drama can unfold i would encourage people to bring along your lead animator to any motion capture shoots that you're doing to watch that data capture and work with the team and some of these are slightly more common sense within within motion capture but blocking your environments and using accurate blocking models makes a huge difference and making sure your props have got accurate grip points means that extra step of accuracy will save you a whole load in the long run and what i'd also say is as you're rehearsing refine your action don't be afraid to cut don't be afraid to to cut out all that fat and flab that's sitting in that

sitting within those performances the great thing about that last note is that it nicely segues into my next headline the theater is refined now of course most xr experiences these days are the results of a great deal of thought time and effort but one thing that struck me about theater as i worked with the rsc over the course of the year was the stages and levels of concerted refinement that took place i was reflecting on the difference between what in games and films we would call the pre-production and production processes and theatrical rehearsal processes now on the surface these two sets of processes can seem superficially similar but there are quite significant differences in how they are structured so i'm going to share some experiences i've had in rehearsals in the past and give you some examples now it's worth saying i'm talking here about my experiences of theatrical process there are many and they can be very short sometimes too short

and can also be entirely different but usually there is somebody with an idea of what they want to present often this is embedded in a script a story to be told staged or revived in theater text is often the anchor that holds the production together a backbone or an armature which the different elements can be hung off if we're actually honest do rx xr experiences have that backbone that anchor i've been involved in many games where the vision the central idea of the game still isn't truly resolved even at launch and if you're still finding the core the truth the vision of that game right to the end of production then your development team is swirling around with that backbone you can see here a typical theater process this can be anywhere from from one year to possibly even to five years long with a long script development process a design stage where

staging lighting uh visual design is developed casting can be a year before music composed alongside that as well and then your visual graphic design is often chosen much that's long before the rest of the production is developed it's an elongated experimental period and there's loads of really interesting uh opportunities for ideas to be developed lots of discussion uh dramaticy as well is in there which is about editing the drama with where once we which was uh the ultimate aim for our magic leap rsc event we followed this pattern uh not with that length of duration but it was a cross-discipline creative process with a real creed freedom to ideate we came up with loads of ideas loads of images throwing lots of things around we explored the text together reading the whole play aloud several times and just listening to the changes that's the spoken text brought

we had space we had time to immerse ourselves in the project we looked at loads of different images related to to classical shakespeare to old traditions of warwickshire what does a forest mean and other things like that as we progressed we created a things we know to be true document to begin to cement some of these ideas down and this is actually crucial for a design process being able to know when to lock ideas down and to begin to create the foundations upon which your creative project is going to be built we had a lot of unknowns relating to budget performance dates technical support for app development so we focused on elements we could control and this was a fantastic creative process now at a certain point usually dictated by budget and release date a theater

company will move into actual rehearsal and this is the meat of theater making actors directors producers and other creatives in a common space working on the production theater is rarely a pure dictatorship and i do choose my words carefully there truly successful theatre empowers the actors and creatives so the process so the process is truly collaborative the production is discovered through the process of rehearsal it is continually refined in alignment with the overall vision of the piece well there's often is that there is an intense focus and an honesty there and actually i think that's really something we need to value and we need to encourage within our within our games as well and often it's not too late to amend for where once we we actually had a very short rehearsal time as we pulled

together the technical aspects of the magic leap experience the music the physical props these physical props featured projection mapping small physical details that audiences could interact with and it was all due to be set within a cd bar in texas during the south by south west festival during rehearsals in stratford and following discussions with a wide range of test audiences we heavily reworked sections of narrative to tighten the experience and to make the narrative more compelling this was really difficult and actually controversial amongst us as a group likewise certain technical elements just didn't work so well in magic leap and so we revised the conclusion of the piece that the ultimate end point simplifying and streamlining the ending ultimately the piece we prepared was more accessible and more practical than our earlier version and was ready for first night

a night that sadly never came to be and that's the presentation that's the final stage of the process um i've grouped tech week into here um this is talking about theater where it's that really important moment like we said before where it's the coming together of all those elements it's very similar to the final qa phase where all that you have within game dev where all the nasty bugs come out and lighting costumes sets music will all be in place there's actually a very fast period of evolution that takes place then in theater the performance can still be developing at this stage responding to the moment plane isn't even fine at this stage after even after the first night the great power of theatre is that it's unique every night a special occasion every performance performers are working with the audience so relating this back then to game dev and xr dev

we have our idea and usually never enough time for anything but we should and never enough budget but we should learn from theater about the design process that pre-production shouldn't rush too fast to prototype do we actually give our creatives time and space to actually think about ideas to play with crazy notions to experiment we should elongate this process allowing time for ideas to gestate trusting creatives to be creative if we have narrative elements then we need to bring our writers in earlier than we do we need to get them get them to engage with the earlier stages of production so their work is fully integrated into the whole involving cast earlier as well pays huge dividends performance experiences can really help shape a piece of drama

particularly if you employ someone like victoria like we did it on battle sister who's done so many so much games work before her experience was so valuable to us as we developed ultimately this process should be about refining and defining a vision for the project that can stand up to argument and scrutiny this means having those difficult conversations and working through the conflicts a solid vision will be worth that pain in late development worth worth that pain and data development so how do we define a shared vision well ultimately it requires a great deal of negotiation and discussion between all the relevant stakeholders and should be laid out and communicated to and with the team really clearly this becomes the test that the entire project this becomes the the test point where we can test our entire project against and it's very important during the later production

it's very important to form a company of people that trust one another now there are many ways to do this of course but ensuring that there is a safe space for people to express ideas without them getting immediately shot down is vital many theater companies spend a lot of early rehearsal time just getting to know each other socializing learning about each other's backgrounds makes them more cohesive and more creative unit and ultimately a unit that trusts itself with ideas keeping them nebulous to begin with and then beginning to narrow them later it makes a huge you know can be really really helpful things like myroboards are really great for this feeding in the advice of specialists writers composers actors to help develop your vision and then ultimately your directors must direct whether you've got one director or a series of directors in different disciplines your directors need to take you through

because group think rarely arrives at a clean and coherent hole testing the idea with your friends family market research as well is is really a great thing to do all feedback is good even the dif even the negative stuff you just need to work out how to interpret that and ultimately you need to take risks you need to be creative in that when you're making your vision so once we move through into full production this is the meat of game dev uh where we actually make the thing like a theater rehearsal this should be a group of professionals enabling each other and working collaboratively collaboratively towards that shared vision held a loft usually by the director this could be a longer period of time and so we need to make sure that we've got enough stamina for that and we need to remember to keep refining as we go forward sometimes it can feel like a slog just to just to get everything over the line um

but that's that's fine we're often still answering pre-production questions and that actually is a problem we should try and stamp out any of those pre-production questions which we should have answered way back in pre-prod now change will of course be needed as we're doing this um but we should try and make this subtractive change rather than additive let's try and cut things out let's try and tighten up the experience rather than bloating it user testing as well can be really helpful as mentioned in the previous if it's done right and if you've got somebody who can actually be interpreting that data for you so the director needs to make time to review the work as it as as it progresses and this importance of the review to this process is really really key production is a long slog and so having those regular moments of review and retrospective is is vital to make a compelling experiences agile can help with this

although there are other ways you can do it you need to make decisions and that's ultimately that's the director's job but individually we need to make decisions as a dev team uh not wallow around in in indecision at this stage that was our pre-production process where we were wallowing now we're making decisions but it's it's rarely too late to change your mind either keep cutting things if you need to most plays are flabby most games are flabby let's tighten it up as we keep going so as we come to release and the joy of those final few weeks or months of qa where all those bugs start to come out we need to still consider and think about refinement things can still be cut although no adding please remember that there's a content lock and we should plan for post and we should have a plan for post launch in this ever connected age the potential to evolve an xr experience um post launch is actually really huge

but we do need to allocate budget to allow for that think about how long you want to support your experience post first launch and have a plan for how you'll measure data to inform that process in larger companies you'll be blessed with data analysts but for smaller xr devs who'll do this how will you evaluate that feedback and feed that back into the development process this is exactly the same process you as you would do in a theater production and and we need to listen to those reviews until i come to my last my last main headline theater is so what do i mean by this well uh ultimately theater is confident it's brash it's political it's resilient it's fun it's challenging it's cheap it's expensive it's unafraid of risks it's dangerous it's exciting it's nuanced

but it's also rough it's resourceful it's ever-changing it's human it's now it's unapologetic it's expressive it's interactive it's surprising loud a community a place to create it's cutting edge it's redefining itself every day it's fresh it's immediate it's relevant it's powerful a lot of words i know but we need to take that from theater theater doesn't try and defend itself it just is it's all of those things at once playwright howard barker says the weakness of contemporary theater is its fear of the audience the horror that they may not understand the scene that they may lose the thread of the narrative that we shall be accused of elitism if meaning becomes obscure this cripples the theater with his false democracy he's talking about boring theater but what he what he's loving is exciting theater so what about us well with our xr

experiences we need to be confident in what we make we need to have that theater confidence we don't need to defend uh what we're building whether it's x whether it's ar whether it's spatial computing whether it's vr we need to take risks and be dangerous this is an exciting time for for this medium we've got loads of really creative new things new things every day new ways of using technology new ways of interacting we need to trust our audience to engage with what we create ultimately we don't necessarily have loads of money so we need to be resourceful and we need to find new ways to solve problems and one way of doing that is we need to be a community that supports one another and it's open we need to share those learnings with each other build each other up so in summary looking back through through what we've been talking about theater is spatial

we design it for our spaces we need to author the experiences that we make to maximize immersion we need to stage our action within defined spaces we need to embrace the theater of the interpersonal understand the relationship of your project to the audience we need to find that human contact we need to build responsive systems so that your audiences know that that they're there we need to be collaborative with our creatives theater is refined so we need to define that vision we need to build our company our company of actors our company of creatives and then we need to review and we need to direct our work and ultimately theatre is we need to be confident we need to be a community we need to take risks and ultimately we need to create thank you so much for listening

pixel toys is hiring uh i'm always happy to talk through any of the things that we've been talking about today you can find my twitter handle just there and my emails there as well so i hope you've enjoyed the talk and if anyone's got any questions i'm very happy to answer [Music]