HoloLens and Beyond: AR Game Design Challenges and Open Problems

[Music] so thank you all for coming welcome to hololens and beyond my name is bart Ned Lasky and I'll be discussing augmented reality game design challenges and open problems so just a little bit about me I'm not actually a game developer nor my dog mental reality developer but I've been spending a lot of time with it lately just had a personal interest you can find out more about me by visiting my website augmented reality is set to explode I think everybody recognizes that at this point and it's an exciting time right now I think you can divide devices up into two categories mobile and headset Mobile is pretty ubiquitous I think almost everyone in this room probably has a device that can do AR but how many people here just a show of hands have used an AR headset Wow so actually quite a lot more than I expected so right now the the hololens is the leading edge device despite being two years old and that's what I've been

using you know to do my AR game development experiments just an overview of the capabilities for those who might not know it's a fully self-contained computer it's it's untethered nope no connection to Wi-Fi and necessary you can walk around the room do whatever you'd like it has a depth camera for acquiring the geometry of the environment that you're in as well as detecting hand gestures now gesture support is very limited it's not like a leap motion device that's why you won't really see a lot of hand stuff going on in my in my demos it has excellent spatial audio capability that's often overlooked it can do 3d positional audio very convincingly and it has voice recognition so I've been doing a lot of just little prototypes and games in my in my spare time two of them are up on the hololens app store and I'll just kind of go through those really quickly to show you kind of what I've been working on [Music] so this one is just a straightforward helicopter action game controlled by an Xbox 360 pad and it was inspired by

Electronic Arts strike series if everybody remembers gentle sprite jungle strike or desert strike back in the early 90s that was my inspiration for this the the next game that I've released is a very different it's an educational game it teaches you how to ask for and understand spoken Japanese directions you play a little bear you have to guide them around a town delivering honey and asking for the location of the next customer and there are segments throughout the game that will introduce progressively more vocabulary so you actually look away and take like a little mini lesson you being a good post office buing hair salon [Music] Masuda pet fat minnow meets yo you got a NEMA gotta go straight

[Music] Ximena et Ossa daddy NEMA got the turn love at the second Street so that's the the gist of those two games so let's talk about game flow first of all a typical AR game involves scanning the environment first so the device can acquire the 3d geometry of the of the level and then that is retained placement comes next this can be done automatically you can place units based on the geometry you have or in some cases scanning and placement are actually the same thing in doko desuka the user places things as the depth camera is turned on and there's no actual pre scanning phase so once you're done placing it's time to play and you can optionally continue to refine the

scan and use the depth camera during play time challenges an AR game design I think can really be broken down into two big broad areas Nats challenges with the unpredictable environment and challenges with the players on familiarity using an AR headset the environment is obviously unpredictable you have no idea where someone's gonna be playing the game it could be here in the bathroom for whatever reason or on an airplane you have absolutely no idea but your game will be expected to work in any environment the placement and level design challenge I would say is mostly a technical challenge and I like to think of it as topology and context titute the two big factors here and what I mean by that topology I mean you know does the object actually fit within the topographic area that you have physically without actually you know asking whether it belongs there so for example why are these crystals on my toilet I have no idea they probably don't belong there context would answer that question and that's a matter of detecting objects so in this photograph the character was

actually placed after the couch was detected another design challenge that I found personally is is the issue of scale so in a traditional game you have complete control over the the scale of the levels they use perspective and so forth that's not the case in AR and I found that you want to have your holograms be relatively large to be kind of compelling and also the hardware doesn't really have the fidelity to support you know micro machine size vehicles running around but that also introduces the challenge of you know how do you scale for example physical buildings if you want to have them realistic compared to your objects so it's an artistic issue in this screenshot it's hard to see but the helicopter is way too big relative to the building and it's something that I never really thought about but I think as an artist or as artists team you might want to kind of consider that before embarking on an AR game the way placement works in the helicopter game I want to start off by having you know I want I want the maximum distance between structures in the games so there's a

helipad in a building and ironically this screenshot shows an erroneous placement where they're placed right next to each other that sometimes happens the constraint solver fails I do this by generating multiple different you know sizes of objects and different separation distances I do a bunch of trial placements and I pick the one that scores best vehicles are much more straightforward I distribute them between elevated surfaces and the floor oftentimes the scan will not have any elevated surfaces in which case they just fall back on the floor now there's a big opportunity for people that want to kind of move the field forward to I think to develop better constraint and placement solvers the options right now there's a few of them they allow you to kind of specify some basic constraints but they're not very flexible I think there's really a huge amount of opportunity to really do something much better dynamic navigation planning is a big thing being able to incorporate you know new areas that are scanned dynamically into like a much more robust navmesh type system I think that doesn't really exist off the shelf yet and the

big one is object recognition segmentation to provide context so that'll allow you to do more interesting and relevant placements around the room or whatever area you're in and of course facilitate character interactions with the environment and on that note you know what can we do with object recognition when I when I took this picture I immediately thought I don't know how many people remember Duke Nukem the the iconic alien scene right but you know this is just kind of silly but obviously you could imagine doing much more powerful interactions and in the way that pokemons go kind of inspired people to go out and explore just get out of the house you can imagine in the future having games that encourage people to just go around and explore how different objects interact with their characters or their game like what abilities are unlocked and what could possibly happen I think that's a hugely powerful aspect of AR that that I expect to see really explode the next couple of years this is just a wish list for placement solvers I'm gonna just skip over this and leave this for people who can you know want to download the talk later while I was

developing a helicopter game one challenge that I encountered was the AI helicopter had a lot of difficulty navigating around the room rooms were actually very cluttered in this case you can see it bumping into my what the hololens sees is my office chair so I tried you know basically just ray casting and just a kind of a heuristic approach I you know since I don't have control over the level I thought that I can kind of get away with this well it turns out it doesn't really work very well so I had to essentially create like a done you know a dynamic aerial nav mesh of sorts now you can see the helicopter is able to navigate around the office chair without any difficulty so it's a you know it's an interesting problem to have to consider you want to consider where the empty space is as well as where actual items are to accomplish this I just basically sampled the space ahead of time after the scan is complete here you see what are basically the center points of volumes and there are edges between so this forms a graph the yellow color coding indicates the largest island in the

graph so white nodes are not accessible from the yellow nodes and vice-versa and green is a particular pathway that you can take and you have to build this essentially dynamically when the level is first scanned because you don't have control over the level spawning vehicles in the middle of the game is a challenge a design challenge I would say and I got around the way the way I do it in my game is I have these so it's kind of a fun effect but a little bit gimmicky at the same time you don't wanna over use it for aerial vehicles I use flyovers which makes excellent use of the spatial audio capabilities you can really hear the helicopters coming up from behind and over you see and this problem with static scenery so the room is not going to change the players will be playing particular physical location you can't

change it and that actually deprives of the prize deprives us of a key reward mechanism and that is just just seeing the environment change so for example in a traditional game like here's sonic the players abilities stay the same throughout the game but a big motivator for going forward is seeing the you know what the next level actually looks like and I never really realized how much games actually depend on this reward mechanic and it's totally absent and augmented reality unfortunately so you kind of have to think about ways to work around that limitation I think one solution that's very clear is to empower the player to alter their own reality I've been told that Microsoft's acquisition of Minecraft was actually largely driven by their mixed reality plans going forward so this is kind of a concept image of what minecraft might look like but even without doing you know what kind of a sandbox totally open-ended game I think you can adapt a game like say a platformer while introducing a level design and alteration game mechanic that gives people the freedom to alter their environment so I developed this prototype as part of a kind of short competition that I entered last year and

you'll see megaman running around and collecting power-ups that allow him to extrude different kinds of platforms and the objectives kind of get to the next powerup [Music] [Music] [Applause] [Music] this is all controlled in the Xbox controller in my original idea you know you can imagine taking this to a much greater extent not just platforms but having all kinds of deformations to the level you know say pushing things into walls carving out tunnels you know bridges but in the game I would imagine that the entire room would be kind of transformed all kinds of interesting elements and I think that this game mechanic would work for other kinds of games like say a Bomberman type game or some sort of a competitive game it's something that I haven't really seen done and I think it's an idea that I would personally like to revisit once

more a our headsets become available and kind of commonplace so I had a lot of fun making it when people that have tried it out I've had a lot of fun playing it another big design challenge is the fact that the player controls the camera the players head is the camera much like VR you you can't really force a player to gaze in a particular location unlike VR however at least home VR you have near unlimited ability to walk around you can walk around the room without fear tripping over things because you are still in your real space and that should be leveraged I think but the narrative challenge still remains and I think you can kiss cinematic sequences goodbye for the most part but interestingly on mobile they're still perfectly doable you can you know switch between a mented and non augmented mode so that's one key difference between the platforms I would say something that I encounter a lot when I demo the hololens 2 to friends is what I call the freeze so here my coworker kind of forgets that you can walk around until I point out that hey you know you can actually move

around and get a better view of things it's it's surprisingly common and I think that since a our devices are gonna you know be a novelty for several years and you want to kind of keep in mind demoing these things you have to come up with strategies to draw the player out and and make them realize that hey you actually can walk around you're not to stand still it's not gonna disappear if you move around a big part of I think why that happens though is the limited field of view something that the hololens has been criticized for and I know upcoming our headsets kind of have the same limitation the videos that are recorded have Holograms present throughout the entire frame in reality there's there's kind of a clipping window that's quite small and it makes it easy especially in the helicopter game to lose track of things I use guidance arrows to point out where people should be looking but I found that kind of like the simple guidance arrow that most people are familiar with it they're gamers which uses the shortest angle distance it's very confusing what happens is people look down sometimes if there's something behind them and then they don't see that thing there and they just kind of like they're like well I don't know what's going on and they totally ignore the guidance arrows and they get confused

what I found works much more effectively is just simply limiting it to horizontal movement and then once something's within like a range of then if you know an angle sweep then do an up-and-down guidance and also just adding that look text made a huge difference when I started demoing it to people and I think you know there's there's room to do things like seamless confusion detection which is something I've kind of worked on haven't gotten great results yet but the idea is knowing all the stuff that's going on in a game where the players looking or you know whether they're like looking around rapidly you can figure out whether someone's confused and perhaps offered guidance the problem is you know you can also incorporate verbal cues like if people will often verbalize their confusion the challenge of course is limiting false positives many people might you know there's lots of times when you might want to look away intentionally like say you're talking to somebody and you don't want to start flashing indicators in front of the user so far my ideas that are kind of shown are not necessarily a are Native they're just adaptations of existing games but one thing that came to mind is that you have a lot of information about where

the players head is at any given you know frame so why not use that as an input device so I came up with this little game concept that I think is closer to something that's really a are Native it makes use of you know the environment interactions and that kind of room scale experience and it's something that really can only be done in AR and you know people that have tried it have had a lot of fun it's a very kinetic experience and again something that I think I'd like to revisit further if I had time Ferg anomic an issue that came up in doko desuka the experience like many takes place on the floor and it's very uncomfortable to look at the floor for extended periods of time unless you happen to be guest on from Beauty and the Beast most people don't have that kind of neck strength so you want to think about that in doko desu ka' the solution was proposed by a co-worker it's really simple you just allow the map to be elevated and you know it works great simple solution but definitely something to keep in mind it's more comfortable to have your head looking up than it is down now to really take advantage of

augmented reality or mixed reality I think you want to kind of get away from the design language of 2d screens unfortunately I know I'm guilty of this myself a lot of applications have you know 2d floating menus but what's really a lot more effective than pasting something in front of someone's face is to just make it incorporate your UI elements as part of the environment so for example these buttons are 3d objects you know in the level itself also what you can do is is map you know different kind of logical things that happen in the game to two different spatial areas in the room so in doko desuka the original idea was to have an instructor that stands off to the side I ended up with a blackboard but the idea is that you look away from the game when you want to learn or review vocabulary and you kind of do that mental context which the game knows you're not looking at it and then when you're ready you go back to the game and I think that's kind of a very natural feeling very intuitive way to leverage space in UX design so it also a key differentiator between Aeron vr of course that you can do this kind of like room scale stuff much more easily so in

summary what have I learned about making things fun I think what I've learned is that you want to keep a casual audience in mind it'll be a while before these things become widely accessible you'll have a lot of you know non hardcore gamers using it and you want to be attentive to that to their needs in kind of drawing them into these experiences you can adapt many existing game mechanics but I think the real payoff is in developing a are native content stuff that can only work on these platforms a big part of that is making use of the environment it adds that wow factor and really that's that's what makes things really AR native so there's there's a ton of opportunity to do creative things here and just keep in mind that people enjoy creating and sharing things and the ability to change their reality like feels magical to them and so you should really keep that in mind I think when designing games going forward for this kind of medium I think there's a lot to look forward to in coming years obviously improved headsets and also the AR cloud so for those who don't know this idea of having a digital soft copy

so to speak of the world everything kind of mapped out and accessible to facilitate multi-user experiences you know persistence and I think this will open up the door to what I would call like you know world scale location-based gaming and I wanted to leave you guys with this little prototype that I made a little demo using a major motion picture franchise for the art of what I think gaming might look like in 2021 [Music] agent Nicole do you read me copy that I've got your video and sound ready so this is the place doesn't look like much I'll bet them just the way they want it all right I'm going in [Music]

I'm picking up five emotion be careful [Music] this doesn't look good [Music] [Music] get out of there and for the exit so so is it is it actually arriving in 2021 well I think that depends on people in this room I hope that it becomes a reality and if you guys don't make it I think maybe I will so that's pretty much it I think we have ten minutes now for questions if anybody would like to ask any questions feel free thank you nicole

lazzaro from as you design I was curious what do you think is the largest you know unsolved problem like what would you like us to to do next for you know creating good experiences for our games what is what is the largest unsolved problem I think that's a very good question I think gosh there's a lot of stuff but I think that if I had to pick one thing that I certainly want investigate going forward is it's the object recognition segmentation making that much more you know easy to deal with I think that would really you know facilitate a lot that and and of course you know just improvements and sensing and so for betting object recognition if I had to like list one thing that I think could make things really magical so you know that Bai difference between a coffee table and a couch because your character would do different things if it was on exactly yeah being able to detect that

quickly and having an you know a way of robustly you know categorizing what the play space looks like I think would be you know just super amazing yeah I want to play a game where that knows that the best place to hide the body is in my fridge exactly right right thank you for developers what other tools are available for geo-located based gaming other than the new Google Maps feature and map box in terms of geolocation I'm actually not sure unfortunately I wish I knew I know that Google recently released that you know API on the hololens right now for example I there's no real built-in geolocation capability there's no GPS you'd have to do it through Wi-Fi or something I really don't know and that's I think an area that's being I think there are lots of like small startups for working on things but unfortunately I don't have any understanding of that yeah when you were scanning the room for the game what's the time frame for like the calibration

of the room between scanning it and starting the game that's a really good question the question was you know how long does it take from scanning the room to starting a game and the room scan itself can take quite a while I use the spatial understanding library which which post-process is the the raw spatial mesh you can further so it takes it can take a bit of time depending on the play space to scan the space so you're looking at like you could spend like 30 seconds to a minute to two minutes scanning if you want like a you know the full area scan and then on top of that depending on how you implement your placement solver after that it can take anywhere from like 10 seconds in my room to place the objects - I've seen some places where gets confused and spends like almost a minute now that's that's partly because I you know didn't use the most efficient way of placing things and invoking the constraint solver that I think can be easily improved but you know you're definitely looking at sucking up about a minute of time and so the in my game once you've scanned a space I make sure to retain that so if you start a level over again it asks you do you want to just reuse the previous scans you have to keep going through and doing it it is slow though yeah how long before it's

not exact real-time but near real-time where you could like walk into the bedroom and just delay you for a couple seconds or something and then kind of pop up that's a great question I think you know I don't I think it's really just a year or two away so that the error cloud stuff that I mentioned that's kind of that idea the idea is that they'll be able to kind of crowdsource scans and so when you walk into a place it's already been scanned it recognizes that and loads the room data I'm you know I'm not on one of these teams that develops the hardware but I from what I've kind of heard leaked to me it sounds to me like this problem will be alleviated a lot within the next year so you know I think not very we're not very far off so certainly within you know two years when this stuff is more mainstream I think you'll see the scanning phase be done much more rapidly cool great talk thanks for sharing thank you thank you I noticed that in some of your examples you used a gamepad for controlling me the game and in some of them it was gestures what do you think are the unique challenges as far as control

space between like the differences between those that's that's a good question you know challenges with regards to control I think what I've noticed when people first use these devices is they start reaching out at stuff so obviously having hand gestures is super super important now the problem is though I feel that once you get beyond that initial stage so you want to have that you always have to have that to be intuitive I feel it beyond that though hands are not very intuitive and in fact if you think about you know what hands are used for they're used for manipulating tools not like pointing and waving at things and I think that you know people that are very zealous about using only hand gestures are missing a key part of like you know the human anatomy and why we have hands the xbox360 controller I think is unfortunately not something Microsoft's is going to move forward with in terms of a control device but I find it to be extremely useful i I personally prefer it to like motion controller type controls I think a combination of something like that and and hand controls have been great but if you like you know you really want fine precision control when you turn like design something for example something like a controller actually works really well

and I like the fact that it's like one single things I can take one hand away and start waving around you know potentially and then you know return to my controller so I think realistically though it's gonna be a combination of just intuitive hand controls as much as possible you know picking up voice and then something like a motion controller I think it seems to be where the industry is heading Thanks thank you at a question about redressing a reed reconfiguring the entire scan so it seems like you have a lot of depth information already and for example with the military game how come you can't cover up the bed with it's a procedural texturing or you know kid bet like modeling modular modeling that could actually cover up the entire bed okay that's that's a really good question so you know he's asking about why not generate more procedural geometry to

kind of cover objects and you absolutely can two things though number one since I'm not an artist and just working on my own it's been really hard to do stuff like that but a bigger challenge on the hololens in particular is that very full rate limited so you know it's very hard to get things to run at 60 frames per second which is what you want and if you were to kind of you know generate a bunch of geometry over this facial mesh so in fact what I actually do is I actually call parts of the spatial mesh I didn't show that slide but I actually removed parts of the spatial mesh that are not absolutely necessary for occlusion I retain them for collision just to kind of lower the fill rate requirements and you absolutely could with more powerful hardware I'm looking I think we're all looking forward to better GPUs on these things and I think it's a fantastic idea one potential design issue though is that when you try to render something like really big like that that small fo that field of view limitation can make it look kind of weird like you know you look at the bed and you'll see like an area that has a geometry but then like the bed is like still the bed out here so it's I guess that's kind of an

artistic challenge but to summarize that one big issue with the hololens right now is the performance it's it's a very you know it's got a very weak GPU and but you could absolutely do that and I think that's an area of further resources that research that I think would be you know really fruitful to try have you come across anybody to try I'm not really I've seen the the closest I've seen is just people kind of placing I think pre-made maps or maybe kind of procedurally generated but kind of flat maps on top of tables things like that I haven't seen something where you have like a conformant you know scene like we have like say something conforming to all sides of like a bed for example or a coffee table I think it'd be really awesome and I would absolutely love to see it yeah thank you I guess that's that's it yeah I'll be hanging around for a little while probably outside feel free you know if you see me any time during the expo to stop and say hello if you're lucky I might have my hololens on me some of the days here and thanks a lot for attending the talk and I hope

you guys do some awesome stuff and I'd love to see more you know more AR gaming in the next year you you