The Reality of Making Hololens Games

[Music] I'm Ralph barbagallo here to talk about the reality of developing Hall lens games um how many people here have developed for Hollens at all oh okay a bunch so I think you know this is sort of like if you've developed for Hall lens this might be maybe a little basic but if you've just started developing for Hall lens or you're investigating it and this is kind of like a brain dump of what I learned developing my first Hollens game which is ether Wars and Ethan wor is kind of like a um oh nobody can hear me oh this is apparently I'm supposed to use this mic all right well I will use this mic when it doesn't fall over okay so basically this is a brain dump of what I learned uh developing ether Wars which is uh a real-time strategy game it's a mixed reality realtime strategy game it's kind of like a combination between you know Starcraft 2 and like playing with toys in your room uh

there's a little video of it here which is um basically you know it's using the hall lens's ability to know the shape of the world you're in and your position in the world so in this case you know these little ships are coming out of the wall they're attacking this base that's floating around in your room the it's not in this video but there's actually like ground units and stuff there's tanks and uh bases that you could build on the floor and stuff and it's using you know Holland's ability to to you know sort of like build a 3D mesh out of your world and play off of the surfaces in your world so it's kind of like if you've seen those weird commercials with Arnold sports Ander for mobile strike but it's really just like a regular mobile game you tap your you know your kind of hamster wheel Type Tap finger thing but in the commercials they show planes blowing up conference room tables and um you know bases and stuff all over the office and tanks driving on your floor I'm trying to make that the actual game which you could kind of do with Hollow lens uh you know it's it's it's

early but you can kind of get there so we're going to talk about how you develop this type of game uh or really any game for Hall lens your first step is of course getting started you know you need your equipment and so you know you need you need a hollow lens obviously now there's two Hollow lenses you can buy they're all the same one just happens to cost $5,000 the other one cost three but there's really no difference between them except the $5,000 one has like some firmware stuff that you probably don't need so you can get the Developer Edition and you need to get a window 10 PC you can kind of develop for it on Windows 8 but it's kind of weird CU it's a the hins itself is a self-contained Windows 10 PC so I mean I you might as well just have a Windows 10 development machine and then you need Unity 3D 5.5 beta or you can use 5.4 if you use the hollow lens technical preview so if you're stuck on 5.4 I'm still on 5.4 I use the hollow lens technical preview but if you get 5.5 beta it's all in one build and basically what this means is if you check you know V r support in your

player settings in unity it Hall lens will just appear as another SDK you can support so if you've developed applications for Oculus Rift for instance you're halfway there because they kind of treat the hollow lens as any other VR platform so you know the main camera is your head model so you know just basically works like that and of course holographic Academy I recommend everybody go through it it's Microsoft tutorials that they uh put on their website it has it covers every major functional block of hollow lens and uh some of the code in it is really good too so I've actually cut a lot of the code out of their examples and put them in my own projects and speaking of which there is hollow toolkit this is the next thing you need to look at when you're getting started and Hollow toolkit is uh well the hollow toolkit Unity version that's on GitHub is part of the greater Hollow toolkit SDK and so Hollow toolkit itself is sort of the Native SDK if you have your own game engine written C++ or whatever and you want to have it work on Hollow lens you can use the hollow toolkit native SDK and then this Hall tool get Unity

SDK contains all the prefabs and scripts and build scripts and stuff that you can use to build uh Hollands games it has a lot of common functionality you can see here input sharing spatial mapping all the basics are covered and it's it's really good it's updated constantly in fact a lot of the information I give you in this talk is probably going to be obsolete by the time I'm done because there's probably been like 15 polls by the time you know while we've been talking it's it's super active it's contributed by the open source community so Microsoft and the community have been adding a a lot lot to it so uh and I always start a project with it and then I always keep it updated i' I've never had an update break my project they've been pretty good so the first thing we'll talk about in building Hollens application is the interface because this is probably the easiest place to start you know it's probably the least sexy part of making a mixed reality application but it's it's required and Hollow lens Microsoft's whole sort of interaction model they use for hollow lens they say Gaye gesture

and voice is basically uh how you're supposed to interact with a Holland application meaning you know you look around with your with your head like in you know like if you've Ed any VR application and then when you see something going to interact with you use a gesture which is this air tap thing which we'll talk about or you use a voice command because Cortana is built into the hollow lens so that's kind of their interaction model you can see an example of it here this is uh video taken in my office of The Ether Wars dialog system now my office is a mess that's there's a reason for that cuz in AR a messy room is better than a clean room gives you a lot of stuff to track I'm going to use that as an excuse but this is a you know World space canvas floating around in my room um and you can see a little circle there is your cursor and it's pretty simple when that cursor looks at that scan button it'll highlight yellow and I select it with an air tap which is kind of out of the camera range and it selects it pretty simple stuff this is a a world space canvas which everybody's seen in unity no big deal I think I put a collision volume around it so that the cursor has

something to bounce off of but bottom line is it's just a standard curs standard World space canvas now the unique thing about this interface and Hall lens which I didn't really show in this video is that it it floats around in front of your vision if I walk around the interface will sort of follow me around and if you turn it'll sort of scoot into your vision and stuff it keeps it floating around in front of you the way that is done is use this thing called a tag along and this is in Hollow toolkit one of many you know easy to use scripts and prefabs this tagalong script or behavior that you add basically does what I what I was describing it it keeps the object sort of a certain fixed distance away from you and as you walk around it follows you around and if you turn your head the little thing will Scoot and try to get into your vision so you can see it you can apply this tagalong script to any object so you know let's say you're making a game and you I don't know you have like a little little sidekick or familiar or something that floats around near you you could use this tagong script for that but in the case of an interface you know you want the you want it to always face you so that you can

access the buttons so that's why I add that little billboard script there so the interface always faces me that's also part of the hul toolkit so once you have your little interface floating around how do you uh select stuff in your interface you have to use a cursor and once again you know it's kind of like making games with Legos these days once again there's a cursor prefab in the hollow toolkit you could just drag that into your scene there's a couple different versions of it I took one of these and I customized it for my own uses but they're pretty pretty well-rounded you just drop that into your scene and then the way you have that cursor interact with the interface whoops let me go back one is uh you use this hollow lens input module now this is you know if you add a world space or you add any canvas to a Unity project it'll add an event system object which has like the standard input module which pipes all the mouse input and stuff to your canvases so what they've made is this hollowland input module which basically routes all the cursor input through the interface so

that you know it when you're looking at a like you saw in that video the scan button gets highlighted because you're looking at it this is all done through the Hollin input module so so far to build a Hollin interface I've had to write no code it pretty much works with the existing Unity interface if you just you know follow these steps but once so now that you were able to look at and highlight the guey you have to like do stuff with it so now we're going to talk about the gesture part so this is the air tap I mean I don't know if people have used Hall lens you basically stick your hand out like this and you sort of tap your finger like no knuckle it's like a really hard gesture to do nobody knows how to do it you know when I do these demos this is a picture of one of my demos I have to spend like 15 minutes showing people how to do it because nobody really it's a really weird thing I'm sure it's the easiest gesture to be able to detect in a self-contained device but it's kind of a tricky a natural thing once you get used to it it's easy but you know for people who never use it it's a little tricky this is the main interaction method for Hollands you keep your you

know hand sort of in front of the camera and it's basically a mouse click you know the air tap itself is just kind of like a mouse click which you would think doesn't give you a lot of interaction possibilities but inside that gesture you have sort of a bunch of different things you can do so you've got the air tap which is basically a mouse click but you have the you have a couple different phases of this tap so you have the ready position your finger's like pointed up like you're about to about to click that's kind of the equivalent of like a mouse over almost like a hovering action you have the hold so you can click and you know like a mouse down event you can just hold it and then you have the drag and so you know one of the tricky Parts about teaching people to use Hall lens is they when when you immediately put it on people start like poking at the interface with their fingers and you know the thing is is that Hall lens doesn't know the absolute position of your finger in World space so you can't actually touch anything with it but it does know the sort of Delta between when you held your finger down and where you moved it so you can use that Delta as a drag or a scroll or even like sort of

like an air joystick type of thing where you can just you know do a cross in the air and control an object and it's pretty easy to do there's a thing called the gesture manager it's another Behavior you just drag into your scene and you tell the gesture manager through the through the API which gestures you're looking for and then you just assign delegates to the gestures and whenever it detects an air tap it'll call you delegate like it's that easy the final well the other input method is voice and you know Corton is built into the Hall lens it's a fully self-contained Windows 10 PC that you're wearing your face and so you can navigate the whole interface with Cortana just like you can in Windows 10 but you can also use cortana's voice recognition capabilities in your own app and so in my particular game to start the game you have to say the word play and I do that by using yet another easyto use script in the hollow toolkit which is the keyword manager and you can see here there's like an array keywords and responses I only have one of them open there's four voice commands that I'm using here but the keyword here

is play so you just type in the word you're looking for and then I just drag an event or a delegate to the response event here and so it's going to call this onplay command when uh you know it'll call that play command to start the game it's that easy if of course this is an SDK you can do this through code if you want to like for localization you want to switch to different languages and stuff you can do that but it's it's pretty simple so next we're talk about spatial mapping so like the the interface you know that's cool whatever if you've made a VR game you've pretty much probably built the same interface a million times it's really nothing unique to mixed reality but here we'll get into the capabilities of the hall lens that make it a unique platform so spal mapping is the ability for the hollow lens to sort of build a 3D mesh out of the world you're in and track your position inside it and this has a lot of implications for what kind of applications you can you can design you can see here here's an example of it

where um it'll just start making a mesh when I start triggering spatial mapping out of my office the mesh itself you know it's it's it's it's cool it's not super duper accurate but it's pretty fast how it builds it and it's uh very useful even though it might be considered a a little rough I might be cheating here a little bit because I the hall lens does cache the scan so if you're in the same room and you turn on the spatial mapping again it will kind of like load the previous scan and then progressively add to it but it's still it's pretty performant pretty fast and pretty easy to use if you want if you want to build a 3D mesh out of your environment all you do is use yet another prefab the spatial mapping prefab in Hollow toolkit and you can see here that the spatial mapping Observer is the behavior that starts building the mesh so the tri you can govern the resolution of the mesh here you can like how many triangles per cubic meter and then the size of the mesh in this case you know uh one one unit is 1 meter so this is a 10 x 10 x 10 m volume and then the special mapping manager

script is what governs sort of the properties of the actual mesh that it generates so in this case you can give it a physics layer so if you want to do Collision against it you can tell what layer to use and you can apply material to it so I'm using the wireframe material here that's why you can see all the polygon edges but in some cases you might want to not draw to the to the uh color buffer let's say but only draw to the depth buffer so you can use a different kind of material if you want to have like for instance the real world a clude a virtual object you can just draw this mesh to the depth buffer and then you you can have you know for instance a virtual object go behind your bookcase or behind your door and it will be uded by it so as an example of what you could do with a spatial mesh this is an example of me using it for Collision so you can see here and this is also an example of the tap tap and drag but it this little drone here is going to hit my ceiling and you know it's just a mesh collider boom it just stops it hits my ceiling if I didn't draw the polygon edges you know it would be a pretty convincing match for the real world geometry you know when you start using this mesh occlusion it's not really

accurate enough to really give you a convincing effect of uh you know characters walking behind stuff but we'll get there at some point you know so you know there's a lot of great possibilities of making a game that plays over your real world surface when you can build a a mesh collider out of your environment but you know that that's cool and that's it's largely cosmetic to a certain degree you know you can make some interesting games and uh applications this way but what you really want to know is you really you really want to know more about your environment and more about you know what is in the room because when you're making a game in mixed reality you know you want to know like especially people can play the game in any arbitrary space you're going to need to know where stuff is like I I need to know where there's a chair or where there's a table or different things about my environment I need to know you know where's a good empty space to put a game character or something like that and so you can do that by analyzing the mesh that's generated by the spatial mapping system but they've actually done

a lot of the hard work for you by providing this spatial understanding module so I don't know if anybody's seen um there's a game called fragments which there's like one point in the game it's a Holland game of course this character just like walks into your room and sits on your couch and starts talking to you and it's like the craziest thing I've ever seen and turns out that the developer that built that game I think there's called a Soo I don't know how to pronounce the name but they built the spatial understanding system which basically creates a new kind of mesh where which I guess generates like a spatial mesh but then turns that into a voxal volume and then turns those voxal into what are called surfes and I'm sure it's maybe like made up from marketing purposes or whatever but it's a thing and using this new kind of data structure they can analyze the space and you can query it to figure out where stuff is so to sort of make that a little bit more um to show an example of that so here's a real simple case right I'm just doing a simple Trace through the voxal volume and it's telling me Oh this is a wall I mean you could do this I did this with the spatial mesh anyway

you know you just check the normal of what comes back and you can pretty much determine you've got a wall and that's cool but you can see here it's like here's a wall here's a floor but now look at this I'm going to say hey let me find a chair so I tap the chair and this little blue thing flies down and there it goes I found my chair which like seems you know to me to the untrained eye seems kind of magical uh but if you look at how they well I mean internally it's kind of a black box I mean it's not I don't think the source code to this is actually provided oh I could be wrong I haven't looked at the native SDK too much but the way you query this system is sort of you have to kind of like create like a heris you kind of have to say hey I'm looking for something where the back is twice as tall as the bottom part and the bottom part you know faces up and that's what I'm going to call a chair and I'm going to search your uh data for that chair so you can Define objects in all kinds of different ways and then it will go try to find the closest matches to them and bring back all the results you can also

well again it's more details here so you can you can also say I'm looking for a safe space like I'm looking for like an empty space that's this big so I can fit my game character and I need it like close to a wall you can give it all these kind of um queries and it will come back and return to you data that tells you where to put stuff if there are places or where these objects are if there are matches so yeah it's it's pretty incredible it's it's it's not super duper performant you know like you these are the type of things you need to do before you play the game even with spatial m you know I basically uh scan first then play but um you know you can cach all the locations and stuff and then use them later so the next next uh thing we need to talk about is persistence so in you know a regular game you know you're just kind of like I can drop in Skyrim I can drop my sword at the XYZ coordinate you know one two3 and then I load up Skyrim next week and it's still there you know big deal right games all just use a coordinate system but in the case of the real world what is the game coordinate system of the real world you know when you turn on your Hollow lens you're at

0000 of your game World wherever you're standing or you know when you let's say when you start scanning a mesh of a new room that you never been in before you're now at the origin of your own coordinate system so you have to be able to place objects in the real world and have them stay there so what is the coordinate system of this table and then how do you tell another Hollow lens where this object is in the real world when they both have completely different coordinate systems internally well the key to that is the spatial anchor system so what they do with this is it's kind of weird it's like you could create coordinate systems inside the real world basically you add the spatial anchor Behavior to a script and what ends up happening is it anchors that game object to like a chunk of the room scan and and then the room scan itself gets refined over time so you might see those spatial anchor sort of like slightly move as you keep playing the game but basically you can attach objects to that parent object that's has a spatial anchor attached to it and now that object is anchored to this chunk of the room skin in the real world when you when you turn on the device again you can use the spatial anchor store to load

and save these things it will load up the spatial anchors and try to like find that chunk of the scan again and place that object there again so you're basically sort of uh saving a little chunk of the real world with your virtual object and then it's going to place it there again also this is how you do multiplayer experiences what they call shared Holograms so it comes in handy because for instance there's this uh JPL experience where it's like a Mars uh experience thing where you all can walk around this room and sort of explore Mars together everybody's in a room wearing a hollow lens and they're all seeing a quote unquote hologram you know in the same position every Hollow lens can see the same shared H hologram and the way they do that is there's a sharing service that comes with Hollow toolkit which is kind of like a regular game server written in C++ and what that does is it just sends those spatial anchors over a network because it's just a chunk of data so if I spawn an object at this location then I create a spatial anchor there and I can send that spatial

anchor over the network and then the game server will send it to all the attached Hall lenses and then all the other Hall lenses will be like looking through their room scan to be like oh I I know that I see that chunk of table I can match that and they put the spatial anchor there so that's how you can have a multiplayer experience where everybody shares the same hologram I mean in theory because I've done this a bunch of times and sometimes it works sometimes it doesn't I think sometimes it depends on if the room is scannable and trackable enough or whatever but like I've seen the spatial anchors up here like 2 feet off like you know from where they are other device it it doesn't work all the time but uh yeah essentially it's just sharing data over a network it's it's not a big deal finally building and deploying on the device you build on the device you used to have to build a visual studio project which you kind of still have to do behind the scenes then build inside visual studio and copy that to your Hollow lens they now have a script in the hollow toolkit that does that with one button but once you get on the device you're going to come into you know you're going to encounter the limitations of the hardware of which there are several the fov is the one

everybody talks about you know it displays an object sort of like you know it's like a postcard sized you know augmentation in front of your vision um but you know that just means you have to redesign your application a little bit so you know you might have to if you have no peripheral vision a game where somebody has to walk around and look for stuff is going to be kind of problematic so you might want to have like little floating arrows or whatever that guide the use a where to look and in fact in Hollow toolkit there are some prefabs that do just that they they're like sort of like guidance you know tag alongs they kind of point at and tell the user hey you should look over here also you're going to run into battery life CPU GPU limitations if you've developed for gear VR you're probably in a good position because it's you know it's a mobile device Microsoft recommends you know like under 100,000 polygons per scene which is a lot of polygons when you're not drawing a background so it's pretty good you the performance is pretty good battery life you know you might get about two hours of constant use uh on one charge which is also pretty good and then also there's control constraints so you know using air tap could be kind of constricting if you find that to be a

problem you can use an Xbox One S controller with it so I'll leave you with this I'll just say it's called mixed reality for a reason so when you're planning your Hollow lens content I've see a lot of games in the App Store right now that are just VR games that float in front of your face which is like you know a cool parlor trick but it doesn't really do anything that you can't do in VR or any other type of platform like that with mixed reality you know you can use the floors you can use the ceilings you can you can make persistent objects appear in reality it makes kind of a pain to debug because you know instead of just like moving your game character to a level and seeing if it crashes you actually have to walk to your some place in the real world and test it makes it a little bit more annoying to develop that way but that's that's the unique property of this platform I mixed reality I think honestly is going to be as disruptive to software as like the app was like 10 years ago we're just getting started so when you when you start develing for Hollands you really need to use those those unique abilities to really understand your environment to make a compelling experience so I like some time for a Q&A but um if you you know if

you want to contact me later you can contact me on on my website on Twitter whatever I'm pretty accessible I'm on LinkedIn you know any way you want to contact with me just you know you can figure it out but uh yeah if anybody has any questions feel free or not but you know okay there's a microphone there oh like sure so the the shared mixed Holograms shared mixed realities shared yeah shared uh yeah whatever shared Holograms so then with the hollow tool kit then it's done through a PC server is that correct yeah which is kind of annoying you can rewrite it through I haven't done it yet but you can use unet because it's just sending data they just don't have an example of it because it's weird the hollow toolkit sharing example is a complete Game Server that does everything unet does I don't it seems like they did a lot of work that they didn't need to do it has lobbies and you know message passing and stuff but yeah you can just use unet because it's just

sending data thanks sure kind of uh what were the major challenges for you and what's next based on the learnings that you've had uh well you know M debugging like I said like having to walk around my office to debug is a real pain like it it just makes the development time a lot longer like the bill test debug Loop is so much longer than VR and also you don't know if it's going to work like I have to go give a demo summer I don't know if it's going to work in their office I've been like my game needs to use the ceiling and I've gone to have meetings and I'm like oh my God there's no ceiling in this room like it's too far up so you really have to test in a lot of different physical environments so it's it's almost not a programming challenge it's more like a practical challenge of how to debug something that can be played in any arbitrary space basically I would say I would say that's the biggest challenge hi uh back to Shared experiences um would you say uh you showed an example of more the educational aspect um is it a little too

early since the anchors are way off to go about with uh doing a game or would you would you recommend waiting until maybe I mean I I I never recommend waiting for any reason because I'm always I always like to do something first so sure go ahead I mean I'm planning it with my game I just haven't gotten to that point yet I I'm I'm going to design the game around the anchors though because originally this game I was working on of for Project Tango and that one just has like a single coordinate space for all the devices when they're all sync up and you can just kind of create objects and they theoretically would appear where they're supposed to this one is a little different because you have to have them with these spatial anchor so you kind of have to be a little different but what you can do is you could do things like I'm going to shoot this I'm going to launch this ship from here to here there I'm going to spawn it at the spatial anchor but then release it from the spatial anchor and it just goes through normal coordinate space and then attach it to the spatial anchor when it arrives you just kind of have to like play with a little bit now whether the spatial anchors are accurate enough as a whole different issue but that's just um you know that's just going to be a factor of testing and making sure you're playing the game in the right space at some point with mixed reality you're going to have to tell the player look you need to

turn on some lights or whatever like they're just they're not going to have a trackable space so this is regarding the shared user space do you get like registered uh player position for all the players who are there like let's Suppose there are 10 people who are on the Mars so they actually know where they are located yeah they they in the example I think it's yeah it's like I think it's the final example in Hollow Academy they show that where they actually have like on the other players like they put like a little Avatar head over the other Hall lenses so I think that um Bas I I I it's been a while since I looked at it but I think that you know whenever um a player I think joins the session they create a spatial anchor of themselves and then they kind of update their position through the server and it's like kind of accurate it works a lot better than I thought it would work so it kind of works Ralph uh talking about waiting we are I'm I'm on the investor side of things uh we have a fund to specialize on on uh augmented and and virtual

reality one of the challenges that we're finding is to convince people LPS or people that are willing to put money on this kind of uh future technologies that that we're not like 10 or 15 years away from this they are very uh not skeptical because they are excited about what is going on but they think that these kind of things are like in 10 years I know that hland are is is not is not very public about timing and Tango it's uh in my opinion has been slower than other um that should but what are your thoughts in one monetization opportunities for these kind of opportunities specifically around games but in all Industries and and number two any thoughts or recommendations about uh you know educate these uh people well I mean I just don't try to waste time educating investors because it's pointless I mean like I agree I mean honestly do you want to spend your life having meetings with

these people I don't so I'd rather hang out with developers who are smart who know how to do things and we can make stuff and I'll leave the going to lunch with you know the investors to some it's somebody else's job so yeah I don't I don't have any answers to that question because I think it's it's impossible uh the it's actually might be 10 years what whatever this is might be 10 years away I mean as far as like a consumer device maybe but right now two-thirds of my business is like Enterprise so you know I've had a lot of interest like Warehouse leg Logistics construction you know building information modeling visualization it's we're but we're still I mean I kind of feel we're weird telling people that yeah I can deploy this now because there's still a lot of variables with the device I think right now you can like build the application but you might not be able to launch it in a real battle tested Enterprise environment at this moment uh but you might be able to do that next year so like Enterprise maybe three years away or something consumer I don't know I do think that at

some point like every device on the planet is going to have some sort of like dual camera slam solution in it and people will be like oh of course it's obvious but when that point is I don't really know and uh I I just think that um you know but but what what has happened to me is like I've had clients like real giant clients come up to me you know huge shipping companies like we want to use this technology now because we have an actual problem to solve and we think you could do it with this Tech and we're willing to suffer through all of the you know beta problems of small fov and stuff like that to get a job done so I think that's where the money is now I mean it's not a huge Market but there are like a bunch of big clients that really need problems to be solved and they can only solve them with this Tech so I guess that's yeah that's that's probably what I would say about that sure all right well I'll be around so you know see later [Applause]