Evolving Mixed Reality: Designing on the Real World

[Music] hello thank you for coming to our talk today on evolving mixed reality i'm jono i'm samantha gorman i'm ron gull uh brian schwab uh so before we really launch into the talk uh just want to kind of frame the space that we're talking about a little bit um so right now you know a lot of ar that we see looks like this a lot of it is sort of single plane user directed um you know tabletop sixth off ar and people have done really really cool incredible inspiring exciting work there uh but i think that also everybody in this room probably has this sense of like this is the first step towards this uh like true mixed reality thing so to draw a couple examples what we mean when we say that um you know that can mean everything from immersive games xr xr uh everything from immersive games to

industrial training to uh iot controls i'll throw more examples out of the kind of stuff you're thinking about i mean the thing is is it's it's where we're going is it's it's just compute anywhere pixels anywhere with whatever device you use that's where we want to hit we're a long way from there absolutely uh so let's meet each of our speakers ron hi everybody so my name is ron gal and i work at the interactive media group at microsoft research just to make sure everybody understands i'm not part of hololens so don't take anything i say as something that is going to happen with hololens we are working with them from like in microsoft research but i know as much about them as i know about magiclip so we contribute ideas we have no idea what the what they are doing with them so in our group we are working on several things that

are relevant to this presentation and also on other things we started to focus on ar and vr and xr sorry about five to six years ago and we did a couple of projects in our group we developed devices we researched ideas of how to design to these environments and we have a lot of fun doing it and it's pretty cool but that's it um hello so i'm samantha gorman and i'm the co-founder of tender claws and we're about a 10-person studio and um we're often we're artist directors designers that are often commissioned to think creatively around emerging tech often at the same time as that tech is actually being developed so this is i'm going to talk a little bit about tendar which was our collaboration with ar core right when ar core is being developed

um and as a case study i was interested in going kind of away from tabletop not because i don't think it's a really great area but i want to do something more room scale with ar and it's tendar is essentially a long-form ar content app it's about three weeks of content and it reacts meaningfully to the user's environment so it has modes that can address an a user at home a user in the outside world multi-person and social use and at the core of tendar's design is a strategic use of our device of on device rather um mobile vision such as object recognition sentiment analysis to create a virtual friend and pet that can evolve over the three weeks by feasting on the player's emotions the tears and joy of you and your loved ones and responding to objects in the player's world so tendar is actually we're going to

talk a little bit about machine learning later but it's a fictitious company that ostensibly created this virtual pet guppy to improve its model for emotion recognition and by inviting users to build a relationship with this guppy it teaches as you play the game a little bit about what machine learning is and how some of the object recognition in the game works itself guppy in itself is a computer vision model that's inspired by various neurobiology research that attempts to emulate and automate how real vision systems operate and identify patterns it's kind of both a literal and figurative representation of machine learning since as you are playing the game guppy must eat and absorb the image the data sets the images of your emotions to survive and by eating those emotions he's being trained on how to process and understand human emotions more accurately and you know hilarity ensues as things kind of

devolve over time here's a brief teaser so cute and pretend there's like you know fun instrumental music in the background okay so some of the things you're seeing in the video um there's various modes using both the front and back facing

camera of obtaining emotion from a multiple not just you as yourself but multiple people in the space you can pull off the emotions from their face they turn into kind of these wonderful digestible blobs that guppy can eat and sample and then give you a fortune of what you're really feeling um so there's a the sentiment analysis aspect you also see some of the object recognition which is a context dependent on the various objects that can be in any of these spaces that the user will inhabit and there's hints of the game like how to find them um some of the things that it also does is it thinks about well i guess to step back for a second and we can talk more about this but the challenge of like why we did this project kind of early on into the ar space was that our main interest in xr in general is that we see it as a rich space to prototype interfaces and models that kind of hint at or lead to a future of spatial

computing that is sort of the forerunner of like what it will actually be in that environment so we value interaction design as a key key element for making these spatial a digital content that feels like truly present okay and next and there's a actually a module that's not just world-facing but that's self-facing where this is um through you can decide to make the food you're feeding your fish by actually engaging with the fish and teaching it and you know emotions and it's kind of this like emotional dialogue you're having with the pet that then like results into this very um i guess kind of like a little bit of upsetting you know distillation of your emotions into food pellets that you feed and that's how you know the fish gets his image that data and then you get this wonderful little like fortune from tendar that analyzes like yes this is accurate this is how

you feel um you know you can choose what to what to do about that so one of the things that i think this panel in particular is relevant is that tendar has to work with all kinds of player context and aspects of change to their world um how do we design for a space where the context the content should be anchored to the user's world and have meaningful and have meaningful semantics with users worlds without feeling overly tethered to a surface it can happen on a bus on a grocery store and a toilet you know you name it act the other sort of unique thing about tendar is it's thinking about ar for social play so there is a module that we often show at conferences and galleries that is for two players and one of the things that is happening there with the front facing camera is the the emotions

of the two players are actually playing off each other and the audio is splitting to different ears so the different players are getting kind of um live procedural updates on what the other player might be feeling which then like changes how they're engaging with each other on screen but it also changes how two users can engage physically in the space with a device and what that user intent means for like moving an ar device around that space and how it engages changes the design of the experience or the application all right um hey guys uh my name is brian uh i'm working as a director on a thing called the interaction lab and so if you go to the next thing it's basically a rapid prototyping group uh over the last four and a half years we've probably made six or 700 prototypes you can just pop through them i don't care we did a bunch of things we did first turn on of all of the hardware and software and perception features as they came online we tried to give really fast

feedback and guidance to those features as they got built we tried to like think about how you would consume those features as a dev so we could like actually iterate a little bit on the sdk before it got there we did a bunch of documentation and you know best known practices and then we finally did a lot of knowledge sharing and then slowly over the as we walked as we walked up to launch we kind of like pushed a little bit further in that direction and the team about for about the last year or so made a bunch of mr um little moments called magic kit that was uh a bunch of stuff that we put out there specifically just to give some tiny little taste of spatialized computing and then i like all the source code in in unreal and unity so that people could actually pull the pull the dials apart and see what made it happen and why and when it fell apart and which perceptual cliffs were important and which one weren't and why everything was doing it so the source the source is all up there on on our website hey i'm jono uh i'm on the labs team at unity uh i'm leading ux dev for uh mars

mix and augmented reality studio and that's a upcoming set of tools for building xr experiences mrar experiences um with a more visual approachable workflow so for example we do simulation we'll talk kind of length about simulation in a sec but this is an example of how this example game content would fit into this simulated living room we also have these uh visual tools for specifying the kinds of parameters of objects that you're seeking so here i'm saying like i want my hero character to be on it's going to be on a surface the surface is going to be you know in some range of size and some range of elevation you can like see that here abstractly and then also simulated see how it actually matches um of course face content right is a big part of commercial ar right now so we have workflows for that and you can see here that i'm simulating live against the webcam notice that we're not in play mode this is at edit time um i think this also kind of shows you the

shape of things to come in terms of uh augmenting individual objects so i can say like i have this canonical version of an object that i can mark up and then you know see how that fits against um you know real fed examples uh here's that first scene that you saw running running on device um you see a little sample of our some of our procedural tools and utilities here uh this this has a rule in it that says um see any horizontal surface that's not the floor should fill with this terrain texture the strain material thing and then also those and the floor get nav matched our character can walk around and then also all the raised surfaces should build a ladder down to the floor um so you can see all those elements and one of our goals with this project is to be able to open up xr experiences and extra development to people who are not programmers necessarily uh so to that end everything you see here has no additional scripting like this is all just kind of out of the box tooling and then also we're looking at tools on

device this is a design mock-up of using a phone to capture a space and then to start to identify layout and rules and yeah content descriptions that you want right there you know in the real in the real space so ultimately you would pull this into the editor and use use this room as a simulation that you can test against and also use um the rules i'm starting to set up here as the beginning of your content scene so i'm saying like if i find you know a surface that looks kind of like this one then put my put my hero on it and that would generate that rule that you saw earlier okay so that's who we are and kind of where we're where we're all coming from so hopefully now we you know spent the setup to give you an idea of the kinds of things that we're interested in in the xr space and where where we're going with all this so let's talk about why ar is complicated what are some of the many challenges of it want to start us off um and in terms of us we are trying to

think about how to compose an ar experience that could be transferable to different environments of the user and still have semantic significance for those environments um and part of one of the things that we were thinking about is for instance all that the objects that guppy can recognize that he can respond to and that the user is kind of there's easter eggs and it's seated throughout the game what these objects are but users can go out and find these objects in their world um and they started with like a thousand we can talk about this later and then worked our model down to about 200 of these um and trying to figure out um how like if like a fire hydrant in like one context may not look like the fire hydrant in another context so how do you design um the the ar experience to be able to like recognize those and like the interface that comes off of it and then like the visual design will be different depending on the different shape for what's on the screen and what the user

encounters um so thinking really trying to think broader than um just like mapping a space to almost mapping a world so actually i had a question i would ask you about that one like since since you recognize all these objects that guppy can respond to like how do you um you know how do you keep people from missing so much of the content that is there or that is possible yeah um so the actual content is actually um it's like writing worked with eight writers over five months and it's like a whole giant corpus of text messages that is just partially generative and partially written that can come up in response to things the user does with guppy in the ar world um so there's like weeks of content but um parts of those peppered in within those messages is kind of hints about he's learning about the human world you're training him to like see and find objects of like things he may want to may want to see um

so you're like oh you know i i heard about this this type of device that you know does x y and z so that's a more explicit example but that can get players to like go out and bring their ar you know application into other spaces in the world um if it's something that you can find at a grocery store for instance or um you know a like a car wash um there's also a module where guppy sends you on excursions explicitly to like for instance the grocery store to talk about what he wants to see in the fish aisle um so you know there's elements that can work with that i i i actually think that one of the biggest challenges of ar is not just that the experiences themselves are tough to make it's that the player expectation is essentially nil like they don't know what to think or do or if they can even move or if physics works or whatever and so like one of the challenges of making experiences for for xr is to quickly and sort of transparently educate your users on what the experience is even

going to do and what they can expect if i'm sitting in front of a screen i know where all the pixels are going to come from i know where the controls are and and now that's all gone and so like like people don't even know where to look unless you explicitly kind of make that contract with them very quickly and they don't know what to do if you don't make that explicit contract with them fairly quickly and that's one of the biggest challenges for now that's sort of a moment in time challenge right yeah and in my case the we we started to work on this area about six years ago when devices with an array of sensors starting to appear and we faced more you know we were in uncharted territory and i'm coming from the background of geometry processing and then we found ourselves with a lot of data streaming fused to some extent into our application and okay what do i need to do in order to create a simple demo and then we start to find

out that design tools are really missing so you get a lot of data you get a lot of intelligent analysis of what's going on in the scene around you but as a programmer not that i'm an xr developer but as a simple developer with tools like unity on unreal i'm able to create stuff but what stuff should i create and where should i put stuff in in order to adapt to the environment around me and this is what led us to start thinking about design tools that will help developers or will allow developers to work in this environment without going through all the low-level geometry analysis and whatever well so to that to that end you know of course one of the maybe the biggest challenge one of the biggest challenges is just the unknown nature of the real world right we don't know where the user is going to be samantha's been talking about that of you know are they on a bus are they in a

field are they in the grocery store um and how do you how do you deal with that um so ron you wanna just kind of keep keep running with the rules there was something i was going to say about the actually from the dev perspective the most challenging thing and why it's important for the tools that these guys are building is getting everything to play nice together especially we're trying to enhance the future of like xr by adding all these other like world sensing modules just that the combination of getting those things to play nice is difficult and that's why what it's some of the what these guys do is so important absolutely yeah we're going to talk about the challenging matrix of devices and capabilities and all that craziness uh so so this you know the the nature of the thing that we're trying to get at right in a lot of ways you could think of it like css on the web where we we talk about like i want i want to specify some parameters and some rules that will let uh you know in the css case if my browser window is really small or if it's really large my content will you know intelligently adapt uh in

this case we say you know if the user's in that room or that room or the bus or whatever that their content adapts um so to that end uh you wanna take this one so continuing on what i said before we started in a more of an a engineering of engineers trying to find a solution to a cone problem we wanted to create amazing magical vr and ar demos and we had the mesh detected by some kind of depth camera and then we asked ourselves what do we need more so we started to throw balls in order to create magical physical simulations and then we found out about holes in the measures and then we thought about how to fix this problem and how to allow a more plausible kind of experiences and then we started to think about okay suppose i want to create an application

where should i put stuff where should i hang my synthetic objects in the scene i we started to think about consistency adapting to different environments adapting to different environments from the first frame or asking the user to scan more data in order to to adapt even better and the virtual objects needs to sit in nice locations nice regarding color regarding contrast i don't want to see especially with additive devices i don't want to put stuff on a window facing the sun outside all kind of rules that when you think about them seems simple but we didn't want to go through this pipeline every new demo that we want to do because we said to ourselves this is a common set this is the basic point i need to start i need to have tools to start developing immediately and of course we took into account a

multiple users in the same environment that i want to put even the simplest thing i want to put something in the room that is visible or is viewed by two or more users then it becomes a simple point that you can fix or you can solve theoretically very easily if you have all the data and this is what we wanted to save the developer don't take all the data we will try to help you in this regard so and of course multiple a application in the same area i want to open a new application i want the previous applic open application to move gracefully and allow me or allow application not to fight over real estate of my screen and over my pixels so so we developed a simple service called flare this is before xr so it's fast layout for augmented reality and we used declarative rules which are very simple you just state

the rules that you want your experience to follow such as i want these tv screens to be hanged on a wall and this brings us to a another point what is a wall and how much semantic information do i have about the scene so we started with put it on any vertical a flat surface we don't care if it's a wall or something else but the nice thing about rule based system is that the more information you have the more intelligence your rule can become and then we started to figure out ways to solve such a rule system because this is a a kind of optimization problem but a very non-convex optimization problem and we started to look for iterative methods and iterative on purpose in order to be able to invest as much time as we

have we wanted a solution we wanted a good solution but if we don't have 30 seconds to invest in it we wanted a plausible or as good as possible solution in two seconds and [Music] we started to we we used a couple of methods of solving it and we achieved a very surprisingly good results on very low end devices and this is what motivate us to continue to look into the matter so this is a simple example and and i wish i could play it yeah yep okay so this is an example of building a racetrack in the room and the rules that were used to construct the racetrack is put a number of points in the room at a certain distance range of distance between them and that are visible each point can view

the next point view in the sense of there is nothing blocking the the path and we construct the racetrack using the spline between these points and when we increase the number of points we see that we adapt to the environment the the racetrack is created larger and larger in the different and of course at some point you might not be able to adhere to all the rules but if your rules are valid in your environment you'll get a solution which is surprisingly nice we were surprised at least i personally was surprised [Music] and these are this is the same example and the nice thing is that it's the same set of rules to design all the to to construct all these examples and we found out that rules are very easy

to teach people but it's not as clear as the problem is that you always as a human needs to decide okay i know the rules but which one has more priorities and which one i want to follow no matter what and which one i can allow to to degrade from or not fulfill fully so it is a kind of art but at least the basic idea is very simple to explain and we actually had developers work with it and it needs getting used to because you you find out the difference between in the user frost room and visible to the user because you have stuff that can block your view and but but it's very easy to integrate multiple users multiple applications additional semantic data that will become available over time object recognition and whatever so and and this is where the connection

with unity happened because unity apparently is going more or less in the same direction and then we yeah we came across the flare paper um as we're working on on our project and uh it was very exciting to us because we're like yeah like this this is exactly exactly the kind of approach that we're that we're taking and um you know just seeing the results that you that you had there was was very uh encouraging on that route um so yeah let's see do you want to talk about this one uh this is another example of um graceful adaptation so we we constructed a set of rules to put the creature in circle facing a middle creature we didn't define the radius of the circle and when the scene change or or obstacles appear such as in animated or human goals and disturb the the the previous

arrangement we simply added another rule to resolve but be as close as possible to the previous solution so everything is very it's like lego everything fit together the only question is how much time do you have to invent to to invest in order to find a new solution and it's highly the minute you have a strong gpu around it's a matter of milliseconds on low-end device it becomes an issue so yeah see yeah i mean like like i was saying we're really excited to find flair i'd recommend everybody read it by the way just google flair microsoft research paper you'll get it uh really good read um when we read it we said i guess this is exactly you know the kind of the kind of thing that we're talking about um so we we have very analogous uh systems to everything that ron just described uh we talk about real world objects which is a a scene object in your hierarchy that represents a real thing so i could be like this this game object in my

scene represents a table in the real world um the complexity of course is that table may or may not actually end up existing so that's a whole thing real world objects are made out of conditions so you can see in this object i'm defining so i have an object in the inspector there called floor i'm defining it with a tag condition looking for the tag floor that's provided to us by magic leap and hololens and then on the other platforms that we support that don't give us that for free we we do some extra logic to fill that in so that we can just say every platform knows what a floor is um that's actually enough for the floor in this case i've added some extra conditions just to illustrate the case a little bit more i could also throw something on there saying i'm looking for something that's horizontal i'm not looking for something that's vertical or something that's off axis i'm also i also have a condition there for i want a surface that's of a particular range of size that's acceptable so we use those conditions kind of stack them up to describe that real object in this sort of atomic way and then as ron described right we then

have this this rules concept that sort of takes all that at a you know at a higher level we say like okay when there is a floor stick water on it when there is a raised surface thick grass on it et cetera right you can see those rules kind of in the lower left of the screenshot there and see they're all matching in this simulated kitchen environment here and then yeah ron was also just describing degradation right so so we we call those fallbacks and we say um basically you can describe the sort of ideal case where you're like oh i have this app that like you need a two-story building and six different walls and three humans and have some crazy set of expectations but then say if that fails then here are some more you know some simpler simpler simpler simpler cases down to the point of like oh i don't even have tracking now what right now you're in a threedoc experience or you're in a 2d experience at that point um brian you mentioned at one point the basketball thing do you wanna talk about that yeah so i was gonna just mention um this isn't super new like like we have had very complicated game ais in the past with

that that use very similar systems uh i worked on um a basketball game for sony computer in the early 2000s and we had almost uh five megabytes of of ai data that was it was it was just pure html of of situations everything from oh if if you've got the ball in front of you and and and nobody in front of you take a shot all the way till in the last three seconds of the game if you're this particular player and the guy in front of you has a broken leg to do this thing like like all the way down to that level of specificity and so these systems have been around for a while i think that they're not typically needed for most for most games it's more games that have uh that had a high degree of sort of knowledge base which the real world represents the largest knowledge base that we need understanding that that there is and so that's why they're starting to make kind of this resurgence uh the other thing that's nice about those rule-based systems is that they're human readable you can look at them as a human and you

can say i i understand that in the last three seconds if you're this guy and there's this blah blah you can actually understand that very easily and so the reason why we had these huge ai systems on that particular game was that by being human readable we could have a small army of hardcore basketball game designers who could like crank crank stuff out um year after year and add to that massive database of ai scenarios that that it could respond to and so it was it was sort of a beautiful a beautiful system so that so that you know non-technical staff it was much more accessible to them so on the point of ai making a big comeback here um we promised we'd get into machine learning a little bit do you want to take us away there um yeah i think when we were uh talking about this panel before we had some interesting discussions and an interplay about like the use of machine learning um in our case i think it's actually there's it's such

a wide topic but um ways to sense the player's world and ways to use existing machine learning mod models and modules to incorporate that into an ar experience as a dev as ways to uh sense the player's world and give you more data i think is a really strong push towards the future of how we're we could engage with ar and for us in particular no tools existed to really do that well at the time so it was learning a lot about how to operate within different systems and bring them into unity and for we actually use parts of a tensorflow and as part of the object recognition and tensorflow was not at that point uh compatible with unity so that we had to kind of build a wrapper around that and um

then essentially take the model one of the models we were using in tensorflow was some one that was created at stanford and it can recognize up to a thousand like a thousand plus objects but the image set it was trained on was mostly dog breeds um so this is where kind of like the design thinking of being a dev working with like machine you know well machine learning models comes into place is that we knew that we wanted the fish to um have an easter egg where he could maybe identify all to take advantage of all the specific dog breeds so we can very sensitively know the difference between like a german shepherd and schnauzer but like very intense and uh in intensely because we were we took that model we binned it into objects that were recognizable and objects that could make creative design decisions for what would be in the player's world and there were some very obvious things they couldn't recognize so very early on we had to make the decision that okay we're going to just spin x number of these objects into chickens so all types

of you know like animals that walk in this particular way have been chickens and and narratively the fish is going to have a fixation with chickens and therefore that's why in this machine learning model we're going to compensate that with narrative by like just you know talking about his chicken obsession um yeah design designing around modules is like one way to you know i think incorporate that into like the development of a game the the thing is is that anytime you have a massive soup of data and you're looking for temporal patterns machine learning is an obvious choice and and because this is a new field a lot of what we're talking about is machine learning but don't don't be put off a lot of what we're also talking about is stuff that essentially needs to be built at a very low level so that most people can just use it as a function like where's the nearest table it should be a function you can call as opposed to let's quickly ram through all of the geometry and run a learning blah blah on it and so that we can find the flat surfaces and we'll declare certain types of flat surfaces to be tables that's the thing that the low level

should do and a lot of what has evolved over the past four years is we've gone from you know almost no semantic understanding of the of a geometric soup to to we're getting more and more and more through machine learning and other analytic methods we're getting more and more semantic data that we can start to provide to to to developers yeah we're debating um know when we were talking about this we were debating should we should we tell everybody in the room go learn machine learning um and you know i think brian just just touched on that a lot where like in a lot of cases that's really on on us right it's on the platform holders here to provide that uh and that you know i mentioned earlier that one of our goals on my team is to uh make this approachable to people who aren't programmers uh so we like like brian was saying like you know we want to be able to just provide a model that's like this one recognizes tables and chairs and household objects and this one recognizes types of dogs and you can just you know plug that in and use it that said

yeah i think we would all encourage everybody to at least like get the shape of machine learning just like understand like what is it like what are people talking about and um you know what are the what are the general techniques used there so even if you're not you know a hardcore programmer even if you're not gonna do you know make your own model um it is still just a very interesting thing to know about and it will help a lot in this extra yeah from a dev perspective i think that's true i think understanding a little bit about it helps underpin certain design decisions that you can make in terms of knowing what's available um to you and what types of things you can do 100 who wants to talk about the user so the user is just as varied as the environments uh uh you know even if you talk about just the quote-unquote human hand being this natural input it's like maybe like what if you have rheumatoid arthritis what if you really really favor one hand over the other even if you are left-handed you like using your right hand what if you just don't culturally think that this means yes you

know what i mean like like there's a number of things that that have absolutely no bearing and so user input is in many ways almost the same problem as world understanding is is user understanding and so a lot of what we end up doing on this side is is the exact same problem with a different user with a different data set like in exactly the same way that head pose is a fused input with a number of different things hardware software and and human understanding right that gives you head pose knowing where the hand is going is very much a hardware software and a human understanding problem you no longer just have a joystick that you're reading the value of an of an analog float coming out of it and so um this is also a very tough problem and it and again is requiring the platforms to to up their game as far as delivering uh solutions to to this kind of understanding for for the average step the sake of time we're just gonna plow through uh procedural content learn

up on it same same idea as uh machine learning like it'll it'll do you well to to check in on like generating meshes um but like i saw earlier like we're gonna be providing some utilities for that and you know there will be more utilities but it's it's important does anybody want to touch on that before we roll along so let's talk a bit about uh some specific tools and workflows that that uh you know that are there or that are coming online for one thing simulation i mentioned earlier this is the simulation view inside of mars and here i'm just going to kind of click through a couple different spaces like how would this content work in this room versus that room versus that room versus anywhere else and here i'm swapping into a secondary simulation view where i can actually simulate as a device so here you can see that if you look in the left panel you kind of see all the rays and the the planes the surfaces being generated by that device so this is a very coarse version of like generally this is how a phone or a headset would would see the world

um so in this way you can you can see like as i move through the world how would i expect my content to adapt and you can you can pick up a lot of issues there and another tool that i that i want to point out is um debugging in the editor uh so here i see in my simulation that one of these trees didn't show up uh so i'm going to this compare mode where i'm like why did it not show up on this surface that i'm expecting and sorry those the text is a little tiny there but what's going on is uh in the inspector in those conditions that i was describing earlier we're showing like this one is failing this one like the surface is too small so i say okay you know adjust for that condition include that also and now you see it pop in there so i think that the idea of being able to ask your content like hey why aren't you matching against this this data that i expect you to work on is very complicated in this in this context and very useful to have some some tooling around oh so here this uh there's a one of the things was you can just fly around a

little rc car and so i'm just flying it around real quick then i push pause bam and now what i can do is i can scrub backwards through all of that data right there in place standing in the same room i can scrub backwards i can scrub forwards i can realize there's way too much crap so i can like go over to the panel i can turn off a particular signal i can then go back and forth and watch what the events are happening when they're happening and then right there in the place i can go oh this thing isn't firing because this particular collider is turned off because there's a ray of light that's coming into the room right here that's that's causing something um this kind of like in situ debugging is is almost crazy important for for bugs that only happen in a particular room with particular lighting and a particular moment basically and being able to just sort of pause right there so we've made this module this recording module that you can just sort of drop into anything and like you've got a little recording window that you can kind of go backwards and forwards the awesome thing is hit

pause there you can also like record what would what would be seen through the device off to the side so that you can kind of like look at where you are right now and also put your finger over in into the into the stream and see the video from the the device i view from that part of the stream so you can compare two streams right then and there and then lastly this is all in world data but you can bake it out and just call it relative and then just take the data set back to your desk and try and debug some things like from your desk as well right and so it makes a nice little data set instead of a fraps video in your bug report you can just re include an entire huge data set and say the bug happens at this time and and people back at their desk can try and debug it by by importing that data set directly uh so let's talk about the ecosystem a little bit um and one one aspect of this um yeah so so brian and i and our organizations

obviously have been talking for a long time about um you know i think we we have very similar and inverted goals with each other where we are trying to build tools that will let developers make experiences that will work on any platform right that's kind of our our whole thing right um and i don't want to speak too much magically but generally the inverse yeah we're trying to allow creators to use any engine they want to to make that content and so in in the end like like we have a particular set of hardware functionality uh uh and what unity is trying to do is trying to basically make a set of a set of authoring tools that that are functionality agnostic right so you you not only have to abstract away the hardware you have to abstract away the features themselves like like so that another human being might be an additional feature to an experience which would lead you to a multi-user right it's not just about

hardware and so this sort of abstraction layer is not only going to allow different devices to run but different devices with different functionality in different environments that afford different experiences all have to be abstracted out so that so that not only can the the editor tell us what it needs to tell us but that the devices can respond to the things that make sense i know you've had uh yeah from a dev perspective um we worked with uh this project on air core and long you know it's a it's out on android but um you obviously want your game to be across as many platforms as possible which is one of the reasons that you can use um tools for unity and create with unity um and from uh you can say the other guys too it's okay you know well we do you know we do work uh but uh it's really actually the creating the ar across multiple platforms is not so hard like there's no barrier really to us bringing this to ios it's more that if you are doing you're adding on these

other modules of sensing each platform can have its own kind of um i guess like ways of doing world sensing that don't necessarily come across so if you're trying to advance ar you know in that way then that that is more the sticking point rather than like bringing the ar content across platforms at least in our experience well it's so interesting seeing seeing your app that does all these things that we haven't seen uh many things we haven't seen in that before and many things we haven't seen simultaneously in the same app before and i think that seems like it speaks to the the fact that you really dove deep into into a platform versus trying to make something that is right yeah and that's one of the advantages of diving deep into the platform is that you can push innovation in certain ways um but then you are tied to the vocabulary of working within that platform i just wanted to show this little table that we have on on my team uh the y-axis there is

platforms whether that's hardware devices or software platforms and the x-axis is functionality so it kind of gives you a sense of what what it's like trying to support all these things and we want all this to work nicely together um so i just thought it was fun to see we were jokingly saying that in the 90s we saw this with 3d cards and in the 80s we saw this with sound blaster equivalent cards so this is not new either yeah not a new problem but an interesting space for sure is there anything i also want to touch on on uh ecosystem i mean i think that what what you should be the big the biggest part of what you should be taking away from this talk is that the evolution proved just how much data there was and how much we need tools to support the use of that data i think in exactly the same way that like in the early 90s let's say when shaders became a thing suddenly we realized well not just shaders but like bump maps and light maps and normal maps and like like suddenly the tools to make good art became

tenfold more important than it was when you could just make your textures and apply them to your you know 200 poly model and so uh likewise now in order to to sort of get the reality flavoring behind all of the things like not just the art but the in the interactions and the ui and the placement and all of the things you have all of this extra data and so we're incorporating this explosion of of tools in the ecosystem to try and get our heads wrapped around that and to get people to be back to the productivity level that they've had for a long time now because we actually have pretty shockingly good tools nowadays to make screen based entertainment uh or or just regular applications but those tools are largely you know the the microsoft paint of going forward is in that they are sufficient but they're somewhat basic for for the level of data that we now need to push forward so let's also touch on uh you know we're

talking all xr here right um iran i know you had some interesting other spaces yeah so in our case we took the same rule-based system that we developed and and knew how to solve fairly quickly and try to see and and this is something that are really really hopeful will happen when unity will release their a rule system because rules are amazing it's stuff that you can't even imagine yet so we applied it we said to ourselves okay so layout of synthetic objects in the real world is one thing but what if we want to layer to create a layout of text over an image then we have a set of rules that are different here we have color and contrast and and font size and whatever but the idea is the same so here we took the metadata of the image and and created a couple of examples that again they are not perfect and they are not ready to be productized but they

show you the power of rules as a way of thinking about problems the way you have a solver even if it's not a good solver it gives you a starting point that will allow you to see things differently that's what we are hoping for and we of course we can't uh do stuff without machine learning so we are using the same set of rules to arrange a a for furniture arrangement and creation of environments because right now we are testing machine learning models on completely synthetic render in order to achieve high accuracy object detection and these are a couple of examples that we are working on right now the some of the rules are created by a set of rules not feng shui but similar walkable areas facing tv

in such basic rules and the objects themselves in the scenes are placed using a set of rules to place them in a natural as much as natural conditions as possible and we are checking to see how can photorealistic rendering right now or or physical based rendering can help in easily create models such a pluggable models such as recognize a set of kitchen tools recognize that in the future to the ecosystem oh yeah i also found it really interesting the uh upper right example there where right using the rules to then generate a layout of furniture right um i found that example really compelling in particular because like we really want that functionality where we could just generate a billion rooms to go test your content against and say oh your content works in like 80 of rooms according to our generator um but then brian i know you uh had some ideas about that i just yeah the thing i would watch out for is that

watch that your room generator isn't too simple because like it can make rooms that are nice and clean and then your algorithms may only work in nice clean rooms or it might make things that are you know who would ever put a television in the middle of the room and it turns out that in some cultures that's where they always put the television you just always have to watch out for implied biases in your in your generators at that point and if you have algorithms that are learning off of those generators you have to you have to watch the the cart isn't pulling the horse and that sort of thing and so um you know we we we've used this a lot though like like like he said uh synthetic data sets end up being the thing that gets you a large corpus of your training and then you and then you try and layer it with with with real data sets as you can collect them but uh um this is very well known in the magi in that what's it called machine learning you know community there's there's formulas for how much synthetic data is too much and that kind of thing already so so again uh um this this sort of stuff is

is it's just hard problems and there's just so many of them you know so it's good stuff though uh one one application of this stuff that is near and dear for me is um applying this sort of rule-based thinking to strictly vr applications um you know right now in room scale vr uh if i imagine probably many people in this room have probably done room scale vr stuff and kind of arguably the best you can do right now is sort of like fit a pre-built uh virtual room into the size that your guardian will allow like oh it turns out you know you're in a big empty room we can stick a really big virtual space in there uh or you know you don't have that much space so we're going to stick a smaller version of that in even that is pretty relatively rare uh so i'm really looking forward to being able to apply this sort of rule-based stuff to you know the system can take your your guardian boundary shape

and on you know more advanced systems than that it can take the actual like mesh of the world for headsets that have cameras on them right um and and actually you know give you a full-blown vr experience that also actually matches to your to your real space which is just another flavor of the same problem of course and there is so much that we could not jam into this talk um yeah when you think about think about like like what i said earlier was you know you have to quickly determine what the rules of engagement are for this particular experience well imagine if there's four experiences all running in the same room at once like like how do you how do you tell people that or or do you limit it or do you not or whatever like the the the thing i would say is that this is a very wide open space xr is huge and it's and it's very infantile and like i would say that if if this sort of stuff interests you there's a huge amount of companies that would love to you know if you love building tools and you love solving problems that nobody has solved yet i would say please come and talk to

people because there's a lot more tools to make and there's a lot more rules to to figure out and and we need to get to the point where we have those high-powered tools that that like allow everybody to start making this space because this is where human computer interaction is going soon soon there will not be screens up on the up on this wall we'll all be sitting in we'll all be talking we'll all be interacting with something that's floating around and we'll all have a little copy here and we'll all be talking and i'll be collecting a bunch of stuff over to the side and i'll get flagged eventually that one of your guys's questions is pretty big and so i'll start to formulate an answer and then i'll push it out to the crowd we'll be doing stuff like this very very frequent very very quickly i don't i don't think it's even 10 years out personally and so like this is where we're heading this is why we want to do the things that we're doing because we're sick and tired of watching people walk around with this little screen in their hand and they're not connecting with each other and what we really want to do is share compute with each other

and share pixels with each other and this is really the only way we can do it but it's hard oh yeah um there's definitely things i could sorry touch on about like privacy and ethics um i did want to just back up a little bit to the procedural and the fact that like something that may be useful to discuss about how um the tendar was created in terms of like if you are creating a character and that's not necessarily tethered to a surface how do you get it to move and act in a room that's it feels more natural and and that there's a like very complex algorithm behind how the fish swims and moves around that room and at the point when we were making the app that this semantic definition of what a wall is was not available to us so um one of the ways to get around that is to use the point cloud data and then we were able to like raycast to figure

out kind of the boundaries of the room and generate this nut model so that the the character can move within that space so it's a mix of like you know generating from the procedural that was helpful um happy to also talk about privacy and ethics too there's tendar starts with a disclaimer that all the data is on your phone um and that is true it's all models uh that are just saved to your phone even though the company is a kind of fictional speculative fiction that talks about you know gathering emotion i always like having um permissions privacy and ethics on on any talk that we do in the space when we're not directly talking about that just because it's obviously such a such an aspect of all of this and you know it's becoming more and more part of the cultural conversation which is wonderful um but i just kind of want to bookmark that in there of like make sure when we're having these conversations that we're also having those conversations too um but yeah like like brian was saying there's so

so much in this space there are so many unsolved problems and you know any of us who are who are like cracking into a particular part of it we're doing that knowing that there's all this is like infinite amount of stuff on the sides that that we're gonna have to address in time as well so like i'm sure everybody in this room is doing you know is working on on a particular piece of the puzzle and i'm really excited to see this all coalesce in the next coming years that's what we got thank you very much and uh let's if you have any questions please remember to use the microphones on in the aisles there thank you cool total understanding i have a question this is a song okay um in regards to understanding the diverse worlds that are out there and your diverse users have your companies found good ways to understand how to make your experiences inclusive

like how do you understand the variety of living room setups that are on the world the variety of gestures that people understand and how those relate to their cultures at scale i mean quite honestly it's just a massive data collection program that we're undergoing right now like like you know every every region that we're going to go into as as a hardware developer we have to go in and actually find that information out and and in exactly the same way that like when i worked at blizzard and we put hearthstone out on 19 different languages we had to like get all of the text translated to those 19 languages using native speakers it's sort of the same thing you have to go out you have to do the hard work in order to do that cool thank you yeah i totally agree with that i mean i also want to put it out there i know this is you know always always said and always understood but i really want to underscore it here that like i think it's super important that you work with people from different backgrounds to be

asking those kinds of questions and to be calling things out as they see them of like oh hey that is an assumption right there that's a bias um so yeah plus one always gather your group thanks guys thank you hey guys got a quick question i noticed if the device that you're using is stable then the animation plays smoothly but if there's any type of motion it seems very glitchy or is there better stabilization or frame to frame blending because if you're moving around there any augmented reality that i've seen so far just is kind of jerky is there is that a software hardware or is that something moving into the future that's that you guys see being fixed or do you guys don't think it'll be fixed no i mean you you probably have just seen fairly early examples i i mean i see stuff every day that doesn't feel very jerky and so like like the the thing is is that especially with

you know head mounted systems the frame rate frame rate is super important because you're dealing with a physics prediction problem and the smaller the delta t on that physics prediction problem the better the prediction and so like the faster the system runs the better like like exponentially better the the head pose gets and so if you've seen an early system that's running at a rock solid 30 frames per second it's going to be jerky just because when you move the predicting model module can't really predict with with a third of a millis you know with 300 milliseconds where it's going to be whereas if it's running at 90 or 120 hertz suddenly that becomes silky smooth okay and for devs the things we most care about in shipping is frame rate and and smoothness so yeah awesome thank you guys can we switch uh what's that hi um i'm doing a graduate project uh on ar in the environment and i was wondering do you guys can you recommend some solid

resources for us non-engineers who are trying to sort of reverse engineer a lot of the stuff we're finding in tutorials then they're too old and so we're just uh we're basically trying to do planar target image and gps tagged 3d objects so come come talk after afterwards we can toss you some stuff it's it's difficult on camera to give you recommendations yeah let's just work it through like i could name a few people but we're not supposed to like pick sides yeah i wanted to touch on something you guys had mentioned early on about how to train people to interact with these spaces everybody knows how to swipe to unlock your phone but we had to be trained to understand how to interact with that screen space so what do you guys see is the future of a design language that we can all adopt that is easy for people to understand we don't have to build that into our experiences yeah i mean that that that is very very much what is being built right now that is that is

the the bastion level that is forming under our feet as we run down the hallway right and so like like i would say that if you have interest in that work please get involved in it because like we're we're building it right now and that is a moment in time problem i see little kids now walk up to stores and like touch the window because they think it's going to do something and so like like they have an expectation right off the bat that a shiny glossy surface means i can do stuff with it and we're getting to that point now where like i can put a device on people's head and they and they stick their hand up into the field of sensing right off the bat which you didn't have even two or three years ago right and so it's it's we're getting in that direction but there's so much work to do i mean it took 50 years to get swipe right and so like like it's gonna take us a chunk of time to get a good set so it's a chicken and egg problem as we develop the tools yeah whatever sticks best is gonna well and there's also there's also a stepping stone problem people currently have their expectations here based on desktop and and mobile

here's here's what we could even potentially offer them but they wouldn't even think to try it they have to do this and then you have to give them this and you kind of have to step them up there and so a lot of times we end up having tutorial levels that have input schemes that are more based off mobile and then slowly you introduce them to the fact that they don't have to use that input input scheme just over the course of one tutorial you translate them from nothing to a touch-based thing to a gesture-based thing and then they spend the rest of their time in the app in gestures and they never even use the touch stuff like like it's important yeah and i think it's a function of both how will these platforms be adopted by the the public because magiclip just came out but you can see in the hololens history the initialization the realization that it's not a good practice and then the changing of the mind and then so it's an iterative process that a and because the

adaptation is not you know it's not millions of users sending data we don't know what will work and we are learning as we go so as brian said get involved yeah certainly well thank you for the answer and i think it's like in general actually it is a question also of user interface and design on this on particular applications like one of the things we were doing was using like thumb prints in certain ways and indicating graphically that the phone could swivel but i think that with more head mounted displays now it will be more apparent like the way that people will be trained to look around the world but when you're making a mobile application they're going to still look at it as if it's a 2d monitor so like in order to like collect the emotions we have to kind of train them a little bit to like actually look around and it is part of the user the actual um graphics on screen yeah that's what i put out there that um every day on on ud's internal slack uh and i'm sure this is the same at epic and microsoft imaginables um people

are just posting all day like look at the cool things this user is doing look at what this person made look what this person made um so yeah like please keep making that stuff and share it and tweet it and do all that because like we we see it and it does work directly back into the tools that we're making the conversations we're having and all of that very exciting taking off but i'll just ask one okay please uh device or server-based ml solution for spatial recognition 5g is that a solution or we should still rely on the device itself sorry repeat the question one more time um server base or device based spatial recognition so you can walk and determine the space without staying in one place so what is the question server services yeah what do you think is going to take off better server base yeah yeah yeah so so the heavy duty

stuff will always be in the cloud but you need the responsiveness of on device uh yeah i mean even even like let's you know let's say we did jump to 6g uh um i would say that we we still have a latency issue especially when i was talking earlier about the fact that you need in some cases to have really good head pose 120 hertz you know sort of frame rate and and nothing on the cloud is going to be at that rate let's say google does give you the data of 3d space around you they do the google 3d map and you can pull the data from google will that help oh sure i mean i mean don't get me wrong there will be a ton of applications where stuff on the cloud makes it down to the device in time or or or in a way that makes it super usable it's just that because of the latency like like if i'm going to do completely recognizing stuff on my hand i don't i don't want to be sending that to some cloud and then

bringing it back down when i'm when i'm specifically trying to do very very hard fast precise work yeah like like off of my hand let's say so so the question of for instance analyzing a room can be done on the cloud because it's nothing to it's not very urgent to i can stand like three seconds or four seconds until you get as a result but identifying your dog running into the room [Music] needs to be immediate you can't tolerate latency there otherwise things will go very wrong so yeah both solutions will be applied let's cut it there and take it to the hallway thank you very much for coming