Integrating 2D UI with VR Environments

[Music] you can't hear me oh okay there we go um integrating 2D UI with VR environments probably already read that on the program who am I um my name is Brad Davis I've spent over 20 years doing software development um I've been experimenting with VR since the kickstarter Days the original devkit 1 uh I built the first Oculus sorry the first Java bindings for the Oculus SDK uh I've co-authored a book on developing for the Oculus SDK called Oculus Rift in action uh for the past couple of years I've been at High Fidelity we're working on creating an open- Source framework for an actual metaverse distributed servers anybody can build things nobody's in control

uh and among other things there I've worked heavily on integrating the 2D UI systems into the 3D world uh so that covers really both of these points so why 2D user interfaces uh VR is this massive shift why do we still need 2D for anything right well 2D UI is everywhere and it's familiar to everyone everyone who's already using a computer um so that's a good argument uh also VR I think as we evolve we're going to see uh VR metaphor replace a desktop metaphor where a VR environment becomes your place to work in we're already seeing moves towards that with things like big screen and virtual desktop so we have to understand how we're going to integrate 2D UI into that 3D

environment also many things don't benefit from any sort of 3D UI in fact very few things that aren't inherently 3D like uh a topographical map or um a body scan uh these things don't uh benefit from just being flashy and in 3D um while 3D representation in a VR headset can give you more information about an object like depth it can also obscure information uh an object that's in 3D part of it is always going to be facing away from you so with the 2D UI that's not the case um also the technology isn't necessarily there for translating some things in the real world that we have sort of either full 3d or semi-3d interfaces for uh take for example a light switch on a wall uh a light switch on a wall and a checkbox on a 2D

dialogue serve the same purpose they uh have one of two states they indicate to you which state they're in uh to the extent that a wall switch has a 3D interface um that's useful because I can reach over and touch all white Switch and determine what state it's in or change its state without looking at it uh and as nice as the Vive controllers and the Oculus touch controllers are I don't think there's any amount of little haptic buzzing that's going to replicate that so I don't think the technology is necessarily there for translating existing 3D stuff into the VR world yet and finally we need to walk before we run um I've actually heard this uh in a couple of other talks but the idea that you know when film curse came about we had to learn the new uh vernacular of film the concepts you could translate

and film that you wouldn't work that wouldn't work on a stage how to interpret stage directions from Shakespeare uh between uh a baz lurman production of Romeo and Juliet versus you know a uh an original Stage production they're vastly different and it took time to learn the new techniques and it's going to take time to learn you know what new techniques that we can apply from VR in 2D UI to enhance it perhaps make it 3D but that's going to take time um I like to break down the the concept of input styles into three different categories that are essentially layers uh the the first kind of style that you have is the console style and this dates all the way back to the original Nintendo Entertainment System where you have up down left right enter and back and maybe you have a meta button for you know accessing the o

and we've seen this uh style mirrored all the way up into current consoles and for current console os's you still use these basic six functions up down left right enter back to navigate the OS to navigate menus and games and it's very simple and easily conceptualize it enables navigation uh above that you have sort of the mobile style and I call it mobile style because it's best example if IED by uh cell phones and tablets where in addition to simple navigation you also have a point a touch point on the UI surface an XY coordinate that means the user is interacting at this point on the surface um whereas console style lets you enter text by you know left left left left left left now I'm on the q key and I can enter Q uh touch surfaces allow you to much more rapidly enter text uh I still wouldn't want to write a research paper in it but it's there um

and then finally the the the top of the food chain so to speak is the full PC style input where you have a keyboard and a mouse and this enables in addition to uh whereas the mobile enables dragging and drawing operations uh the PC style enables massive amounts of text input and manipulation and also enables enhanced dragging and drawing operations um if you think about 3D manipulation programs often uh touching and dragging with one key press control will rotate or with shift will Zoom uh so it enables further kinds of input interactions um I also want to kind of divide up the kinds of UI that you can talk about in terms of uh internal and external uh external UI is something that is UI that's part of the scene uh

an ATM screen on an sorry a screen on an ATM in an inworld object is external UI um whereas something like a browser window is something that's internal UI it's private to you uh in a multi-user or social VR experience external UI is something that's going to be visible to everyone internal UI is going to be visible only to you um most of the stuff I'm going to be talking about most of the interesting problems I think have to do with uh internal UI presenting um the customizability interface to a user uh I'm trying not to zoom through this but I'm also trying to make sure I have time for questions so external UI uh external UI is conceptually easy in terms of rendering you don't have to

think about how am I going to position this UI relative to the user it has a position in a scene if you're rendering a cockpit and there's a UI element in the cockpit that has a particular place it goes and just as you're rendering the rest of the cockpit you're going to render that UI element um also there's no conflict between the depth that the object is and occlusion um I'm going to cover this in a little bit more detail later but essentially if there's a UI element there and there's something in the scene that's closer to you than the UI element then you just have to deal with that that object obscures the UI element and you have to you know figure out how to get it out of the way maybe that's part of the game play maybe that's just where the user stuck their head um as compared to internal UI though uh external UI is going to be harder for picking um if you have an arbitrary point in space and a ray

projecting from that point um both for internal UI and external UI you're going to want to be able to uh say where on the XY UI surface does that point intersect now for internal UI which I'll cover brief in a moment um you're directly in control of that for external UI the position of that UI surface is completely relative to you know where did the user stick their head and where is that scene and you have to have some sort of picking Logic for it um hopefully whatever rendering system whatever rendering engine you use lets you say okay I need to do the triangle intersection I need to know exactly what point on this rectangle I'm hitting and you can translate that into XY coordinates also for external UI if you have multiple elements of external UI um you need to be able to find some way to manage Where is the Focus if I provide a

user input does it go to this element or does it go to that element um Elite dangerous is a good example of uh gaze Focus uh it has multiple UI surfaces that you can activate by turning your head and focus on that and when you have that thing focused then that's where the input goes uh you also have the same issue if you have both external UI and internal UI that are aail ailable at the same time um one thing I wanted to note is when you're building external UI in particular you have to beware be aware of making your users uncomfortable um the uh the the Vive controllers in particular when you're doing picking um tend to shoot the picking Rays out of the top of the Vive controller so if you're holding like this basically the rays are kind of going along the axis of your thumb but

uh if you tried to use a touchcreen with your thumb that's below you that's going to be really painful you want to avoid anything that's going to bring the elbow up above the uh the hand in order to do that um also if you're just using sort of gaze activation again anything that's kind of below a few degrees of uh The Horizon it's going to put strain on the neck as you have to look down and especially if you are doing detailed G Focus um you don't want to have to make have make your users have to make fine motions with their head as they're looking down um you want to kind of avoid having them crane their neck down if at all possible so for internal UI uh you have to decide whether or not uh and this may be not your decision but something that's just dictated by your application or game uh is your UI going to be transient or persistent um a transient UI would be

you know a menu you bring up to say Okay I want to load this other level I want to do this other thing um but a persistent UI might be you're in a social VR environment and you want to have a browser window that you can refer to while you're talking with this other person so is it going to be something where uh interacting with the 3D scene and interacting with the UI need to be essentially almost concurrent or is it going to be something where uh it's sequential you're doing one or the other uh when you're dealing with internal UI uh you're directly in control of how that UI is presented to the user and uh you may have a desktop metaphor with lots of different elements that the user can move around you may have lots of individual elements um that are each rendered individually or something in between uh but however you have it you're going to

have to figure out a way of how do I inject these into the scene um open VR and Oculus both have uh support for overlays or layers that they will inject into the compositing scene for you into the compositing process for you um but they're fairly Limited in scope in terms of how you can use them um they they only rep um sorry Oculus will only allow you to render to a flat planer surface um open VR will mostly only let you reenter to a flat planer surface and then composite that in uh it will also let you render one curved surface but you don't have direct control over the amount of curvature and uh most of the feedback I've seen most of the um most of the talks I've seen when they're talking about UI uh really recommend that you want to have a curved

surface um that's I'm sorry I'm getting off track um when you have these internal uis that are projected into your scene but are not part of the scene you have the potential for uh oclusion or depth disparity one or the other um generally you do not want to render a UI closer to the user than at arms length um it can become tiresome because the depth cues between accommodation and convergence uh will disagree so you want to have the UI large and Far Away um but when you have a UI that's large and far away that also means that elements in the scene might be poking through the UI uh and you can either have that be a situation where uh the elements poking through through the UI then olude the UI

limiting its usefulness uh or you can have the UI simply overlay the 3D scene the problem with the lad is that if you have the UI overlay the 3D scene then you get different depth cues um if there's an object that's poking out below it and one of your depth cues is telling you that object should be closer uh and your other depth cuers are telling you no this overlay is closer because it's obscuring the other object and that can cause some discomfort um sorry uh if you're creating a desktop metaphor where you're actually creating an entire window system that requires a you to do a lot of work right now there are UI libraries out there that will render UI for you uh but uh generally they won't do a windowing system for you at most you will be able to render a bunch of

individual elements and you'll have to create uh the concept of window decoration or Z order things like that and then um finally on internal UI don't ever pin the origin of the UI directly to the Viewpoint um if you want to make text more readable um or want to make various parts of the UI more accessible it's important that a user be able to lean their head left and right or lean in and get closer to the UI to gain greater Clarity on it um you can also get about an order of magnitude more readability out of text even if you can just move your head in almost you know tiny amounts uh and see the text change you can make out the text shapes better

see so compositing and projection so when you have internal UI you need to composite it into your scene and you need to choose a projection um like I said the existing sdks have support for overlays but they're very limited uh they will only allow you to do compositing over the scene they will not allow you to do compositing into the scene where the UI can be included um whatever projection that you choose is going to lead to some kind of distortion of the objects and when you're doing uh picking against a projection the more complex the projection that you use uh the more complex that mapping from a ray An Origin and array direction to an XY coordinate on the UI is going to be

see um open VR does help you in that in that um they actually have an API that says uh given this Ray and Direction uh what is the XY coordinate on this overlay service that I'm hitting so for the issue of uh occlusion versus depth cues there isn't really a general purpose solution um if you're using a transient UI uh you can optionally fade the 3D environment to a mono environment while you have the UI open uh which eliminates any potential for competing depth cues but the fade effect itself whether it's immediate or over a small amount of time can be discomforting um the other option is you can call the problematic objects um but

coling objects can result in like bisected avatars um or important items missing from the scene uh for our purposes at the moment what we just do is live with the competing depth cues okay so integration details what are the requirements for a UI library that you want to use to uh integrate into your 3D application it has to work on your target platform um this may seem like a no-brainer but you have to think about the platforms you're using now and the platforms you're going to be using in the future if you're eventually thinking well we're going to Port this to Android you don't want to have to use something you don't want to be using something where you're going to have to rip out all of your UI

and rewrite it um you have to use something that has capturable output uh ideally you want a piece of software that will render a UI directly to a GPU surface uh a texture um either a direct 3D texture a Vulcan image or an open jail texture less ideally you could use uh something that will render to a uh a a image in CPU memory and do the transfer every time the UI changes but that's going to be consuming a scarce resource um namely CPU GPU transfer and GPU memory GPU uh processing power and finally your library has to be able to accept programmatic input uh most UI systems just accept native events from the OS or from whatever uh framework you're using they uh you want to be able to tell the

UI Library you're using I don't care where the OS says the mouse is I don't care what the OS said the keyboard input is I'm handing it to you uh because you're going to need to be able to do translation uh desirable for a UI Library uh you want it to map to a set of UI controls that are familiar to everyone the checkbox the button the slider the scroll bar whatever um you don't want to have an in-house designer who comes up with a UI design and then discover that it's going to be a gigantic pain to implement because your UI Library doesn't really support you know all of these common features you want to have good tooling support uh the ability to have lots of easy to use tools to build your UI uh things like Android provide that

uh most UI systems have some sort of UI mockup mechanism um if it's something that provides good HTML rendering then that's automatically going to hit the first two points and you want it to provide a rich Focus model uh in particular it's uh valuable to have knowledge of when the UI is focused on a text element in order to provide inor um onscreen keyboard elements so the qml sorry the UI libraries that I've worked with uh I've mostly worked with qml in the past couple of years uh that's part of the QT framework uh it hits all of these notes uh qml will only render directly to an opengl Tech I'm sorry qml is a general purpose Library you can build a native application with it and it will render

to the screen in terms of rendering to an offscreen surface it only supports openg GL uh crazy Yeti GUI and lib rocket are both more special purpose libraries uh they're targeted at game makers who want to build some sort of UI for their application uh they're a little bit more limited in terms of their features uh but they will both render directly to to open GL and to direct 3D so um when you're dealing with input uh most UI libraries are typically built around the concept of keyboard events and mouse events um so if you have controllers the Riff controller the Vive controller uh the Oculus remote you're going to have have to have a translation layer

there sorry there we go um keyboard events are fairly simple usually there's a straight one toone correlation between whatever keyboard event your framework supports and what whatever keyboard event the uh UI Library will support Mouse events are a little bit more tricky uh you usually need remapping of some kind uh but naive remapping can also be tricky um one of the issues is uh in VR you might have a very large offscreen UI surface that's acting as a a kind of virtual desktop so if you have a you know 4,000 by 2,000 UI surface and you have an onscreen Mouse onscreen

window that's you know 100 by 100 or something more reasonable um the mouse movements are not going to map one to one uh so it's better to keep a sort of virtual Mouse position specifically for the offscreen UI and use Delta Mouse events to move that around rather than try and do an absolute mapping um hand control ER picking into the UI surface is tricky uh as I said open VR has helpers for converting um if you use a simple projection on which to put your 3D on which to put your 2D surfaces um then you should be able to create a simple mapping between uh 3D rays and the offscreen surface

so for output uh you've got two levels of rendering you have one process you've got one uh you have your UI scene which you need to render sorry your UI that you need to render and you have your scene that you need to render and you need a step to composite them together if you're not relying on the SDK to do that compositing um you want to make sure that you use a rotating buffer of output textures you don't want to just use a single texture for rendering your UI to the texture and then composing it into the scene and then waiting and doing that again uh because you need one version of the texture which is being used for reading one is being used for writing and one which is in flight so typically you want a rotating buffer of at least three textures to hold hold your UI um ideally you want a distinct

rendering context for your UI you don't want the UI rendering to be blocking the main rendering you don't want the UI rendering to be changing the state of the open GL State machine or the direct 3D State machine uh while it's doing that and then synchronization can be a pain if you're working with if you're familiar with working with fences and uh passing textures between multiple open jail context um you have to be very careful about uh fences and flushes and making sure that even though you've completed all the commands to render the UI the GPU hasn't necessarily finished the work so you need to use synchronization Primitives to protect against um rendering corruption so tips for designing UI for VR um you want to go big with

your elements uh I recommend designing as though you had a fairly low resolution screen compared to current screens um you want to avoid controls that require fine motor skills uh one of the recommendations I I have is that if you uh if you have a touch screen available um design your UI and then practice using it at Arms length with the touch screen cuz that's going to reasonably accurately simulate the kind of accuracy you're going to have when you're using hand controllers and a pointer um and be cautious with transparency uh it it it might be very tempting to say okay uh I've got this 3D scene I want to make sure that even though the user has a browser window up they can see the 3D scene partially be behind it um this ends up becoming uh a distorted mess you can't really make out

the text and you can't really make out the 3D scene you don't really have any control over uh what's going to be behind it necessarily so uh avoid transparency particularly in elements that are going to be containing text or might contain text you know it's fine if you want to use partially transparent elements for window decorations but um also in terms of avoiding fine movements um you know compare a a number field that has little spin box controls that are wedged in there uh you want to avoid that if you want something that moves a number value up and down try and lean towards a giant dial like you would see on a [Music] tuner yeah okay um also more tips uh you want to optimize for multiple kinds of

inputs um and in particular you want to look at the inputs that the user is actually using at the moment um I found a number of applications when we first started working with oculus touch it worked with uh steam VR applications but a lot of the steam VR applications were uh working from the assumption that you had track pad touch pads not joysticks for the analog controls um which meant some of the uh some of the tools were just much harder to use than they needed to be because they needed they expected you to be able to really finally control the XY coordinate of the joystick in a way that's easy with a touchpad not easy with the joystick and also um I can touch type so you know if you've got a text field don't force me to use the on screen keyboard um just try and be aware of like different people are going to to use different inputs and try and put in

the extra effort to customize the functionality of your application for whatever kind of input they're using um and then finally you want to minimize friction that they have they encounter when they're switching Focus between the UI and the world and vice versa um I don't want to have to pardon me uh I don't want to have to uh hit a button or jump through a bunch of Hoops in order to be able to look at a browser or look at the person I'm talking to uh I want to be able to just turn my head uh so that's uh and that's pretty much it hi Fidelity with hiring uh you can contact me uh on Twitter or email and uh with any questions you might have um also we do have a little bit of time for questions um so if anybody has anything they

wanted to ask please step up to the [Applause] mic hi so you see see these sorts of things translating oh uh translating over to augmented reality it seems like a lot of the lessons are sort of one for one do you see any big differences um for augmented reality I kind of feel like uh sorry I've got something caught in my throat for a the question is uh how do I see them translating into augmented reality uh I kind of feel like in augmented reality uh you expect to see even more UI elements but they're going to be sort of somewhere on that spectrum of

internal UI to external UI they they're kind of going to be attached to things that's what you would expect from augmented reality you know I see a person walking down the street and they've got a tag floating off of them with you know hey I'm uh I'm this person you know here's my family's names and you know here's how I should greet you and here's how long I've known you um so I I would expect a lot of the same uh uh information to apply uh I don't um I I'm not I kind of feel like augmented reality is is providing rendered information to you but doesn't necessarily expect you to provide a lot of feedback into it um certainly you'll you'll want to have some sort of UI for overlays and they'll probably be a lot of the same uh uh same information you'd want sorry the same uh topics would apply to augmented reality in terms of

like the integrated UI it's like oh I'm going to activate this menu and I'm going to bring up this overlay uh on the real world uh so I I I'd be mostly focused on uh the things I was talking about in terms of uh internal UI thank you time okay well thank you