Creating Social Experiences in VR

[Music] all right thanks everyone for coming let's let's get started my name is Mike booth I'm a product manager on the Facebook spaces team I lead the development of Facebook spaces so back in April we we launched Facebook spaces this is our our MVP Facebook social vr experiment basically we included in MVP sort of the bare bones of what we felt like we we needed to get something out into the world and start seeing what people actually do with it and and learn because we all have wanted everyone here knows how early it is for VR and there's this we don't know much of anything and we wanted to see what we can learn from this experience so this MVP included stylized avatars the goal of the avatars is that they're you know friendly and inviting and charming but they're automatically generated through machine learning to to look like you we try to remove the

friction of getting into VR that way you can experience your Facebook content 2d photos and videos 360 photos and videos you can take selfies from inside of VR and post them out to VR where we're trying to cross the boundary between VR and 2d Facebook as well Wow we included a marker that lets you draw objects in space to give an open-ended bit of creativity to see what people would do with that and our reach goal was messenger video calling which we'll talk about a little bit as well well you could actually make you messenger your video call from inside of Facebook spaces out to the real world and people would answer their phone and see a cartoon version of you talking to them from inside of VR so our philosophy of why we built Facebook spaces really comes down to to three things the first thing is being in person with your friends regardless of where you are and I'll talk a little bit more about this but the the feeling of being present

with someone else in VR is amazing I think that's the the core piece of magic that we're sort of hanging all of this on that when you're in VR with another friend who's in VR they feel like they are right there so what better way to connect you with people that you care about then in social vr the second pillar is personally meaningful content so once you're together in VR the obvious question is what do you do and as Facebook we have access to all of this content that we know is interesting to you and your friends because either you posted it or your friends have posted it or you've reacted to it and shown interest in some way so we need to expose that sort of content to you in VR so you can consume it specifically with 360 content in a really powerfully immersive way and thirdly you want to spend quality time with your friends doing things that you enjoy and this is sort of really what we're pushing on now is now we have this sort of bare-bones system where you can get together in VR with your friends but we need to flesh

out that that the sort of ecosystem of things to do in VR and spend quality time together so I want to talk the rest this talk I want to go through what we've learned so far in building Facebook spaces and seeing what people are actually do doing with it so first is representing you now I know a lot of us in the audience are game developers so there's the notion of an avatar you know this has been in video games 2d and 3d for a long time in VR things are a bit different where it is an avatar but this is literally you this is your skin when your friends are looking at you and VR they're actually looking at your avatar so it's really important that we get this right so this goes for me all the way back to oculus connect 2015 when I know I was ah it was III I think where I did the oculus toybox demo for the first time now as cool as it was to play ping pong and reach out and pick up objects in VR and stack things and all the fun physics things the thing that

really blew me away was my friend Jason Holdeman was the one giving me the demo he works at oculus that that floating blue head and those disembodied blue hands was Jason like it literally felt like Jason was right there his voice was coming out and it really comes down to you know the the the six degree of freedom tracking of your head and hands all that subtle body language the way he holds himself how he uses his hands when he talks it all came through and I've been in the game industry for a long time I've seen all the Triple A stuff which is all amazing I was up way too late last night playing pub G this was the most impressive thing I had seen today so how can we take this and like add add to it expand on it make it even more intense so when I when I started with Facebook it was just me I didn't have any artists so I hacked this up trying to experiment like could we do eyes can we do emotions and ended up with this really creepy cross between Max Headroom and Charlie Sheen what was really apparent from this is when you're

in VR with this guy he is really creepy and this is not what we want to do but I could make eye contact with it and I could use a like a thumb stick on a controller and sort of lurk between emotional states and I could see the emotions on their face so it was it was interesting and how could we actually make that something that was not awful so we actually got some real artists involved and we built these early stylized humans and they're really charming they look they look great in 2d and this is where we discovered that there's a big difference obviously between 2d and being in person with something in VR when you get when you're in VR with these avatars there was something just a little off-putting and unsettling about them part of that was the eyes or eyes kind of big and spherical and bulbous but what it really was is the mouth and specifically which I'm not showing here with the mouth motion the fact you had this smooth mouth and it was smoothly lurking between shapes but it wasn't quite right made you just not want to be in the same room with that thing so that's a little

bit hard to see the red rabbit on a black background where this took us though was what we ended up calling the rabbit are and this guy there's a whole host of things that we learned based on this first of all there's there's a torso and there are arms we're inferring that based on where your hands and head are the fact that we can do that and it's reasonably believable was was great now we don't actually show that for you because if you look down your actual arms won't line up and that was that was that was that didn't work but it works great for everyone else in the space another thing that wasn't obvious is that the rabbit arts eyes are 2d they're just 2d sort of two knives pasted it on their head but you could still make eye contact and really feel like you're making eye contact with another person in VR with 2d eyes which is great because asset wise they're really quick to iterate on and very cheap assets to make also the emotive the ability to make expressions and have those expressions read worked really well and the secondary motion the

giant rabbit ears that are flopping around the hair are the whiskers etc the amount of expressivity that the secondary motion brings to the table is really useful especially when you have these limited channels that you're talking through adding everything we can add to that to bring more personality is great the last thing you'll see here is the the gestural most we'll talk a bit more about these but you know the ability to put your hands on your cheeks and make an emotion happen with something that people found found really really fun talk about that in a little bit okay so as charming and fun as that was and the number of things that we learned from it Facebook is really about connecting you with your friends and presenting your authentic self and as I am NOT a giant orange rabbit this wasn't going to work so for f8 we took a step back and we revisited the toy box and we had an artist to use our VR markers to basically do a character drawing on these VR these toy box heads and this is myself on the left and Mike's rep for

the CTO on the right and we use this for a live keynote demo at a fa 2016 and that seemed to work really well I mean these are very simple but it was very clear who Shep was and who I was so moving on from that for oc3 last year we did another live demo and we at that point had settled on these sort of stylized cartoon avatars where again like this is more or less me is not exactly but it's friendly and charming and stylized but it has all the things we learned from the rabbit arm there's a torso their arms I can emot-- I can make eye contact and this was this worked this worked well for our demo we continue to iterate on this and with time we shipped our MVP in April of this year this is where he landed and again we're taking more of the learnings from our previous experiments and you can see here like the eyes have gone back to a 2d eyes and just general refinements to

shaders and forms and whatnot and we've continued to iterate on this to our current avatars right now with just subtle details like now we have more controls for the user to go in and like in this case I was able to have a different color to show that my goatee is gray and my receding hairline is brown so this leads us to sort of our golden rule of avatars VR for VR first one is you should be happy with how you look and I'm reasonably happy with that guy and the second one is your close friends who recognize you so this is in other words like this does not have to be a perfect representation not everyone in the world needs to be able to look at you know an avatar and go oh that is exactly that person but if you're in context in a social environment with friends it should be very clear you know oh that's Mike that's so-and-so that so-and-so so how do we create these avatars so here you see our current interface in Facebook spaces where again

we're using machine learning so you log into Facebook and we can bring up photos of you and you basically can pick one of those photos and it presents you with several guesses of what your avatar might look like this was important because initially we really wanted to remove the friction so we just would as soon as you grabbed a photo just put a guess of what you look like in the mirror and as good as the machine learning algorithms are they're not always perfect so sometimes this was jarring because it felt like Facebook was making a very clear judgment on how you looked which is not our goal so by presenting you with five choices it's it's clearer to the to the user that these are guesses pick the one you like and if you don't like that one grab another photo and kind of keep keep going until you find one that you do like even once you have selected the one that you like best you then are free to modify any part of it so you can change colors change eyes change shapes change those shapes etc and as you can see with

with this example like oh that particular one didn't have glasses or put glasses on I'll set them to blue on the left there you can see we have a couple of images of pre-saved avatars so if you've made you might want to have an avatar for different occasions so you can preset some avatar and bring this up and quickly switch back to an earlier creation you've made but the key point here is that you have the final say on how you look okay so now we have these avatars we automatically generate them for you give you the tools to tweak them just how you like how do we actually bring them to life so it's actually you coming through so this is a really interesting example these are white dots moving on a black background but when we look at this we can't help but see a person walking because these dots were motion captured and our brains are wired to see that sort of biological motion the good thing about this is anything

that is captured in this way will seem alive and the current state of VR at least with six degree of freedom tracking is basically a built-in motion capture studio so if you're wearing a six degree of freedom headset on your head and you're holding six degree of freedom trackers in your hands then we get this biological motion and it comes through as long as we don't like we did do some experiments where we try to do some intermediate animations and whatnot and that washed out that signal we don't want to do that basically that's signal we want to leave alone and let's just come through in your avatar because that's carrying all of your body language and mannerisms and it's really powerful so obviously we can't see where your eyes are looking in this headset but we can infer based on where your headset is looking and what objects are in the scene what you're probably looking at and we make an educated guess and it actually works quite well again you know we've we've chosen the environment carefully and people are the most interesting thing generally anyway but this really helped to have eye

contact and you get quickly to a place where arguably this is more interesting than a video conference because you can tell where people are looking what they're paying sorry where they're looking what they're paying attention to whether they're engaged or not and you can get gaze following as well where someone would look at something and you see them looking at that and then you look over to see what they're looking at in addition lip-sync is really interesting and important but in a particular way so with this lip-sync you can see it's very simple and very stylized we're using replacement animation like an argument or a like a sort of thing in our earlier experiments we did try as we mentioned earlier like having more realistic lips and doing all the proper things with blend shapes and lurking between all the Visine and it was never quite right because we're just inferring based off of the audio and that just seemed creepy it was close but not exact so we're in the uncanny valley so we backed off and we intentionally went very stylized with Visine that just snapped two different shapes but this works really well because your brain

looks at this and says oh that person is talking and it looks like they're saying what they're saying because most of the Visine is match up but you do not believe it's real so you're not trying to believe that this is a real thing in front of you now we talked about this with the emoting this is really this is really important and something that that you know we're pushing on is that it's not enough to be there and to look like yourself maila make eye contact you want to express yourself and these your friends who are going to do things that will make you you know angry or laugh or or or smile or be surprised and whatnot so we need some way to at least start to be able to do this so this is another example of the gestural notes so if you take your hands and put them on your cheeks you will automatically express this oh my gosh expression we have a whole set of these we're working on where I like to put your hands on top of your head and then you're scared you bring your hands up over your head in this hooray like you saw with the rabbit are you do like this and you're confused and what's fun about these is they're inherently social so you'll see someone do this like oh how did you do that now you just well you just put your hands on your cheeks and they try it in the mirror like oh that's great and you can

propagate that on that way so that said if you are say putting on a show maybe your live streaming is we added the ability to to livestream from VR out to Facebook a few months ago you want to have direct control so we also allow you to use the thumbsticks to basically puppeteer your face so in this case if you push both thumb sticks together then your your avatar will make that same ohmygosh face but it will stay there as long as you hold it and you can quickly switch between emotions this way so you can build up muscle memory and get pretty quick at this but clearly whose early days and this is very much like VR emoji as in a text message you would add emoji but the fact you can do it all at all I think is really important because at the end of the day what we want to have particularly for Facebook social vr is we want to have avatars that are recognizable by your friends and are interesting and expressive enough that you actually want to take selfies of and post a facebook for example physical contacts a physical

contact I'm using I mean you're in VR it's not physical but if I reach out and actually make contact with your avatar we should recognize that and something something that should happen this is I think a really interesting example so we were at a previous trade show last year we were demonstrating this but spaces at the time had a bug where you can reach out and shake hands with someone which is actually really cool because you're holding a touch controller so when you actually grab the other person's virtual hand you're squeezing the touch controller and it really feels like you're holding their hand but there was a bug so when you would log in the first initially your hand you'll be holding hands they were just your arms were stuck together so we had that we had we had a person come in and we're gonna show them spaces and we had Alicia are our producer back in Menlo Park ready to meet them so he puts the headset on and he finds himself standing about this far away from a woman he doesn't know staring into her eyes and holding her hands and he blushed he blushed this was

awesome he physically blushed his lizard brain believed it enough that he turned red and blushed and he took off his headset he said I need to tell my wife the fact that we can you can't fake that right the fact that we can induce that sort of level of reaction is super powerful and gets me very excited about what we can do in social vr so here's another example high-fiving a friend like we found effect there should be a particle effect whenever someone actually makes the effort to do this we should recognize it and celebrate it now as powerful as VR is you really do feel present with with these folks right now Facebook spaces is all about connecting you to friends using the friend graph in Facebook but some people have lots of friends and some people have you know that guy so you want to make sure that you feel safe and comfortable in VR so when we launched we have these safety features

for example on the left is the safety bubble so by default nothing can get within about arm's reach of you unless you allow it if someone tries to reach into your space or put their face in your face they just they vanish you have to have to let let them do that for that to happen on the right you have a wrist menu so you have a watch basically and on the inside of that watch is a big pause button so if someone is being obnoxious doing something you know like or someone puts up a really high-resolution 360 video of a super violent rollercoaster that does not make you happy you can just hit the pause button and stop the world everyone disappears all the content shuts off you can gather yourself and decide what to do you also have the ability to mute and block and kick people that are in your space as well so we we really take seriously trying to keep things friendly and safe okay so that's that was you and how you are in VR let's talk about your space so when we started this project like my assumption was that we would be hiring a giant team of 3d artists we'd be billing this immense 3d space that

we'd all be exploring and whatnot and this is one of our early experiments where this is like a clubhouse with all sorts of things to do for what we're really trying to do of frictionlessly bring people into VR connect them with people they care about and share personally meaningful content this was a distraction because locomotion is challenging we immediately found that like okay well there's the problem of what do we decide to do how do we get everyone to do this thing and not that thing it wasn't focused on the same things we were trying to accomplish another experiment is fishing fishing as an activity is is ideally suited to socializing right it's it's not so intense that is drawing all of your attention and you can hang out and talk while you're fishing and occasionally exciting things happen the problem here is that we literally got lost in the woods it's hard to see but on the other side a little lake there is the other avatar and they're way far away and the audio falls off and like people literally be like hey where'd you go and they were trying to find each other that was not that didn't fit with what we were trying to do another thing that everyone

wants to do thinks they want to do is I wanna I wanna have a space I want to decorate it just the way I want I want to paint on the walls just so and put all these furniture here and whatnot and this is true and is is definitely has a place and I'm sure that we will come back to this and obviously we invested pretty heavily exploring this but ultimately if I want to meet a friend and catch up with them and you know see some latest videos with them or or share some personally meaningful content with them if I have to start VR and go in and select all my furniture choices and material choices and decorate my space and light everything and have my friends come over and evaluate how they think I am as an interior designer like that's a lot of that's that's friction so we went in a different direction where we ended up is this this blue table at the bottom there basically the idea was a dinner party for you and your friends so it's you and three of your friends in the space there is a table that you're all focused around and you always know where

your friends are okay so the table itself it was not clear at the beginning that there should be a table at all and we sort of evolved our way there over time and this was our first demo with the live demos rep at f8 so here very literal right okay we have a table and here's all the parts and pieces we need to put on our presentation obviously that was not going to scale you know like we have this lovely on the on the left there is our bowling ball rack with our four photo spheres that we're going to show for the demo that's not going to scale to the millions of photos that are on Facebook where we've ended up is this sort of arrangement where we have a table in the center of the table is a display system so you can take content put it there display to the room you have a tablet with a recognizable interface that you can access things like a shelf full of tools that are all useful so we've been sort of iterating our way figuring out VR UI which VR UI

is at least as challenging as avatars and everything else that we've been doing it's extremely hard for example here is here's what we started with with photo spheres this is super fun magical charming right you see this like magical orb you can pick up and look at and you can orient in just the right way and you put it on your face and boom there you are not super discoverable people did not they were mystified by the orb it was great but no one thought hey I will shove this on my face and that's how I will display it so what we did is we made this display center in the center of the table and although you don't see it here as soon as you pick up content that can be displayed the center of the table will glow so you know like oh it's glowing maybe if I put in the center of the table it will display and that was much more discoverable we lost the ability to orient the photo though right where you could sort of look at it so I want the Eiffel Tower to be there and then you put it on your face so we had to iterate our way through that plus the center of the table is like the most

valuable real estate of the table so if you actually want to do anything else like say play a board game or something the display center made it very difficult so where we've evolved now and we're just releasing this soon is that we've moved the display center to hover above the table and only appear when you when you grab content that can be displayed and as you saw there when I when I brought the 360 photo you can sort of put it in the display Center and turn it and orient the the display while keeping the portal only so big because if we had the whole world swinging around you that that was also could be upsetting so this is a schematic of sort of where Facebook spaces is right now for voi VR UI and the different affordances we talked about the table and the display center whenever you put 2d content into the display center it goes on the big display there the dock in the projected UI so you have a watch that is on your wrist first of all it shows the time which is really freaking useful in VR to know what time it actually is but any messages that come in will appear there from messengers and

whatnot so it's sort of your your notification center of what's happening and then any object will see this in a bit any object you pick up will have a context-sensitive wrist menu for the object that you're holding so for example these are dice which we've just recently added to spaces and you can see that as we interact with the dice the wrist menu updates like about holding a dice in my hand the wrist menu becomes you know context sensitive to that dye so I can add more dice I can gather the dice I can delete the dice etc it's one thing that you have to learn but once you've learned that every object in the space has a similar menu that's appropriate to that object so this is this is another interesting bit of VR UI so this is distance grab that we're starting to play with I know there's a lot of VR systems out there that allow you to grab things at a distance and bring it in in the case of social VR there's a couple of additional things that you know we need to be thinking of this is not implemented yet this is just

early experimentation but you know if I want to pick up that that red tube I can't just teleport into my hand what we need to do is we need a way for me to say I want that red tube but all of the other people in the space need to see that hey mike is going to take that red tube and allow them to see what's going to happen and be able to intercept and changed their mind basically like I'm trying to steal something from you you need be able to grab and say no okay so how do we what what are we doing in the space we need to bring in things to do and when we launched spaces the big the thing that we had that enabled this the most was the marker you could pick it up you could doodle that's great but really it's a 3d printer so in this case you know we could play a game of tic-tac-toe in space which is fine but you could also spend a little time and like draw all of the part all of the pieces of a chess board and then bring in a 2d photo of a chess board and actually play chess you can save all those parts away and bring them back whenever you want so you know we're trying to create an

open-ended verb to see what the community would do with a easy way to construct 3d objects in VR this was also wonderful for collaborative design this is a kind of a mock-up of the Avatar creation that you saw earlier we actually did this in VR with like Christoph Xavier is the lead designer on my team he and I were in VR together with markers we would just draw different affordances and shapes and whatnot and arrange them in space and pretend to use them and see how it felt that was wonderful you can also in this case like one one of my goals for the team is I want to get my Dungeons & Dragons role-playing night back together all my friends are scattered along the West I wanna be able to go into VR and have a D&D game night so in spaces if you go look at the drawing library there are dungeon pieces for that reason but you could also see a little hard to see here these these these fighter characters and the dragon this is something that we're just adding two spaces called photo cutouts where basically you can take any 2d photo including your PC desktop which

is interesting and with a cut tool you can trace lines around and cut out shapes and bring them into VR with you so you can make any any board game piece any stand the mask etc so based on what we learned from markers we're going to photo cutouts to allow people to be more creative okay so this is a this is our current UI for exploring Facebook content we've recently added a search bar I immediately use this I I grew up in Iowa I'm from the Midwest so I went in there and I hit the search bar and I typed in Iowa and I immediately saw a bunch of 360s of the Midwest and I was able to like sort of wander around my hometown and look at you know rolling hillsides from Iowa it was great so now we've really opened up anything you can think of you can search for and find I'm gonna have to go fast cuz I'm running out of time okay so I mentioned the PC desktop so this is actually your PC desktop in this case this is Brian's Instagram feed and his

favorite dogs you can see here he grabs the cutout tool and he traces around his his puppy and he will get a cut out of his dog now the thing with the PC desktop is this literally your PC desktop everyone in spaces can bring in their PC desktop so you can have land party where you're each playing whatever game that you want to play together you could be all over the country but you could be together in VR doing that and any any website anything on your PC etc you can access in spaces if you put your PC desktop in a display center it displays on the big screen for everyone in the space to to watch as well so again looking for open-ended verbs to express creativity in VR and see what the community does with these tools we talked about dice there's a couple this is a good example of things that we can do where there are familiar objects but we can do things make things a little easier in VR so dice have the properties they do in real life you can pick them up you throw them they bounce around but of course we show you the numbers in the sum total because why not but also things like in this case these two are

playing liar's dice it's a little hard to see but you can take your dice and make them visible to only you that doesn't make your dice invisible to everyone else but what they see is instead of numbers they see question marks so we can roll the dice we can play liar's dice we're bidding back and forth and then when it's all over and someone calls you a liar you can say show two friends and then all the question marks turn to numbers and you can see what it is essentially having a dice Cup in a different way and of course one of the most fun things in spaces continues to be the ultimate vr photo booth where you can take all these creative parts and pieces that we're providing you and people just have lots of fun goofing around their friends and then ultimately they take they take a selfie and they post it out to Facebook where you know the two billion people in the facebook graph can see this and start to see what is this VR thing and oh hey this VR thing is interesting and I might be I might want to get into it so this gets into opening up VR for everyone because the goal is VR is for

everyone selfies obviously are the biggest thing they're the simplest thing to start with and as we just described they're a lot of fun but people out on Facebook who don't even know a VR is can start to see these artifacts and ask questions and get interested we also as I mentioned that launched we integrated messenger video calling so from inside of spaces you can make a messenger video call to someone just out in the real world and they will see a cartoon version of you and you and they can interact with the space with you we've also recently allowed you to actually receive a keyboard inside of spaces so you can respond to your messenger your messages this like search it's really simple but when you go from 0 to 1 like I can't do this at all I have to take off my headset and go back to my computer to respond or my or my phone or I can just quickly respond from inside a VR it is it's it's really game-changing unless you just stay in VR and it feels like everything is wide open in addition we launched the ability

to go live so this is a example I was actually on an airplane 30,000 feet in the air on airplane Wi-Fi watching a livestream of my team testing live-streaming and I commented I'm watching you from 30,000 feet so when you're in spaces you see this sort of magic mirror of the live stream that's happening so you can see what everything looks like reactions that people gave like hearts and likes they come popping out of the screen and any comments show up next to the magic mirror so what they're doing here is they reached out saw my comment grabbed it and pulled it into this space and showed it back to the camera and when I'm sitting on the airplane I saw them do that that was just it was amazing it would feel like I was involved in the in the VR experience we plan to do we want to do a lot more with that in the future here's some examples of live streams from the community and there on the right that's a live stream from slate slate has started to put on a weekly show called conundrums where they bring in celebrity guests and they do a quiz show with them from inside of spaces usually Lindsey is in New York and their guests are often in LA in the future we

really want to get to this people centric computing platform and we really want to get third party developers involved we're starting to look into ways that we can make this happen so in this case you know if I'm there with Rachel like I wanted to connect with Rachel first then we decide to play let's play a game of pool we didn't launch a pool game app and then invite Rachel in I was there with Rachel and we brought the pool table and we might decide hey this should be in a pool hall so put a 360 of a pool hall around let's do it in a in a back alley because I don't know why not and then while we're playing pool we want to listen to music so perhaps Rachel brings in her boombox where she can stream her favorite music service these things should all be built by third parties they should be brought into spaces and we're trying we're just starting to explore how we can make that happen so as tough as VR developers think about how spaces could be useful to you and let us know so again VR is

for everyone and we feel like this is one of the things that we're trying to communicate at Facebook I have been in the game industry for several decades I was up late paying playing pub G as I mentioned I understand that VR is amazing to do awesome games where you blow up robots and shoot zombies but I think to get to the kind of scale that we're talking about with two billion people on Facebook it's important that we communicate that VRS for everyone and find other ways to express the kinds of things that we all know VR can do again this journey is 1% finished everything that I've said and displayed here could be entirely wrong I might change by next year but these are the things we've learned so far that we wanted to share with the community so if you want to learn more facebook.com slash spaces thank you very much [Applause]