The State & Future of AR Games: Rose-Colored Glasses

[Music] yeah we're going to talk about a are today the talk says the state of AR in the future and I have a confession to make I don't actually believe in predicting the future I think it's kind of hard to do but I am a big fan of a quote by Ellen K which is the best way to predict the future is to invent it or to build it and so I think what we're gonna do today is we're gonna talk a little bit about our learnings at Niantic having launched some a our products and run them over the past couple years and then I'm going to share some about our thoughts about stuff that's under development right now so stage one is looking back stage two is about what tech is getting built and the importance of that tech and then stage three of the talk is going to be about what we think the implications of that technology are for the kinds of products that we can build whether they're games game

developers in the room fair number good so we're talking about games we're talking about non game applications as well so looking backwards let's start with just a little bit about Niantic I know some people know us as the company that made Pokemon go or the company that's spun out of Google which is cool that's what we are but we're our own thing too and the thing that we're kind of proud about is not in a chess PD way but proud of is why we're doing what we're doing and so I want to give you some insight into why we're building these products these things never really work when they're supposed to work I can push this button so why do we do this stuff so the journey actually started quite a ways back for us there's kind of a through line that connects so for me it started when I was a kid actually I grew up in a small town it wasn't a lot

to do there but at one point somebody came and they dropped off grocery bags full of old National Geographic magazines and for me I fell in love with a maps that you could pull out I can still see the one at the Nile Delta and all of the kingdoms just amazing stuff so I spent hours with those maps of fears later I saved up my money from mowing lawns and bought a computer and so on that's kind of my whole story right there like a bunch of National Geographic maps and a computer and basic and no money to buy any software so then things happen but that led me through a series of events to meet some really cool people including a guy named Phil Kessel and Phil is the CTO of Niantic back then phil was an engineer that I met out of Silicon Graphics and we and some colleagues chicawa Zama avi Barzeh of mark abend Michael Jones Brian McClendon we founded a company the company was called keyhole it's a small company we didn't have a ton of resources but we had kind of an

audacious vision especially under an especially audacious vision for a company that size we wanted to take 3d technology new digital satellite imaging technology and broadband and make a map of the world unlike any map that had been made before a map of unprecedented eat ale built on imagery for the entire planet so we got started with that we actually were acquired by Google along the way and with help from a bunch of other talented people inside of Google Bret Taylor Jim Norris Andrew Kermes E to name a few but actually a very large team we were able ultimately to realize that vision to build the map and really to put the whole world on your desktop initially and then later we brought it to the palm of your hand when we brought it to iOS brought it to the iPhone and brought it to Android so after we built the map at that point there are thousands of mashup sites you know people doing all kinds of crazy things

with maps and so we started thinking about what what can you do with a substrate that we've created this Geographic substrate of the world and we also got interested in the kinds of devices that people were tinkering with that look like they might come after the cell phone they might be that next thing that would come there were early versions of wearables watch type devices people were playing around with HMDs different kinds of AR glasses and so the thinking was okay Maps location information the real world these things probably combined in some interesting way I got interested in part because I became a parent I had kids and you know when you become a parent it's like it's your chance to do everything over again it's great it's like you reset and so I was thinking about all the places I wanted to go and explore with my kids so we started Niantic with kind of those themes in mind like let's see what happens with maps and location wearable technology to build stuff that

brings those worlds together the digital world in the physical world so we built some products by the time we spun out of Google Pokemon Go is in development and we had a pretty clear notion at that point in time of kind of what our value would be the things that we wanted to I mean with software you guys know how it is like you can build anything you know sometimes the hardest thing is to decide what you actually want to build so we said well let's put some parameters around our efforts here to guide us in a particular direction and the ones that we came up with based on our experiments and based on the early games that we pushed out to users and beta tests and so on were these it started with this idea of exploration so sort of the key thing that we settled on was you know in every neighborhood there's probably a story that's interesting there's a mystery that can be built there everywhere in the world so it was this idea of exploration the second thing we settled on was you know being computer people and spending a lot of time at our

deaths you know we realized it hey everybody needs a little nudge sometimes to go outside and get their 10,000 steps in or get their daily workout in maybe some of you have had a chance to check out our garden over outside the Metreon good if you get tired of sitting inside we've created a nice space for you out there and we heard back from some of the early users of our games from ingress that people who didn't consider themselves athletes really appreciated that gamified kind of nudge to to exercise we we adopted that as one of the three principles and then the third one was social so the other thing that you know and this really came from feedback not from originally from our own in sight but from what people told us when they were using our products was that they loved the opportunity to meet new people and have a thing to do together with their friends or family or parents with their kids outside like an activity that people could do together just lots of love lots of fan mail lots of positive reinforcement around that so we adopted that as the third principle

so this is sort of guiding us we track our progress I mean just to give you a sense of scale of you know how this is going we measure various things in terms of the exploration piece we measure the total number of places that have been observed as you know special unique places in neighborhoods where people lived that people have photographed and named and described and added into this global game board so that no numbers in the many millions we've opened up that submission process in Pokemon goes so it's it's growing rapidly worldwide so this idea of stitching together a global game board a place that we can play out of all the interesting nooks and crannies in the world it's going pretty well on the exercise front this one is just you know it blows us away we check this number it's like bigger every time you check it it's great billion total kilometers walked and it's interesting I know here at GDC maybe four years ago five years ago there were there was a

speaker talking about some of the early efforts to do outdoor games and the conclusion of the talk was hey people aren't going to go outside and play give up like gamers want to stay seated in their couch and I don't know if it was just timing or the evolution of the technology or that right combination of gameplay and design and tech but that's not true like people like to go outside and they really enjoy that that part of the experience in terms of social we measured in two ways we measure it in terms of like the formal connections that people form in the game we added the friend feature and Pokemon go last year so you can identify your friends you can exchange gifts with those friends you get bonuses for playing together and raids so now there are over 190 million friend connections so lots of activity there and we also measured in terms of the events that we has anybody been to a go fest event or a Pokemon go community day okay decent number on so if you haven't been to one

these events look a little bit like music festival if you've been to like Outside Lands or Lollapalooza or something like that may be combined with like a healthy dose of comic-con with a little bit of 5k if you can like combine all those ideas so people coming really fully you know into the game living out their fantasy of being a pokemon trainer or part of that world and families have come together groups travel from across the country carpooling together people travel from continent to continent so really neat these have been very successful for us last year we had three million people in total come to these events and single day attendance at a single event we've had events with over a hundred thousand people so this idea that games can be part of that festival like outdoor world is very much true so just in terms of catching up and looking back retrospectively you know a lot of

people know us as you know a company that helped to pioneer this idea of consumer AR the first game we launched was ingress back in 2012 it's going very well it's played in a bunch of countries I think 175 countries around the world we've helped hundreds of the events of the kind I was describing we were doing it with ingress before Pokemon go we launched a new version of it last year ingress prime and this was our learning place this is where a lot of the feedback that's informed our later work came from Pokemon Go continues to set records in terms of users revenue it's now over a billion installs in total over two billion dollars in total revenue and a really awesome amazing global community and we're continuing to obviously invest heavily in that product we recently rolled out our or a our photo feature in Pokemon go I don't know if anybody's tried that it was the third

version of AR in the game so the very first version Pokemon go had a really crude implementation that they are we did a r+ about a year and a half later and then this is our third or third iteration of figuring out how is the best way to actually utilize AR in a helpful and fun way with within a game so check it out if you're kind of looking to understand our our thinking about this at the moment next up for us is Harry Potter wizard unite hopefully there are some Harry Potter fans out there so this will come later this year I will play a short trailer for you I don't know about you but when I hear

that music I'm ready to play the game I love the world of Harry Potter my kids grew up with it reading the books I read them aloud to all I have three kids to all three of them at this point so we were excited about this product we just gave a first look to the press a few weeks ago released some of the first screenshots and video of game play so we're looking to this as a platform to evolve our work around real world and around augmented reality a lot of people talk about AR and AR games by the way and we really prefer the term real world because we think it's that fusion of the digital and the real world that's the interesting part of it and they are kind of you know sometimes you're building AR and it's that technology of I've seen something

through a viewfinder which is a certainly part of what I think of is the full breadth of augmented reality but it's not that's not all of it you know it's really about how you integrate that with the physical world okay so behind these games is a platform and this is the set of tech that drives the games it's evolved with each of these products we actually have several projects under development that have not yet been announced that are built on the same technology stack at present we have as many people working on the platform technology as on Pokemon go itself so there's a robust effort to mature this and productize it this tech is the part of the system that does the heavy lifting so that we can all play in a single game instance so if any of you played World of Warcraft like I did back in the day of course we had to pick what server to join when we wanted to join a massively multiplayer game the innovation here is that this system holds state for all users in a single instance if you will of the game

it's of course striped across tens of thousands of servers but you don't have to think about that as a player the system also supports all the social features that I've talked about all the mapping of the location information that's integrated into that really all the support that you need to develop and publish a game like this and of course advanced AR and we're gonna spend maybe the rest of the talk talking about the advanced AR part that gives you a common layer between AR kit and a our core and also provides a way for us to extend that functionality with cloud-based AR capability and that's where a lot of our R&D is is is going we do have a developer contest that's open right now the Niantic beyond reality contest there are a million dollars in prizes for that contest the it's open for submissions until the end of the month there will be a sort of game jam session through the summer who will be working with the finalists and we've set aside actually

several tens of millions of dollars to fund hopefully some projects that emerge out of that beyond the prize money and also other projects that will identify next year so something to check out if you're interested in building products like this why do we want to see more games like this one of the reasons that we do or events like this so I talked about our Niantic hosted events within that are a subset of things that we do with civic organizations we think about technology there's not a bad press about tech right now I think the tech can actually have a very positive impact on the world and this is one example of that we partnered with the knight foundation they do a lot of work in cities that are distressed that are trying to rebuild that have serious economic and social problems and one of the ways they do that is they host festivals to get people into the city and just walking the streets meeting their neighbors rediscovering what's there so CicLAvia and Los Angeles is an example of that the Philly free streets

initiative is another there's a UK org that we work with it which is independent of the knight foundation called big heritage and they're devoted to teaching history to kids by having these big festival style events families come together and then they learn about the history of the town in this really fun interactive way so in all these cases we're overlaying gameplay onto an existing kind of civic festival dramatically increasing their attendance and drawing in an audience that wouldn't otherwise probably attend so we wanted more events like this and that means opening up our platform so that many people can build hopefully really fun experiences on top of it okay so what have we learned you know this is the part where we share some of our stubbed toes some of the mistakes that we've made you know the first one is really a message of encouragement about the potential to build games that are broadly inclusive and broadly appealing

whenever I first started investigating the idea of us doing a game I was doing my research and people talked about casual games and mid core games and casual games had a certain look and feel they attracted a certain audience they had strengths and weaknesses mid core games were for real gamers and they had a completely separate dynamic to them and then there were you know hardcore games or like the real serious console games we didn't want to put ourselves into a single category we wanted to build something that was broadly appealing but you know we had ambitions around retention we wanted it to be financially successful and so this was this was a risky shot for us and what I'm reporting back to you is that it is possible to build this kind of game and I don't think it fits an existing category I want to call them accessible games you know games that are gonna appeal to males females people from all walks of life multiple demographics and age groups

with go today based on third-party research polls that have been done we see more than 40% of the players are women we see underrepresented minorities in excess of 30% which is great that begins to approach what the world looks like in terms of those distributions so gamers not a subset of the world gamers are just they look like everybody else and we've seen very good long-term retention good good spend in terms of the revenue so it's possible to build broad games that keep people for a long period of time playing very intensively so mid core casual you know there's something in between I think so AR I want to tackle this head-on because I know you guys many people are probably curious about AR many people are thinking about starting a our projects and you know we've had some learnings here right AR in and of itself is not a magic

bullet for a hit game there is some real drawbacks to it the ergonomics are one of them we know that oops we know that people really don't like to hold up their phones for a long period of time we think the right session length for holding up your phone is not more than two to three minutes for example so if you think about a are you want to think about it as a type of play that happens within a broader game where a lot of the gameplay is not happening in a and in AR mode social stigma so when you hold up your phone and you wave it around it looks like you might be taking a picture of everybody who's standing around you this is a big drawback to a AR in certain situations and it's something that really needs to be designed around the other thing is just efficiency and by that I mean two things device efficiency so there's a power drain that one has to take into account and there's also just game play

game loop efficiency if people are power leveling through a game they're not maybe happy to take that extra time it takes to instantiate an AR session and play in that way so short sessions and really thoughtful design we think the right approach here is to design things for AR that really can't be done outside of AR so to try to think of game mechanics where you're in AR mode and it's obvious why you're there you're having experience that you can really only realize through through AR the other thing that I want to sort of flag here is you hear a lot of hype in the industry about augmented reality here from large tech companies other than us like the really really big tech companies you hear us talking about AR it's important to understand that when sometimes when we talk about it and certainly when some of the other companies talk about it really talking about the potential free R in devices that are that are going to come after the phone so there are inherent limitations to AR

in phones I for one don't believe that AR is gonna take over the phone and everybody's just gonna walk around in AR mode all the time it's definitely not going to happen but I do believe there's huge potential I believe that potential is going to come in future devices having said that there's huge incentive for all of us to try to learn about AR to learn what works and doesn't work to evolve the technology on mobile devices today so a lot of the interest you see from other tech companies that I won't name by name and Hoss is trying to figure out this technology and advance it and evolve it while this other kind of hardware is being developed so what's the killer feature for a real world AR game you know some people a set of us a lot and I don't think it's the AR I think AR is nice embellishment on top it's the icing on the cake the things that we think are really the killer features for games of

this type are the ones that we adopted there's our core principles that's why we adopted them but it is exploration and exercise and real world social interactions in terms of the you know I said earlier we had three million people come to events that we host Pokemon go players organize their own event once a month it's called community day we support it with a little live ops programming behind the scenes but there were over a hundred million players that participated in Community Day last year so intense interests in people getting together in real life and we you know we think there are reasons behind the exercise and the real world social that are pretty obvious both of those things make people feel better ultimately so there's a there's a feedback loop there that's that's driving that okay so that's the past all right now it's time to look towards the future so what what what is the interesting technology that we're working on and other people are working on what they are we're gonna get into the meat of that so the way I like

to characterize it is connecting the atoms and the bits so if a are sort of v1 is open your phone see a hologram maybe it is sort of situated more or less accurately in space it's blue to the ground doesn't really know where it is it's not really part of the world it's sort of a parlor trick it's a it's an illusion that's fun but there are limits to how far you can take that so AR v2 is about really connecting those bits the thing that we make in the computer with the atoms of the world and having those objects be grounded in specific locations being aware of those locations being shareable and being persistent so when we think about what's next there are three kind of key areas of technology development that we think are essential to realize that vision that I just described the first is mapping and I'm going to talk about each of these in more detail the second is

understanding or interpreting the flow of pixels that's coming into the device and the third is sharing the ability for all of us to have a common shared experience with AR none of these are really possible with today's technology so each of these three areas are new things that need to be invented so let me talk about mapping first I think this is it is the one that's near and dear to my heart it is a continuation in some ways of the work that we did on Google Earth and Google Maps but it's really different in some ways as well okay because we made Google Earth and Google Maps for all of you we made it for people it's designed to be a fun friendly accessible UI for human beings so that we can never be lost again we can navigate wherever we want in the world the AR maps that we're building our built for machines okay so it's a very different kind of map in some ways now I want to say when you say we're building maps or machines it

sounds like terminator or something like that it's in the service of helping us be better human beings and that's a really important thing for an antic I think you've seen I think you've seen some of our thinking around that is to allow us to better connect with the world and to have shared social experiences with one another but fundamentally we're building we're building these maps for computers and those computers are cell phones today those computers may be glasses in the future and by the way they may be robots beyond that because robots that want to move around in the world need that very same kind of precise localization so what the map does by accumulating data is to allow the computer to know X Y latitude and longitude exactly where it is in the world down to an order of centimeters that's what we need for AR preferably like single-digit centimeters and we want to know the sixth off orientation so we want to know exactly what that device is pointing at so that we can interpret and add to that scene so you sort of go into the next deeper level of

what does that look like you know it looks something like this so you start with lots of images in different ways those images might come into a system but from that you're gonna derive a very very detailed three-dimensional understanding of space and you have a bunch of points it almost looks like a solid physical object but that is a collection of data that lets us understand a specific location now you can glean from looking at this image this is a small statue lots of photos lots of points that's a lot of data I mean you think about think about the entire world like think about all the places that human beings can walk in the entire world if we want AR to work out of the box without having to remap re localize do a lot of work every time we want to use it and I think it's really really important this Maps got to be created ok and point number two is that at this scale the world is not static

like stuff changes all the time it's incredibly dynamic so this is a kind of map that it's probably a map that never gets finished really this isn't this is this is like a lawn to emo or as soon as you get done on one side you start over on the other side as if he continually rebuilt and refreshed and extended so in our view we think this kind of thing can really only be built in a cooperative way by people that are using a system so a collaborative ongoing effort to constantly map and remap the world in terms of a more concrete example of a system working that uses this technique I'll show you an example of a system from our research team and this will show you that evolution from photos to points to putting Holograms into the world so this is the Ferry Building and Downtown SF maybe you'll have a chance to visit if you're from out of town a really cool place so lots of photos are

needed from that we get what you might think of as a point cloud the points that you pick are really important once we have that we're able to put things into the environment and importantly because it's in the cloud you can come back the next day and see things and exactly the same location as you saw them on the first day with this technique we can put Holograms in place over very large areas indoors and outdoors and finally everybody who's having the AR experience is seeing the same thing so it's really as if the hologram is part of the world and not the sort of manifestation that you've created okay so that's mapping the second piece is understanding so point cloud there's a bunch of points if we want our games our characters the story that we're telling to interact with the world in a meaningful way we have to

imbue the system with a little bit of intelligence to know what's happening what objects are in the sea what are all of those pixels what are all of those points so that falls into an area that some people call segmentation we're talking about various techniques and there are multiple techniques I'm going to show you two involving deep learning neural networks to essentially decipher what we're looking at and that opens up all kinds of possibilities so we would think about a vocabulary for AR in this case if I want to put a vos on the table and I want to put Pikachu in the chair where he belongs with just the point cloud you can't really do that you don't know what it is you know the shapes you can put something on top of something or hide it behind something but you can't really do things that are semantically interesting so what neural networks enable us to do are they enable us to segment a scene like this and this is real a real screen

shot by the way this is a made-up artistic creation and lets us identify what's in that scene with a pretty high degree of probability so we know at a high confidence level that that's a monitor we know that's a chair we know this a table and so on so we can actually then respond to that and we can put Pikachu in the chair and a boss on the table even more importantly we can put combi with our flowers in the garden and we can put sie doch with his duck friends in the pond so that's one type of email that can be applied to a are that I think for us for game developers people want to tell stories is really important this is another this is another type so I talked about mapping the world and how hard that would be with all the stuff that changes some things actually change second-by-second so the idea of mapping them in advance is a non-sequitur people walking down the street cars driving down the street

you can't you don't create a map of that so that's something that you actually have to understand in real time if you want to augment the world in a way that's realistic so this is another piece of work that we're doing and this is a deep learning solution that basically can provide the depth of all the pixels in a scene in real time so rather than mapping in advance everything that's happening here we're deciphering depth in frankly a dark kind of a dark and mysterious way in terms of how these neural networks work but we can understand the depth of people as they walk in cars so that nicely complements the map piece so you can build a map of all the permanent structures the stuff that doesn't change and then for the really dynamic stuff of people moving around and cars moving around you can add this capability and it really is quite amazing to me as somebody that's worked

around computers for 25 years this kind of thing is even possible but it is the rate of change here is phenomenal that allows us to sort of fix AR in a way that it's really broken today so if you've used our products or others you will know that if you drop your Pikachu for photo mode into the scene and somebody walks in front of Pikachu or other things happen that illusion of reality gets broken pretty quickly whenever the character is not properly occluded behind things that it should be occluded by right so we want to see Pikachu ideally here be hidden behind objects and people when it should be that will help us believe that this hologram is actually part of the world of atoms the one that we live in so when we apply the system that I was just describing to the same situation we see that as people walk in front of Pikachu and his Pikachu moves around these planters that he's properly hidden and that's what we want we want to create believability we want to make these

atoms and bits indistinguishable from one another that's the ultimate goal okay so that's we talked about mapping and we talked about understanding the third kind of key piece of technology that I want to describe and show you some of our work around is around sharing so games are played between people games are inherently shared experiences those of you who are building console and PC games I'm sure have dealt with latency in a multiplayer situation it's really hard to share lots of data between users especially when you have a fast-paced game and things are changing all the time and you need to keep everybody updated and locked and we know that you know with online games sometimes we can cheat because you don't really know what that person is doing and so if you lag behind a few frames because there was a hiccup the network for a brief period of time you can sort of fudge it and then make up for it with AR that becomes a lot

more difficult because you actually see the atoms the people moving around you see other things that are real visually instantaneously and so the additive stuff the Holograms are you're adding into that environment need to track with a with atoms really closely or it's immediately obvious that you're out of sync and things are broken so you know why do we care about it we care about it because we want to make experiences like this better we want to have a truly shared immersive holographic AR experience so we've been working to figure this stuff out and we built this which I'll let one of our amazingly talented designers introduced David Holland and this is codenamed neon so what you're seeing here is high speed low latency shared AR so you have shared resources on the ground so think of those as pac-man pellets those are energy globules that you need to absorb in order to attack the other

players you'll see see people walking around so each person is being tracked in real time in this environment and this is all footage taken directly from device there's no CG applied here and you're seeing projectiles fired between the users so that's all being tracked na are all being shared to all the users at the same time it's really hard to do that right now there are just some laws of physics that get in the way you need around 10 milliseconds or so for that to be imperceptible okay and if you were building something today that has two round-trip to a server over the Internet you're talking about 100 or hundreds of milliseconds typically obviously a lot of folks are trying to get those Layton sees down for many applications including cloud gaming but it is a big problem that has to be designed around we found some ways to do some clever things so that we can do that on in many situations today including in some 4G LTE environments but it's it's not easy

this is another example so it doesn't have to be about shooting let's see Lena panting she's gonna is tun hinge she's gonna introduce us to this so this is a puzzle adventure II kind of shared a our experience that we also built just to kind of explore a different dimension of gameplay I was a huge mist fan so I particularly love this one so we'll see in a moment a hologram that everybody is sharing we actually cloaked the other players just to sort of add to the to the ambience that's really hard to do and there's some interesting work to be done to really close and quote people and in a more intricate way but this allows a complicated puzzle to be manipulated by all the users at the same time you have to talk to one another and synchronize your actions in order to unlock the same

and solve a puzzle so the whole idea is to give people reasons to talk to one another right I mean that's really the heart of social gaming you're creating a context we're giving people an excuse to be social you're making it easy for people to be social with one another [Music] so that's Stonehenge really hoping we get to turn that into a full-fledged game so on 4G networks you can make this stuff work but it's hard 5g for those of you guys that are you know read the tech press or into networking technology you know that there's a huge amount of hype about 5g is probably one of the most hyped technologies that's been around in the last several years a lot of people would say it's overhyped there are some real advantages for AR so

I'm personally grateful that we have 5g networks coming it's going to make AR much easier for us number one latencies are lower so you can actually get Layton sees between 5g and edge computing which is part of the overall paradigm that's rolling out you can get 10 20 second main Layton sees on a regular basis the other area it helps us is that 5g supports a far greater density of users in a given area so if you want to do mass outdoor AR gaming where you have lots of people playing the same space 5g is super helpful it actually supports by spec three orders of magnitude more people per essentially cell tower real-world performance may vary but nevertheless it's a much better architecture for sure they are so we are forming partnerships a lot of the companies that are building the networking infrastructure to make this possible to help them do it in a way that's going to be good for AR and to also really be on the cutting edge as this technology lands in consumers hands

so folks like twitch Telecom SK Telecom and Samsung are all instrumental in the backend systems that are going to enable 5g for consumers ok so we talked about history and we talked about the tech that's being built so now we get to the fun part ok so this technology exists what are the implications for the kinds of experiences that we can build for the kinds of games and non game experiences that we can build so I wanted to say something thoughtful about this and not just show a bunch of demos so broke it out into a couple of categories I really think about it in terms of two fundamental kinds of interactions that we can do we can add information and services to all of the atoms in the world so all the physical stuff can kind of work like the web like you can connect to information and you can make it do stuff through web services and then the other major area

is to is is to add things to the world okay and I'm gonna talk about each of these separately so in terms of information interactivity as I said it's about information and services and in terms of adding things I want to talk about those as augmentations just generally and here in the first case we're talking about connecting atoms to bits in the second case we're talking about making bits that can really fool us they can realistically masquerade as atoms so we want the bits to feel and act as much like real atoms as possible when we think about the first problem and what we might be able to do with it this is really the a our vision that we've been sold by sci-fi by movies this is about walking around with glasses and being able to know more about the world

so you might walk to the Ferry Building and ass up and you might learn about the history of that building and who designed it and why it looks the way it does right now you might walk up to a piece of artwork and understand okay who is the woman depicted in this statue you might walk by a restaurant and understand what Yelp has to say about about that particular restaurant maybe you're running late for a dental appointment you run it to a building it's got 20 floors you know your dentists is somewhere in some office you can know exactly where they are and how to get there so incredibly powerful that's just learning then there's a whole nother layer that we can do here which is to enable interactions so we can not only consume information all these things can become interactable so we can perform actions if we walk up to the bus stop and we see the bus rolling in and we don't have money on our card you can transact you don't have to go physical terminal I go to the airport you can check in when you arrive at your hotel you can check into the hotel think

of all the places in the world today where I mean we've tried to kind of do it by installing kiosks and things like that all of that can be just a surface that becomes an interactive surface for for for services now people have thought about what this might look like and one of the best-known of those is gun mkg Matsuda if you guys seen his work by the way the hyper reality film series I know many of you have I'm gonna play about a minute of this just to remind us of it it's interesting let's just take this in this is a possible future that we might live in I already feel stressed and nervous this Pony so your proof is

already stored in the medical staff you know I don't care for another DNA scan [Music] so the system knows where we are we're gonna find out in a second that we do you know when to get off the bus [Music] let's look at kiddo healed but how long do I know this [Music] we'll be on you that looks helpful do three times when I might like to see a path [Music]

it's really crazy as this part where we should go she's going outside now the world is plastered with ads and distractions so it doesn't have to be like this I had a chance to meet actually I was at a talk it was kind of like this I was talking about hyper reality and I was talking about hei Matsuda and how smart he was and this guy on the side of the room hello businesses I'm Katy so I had a chance to meet him and I learned about his work so he's not showing us that because he wants the world to look that way he actually follows something that's called there's a school of design called critical design and the whole idea there is that you make things you design things to question certain things in the world and so really he was very deliberately challenging what the world might look like if sort of current trajectories continue into the future

and we build they are not in a very thoughtful or considered way and it's a world where you're tracked everywhere you go where giant companies know everything about you your identity is constantly at stake and the world itself is noisy and busy and literally plastered with distractions so that's out there and some people would say hey AR is a bad thing because we've seen this vision of how bad it can be the point I want to make to you all is that it doesn't have to be that way and it's up to us to make it the way we want it to be and there are some incredibly powerful things that you can do so I don't think we should be scared by Katie's vision I think we should be informed by it and actually I know he's continued to work in the field of AR after that and he's helping do some of the work to really define how AR should work but I imagine a future where with the kind of map that we described and with the kind of understanding that we

describe that we can have experiences like this where we can learn about the world in a deeper way and we can access things like I think like riding public transit I try to ride the subway or the trains or the buses in every city I go to and a trouble fair amount around the world and it's it's like the most complicated adventure game that you've ever played trying to figure out how to buy a ticket and navigate through the terminal that can all be so much easier with a R and that would be a social good if more people rode public transportation because they could understand it it helped them get places and a more reliable and easier to access way than be fantastic and it's one of the things that I think AR can do for us so just to address some of the you know if the more technically minded people out there might want to challenge it for you assumptions here you might say hey you know you can do some of that stuff with an app today like they're a bunch of apps in the App Store and that's true but it was a huge amount of work on you as a user you're kind of in-between you're the glue between the world and the digital world the atoms and the bits so you have to go find the app install it all the apps work differently I for

one know that I'm in venues all the time they say hey use our app and I don't look I don't download their app because already have a lot of apps on my phone and it takes a while and it's complicated so I don't think that really is the solution the other sort of counterpoint would be well maybe you don't need this kind of complicated AR cloud that you were describing to do this maybe there are other ways to do it like you can have QR codes on things or you could have Bluetooth beacons that tell you where you are and what's going on you know I don't think we really want to put a QR code on every object in the entire world I think that would be kind of hard to do and you know would add a little bit of pollution to the world I think that would that would be a bad thing it's not really a very practical idea Bluetooth beacons are even more expensive and more difficult to deploy as actually many companies have tried to deploy large networks of Bluetooth beacons and it's expensive and hard so I don't think that's really the answer either even like raw computer vision

we're all computer vision in some cases can tell you hey this is a Starbucks or this is a bus station but has a very difficult time knowing that this is this specific bus station this is this specific park bench a specific anything because it's trained to recognize patterns and specific things are hard to pick out from all the things that look alike frankly so the AR cloud is essential even if you have good good computer vision so let's move on to the second class of things and I have some examples I want to show you here too so we're talking about augmentations we're talking about putting things in the world that look like us or they look like creatures they look like guides so these are things that might entertain us they might do useful things for us or they might even help social interactions between people when we think about games you think about Pokemon you think about adventure games you think about putting fictional characters into the world it's

kind of an obvious thing that we want to do with the technology but if you tried to do it right now you might run up against barriers that you weren't expecting like you know one thing that I can imagine is okay imagine that we want to have this great noir experience in San Francisco we want to be Humphrey Bogart we want to go through the city on an adventure we might meet a hologram so imagine that there's a hologram standing down on the waterfront of San Francisco if that hologram doesn't know where it is if it's just something that you popped up in your phone but it's the same experience that could happen anywhere there are limits to how sophisticated that interaction can be but if that hologram nose is standing on a specific street corner in San Francisco and it can direct you a half mile away to Coit Tower on a quest to find a clue that will lead you on the path of the Maltese Falcon that would be incredible but to do that we have to have Holograms we have to have augmentations that have contextual awareness we need the AR map

for that enable for these things to know exactly where they are in the world we are the other thing that it gives us and we talked about it earlier after rebuild O's but again is that it allows us all to share that experience so it's not you know if only you can see something there's a word for that it's called a loose ination I think sometimes that's a bad thing but if we can all see it it's an amazing shared experience which we all talk about so the sharing part that everybody's saying the same thing part this is really really important we've been about this Niantic our designers and artists have said we want to build we imagine our festivals with real-life gems and pokey stops if everybody can see and understand together even going back to the early days of ingress we were thinking about this more recently we're thinking about non-game applications and what these kinds of holograms can do the technology is not

there yet but I think you saw from the preceding section that we are getting closer this is an example of one of those baby steps forward so this is a project that we did with our colleagues and partners Softbank in Japan and it's done on Holland's one not Holland's two and we actually captured this footage at the top of Roppongi Hills I don't know if anybody's been to Tokyo but probably some of you have and you know the big skyscrapers so you're when you see this imagine you're on the top floor of the skyscraper you know we're gonna look out the window in a second you'll understand why so as we look around the room we're seeing and everybody who comes through the experience we ran a bunch of people through this everybody saw the same things in the same location so they were shared experiences as I was talking about we had specific Pokemon inhabiting certain parts of this pretty open room

and then when you walked out to the window and looked out across the city you could actually see all the pokemon gems where they're supposed to be we can see our legendary birds flying through the air I think Moltres is in the background so it was really exciting for us because was one of those glimpses of the future you know you can only do this in a limited way you can prototype under certain conditions but it is is a glimpse of what's going to be possible in the not-too-distant future okay so I'm gonna wrap this up I think we're getting close to time so we've talked about AR and we've talked about the history and the things that we've learned we've talked about the technology that we're building and last we talked about some of the product potential and I think for you product designers and game creators out there your mind is you know you probably have a thousand ideas of stuff that you

would like to build so whenever people ask that question well where is AR in the sort of hype cycle or whatever you know is it op is it down I don't really care about that I you know the question that's interesting is is it something that is meaningful with a capital M like is it something that's going to affect our lives in a significant way and many people's lives in a significant way and I absolutely believe that it is you know what is happening on handsets today is a warm-up that's the pregame what's gonna happen with glasses in the future is the real deal but I think it's one of those platform shifts you see it once every 10 or 20 years you know and I've lived through a couple we saw pcs we saw the cloud we saw mobile I think AR is one of those shifts so it's something that I think is worth our time to understand it's worth our time and money to invest

early in and I think it's going to be hugely important rose-colored glasses I want to touch on that that was part of the title of the talk people have knocked a are because they've said oh I didn't switch my slide to the beautiful closing slider perfect rose-colored glasses so some people have said hey girls like rose colored glasses you're not you're taking away from the world people aren't seeing the world as it really is so it's a bad thing I want to push back on that a little bit you know a good pair of sunglasses if it's right outside they actually don't get in your way of seeing the world they help you see the world more clearly they eliminate the glare you can focus you can see things you couldn't otherwise see and that's how I think of AR we're adding things to the world but the things that we can add can deepen our appreciation for the world and our understanding of the world I think there are two primary things that that they are can can do for us one is

it can allow us to see the world in a fresh way by causing us to pause and seeing history everything that's interesting about the world brought forward to us I really think there's an opportunity to deepen our connection with the places that we live the second thing that I think is incredibly important about that is that it allows us to do it in a shared way and to have experiences together to have these adventures in real life with our friends with our family and with our kids and by deepening our connection with community in a physical sense and deepening our connection with community in the meeting our neighbors and doing stuff together with our friends sense I think is a really important step forward for technology and in frankly I think it's something that we need I think it's something that the world needs so you've seen how it can go wrong so thank you khe for showing us how horribly wrong they are could go but I believe that the positives here outweigh that and you know I want to put it to everybody here

you know I there's a Gandhi quote which is be the change you want to see in the world in our professional lives I think the quote is it is really up to us to build the future that we want to live in right it's up to us to build the future that we want to live in that we want to exist and more importantly it's up to us to build a future that we want our kids to live in so if you guys are inspired by the opportunity here in Austin AR reach out to us we have a contest we're looking for allies but there's a lot of work to do and I think it's important work thank you [Applause]