Niantic's AR mapping lead on using Earth as an AR gameboard

[Music] [Music] hi everybody I'm Ross Feynman I'm from Niantic labs I'm gonna be talking about treating earth as an augmented reality game board so one of the nice things about coming to these events rather than going to like some of the computer vision mapping ones is most people here have heard of Niantic most people refer to us as the Pokemon go company so a little bit of my background just to kind of get to you all situated so I was recently the co-founder and CEO of Usher reality we were founded in 2016 on this was well before like augmented reality was cool before Pokemon go came out this was when actually even before oculus

came out and so then we were working on doing 3d mapping on devices we went through Y Combinator last summer we were venture backed and then through a crazy fall if anyone here in the augment aerospace knows the AR cloud kind of buzzword was created we were one of the companies working on that and then we were acquired by an antic at the end of last year so I lead up the 3d mapping efforts on for augmented reality so have you think about this in the gaming context that's kind of building up the game board that a lot of the players can use off of it so a little bit of background on Niantic because most people know about Pokemon go first ingress second and not much else after that so it was founded in 2010 actually the former Google Maps team so they built like Google Earth Street view an entire geo team on so then they spun out like how can use these style of maps to kind of view the world differently and have experiences with them Google then spun them out with the reorganization

for alphabet I think that they regretted that shortly after 2016 on and some of the games that have been developed so then there's anger aspo kamongo I've been told that marketing people all that I can say officially is Harry Potter has been announced also one of the people one of the things that people don't know is actually a field trip which wasn't fully a game but an experience that people could kind of explore the world around them and kind of learn about say the Wikipedia articles about what are the different places and different things to kind of explore and expand the world so it's kind of a stepping stone to first get information about the world and have an experience in the world and have a worldwide experience in the world and continuing that on with Harry Potter now it's kind of sets the stage for the background now I want to talk more about oh and world's largest pay our user base right now so we have a very large slice of a small pie and then we're hoping that the general pie grows much bigger over time but the goal of this talk

because I want to talk about interactive augmented reality so how can you use the earth as a game board in a sense how can you begin to interact with it and then how to think about doing like planets scale augmented reality so what changes when you're thinking about games that are kind of in the real world where people are running around and you have tens of millions hundreds of millions in our case of people exploring in the world and kind of doing things so whatever kind of frameworks much to think about that because mented reality is in the very very early days of it so think about it's like where mobile was in 2007 it was kind of the first releases of like simple games and it took a while for people to begin to figure out what are the mechanics so if you get into this one of the big questions that I always like to start with is what is actually new about augmented reality from a gaming perspective so what are the tools in which you can use so I've found - first is that the position of the device

matters so like you can view things in different angles if you have a headset on you can walk around you can dodge things on so that's actually where a lot of development and augmented reality is today it's kind of using the device positioning kind of shooting at stuff moving around it's on the simpler side but it's a good step forward and then what I get really excited about where I view the future of gaming augmented reality is is the real world becomes content so how can you have games that begin to interact with explore and take advantage of the real world so for an example like using the device position like there's this demo that came out I think last June or July from a our kid after Apple's announcement so here you can actually use the phone as a paintbrush in the real world for my own company we developed a quick concept of like a sure sketch for extra reality on and then this kind of blew up and I think is a really nice application of

how you can use kind of the position of the device to create an experience so then people would draw things in the world and then explore and like take videos of them so but what I want to focus on is for this talk the real world becoming content so I view a lot of the stuff that's being explored right now is the position of the Vice but then there's so much more potential and what people can do for the real world so for this I kind of break it down into four levels of interactivity so if you think about like the first level is overlaying so it's kind of digital content as displayed over the camera and view so this is actually what the first version of Pokemon Gil was if everyone knew that it was come the most basic version of augmented reality and like it doesn't quite live in the real world but you can do kind of interesting cool things with it so the number of pictures and videos that went around with kind of staging all this after Pokemon go released is I can't share with you some of them on no

one should see those but it begins to get kind of digital content and it fulfills the fantasy for people um and that's really what kind of a lot of gaming is is what is the imagination what is kind of a hook that brings them in but as we get into more like longer term sustainable interactions we need to think about kind of anchoring in the real world which is where the current state of technology is today digital content is really connected to the real world you can walk around it you can explore it this is like you draw things you can leave it there there's a lot of really wonderful new technology that's being developed in our space in terms of having this be persistent having it be seen by multiple people kind of having it really live in there rather than just for one individual session so that's kind of on the spectrum at the moment so one of the things that I get start to get really excited about is as you begin to get into interacting so and take a

little bit a moment here is how do you bring the physical world into the digital content and how do you begin to have them mesh seamlessly so this is I view really exciting concepts this was a demo we did for my company before we joined Niantic and can't share any I I can have a lot of freedom to share the stuff that I've shown before for my company there's exciting new stuff that will kind of wrap up nice romantic but in this case it was using colors as a resource for a game development so then you could actually mine the colors out of the chairs and actually over time the you could set a shader over it and then it would take the color out and turn it slightly more gray over time then you'd have to move your mind around then the color would come back so then if you think about this this is taking in on the kind of the colors of the real world and then it changes the game strategy and the dynamic based off of the location

that's a very simple thing to say but very complex in terms of the game mechanics because okay if you want more gold you can put up more post-it notes and then you can imagine people like rearranging their living rooms or like okay I need to have like a pink resource and they paint all their colors pink and I like oh my god I need more blue and then they paint one wall blue and one wall pink you can imagine the kind of really excited people that we run into a lot of really excited people at Niantic and ingress they're very enthusiastic people but then here you can imagine how do you bring in like the objects of the of the real world so then this is kind of an interesting concept because if you think about gaming before it was really what can people develop on their own sorry what can developers develop on their own and then the longer-term experiences are not just okay what is the content that has been created but what are the people that you're playing with so say that you're playing Halo like the first time that you play the

game you go through and it's like okay you spend like tens of hours kind of going through it you can play that maybe four times and discover other Nicks and crannies but then if you go into like multiplayer systems where it's kind of connecting people across scope and it changes from what is the content of the game to what is the strategy associated with other players now for augmented reality you can also include other players with a lot of the new technology that's being developed but you can also kind of think about the real world as content so then if you can bring in like the colors the objects the environment as a whole into the game which a lot of Technology and the infrastructure is being developed by other companies Google made some wonderful announcements then you can actually have kind of two sources both other people and the environment that you're in so if you think about like tower defense in your living room and then you need to like rearrange the cushions and then you need

to put up digital turrets and everything like all this stuff can really affect how people play in the games so I think this is a very important step and this is actually like colors as a resource this is available like everyone can use it there hasn't been that much development but it's one of the things that I'm really excited about and then I view like where this is going longer term in the future it's kind of affecting so then how can digital content affect the real world so this actually came out of demo that we were doing goodness about two years ago now where we had like this digital cat like running around in the real world and then people's okay can the cat move the cup and then the cup was real the cat was digital and it was like no it's you can't change remove things in the real world it's just the camera but it kind of fulfilled the fantasy enough that people kind of expected it and when you like mention that no it's the cup is real the cats not then be like oh okay

so it's actually a sign of like people believing the experience but afterwards we're like oh how could you actually move the cup and then because you have control the camera feed you can do kind of interesting things with it so I actually view in the longer term you can actually like use a lot of very complicated systems to kind of bring in the content of the real world and begin to like makes like modifications to it begin to change it so then you can like actually end up with a matrix style thing where you can begin changing the real world and to show that this is impossible with kind of older technology and other things that there's still a lot of development here not saying that any of this is available right now but just kind of how it's going is this is actually from SIGGRAPH and 2014 wonderful paper I forget who the author is already give them full credit but its kind of offers a telekinesis mode where you if you have a model of the origami you can actually from a single image kind of fill in the

back and then have things fly around so imagine that if you could do that with like your cup can become something and then walk around on the table and then like quickly move back and you have these little experiences so like this is still in the in a future technology but this is at least view like the end of the spectrum of what is capable for interaction and then the cool thing about all this on at least from our standpoint what I hope many other people get to is doing this at what we call a giant I antics scale on so how can you do this for kind of cities countries continents and like the entire globe so then what changes when you can have say some gamers playing in one place and another place and other gamers playing in another and how can you begin to connect them together because you have knowledge about these different environments so again sadly the example that comes to mind is a Harry Potter one I've been told I can just say that's announced but there's kind of cool

things that you could imagine connecting different parts of the world and relating different objects together and kind of bringing this at a very very large scale so then people can feel connected not just in the same location but across locations and then you can have interactions at that level so summarized on this and love to take questions afterwards on I believe that the new game board is not digital it's not in our imaginations it is the real world and then we can add on new layers on top of it on so we are just at the very very beginning of augmented reality it's kind of one of these really exciting creative opportunities and since that a lot of these questions and how it will develop they're just unknown so we heard about like the kind of amazing growth about mobile like for augmented reality this is just like we're scratching the surface of what it can be because a lot of the tools and even the creative like what do you do with it are still being explored at the

moment and actually there's a lot of really wonderful like indie developers small studios larger studios like us that are really taking heart looks at how to define the space and look into it there's new differentiated AR mechanics so in the mobile space even though mobile AR is in mobile people you you can have things that very much standout just because of the nature of the game it's a new medium it's not saying to create like candy crush is addictive it's a game as it is but like that ney R doesn't make sense but what are the new mechanics that you can begin to do in AR there's new technical ones the position of the phone and the real world becoming content but like what are the creative mechanics that people can develop off of that and then lastly like I believe that games can and should interact in the real world so we found with pokemons go and even ingress and do the lesser extent field trip before like when you get people out there and like exploring in the real world

on I actually met my girlfriend based off the plane not fully playing but talking about pokemons go and everything else but no it's this really wonderful experience that has kind of natural viral effects if you kind of people are rushing off the bus and then you kind of there's wonderful videos of people in like Tokyo like a mob of them and taking over the streets which is terribly unsafe and we felt terrible about that but like they're like a thousand people running through the streets like because rare Pokemon will stop there like there's a lot of wonderful social interactions that come out of it so there's kind of these new areas that not even we knew when we were getting into this that I think have so much more opportunity now with that I'd like to thank you all and opens up for questions was great I think it's whole area like you said this whole this idea of how much we've yet to find out about what's possible its kind of key to me and you know what I find really fascinating like you say the world is as a global level

is now our playing field and the amount of how we collect data for that how we share information how we can leverage existing pieces you know how much of a challenge I mean obviously you guys got huge amounts of work on this but for people who have maybe coming in as an India how do you think you know the indie communities gonna be able to take advantage of that kind of work that you can you guys or other companies are doing to make it possible to make the world a playful space so I think that there's two things one is in terms of I'm encouraging Niantic to share more externally like the lessons learned associated with like okay what how do you deal with like hundreds of millions of people and then kind of games on that level because there are a lot of the questions out there and then for so I think it's just getting more of a community around the space so that having people talking and exchanging ideas because on there are these like little niche groups I found all over the world that are exploring different augmented reality

aspects and then just connecting that and saying hey you should you two should talk on then it ends up getting a good connection in there and I just view the AR spaces so early at the moment that there hasn't really been on this kind of condensing of ideas we're trying to start to nurture that at the moment but from a technological standpoint um there's a little kind of lot of data associated with it like based off lessons learned from Pokemon go and everything else like we're kind of opening up a our own like kind of real world platform I'm associated with this so then people can say like okay well this worked for all the Niantic games and then canvas work for other studios and then offering that as an open service independent of any of the other systems on top of it yeah yes - again if I'm guessing again I think you covered this it's this whole idea that there's actually a whole bunch of layers going on here there's the materiality part yeah which is fantastic there's a location part there's the big data part

there's the social part or all of which I've heard different ways of looking at experience that my favorite all-time location well moving location type of experience was actually a game done in like 2006 I think it was it was called triangular oh I think I may have made you know basically the idea is you had you know ten people you know five people in each team and they had to make a triangle it was 150 meters on each edge and when you managed to make that triangle you could press a button and capture the enemies who were inside that triangle this is completely insane but it played like paintball and I mean no commercial model but I think not here at the time we're still a thing and they bought it but it's the fact that you can see Pokemon at one end and you can see try are another end and that's not just a spectrum there's a whole array of potential new concepts I think oh yeah totally and even like ingress people were one

person flew from Europe to Hawaii to like complete a mission with the team and everything else and like that's dedication to duty totally I mean I've got a texter who takes me to all the airports every time I fly a lot and he's a massive massive man Pokemon fan and he and his nephew spend every weekend getting together with clustered groups I'm using some of these forums and they just go and do missions together and it's so it's incredible a social experience I'm Hawking because I like talking about that stuff anyone got any questions anyone who saw something different yeah okay for my question I was like asking if I remember a brutality became a norm in like marketing so I'll say if like coca-cola would get you hold your phone over coca-cola something would happen and then you would have a chance to win or something

would that drown out like the the wonders of having argument or talent around reality or do it like personally I don't think so on so there's kind of been different marketing campaigns throughout the years like some of the marker based on systems like that so in terms of those like having a digital avatar II kind of like run around in different commercials in a sense like kind of ruin the snapchat experience there bit mo G's in the sense I don't think so so I think as long as it's you have a mechanic associated with it on that is more in the gaming space then you can fairly differentiate and differentiate them cool and on that note Ross thank you very much thank you [Applause] [Music]

[Music]