Three Years of Tilt Brush

Um, all right. Hey, thanks for coming to our talk. My name is Patrick Hackett. This is Drew Skillman. And uh, our talk is called three years of Tilt Brush. So, uh, really quick, if you're unfamiliar with Tilt Brush, it's a virtual reality painting application. It's currently available for the HTC Vive and the Oculus Rift with touch controllers. Um, Drew and I created Tilt Brush three years ago and for the next about 45 minutes, we're going to talk about the point up until now. So, when we first started uh, writing this talk, what we did is we we put together like a timeline. Like we we laid out a timeline, took a bunch of events, put them in chronological order. And then as we went through them, the story that came out of it, what we realized is that uh, it's just two naive game developers applying game development principles to a whole bunch of situations that were not video games.

And sometimes, a lot of times, it was very beneficial for us to think in this way. And then there was other times where it was clear that like we didn't have a clue what we were doing. Um, I just wanted to pause for a moment. As a show of hands, like who would identify in this room as a game developer? Awesome. That's a that's a great mix. Okay, but really the important part and really our hope of what uh, you take out of this talk is that game development principles work outside of games. And again, this talk is going to cover the past about three years. So, starting mostly in August 2014 and then leading up till now, September 2017, and we put in this this uh, timeline right at the bottom and we'll put dots on it so you can kind of follow along. Awesome. Uh, so before we get to Tilt Brush, um I

want to talk about some of the backstory and uh give you guys some context on where we're coming from. Patrick and I, we have our indie roots uh really come from a game studio called Double Fine. That's uh not far from here. A lot of you may have heard of it. Um we worked together there for probably 8 years. Um and that's really the story where the story begins. Uh we stumbled into a project for the Kinect called Kinect Party. Later became or actually it was Happy Action Theater then it later became Kinect Party. And that that without we didn't realize it at the time, but that uh it was a magical moment where we kind of got pulled into this uh this awesome world of mixing technology and art and game development and trying to push on new technologies uh in in just innovative new ways. We got so excited about it, we decided to uh formalize these efforts. We invented a name for ourselves. We called ourselves the Department of Future Tech, which was kind of a joke, but also really serious. Um and we tried to get as much cutting-edge hardware as we

could. Um you can see the Gamasutra article we wrote was really just like begging people to give us their crazy VR gear, their depth cameras, their AR gear. We just wanted it all. And we also jumped on the DK1 Oculus Kickstarter. Uh we immediately started doing late-night game jams at work with, you know, the two of us and a bunch of other people from Double Fine. Here's Tim Schafer, the head of that studio, checking out uh just random prototype we put together called Moon. Here's a Razer Hydra duct-taped to a Nerf gun. And this I think is uh one of the first times that that a bunch of us got to try 6° of freedom tracking for the first time. Albeit, you know, very mediocre 6° of freedom tracking. So, that's rotational plus positional. And this is Paul Debevec, it's pretty dark uh GIF, but that's Paul Paul Debevec, co-worker at Double Fine, and he's on the Tilt Brush team now. And you can see he's got some Hydras

duct taped to his head. Actually, you probably can't see, but take my word for it. So, we're we're just playing around with all of these new toys when suddenly there's a huge shift in the industry. Can't really like it caught us by surprise. Facebook bought Oculus for $2 billion. This was the first signal that things were happening. Like it was not just like late night game jams anymore. And around the same time we were incredibly lucky to get to try out Valve's Tag Room. It was, you know, absolutely, you know, vastly beyond any of our expectations. I mean, anyone who's had a chance to try it knows how pivotal that moment was and how transformative it was for like the whole industry, and we're all here largely because of that Tag Room. And around the same time, a co-worker at Double Fine, Lydia Choi, left to team up with ex-Bungie dev Brian Sharp to build a virtual reality sculpting tool. So, over the course of just a few weeks, it became super clear to us that there

was a VR train, and it was leaving the station, and we needed to be on it. So, that's what we did. Um first lesson in being naive, the train didn't leave for a little bit longer. But, we formed a company, called it Skillman & Hackett. Clearly influenced by watching too much Mad Men at the time. And we got to work immediately. We weren't we weren't really doing games. We had this motto of not necessarily games. We were doing a lot of the same stuff, though. It was just, you know, a lot less people. In fact, we were working down the block. We were a block away working out of Drew's apartment. And really at the time there was just a ton of room to explore. We picked up some contract work to pay the bills, and then really just spent our remaining time mashing different pieces of hardware together. In general, at the time we were using the Kinect, Kinect 2, uh DK2, Hydra controllers, Leap Motion, cell

phones, and then really just using Unity to glue all this stuff together. So, there's some random examples of of like prototypes, things we were making at the time. We did a series of music music visualizers for what would eventually become the Gear VR. We worked with Leap Motion. At the time they had a uh an update to their SDK coming out that had greatly improved hand tracking. And so, we we built a couple of like showpiece demos for them. This is another one with Leap Motion, but this was a uh this is like VR in Leap Motion. So, it was um you know, taking the Leap Motion and mounting it to the front of the DK2. This was a user interface mock-up for how you could manipulate a map in VR. Another project we did was a collaboration with Tyler Hurd. Uh we'd worked with Tyler at at Double Fine previously. He was now independent. He made this short called Butts, and we helped him bring it into VR. And uh and yeah, so that brings us to

the point in the story where we we uh really stumble across virtual reality painting. Uh here, this is one of the earliest GIFs we have of of the prototype that would eventually become Tilt Brush. And arriving uh arriving at this prototype was really to quote the great Bob Ross, it was the result of some some very happy accidents. So, uh I'm going to describe to you the prototype that we were trying to make, and then share a couple of the happy accidents that led us to virtual reality painting. Um so, this prototype, the idea was it would be this I should It's little more background. This was over the course of like probably one day. So, this was not a serious prototype. This was just like a what might happen thing. The idea was uh take a chess game, use an iPad as input, but you're wearing a DK2 headset. And then if you can track the position of the iPad in 3D space,

then you can use it as input uh and you could reach out and touch in VR and move chess pieces around. So, the chess pieces could be rendered in virtual reality coming off of the iPad. You could reach out, move stuff around. At least that was the idea. Uh the first happy accident is that it's way easier to use a Wacom tablet as input for like a Unity experiment than it is an iPad. It's like, "Oh, the Wacom just behaves like a mouse. You're done." So, in the spirit of working fast, we substituted the iPad for a Wacom. And the second happy accident uh is that I added debug trail rendering lines to the chess pieces because they were of course broken and behaving in all these weird ways. So, now I've got So, now I've got a stylus and I'm moving chess pieces around, but the chess pieces are leaving these ribbons in the air in that perfect sweet spot. You guys if you've done, you know, like positional uh you know, full six off VR tracking, you know that there's this there's this sweet spot right in front of you where it just feels absolutely

amazing. And that's exactly where I was drawing. And we had, you know, these are the same ribbons that we've been using for the last whatever 5 6 years on every single uh game we made and every single prototype we made. It was all using just ribbon rendering and visual effects and particles and layering all the stuff on top of it. And then the last point is that you could rotate this chessboard and then move the chess pieces around. And so, now you really were uh drawing in 3D space, but absolutely uh by accident. Bob Ross would be very, I think, proud, I like to think. And really one of the reasons that I'm such a proponent of rapid prototyping is that they they really do let you um find these happy accidents. And and also, if you do a bunch of prototypes, it can help develop your intuition, which is a point I want to stop on for a moment. Rapid prototyping, it builds your intuition. So, we'd been working as fast as we could for the last 4 months, and we'd done a ton of prototypes, and that gave us, you know, that gave us the perspective so that when we saw this one, we knew it was special. We knew it needed to be pursued.

Question is, what next? So, we have a prototype we're really excited about. Where is it going to go? And there were a few a few things we had going for us in terms of providing context for for where we could take this. And the first was we had a music visualizer UX experiment that Patrick had made from much earlier in the summer. And it it explored keyboard mouse input with the DK2 headset. Now, back in these days, there was no VR controller, like those didn't exist yet. So, keyboard mouse was really the only thing you had to work with. And this experience was really compelling. Um it was it was kind of surprising, but it worked it worked remarkably well, and this UI would become kind of the signature Tilt Brush UI. And the second thing we had was a really cool additive kind of particle bloom-based style that was built up, I mentioned from from many years of of experimentation. And we'd done enough prototyping in VR with it to know that it would work really well. It was like a very very obvious fit. Just take all the work we'd

been doing and slap it on the lines that were coming from this Wacom stylus. And third, we were still super good friends with the team that had left Double Fine to work on that virtual reality sculpting tool, and now that tool is maturing into Oculus Medium. And everyone's simmering in the same creative juices. You can see we're all lined up around Brannan Street, just a few blocks away from here. Um and so so we knew that there was a precedent being set for VR art tools. And And helped us connect the dots along with the ribbon rendering and along with you know the other work we've been doing so it started to paint a picture of where we might be able to take this. We we still had a lot of questions over whether painting would you know be viable as a form of VR content creation but we knew we wanted to give it a shot. Right okay so Drew throws this prototype over the wall at the time it's literally named VR chess.zip and I I say over the wall because the dirty secret of Skillman and Hackett was

at that time we didn't have source control and so we're passing this thing back and forth. In this GIF the first thing that I added you can see the first glimpse of the UI it's fixed flat in front of you there was a color picker some different brushes. So we continued to work on this for a couple of days and we got a very well timed email from Johnny Ross. If you don't know Johnny he's the co-founder of Mindshow which is a really great storytelling VR application. We'd met Johnny at an early San Francisco VR meetup exchange contact info. At the time he was running VR LA and he contacted us because he had an idea for another event. He was setting up the Proto Awards which was going to be the first virtual reality award show. He wanted it to coincide with Oculus Connect 1 reaching out to people he knew to see if he could dig up some entries. Luckily for him we did have something we wanted to submit and unluckily for us

really what we had was a two-day old prototype and the conference was in three weeks. So these kinds of forced deadlines were you know common to us from games and to be honest better or worse they're hell of a motivator. We really thought that the proper unveiling of Tilt Brush could be at Oculus Connect with the submission to the award show maybe we win an award and so to do that we wanted to put together all the materials of a proper product. So, we needed in 3 weeks a name, a logo, a website, a trailer, and we actually had to get this thing to a point that we could submit it to an award show. So, we got to work. All right. Names are hard in general. Uh these are some chat snippets I I dug up and uh we had a lot of bad names. Virtua Sketch, Super Sketch Box. Uh I remembered looking through this that at one point I woke up and I was I

was positive. I came up with the perfect name. I was so excited. Told Drew it should definitely be called SketchUp. Drew reminded me is a very good application that already exists. Uh we liked Tilt Brush initially when we came up with it, but honestly we thought it was a little bit too limiting because we recognized that this application could be much more than painting. For the logo we worked with Drew's sister and brother-in-law who run a graphic design studio outside of Seattle. They had already done the logo for Skillman and Hackett. Uh and we whatever, we bought them a really nice bottle of whiskey to kind of expedite the order through. When I was looking through old email I thought it was funny that I was concerned the colors in that first logo were a little too Google. Uh and also you could see that I was I got pretty excited when I actually saw the one that would become our logo. And really I apologize for the cursing

in these slides. Like we didn't really expect to be presenting this stuff ever. Um we weren't trying to come up with a slogan, but uh Drew came up with a slogan one time and so we mocked it up to see, you know, how it looked. Looked pretty good to us. Couldn't use it. Uh and then we got a trailer cut. And you like was this was made for the DK2. It was centered around keyboard and mouse. You can see the old UI in this video if you look closely. It was based on you know sitting in a desk and so the UI was this cockpit experience around you. It's fundamentally different from what you would experience if you tried this on the Rift or the Vive. And that was it. All the pieces done, ready for OC1. So yeah, as Patrick said we we got it together name, logo, website, trailer and an actual playable prototype as opposed to the two-day-old experiment

that we had and we jump on a plane and we head to Oculus Connect 1. The first thing we do is we set up camp in the demo area and we skip every talk and we just start putting every single person we can find through this hot off the presses demo. And a bunch of things were really special about that first Oculus Connect. There's an electricity in the air. Anyone who was there knows that it was just like a very special time. But there are two connections in particular that were critical to the future of Skillman and Hackett and and also to Tilt Brush although we didn't know it at the time. First was Chet Faliszek who I'm sure many of you know. We tried to track him down and did and and had to convince him to let us put it on Steam. You know for those not familiar, Chet has had this huge role getting the whole VR industry off the ground and he's definitely a big part of why Tilt Brush exists today. Second connection we made was Andre Doronichev. He's a product manager at Google and I think Tilt Brush would have existed without Andre but it never would have become part of Google and it never

would have received the global recognition that followed after after we joined. Okay. So we weren't actually just there to play test and put it in front of people. We were there for the Proto Awards. That was the big motivator and the and the source of this this kind of deadline that we imposed our on ourselves. And Tilt Brush had been nominated for best GUI, which ended up being the first award that was announced, uh and we won. Uh but we were woefully unprepared. Uh I think I remember I remember being shocked and surprised and going up there with Patrick. I think that I swore and Patrick thanked his mom or something. Uh but it's it's worth really uh calling out that what this ended up being for us was one big playtest. Um you know, the importance of playtesting is one of the most important and critical lessons we took away from our time at Double Fine, and we were paying super close attention to the expression of like every single person's face after they tried the demo, um looking for responses, trying to see what people liked. And at the Proto Awards, you know, we saw that Tilt Brush resonated with the jury.

On social media, it it blew up. Um you can see that un- unexpectedly we'd left this experiment slogan experimental slogan somewhere on our website, and it was getting scraped by social media. We felt embarrassed about that. Uh the launch video hit 100,000 views within a few days, and so like the results of this uh this public playtest were clear. The world was interested in virtual reality painting, and uh we wanted to give them more of it. It's also worth calling out that this was the moment when we really stopped being a rapid pro- prototyping team and pivoted to become a true product team. All right. So, now we're home from OC1, uh very excited about the positive response, itching to get back to work. This point we're in about start of October 2014. We're moving really fast, adding a bunch of features, features, features. Uh these GIFs show off a bunch of different brushes that Drew was experimenting with at the time. And we started really having long

conversations about what a brush could be. Um we proved that, you know, ribbons were satisfying, but what if brushes were particles? What if they were light? What if they were sound or models or behaviors? Now, we started adding features like the ability to export animated GIFs, to import models, real-time lighting. And while we're adding all these features, we made it a point to go to like every single local VR meetup to show off the build, get feedback, meet everybody else in the scene. Um it really was this this large running user test with the public. And around early October, we received um a mysterious summons from Valve. Uh they invited us to Seattle, didn't offer much information, you're not going to say no. Um and at at this point, this meeting is fairly well documented. It was the unveiling of the Lighthouse

technology to a small number of developers. Um a chance to give feedback and kind of an ask to, you know, do you want to participate in what will be the unveiling of this new headset built by HTC? It was an awesome couple of days of brainstorming with, you know, like-minded VR folks. But what this resulted in was Skillman and Hackett obtaining a V negative one Vive headset. So we had officially achieved the unthinkable goal of getting Tag Room quality VR in Drew's apartment. So by every metric that mattered to us two, our company was an outstanding success. Uh it also meant that we had to rethink our entire applications user interface. The uh the six-off controllers were way better than we expected them to. And remember, like Tilt Brush was this desktop-based experience. You build things right here, and the possibilities that room-scale VR gave us were totally

uncharted, and so we had to rethink all of these things. At the time, I sent an email to all the other developers explaining that we got our kit set up and I included this image to show how we felt at the time. It's Neil Armstrong. Uh unfortunately, now we're getting into November, we've been focusing full-time on Tilt Brush and we hadn't been doing any contract work. So that means we didn't have any income. But what we did have was a plan for not only how to build out Tilt Brush as an application, but how to build it out as this ecosystem. Part of this ecosystem relied on lots of users consuming these VR sketches. And at the time, we were really convinced that if you create something in Tilt Brush, it should only be viewed in VR. So we needed a product that could reach a lot of users. There's no better candidate than that than Google Cardboard. So we reached out to Andre. I asked him

for an embarrassing picture and that's what he gave me. Uh and lucky for us, Google at the time had a they had a big Cardboard push that was coming that December. So we worked with them and got Tilt Brush Gallery ready for their event. Tilt Brush Gallery was a Cardboard application where you could view a fixed number of sketches that Drew and I had did had done. Uh they were all preloaded on the on the application. Doing this was another, you know, big four sprint for us. Uh but the work that we were doing was also going to be immediately beneficial because we had to turn our attention to the Vive shortly after that and start optimizing for 90 frames a second. So uh as Patrick said, we got the Cardboard version of the app launched or at least a demo of what it might be. Um and next on our list was uh seeing what would happen when we took the keyboard mouse version and really pushed on those six off six off controllers,

those six degree of freedom controllers. Uh Valve uh offered us an opportunity, a super generous offer, to go hang out with them in the Bellevue office and embed ourselves with the VR team and uh see what we can make happen. Uh it was an incredible incredible generous offer. We took them up on it and we called it Valve Camp. So, our goal was to build a GDC demo experience. Uh the reward, if we were successful, was going to be inclusion in in Valve's uh GDC showcase and the unveiling of the Vive. But, we only have 2 weeks to do it, 2 weeks at Valve Camp. Uh the experience itself needed to fit inside of a 3-minute window, a kind of a 3-minute time box. And of course, our approach was to lean on those, you know, indie lessons from Double Fine and uh in our independent days. Work hard, focus test constantly, and iterate like mad. But, Tilt Brush uh in its current form was way too complicated. We put people in front of

it and uh play tested it on, you know, Valve employees and other people and it uh it was just it was just way too much for people to wrap their heads around. All these features um they didn't make sense to people. And so, in classic style, we cut cut cut. We really just like cut everything. Um our goal was to distill the core of whatever was making Tilt Brush resonate with people and cut away all the fat, cut away everything that that was distracting from that experience. And so, goodbye refractive brushes, goodbye audio brushes, goodbye lighting, goodbye model import, goodbye volumetric brushes. Just cut cut cut. But, it was worth it. Um those cuts were replaced with a massively simplified paintbrush and palette style interface. Uh we tried probably 10 different types of, you know, true VR 6-degree of freedom interfaces, but but no one was able to wrap their heads around them in the in the 3-minute window. That 3-minute window ended up ended up being so crucial to kind of like honing the experience. But the the uh paintbrush palette

interface, it was working. It was coming in blazingly hot towards the very very end of that 2-week period, but we were we were feeling good about it. Our confidence was confidence was growing. We asked Chet for one last play test and he offhandedly said he had someone in mind. So so we go back to our desk, we're ripping apart the build and we get a tap on the shoulder and it's to our surprise Valve CEO Gabe Newell as our last play tester. And the funny thing about VR play test is that the person in the headset can't see the developers. That means you don't have to mask your expression. So you got to picture Patrick and I watching Gabe in the demo and just making a face like while we watch Gabe start to use the new interface and brushes. And we also didn't have to hide our relief when he said that he liked it. And when Gabe Newell says that he likes something, you feel you feel pretty good about it. So at that at that point we called it, GC demo was done. And we headed back to San Francisco.

In our carry-ons we had new upgraded V0 prototype hardware to play with, did a week of bug fixing and then the vibe was unveiled. This was another one of those super special times where my old apartment turned into one of the most like futuristic studios in all of San Francisco at least for a few weeks. We were trying all the other devs demos, we're trying alchemy stuff, all kinds of things. You can see Patrick is fishing in the top right corner there. And the vibe reception, you know, couldn't have been better. It was it you know was enormously positive at that year at GDC. And Tilt Brush made it into the GDC demo loop and was receiving tons of great press because of it. All right. So now we are post GDC, March 2015. What do we do now? Reception at GDC was really stellar. It it was clear to us that continued effort pushing on it Tilt Brush could be a huge success. It was still the two of us at the time, which is great for keeping overhead low,

but also, you know, limits the speed we can move. And contract work was really just not appealing anymore. Uh we wanted to just keep working on Tilt Brush. What are our options? Being in San Francisco, going to these meet-up scenes were often encouraged, strongly encouraged, uh to take on VC money, so that was certainly a ready option. Joining a larger company would provide resources for the team. Or we just keep moving forward as we'd been going. We pick up some contract work, try to pay the rent, uh and then just do as much as we can in our free time. Uh this was hard. We thought leaving Double Fine to start a company was hard. It was actually very hard. Um this was a magnitude harder. So, what do we have to consider here? First and foremost, we live in San Francisco. It's expensive. Uh contract work is going to require time. All the time we

put toward that pulls away from Tilt Brush. Creative ownership is an absolute must. I mean, honestly, it's it's how we ended up in this position in the first place. It's why we created Skillman & Hackett. And this was a unique project. We'd done, as Drew said, we'd done lots of prototypes. We knew we recognized that this one was going to be special. Uh and we thought that it truly deserved care. So, on April 16th, we announced that we had joined the Google VR team. We had a lot of conversations with Andre and his team about the direction of the industry. It was clear that where we hoped it would go, they hoped it would go. Uh Andre really recognized that Tilt Brush was unique. This move would allow us to keep the lights on, and the important part to us was that we were going to hold creative control over the project. So, really in Google we had found a perfect match for Tilt Brush and a partner that we could grow this industry with.

So, that brings us to April 2015. And we had those kind of like broad meta questions of where we going, how are we going to do this, what are we doing? We hadn't We hadn't figured out, but that meant that it was time to get back to work. We had to ship Tilt Brush. The tentative tentative date for the Rift and the Vibe were the holidays, so we knew that we had about 8 months to integrate ourselves and our work into this new company, turn the demo, the GDC demo into actual actual product and launch. Our strategy was going to again be to apply those indie game dev lessons from previous previous lives of rapid prototyping, fast iteration, extensive play testing, but we just had to do it Google. And everybody was on board with that. People people were excited for us to kind of bring the approach that had been so successful in our independent studio to Google, but it was less clear how to do it. And a big question is would it would it work at a company like Google?

One of the first hurdles was really understanding just how to apply them at all. Patrick and I had never worked at a big company before, and we were we were let's say a bit naive or a lot naive. And one of my favorite examples of this is is our approach to getting help modeling our environments. This was like right at the beginning. We just started Google and we just we just needed to make some more environments. Seemed pretty simple. So, our idea was let's just hire our good friend and collaborator Tyler Hurd. We'd done VR Butts with him. We've been working with him for a while. And this is how the collaboration worked when it was independent and we were working with him on Butts. Said, "Hey Tyler, you want to work together?" He said, "Sure, send the contract over." And this is what the contract was. It was made in Microsoft Paint. And uh equally good is the file name. Um

So So this was just this is a good example of kind of where we were coming from and uh Google legal team The Google legal team is really really really really great. Uh they're really great. They softened the blow and this is not a joke. They softened the blow with homemade cookies. You guys cannot do this anymore. You are a Google. Can't make contracts in MS Paint anymore. Uh So that was one hurdle. Um then we also needed to uh accomplish a bunch of firsts for the first time and we had to do it before the holidays. We needed to launch on an engine that was new to Google. We needed to launch on hardware that was new to Google. And we needed to launch on a distribution platform, Steam, that was also totally new to Google. And Patrick and I really didn't understand the magnitude of these challenges. Uh again, we were naive, we're coming at it from mile-a-minute

indie development, who cares, MS Paint contracts, get it done. Uh and luckily luckily Google uh stepped up immediately and they helped us. It turned into a big team effort with a lot of folks behind the scenes kind of like pushing to make this happen. Um and so So we were able to get these firsts uh you know, accomplished at Google, but it was way harder than we expected. Another challenge was growing the team. Um around this time uh VR was exploding. Um people from games were getting absorbed into to VR left and right and it was it was super hard to actually just go from the two of us to a team like we'd imagined. Ended up taking a lot longer than we expected. One of the great things about life at Google is Google scale. Um, Google operates at at this massive scale and it and it afforded us some really special and unexpected opportunities. Uh, one of them is that we got to collaborate on this mixed reality launch trailer with uh, a visual effects team and a creative agency that did work on The Martian. Uh, which was, you know, coming from I have

a visual effects background, so I was super excited about this opportunity. Um, we actually gave them like we approached it in a very indie fashion. We gave them our source code, so they had our Unity project and they were you can see this is behind the scenes footage from the shoot. They're doing camera matching and then re-rendering from Unity and compositing things back together. It had that like indie vibe approach, but it was a way bigger and more coordinated effort. Another great example is that uh, when we joined Google they sat us next to the Google data arts team, um, which you may be familiar with some of their work and if not, you should check it out. It's they do incredible incredible things and they sat us down right next to these guys. And so of course, through hallway conversations we uh, just like ended up doing a collaboration uh, that was called virtual art sessions. And they had they had two connects on opposite side of the room. They would capture artists kind of volumetrically, capture all their paint and they built the whole thing into a WebGL experience that you can go check out. Uh, again, you know, like massively bigger in scope than anything we would have bitten off ourselves. Uh, we did a collaboration with Glen

Keane, legendary Disney animator, uh, and uh, worked with him on a future of storytelling presentation. And in addition to kind of resulting in a video that went viral and exposed a bunch of people to Tilt Brush for the first time, uh, I got to try my hand as a roadie. You can't see me, but I'm in the darkness back there running back and forth. Uh, right before this video gets taken, uh, someone another stagehand steps on a power supply and shuts down the vibe and I was like trying not to throw up as I clicked buttons and rebooted machines and was waiting to see cuz this was back when everything was just so hot off the presses. You never knew what was going to happen. It was absolutely terrifying. Uh And we even did a Mountain Dew NBA collaboration. Um definitely definitely not indie. Uh so yeah, a bunch of examples of of Google scale there. And really the way, you know, looking back, the way I see it is we went through trial by fire. We had 9 months of of really tremendously hard growing

pains. But we came out came out of it on the other side with a 1.0 Tilt Brush release. But way more important than that is is at this point we had a team. It wasn't just the two of us. Uh we were shipping Tilt Brush with the team and it was it was, you know, it was better than we could have imagined thanks to all the new faces that we had had working with us now. All right. So now we're into 2016. Uh we're a little bit more comfortable, really trying to retain this independent spirit. In particular, we maintained a a really fast pace, instrumenting 6-week sprints. Led to a lot of launches um and us bringing back all of those features we had cut. Brought back animated GIF export, export FBX, added a teleport tool, which was a huge oversight to begin with. Um later in the year we added the ability to scale and rotate the scene, started building out more pro tools. Um and a pain that we kind of just had to become accustomed to was we were now in full production on live product. And

this means that it is of the highest priority. It still is the highest priority that we do not break existing art. So any new change that we put in can't affect the art that was created in a previous build. Before, you know, we'd launched, we could safely prototype, scrap, throw things away, whole concepts, didn't matter. Um but now we had to live with these choices. In addition, we were rewriting huge sections of the code because, you can imagine, this was all like prototype quality code. You can't build off of that. And we were hurdling walls. Um Drew and I had this had this motto during the Skillman hackathon that you run as fast as you possibly can. And if you hit a wall, you just turn and keep running. Uh unfortunately, we're now in a place where we had to solve those hard problems. So, you can't just turn. You have to hurdle these walls. Or as we like to joke sometimes, um you're running so fast you just blow right through your wall. We continued to prototype as much as possible though.

Uh most of these prototypes will never see the light of day. This was a prototype I did for a connect the dots game that would be inside Tilt Brush. It was pretty much as fun as connect the dots is. Uh this was a portal brush that using the camera on the front of the VIVE allowed users to peek out into the real world. This was an early version of constraining brush strokes to uh planes or other shapes. And this would actually eventually ship as our guides feature. Early version of the audio reactive mode. Uh and another early version of a a feature that just shipped, a um a quick tool interface. Uh so, in addition to prototyping like mad, uh and we were still we were still play testing like mad. Again, the indie spirit of Google wanted to keep it alive. Um and it had gotten us really far, you know, like we've talked about how it was

it was huge to Oculus Connect 1. Uh really defined Tilt Brush those 2 weeks at Valve at Valve Camp. It was through those play tests that Tilt Brush really took its final shape. And at Google we wanted to go even bigger. We wanted to play test on casual users. We wanted to play test on our, you know, peers, our co-workers, other Googlers. Uh Uh, but we also wanted to try something new. We wanted to see if we could see what would happen when we, you know, we play tested and worked with some of the world's most accomplished and impressive artists. Uh, and we did it through a program called artist in residence. And so, through this program we engaged with about 60 artists. And worked with people well, people like uh, Glen Keane who I talked about earlier. On the right here, this is Victory Dance by uh, by him and this was before we had world transform. So, every one of these strokes uh, Glen had to actually walk to and place in room scale, which is a crazy time to think about. Like he had he probably

walked 10 miles to make this sketch over the course of a week. He was crawling, climbing on things. It it was just a demonstration of this like grit that that the world's best artists have. Unbelievable. And he would he would give us feedback in the form of beautiful drawings and sketches of strokes he wanted or brushes that he wanted to be able to to you know, use. Another great collaborator was Peter Chan. He's another legend. He did a bunch of work at Lucas Arts and Double Fine Games. He's worked on a bunch of Hollywood movies. And this piece is called The Rescue and it's the opposite. This is we worked with him while we were developing the world transform feature. And so, he he used this world transform feature to feature to build this huge kind of like epic snowscape that completely brought Tilt Brush to its knees. It was so massive. It runs at 90 frames a second, but it takes a little while to load. So, we need to optimize our loading before we can share that. And here's some feedback we got from

Peter. Again, it's just beautiful getting getting getting feedback from artists. Uh, Chris Pronaoski from Titmouse did this sprawling comic that you explored and like walked around room scale reading these panels. He also took incredible notes and we captured we used Chris for some promotional footage and so he actually wrote this in his notebook which I think is hilarious. Write fake notes in notebook for shoot. I am writing these notes now. And again another collaboration with Tyler Hurd. I'll let the gift do the talking on this one cuz it's really weird. It's weird and great. Su-Chin Chung is a New York City based artist and she really she does beautiful work but she also really pushed us on our export. She was asking for splines that she could bring into her the rest of her workflows. It's just a cool example of some of the unusual unusual

things that artists ask for that maybe a regular play tester wouldn't. So all this art and tons more is on the tiltbrush.com/air website. I think it's an incredible reflection of of Google scale and it's also something that never would have happened as the two of us in a little tiny my old apartment. Um I want to I want to take a moment and call out the incredible team behind this program. Again, Patrick and I could not have done this. This is like tons of people making things happen at Google and those people who built this program and the artists who participated in it had this pivotal and critical role in defining the path that Tilt Brush would take. So that's that's like an example of Google scale, right? On the opposite side when we're working at Double Fine and more generally in the video game independent game scene we learned how important it is to participate in and really take care of your community. We spent a lot of time doing this at

Skillman & Hackett attending all the local meetups and meeting folks and we really tried to carry these ideas into Google. Um in particular I spent a lot of time on Twitter. Uh this is a capture of TweetDeck um where I search for Tilt Brush as one word and two words because, you know, people aren't consistent with it. We also maintain [email protected] which is very strange for Google. They do not normally have a public-facing emails for their internal products, but it's been an enormous help for us to communicate with the with the community. Glowing example of this is having a Twitter conversation with a gentleman named Tim Aidley. Uh, he had reverse-engineered our save file format a long time ago, was creating 3D models with it, followed up over email, and well, eventually we hired him. I also maintain a ridiculous spreadsheet of feature requests. It's in no way scientific, but it's a great pulse on feature requests that

come in through that email or through Twitter. And this really does help shape our future roadmaps. And really the only reason I'm showing all this stuff is to emphasize that the community should help shape your product. Um, we've said this a few times before, but you know, at Skillman & Hackett we gain so much from attending all those local VR meetups. So we wanted to pay this forward a little bit. And so we hold developer meetups at our office where we let people demo their experiences that they're building or just eat pizza. Uh, this was kind of a scary moment. This was the moment that that our community kind of started to take on a life of its own. Marvel worked with three artists, Danny Bittman, Steve Teeple, and Stuart Campbell to create promotional art for their upcoming Doctor Strange movie. Now, like the art they did was amazing, but the the crazy part about this is we didn't have any idea that they were doing it. We didn't know about it until the sketch was done and they sent it to us. It's like, "Hey, check this out."

And it was this sobering moment where we recognized that people were making a living off of this tool that we're building and maintaining. And I really like to think that the lessons that we learned about community management from the independent game scene is had just a huge influence on the product. Um you know, it's really the dialogue with your community that allows us as product creators to empower the real storytellers. So, uh that brings us to January 2017. Uh a team has gelled. We're hitting our stride. We've kind of figured out how we can operate at Google. It was a challenge, but we we figured it out and that means that now we can move way faster than we were. Um and we're doing all kinds of cool things. We're pushing in new directions. Uh we launched something called Tilt Brush Toolkit, which lets anyone bring Tilt Brush art into Unity and preserves

shaders and materials. Uh purpose of this is to allow people to breathe life into the Tilt Brush art. Uh so, some examples here. On the left is an a music video that Titmouse Animation did. And on the right is a game called Paulo's Wing that was published on Steam. We're also uh launching fast and the launches are bigger now. They're a lot bigger. We've launched on Oculus. We're adding new lighting features, custom environments, detachable panels, a bunch of polish. Uh and we finally got to something that was on on a road map, you know, 30 slides ago, uh sharing. We got deep sharing integration. Uh and what that meant is that anyone can share sketches and view sketches from uh vr.google.com/sketches. And we even have support for other Google products inside of Tilt Brush. This is a the Blocks library, which is support for Google Blocks that happened in our most recent launch.

So, that's it. So, 2 and 1/2 years in and we've been running this this grand experiment with Google and we've proven that it can be successful. We've capitalized on the benefits of Google scale, tried to really maintain these core principles we pulled from the game industry and from Skillman and Hackett. And we hope that, you know, these These are lessons to show that these game development principles can be used outside of games in very different atmospheres. And it's also a validation of prototyping as an exercise for production and recognizing when you should really drop everything and and start running at an idea. So, we want to say thanks, but really quick before we jump to questions, couple things. Okay, it's common in games to have a credit section and a special thanks section. So, we wanted to let you know that there's a hidden credits sketch in Tilt Brush that as far as we know has never been found.

And we're not going to tell you how to find it, but I just want to let you know that it exists. And likewise, we want to I mean, there's so many people that were pivotal in Skillman and Hackett and Tilt like the last 3 years have been this wild wild ride, this incredible roller coaster. But, I do want to thank a few people who who are pivotal pivotal in the creation of our our indie studio Skillman and Hackett. That's Tim Schafer, Aaron Davies, and as mentioned earlier our friends on the Medium team. And a huge shout out to Chet Faliszek and Andre Doronichev who saw something special in Tilt Brush and and actually did things that brought it to life and I honestly don't think it would we'd be standing here without them. So, tons more people to thank, tons more people I see you in out there. Thank you.

Uh yeah, so questions? I think we Yeah, we got a few minutes. Hey, so uh I'm a designer not an engineer, so I don't necessarily get into arguments about tabs versus spaces or Unity versus Unreal. But you did mention that Tilt Brush was being built within Unity. Is that still the basis of of the product with the product constructed or have you had gone lower level or or anything like that? Yeah, no that's a fantastic question. Um we are still built in Unity. Um we've been uh optimizing aggressively in order to like in order to accommodate the work that people make which has been explosively growing in scale, but we haven't hit the limit of what we're able to do inside of Unity yet. There's still optimizations to be made. There's still a lot of room uh for us to grow there. So currently we're still in Unity, yeah. Um so I actually tried Tilt Brush. I love it. Um It's a it's awesome program. It's like

very immersive and I've spent hours and hours with it. So but my question is more on cuz I'm also looking at Quill. So I'm wondering what your guys' thoughts are on Quill and maybe uh where you see if if you're going in the same direction or if you're not going in the same direction, where are you taking Tilt Brush? Yeah, that's a fantastic question. I love Quill. Uh incidentally I had that picture of like Double Fine, Skillman & Hackett, Medium. Also that's Quill and it came later in the story, but that's also where Quill is. So we had lots of conversations with Inigo and those guys. Um they're incredible. Uh I think that for us Tilt Brush we've just taken a little bit of a more um broad view on what a brush can be. Uh and I'm not going to speak for Inigo or the Quill team, but from the outside it looks like they're very laser focused on um unlit strokes and like and animation. I've seen a lot of animation, but they're really like like it has a very

beautiful uh beautiful specific look. In Tilt Brush we've taken uh much I think more um open-ended approach to brushes. Patrick mentioned that earlier we were like, "What can a brush be?" Like once you're in VR and you're laying strokes down, well those strokes could turn into anything. We've really tried to push that in terms of particles, unlit brushes, lit brushes, animated brushes, objects, lighting, set design. So so that's really like kind of where we've been focusing our efforts. Uh but I think they're you know Quill's a beautiful tool, so I like it too. Thanks. I wanted to ask about the impact of your product uh not that you've generated necessarily, but that the artists that use your product generate. Uh so the first part of it would be what's the biggest impact that you've seen um from artists that use your product so far and uh in the future, you know, like

a pencil for instance uh has a big impact not by the person who made the pencil, but by the people who use it. And this is really a new uh medium. Sorry to use that word. Um and uh so I'm curious how um you plan on evolving your tool to allow artists to have impacts, whether that means like entering into the movie industry or the video game industry or having Donald Trump sign a new executive order in Tilt Brush or whatever it is. It's an idea. Yeah, yeah, yeah. I mean I I think um um I I the answer there is consumption, and I think that that is almost a like VR level problem, right? Like um you know, I said at the start we're like, oh, if you build in Tilt Brush, you should only view it in VR, but then we saw how powerful it was to for people to

integrate Tilt Brush into their standard pipelines. So, we added, you know, export features and a and a lot of care so that you could take it out of Tilt Brush. Um and then we even added much later the ability to to share to the web, and so now you can view sketches on your phone. Um and so I I think to maximize impact, you need to maximize that part of it. You need to maximize the ability for someone to use Tilt Brush or Quill or you know, any other VR uh content creation application and then get the get it out. Yeah, cool. And I'll I'll just add that I I uh for me when I'm working on Tilt Brush, I like to I I nostalgically think back to like as a kid using like Super Paint and MacPaint and like these little big giant pixels, maybe it's a 200 by 200 canvas or whatever. And the tool was the tool was ridiculously immature, but it still uh made me think in a way that I think my brain was ready when like actual art

creation on computers happened. And so, I don't see it necessarily as a short-term effort, but it's more like you said, like with the Air program in particular, our biggest goal is just legitimize it as a new medium. We want people to understand new medium and to be thinking about how it will impact their lives, even though, you know, who knows like how that impact will ramp up. But it feels like it's on a you know, there's only it's growing and it's going to be changing people's lives. Thank you. Yeah. This is less a question and more just a comment, but um really love Tilt Brush. First tried it at a private Valve demo, and I think my exact comment was, this is the coolest I've ever seen. Thanks, man. And we work at the NASA Jet Propulsion Laboratory. We actually use it to prototype UX for AR VR experiences that we build. So, not all of them, but occasionally we'll go into it and we'll say like, "Well, what do we want?" and we'll start drawing it out in Tilt Brush and then we can actually go and refine

and build it in our own tools, but it's a really great way to sketch that out. So, thanks for making something like that possible. awesome. That's incredible. Let let us know what's what what you need, what's broken. We don't want any, you know, rockets crashing. Hey, so given that we now have ARKit and ARCore, have you thought about building Tilt Brush in AR? Yeah, I mean we've always talked about uh uh we want to build Tilt Brush for whatever platforms make sense. You know, like um we built that Cardboard viewer a long time ago Tilt Brush Gallery uh a long time ago, but it didn't ever make sense for us to try to build a creation experience for Cardboard because we feel that it requires six off head tracked six off controllers. Thank you. Yep. Hi. Um I love the Tilt Brush's selection of the paints, but also the UX decisions

and how it uses the metaphor of the palette. And since you guys are so um one of the very early uh VR apps that was available, I was curious what your process was like and how long it took to arrive at that sort of decision. Yeah, that um that mostly happened at Valve camp. Um you know, like when we showed up there we had this DK2 version that was all cockpit and um I think we had a slide in there that we pulled out that it was making fun of ourselves for not capturing more like iterative footage from that time because a A of our um a lot of the things we tried would show up later. You know, what we first had okay, the the panels are around you. And then we thought, well, you're in this room scale experience, maybe they just float. Maybe they just hang out and you can move them around, but then people didn't know how to interact. Uh we tried what we referred to as Batman belt where you press a button and they attach to what we could guess is your hips. And then you do this, but it was distracting. Um

flavors of all of those that had springs attached to them, they would follow you around. And eventually it it just um whatever, you know, like let's try one more thing where we just you know, dismiss this controller. I think there may have even been a time where we were talking about Tilt Brush would be a one controller application. And we thought, well, let's use it and maybe attach, you know, the uh panels to it and then things grew out of that. Yeah. It was really, I mean, to your point, it was it was very unexpected. We we didn't we didn't think that prototype was going to work. We were just really lucky to be working on it at Valve with a dedicated play testing room and people running through who were actually able to give us a thumbs up, thumbs down. So, that that really helped us find it. Awesome. Thank you. Hi. Um most mind-blowing thing I've experienced in VR is scale change in Tilt Brush. And I'm wondering if you can tell me if

you have any further plans for developing uh that functionality. Yeah, I mean, um uh that feature took longer than we hoped to actually push live because it's a delicate balance between simulator sickness. Um you know, we had we had prototypes that uh you could flip the world over, like the environment over. Um that made me nauseous immediately. Um and so that's why you can, you know, you can only spin and scale. Um uh there are flavors that we've talked about of, you know, being able to manipulate different canvases, different layers, um, and have full control over those, but um, yeah, that's it. I mean, it it really is a balance. Uh, you need some sort of frame of reference if you're going to start whipping the whole world around on you. The squirrel helps. Yeah. That was Drew's idea. Uh, excellent uh, presentation. Thank

you so much. Um, has there been any discussions to expand Google Tilt Brush into a kind of Adobe After Effects for VR or to allow for hotspots and interactivity inside various scenes? Hm. Hm. Yeah, that's probably on the on the spreadsheet of future requests. Uh, ha, ha. You know, one of the things that we've really tried to hold on to as we build Tilt Brush out is that core experience that was received so positively, uh, during GDC, during the unveiling. There's still a lot of people, I see on Twitter, that, um, you know, tweet things like, "Just tried out VR. Uh, tried Tilt Brush, love it." Right? Flattering, but it's also like, we just gave a presentation on we've been working on this thing for 3 years. And so, there's we almost feel like there's a little bit of an obligation we have to keep the experience for the first-time room-scale VR users. Um, and so, that's our

balance. It's like, how do you how do you maintain that piece while expanding for our pro users and enabling artists and like, really graduating it to the 2.0 version of Tilt Brush. Um, Yeah, the only the only thing I was going to add to that is just that we've also stayed true to our our rapid prototyping roots and we literally have hundreds of crazy prototypes exploring every single thing we possibly could. And, uh, and we are going to try and follow the you know pattern defined in in the deck and like let the let the magical prototypes kind of like guide the way and we've had a few of them. But you know it's like the fun thing is no one knows what's what's going to happen and which ones are going to win and where they're going to lead. So we're just kind of along for the ride. All right. Awesome. Hey, thank you guys so much. Thank you. Thank you.