Controls You Can Feel: Putting Tactility Back Into Touch Controls

[Music] this is controls you can feel I'm Zack gage there's a hell of an amazing echo going on for me so that's cool this is a talk on why tactility matters and how you can build that into your games if you're doing touch but I'm also talking about kind of any kind of system so this isn't just touch this is if you want to make a game for anything you've never really dealt with how to think about the controls so the first thing is I got to say I'm not a scientist I didn't do research for this stuff I'm an artist and I've worked on lots of different things and I've spent a lot of time thinking about interaction I've worked on large-scale installations I've worked on small-scale iPhone games I've worked on weird art games I've made chat rooms or chat software I've done small installation art I've even made aggressive books that take advantage of how you're interacting with them so I designed and thought about interaction in literally dozens of different contexts that are all really weird

digital and non-digital and in my experience the only thing you really need to do to design quality interaction is just really pay attention to what you're doing and put in legitimate thought into thinking about it and I have a couple rules that should help so when I talk about tactility I'm in this presentation I'm not just talking about you know what you feel with your fingertips which is usually what we think about when we think about tactility I'm talking about the entire physical experience of interacting with something so how you hold it how your body understands it everything so as far as I can tell the root of a lot of unsuccessful iPhone control schemes comes from designers just not realizing that the touchscreen is something that they need to design controls for and beyond that you have to design not just good controls but you have to design a game that's supportable by the type of controls that you can design so most of us who have been thinking about making games for a while should be pretty familiar with design and control layouts

as in like designing what buttons on a controller do something and because we've been making games with this interface for so long it's really easy to forget that that interface was designed in the first place and all the games that we make are based on that interface as much as their based on any creativity that we're bringing to the medium so designing for touch interfaces is really different than just designing what key or button does what just as designing a game for the Wii or the Kinect is really different it's the closest you're gonna come as a game designer probably to designing whole new physical control schemes so a long time ago Nintendo got to invent things like the B button and the d-pad and the Z trigger and just like them today when we make touch games we're inventing ways of people physically interacting with our devices which is pretty exciting I think sure you could say this isn't really worth it and just in case for some reason you stood on the line to get in here and you totally disagree with everything I'm going to say let's just look at a couple of really successful

iPhone games so Angry Birds fruit ninja Canabalt you've probably seen all of these games these all have really successful touch controls and I would argue that that's the driving reason why they've been so successful especially things like fruit ninja cut the rope Angry Birds and in case you're worried about unique controls and you think that maybe you won't be able to port your game if you have unique controls let's just consider a couple reasons why that's not a great idea so first you got a rock with your software when you first put it out you can't put out something that sucks and then expect to port it to something where you'll have easier controls you want the best controls in the first thing you put out also games with unique controls tend to have more success or be capable of more success which is why everything you see on the iPhone store that's popular either has great controls or is a copy of something that had great controls and the other thing is the iPhone is the mobile console to beat everything's got touched and it's going to have touch so you can probably port okay so how do we do this

how do we do this special thing so I have two rules any new successful control scheme should follow these two things one it needs to be comfortable when gestures when players make gestures or moves those things should feel good they should probably feel great but they definitely shouldn't feel bad it's really easy to think that a gesture that doesn't feel very good won't alienate a lot of people but it will and if your gesture feels extra good that'll get people into your game way more so especially if you combine it with a really awesome thing happening in the game so pulling back the countable the catapult in Angry Birds feels good pulling the trigger and the shooter feels good there are reasons why those tropes have really lasted and then the second thing is your control scheme really needs to inspire confidence in the player it needs to provide primary feedback that allows practice players to be a hundred percent confident in their inputs and this is a little bit more complicated so I like to think that feedback comes in two categories primary and then an optional reinforcement and

secondary and I split them up like this because although they are both important there each important for different reasons and you can't compensate for having poor primary feedback with great secondary feedback or vice-versa so what makes up these so primary feedback is information that players get from the body this is the tactile information and this could be coming from your sense of touch at a thousand frames per second it can be coming from your muscle memory or from your understanding of body position which is called proprioception and the success or failure of this is based entirely on your games control design and this is where a lot of iPhone games go wrong and this is basically what the focus of this talk is going to be then there's kind of an optional reinforcement that can come from audio cues so this would be something like the jump sound in Mario and then secondary feedback comes in through the eyes and this happens a lot slower at sixty frames per second or less depending on the game and this is never the solution to primary feedback I think probably a lot of you have great skills of designing secondary feedback and so I'm not really going to talk about that and the goal of these feedback mechanisms

then is to convey the game state of the game to the player as fast as possible so that they can really successfully embody the character that they are on the screen or that if there isn't a character that the world that they're interacting with feels real essentially the goal of these mechanisms is to make the action of interacting with this world as completely invisible as possible so that the player can spend the most time thinking about interacting and with the world and not thinking about interacting with whatever their devices so to illustrate this clear we can look at how information is coming when we play Mario and if you weren't thinking about control design you'd probably think alright I'm going to push right Mario is going to move right I'm gonna push a Mario move a or Mario will jump and this kind of feedback is really important to new players this is how they learn the game they try something and then they see what happens but this visual feedback is actually the secondary feedback the first feedback is coming through your fingers so when a practice player plays Mario she presses right and she knows that Mario will move right before she

sees it and this is easy to demonstrate if I push the a button here what's gonna happen next we don't need to see the next frame to know that Mario is going to jump and that's really important to have this intuitive understanding that Mario will jump when you push the a button that it's because I can feel the button go down it's because my proprioception is telling me that my thumb is in the right place this all maybe seems kind of obvious but just in case you were wondering what would happen that's what happens and when a novice plays mario they might not be familiar with the controller they might not know this stuff they might not be able to tell the difference between the a and B button by their finger proprioception they might have to look at the controller a lot and look at the screen and this is normal but the difference is if you don't have a good controller design nobody ever gets past that they're always looking at the controller or even if they're looking at it ten percent of the time that's still very novice play and it's not the kind of thing you want from your game so this is easy to demonstrate to if I'm a novice player and I'm pushing one of the

buttons on the right what's going to happen we have no idea the only way we can find out is by waiting for our eyes and our secondary feedback to understand what's going on and this probably seems obvious like this is not super complicated stuff but the thing that makes it tricky is that it has to be a hundred percent it has to work every single time so if we go back to the screenshot and we imagine I'm 80% sure I'm pushing the jump button that's not good enough 90% sure if I'm wrong Mario's probably gonna get killed by this Goomba which would be really embarrassing in front of all the friends that I'm playing attender with so it's got to be perfect I don't want to ever be worried and so when we take this back to the iPhone context you'd be really shocked at how many games fall into this good enough trap so geometry wars is a game that uses a virtual joystick that centers based on your thumb placement so you you put your thumb down in its center that's pretty easy to understand sorry thumbs look really when you take them apart from a hand so I move it right and that's a gesture and that's easy to understand because of my proprioception I put it down I moved it

and the ship follows my movement right that's pretty easy to guess I can understand what's going to happen from the screenshot and just from my understanding of my body but if I move my thumb to a different position and we look at the picture and we start to think about all right so now where's the ship going to go and think about how much longer that took you to figure out then when you first move the thumb and you can try it with your finger in the air and the problem here is that we're taking all the benefits that people are gaining from their proprioception from their understanding of their body and we're then mixing them with this other system with this spatial comparative reasoning system and so now instead of a system where we understand everything intuitively we have a system where you have to think about it and thinking about it takes so long that most players playing this game will now go back to relying on visuals they're gonna go back to that secondary feedback they're missing all the first feedback and that makes the game a lot harder to play and it's not just with these mobile joysticks it happens with fixed

joysticks too so I have a thumb box here that box the where 44 points is coming from is apple's guideline for how big a button should be when you're looking at it to be easily pressed as 44 points points are like pixels with DPI so you could imagine on a phone 44 points would be 44 pixels on a retina device it would be 88 pixels it's just size so they say that that's how big the box needs to be to be hit so we can imagine that on a joystick that's fixed a player puts his thumb down there's probably like a 44 pixel variance in where exactly their thumb position is and maybe if they're really good it's just a 22 pixel variance maybe we could make it smaller so that's half of apple's research number and they've done a lot of research on this but that's still a ton of directional options for this ship for this ship you can see from the pie chart there are all these different places that that could go two of those directions put me into enemies one of them is in the clear if this is the gun what am i killing all this stuff is things that players worry about where they're playing a game and this is all assuming I actually remember the exact

location of where that joystick was and when you play something for long you forget about that virtual thing so how come that works with analog sticks how can we don't have this problem so with analog sticks you have a spring in the center and you can always feel where the center is and the amount of torque on the spring is how far away you are from the center all this stuff was really well thought out when it was originally designed and we don't have any of these kind of affordances when we're dealing with virtual sticks all that information that we would normally have gotten from some mechanical process is coming from inside our body so now you might think like alright well maybe it's impossible maybe we can't do virtual d-pads and that's not entirely true super crate box has pretty good d-pads I would argue that you can't do a virtual d-pad as well as you could ever do like real touch controls which I'll get to later but they do kind of work and so we would wonder alright so how did they pull this off and it turns out it's actually not very complicated if Apple is recommending 44 points for a button that you're looking at we would

imagine you'd have to be a little bit bigger for one that you're not looking at and it turns out that great boxes are 75 by 75 points which is a pretty good margin of error there are only a few buttons you're not moving your thumb a lot and those buttons are only doing one thing they're not analog joysticks that could be anything they're yes or no and so it turns out that if you have a practice player who's okay with playing with virtual joysticks this is with virtual d-pad you can be pretty confident that your actions will be portrayed properly when you play this game but there are people out there who will play this game and it feels comfortable like a d-pad not rocket scientists science this is pretty straightforward you just have to think about it they just made the buttons really big and if you think about then the other thing that's sort of coming out of this is that there are certain games that you just simply couldn't do this with great box work because you only have two buttons but if we need 75 point buttons a d-pad is just not going to cut it and the other scary thing about a d-pad is when you have all those

directions even if we didn't care that it was taking up basically the whole screen you're asking somebody who's playing it to move their thumb a lot and the more you move your thumb the more you forget where things are that you're not looking at so there's a reason that the Jaguar sucked that the controller didn't work you have to look at that all the time so let's look at a case study here then so we could look at tetris tetris is a really good thing to look at because they released it and it was bad and then I released a game that was kind of based on it and then they released a Tetris that was really good so we have one bad thing and two good options so the original Tetris people move people interacted with this game by dragging the block left and right swiping down or tapping to rotate so if we start with what went well tapping to rotate works pretty well those are basically two giant buttons we know the buttons work and it worked great but that was it they didn't get anything else right sliding is problematic because the cute the Rope basically the

columns are only ten points and your thumb is huge your thumb is forty four point variants so you're not moving your thumb very far and when you don't move your thumb very far it's really hard to intuitively understand that there are not a lot of things situations in real life where we're only moving our thumb centimeters and so our proprioception isn't really geared to understand that kind of movement also the swiping down is a huge mistake swiping down is not a comfortable gesture and this goes back to that comfort thing that I was talking about if you look at Apple's UI you'll notice that there's very few situations where you ever swipe down all the menus that you scroll start at the top and you swipe up and that's because swiping up feels great swiping down feels poorly and you can feel that if you try it with your hand this is not a comfortable gesture and swiping a piece down and Tetris should be the most exciting thing you can do because you do it when you've made a decision or when you're gonna get rid of blocks that should feel great mixing it with something that feels bad is not a good move so the game that I

built that was kind of based around that was called unify and I solved a lot of these problems by flipping the phone on its side a lot of things are more comfortable when your thumbs are more extended when you're holding a larger thing that you're not cramping your hand around so the way that unify worked is you had two pieces one on the left and one on the right and the bottom row from Tetris was in the middle so rotating is just tapping and it felt good sliding pieces up and down feels pretty good because these rows are so much bigger than those time roseann Tetris and sliding in actually putting your two thumbs together is a really good gesture it feels really good players really like it especially when you put them together and then blocks blow up off the screen so I got a lot of mileage in this game just from that gesture feeling good and being able to do it over and over again and I noticed that even in prototypes the other thing is we added really strong physical reinforcement sounds so the sounds that come in this game aren't just digital bleeps and bloops they're very physical they're like clicking and that makes you

feel more like you're interacting with a physical thing so this was a pretty simple set of changes but it resulted in a really differently structured game and this is a really important point that I want to make is don't be afraid to let changes making in your controls don't be afraid to let them bleed into game play it's really important that the game play takes cues from your controls so let's look at how tetris solved this so they unlike they couldn't really do the amount of changes that I did because mine resulted in a totally different game that didn't feel like Tetris they wanted something that still felt like Tetris so they actually did a really good job and they did it by not being afraid to mess with some of those really core mechanics of Tetris they were trying to hold on to the core of Tetris but they could change some of the mechanics and so what they did is they have these options of where you can drop a block they got rid of the sliding entirely now you just tap where you want to drop it and they only give you four options and the reason for that is because they need enough pixel variance so that you can be super confident when

you tap one of these options that you did tap it properly that's pretty close to Apple's number and it works perfectly the other thing that they did is if these aren't good options you can tap that cycle button again and it will give you four new options and it will reset the time that you have so you could just keep tapping the cycle button forever and never drop a piece but what they did that was smart is they basically took the time it used to be in Tetris you were under time pressure and you were interacting and they noticed that the interacting wasn't great so they separated them out they made a non interactive mode where you're thinking under time pressure and then you just make a choice and it works and it actually feels a lot more like Tetris than the old terrible version of Tetris but okay no more virtual buttons that's not probably why you're here how to make good virtual buttons so let's look at some games that didn't do virtual buttons and did good flight control was a really great example drawing a line feels really good the planes are easy to press you never make a mistake but there

are some mistakes that you do make and flight control and that's when you draw a bad line but that's not an input confidence that's the player just not being skilled enough which is the kind of mistake that you do want to make and they were really smart they paced their game really slowly so that you could actually get that secondary feedback you could draw a line and then you could look at it and be like oh that's not the line I wanted to make and you draw it again and it's really important to point this out because this isn't a decision that they made because their control scheme wasn't successful because players weren't confident this is a decision they made to enhance their control scheme this is where you have something that works and then you design your game around it in a way that supports all of the things that are great about it and tries to draw attention away from the things that are bad if this game was really fast and you had to draw exact lines it probably would feel terrible but that comes from the actual game mechanic and not from the interaction the interaction is actually very successful and players playing it are always confident I don't know if you've

ever tapped a plane and it didn't work I've never had that experience and that's really telling fruit ninjas another one that does it faster fruit ninja you swipe and it draws a line and it feels great players can understand it you get incredibly great visceral feedback because swiping feels good and then something blows up every single time and the other thing that they did which is smart is that strawberry up there is 40 by 40 points you're gonna see that number everywhere when you start to research this stuff there's nothing smaller than that because something smaller than that would be really hard to know that you hit it and the thing that you really want is to build a game around supporting the confidence that your input is generating in players so I have one more you really want players to blame their resource their reflexes when they lose and not the game controls so I have one more example of a game with even faster analog input and that's this game bit pilot that I may which was really born out of the desire to have an iPhone game that was really

intense that wasn't slow movement and planning where you could actually input it directly and do these really great sweeping moves where you could really become an expert in the controls and so this might look like a standard twin-stick asteroid dodging game but it's actually quite different so I'm gonna show a video so the way that this works is that when you swipe on the screen oh man that is super dark I may need to adjust I don't know is there a way we can turn down the light slightly for the video No okay so here you can see the thumbs moving so the way that it works perfect thank you is you swipe with one of your thumbs and the force of your swipe is given to your ship like in a fraction of the force and so if you swipe with two thumbs the force of both of your thumbs is given to your ship and it's not one to one it's not like if I move my thumb in a circle the ship will move exactly in a circle it's a you're kind of pushing the ship and this works in a lot of ways because of the way we think about gestures when

I gesture when I like I'm just sticky lating with talking I'm not doing something that's very specific I'm doing something that's kind of vague it's sort of a hint and so in this game you're hinting at the ship what you want to do but you can make big big gestures if you move both your thumbs at the same time but really most of the time you're just hinting at it and because you have to hint at it a lot the player has to make a lot of decisions all the time and that takes a lot of the stress out of what those decisions are which is really really helpful for the player and it makes you able to free up your visual perception to pay attention to all of these tons and tons of asteroids that are moving which you would normally not be able to do if this was a dual stick game you'd be paying so much attention to where your ship was and if it ended up where you wanted it to be that you wouldn't be able to pay attention to how these asteroids are moving if you notice here there are no visual cues as to where those asteroids are going you actually have to keep track of them and remember where they were okay so that

that was a lot of different things that I covered and probably you're not going to remember all of them exactly so I'll sum it up in a couple ways that should be helpful so the first thing if you're going to do this is you need to be flexible with your game design you need to understand that games are as much a product of the interaction possibilities of the platform that you're developing for as much as they are of your own creativity and it's actually normal and great select controls influence the direction of your game this is incredibly common so a genre that works on a controller might not be viable on a touchscreen might not be viable on a Wii that's okay the other thing is you need to be really serious about thinking about these things controls are really important they're the way that we're interacting with the games that are playing and and if you're not paying attention to it you're really doing a disservice to all the creativity that you've put into your game in the first place a skilled player needs to be able to know what the input that they

give to a virtual environment will do before they see it just like when you walk down the street you don't hope that you're not going to fall over you know and that's how we need to be able to interact with everything in a game and then the last one is once you have good feeling controls once you have players who are confident you need to make sure your game supports them that it supports the strengths of the controls and not the weaknesses that it integrates those two things well so this stuff isn't rocket scientists science wow it's not made by rocket scientists and it just requires a little bit of time and a little bit of experimentation and these three things and when you get it right there's nothing better because your controls become invisible and instead of concentrating on your controls players concentrate on playing the hell out of your game also these are good points for being a good human being that's totally unrelated but I think it's important ok so thanks that's my talk you