The Invisible Escape Room: Adapting Jack Ryan for Alexa

My name is Dave Grossman. I will be your your host this morning. Uh this is my trophy wall with which I shamelessly hope to coerce you into uh trusting me as an authority on on subject matter here. This is uh I've been designing and writing and directing uh story games of various types for about 30 years starting in the upper left there. This is the laser I'm sort of pointing. Uh with the Secret of Monkey Island at at uh LucasArts and Day of the Tentacle. Depending on how old you are, looks like maybe kind of an old crowd, you may or may not have have grown up with uh uh Pajama Sam. And uh then there was uh there's a large section in here that represents my my tenure at Telltale Games, um may it rest in peace. Uh where I was directing the design and writing there for about a decade. And then down in the lower right-hand corner, uh you see um some stuff you may

not have heard of before. Represents uh my more recent work. It's uh they're audio games. Five years ago, I became one of the founders of Earplay, which is a platform for making stories that you you play with your voice. They they use voice control uh and are audio only. We did this first on mobile phones, uh but then other other platforms started to come along. There were uh voice assistants, uh Alexa and and Google Assistant, and then more recently Bose made these glasses frames that have some interesting hardware in it. And we expanded to support all of these platforms. And in this talk, I'm going to mostly talk about the voice assistants, but the lessons apply pretty pretty broadly. Uh we've done quite a few of these things now, actually a lot more than this. Uh the engine handles all kinds of stuff, but they are mostly um kind of uh branchy choice-based things in the sort of a Telltale Games model. And there's some specific reasons why we like to do it that way. I'll get to in a minute. They tend to be kind of big compared to other skills that tell you the weather or whatever that just you kind of in and out, ours you may play

for an hour, two hours, three hours, some of them. Um it's still though uh not uncommon for them to be designed and written and sometimes implemented by a single person. And so I want to give you a window into what that process looks like from the ground up. Um I'm going to talk about what we normally do, how we how we normally do it, and then uh what happens when we break our own rules. Um so this will be a kind of a rambling uh cautionary/inspirational tale. Uh and hopefully it will have some useful nuggets in it for you. I'm going to talk about the technology um a little bit. I'm going to get into language models quite a lot. Uh I'm going to talk about user experience design and touch a little bit on the on the writing, some some tips there for for doing stuff. Um these are kind of the the things I think of as the core of of narrative design, uh and I I think of a narrative designer as being probably the most appropriate person to do this by themselves if they were going to do that. So, just show of hands, who in who in the audience self-identifies as a narrative designer in some way or wants to be like a good

good good you're the right crowd then. And and how many have actually tried to do this, tried to make a voice skill or a voice app or anything? Not many. Okay. But some. So, that's that's good. Good on you for for giving it a shot. It occurred to me last week that uh this talk is kind of a travelogue. Uh so, let's learn about the country that we'll be visiting, which is the country of voice. It has some peculiar characteristics that other countries don't have. Uh for one thing, your your input to one of these stories is intermittent. Um the the story is not listening to the player all the time. It can't, it's not allowed to for legal reasons, privacy reasons. You you as the developer are not permitted to just open up the microphone and listen all the time. So, turn taking has to happen, and it's super important for the audience to know when it is their turn to talk. So, for that we use very explicit prompts. Um and you could use a sound effect or a bell or something like that, but we like to stay in fantasy with everything if we can. So, the way we do it is we just have a character ask the player a direct question at some point

in the story, and then you the player are expected to answer that question. That's how we let you know, "Okay, it's your it's your turn." There's a writing corollary to that, which is that that is the only time that anybody asks anybody in a a direct question in one of our stories. Uh and having having to write a whole you know, a couple of hours' worth of audio in which the characters do not ask each other direct questions and never uh speak in rhetorical questions strikes writers as strange and uncomfortable at first. Uh it's not though as hard as it sounds. You get used to it. Um second characteristic is you must make your decisions quickly uh in these in these stories. Alexa, for example, gives you 10 seconds to talk and then it'll shut the microphone off if you haven't started saying something. Uh it'll it'll repeat the question one time and then it'll shut the skill down entirely. So, you have to kind of be able to make that decision quickly. Uh this is mostly also about not leaving the microphone open for uh privacy reasons.

So, for that we we just make sure that when it is your turn to make a decision, your options there are are clear, simple, and and they are few in number, and we enumerate them for you right then and there. And And so, that dictates the nature of how you would write that that prompt. It's sort of a uh a quick summation of of your options at that point in the story. Thing three, voice recognition is really cool, but it is not magic, amazingly. It's technology. Uh so, with someone who is mumbling or turning away from the microphone, or maybe standing in a kitchen where dinner is actively being prepared, the sound can get kind of muddy. And if you give the voice recognition muddy sound, it has a much harder time doing its job. It's easy for it to get confused. So, we try to make uh make it easier on the voice recognition itself by uh making its decisions easier and we and we make the things that it is listening for also uh clearly distinct and and few in number. So, we offer the player clear options, but we also offer them specific

ways to say what they want and we carefully manage those phrases, again part of the writing of these things, so that they do not sound similar to one another and all of that makes it easier for the voice recognition to do its job under difficult circumstances, which difficult circumstances are more common than you might think. And these prompts tend to get uh a little a little awkward and they they sound they sound stilted and they stick out, but actually that's good because it makes them work better as prompts. You sort of sit up and take notice cuz somebody's talking funny at you uh and you go, "Oh, it must be must be my turn again." Uh so, we will actually often go so far as to capitalize while we're writing the the key phrases in the prompt to get the voice actors to hit them harder and sound even more um stilted and and authoritative. Uh so that then you're you're you're again you're sort of reminded, "Hey, it must be my turn." Realism is not our primary concern here. It's it's functionality. So, putting all that together, we we wind up with this model where the stories are turn-based and they're

structured so that we can uh focus all your options uh on a single moment at at any time and then those moments are embodied in these prompt questions of two or three distinct actions that that sound quite different from from one another. That's the usual model. And and and these are good rules. However, if we followed them all the time, this talk would be less fun. So, uh last year, Amazon Studios got in touch and they asked us if we wanted to do a story on Alexa and exclusive um uh next to uh this um new show that they were making, this Jack Ryan show, which sounded kind of cool. So, we said, "All right, we're in. Let it We live for that stuff." Uh so, we delved a little bit deeper with them and the inspiration for what they wanted to do was this escape room that they had also commissioned as a promotional effort for the show and they had run they built it and they ran some teams of influencers through it and they did a twitch stream of that and it was I watched it it was it was pretty fun. Everybody had a good time. And one of the peculiar aspects of this

escape room was that Alexa was built into it. There was a device in every room and it sort of um uh became part of the puzzle so on. And their idea was they just they just had such a good time with the escape room they wanted to bring that feeling to a Jack Ryan story on Alexa that you could then experience in your own home. So, what that meant was we were doing an adaptation of something that was already a kind of a loose adaptation of a TV show that we hadn't seen and we hadn't read. Um That's okay. That's that's that's just what the job is like. More on difficulties getting to see the show a little later in the talk but for now uh we wanted to um we wanted to make this story feel like an escape room and and central to that feeling is this idea of exploration. You want to sort of poke around and open things up and find clues and put them together in a very sort of self-directed and free-form kind of a way. And so one of the first decisions that we made was we would take our usual wisdom of this connected structure of discrete moments and throw it out the window. So, instead you'd be constrained only by

what chapter you were in or what location you were in. And instead of asking those very pointed questions the the nice prompts of uh here are your three options we would ask the much more general question, "Okay, what do you want to do next? Tell us anything." Um so, we were already getting ourselves into trouble we knew at this point and uh the problem was something that we were already aware of from the days when adventure games allowed you to uh type in your input at the keyboard uh using uh the English language. And the problem is that there's a huge difference between what you can express with your input mechanism and in both cases here, the English language, capable of expressing just about any thought you can have, and the stuff that's actually addressable by the game, which is some tiny subset of that because the game is is made by human beings and in a finite amount of time. You just can't cover all that space. Uh so, all this blue space here in the middle is frustration. Um the audience

gets really frustrated with this stuff because we're not just allowing them, we're actually encouraging them to try things that are doomed to failure. Which is not a very friendly thing to do for your audience. Um notice this is a problem with voice control in general. This sort of unfettered freedom of speech is the fourth characteristic uh of the of the medium. That, by the way, is the Constitution of the United States, freedom of speech. You get it? Mhm. So, we've been avoiding this problem up till now because we've been using these clear options with specific phrasing, and that uh clarifies the addressable space moment by moment for the for the player. One of the reasons why we do it. So, we we realized that open structure or no, we still needed to clarify a useful range of actions for the player. And these are the ones we picked. They're basically what you would find in an adventure game of the sort of point-and-click variety. Uh and they're solving the same problem uh that they do in those in those old adventure games. So, the idea was that any time a player would be able to uh do any one of these nine things,

and there would be a a list of about 50 targets in the in in the game, people and objects and stuff, that you would be able to do these things to. So, the ramifications of that bring us to the next step in our uh awesome uh travel uh paradigm. Uh let's talk about language for a second. So, how many in the room know when I'm when I'm talking about voice recognition, what I mean by the phrase language model? Anybody encountered that? Okay, not too many, so we'll we'll explain. Uh so this is this is actually kind of a big focus of the talk because it's it's totally unique to voice recognition medium. And even if you aren't the one putting it together, it will affect your life. It affects everyone's life. So, the idea is there is this voodoo black box that exists between the player and the story. And when you're when you're playing one of these things, you say something, and then that thing that you have said gets uh digitized into a waveform, and it gets sent off over the internet to

Amazon or Google or whoever it is that's doing your your voice recognition. And then you what you get back from that is two things. You get an utterance, which is its best guess at the actual words that got said. And then you get what's called an intent, which is the important thing for you. It's a tokenized meaning. Uh so it's it's uh and you help define this. It's basically um in the in the space that you need it to be in for your uh your particular skill. And you need it distilled down like this to make your life possible. You don't want to be messing around with processing the natural language. That's an AI hard problem. Leave it to the professionals. Don't try and do it. Um but you do have to help this black box out a little bit because it's not magic, it's technology. And what you do is you provide it some sample phrases, and you group them into intents and label them according to what you your own your own intent names. That's sort of a a rough guide for uh the black box uh for how it's going to interpret meaning in your

story. So, this example is actually is all about uh just wanting to know what the temperature is outside. Um there I've given it some phrases here that address different things. There's one that's doesn't mention the the hot or cold at all, it's just about wearing a coat. Uh notice that I I will get now into a little parser problem of my own here. There's no way I'm going to be able to enumerate every possible way in the English language that you can say something that indicates an interest in in what the temperature is outside. Uh fortunately, Alexa, when you send this off to to build the language model, will happily uh expand on the examples. Uh that's good. It does it invisibly, which is less good, so you sort of don't know what's going on there. Um so the good news is it it will recognize things that you didn't think of, which is great. Uh the bad news is it will recognize things that you didn't think of, and so you'll get some some false positives in there. Uh which is another reason why we try to be very specific about what our users should say.

So, coming back to this Jack Ryan thing, now we've got these nine uh actions and about 50 objects. Um you multiply those those together, it's about 450 intents. We we tend to have a lot of intents in our skills, maybe 100 or so, but this is already on the way to crazy town. Fortunately, there is a uh a feature that's supported by a lot of these things um that you can use to ameliorate that. Uh Amazon calls it slots, we call it entities internally. Um the basic idea is that it's going to allow the machines to do more of the heavy lifting around all of those intents and that 9 * 50. Um so, the way it works is instead of having a separate intent for examining the drawer and examining the cabinet, examining the padlock, and specifying those all uh individually, you'll just have a one way of saying, "Okay, I want to examine a thing." So, that utterance that thing will have a "examine the object, look at the object" and different ways that you might phrase that. And but you leave a a placeholder in for the object, and then elsewhere

you define that list of what can go in there, and that's there that's your 50 objects, and each of those can have its own little set of utterances that that define it. And then the computer does all the hard work of of meshing those together. And then when you when you use it, what you get back is both the name of the intent, "I want to examine something," and also the target of that intent, "I want to I want to examine the desk." So, that takes your 9 * 50 and makes it into 9 + 50, which is much more manageable. Now, we hadn't uh used this much, so uh I built a little uh test case prototype out of the first scene, and I threw some some inputs at it that were uh designed to be as confusing as possible. I I had some utterances in there that differed only by one syllable and meant different things. I swapped words around intended for it to um still recognize that. I used switch as both a verb and an object. Uh it did really well. I I did none of that stuff confuse it actually. I was very impressed. Context helps it to to get the right idea from these things.

So, eventually we would get into trouble with some of the built-in slot types that they have, uh but that's later in the talk. For now, I I'll just give the general advice that whether or not you're expecting that language model to be problematic, you should test it as early as you can because it'll come up. And again, don't forget about real-world situations. If your office is like a nice quiet place, try confusing the thing some some way. Challenge it. So, now that I've got my technical proof of concept, I started thinking about breaking down the actual story. Um in our amazing travel travelogue paradigm, I'm going to call it itinerary. Now, the source material in the escape room, it's it's more premise than story. And the idea was you were CIA agents and you're in the safe house run by the CIA, and for some reason it had these uh these Echo devices all over it. You started in a small room, and then you broke your way into a big room for reasons that were not clear. Then you uh your handler calls you on the phone and says you got to find this

McGuffin that looks like this. So, you uh solve some crazy puzzles. And then you find the McGuffin, and then your handler immediately tells you you got to find another one. So, you solve some more puzzles. And then a cool thing happens. Your handler, Foxtrot is her name, shows up in person, and she takes the things from you that you found and then she kidnaps one of the people on the team and takes them away. That's cool. Uh then everybody gets thrown into a death trap and then they escape and then that's pretty much it. It does There's no ending to this story. You You don't find out what happens to your handler or the stuff she's stolen or anything. And also um typical of of escape rooms, there's a lot of ludonarrative dissonance in this thing. There are a lot of puzzles that are built using thematically appropriate elements but which once you get them into puzzles and think about them, they don't really make much sense in terms of of either story or character. This bothers us. We're a story story studio. We want to do a little a little more than that. So we Most of that stuff got thrown out.

Um in an adaptation, you want to keep whatever works and is strong. For us, that was this betrayal thing was the coolest bit. Uh so we made that the middle of the story and and now it has some some build-up and some consequences. We've decided we'd have the the team dynamic rise and fall a little bit and there was a test of the kind of cohesiveness of the team after that moment of betrayal as after you've discovered that there is a mole on your team, there's some sort of infighting and you're asked to take sides in it and it's all good stuff and then way down at the end we added you get to try and catch the bad guy which seemed important somehow. Um Notice that most of this is now about the team and the people in the team. It's always a good thing for a story is to have people in the middle of it. Um back to this idea of itinerary for a second on a trip. You want to go to some different places at your destination. You have different things to see and different things to eat. You go to Spain, you want to you go to Barcelona and you go to Granada and they're very different from each other and that's good. You want some variety. Similarly,

in your story, you want a little variety in in tone and in tension. So when I get to the next step, which always for me is to do a broad outline of the the whole thing, 10,000 feet, just looking down, um I decided this time to give each chapter an inspirational tag, just one word about what it should feel like to be in that chapter. Uh, as a guide for making the pieces different in terms of what was the player activity in there and where the where the tension comes from. Escape rooms typically use the giant timer on the wall taking down an hour, um, as as their main, uh, tension generator. Um, we kept being pressured on this project to include them literally. These people are like, "Well, okay, this sounds great, but where's the timer?" Uh, we thought we could do a little better than just having a number. Um, stories are good at tension and so are games. So, let me let me get a little bit into the the kind of breakdown of the different chapters. The first one was had to be both the hook and the tutorial for the experience. So, it needed to be simple, and needed to be short, and

needed to be fast-paced, and it needed to be exciting. So, we decided to, uh, chuck the timer in that scene and replace it with something you can't really use in an actual physical escape room, which is a death trap. We had a room that was filling up with water and a character who was in that water, uh, giving you very uh, frequent updates on how she was about to drown. And, uh, you were basically talking to her on the phone and and trying to talk her through a puzzle that was going to get her, uh, out. And so, the water is getting higher and higher and time is running out. Uh, you can actually fail this scene, um, and she will pretend to drown. Uh, we want you to worry about, uh, the threats that we're making and and take them seriously as both as a player and as as a person invested in the story. So, it's common for us to put something in at the beginning that you can fail and kind of have to start over, uh, before you've invested too much time. We refrain from actual death in this scene. It's a little too early, narratively speaking, to be killing somebody off. So, this is actually it's a training exercise and they're all kind

of just pretending to do their parts, but, uh, you can fail it and you have to you have to start over and do the scene over again until you get it right. We also at this point introduce the the idea of this bad guy, Kessel, who is the the architect of the original version of that trap that you're essentially training on. Um he's a a terrorist guy who not the um the guns and and bombs uh authority, but he's the guy who moves money around for for all the other terrorists. He's um uh smart guy who kind of likes to uh challenge himself against uh you, the allegedly intelligent CIA agents, by trying to catch you in traps. And also he's the kind of person who when he leaves his Wi-Fi password around on a post-it, he'll do it in code in some way that only he makes sense to him and and that you will have to sort of puzzle out. Um this obviously is always is just a way to ground all of the crazy uh escape room kinds of things you're going to do uh in the in the story. We wanted them to at least have an excuse for why they were there. And so most of the most of the story takes place essentially at

his house, which is a crazy place. His name, by the way, was very poorly chosen. Now without come later, I'll explain why. Um so in the second chapter, uh you and your team get together and you go to Berlin, you go to his uh his headquarters. Uh he has several, but this one has just been discovered by uh your your compatriots. And you're there, you're doing exploration and and and puzzle solving. This is the the part of the story that feels the most like an escape room. Uh there's codes and and and things like that. Um you're you're uh you're discovering the secrets of the place. Um the the difference between this, I think, and a regular escape room is that uh it you're you're more of a safe cracker and it's all about breaking into things rather than breaking out of anything. You're you're trying to get in under the secrets. Uh it didn't make much sense to put a timer on the scene at all, so instead what we did was we put some traps, booby traps in the room, and sort of made a lot of noise about those, and we we set one off pretty early in the scene so that you would take that seriously. And then they uh they find out about that there's a mole uh on the team and uh and get worried about that, and they

start um start sniping at each other, and and also at that point you find it a a another small room uh which is where these people keep all their records. And so the the player activity changes here. It becomes more informational. You're looking through files and and uh finding stuff in photographs and combining things, listening to audio recordings uh all gathering evidence with this idea that you're going to find out who the mole is and it's becomes increasingly uh likely that it's somebody on your direct team. Uh you find evidence pointing to both of the people who are physically with you in the room and they start getting upset with each other about this and they ask you to take sides and that's where most of the tension comes from in this scene. Again, there's no timer on on on that scene. Then uh Fox Trout, the handler, who has been offsite until now, shows up and betrays everybody and then knocks them all out. And then uh then you wake up and you're you're in an actual death trap and and it has an actual timer finally, but we've attached it to a bomb just to add a little extra oomph again, something

that you probably don't want to do in a real escape room, but we're not in a real escape room. And what the player does in this scene is a little bit adventure gamey. You you uh put together a thumbprint kit out of stuff that you find in the room. You go around, you're just trying to find the right thumbprint that'll let you unlock this phone that's going to let you stop this bomb before it blows you up. Also, one of the one of the team members is missing now and you're not sure if he's in league with the mole or what. You don't know why he's gone. You don't think about that too much uh right away cuz there's a bomb ticking down. But as soon as you as soon as you uh deactivate that bomb, you get into the next room and you find him and now he's been uh locked into one of these uh one of these water traps. And it hasn't uh you recognize it immediately from having trained on it at the beginning of the piece. Um and you it hasn't been triggered yet and you know how it it gets uh it gets triggered if you uh if you turn off the lights. Um so there are it would be possible to leave him behind and there are various practical and emotional reasons to do that. For one, you're still not really sure if he's on on your side or not. But

also practically speaking, Foxtrot is now getting away with the the things that you spent time finding. And you have to make a call as to whether you want to spend the time to try and get him out of there and take him with you. And it's kind of a either way, makes sense. The trap turns out to have some twists so that your original solutions don't quite work. Makes it sort of a trap revisited. And both of these things also they have time limits on them so you can fail them. If you fail the the bomb, again, no one is killed by that but it does destroy all of the evidence that you were hoping to take home. So you'll take it on the chin from your supervisor if you let it go off. And if you let the the water trap fill up all the way then the guy in it does die. And you will also take take it on the chin from your supervisor for that. Doesn't like it when you fail stuff. Then the final final chapter is more of a chase and and Foxtrot is is getting away on an airplane and you find her car and you're just kind sifting through it. The puzzles are informational again but but on a timer this time. And you're just trying to find out like

where she's going, what flight she's going to be on, what name she's going to be traveling under so that you can locate her and stop her. And since since there is a limited amount of time before the flight takes off, you can fail this scene too. Doesn't uh ruin everything but it does mean that you don't catch her and this guy gives you a gives you a hard time about it. He does your performance review at the end. We I think there's about nine or 10 different ones that you can get depending on what you have screwed up and what you haven't. So this is a guy from the show. It's James Greer. He's the director of the whole TFD operation. Speaking of the show, by the way, we haven't seen it yet by this point in the process. We've gone pretty far. Stay tuned. So next step after that outline that I we always do is a is a detailed design and the idea there is we're going to break it down into what all of the individual interactions are going to be in the whole piece. And that's normally for us is about engineering a lot of these moments of of discrete choice. For this project, we're talking about sort of what are the objects in the in

the chapter, what happens with you when you interact with them, how does that drive the drama in the in the story, and how does it how does it resolve? It's basically adventure game design, which fortunately I know a lot about. I usually work out the dynamics using traditional methods. And adventure game design is is kind of a different talk entirely. One which I would need another hour or three to cover. So, I can't really do it justice here. So, instead, let's talk about what happens when you do do adventure game design on an audio platform. There is one thing in particular that comes up here that doesn't come up elsewhere. Um it has to do with memory. Now, normally in a game when you need to interact with a whole bunch of stuff, uh you have a handy persistent visual like this to remind you of what all that stuff is. Just uh there's a lot of things to remember. Uh but on a on an audio platform, you got nothing. And we run very quickly up against the limits of human short-term memory. And

let's Don't take my word for it. Let's try it. Um So, imagine you're playing my game now, and you're standing outside the lair of a known terrorist, and you speak a code phrase to get in and the door opens. You go in, and one of the guys on your team starts describing the room to you, and he says, "Oh, there's a there's a desk here. It's kind of a cool-looking old antique one. I think probably uh probably bought at auction. Um there's some some old rusty old lockers, kind of very industrial, bolted to the wall over there. They don't really match anything in here. Uh there's a nice-looking map on the wall. It's framed, and oh, the frame has hinges. That's very interesting. Uh right next to that, there's a cheap poster. It's a periodic table of the elements. There's also a an old CRT monitor. It's got to be 15 years old at least. And on the on the floor, there's a rug, which is maybe the most garish thing you've ever seen in your life. It's all primary colors, clashes with everything. Well, let's start at the beginning, you say. Let's look at that desk. Well, okay. Well, it looks like somebody's been been doing some work here. He's got a notebook and it's open and there's a

paperweight that's holding it open, a little glass one, kind of nice. Uh there's a bottle of something he may have been may have been drinking here and also the desk has a drawer and that drawer is currently closed. Oh, well, we can't have that, you say. Let's open the open the drawer. I want to I want to see everything. So, the drawer is is stuck. It doesn't appear to be a lock on it of any sort. It may just be stuck. Um one of your one of your team members is carrying a knife and she offers to jimmy that drawer open for you. Yes, please do that, you say. And then a booby trap goes off. Kaboom! She's she's injured and there's a little sequence where you have to kind of patch her back up again. The end of that sequence, she's like, all right, I'm ready for action. What next? Now, possibly you do have a very concrete idea of what you want to do next. But, likely you have also forgotten at least half and possibly three quarters of the objects that have been mentioned so far. And that problem only gets worse as you go further into the into this game, which I said is going to take more than an hour to to complete. Now, normally we don't have this problem

because we've been orchestrating the context so that you don't have to remember more than three things ever. But, because this is a broad and exploratory experience, we have to take measures to aid the memory. Our case here is is kind of extreme, but the lesson still applies pretty much everywhere. Memory is a problem in an audio medium because you just don't have those persistent visuals reminding you of anything at all. So, as a general writing note, you should be repeating the important information more often than you think you should. Again, make it sound a little bit stilted if you have to because realism is not the goal here. It's it's sort of game functionality. Uh in our case, we could implement a simple list of available objects on demand and we did do that. However, it's a little bit unsatisfying. It's the list gets long, and it's hard to process while you're listening to it if you if there 12 or 13 things. Uh and it takes you right out of the fantasy to have somebody just listing off the objects in the room. So, one of the better and more effective things that we did was to contextualize

that information. We wrote a bunch of status reports. That was just a thing that you could ask a character for. You give me give me a status report. And instead of just listing everything for you, here's all of the context that you have encountered so far, it would sort of take that into account. What have you finished with? What have you not started? And what what maybe have you been working on recently? And so, it would focus on only the things that would be interesting to you right at this moment. And this is part of how we're directing the experience that that no one can see is by kind of focusing the player attention with these things. You can also do a lot of redirection with uh the responses to other player actions. It's kind of like doing hints. When you do hints in an adventure game, you sort of if if someone interacts with an object and they do the wrong thing, you sort of hint them towards doing the right thing. And if they do do the right thing, they sort of hint them to what's the next interesting thing. Um that all that stuff can be used uh to hint at drama as well and kind of direct

the direct the story experience. Uh these status reports are doubly useful. Remember that we had this constraint of limited time. You had to answer quickly, or the skill was going to drop. Well, it turns out when you're solving puzzles of the escape room variety, this is sort of inevitable that the you're going to have to stop paying attention for to the skill for a while and write something down on a piece of paper, and either you will pause it, or it will drop of its own accord, but either way, you're going to have to start it up again. And so, we needed a way when someone was starting the story back up again, whether it had been 1 minute or 5 minutes, or they had gone to work and dinner and come back after 12 hours, uh Uh, a good way to just sort of jump them back into the story and sort of remind them what they needed to be thinking about and these status reports were perfect for that. So, anytime you did start the skill from from nothing, if you had a bookmark in the middle of the story, we would throw one of these at you. Super good. So, now ordinarily, after I get that detailed design all done, what I like to do is start the implementation and create a skeleton upon which I can

then write all the dialogue. I leave the dialogue writing till the very last if I can. However, I don't always get to do that and in this case, we had a a script approval in the contract and our partners needed a fair amount of time to to get that turned around. There's a a whole chain of custody that goes from you to the producer to takes a while to get to the actual filmmakers or or um TV people to read it. Uh, so the script got written on paper first. Um, this probably happens, I'd say, about half the time. We wind up having to do this just the uh, productions being what they are. Uh, the script, incidentally, is about 160 pages, including a bunch of pseudo code in it to explain how the different pieces fit together cuz it's not entirely clear uh, because this is a non-linear sort of uh, sort of a thing. Uh, also, incidentally, right there at the end of the script writing process is when I finally got to read some of the scripts from the show and actually see the show. Um, time for a little sidebar about studio security in Hollywood, which is

fun and interesting. I'm going to preface preface this part by saying that uh, our our partners have been fantastic about this stuff. They've always been really helpful and pleasant. Uh, I've had good experiences with them. This project was no exception. Amazon Studios was really good to us and I I would work with them again in a in a minute. Um, what you run up against though are processes. They're not designed for you. Um, the processes for working with anyone outside of the studio are mostly for people who are doing things like making action figures and lunch boxes. And those people don't need the kind of access or feedback or level of detail that you do when you're trying to write a complimentary piece of fiction. On top of that, everybody is super worried about leaks and you cannot blame them for that. If the there's recent history has has borne out a few things and if your show winds up on the internet before it winds up on your own streaming service, that's really bad. So, consequently, as a as a partner, even one under non-disclosure, it can be pretty hard to get hold of a script.

Now, this Jack Ryan project was actually pretty smooth. They were they were willing to let me have stuff, but what they didn't trust was the internet and the mail. So, they weren't going to send me anything remotely. I had to go and meet with an actual person and she would screen the show for me and then she actually handed me a pile of scripts at that time that I could take home and then use as reference. The reason that it took this long is that we were all really busy and our offices were 400 miles apart. I'm not in LA. So, uh it just took a little while to to get things going. I've definitely had worse. In fact, let me let me explain about that one time when I had worse. Uh one time I had to fly to Los Angeles and I they took me up to a weirdly deserted floor at the top of a tall building and somebody brought me the script and he took the little pin out of the script so that I could move pages individually and he brought me a notepad for taking notes on and a special red pen for taking notes with and then I sat there with the script for a couple of hours and then they came and they took everything away and they scanned my notes. They did not just hand me a thumb drive or anything. No, I had to fly back home

and then they sent me the notes a few days later inside of a secure app that I had to put on my laptop and I had to go through a vetting process to get a login for that app and the notes, of course, couldn't be exported from that app in any way. You had to view them only in the app. They were encoded. Uh and then there was a second app that I had to install on my phone to do a numerical two-factor authentication every time I entered the password, which was about every 10 minutes unless I kept tapping the window, which of course I didn't cuz I was trying to work and I just wanted to have the notes up there on the screen next to me. So it was like having the notes inside of a safe though when the door kept kept closing on me. That project turned out fine, too. It's just a a little bit inconvenient. I just want to let you know what you might run up against. Anyway, 160 pages of script later we got it sent off for approvals and now I think I can finally do the implementation. Yay. I like that part. Um some people do use code for this. We have a tool that that

lets you draw little boxes and put stuff in it and and connect them to create the structure. Um this isn't it by the way. This is a Visio drawing. I was asked not to show an actual screenshot of our tool. It's some kind of a trade secret or something so I took that out. Um but it basically it looks kind of like this and it lets you uh make little boxes that have the the dialogue for each each piece and your audio is going to get popped on there and then there are different kinds of transitions that connect them. Some of them are direct and some of them are like you're checking conditionally some things that may have been done previously in the skill or you can check the date even or something like that and just go to to one of the different set of destinations based on that. And then sometimes you can make it stop and listen for some input from the from the player and go different places based on that. So it's good to to get a working version up as soon as you can. Like I said, I like to do it even before I write the dialogue if possible and the reason for that is that you want to you want to test this stuff. You want to get it in front of some people who you don't know and who don't know your skill and

who don't know what it's supposed to be. And that way you can test all your assumptions and and fix some problems. Uh the more more complicated your piece is, the more important this is. Uh you start with a text-to-speech voice or you can even do sometimes we'll do a scratch version just me sitting at the desk like speaking into the microphone uh before you spend the money to record actual actors cuz that is by far the most expensive thing that you will do in the whole process. You want to fix fix stuff before then. So, speaking of fixing problems, let's talk about some stuff that came up cuz it's always it's fun to hear about other people's misfortunes, isn't it? Uh there this is the first full test of the language model and there's always some things that come up with that. We just we try to be careful with the writing but it's sort of not possible to avoid everything. Um things like door and and drawer, like we've got two objects in our list that sound very much like each other, easy to confuse. This tends to work out though okay because the door and the drawer are in different parts of the story. There

aren't any times when both of them are both available and interesting. So, we can combine them essentially and assume that the player is referring to whichever one of them matters right now. There's there's two ways to do that. We can either do it on that structure that I just showed you and we have an input for door and an input for drawer and they both point to the same thing. Or you can literally combine them as as part of the language model and just make them synonyms of each other and you'll just get one token back that's, you know, door drawer or something like that. We had a few other things like that in the skill. Um note in the notebook, the table and the periodic table, confusable under some circumstances. Um again, they were in completely different chapters so we just combined them. Uh we we did have to change a padlock code. We learned by experience that eight is perhaps the hardest number to detect of all of the numbers. Um it's it's a really there's not much there with an eight. Eight eight it's just like this short guttural bark. It's like a tiny tiny amount of sound. So, it's hard to compare that to anything else that might be a tiny tiny amount of sound. So, we

changed it to seven and this was a puzzle that was actually based on that periodic table. Uh so, we're actually exchanging oxygen for nitrogen which, upon research, didn't make no sense at all. It was fine. Um okay, remember this guy? I said his name was a bad choice. Uh, anybody want to guess why given what we've been talking about? Yeah. So, there's a built-in required intense on Alexa, which by convention ends the skill if there isn't anything hanging around to be canceled, which there never is in in our story. And it's what we call a universal intent, which means it's valid at any point in the skill. Uh, so we can't do what we did with the door and the drawer and make some kind of positional decision about which one of those it is. It's just going to get confused and it's going to send a cancel intent, and we're required at that point to shut the skill down. So, this guy has Thank goodness we tested that before we went and recorded anybody expensive saying his name. He has a completely

different name in the skill now. Never name anyone Alexa, either, in case you're wondering. Not Not Not that we did that, but yeah. Uh, there are some other built-in, um, intents. Stop is one of them, and that obviously does stop the skill. Um, we already knew never to include stop as a player choice. It's just one of those things you avoid. It comes up a lot, actually, when in a story you want to stopping something to be one of your options, and we just avoid that if we can. Um, the list of phrases that are listed that are listened for for these built-in intents is not public. Uh, you don't know what what's in it, but I can tell you now from personal experience that one of the synonyms for stop is that, turn off. So, unfortunately, we had more than one puzzle that revolved around turning something off, so we couldn't really engineer that out of the story at this point. Fortunately, we got we got like it doesn't really cause trouble too often because if there's context for it, if you're turning off something, the the language model is smart enough to pick up on that, and it'll send us a turn off

the thing intent instead of the turn off the skill intent, and we don't stop. Where the trouble happens is if somebody is mumbling or turning away from the microphone at exactly the wrong moment, uh, that can take part of the speech and make it sound muddy enough to be misinterpreted as background noise, and then you lose it, and you just get the turn off, and then it does ask you to turn the skill off, and you have to do it, or you will not pass certification. Exit is another built-in thing. This is like a hard reset. That you can't even be nice about it. With a stop, we get to send little messages to say thanks for playing. With exit, it just cuts off instantly. Exit also has some hidden hidden synonyms. Uh among among which we discovered was closed. Now, what are the odds? Out of nine actions, we pick two that are hidden synonyms for please stop the skill now. It was Here again, because of because the players are providing some context about what to close, it usually doesn't get confused, but uh had I known in advance about turn off and close, I

would have tried to just design the thing around them, and I would advise you to do that, too. One more thing about the language model. I'm certainly talking about the language model a lot, um Amazon has some great pre-made slot types for for recognizing things like numbers and the names of cities and the names of countries, things where there are a lot of elements and potentially a lot of ways to say them. If you think about the numbers, you know, 1,017 isn't the only way to say that. You might say 1017. You might say 1017, especially if you're entering a code on a keypad, and every number has that same problem. So, rather than actually build that thing yourself and list out all of those potential ways to say those things, it's good to have something provided by Amazon that you can just use. And I got excited about the possibilities of these, cuz they're all kind of good for uh code-based sort of puzzles. And I I used four or five of them in the in the skill. Well, one day we we discovered that um no had stopped working for some reason. You couldn't answer no to any question in the skill, including would you Are

you sure you want to start over? Which is a good thing to be able to be answered no to. Some investigation revealed that no was actually the word was getting returned correctly, but it was being flagged as a country. Go figure. Um, country was is a big intent. There's a lot in it. Um, and so it would capture a lot of extraneous things that that weren't listed somewhere else in the model. So, if you like if the player just said banana for some for some reason, it would say, "Well, that sounds like a little like some of the countries in my model. So, I'm going to I'm going to flag that as a country." It tries to tries to put something into one of your boxes as much as it can. And that's and so that was one of the bigger boxes that we had. So, a lot of stuff went into it. This is the only time though that something that was literally specified as part of another intent got trapped by that somehow. And we cannot explain this. We have no idea why that happened. Uh, we don't know what's in the built-in intents. That's not public information. And again, we don't really know what the language model does with the the data that we give it up front. It's it's voodoo. It does not explain itself, and

that's why you need to test it, test it, test it, and then test it more. Uh, as it turns out there are not so many countries. So, we just took that built-in intent out and made our own, and everything was fine. Now, here's probably our biggest problem. It's nothing to do with the language model. It's all us. Um, once we got this thing in front of real people, uh, we realized we had not been nearly clear enough about this list of actions and the fact that that was all the all the useful things you could do. And people were being plagued by the parser problem, which I mentioned earlier. Uh, they were trying to do all kinds of stuff. They were trying to chisel things and climb things and make a rope with their clothes, and they were getting frustrated. And they were quitting the skill and sending back bad reviews from the uh the test bed. Fortunately, this was not live players. This was test players. Uh, so in retrospect, our tutorial was was not at all well designed in this one respect of establishing that addressable space for the player. So, we we added some some extra guidance from Alexa here, put some more some more

comments from her in there, including a lengthy, lengthy monologue that comes up front before the story even starts. It really just hits you over the head with like, "I'm going to give you this list of things, and that's what's useful and nothing else. So, stick with that." Um the advice here is just be as heavy-handed as you can about describing that addressable space. Uh it's not possible to overdo that, and it's super important to get that in the player's mind. We also um we discovered as we always do I shouldn't point my head directly at this mic, I don't think. Uh we discovered as we always do that that a couple of the puzzles were too hard. That's we just expect that. We just we just don't know which ones it's going to be, honestly, until we get it in front of people. Uh and mostly what we do for that is we just wrote more hints. Uh the hints in this game were were in fantasy, you would just call headquarters, and they would sort of contextually try and give you some advice on whatever you seem to be working on. So, they're actually kind of fun to fun to hear. Now, eventually you have to stop fixing stuff uh because your recording date comes up

on the calendar. It's some some uh minimal number of days before the date you actually have to ship. You have you have pushed it as late as possible. You've pushed it a couple of days past that, but eventually it it comes up. Uh so, you got to do it. And And you may not be doing recording. You may be making a skill where you want to do everything with text-to-speech. And you can do that. Some people do do that, and it's cheaper, and it's easier, and you can fix stuff right up right up to the last minute without uh getting anybody in the studio. But, we feel that good drama and good story need good actors. It's just kind of a philosophy of ours. So, um that's Cat Cressida, who is a a good actor, plays Tango in the in the story. Um now, audio production is kind of a a known quantity that you should be on another another track to really hear about, so I'm not going to get into it too much. But, I'll say this about the writing for it. Um this is a a non-linear uh kind of experience. Like I said, it's broad and exploratory, and you're recording these chapters full of audio for things. And so, you need to be aware of the

potential differences in tension and tone between the different lines because they may be playing next to each other really crazily out of order. So, it's good to think about that a lot, kind of compress everything into a smaller range of tone and tension within a scene and just kind of go for an overall tone and annotate them like crazy. Like this is tension level two and this is tension level four and this is tension level six. So, that way you can have like for example in the scene where the water is filling up the room, a real rise in what you're getting back from the character who's who's in that trap about how dire things are and yet still have it mesh with the responses to your puzzle actions that may be interleaved in there anywhere. Some other stuff you might want to know about as a as a writer on on Alexa specifically you have a time limit between your player interactions. You're allowed four minutes of of audio in between. Now, four minutes is a long

time. Let me tell you that is an eternity. Writers in the room you really have to keep the pace up for for interactive audio. If you go much over a minute between the player actions in one of these things it's it's even more important than than something where you have a visual. If you go more more than a minute that better be a deliberately slow scene. And if you ever get to four minutes I'm coming over to your house and smack you on the head because it means you've forgotten that the player is the most important character in in your piece. Just ignore them. One other limitations specific to to Alexa is you get you only get five audio files. So, if you're doing something complicated and you thought, "Okay, I'm going to be clever and this character who may or may not be in this scene I'm just going to record their lines separately and have them in separate files and drop them in in the places where they would go." Well, you're going to get to five different files pretty quickly and then it's just not going to work after that. It's very restricted if you're if you're doing anything complex. Also, uh gapless playback is not guaranteed um on most of these platforms. And so, uh

again, if you've got something uh sort of going on in the background, your scene is in a diner, and you've got you you want noise in there, plates clinking, and stuff like that. Uh you won't want to hear like these weird silences in the middle of that. So, you what you're going to wind up having to do is instead of piecing it out and playing several different files back-to-back, you have to make one big one, and you have to make a whole bunch of different versions of it to uh to take care of what are all the conditions that might be possible going into this particular moment. Uh I try to limit that to 32 versions just to keep myself sane and to keep my uh sound designer from coming to my house and doing mean things to me. Okay. Pretty good. Uh now that your audio's done, you kind of plop it into the structure, and uh and then you're you're kind of ready to go. Once Once that piece is is done, uh there's one last little formality you have to go through. You do There's a certification process. Uh Amazon's got to sign off before you can go live. It's like going through customs. And a lot of

the things that they will flag for you are functional things. The engineers will will worry about those. But there are also some uh content policy guidelines that you should just be aware of. Sex and violence is uh that's a typical thing. I mean, every platform has that, right? Don't they? Violence, by the way, includes unsettling content, {uncote} um there was a demo that we made for a ghost story a couple of years ago that we think was perhaps the first skill to be rejected on that basis. It was just a little bit too um graphically grizzly and and and descriptive. Uh the ghost story was fine. It was just the the demo that they wouldn't let on there. So, watch out. You're also you're not allowed to run ads, and you're not allowed to recommend other people's skills. That just means Amazon doesn't want you um getting in the way of their space of being a store. You're not allowed to be a a store there. They do that. Uh which is reasonable of them. Uh hate speech is explicitly prohibited. I was at Twitter, please take notice. I was glad to see that in there. Um there are lots more, you know, maybe just uh look them up

before you do anything questionable. Now, uh Jack Ryan only failed certification a couple of times and it was largely on these these technical grounds. They were and they were really fast about it. They they got back to us um within uh usually uh within a day and then we would fix stuff by nightfall and just kind of send it back. The whole process took uh I think less than a less than a week and then we went live. Uh fortunately uh be in time for the the launch of the show, I'm I'm happy to say. Um Now, we can do we can do dynamic content with our system, so we continue to uh to monitor and and tweak things uh even after launch. But basically, that was it. We have we have arrived at our destination. Uh now you you are going to get momentarily uh an email. I'm going to just uh remind you to fill out the evaluations for the for this talk. Um the recommended uh excellent rating is is highlighted for you. If you ever want to hear me speak again, go ahead and give me that and then we can move on to uh I think we got

a few minutes for questions. That's good. Thank you. Another microphone somewhere? Uh I don't have a way to do that. It would have been cool, but it it is out in public and it is free. You can just uh ask Amazon. It's it's called uh uh Jack Ryan November morning and if you just say open Jack Ryan to Alexa, then she'll open it up for you. I just wanted to uh thank you again for the talk, but uh wanted to ask um do you use Alexa like the voice to text or text to voice? Um do you use Alexa's voice uh sometimes and what are your design considerations when you're choosing voice acting versus Alexa? Uh well, we generally treat Alexa as sort of the host of an experience. So, she's she's always outside the fantasy. Uh just kind of handling a little bit of maybe tutorial stuff. Uh kind of getting you ready or any kind

of help messaging that you that you want to ask for in the middle of the skill. And then everything else we try and keep in fantasy which which doesn't include Alexa. And even though in this thing the um the escape room the original inspirational material included Alexa as as part of the of the story. It rapidly became clear that we weren't actually going to be able to do that for various reasons. One some were functional because we can't from within our skill access any of Alexa's other abilities which is the whole reason that she was in those rooms was to add things for you and stuff like that. So, we would have had to recreate all of that. We didn't want to. And the other is that uh Amazon is understandably a little bit protective about what words you put into Alexa's mouth. And if you make her a character in the story you got to portray her in a very particular way. We didn't want to. And is there any reason you wouldn't just voice act everything that Alexa had done? Uh like beginning tutorial middle parts, right? Well, I guess why would you use her versus everything Oh oh oh

as opposed to having a human sort of be the host. Sometimes we do have a human host and usually it's me actually. We have a few things where where I am I come on to give you a few a few pointers. Uh the tricky bit about that is actually just updating stuff. Like I said, we can we can update live after we've after we're already out. And if you decide, "Oh, we need to change this messaging." It's easier to do it with text to speech than than sending somebody into a studio. Now, if it's me uh you know, that's not very expensive. I'll just go and do it for free. It's fine. But uh uh if it's anybody else then then it's trouble. Hi. One more and then we've got another one. Don't worry. Yes. Uh great talk. Two-part question. Uh your 50 objects, was that for each chapter or the whole story? And in hindsight, was that too many objects, just enough, or not enough? Ah, okay. Was it Okay, so the 50 objects was for the was for the whole story. And actually, it's not even all objects. Um about six or eight of those things are people. And you use them in in different

contexts. There's one uh one of those actions was contact somebody. So, you're allowed to kind of call headquarters or try and call your boss or um the various other other people. Um yeah, that was that was actually plenty. That was plenty of stuff to do um you know, kind of more than an hour worth of material in and the that second chapter actually feels very big even with the the amount of stuff. There's probably only about 15 or so things in there. Um yeah, it's pretty That was plenty. I would I would hesitate to add any more than that. Thank you. Uh you're placed on a lot of choicy stuff. And now there's this sort of parsery text adventure. Uh do y'all feel like oh yeah, now we've discovered, let's do let's do lots of parsery text adventures from now on? Or do you feel like, oh man, uh that was a fun experiment. Now we're going to go back and do choicy stuff for the rest of time? Oh, okay. So, he's asking is is this now, having done this, is this going to change our paradigm and now we're going to do this all the time?

Or are we going to go back and and and just do the kind of momentary choice-based stuff? I think the answer is neither of those is the case. Um we this was uh you know, it highlighted some problems for us, but we figured out how to deal with them. And we certainly could do it again. And and I think given the right situation, we will. It's a very different feel from some of our other stories. And so, those are those are two models that I would use in completely different situations. And uh I think we will not limit ourselves to those two under any circumstances. One of the cool parts of my job is I get to find new new ways to use the technology, which is uh I'm looking forward to doing more more weird stuff this year. Thanks. Thanks for the talk. Um I was wondering, having worked with both on both visual games and audio games, what you prefer about working in audio? Ooh, what do I prefer about working in audio over visual? Um mostly I like the the the sort of um

inexpensive, comparatively, uh a kind of garage operation feel. I like to work small. Um that lets me experiment more, take more risks, and and and just generally have more control over things. Uh I I can I can change a lot of stuff up until almost the last minute, you know, but we're doing voice recording, but that that comes pretty late in the process. Uh I've always liked that from from the beginning of the, you know, some of those earliest graphic adventures that I worked on. One of the reasons I liked them was the art was so simple that we were, you know, we're basically kind of doing puppet theater, and we could just see what we could make out of that without uh spending too much money trying to trying to draw everything and do uh uh you know, speak match the speech and stuff like that. Thank you. Hi, again, thank you for the talk. Oh, sorry. Hi, uh obviously you're a pioneer in this field, and you've done a lot of great work. What and you've got a lot of invested in your plate. What do you see as the future um What do you see as the

marketplace um in terms of the growth of the marketplace for these sort of audio these audio games, and what do you see sort of the future this going growing? What What kind of technology and aspects do you see coming next on the horizon? Marketplace and future. Ooh. Okay, so that's a that's a pretty broad question. Sort of What do I see in the future in terms of both both marketplace and and technology. Uh how much time we got? About about a minute. Um Okay. So, uh I I think we can I think the marketplace is is still kind of nascent. There are ways to make money in it, several of them. Um nobody sort of had a you know a breakout like this company made a billion dollars doing X Y or Z on it, but I think it's it's sort of like early days of other platforms, phones and and things like this where uh you're seeing people now starting to get into it and they're starting to have lots of these machines

in their houses that just listen to them all the time. So, certainly there's market penetration. There's interest in having things to do on the on the platform and there's monetization is is built into it now. So, there's there's just kind of um uh the things need to get shoved over the edge now where people are comfortable doing that. And I think that's that's right where we're at now and I think that'll happen in the next 5 minutes. Okay, and just Google the Amazon versus the Google. Which one do you prefer using? Which platform? Oh, I try to be agnostic about that. Okay. I try we'd like to publish on both if we can. Usually it's more of a business decision as to you know, they like exclusivity one the one or the other, but um yeah, we can we can we can go iOS and and and Google and Amazon and and Bose glasses frames if we want to and that that's my preferred thing. your future tech then. Yeah. So, we're we're at an hour. I think we have to go to a overflow room. Is that

am I accurate about that? I Yeah, okay. All right. We'll be out out across the hall in a in a second and thanks very much.