The Science of Desirability

[Music] today we're going to be talking about what is desirability as we define it in our research lab how specifically we study it and go over some familiar examples and then look at how we put this theory into practice and creating a new game that's called chef league that's launching very soon it's exciting all right so what is desirability look at this can ah what did that sound uh desirability is oh we've got to listen to it one more time it's about expectation and anticipation that an experience will be all that we hope or at least what we hope and maybe even exceed those expectations and this if you saw this can after hearing that sound maybe your desirability uh your

desire for that cat whatever the beverage is might go down a little bit um that's followed by an emotional appraisal of reality of the real experience that we encounter after that moment of anticipation so if the liquid that you find after you see this uh you hear this sound and open a can is this chickpea or maybe that's what you were really looking for and you're you evaluate and appraise that as uh positively but maybe you were pretty excited to get fat tire beer or something like that you'd be more excited if that's what you get or maybe that sound was making you think of soda and coca-cola might be what you want so a simple model that we can use is along the uh the y-axis there is is a person's point of view so that's their sensory anticipation their values driven or signaling anticipation and their emotional anticipation and we can have that first sound that we hear of the can and then we see yep it's a can so if you

are thinking of something that comes in a bottle maybe the can changes that and so depending on which one of these people you are hoping for a fat tire hoping for chipped beer or hoping for coke um this is going to play out differently for you all right the people who got fat tire in the can or the people who at the top of this are going to appraise that experience uh most positively of the others all right but the reality is that desirability and studying it is a is complex so it differs by people it differs from novice to expert it's always changing brands often get blindsided because of confirmation bias and under invest in listening to how it is changing over time i don't know if you're a fan of the dos equis ads but they this this says he gave his father the talk and it says can you get facebook on this thing and um i thought they were clever hats

um so how do we study desire desirability conceptually uh in from our point of view we're looking at it related to language and related to learning so i want you to all to think of a song that you recently came across that you heard and you liked just get a song in your head think about that everybody has one something all right and then i want you to try to articulate again in your head why you like that song specifically so for most of us we notice things that we like or dislike but we actually kind of struggle to describe them very effectively so you might be i don't know you might be into cardi b or this latest lady gaga bradley cooper song um any of these but we're sort of we're trying an experience we're noticing we're liking it but we're not doing this next step which is being able to know all of the labels

that effectively describe why we like that thing and enable us to find more of that thing efficiently i'm interested in this for a second reason which is related to how people advance and creativity so at the bottom of this uh triangle we have a little mini creativity that's when you're able to be creative for you so like maybe you put coconut milk in your coffee that's the first time you've ever done that that's creative for you but it's not actually that creative broadly speaking middle creativity is when you're creative in your in your domain and big creativity is really often what we're talking about when we talk about innovation and groundbreaking products i'm really interested in empowering more people to get to big c but it really starts with empowering them to get to little c and that's related to whether they ever develop that language to categorize and find experiences that they might

enjoy so most of us really instinctively evaluate what we like or dislike without learning to articulate why and i study uh how like what people mean when they like or dislike something even when they lack the language of of describing what they mean and part of how i do this is by looking at people who go further than that so there are some people who go further and they develop a desirability point of view that they can articulate so we go back to the song that you're thinking and how you try to describe it in your head this is what uh what experts use to describe whether they like or dislike a song a specific type of expert this is like a sensory evaluation type of expert so they can analyze in their minds the uh the tempo the tertiary time signature and a lot of other things that most of us lack the language to even really um differentiate our

experiences based on that so this uh this is more what that site that full cycle looks like so up in the blue corner you have noticing the songs you like or dislike and then learning the labels to describe why you like or dislike them and then getting to the point where you have clear criteria to structure those experiences getting feedback this is where products come in do products allow you to get that kind of feedback or not on your successes and failures and trying to find new music that you like learning to get to the point where you can critique effectively music that you don't like and learning to get a differentiated point of view that point of view helps you more efficiently find new things so some of the benefits of being able to have this full cycle are you are faster at analysis you're faster at search you can be more experimental possibly

even more creative you spend less time sorting through experiences that you really aren't going to like and you also because you can articulate your voice your voice is heard more by designers so you are more of a participant in the design of the things that you might later enjoy all right this applies to a wide range of fields some of you might know this scene from the devil where's prada i don't know if we have um people have seen and said she's she's looking at these two turquoise belts and she says oh they look they're so different and she's making a joke she thinks they look exactly the same and then she gets schooled by miranda cares [Music] who is the anna wintour character in this movie but anyway you i'm not going to go into it today but you can apply this across a wide range of fields beyond flavor beyond music beyond games many different types of

products and i'm happy to answer questions about those later if you have them so next how do we study desirability practically speaking so we've gone over what it is conceptually how we think about it but how how do we do it so we start by analyzing qualitative and quantitative reviews of experiences we start with small data sets and then we build much larger data sets we distinguish experts from novices and so we look at sort of where they are on that circle that i just showed you we turn distinct patterns once we have distinct patterns we turn them into algorithms that we can run and eventually create little simulations sort of ai that can make choices in our in our system i'll explain this a little bit more in a moment and then we also develop ways for us to continue listening so that we can avoid the trap of confirmation bias and then most importantly and this is a thing that i hope you'll take away from

this talk we also try to build into these products ways to help users develop that their ability their own ability to articulate and understand their point of view because that also makes them better at signaling how we could improve products for them so we're also in this case not just researchers so the game chef league that's coming out um this gives you a little overview of what it [Music] is [Music] so in this case this game is meant to be fun first but the more that you play it you also learn from the chefs that we have

built into the game they're real chefs and there are little chef ais that are based on the data that i'll walk you through here so i this game is also very personal to me in the sense that this is not funded by grants or anything else it's taken me three years to build it uh it's been brutal uh but it's so fun and three years in we're we're on pre-order in the app store right now and uh and i still play it and i think that is kind of amazing because you all know how exhausting it is to make a game but i really love reality chef shows but i'm also a pretty introverted shy person and i don't want to be on reality tv so i thought that it could be sort of interesting to learn from these in a game and i like could you beat bobby play every day you know on your phone sort of privately and so it took the first couple of years to build the data for this and then it took the last year to build the game so it starts back in 2015 with this flavor genome project and so we started categorizing how food tastes

to humans which is actually the poorest data that we have on flavor so we have a lot of data on distribution of food and beverage items we have a lot of and very rapidly accelerating scientific data on flavor compounds that are in different things but we don't have a lot of data on how food tastes to people and the diversity of pallets that we have and so we started looking into that and linking it to emotion and culture and cultural associations that people have and then we went into the small scale data development so this is qualitative and quantitative reviews with real chefs we're really focused on how they like did they like something or not and how they described it so let me give you an example of what a helpful articulation versus a non-helpful one so a non-helpful review is that was gross like we it doesn't

give us very much information about what is wrong with the with the item or how it might be improved a helpful review is that this is bland it needs salt and experts know that salt can come in the form of salt but it can also come in functionally it can come in the form of feta cheese or some something that is salty and brings other properties uh to that right so that's what that's what we're talking about when we talk about the difference of someone who has this language versus doesn't have this language uh and then we move to much bigger data sets so at this stage we're looking at about 1.5 million different reviews from real chefs of real recipes we begin to be able to differentiate people just be based on the language that they use by expertise and by degree of experimentation so we also know did they make a thing so the experimentation is being done that way

so this is an example of a an expert palette versus a novice palette that we were able to ascertain from video so we can we can then build as as i said before we can turn this into algorithms so we can begin to move through new uh recipes with these algorithms and see go back and forth between real chefs and this data to see how that matches up so there was a show it's not on the air anymore it's called the taste it had anthony bourdain bless him nigella lawson ludo um and it's had some other celebrity guest chefs so we look we analyzed the history of that show the show is really focused on one bite of food and then they don't see they're blind to who made it and they talk about it and they um give it an evaluation of this person moves forward or not and so we looked at the raw data for each of these and then we could also

simplify that eventually to see that there are distinct patterns in how these chefs are voting on this reality show and because this show is blind to the personality of who made the food it's particularly interesting from an analysis point of view we did a bunch of other experiments we looked at chefs primarily in the boston area and learned how they think through flavors and particularly how they invent new dishes and what that process looks like so that we could modify our our ai chefs in in this game and we've collaborated with mit and harvard on some different things this delightfully paired feelings for dinner was looking at this was an emotional menu there was no food or beverage item listed on the menu the menu items were things like a slap in the face and and then that got interpreted by the chef and the sommelier team

and then we built the first version of our software and we hosted a dinner where it was sommeliers versus software for um pairing um foods and beverages and then the audience decided what they liked better and um this the software did pretty well it did about equal to the smallies um which was better than we had hoped at this point so this is an example of what that early software looked like you could go through it and you could look up any food any beverage and it would tell you like critical factors related to it and it would also tell you what chefs think this goes best with this was interesting it was interesting for a very nerdy person like me but i don't really think it's a product that other people really wanted so this we hadn't yet turned this into a game but it was a critical step along the way

so then we got to the point where we could turn these algorithms into ai chefs that could make choices so we could um with a new recipe data set we could see how nigella lawson would likely vote on new items that would come up and we could differentiate across these three and then the next step was in terms of making it into a game is how can we make you versus nigella or one of our other six ai chefs possible and then that comes back to our language question how can we build this game in a way that capacits you to get savvier about the language that you use and so um takeaways from this before we open it up to a question in in the products that you're building thinking about how you might model the sensory expectations the emotional expectations and the signaling expectations of your

users and the key moments over time and see what that path might look like and how might you add feedback loops particularly if i go all the way back here to this diagram where along this circle might you be able to add feedback loops that would capacitate your users to not only get smarter for their own benefit but also to to communicate more effectively with you about how you might improve your products thank you happy to take questions people have questions yes could you maybe uh describe

the uh the the overall thesis and kind of premises of the desirability lab and like what the what the main kind of like mandate and thesis is sure so um that's a big question we are primarily uh about education like when i showed this diagram here like this is a big part of who we are as a lab i do a lot of work with companies for products and then it's not always about education but first and foremost in in my lab i'm looking at how do we empower our students to move up this you know this diagram to become really effective creators from their point of view and surprisingly one of the things that shocks me the most about harvard students they're super smart that's not surprising but many of them haven't you know they're really good at following the rules but they haven't done a lot of the work of

developing their own point of view that they're going to design from and so that's the thing that i personally care the most about and that my lab is has embedded in everything that we do and then i think to do that effectively it takes us into some of these other spaces that we talked about like i need to be able to take them through this cycle for themselves and it needs to be open to being around their interests i also from a from a broader market thesis point of view i think people are going to get a little bit tired of products that are sort of addictive and not not designed in their best interest and you know call me an optimist but i think that they will really appreciate products that are also designed to help them develop their own point of view and become smarter in different ways and i'm really interested in experimenting in that i could be totally

wrong but that's the that's the experiment i wanna i wanna do i hope that answers your question yeah oh yeah hi um one of the things that your talk made me think about and like one of the really important things as a game designer is getting user feedback doing play testing things like that and i think one of the biggest challenges there is getting good feedback because you know oftentimes people don't know what they want and so that's what this sounds exactly like but it also sounds like this is something that's kind of built up over time and then working with someone over a large period so if you do just have like an afternoon with someone or even a couple hours do you have any kind of methods for getting that better feedback and getting them kind of up to speed on on the language quickly yeah i guess i mean you you're like talking about a little experiment design right so you would have your upfront explanation of what the key terms that we're going to be looking

at in this period of time might be right yeah or just you know you you put your game in front of them and you you say like you know what's your favorite part or like that you know it's hard for them to kind of answer that because they don't really know necessarily what goes into making a game they might just say you know i don't know i like halo make it more like halo when right you know that's not exactly what you're going for yeah it's a it's a hard problem especially in the time frame that you're talking about i i empathize with that i think so some because you also want the unadulterated feedback right so the way that we would usually do that we don't do a lot of specifically the frame that you're talking about but we would do the normal first part and then we would have probably some worksheets that have terms on them that are key terms that we're trying to like so you're you're saying you want it to be like halo but like which of these terms would really is what you mean and i think people

would be pretty good at that it does mean more work for your team to set that up but you're also going you'll get more valuable data from it cool thank you yeah and then like the broader picture how do we how do we educate them in school maybe even so that they're coming out with those terminol that better terminology yeah yeah i don't know how this goes yes please hi uh i wanted to say as a ravenclaw i'm incredibly delighted that you would work towards making games to help people learn because that is totally up my alley but my question was a lot of times when i talk colloquially informally into people co-workers about games this is a designer that really interests me what they like about it but they have a tendency to sort of poach vocabulary from reviews and things like that in a way that doesn't genuinely express their own idea they're just sort of mimicking what they read or heard through youtube or metacritic

or things like that would you have any any way to approach that so that you could get more personal and authentic feedback from them and these are users or or designers on your users um i mean i think it's probably a little bit of a similar answer to the last question uh first of all it's very common that people do that and what he was talking about as well that is normal like the people who manage to get this full cycle are not normal they're very very rare um but that's why like these two brown bits those are the bits that i think are you know realistically achievable where people could over time learn more of the more of the terms and that that we could be building products that give some feedback loops that give them the terms for what they're experiencing and evaluating so some of this i think you can do inside of a game so i don't have it showing up

here but inside chef league there are these ingredient challenges and so it's like you know do you know what a strawberry really tastes like and most people don't they you know they they're up in the notice things you like they know if they like or dislike strawberries um but we then give them this little quiz where they have to move along sliders you know how acidic is it how bitter is it and and the reason we're doing that is to give them the vocabulary and to get them to pay attention to those differences so i don't know if you have the opportunity to build a little bit of that into your game in some way um i'll see if i can yeah and if not i would give the same answer as before how can you in your user testing expose them to the terms that might more effectively describe their experience no i appreciate it thank you yeah thank you some of this also you can do automatically so like in songs you know that they like or dislike this song

you don't need to ask them about the tempo you have you like you can figure all of that out on your own yeah which is what echo nest did and econos was bought by spotify for 100 million so there's some value to getting this right uh yes cool yeah uh thanks for giving this talk yeah so in the trailer for chef league it looks like you are asking uh potential users in the future to parse a lot of data and like assign a lot of variables and there are a bunch of sliders i saw that they're going to match up with acidity and things i was just curious if you could talk a little bit about the ux that went into not overwhelming your users with all the stuff that you're asking them to consider when they're sort of playing through these challenges sure i mean that video is like on ultra speed ready to get into the 30 second window that the app store requires but um yeah they're all voluntary challenges so the ingredient challenges where you're moving sliders are really kind of practice challenges so that's only the users who want to do that that are going to do that the really fun challenges are where

you're competing against chefs and our you know little chef ais and those are to fix recipes and all you do is pick a flavor to fix it so there's no you don't have to move any sliders or anything um but it's a really good point so how do you make it really easy for for people to pick to pick up the game and we've tried our best to do that i mean you know the market will tell us soon when we when we when we drop it sure okay thank you please hi um so i had a question more about like on the disliking end of it um how do you educate people that might be on the undesirability scale of things um so basically what the the concrete example that i was thinking of is often gamers are reacting to things in games without being educated about the issues that prepare those changes so things like social issues or like representation or things like that

when gamers are reacting to things and like um they're not educated about why those choices were made and they see it at a very surface level and they feel like it's i don't i don't know what their problem is but um so how do you educate are there any techniques for like breaking the barrier of like dislike into education yeah let i want to spend a little bit more time with your question let me go back to this diagram here so um i mean i think there's a difference between educating a user and empowering them to use the like more accurate description of their experience right and on that far side you have like what are their expectations so it's key for you to get a sense of what is the expectation of this user that you are then presenting something to them that they're disliking and it's violating their expectations somehow the value isn't really getting to that expectation

because it may be that they're actually their expectation can be aligned with what you're trying to do but uh the way it's happening it's not they're not giving you're assuming what they're what they're saying and they don't have the language to to more more uh effectively articulate what they're saying so this is hard to fill in but i think that is ultimately your goal is to see like what are these people hoping for in terms of the sensory experience in terms of the when i say signaling experience what that really is it's related to meaning and value so they might be they might have a fundamental belief about what a type of game should allow them to do and that's a that's a value that is coming into this uh this simple diagram and uh and you're somehow violating that and as long as that is still happening you're still going to have tension there

but it may be that there's a way to unlock that i hope that's helpful feel free to follow up after if you want more detail for that thanks so much yes so hello and thank you for the talk i want to ask a little bit about your method of validating the quality of the software you've built so presumably you'll give it to some people before release you'll try to figure out if this is going to satisfy them to a certain extent [Music] what is your approach to testing if what you have is you know good sure so i mean it's an indie project so we are a little bit limited in what we're doing um but we first tested with friends and family and then we we have you know qa remote testers that we sorted out a lot of bugs with and then we've done load testing and then we've come back to you know mechanics testing at this stage

um i'm talking more about not the not the bugs side of it but the satisfaction side of it so you mentioned how difficult it is to get the the truth out of people yeah so somebody says you know did you have fun with this app or do you did you like this game like yeah yeah i mean we can see whether they continue to play it i think that's a that's a that's a meaningful one um i don't know i mean i don't think i'm going to surprise you with my answer here we're doing our best to do what you all try to do uh i don't we i explicitly send them a little simple survey after they test it um on like is it fun would you play it again would you recommend it to others and then i look at do they continue to play on their own when we're not you know looking over their shoulder or they're not expecting a survey people have been pretty brutally honest

so like at least so because i also say to them like i want to fail now and so i want you to i want you to be as brutally honest as you can now like that's the biggest favor you can do for me right now so and then we'll see i don't know there aren't there isn't really a game that's quite like this on on the app store there are cooking games uh and they're like animated they're often managing a kitchen kind of games and then there's the food network and so you know i don't think this game is going to be for everybody but i think there are probably a few other people out there who who want this and we will see and even if they don't i'll keep playing [Laughter] be an expensive game for one thank you yes i think they were getting cut off yeah what i call dinner okay thanks if people have more questions i'll just be outside thank you