Lessons from 'Duolingo': How to Make Learning Hard Things Easy

[Music] hello everyone um before we start today i'm supposed to remind you that you're all season pros at this point to please just turn off all noise making devices um and also that a survey will be sent out at the end of this talk just please fill that out it'll be super useful for me and super useful to the organizers of this awesome conference to figure out how to make the conference better for everyone so hello my name is karen ty i'm a senior engineering manager at dual lingo and thanks for joining me today on this final stretch of an amazing conference i'm here today to present lessons from dulingo how to make learning hard things easy now how many people have used duolingo before oh my gosh that's so awesome to see so many of you well if you don't know what duolingo is we're an app that tries to teach you foreign languages for free we have over 300 million users now and we're frequently at the top of the

charts for top grossing and top free education app on both google play and ios so you've probably heard of duolingo but you surely haven't heard of me so let me give you a little bit of background in undergrad i majored in computer science with a minor in neuroscience focusing on reinforcement learning so the intersection of technology and of neuroscience and learning has always been a passion of mine i'd like to think that i also have an unofficial minor in league of legends because i probably spent more time on league than on my neuroscience minor and in the fall of 2012 i joined duolingo as an engineer about three months after it launched so i was able to see the app grow from under 1 million users to now over 300 million and all the lovely bumps along the way nowadays i'm the engineering manager for two teams at duolingo there's the courses team which tries to make sure we're teaching the right content and the core learning team which aims to

ensure that we're teaching that content to our learners without lowering engagement these teams are constantly having to think about how to teach foreign languages to people without driving users away so you might be wondering what's an education app like delingo doing at a place like gdc well it's simple really we want games i mean games are fun we wanted learning a language to be fun so naturally we wanted duolingo to feel like a game when we talk about product when we talk about design about monetization we talk about games not other education apps at dulingo many of the mechanics you see in the duolingo app were inspired by games that you've created but the main reason we're here is that the challenges we face are very similar to that of any mobile game we all live in a world of really engaging apps and activities that all compete for our time as creators of a mobile education app we

had to think really carefully about how we can compete for users time when the nature of what we do is traditionally thought of boring confusing or at best really difficult we try to teach foreign languages and we need to convince users that they should be learning instead of doing something else people are not usually super motivated to work hard at something voluntarily especially when they aren't forced to when there's a teacher in the classroom they can enforce learning or help students get unstuck when they're stuck and school was mandatory so you were guaranteed to have students for a certain period of time and can employ really effective learning methods in helping them learn what they needed to but on an online platform where players are sacrificing opportunity costs of playing something else you have to make sure the user is willing to get through on their own even within apps there may be parts that you need a user to get through that are

just less exciting or harder to get through than the rest of your game and if it's too hard or too boring you'll likely lose that player forever so the gaming industry right now is incredibly saturated and it's definitely not easy to make a successful and appealing game that stands out among the crowd if the rise of hyper-casual games are any indication lowering the barrier to entry on games is becoming more and more lucrative when designing an experience that appeals to all types of players we found that we have to think along a lot of the same dimensions as games do we have to accommodate differing rates of skill acquisition for example some users will be really quick to pick things up and some will just need a lot more time and the same goes for learning a language our games should try to be flexible and allowing slower players to go at their own pace there's also play context players might only have a small window of time to play your game or might they

might be ready to play your game for hours on end if you offer something in your game that offers a satisfying experience for both short and long moments of play it opens up when your game can be played there's also prior skill transfer if a player is already familiar with your core genre or familiar with games at all they'll likely need less onboarding than someone who has never played a similar game before and we need to make an experience that accommodates both of these types of users and finally there's motivation we have users learning a language for travel to connect with old loved ones or in many cases looking to better their socioeconomic statuses we have users who also don't actually care about learning a language at all but just want to feel productive while doing something fun you likely have players playing your games for differing reasons whether it be to achieve mastery to be social with friends or just to unwind and you'll cast a wider net if you

create a good experience for all of these types of motivations so duolingo is at gdc to share how we've tackled the challenge of teaching something hard to all types of users while keeping them around for the last six years we've learned a lot through both successes and failures about what works well and what doesn't when it comes to teaching things to online users i mean in the end every game has a learning curve and every game is going to benefit from having a learning curve that doesn't drive users away let's make sure that that learning curve isn't holding your game back so for the next 45 minutes or so we're going to talk about how to teach stuff while keeping things engaging first i'm going to spend a chunk of time talking about metrics everyone's favorite thing i'll share a bunch of av test results with you of times that our intuition was wrong while talking about how we measure things what things we measure and why we

a b test everything then i'll go over five principles on how to create learnable content we have a lot of really talented learning experts in the company that help inform product and engineering on how to create a good learning experience and good learning curriculum then we'll dive into how to go about presenting that content in a way that drives engagement and is still good for learning outcomes i'll share a few tricks for what we use to make the whole experience motivating and then finally we'll recap with some actionable takeaways and leave a few minutes for q a so first we'll talk about metrics which is likely something all of you have a lot of familiarity with this presentation will be full of metrics to demonstrate the impact that applying certain principles had on our numbers many of you already constantly probably look at user retention ltv dius and all sorts of three-letter acronyms

and many of you have at least heard of a b testing which is a core part of duolingo so really quickly here's a quick overview of what a b testing is if you want to test a new experience you take your users you split them into two buckets on one bucket which we call control the users get the old experience they continue doing exactly what they've been doing in the other bucket they're in the experiment condition and they get the new experience you're trying to test you take a look at the metrics for each group and see which one's doing better to decide whether or not your new experience is worth launching so duolingo is obsessed with av testing every change to the product and we believe it's the main reason we've kept growing to date we've run over 800 tests and through them we've steadily and incrementally increased d1 retention from 13 in 2012 to 55 now and i've picked up a bunch of these results to share with you today simply because

they help illustrate the problems we're trying to solve so here's a quick tip also for a b testing if you're already doing this on your platforms not every user is affected by an experience you're trying to test for example if the change you're testing is on level three of a dungeon you really only want to take a look at the users who reach level three not the users who are still on level two and not the users who are way past level three everyone else is just noise that lowers the statistical power of your a b test and if you don't have many users to start with it will be really hard to run an a b test that gets statistical significance if you don't filter out this noise so if you plan on a b testing things which i highly recommend you could take your analytical tools further by only measuring users that would actually be impacted by your change to reduce noise only count players who would have been exposed or are exposed to the changes you're testing and you'll get a lot more out of your metrics that

way so as a disclaimer the numbers in this presentation are based on relative increase between control and experiment buckets and not absolute numbers unless otherwise indicated cool so the first tv test i'm going to present to you is what i call the capitalization crutch and those who use duolingo are probably very familiar with this so here's an exercise from dulingo with a word bank to help you out and it's asking you to translate this english sentence into spanish using the word bank now can you guess what the first word of the right answer is yeah you actually don't need to know any spanish to guess it's the one with the capital letter right it's boss and users said this made the exercise way too easy and honestly it was a bug that it was capitalized in the first place so we wanted to fix it but since we a b test everything we tested removing that lower casing instead of just doing it so now vos is not capitalized in experiment condition and we were hopeful that this would be

fine and we don't want to give away the first word anymore so this test actually revealed an absolute d1 retention drop of 0.5 when we did this so unfortunately to this day we still capitalize the first word in these word banks until we can figure out something that works better the lesson learned here is that even the tiniest change in difficulty can have really large and significant consequences on retention this is a problem because learning a language well and efficiently is pretty hard and we knew if we made the app way too easy people might just not learn anything at all so here's another one we often ask learners to translate the language they're learning into their native language so if i'm learning spanish i might be asked to translate a spanish sentence into english this is a lot easier than for example trying to translate english into spanish because i wouldn't have to come up with the spanish words myself i just need to

recognize those words and know what they mean in english however it's a lot less useful for me when i'm trying to have a conversation in spanish and i don't get to practice creating my the language i'm learning enough so learners ask them asked us to give them more translations into the languages they're learning because it would let them practice more this was a top request on our user run forum and app reviews etc and we thought it was a really great idea so we ran a test just flipping one out of about every 20 exercises to the harder translation type translating from one direction to the other and on ios this test yielded a decrease of 1.5 daily active users when we did this one out of 20 challenges was enough to have these results so this goes along with the earlier lesson that by making things harder it's probably bad for daus and retention but what was harder to swallow about this test was that what the community asked for and what their behavior showed

wasn't always aligned we encountered this frequently after this experiment as well on other experiments we've run it's always important to listen to your players but it's also good to remember that certain channels like the forum or app reviews might skew towards vocal minorities players have a tendency to propose solutions that they think will solve an underlying problem but might end up hurting instead so try to find the problem instead of taking a solution at face value and don't feel pressured to implement the solutions your users demand if there might be a better one users will often speak through their behavior as well as words so so far we've seen that making things harder even in an effort to teach things better often leads to hits and engagement metrics like daus and retention but what about just encouraging users to do a little bit more right if you do more in the app you're going to learn more language and it's going to be awesome

so in an effort to let our users pick their goals we offer different tiers based on predicted time per day we want you to spend on duolingo more casual learners could commit to just doing duolingo for five minutes a day whereas more serious learners could strive for 20 minutes a day however what we found was interesting using using the casual goal as a baseline is d14 retention for all other goals were just way worse learners who chose the most aggressive goal had 23 percent lower retention than learners who chose the most casual one now of course a lot of factors could account for this it's likely that some of it has to do with our platform being better for casual learners and for more advanced learners at the time but other experiments have also shown us that when learners try to do too much at once they quickly burn out and leave so teaching harder stuff doesn't seem successful and asking them to do a lot

also seems to do poorly sometimes so what things actually help this is one of my favorite experiments this happened the first year i was at dolingo hence the retro art style here on duolingo we presented we present little bite-sized chunks of knowledge in the form of what we call lessons and several lessons make up one skill so here you see a skill with four lessons if you finish all the lessons in the skill you got this trophy icon to light up at the end so in one of our courses we accidentally added another lesson as part of an a b test nothing else changed on this page there was just an extra box there however oddly this increased click rate on the very first lesson by three percent extremely significant result and just to emphasize how weird this was that means when a user got to this page before they clicked on anything three percent more users clicked on that first lesson than in the control condition

just because we don't even know this could have been because the trophy is now closer to it maybe they misread it and thought they'd get the trophy if they like did lesson one we still aren't sure but what this did tell us is that the weirdest things help sometimes and we learned early on that user behavior is just really hard to predict without observation and that's why user testing and a b testing are so vital to what we do day to day all right one more so in the spirit of tests that didn't make sense to us i want to share one more experiment for the first time we decided to allow users to opt in to a change we were making to the inner case that we call crown levels which i'll talk about a little later we have this cute little modal that pops up and asks you hey do you want to update now and try out this new experiment because of our a b testing culture which our users quickly picked up on users in the past had complained about not being

offered a choice whenever we changed major things so we tried offering them a choice of course as an ap test the results were pretty good though we had no gains in da user retention sessions were up by 14 for users who had a choice between updating to the new experience and not however when we looked at the branch of the experiment that didn't give users a choice and just forced them into the condition the metrics were actually way better we believe that it might have been a combination of the new experience actually just being much better but also suspect that sometimes choice might cause regret or delay acceptance we had spent so much effort on this opt-in experience but it turned out to actually not be helpful at all you could see that the the d1 increase is huge there in that condition so what all of these experiments tell us is that what's intuitive isn't

necessarily right all the time and metrics really help us keep accountable by telling us whether user behavior actually agrees when we make intuitive changes when it comes to making learning things in your game better you have to really question if what you think helps actually helps so to do this you need to measure the right things as well you gain a lot of insight by measuring everything you can within reason for example we're able to identify lessons that are particularly hard on duolingo and try to fix them because we measure things like time taken quit rates exercise failure rates everything we can possibly measure about how users are doing their lessons we're able to fix things on a very precise level and target what we need to improve for onboarding or tutorials it'll be critical to see the funnel of how many players drop off along each step maybe there's a step in there that everyone is getting hung up on that you could make better

also regularly question which metrics should be used to make decisions sometimes it's really easy to cheat the metrics so they go in your favor for example we wanted to maximize the six if we wanted to maximize how many lessons get done on duolingo all we have to do is make the lessons way shorter right the number would go up drastically when investigating the success of onboarding or a tutorial in your game don't just look at how many players got through it you also have to make sure that they actually gained the information you were trying to deliver in that experience if you want to make sure your learning experience is both effective and engaging you have to be able to measure both things so the bottom line is measure results not just activity it's great if you run a test that increases onboarding completion and maybe you removed a step to do it but it's likely that users actually might understand less because they no

longer have that stuff that you removed at duolingo we try to hold ourselves accountable by measuring learning and not just completion recently we created exercises aligned to the cefr language standard which is a standard language level framework the questions that we created are independent of our own content curriculum and we designed an experience called the learning quiz which looks like a normal experience in duolingo but is actually there to measure how well our users are learning it helped to tell us not just if we were successful in teaching the things we wanted to teach but even if we were teaching the right things at all we can detect if we're really bad at teaching certain grammar or have holes in vocabulary we're teaching because of this standard for games you'll probably need to create your own internal measurements of whether or not your users are picking up the right things try to bake in learning indicators to see if they learned what they need to

all right so metrics are important let's assume you have a good analytics framework to detect where improvements can be made and you can keep yourself accountable for it the first step of making hard things easy to learn is to make the content itself more learnable we'll go over five principles that we apply at duolingo based on insights from our learning experts so the first one is to quickly deliver satisfaction and what i mean by this is to teach something immediately satisfying first it's important for us that within a user's first few minutes on dueling go they'll learn a full functional sentence in the language they want to learn we don't start by asking users to memorize vocab flashcards or read grammar notes with the promise of eventually being able to use the language we make sure they learn the language immediately we want to give them the satisfaction to say i can do this now and it didn't even

take that much effort so find a small but complete and satisfying piece of your game play and make players gain that experience as early as possible you probably already do this to an extent i mean i don't know any non-tabletop game that requires you to learn all of the rules and mechanics right away before you can do anything but it's likely that there are some things that you can still put off teaching until a little later just to deliver a fun experience a little faster you only get a small window to prove to your users that their time spent will be fruitful and satisfying and fun and if they can't be convinced of this in the first few minutes they might leave and never come back number two on how to produce learnable content is to remember that though instruction is useful at times it does not replace practice if you want really efficient learning often it's tempting to give very

detailed instructions especially when it comes to grammar so here's an example from our klingon course and yes we have klingon on duolingo here's a paragraph describing the grammar of verb prefixes in the third person it says if the subject is third person he she it they and they and has either no object or a third person object etc it describes in great detail when to use a prefix on verbs and this is actually incredibly useful and many learners love this but there are also a lot of users who won't bother to read all of this text and wouldn't understand it even if they did manage to muster up the motivation to read it what we lean on instead is providing a lot of practice for a topic at hand as the learner does exercises in our app with the goal of leading a learner to form their own generalizations so i'm about to try to pronounce klingon so i hope there are no native klingon speakers in the audience because i'm

going to butcher it so for example we might present the sentence yasha mara i'm supposed to say klingon very aggressively is what i'm told so that's what we'll do uh it means does mara understand but means mara do you understand here you can start to suspect what adding bl to the verb does here here are two more examples does torg understand and then torque torque do you understand so now just based on these four examples you can probably infer that the bl is needed on a verb when talking directly to the person but not when talking about them by having tons of practice learners not only have a less intimidating experience but also start to learn things implicitly and create their own rules for how things work

the brain is an amazing thing and it can learn complex things really intuitively in fact it's actually better for long-term retention of knowledge to learn implicitly rather than explicitly the best thing to do is provide enough exposure to encourage this type of learning so in breath of the wild i thought the game did an excellent job at giving really lightweight instruction and tutorials for battling they presented new mechanics in fun ways and shrines you needed to complete to progress one such mechanic was how to pair you with your shield the instructions say press a to deflect an attack with your shield right before it lands for a perfect guard alright cool but the game didn't try to describe to you in excruciating detail precisely how to time it or when to use it or what kind of attacks you could parry you just need to practice to get good at it and i don't remember how many times i got set on fire trying to practice how

to deflect a guardian laser but that's something i just really needed to get a lot of practice on even with perfect instructions i learned implicitly about which animations lined up with the proper timing and what things could be parried and that was not only more useful it was a lot more fun so the third piece of advice on creating learnable content is to cleverly disguise difficult or boring concepts it's a well-known trick to try to pair less exciting things with fun things if you want the less exciting things to get done so in the early days of dulingo on all of our units we had well all of our units which we call skills were named things like present tense determiners adverbs three prepositions we used to have what we call a grammar wall on duolingo a group of skills that explicitly tried to teach you difficult grammar and we found that many users dropped off at this point once they hit this grammar wall

now in newer versions of our curriculum we sprinkle that grammar into skills like recreation restaurant travel we use a theme like recreation to teach you verbs like run swim and play and it's a clever way that we can teach you how to use the present tense when you're just learning about how to describe your hobbies and what you like to do we can disguise a grammatical concept like how to ask questions in a skill about how to ask for directions the learner thinks that they're learning about travel skills and having fun doing it but what they're really learning is how questions are asked in the context of games you can make particularly challenging things into competitions or events you can try to bake more mundane but useful things like how to set certain settings into the game progression itself the opportunities are boundless but your users are going to have a lot more fun if they feel like they're working towards something that they want to advice number four when it comes to making content learnable is to offer a

lot of exposure of that content in different contexts on duolingo we have a lot of exercise types to add variety to the experience and also present many ways to learn something so at one point you might be asked to select a picture of a fish then recall how to save fish and then translate i have a pet fish all in the same lesson you know there are many different ways that you can practice the word fish and by the end of it we want learners to be very comfortable with it going back to the shield pair example from breath of the wild i learned what was the same and what was different between parrying a lynel a moblin or a guardian change up the environment and allow users to apply what they know to new situations you see examples of this in games everywhere like having different stages and items in smash ultimate or using the same items across many different champions on league of legends this will allow users to create correct generalizations

about what they've learned and how it applies in new situations it's also really exhilarating to discover you already know how to do something in a new situation and exposure in different contexts is key in solidifying cognitive learning and finally making things memorable also makes them more learnable this seems super obvious but it's easy to forget to be intentional about it so we actually have a twitter account dedicated to really weird sentences found on duolingo and some examples include there are 1500 cat photos in my cell phone where can we hide the bodies and you do not exist these are real screenshots from real users and actually some users criticize that these are not pragmatic and useful you wouldn't normally say these things when traveling etc but they actually serve a

real purpose for one going against expectation makes things more salient or noticeable you have to think about what the sentence really means maybe second guess yourself like did they actually say that instead of making assumptions in your games find times to make things you want the user to learn surprising funny or emotional in some way but just a quick word of warning for users doing something wrong is also memorable we all know the saying learn from your mistakes because mistakes are really easy to remember and learn from but make sure it's not a negative experience if you plan to rely on this for learning if you plan to lead the user to do something incorrect to teach them something provide an environment where the user isn't afraid to make mistakes or it could be really frustrating all right so now that we have learnable content there needs to be a good

framework to present that content so the framework we'll talk about today is difficulty scaffolding on duolingo it used to be that once you finished a unit of learning called a skill it was just done it would turn gold and then over time it would fade away to remind you to practice it but once you were done you had finished that skill about a year ago we changed the experience to something that we called crown levels it made it so that there were now five levels on every skill instead of just one for each level you gained you earned a shiny crown and we gradually increased the difficulty of exercises you got as you leveled up a skill you could then choose whether you wanted to level up a skill you had already learned or move on to learn new content this feature was critical to us for improving learning it provided a clever way to get harder material for those who were serious

enough to level up and we could finally inject harder and more effective learning material without killing engagement by just putting it at higher skill levels this allowed users to self-select this satisf it satisfied both casual learners who wanted cursory knowledge of the language to very serious learners who wanted to master everything and practice as much as they could so crown levels is probably our biggest experience redesign ever and one of our most successful for new users we saw an increase five percent in da use seven percent in sessions completed we also got a two percent increase in d1 retention four percent in d7 and six percent in d14 retention now if you wonder what i mean by new users we almost always run both new user and existing user experime experiments for every significant change new user experiments are for users

created after a certain point in time we decide what experience they'll get before they even start duolingo so they start with that experience and have a consistent one this removes novelty effect of certain features on the other hand existing user experiments or old user experiments test the effect of giving the new experience to users who have already seen the old experience and changed to the new one it would include novelty effect we saw huge gains in the old user experience as well sessions increased by 23 percent likely due to novelty effect but d1 and d7 both also increased by three and four percent respectively we don't have d14 retention numbers because we launched this experience to users before we could get them since we didn't want to wait any longer the metrics were just too good for both new and old users the experiment was a huge win for both engagement and for learning learners were gradually exposed to

harder exercise types as their account age increased and we were able to expose many more learners to harder effective exercises overall and only when they were ready for them so the lesson here and our advice to games is to find ways to scaffold difficulty it's certainly helped our numbers tremendously provide paths for both breadth and depth so that users can self-select into the experience they want naturally let players who love a feature go deep into it and let casual players stay more surface level until they're ready you'll likely be able to convert casual players to higher difficulty over time if you can keep them for longer and what's more is by having a framework like this it helps with the scalability of your game design if you can create a framework that opens up a lot of possibilities in the future it lowers the barrier of entry for new ideas and lets you iterate quickly at

lower risk we were finally able to introduce more exercise types with high learning value using this new crown system that we've always wanted to but never introduced in fear of scaring away newer users now we just introduce them at later crown levels likewise if you have certain opportunities open up at later player levels or skills you can make those experiences more intense without fear of scaring away the newbies how to present learning content all boils down to balancing cognitive load the amount of brain effort needed at any given time too much of it is exhausting and too little of it is boring our crowns change let users self-select into how intense they wanted their learning experience to be so now that we have a framework we should also try to make sure we don't overwhelm our users within that framework one key thing is to resist the urge to

get to the good stuff right away make sure that users are prepared for it first the boss battle might be the most exciting thing in your game and you can't wait for users to get to it but you'll set the player up for failure if you drive them to that experience too soon also take into account the learning the learning curve of using your game not just what you're trying to teach we wanted to start by teaching really important vocab and grammar right away at duolingo but what led to much better engagement was first teaching trivially easy vocabulary for example at the beginning of the experience we use a lot of names or other words that are similar between the native language and the language being learned even if it's not the most immediately useful this allowed learners to focus first on learning how our app works in the first place the types of exercises we use using hints throughout the lesson in the skill system etc before also trying to learn a new language on top of that learning it all at once would be too

much a lot of games do this well by having an onboarding experience that targets learning how things work like the menu without overloading the user with lore or new mechanics off the bat but in an attempt to not underwhelm the user you may run the risk of i mean overwhelm the user you might run the risk of underwhelming them with things that are too easy or don't take into account prior skill remember to keep the experience interesting for those advanced users too you can get around this by trying to manage difficulty by user cohort so for example we track the failure rate of every exercise on duolingo every time someone has to translate i have a dog to spanish we see how many learners get it right how many get it wrong and we have a threshold where we throw out exercises that are too hard but we separate these based on crown level so users at the first level more beginner users all get the same

threshold as other beginners whereas more advanced users probably can handle more difficult sentences and get them wrong less often in games leagues and leaderboards are a prime example of this where you're constantly in the same grouping as players with similar activity levels you either get promoted or demoted regularly to stay calibrated and elo systems are the same way also in games like starcraft 2 or in civ 6 you might be asked at the beginning to self-report your prior experience so that the onboarding can be accelerated if you're already experienced in a genre provide natural ways for your players to speed up their progression when needed all right the last step after having great learnable content and a framework to make going through that framework a good experience sorry that's sentenced in grammar the last step um in creating a good experience is to design a motivating ux for that experience

and actually a surprisingly large part of making an experience delightful is to just avoid things that are not delightful which brings me to this something that you hear over and over again but cannot be understated don't make users read because users don't read so here's an example where do the owl helpfully pops up on the website excited for your first lesson and tells learners they can hover over words to get a hint whenever they don't know what a word means we even highlight in orange new words to try to encourage people to hover over them and use these hints we provide and this is actually a key part of how duolingo works we don't give you upfront instruction on how language works we just kind of throw exercises at you and convince you that you can do it with just a little bit of help so we found that fewer than 40 percent of our monthly active users use hints at all

and we have seen many user testing videos where learners are really confused about how they're supposed to know the meaning of words without being taught them many say that they had no idea they could hover on or tap words this single sentence was not being read by many people or if it was it was being quickly forgotten here's another example where in big red letters we tell learners why they got something wrong here the learner entered in women instead of women and we point out that the singular form woman should have been used here instead however when looking at the forms around this exercise we'll still often see learners confused about why their answer wasn't correct this user says i wrote women instead of women and feel it should be counted correct it's likely that when users see longish text they continue on without reading it or skim it and forget it

and you'll note here that these are just single sentences that we found our users were just not reading so what i found helpful in the past is performing the utterly screwed test ask yourself if is there a way for the player to recover if they were suddenly plopped into your game at any point will there be anything they missed out on at the beginning that would prevent them from moving on if your game involves reading a one-time wall of text at the beginning it'll be very hard for a user to keep going if they come back after a hiatus but if your game is intuitive to play without much explanation it's much easier to get back into for duolingo resurrected users or users who come back after a long break make up a large chunk of our daily active users so making sure that they can get back into the experience without a ton of additional onboarding is really important for our metrics

besides players not reading this is also a huge reason to avoid text at dual lingo we see lots of text as a possible indicator of lazy design text is often a crutch for solving difficult instructional problems that often have a harder but more user-friendly design alternative the more intuitive your design the simpler your design the less text you'll need this is largely why we didn't have explicit grammar instruction in the app for so long why we keep all of our exercises really short and why we use lots of colors icons animations and game mechanics to convey instruction instead something we've also found works well is to offer help as close to when it's needed as possible for example we provide in-lesson hints instead of asking learners to memorize all of the words ahead of time because even if you manage to have a player read the text they're still not

likely to remember it so offering help at the moment that it's needed is really useful for your players second component of creating a good learning experience is to prevent the player from self-destructing on your game if you have a game with a really steep learning curve it's easy for learners to burn out or get frustrated so pacing mechanics helps with this it helps make sure that you aren't burning through or introducing your content too quickly on duolingo we introduced a controversial feature called the health mechanic in our ios app where health segments refill after a period of time if a user makes a mistake they lose a health segment they can re-earn this health by practicing material that they've already learned and this means users naturally have to slow down if they're moving too quickly however this also led to increases in d1 and d7 retention when we ran this despite a lot of outrage from our users

and it has the plus side of also slowing down learners to really try to think about and get the right answer on exercises and think harder about the learning aspect cramming in general is really bad for long-term learning and retention if you want players to learn a concept and grasp it well users will quickly burn out and you'll have high user churn if you fail to pace them correctly if you want to build a habit of your users playing your games every day it's good to encourage them to come back later sometimes timers and pacing also give motivation to come back and push back against binging and they give a great reason for notifications that also helps bring users back another form of self-destruction we saw was users getting frustrated with themselves if they got several exercises wrong in a row we saw this in user testing so this might seem really silly

but we started adding a friendly motivational message reminding learners that they're still learning even when they get things wrong so duo our helpful little owl pops up and says even when you're wrong you're still learning and this led to huge engagement wins just this simple tiny thing so we saw nearly a two percent increase in daus retention increased three percent for d1 6 for d7 7 for d14 retention this little bubble message here and that's an insane gain for such a simple simple feature if you have a way to detect if your player is getting frustrated setting up just little ways to intervene and encourage them can do so much for retention and finally though it seems rather obvious remember not to not only discourage destructive behavior but to also encourage desired behavior

achievements or quests are a really classic way to do this and you see them in games all the time when we added achievements on duolingo they gave us wins in daus sessions completed and retention but one of their main benefits is also providing clever ways to increase discovery behavior in your game for duolingo some fewer used features like looking at your profile tab or adding friends doubled in engagement over a hundred percent increase in clicks with the addition of achievements and finally if we're talking about one of our best performing mechanisms for motivation it's definitely the streak the streak is how many days in a row you've met your goal on duolingo there is nothing that unlocks when your streak is high there's very little social proof or ability to show off your streak there is no currency attached to your streak we don't get any learning content on your streak

it's simply a number that people are proud of and it doesn't actually gain you anything in our app but our users care so deeply about it and find it incredibly motivating so much in fact that they get devastated when they lose a long streak in response we created the idea of a streak repair the ability to repair your streak if you accidentally missed a day and to illustrate how much this simple little flame motivator mattered we tested offering one free monthly streak repair as part of our subscription perk and when we did this subscription revenue increased almost 60 [Laughter] 60 subscription revenue because users really care about this little flame icon that got them nothing in the app so never underestimate the power of a simple ui element whether it's a motivational message an awesome

animation a color change or a tiny flame in the app that you want to keep burning for hundreds of days you might be surprised how motivating these things really can be so let me take a few minutes to recap lessons today about how to create a good learning experience within your game so here are things you should probably do yesterday but at least do them as soon as possible first identify holes in what you're currently measuring making sure that you have good indicators for results and not just activity create learning indicators to see if players are actually learning what you need them to invest in a good a b testing framework like i said we went from 13 d1 retention to 55 just from incremental changes in a b testing everything remember to cut out the noise from players who aren't exposed to the experience you want to test and you'll get a lot more statistical power that

way use a b test to see how much new features help and hurt to keep yourself accountable improve your intuitions with data and make all early stuff way easier i can't recall a single experiment where making things easier hurt engagement within our app however the caveat is make sure that those learning indicators are in place so you ensure you aren't sacrificing too much for the sake of engagement and then here are some actions that you can look into in the near future for improving the learning curve in your games remember to identify and deliver the smallest unit of satisfaction as early as possible to your players give them the confidence that they can do this they can conquer and get good at your game get them interested in the first few minutes of play and provide that baby step needed to start climbing that learning curve

disguise boring concepts by replacing or pairing them with interesting ones if you can come up with a good way to make volume adjustment fun do it look at what steps seem to have higher user drop off and target those first as areas of improvement scaffold difficulty in your game if you want to appeal to a wider range of players provide paths for both breadth and depth so that more beginner gamers can enjoy your game and advanced players can have a blast mastering it perform the utterly screwed test on your game are there parts of it where the player would be totally lost if they just picked it back up at that point try to create ways to recover at those points winning back users who want to give your game a second chance can be critical to improving your daily active user numbers remove text it's likely you could probably remove half the text in your game

and finally add delight find ways to intervene when the player needs help and to encourage the player when they're crushing it even if it's by really simple ui elements so if you want to talk about specific ideas for your game have questions about duolingo or learning or just want to say hi feel free to reach me at this email i'm also happy to take a few questions now and after the talk in the wrap-up area so thank you all so much for coming out today safe travels home and i'm looking forward to continuing to be inspired by all the games you create in the years to come thank you oh boy hello hi karen hi thanks that was a great talk thank you um regarding streaks did you guys test anything uh like rewards or anything related to streaks yeah absolutely so we are looking into a few things one thing that happens is we

have an in-game currency that we call lingets right now and as you get streaks of multiple 10 you actually get a bunch of lingets with it so that has helped um one thing that we're also testing is how many days to sort of show in regards to your streak like if we could get users to just a three-day streak maybe they're more likely to do a seven-day streak things like that so we're always playing around with streak because it's such an important motivator thank you thank you hi hi um how do you process uh feedback from your users how how for example that one was talking about the one word that was wrong yeah how we process that inside how is the team working on this feedback to upgrade duolingo yeah that's a great question so there are a lot of channels for user feedback um one one thing that i mentioned in the talk too is sometimes words aren't always as indicative as actions so we

try to rely on our metrics as much as possible like are users getting frustrated and quitting right after they get this thing wrong etc but in terms of qualitative feedback and not just quantitative we do have a customer support that processes reports and error reports from our users and a lot of us that work in product also constantly check our user forums and our app rating reviews to see if there's any good qualitative feedback we can gain it is a lot of work to process all of that qualitative feedback but it's also really important thank you thank you hello hello talk thank you i want to ask something but first because i'm not so familiar with your app what sort of monetization scheme do you have yeah that's a great question so we we make revenue from ads and we also do a subscription model so we for a while we did in-app purchases ads as well as a subscription model one of the key tenets of duolingo's mission is we want all learning content

to be free for anyone because not everyone has the the means to pay for education so our subscription model allows users to do things like download lessons offline get a free streak repair and things like that and and we have found that subscriptions are probably going to be our main source of revenue i see so it seems that you're very centric about um looking at the metrics to see what works and what doesn't now you also mentioned at some point that sometimes you can skew the data or look at it in a weird way so you artificially create the results you want what sort of filters or checks do you have in place to uh confirm whether a certain metric that seems to be pointing in the right direction really does so yeah that's a fantastic question and honestly it's it's sort of hard to have something in place because the people who are prone to those tendencies will need to also be the ones that puts checks and balances in place so for example on the teams i'm on with learning

it's our goal to not decrease things like retention and d1 in d1 retention and daus and we have this other metric that we call the learning metric based on those sort of quiz scores on how users are doing and how they're learning that tries to keep us accountable right if we increase engagement a ton by just making things way easier but people are learning less we see that other metric go down so what i would suggest is just going through the thought process and exercise of seeing what things you can cheat on and then making metrics on top of those things to go the other way to keep yourself in checks and balances and sometimes you just need really honest co-workers to help you with that too thank you hi so we make kids learning products for much younger age range and one of the things we always run into is this balance between learning and fun oh yeah yeah so i'm interested in how you guys define like either boundaries or you know guidelines to make those kind of trade-offs yeah i

mean this is this is like the problem that we've had ever since the beginning of dulingo and it's definitely not one that we've actually solved we're constantly improving it but what has really helped is between learning engagement like i said if you can measure both you can see what changes actually improve both and then you can try to avoid situations where you have to choose between one or the other so for example the crown levels experiment i presented was one where we got really great wins in both whenever we revamp our curriculum we also see gains in both so we're sort of procrastinating on how we can do that trade-off by just trying to first target features that would actually increase both of these things at the same time hello hi thank you so much for the talk um i wrote my question down so avian a b experiments that feature large changes such as revamping the entire fundamental curriculum take time to deliver quantifiable results this makes like measuring the result in

a timely manner much more difficult in this kind of situation how do you know if these types of changes are the correct changes and as a side question when is it right to disregard all metrics and go with your gut feel yeah oh gosh that's so hard so um duolingo actually has another app called tiny cards that i worked on for a while and this is an example of an app that didn't have that many users so we couldn't rely on just sheer numbers to deliver really quick metrics results and i'm sure a lot of games are in this situation so sometimes what you have to do is sort of rely on the best practices and measurements and the the tests that other larger companies have already run in order to just make a best guess for what might work for your game i will say that usually it's very rare that we disregard all metrics and go with gut but we do actually emission and core values really helps you keep true to at least things that you don't want to compromise on so for example if we started charging for learning content i bet we could make a ton more money but that is something that we just encode in our values that we're never going to try

um so it's hard right um but i think if you have really fundamental core values in place and build teams around it if you're willing to take risks sometimes for larger features and just see how they do you'll still be able to move quickly even if you don't have huge user numbers to get you results within a few days thank you and just a note i think we're going to be kicked out in the next 30 seconds or so so i'll take one more question and then i'm happy to just head on over to the wrap-up room right after this go for it so when you're running multiple a b test are there any measures or strategies you use to make sure one test is not biasing the results of the other test yeah so the beauty of a b test is because it's a completely random sample um it's supposed to be the case that whenever you have a test that that splits up users into two buckets any test sort of also takes a random sampling of those two buckets so you can do sort of cohort analysis and see hey users in in these two experiments those who really don't interact very well or

you you can see bonuses there so we we have sort of things in place to try to prevent us from doing that but the also key is all of the experiments should be independent of each other so if you have experiment a running um a user is just as likely to be in experiment b or not in experiment b if they're in experiment a got it thanks all right i will i'm very happy to go over to the wrap-up room after thank you so much everyone