Lessons from Duolingo: How to Turn Good Intentions Into Good Numbers

[Music] hi everyone i'm so excited to be at gdc with you today to present lessons from dulingo how to turn good intentions into good numbers this talk will go over how to leverage a data-driven culture to further your business metrics while still staying true to your values i'll start by doing a quick introduction my name is karen shai i have a bse in computer science and a minor in neuroscience from princeton university so i've always been interested in the intersection of education and technology currently i'm an engineering director at duolingo the most popular way to learn a foreign language online that's duo our green owl mascot that you'll see a lot in this presentation today i had the privilege of joining duolingo in 2012 three months after duolingo's public launch and i was able to witness the

company grow from about 15 people to over 300. i currently lead engineering in the learning area the biggest area at zolingo this area focuses on how to teach our learners better and of course i'm particularly excited to be here because i've always loved games my current obsession is team fight tactics though i was playing animal crossing pretty obsessively earlier this year i'm excited to share our approach about what we've learned at duolingo in the past eight years because it's worked really well for us we're currently the most downloaded education app with over 30 million monthly active users we think this popularity is due to duolingo being free and because it's designed to make language learning feel more like a game since i joined there were two main pillars of company culture the first was our mission our purpose was to equalize access to quality education and our second pillar was that we would rely on data and a b testing

for all of our decisions when it came time to mature and become a profitable and sustainable company we had to think a lot about how to do that while still being the company we wanted to be we also had to figure out how to prioritize things that were hard to measure like how well we taught so despite all of the a b tests we've run it still wasn't capturing everything we wanted it to last year we gave a talk on how to make learning hard things easy providing a lot of examples of surprising results we found through over 3 000 a b tests on how to make an engaging product in the q a part of that talk there were a lot of great questions on best practices for using data and what to do when data wasn't readily available or didn't quite capture what was desired so we thought it might be useful to dive a little deeper into those questions this year so let's start with some background to frame the problem at hand the question is this how can we encode

good intentions and values into day-to-day decision making while still listening to your metrics and numbers how do you make difficult decisions when it comes to trade-offs and there are a lot of trade-offs in games you encounter these decisions pretty frequently for example do you want to make a super accessible free game that's widely used or do you want to be profitable sooner but maybe with a paywall and get less usage do you want to do constant patches and ship new features quickly or do you want to have a very high quality product off the bat do you want to make your game challenging and appealing to niche users or do you want your game to be accessible enough that it's hyper casual and appeals to a wider audience there are a bunch of other sensitive gray zones like how much moderation you should do whether or not to take political stances in your product how you approach user privacy representation diversity inclusivity etc and this talk is not about what to

do or what to stand for but it will be about how to develop a process and a culture that lets you stay true to the values you choose while maximizing metrics at duolingo we encounter these sorts of trade-offs every day we constantly have to decide how we can be sustainable and earn revenue while still being accessible to as many users as possible we also have to balance the trade-off between making our app approachable and easy and increasing engagement but also trying to teach effectively by also teaching harder content as well at the right times also one thing on my mind constantly is how we measure learning outcomes which is a traditionally slow moving metric and difficult to capture in day-to-day metrics by the end of the talk i'd like you all to feel confident that you can develop a process and a culture that will allow you to be both idealistic and staying true to your values

but also pragmatic and intentional in maximizing business metrics so we'll go over each of the steps in detail but here's a preview of what we'll go over today first the first step is to document your operating principles and values and i'll explain in the next section what these are second is to set up and encourage a data-driven culture specifically if you want to have good numbers you need to be able to track them the bulk of the talk will actually be around a concept called guardrail metrics which allows you to tie these two together and finally we'll talk about how to account for things that are a little harder to measure our approach is fairly simple but we found quite effective so the first step is to establish operating principles operating principles are a way to put culture and values into practice a commonly heard principle in tech is to move fast and break things amazon has a principle around frugality

some companies might say company needs should come before user needs whereas others might say user needs should come before company needs what's so interesting about operating principles is that good operating principles have very valid counter arguments things that are obvious don't need to be written down but when the company makes a choice on what to prioritize or value that may not be entirely intuitive operating principles are very helpful these are a way to encode your uncompromisables things that you aren't willing to make exceptions for and that you want to preserve once you've come up with some operating principles it's usually a good idea to list out examples of how your company has demonstrated these in the past or hypothetical or aspirational situations where you'd want people to make the right decision so here are a few examples from duolingo to illustrate these in practice one of our first principles is learners first and that means we never charge for learning content

as you can imagine this makes it tricky to monetize efficiently but since it's encoded in our values we've been able to preserve it another principle is to take the long view meaning they'll often turn down short-term significant wins if they have a risk of having long-term detrimental effects another example is all for one and one for all which values cross-team trust and collaboration some companies are very successful by having a more competitive environment but we encoded here that every team should be working with the consideration of other teams so let's say you now have this lovely list of operating principles the first thing to do is to stress test it with past and hypothetical future decisions given these principles would an average person at your company make the same decision you would would they prioritize the right things and when they feel confident they're making the right decision from there you can begin to evangelize your operating principles and establish a culture around them and

then comes the hard but fun part figuring out how to align your principles to business objectives by identifying what key metrics will further those business objectives so having values in a goal without strategy runs a large risk of failure being data driven even with the simplest of a b testing tools allows you to be more confident in the direction you're taking and signals you to course correct when necessary by obsessing over our metrics and only launching things that win we guarantee that our numbers increase over the years in all of our key metrics and since we run thousands of experiments that adds up so i'll quickly go over what av testing is here because a lot of you are probably familiar um the idea is that you can test the impact of a change by giving it only to a fraction of your learners so for example here you could divide your users in half and only give half the new feature we generally internally call these two

conditions the control condition and the experiment condition where experiment generally represents the change that you're trying to test you then look at metrics such as retention or activity for both populations and see how they compare one tip is that when analyzing these metrics only look at users who were or would have been affected by the change and don't include those that weren't so for example if you make a change to level three of a dungeon and only look at you should only look at users who reach level three users who are only in levels 1 or 2 don't even see level 3 and wouldn't experience any change and that also dilutes your data this simple concept really changed our ability to run experiments quickly until we made this change many experiments would be neutral or take several weeks or months to run but after this change most a b tests run for only about two weeks with significant results as mentioned duolingo has run over 3000

a b tests with around 150 to 200 running at any given time this is possible because everyone is bought into the culture of experimentation anyone not just engineers and product managers are empowered to create experiments after going through a short training on an online course there's a quiz at the end to test understanding and if you pass that quiz you get the permission to create experiments as a result tons more experiments were created after this training was launched and this number continues to grow at a rapid rate we also try to make analysis as accessible as possible a key part of our experiment framework is an auto-generated report that updates daily doing all the statistical work necessary to determine if the effects are statistically significant this means almost anyone in the company is able to provide informed recommendations on whether to launch or kill experiments based on the metrics provided in the report

and the final step after experiments are created run and analyzed is to share the findings this is very common for av test results whether positive or negative to be shared in our weekly all hands meetings is a picture of our all hands meeting before code that hit hopefully we'll all be back in person soon all experiments are also sent to an open mailing list of experiment write-ups through these two ways of sharing experiment results everyone gets pretty familiar with how we look at metrics how we make decisions on whether to launch or kill features and what seems to work or not work when it comes to moving these metrics now if you're just getting started with getting more data driven there are plenty of third-party providers that will provide a solid a b testing framework for you and tools that will allow you to determine whether or not those results are statistically significant it took us many years to build a sophisticated framework but even in the very early days we were

able to benefit a lot from very simple testing tools what's more important than the actual tools you use is fostering a culture that loves experimentation for duolingo teams are generally organized around these metrics we have a retention team teams that focus on revenue etc but sometimes that may cause issues when two metrics collide what if there's a metric you care about but no team is in charge of it that's where gargoyle metrics come in a guardrail metric is a business critical metric displayed on all of our experiment reports that should never go down significantly otherwise the experiment shouldn't launch unless there's a very good reason to do so remember those cool automated reports that i showed you earlier they contain around 20 key metrics on them many of which were created by teams to prevent other teams from accidentally negatively impacting their metrics

you can see in this word cloud here i've displayed the ones that we tend to display on our metric reports they include things like app crashes which you may not always think of on feedback email taps for people asking for help retention subscriptions etc by having these guardro metrics on all reports it aligns everyone on what the company values and gives a holistic picture of the impact of the future without them teams might tunnel vision on the metric they sought to increase without realizing the effect that it has on other metrics outside of the team's purview teams can also rest easy knowing that generally if the analysis report looks good and no guardrail metrics are negative they can be confident in making a decision aligned with company interests if something is negative or concerning they also know which team to reach out to in order to come to a conclusion about whether or not the experiment should launch

so i have three examples for you today around the usage of guardrails the first was a situation where we identified the need for a new guardrail then i'll share a case where guardrails helped us make a difficult decision and finally i'll share an example where guardrails revealed something positive and unanticipated so our first example centers around an experiment around our health mechanic similar to many games learners would lose a life or health for mistakes that they made in the app and if they ran out of help that was a signal that maybe they were learning a bit too fast and would need to practice or wait to earn it back health refills were an in-app purchase that we added so this feature actually made us quite a bit of revenue but it also seemed to help d1 and d7 retention without hurting daily active users however what we saw in the metrics was not what we saw in the public forums

and our channels our app store ratings really suffered a lot with learners unhappy with this change we had to manage a lot of community outrage and we suffered for a while with a much lower app rating than usual we discovered this was mainly impacting existing users since it's always painful to have something taken away or changed that you're used to new users who started duolingo with this feature didn't seem to mind it so we let this feature run only for new users for a while and rolled it out to existing users very slowly but because of this experience we added a new guardrail which is app store ratings we could see if a feature significantly changes our apps for reading and this metric allows us to prepare a community response ahead of time so we could be thoughtful about how we execute it and also stay ahead of any community concerns a second example around guardrails involves an experiment we ran on our subscription purchase page

that changed the copy or text of the main button in the control condition the one that users already saw it said try seven days for free we wanted to test a new copy that said start my free week it turns out that the experimental copy made us a lot of money we saw a six thousand dollar day in per day increase in in-app purchase revenue in the experiment condition however looking at guard rails we saw that new user d1 retention dropped by about one percent which seemed to translate into also a small drop in daily active users we hypothesized that the new copy may have been misleading new users might have assumed that after a week duolingo would cost money which was not true due to these guardrails we decided to leave the money on the table kill the experiment and iterate on the copy further to minimize misunderstanding

our monetization team has long had a stance that we never want to trick people into giving us money it's powerful and telling that this decision came from the monetization team itself and was not in force top down because of our operating principles about taking the long view and all for one and one for all teams are aligned to make good decisions for the company even if it requires a sacrifice from the team our last example is a serendipitous one one of our teams made a change that updated the look and feel of our language courses by replacing a button that said test out of and skills with what we call checkpoints which when clicked on would also allow you to test out skills but also become milestones and indicators of progress once a user went past them unintentionally we discovered that this made the test out feature less prominent

our hypothesis was that because learners weren't testing out as much which is a shortcut they were running out of health faster because they had to learn the material more thoroughly free trial starts and subscription revenue increased substantially now at first we were worried that if learners weren't using test out they might have been seeing content that was too easy and they would be bored and leave however since all other guardrails like retention day use and time spent learning were neutral it implied that the learner experience actually wasn't negatively impacted by this change and now we know that test out is a lever that does impact revenue something we would have not thought of in the first place so now that we know what guardrails are about what do we do next well the first step is to identify the metrics that are most important to the success of your product for games this might look like getting

as many users as possible or to be as profitable as soon as possible you could also look at how engaged your users are by checking things like how long they play your game per day or how often you might want to look at latency and crashes if stability is very important to your game and if your game relies on network effects you may even want to measure more specific things like the average number of social interactions per unique user and finally you can track more subjective but still quantitative metrics like app store ratings and social media engagement but some things are just really hard to measure and put a number to and how do you account for those for example it may be that one small bug is insignificant and not very high roi as we call it to fix but what is the impact on the user if they see a lot of these small bugs similarly

how do you measure the impact on your reputation when it either comes to bugs or features similarly how do people perceive your product and your company how do you measure creative and design quality and one that i think about daily is how do we measure long-term learning outcomes the hard truth is in a data-driven culture if it's hard to measure it's hard to prioritize that means that if there are important things that you care about creating guardrails around that are hard to measure you need to find a way to prioritize them anyway we have four approaches to this that we use for different situations the first solution is to simply care about it even in the absence of measurement brand design consistency falls into this bucket for us the approach usually requires an additional process for enforcement or else people may not be motivated or

reminded to care about these principles we enforce design guidelines during development design review and quality assurance but otherwise can't easily measure the consistency of our branding but for example we updated all of our logos to our new brand logo even if it didn't impact metrics operating principles are also very useful when it comes to encoding that something is important another solution is to reframe what success looks like for an experiment for some objectives in the learning area for example we consider a feature a win if it's entirely neutral for metrics but research or learning expert backs up that that feature is better for learning again guardrails are critical for this we can be confident that we aren't hurting by launching a feature we believe in a third solution is to actually measure slow moving metrics periodically some

things are not impossible to measure but may just take a long time to do so for example metrics like brand awareness tend to move relatively slowly so we do an annual study we also do an annual study on employee happiness and when it comes to learning we conduct periodic efficacy studies that look at learners who have been around for a long time it's good to predict things that you'll want to measure in the future so you can set up an infrastructure ahead of time before you miss a window and have to wait a long time before you can do it again and finally the solution that's my personal favorite and generally works the best is to find what we call proxy or good enough metrics we realized sometimes it's better to just be able to move forward even if the direction is not perfect so long as you're generally moving in the right direction one example in the learning area is that

one of our teams is working on making things harder on duolingo without decreasing guardrail metrics which is actually quite hard because if you were at the talk from last year you know that any increase in difficulty actually has a huge impact on retention and da use in this case we're using make stuff harder as a north star that's a proxy for if people are doing harder things they're probably learning more and practicing more of the language they're trying to learn even then we had to have a proxy for what it even means for something to be harder we looked at the types of exercises users do on duolingo and had a general sense of which ones are hard versus which ones are easy for example translating something into the language you're learning is much harder than translating something from the language you're learning into your native language so the tea will consider it a win if they release a change that can increase the ratio of

challenges that make you answer in the language you're learning without harm and guard rails this approach also pairs really well with longer term studies because then you can then check how well your proxy or good enough metrics helped you move the needle on the actual metric you care about so to summarize the four approaches to measuring immeasurable things you can commit to caring anyway and create a process to enforce it you can declare experiments as success as long as they're neutral but are supported by some other measures such as research or subject matter expert you can measure slow moving metrics periodically and try to anticipate how to do so or you could find proxy or good enough metrics to cure decision paralysis and move forward anyway so what has this approach of establishing operating principles

obsessing over experimentation and leveraging guardrail metrics gotten us well internally we've benefited a lot teams feel collaborative instead of competitive through guardrails and a system of accountability there is a lot of trust between teams to make good decisions that don't hurt other teams we also now have minimal debate about whether or not to launch experiments this allows teams to make decisions quickly and shift things faster it also feels more fair and more consistent finally sharing both wins and failures not only offers a level of transparency and accountability into all teams experiments but also helps the company get collectively smarter about the product and what works when it comes to moving our metrics with gardwell metrics we have a checks and balances system that allows us to be confident that we're moving forward

in increasing all the metrics we care about without taking steps back on any of them we went from essentially no revenue to over a hundred million dollars in revenue within 3.5 years and best of all our users are more engaged than ever we have five extra d1 retention since launch while also working towards becoming a sustainable company from a business standpoint this approach has been really effective by having a free app we've been able to attract a huge user base meaning we can run experiments and improve our product very quickly we've also been able to become profitable despite not charging for learning content though this was probably the harder path than just starting to charge for our app and we had to be more creative and careful on how we generated revenue it was the path that ended up being the best for business as well and finally preserving our values is

observed and felt by both our user base and internally as a company we tend to get benefit of the doubt even when we make mistakes and as you all know a loyal user base is absolutely invaluable so let me summarize quickly what we've gone over today and then we'll jump into questions the approach we took started with encoding your core beliefs and values in operating principles remember it's not just useful to list up things that are obvious but rather things that you want to take a stance on as a company then set up a data driven culture that loves experimentation it's much easier to drive up key metrics when you can measure them and get a sense of what impacts them then identify and enforce some guardrail metrics you'll likely find more and more over time but start with the critical ones that are critical to the success of your product and finally do your best to make immeasurable things

measurable whether through process proxy metrics or longer term studies being able to capture the things that you care about is really important in being able to encode good intentions into your product in summary with some intentionality you can drive a successful business and successful product without compromising your core values in fact you'll likely find that you and your fellow employees are more motivated and successful aligning your core business objectives to your core values games have been such an important part of my life and i truly believe that they do change the world for the better i hope you'll join me in being idealistic and create games that we're proud to be a part of thank you for joining me today and now i'm looking forward to all of your [Music] questions