Session: How Does Duolingo Drive Growth At Scale Without Losing Speed Or Quality? - Osman Mansur.

Now we are going to see a really good show reel from um uh Dualingo which I'm really excited to see. And then after that, we're going to have Osman Mansour come and speak to us about how Dualingo drives growth at scale without losing speed or quality. So, let's see what During Let's see the the the reel. Have we got one? [music]

[music] [music]
Hello everyone. Everyone, I'm Osman Mansour. I'm a product manager at Dualingo based out of our New York City office. And I'm really excited to be here today to talk a bit about how Dualingo has grown and a bit about our culture of experimentation. Um, so many of you might be familiar with Dualingo given that we're kind of put ourselves all over the conference here. Um, but if you're not, we are the largest language learning app in the world. But we also do more than just language learning. More recently, we've scaled our product to also teach things like math, music, and chess. And on [snorts] that note, um, as you just saw in the reel, we are hiring game devs in

our London office. So, particularly if you want to work on our music product, please come chat with one of us. Um, but today's talk is primarily focused on the story of Duolingo's growth. So, this chart is a chart of the daily active users on Dualingo since the company went public in mid 2021. I joined Dualingo a few months before the start of this chart and I've been on a variety of different growth teams as a PM. So, it's been awesome to be able to both witness and contribute to this growth over time. Um, if you're familiar with Duolingo, you might know that we have a powerhouse marketing team and a lot of our growth is driven by organic word of mouth, but that doesn't really tell the whole story of how we've grown. A lot of the growth that you see on this chart is actually from product improvements of making the Dualingo app more fun and retentive across all services. And one number that really tells that story is this one right here, which is that out of the 50 plus million

daily active users on Dualingo, over 10 million of them have a daily learning streak of 365 days or more. And having worked on the streak feature and been on the retention team at Dualingo before, I kind of take this number for granted. But it is honestly remarkable to reflect that we've built a product where over 10 million people have been using it to learn something for over a year plus. Um, but as I mentioned, a lot of this growth has come from product improvements and that there's a very special sauce and process to how we've gone about that at Dualingo. As our user base has grown, so too has the number of AB tests we've ran over time. At Dualingo, we love to AB test everything. Whether it's a huge change, like launching a big new bet or a small optimization to a single screen in the app, but across the dozens of teams at Dualingo, each team is running hundreds of AB tests a year, equivalent to

thousands over the course of a year across the company. So recently we've codified this process of testing everything into a philosophy that we call the green machine which I know admittedly it sounds a bit selfindulgent but the basic principle is pretty simple. It's just that we test things and when we find something that works we double down and test even more things in that space. Um but this is very frameworky. I'd love to get into a couple of examples of some AB tests that I've ran as a PM at Dualingo and how it tells the story of what testing and optimization looks like within the product. Um, so to start, one example I'd like to talk about is with our notifications. Uh, if you've used Duolingo before, you're probably familiar with Duo's pesky reminders. And I have to admit, I was one of the product managers who worked on our notifications for a few years. Um, and the test I want to talk about is a copy change that we ran back

in 2022. Uh, this is an example of a practice reminder you could get from Dualingo. And under the hood, the way our notifications work is that we have hundreds of different variants of possible copy that a user might get. And we have a multi-armed bandit, a machine learning algorithm that will pick the copy we think is best for the given user. So amongst our library of hundreds of different copy variants, a lot of the copy we had back then referenced the fact that a lesson on Dualingo would only take 5 minutes. This is in spirit of framing learning on Dualingo like a bite-sized activity you can do every day. Um, but being the curious PM that I was, I looked into our metrics and realized that the average length of a lesson on Duolingo wasn't 5 minutes. It was actually slightly under three minutes. So we ran this test to change every notification that referenced that 5 minutes to instead say 3 minutes and

it happened to be one of our biggest wins in daily active users uh back in 2022. Uh we saw big increase in DAUs. I know that number looks kind of small but when you're thinking about our user base of tens of millions of users these small wins really add up. Um, and the other interesting thing we saw from this test was that not only did it increase retention by making it seem even more bite-sized to jump in and learn some Spanish, it also actually increased the total amount of time of engagement on Dualingo. So, it wasn't as if we were disincentivizing folks from spending longer to do their lesson. Um, so that's one example from notifications. Another example I wanted to cover was around streak flexibility. And the reason I I love this example is because it gets at how Dualingo approaches uh trade-offs between uh user engagement, user growth versus monetization. So if you've ever lost a streak on Dualingo in the past, you and you then came back to the app, you might have seen that there's an opportunity to repair your streak. A few

years ago, this is also an experiment from back in 2022 23. Uh the main way to do that was to pay. So, we have an inapp currency called gems. And if you came back to the app after losing a streak, you'd be able to pay gems to get that streak back. On the growth team, we didn't really love that because it was really like play to win, and it kind of reduced the meaningfulness of the streak, but also the streak was meant to be a feature to drive retention and engagement. So, we didn't feel great about fitting in like a monetization hook within that feature. So, something we tested on the retention team was adding an extra option to that uh dialogue where the learner could do a free learning challenge where whereby if they completed it, they could earn their streak back and they wouldn't have to pay any gems. Uh the results here are pretty interesting. These are big numbers, but you should think about this in the way that this is a very specific user moment of a user who lost their streak and then came back. But what was

very fascinating was we saw a huge increase in retention as you see by the numbers there, but also a really big drop expectedly in revenue because we were diverting users away from that only paid option towards one that was more in favor of retention. Uh but we ended up launching this. This obviously isn't reflective of us losing half our AP revenue overnight, but just this sliver of this product surface. But it kind of reflects how at Dualingo, we will often take the long-term engagement win over the short-term revenue hit, especially as we frame ourselves as a premium product with an amazing experience for uh free users as well. Okay, so those are a couple of examples of tangible AB tests that we've ran. Now, I want to talk about more of the process that goes behind uh these experiments and I'll walk through five best practices that I have. These aren't necessarily official like company

documentation of what we think of as the best practice, but more so like my point of view having been a PM at the company for a few years. Uh so the first one is about developing extreme clarity around metrics. At Dualingo, we segment our user engagement by the growth model that you can see here. Um and this is like mutually exclusive and collectively exhaustive of all users. So wherever you are, if you're an active or inactive user on Dualingo, you fit somewhere here. Uh this model isn't revolutionary. In fact, when our team had developed it several years ago, it was inspired itself by Zinga. Uh but we use it for all our growth work. And I think the place where it is most valuable is we obviously track the health of these metrics, the numbers of users in each bucket as well as the rates of movement between them. But we also look at the sensitivity and leverage of each of these metrics and how much they'll move our north star which is daily active

users. So as a personal example when I started working on the growth team at Dualingo I was specifically on that retention team which owned the metric cur which is kind of in the middle left. That's basically current user retention rate or the D1 retention of uh existing users. But as we were working on that team, improving things like notifications and the streak, we realized that we were inadvertently increasing another metric that was having a lot of topline impact on DAUs, which is IWAR a little bit to the right. It's a mouthful, but it stands for inactive weekly active user return rate. Basically, if you're a weekly active user, but you weren't active on Dualingo today, what's the rate you come back tomorrow? So when we realized that this was such a instrumental metric, we spun off a team that I was co-leading to exclusively own and optimize for IWAR. And since then, that's been one of the most productive teams at the company for driving topline DAUs. All of that is to

say that having this really comprehensive view of your metrics and what metrics move your top lines and then staffing resources according to that is really powerful in this process of optimization. The second point is pretty similar which is about maximizing the power of compounding. Uh this is a lesson that our VP of engineering Sean Colombo loves to share. So I'm going to reshare it on his behalf. But as a thought exercise, imagine that you have a stable user base. let's say 100k users on your app and you have a D1 retention of 80% and every day you bring in an influx of new users equivalent to 1/5ifth of your user base. If those are your numbers, you're going to keep the same exact number of users every single day, 100K. But let's say you ran an experiment to increase retention by 1%. So that 80% number goes up to 81%. quickly just in a matter of a

few days you'll realize that let's say after a week you're at 108k users. Uh so this compounding happens really really fast and it's a pretty like obvious insight of if you ship things fast launch experiments when you have statistical confidence your user base should grow faster but it's really something that we internalize deeply at Duolingo and we apply it to every stage of our process of optimization when it comes to roadmap planning or designing the setup of of AB tests. We make sure that we're able to get as much of the impact out the door as soon as possible because it really adds up when you're stacking several hundreds of experiments over time. Okay, so I really like this point about enforcing a consistent quality bar and it really gets to the title of my talk about how Dualingo has been able to grow without sacrificing speed or quality. Um, and at Duelingo, we do a few things within our process that really make sure we're upholding our quality bar. One is

our very unique product review process. So, at Dualingo, any product change, whether it's a small optimization or the launch of a big new course, uh, almost always goes through a process called product review, where the change is reviewed by our CEO and our leaders from product and design. Um, there's some exceptions, but generally it all goes through the central process. And the thing I found most interesting when I joined Duolingo was that this is a public meeting that anyone at the company can shadow and watch. So, it's meant to be a pretty democratic process. Um, obviously, this sort of overhead can add a lot once you're a growing company and have a ton of teams making a ton of changes. So, this is something in it in of itself that we've optimized over time. Uh this spreadsheet, the the screenshot right here is an addition we made to that process a couple years ago called mini product review where if your change was small and uncontroversial enough, we would have a separate

15-minute meeting every day where anyone at the company who's proposing a change can sign up with their name on the spreadsheet to give a quick one minute pitch about their change. And those same people, the CEO, the product design leaders would give you a quick thumbs up, thumbs down, or some feedback. And that keeps teams operating quickly while also having a central form where all things like strategy or design alignment or making sure that one team isn't misaligned, everything happens in that one room. So that's kind of the process before and while ideas or product changes are getting approved. But we also have a lot of process to help with this quality bar. Um once features or ideas are being shipped uh we do a lot of testing of our own product do dog fooding and we share a lot of feedback internally. Um this screenshot here is from one of our slack channels where most of the companies in the Slack channel and it's just a forum for being able to share feedback with each other uh about the product. And the thing I

really like about this channel is on any given day you could see like a new grad engineer sharing an opinion about the product and then two minutes later our CEO Luis will uh jump in the replies and say he agrees and that someone should go and fix this. Um so this whole process of making sure that we're upholding a bar is really pervasive throughout every step of how we ship product. Okay. Um, next point is about having internal tools that speed up your process. So, we've talked a lot about all the AB tests that we run. All of that is being done on AB testing tools and metrics, data discovery tools that are built in house. And this has been the case at Dualingo many years, even before I joined. Um, if I had more time, I'd love to show you a demo about how some of this tooling works. But the main point I'll make here is that we have a experiments team within the company that owns these tools and works on improving

them which I know isn't like unique to large companies but it is really useful to have someone within the company that can adapt and modify your tooling to to your team's specific needs because every company or every team has different needs when it comes to their AB testing. So this really speeds things up. Okay. So, thus far, a lot of what I've talked about has made Dolingo probably seem like a very scientific process where every part is hyper optimized. Um, but maybe the most important principle is that we like to be a little bit unhinged. So, if you have Dualingo downloaded, you've probably noticed that every few months we'll change Duo's face to be something weird. Um, and this is a process that we kind of discovered on accident, but have come to really like, and it's also been an effective lever at uh, re-engaging some inactive users. Um, but if you skim this DM from our CEO on the right, uh, for context, I was a

product manager who worked on these campaigns for a couple of years. Um, and got this DM from Luis. Uh, but even when we're so optimized running AB tests, there's also times where we're debating things like whether or not Duo should have a booger on his face. So, as optimized as we are, we still like to be creative, fun. We don't take ourselves too seriously. And I think that is the secret sauce that you need to have even if you are running a tight ship with all these numbers, metrics, experiments. Um, so yeah, don't forget to have fun. And I think that's all I got. I don't know how we're doing on time, but we have a couple of minutes for questions.
Couple of minutes for questions. [clears throat]
Got questions down here. Thanks, Sam.
One minute.
Uh hello. Um Chris from Syncing. I'm just wondering how would you advise uh smaller startup companies to approach uh AB testing?

Good question. I not exactly sure where. Okay. way way in the back. Um, great question. Honestly, I might not be the best person to ask since I had joined Duolingo being mid-stage and I kind of took all these processes for granted. Um, but what I will say is especially when you're trying to find product market fit and it's early stages, it might not be the best time to start AB testing just because you won't have the most statistically significant results or you'll be optimizing maybe for short-term wins rather than long-term uh long-term product market fit. And that's honestly the case when we're launching new courses on Duolingo. we'll still run like the launch of the chess course as an AB test where we withhold it from a few users um because it it gives us some baseline but in general we're not really like studying the metrics of the AB test to determine whether we should launch. Um so maybe that's a non-answer but in general uh

you should be careful about when to start AB testing I would say.
Thank you. It's exactly what I tell the team to leave it a bit later.
Great. We have another another question down here.
Thank you.
Could you hold your hand up again? Oh, thank you.
Also, I think I'm on a panel right after this in this room, so hopefully more questions can be answered then.
Hello. I was wondering about the experiment um for district repair where the revenue decreased by I think 50%.
Yeah. How did you convinced your stakeholders to roll out this?
Yeah, this is a good question and the reason I put relative in parenthesis here is I mentioned in the talk to that it wasn't like our total IP revenue that was cut in half but rather the revenue that was created from this uh purchase point dropped by half and yeah the internal conversations for this were tricky. At Duelingo, we have a growth or and a monetization or and uh by design,

we'll often debate and argue with each other of what features should be oriented towards revenue and what features should be oriented towards growth. Um so in this case, we knew that the streak was primarily a growthoriented feature and we'd rather have someone stick around and be committed to Duallingingo than squeeze a few extra dollars from them because they missed a couple days. So in this case it was [clears throat] honestly more of like a philosophical conversation of where do we want our product to be and in this case we wanted to prioritize the longer term engagement over the short-term revenue.
Any more questions?
You're right next door. Have you got the microphone there?
Hi. Uh this was really amazing. Thank you. How do you have a specific pro process after you roll out an

experiment? Like this is your new control. How do you track performance after that? And do you have cases where you go back into experiment same or like very similar things? Do you have a philosophy on how that works and like how taking the learnings and you make sure that you're not going back to the previous question innovating?
Yeah, got it. I guess maybe to give a little bit more color on the AB testing process. I'll go to the slide with the yeah the report. So when we uh run an experiment, we'll have like a internal page dashboard that has the metrics and often what we'll do is when we're ready to make a decision or call on the experiment, we have an experiment analysis process where usually the product manager, sometimes it's a data scientist or engineer will summarize the learnings from the experiment. And we've gotten a little bit worse at this over time, but in the past we'd be really good about sharing out the learnings in an email format. Back then it was to the whole company where like this is what we

learned. These are the key metrics. These are the learnings and next steps and that makes sure there's like uh a collective like learning process where from every experiment we're like taking away what we need to and we're not like repeating mistakes or testing things that had already worked. Um yeah, I think that answered your question. Let me know if I missed anything.