Lessons from Duolingo: From Startup to IPO

Hi everyone. Oh, we're really nice and on the sides here. Thanks for joining me this morning. And super super excited to be at GDC. So this talk is lessons from Duolingo from startup to IPO. And there's actually a lot to get through today. So let me first tell you about what Duolingo is and I'll tell you why I'm here to talk with you about it. So Duolingo if if you're here hopefully you know what it is. It's the most downloaded education app with over 42 million monthly active learners. And it was designed to make language learning feel like a game. We got inspiration from games from game creators like you on how to make learning as fun as possible. We launched in June of 2012 almost a decade ago and we IPO'd last summer which is a really really exciting experience.

So who am I why am I here talking with you about this? My name is Karen. I have a background in computer science and neuroscience and I had the pleasure of being with Duolingo since three months after its launch in June of 2012. So I started as a brand new bright-eyed bushy-tailed software engineer now a senior engineer senior director of engineering. And I love coming to GDC because one is I love games. Duolingo has been inspired through games throughout time but I think you know game adjacent industries have a lot to also sort of offer and learn and share with the game industry. So there I've been to GDC before twice in 2019 we gave a talk on how to make learning hard things easy using data from Duolingo in summer of 2020 we talked about sort of our AB testing framework and how to turn good intentions into good numbers. Was really sad to miss 2021 due to COVID and such but so thrilled to be back here in person with you all in 2022. And what I like to share every time I

come to GDC is my current obsession with games. It is currently Final Fantasy XIV. So if you play this game or want to talk about it for hours please talk to me after. Okay so for today's talk really wanted to reflect on what I thought were important decisions I've observed through witnessing Duolingo's journey from tiny tiny startup of about 15 people 10 years ago to now a sustainable company of over 500 employees. So that's 10 years of worth of learning to try to cram into a less than an hour's worth of time. We have a smaller group here so I definitely want to leave time for discussion afterwards. So we're going to go pretty fast. I like to slow down during presentations but today we're just going to go. Lots of bullets lots of talk topics. We're going to talk about people and hiring and early hiring decisions business strategy early on prioritization organizational structure and some key processes that helped us and finally some anticipated growing pains. Honestly I ended up hiding like 10 slides this morning cuz I was like this is way too much but if you're interested

in sort of more processes we have as well the slides will be available after this talk. So don't stress too much about writing everything down or taking notes. I'm happy to share the slides with you. All right first some disclaimers that hopefully should be fairly obvious is there is absolutely no playbook for how to be a successful startup. If there was one it would be a much easier journey. We came in at Duolingo with a lot of advantages and had a ton of luck along the way. For example our CEO already had a good track record so funding was probably a little easier for us. We managed to hire and retain really excellent people and the market is very different today than it was 10 years ago. And some really lucky things happened. So our owl mascot became an internet meme sensation usage soared throughout COVID as people looked for productive things to do with their time. So we got really lucky in in many ways. So all this to say the goal of today's talk is to describe what happened along the way of a single company's journey.

And it's not meant to be prescriptive because different different things will work for different companies. But it will aim to be generalizable. So the main takeaway I want you all to take from this talk today is not necessarily what to do or what to copy in terms of what we ended up doing but more what were the problems that we encountered along the way and how might you be able to proactively address them within your own environments. All right we're going to jump right into people. So when you start at a company and I guess maybe just for for my own reference and show of hands like who here is like in sort of leadership or management at their companies? Perfect perfect. How many of you are in companies that are like 50 people or less? Okay cool cool awesome. And how many of you are here just cuz you're either a Duolingo fan or you feared like our owl mascot would come after you if you didn't come to this talk. Okay just me cool. Great so people when you start a company and many of you are at small companies you have very very limited resources as you know. And even as you grow like even with us at 500 people now there will

never be enough employees to cover everything that you want to do. So we focused really heavily early on on growing our talent pool very carefully and honestly quite slowly and making sure we were making the most of the people we had and putting them in places where they can have the highest impact and succeed. So early hiring just have a few bullet points of of things we sort of observed and learned over time. For early employees we found that generalists are super super important. In the beginning people need to wear so many hats. Um product QA engineering even design. And and also they tend to have a high level of independence cuz you just aren't going to have a ton of senior leaders to to go around managing people. So we only really started specializing more and hiring more specialized talent much much later and later in this presentation I'll sort of show a timeline of when we hired certain key roles to give you a sense of how scrappy we were able to be at the beginning. Relatedly it's really hard to find perfect hires at the start. You need to find people who are passionate about what you do and willing to take the risk

to to go along for the ride with you. So you have to get really good at identifying potential. You know it will be really hard to find a resume of this person has taken a startup from zero to 100 and you can hire this person. But the thing is if you find someone who has an incredible growth trajectory invest in them a ton. Like I mean a ton. Give them opportunities because they will grow into someone you probably could not have hired. And I think if you are someone who who maybe is not in leadership manager be this person. Really try to be this person. It will play pay dividends for sure. The other thing I want to highlight here is that early diversity makes it so much easier to maintain diversity and grow it. We found that having a diverse team early on made it so much easier to continue hiring diverse talent and being appealing to diverse candidates and injecting diversity after the fact is always going to be so much harder. So keeping this in mind when building your team is what do you want your team to look like and how do you want it to represent your user base going forward

as well. And finally we this is potentially a controversial point but this is something that we did early on that I actually think was somewhat of a mistake for us that we got better at is we we basically we are a tech company we should have internships. Great internships are fun but we originally looked at it as like a hey maybe and we did paid internships of course but we thought you know maybe we could just get some extra resources for a summer etc. Great. What we found is we it turned out to be a little better for us to prioritize internships as a hiring funnel. And really invest in full-time hires of our interns at least early on until you produce the sort slack to be able to to not have to ruthlessly prioritize as much. So originally right we made internship programs because we wanted to offer the opportunity etc. but then we had to really prioritize our senior engineers time in mentoring and training interns and interns were fantastic if you invest in them for converting to full-time hires. So that's what we had focused on. Similarly too I think there is a fantastic place

for contractors freelancers. They are wonderful. But we also early on really focused on trying to hire full-time as much as possible so that we could invest very heavily in our early employees. Contractors are fantastic for well-scoped isolated straightforward projects that have easy handoff but very early in a company we found there weren't a ton of projects like that and we needed someone who who could sort of own knowledge and become knowledge experts over time for for years to come. Similarly when it comes to shaping your team there are a few things we encountered along the way you might already be experiencing it is the hiring bar needs to remain super high and there will absolutely always be pressure to move it down. So make sure you have bar setters and reward them for being bar setters. Um making sure that you're not compromising because it is so so costly to have hired the wrong person. Super super important. And similarly if it will take six months to hire for a role make sure you look around and see if you can grow someone

into it within six months. Seems obvious in retrospect but you know looking around you and having someone grow into that role is oftentimes one of your one of your best bets. We did this very frequently. We had people even swap roles like I think we had someone convert to a PM who was originally a marketing intern cuz he was fantastic. And and these things worked out super well for us. Other thing to to sort of suggest is I think the we tend to focus a lot on sort of the hiring funnel how could we get as many applications as possible how can we you know be appealing to new hires. One thing to note is just a good employee retained can be a lot better than actually being able to hire someone. So a lot of companies focus on competing for talent but they sort of neglect addressing turnover or reducing turnover. But you know if you if you don't lose someone that means you don't have to hire someone else. So in some ways it's it's a very very high ROI investment. And also long-term plays like I said have deep institutional knowledge and help persist culture. So invest in in both internally and externally when it comes to building your team.

Uh at the same time, this is something that I think Duolingo has gotten better at, but was really hard is if you do manage to hire really amazing industry hires, giving them space to sort of innovate and to challenge the status quo um and be accepting of change, I think is super super important when it came to building our team early on. And then finally, incentives should align with what you want people to do. Uh again, these things said out loud, I think seem obvious, but one of the things we realized is career growth, compensation, recognition, etc. really needs to align well um with what you actually want people to prioritize doing. And that includes hiring and interviewing and building teams and persisting culture. Um so, make sure that the the sort of invisible work that is important to your company becomes visible over time. Great. Uh and so, this is my favorite uh quote when it comes to people from our awesome CEO, Luis von Ahn. Um is this is a principle we abided by for a very long time and still abide by at Duolingo is

it is better to have a hole than an A-hole is what uh Luis said. Um you know, we at times we waited on key hires until we found the right person even if we sup- suffered in the absence of that role. And at times we waited over a year to hire those key hires, but we have never ever regretted waiting. Um so, this is simply advice to have at the beginning is it's going to be very tempting to say, "Well, this person is probably good enough." Or uh this person kind of doesn't fit with the vibe or the culture we want, but they're really accomplished. Um we've been able to resist that and we've never regretted it. So, keep this quote in mind. Great. All right, so let's talk business strategy. And this will be like the meteor part of this talk um is the sort of way that we approached uh strategy going forward with with limited resources. Um early key decisions is and this is hopefully familiar with this audience is we bet on making a mobile app about 10 years ago and rode that wave with the rise of mobile apps. And this is very sort of analogous with the rise of

free-to-play over boxed products over time in the games. Um at the time in language learning was very much a sort of you attend classes or you, you know, learn it on a CD, etc. And we kind of had to make the risky bet of saying, "What if we made an app?" You know, people were saying at the time you cannot learn through technology, but that's what we wanted to do. So, at the start and rise of mobile apps, we decided we're going to invest in iOS, we're going to invest in Android, and we're going to have it so that you can learn a language on your phone. Um and so, the thing to sort of always ask yourself because I think this was actually the most critical part to our success is what will that next wave look like and will you catch it or be left behind? Uh I've attended a lot of NFT talks here. So, we will see if if that is the the next wave. Um uh so, the other thing that we decided to do really early on is to not try to do everything at once. And this is very very hard to do cuz you're going to have naysayers, you're going to have people who are pressuring you to work on everything. And we very intentionally decided to focus first on user growth

and then we added monetization and then we decided to invest in longer-term bets like our learning efficacy over time and then philanthropy in that order. Um and we're going to focus a little bit more on these first two, user growth and monetization, cuz I think they'll be most relevant for this group here. Um the other thing that we did very early on is we invested in data a lot. Um AB testing drove all of our decision-making and allowed us to move very quickly at Duolingo and I will talk a little bit more about experimentation going forward as well. So, let's talk about growth stage first. So, we decided that we wanted to focus on building the most engaging learning product we could and ramp up our user base as soon as possible. Um and I will say we did this at the cost of not monetizing early and also honestly not focusing on learning efficacy very early. And there was a lot of criticism about this. But here was our logic is the more users we had, the faster our sort of experiments can resolve into statistical significance in the data we get from what makes an

engaging product. The faster we can iterate, the better product we can build, and then the more users would come because we had a great product. I think this flywheel was by far the most powerful thing that drove Duolingo to its success in the early days is we simply had more people than anyone else did and we could learn so quickly and iterate so quickly on our product. Um we could provide our services for free, attract a large audience, and then we could bet that before we run out of runway, we could monetize quickly enough. Um So, the question to sort of ask too is our moat was the number of learners we had and as well as a very sophisticated AB testing framework. And the question to to ask is then what are the moats that you want to invest in? Like what gives you that competitive edge in the market at your stage of startup as well. The other thing uh that we wanted to make sure of is to stay ahead of the curve when it came to product. We decided that we were going to be our own competitors um and and try to innovate in this space. So, for example, over time we've done several redesigns of

Duolingo and sometimes had to pause our feature freeze to say, "We're doing all hands on deck to update our product, update all of our assets so the app never felt too old." Um and this is a perfect time also to clean up technical debt, to clean up product debt, to clean up product complexity um and was something that we definitely invested in a lot over time. So, I wasn't kidding when we said we do a lot of experiments. Uh this is the number of experiments we create a month at Duolingo. You can see it's hovering around or near 300 experiments a month. That's like 10 a day that we run. And when I say experiment, it's things like, you know, as simple as let me change the copy on this button and see what results in our click-through rate um all the way to this is a whole new feature that we're going to AB test and slowly roll out. Um and we we are able to do this because our user growth, this is our our monthly active users over time. Um you can see we get spikes on New Year's because of New Year's resolutions. Um these two graphs are very similar, right? We can run a lot more experiments with more learners that we have and also running more experiments allowed us to make a

better product that retained and attracted learners over time. So, this is the sort of power of our flywheel. And after we accomplished a massive number of learners, we paused and said, "Okay, well, how do we survive as a company now in terms of monetization?" Um so, stage two, monetization. Now that we had a ton of users, we could have the privilege of of turning a switch kind of to start monetizing and extend our runway. So, the way that we approached this is it was it was honestly really hard. Um we started with ads. Honestly, we didn't like ads. Um we we really liked having a very clean product, but we started with ads because we decided, "Hey, this will buy us runway to figure out a longer-term bet." Um and that longer-term bet was a subscription model. So, Duolingo monetizes through a subscription that gives you some perks. Um and ever since our subscription model, we started uh also exploring the potential of in-game currencies and in-app purchases also inspired by games. So, the other thing that we also added in was starting to bet on new verticals. So, this is the order we did things is

we did ads, we did subscriptions, we're investing in in-game economies now, and we also are investing in new verticals to monetize at Duolingo as well. So, this is the breakdown right now um as of I think a few months ago in terms of where our revenue comes from. You could see it's dominated by subscriptions um but the other sort of assets you see, ads started out being everything and then dropped to 15%, which we're we're quite happy about. Um so, a few things learned along the way from monetization. Um it because we were such a driven mission-driven company at first and people who joined Duolingo early on really just believed in that mission of free education, we had to do a lot of work to align our business objectives with that mission. Without that alignment, there was no passion for for the product. And one of the hardest things we had to do was honestly convince ourselves making money wasn't evil. And I'm I'm not kidding when this like took several months. Um there was a lot of people who thought we would simply dilute our mission or stop caring about our product once we started monetizing. Um and I think the the only way we were

able to really combat that both externally and internally is to really really stay true to being an authentic and mission-driven brand and saying, "We're going to need money if we want to win in this space and if we win in this space, it's going to be good for the world." Um so, we had to convince our employees. We had a lot of like sort of open discussions and forums about it. We listened to concerns. I was I was still software engineer at the time. I wasn't even leadership and management, so I still remember this kind of from both sides. Um and and we did it. And I think one of the things we also did was we we wrote down our monetization principles to sort of say, "Everyone should always be able to learn for free on Duolingo. We're never going to gate anyone with money." And I think that guiding principle is still that one that we hold today. Um our user base loyalty was very high at the time. I think it's it's important to sort of demonstrate your mission and not just say it. Um and it played a really critical role in in how accepting people were when we started monetizing. And I think one of the affirmations for that is once we released our subscription, you know, it came with perks like you can download your lessons offline, you get I don't know, some cute

like skins for the duo, etc. The number one reason people subscribed was to support the mission of free education. Um and I think that was saying a lot in how we branded our monetization and how our intentions were made clear to our learners. All right, so then we monetized. We decided we were going to keep living as as a startup. And this stage I won't spend too much time on. Learning efficacy is maybe not that applicable to games, but the reason I'm bringing it up here is this was the stage after monetization that we decided to to really invest in longer-term bets. Um and we decided to start investing heavily in measuring learning, measuring efficacy, doing curriculum overhauls um because we needed to fulfill the promise of actually teaching you a language, right? We didn't want to be a scam. We didn't want to be uh a place where people only went to casually learn a language and then stop forever. Um what this taught us was actually that some things are not measurable and this was something that was hard for us to learn is as such a data-driven 300

experiment per month company, we had to learn that there was a huge important aspect of Duolingo that we would just never be able to really AB test well. So, we had this conversion in the sort of renaissance within the company that we needed to stop saying every experiment needed to win and increase metrics and instead sort of shifting to let us increase pedagogical like value based on the experts who know how to teach a language and let us simply try not to hurt our guardrail metrics. We went from neutral experiments are good. Um if we are increasing the efficacy of our product. Um so, the reason we invested in all of this was and the takeaway I want you to take from this section is we knew that learning-based initiatives would increase the cap of where we could be 3 years from now. You know, we would hit a hard wall on growth or a hard wall on monetization if we were never able to extend how far we can get you in learning a language. So, the question to ask yourself is what will you be limiting by 3 years from now? What do you think the cap will be in terms of potential of your product? And what can

you work on today to uncap? And we decided after we had enough learners, after we decided, you know, we could be sustainable as a company, this was our cap was learning efficacy. And then finally, fourth stage um that we're still expanding on is innovation and philanthropy. Not saying innovation can't be there at the beginning um but sort of risky bolder bets in innovation. I think you get more slack to to do once you're sort of out of the in the clear. So, we started investing in in-house technologies and tools at this stage and note that we didn't do it earlier than this stage. So, we have our in-house analytics framework, internal tools for curriculum creation, custom speech recognition, etc. But the the main advice here is to not reinvent the wheel too early. So, use existing stuff if it's available cuz the time uh of of your team is the most valuable thing you have right now and build internally only when it breaks or if you've outgrown it. Um also new verticals, bold bets. We had several new verticals. We failed at most of them. We succeeded in a few of them. Um so, we did language certification,

literacy, kids, math flashcards. Some of these succeeded. Some of these did not. Um and I guess one like slight point of guidance and this might be different for your companies is we decided to invest about 70% in optimization in the things that we know would get us gains in building the machine, in running the machine, and about 30% in R&D and innovation and exploration. And we try to design our teams such that that is true. Uh so, a few a few learnings. It's really, really easy to go overboard and say and it's really, really hard to say no to philanthropic things. You know, when someone says like, "Hey, we should do this." You don't want to say no if it's a good thing to do. Um especially with a very passionate employee base. But the advice I'd give especially when you're small is to focus on the impact you're best at making. So, for Duolingo, it was that we were uniquely positioned to do things like support the addition of underrepresented or endangered languages on our platform. Um but one of my uh favorite quotes from our CEO is like, "We like the whales, but we shouldn't focus on saving the whales." And I think that is sort of a sign of, you know, Duolingo's not in in

a position to really go and try to save save the whales or do a lot of other um philanthropic things that weren't directly related to our company. So, really focusing on the things that you can have unique leverage on will help you kind of scope. Um relatedly, right? Prioritization. So, making the most of the resources you have at this early stage which many of you are at um is probably one of the most important things you could do. And you though the strategy I just described already defines what you prioritize, um there are a few other things we learned along the way. Um so, early on the best advice uh that I got from from Luis was to realize the value of my time even as a very, very junior engineer. And I remember this story from when I started at Duolingo is I was I think I was just like mad at someone on the internet or something saying something wrong about Duolingo and he said, you know, it if every employee had a share every employee has a share of like a 100,000 users to take care of, whatever you decide to work on every minute of your day, think about those 100,000 users that are in your care. Um and that helped me really understand

the sort of opportunity cost that comes from going down rabbit holes um and what I'd be sacrificing to go down those holes. So, training this mindset in in all of the early employee base, I think was super, super useful for helping to protect us from ourselves in in investing in in lower ROI work so that we could maximize our impact as a very small team. Um this sort of philosophy will just make a lean work force go a really long way for you and stay focused. Uh the other thing and this is like a quote you don't have to read fully, but the TLDR is, you know, one thing that we learned also is as a small team, it's very easy to sort of like go down the Reddit rabbit hole or read an article about your product or get really angry at people who are wrong on the internet. Um but one thing that we found is we were listening to channels that represented just a very, very small portion of our user base. And that's not to say you shouldn't listen to your users. I think this this quote is a bit strong. But one thing to recognize is making sure you're actually listening to

your users and not just listening to a few of your users. Um so, we had to really, really take user feedback with a grain of salt. Um and what we decided to do is we try to listen to our users through data and their usage instead of directly through qualitative comments. Um a few caveats here is I 100% believe in user experience research. I think UXR is fantastic and it is absolutely necessary. Qualitative feedback is really important. But the type of sort of feedback that we didn't find as useful early on was was things that are not conducted by professionals, right? Like an engineer, me, reading on Reddit that someone is mad at the product. Um you know, that tended to just really be red herrings to the actual underlying problems. Um so, thick skin helps. You know, remember it's the most upset people that that tend to be the most vocal and I think looking at our data and saying, "Oh, actually the vast majority of our learners really love the redesign we just did." versus like reading the really long thread about how many people hated our redesign was was affirming. So, try to use data to to really get a

clearer sense of what the average learn user uh on your product is feeling. Uh another learning about prioritization is it is so, so, so tempting um to sort of greed and uh try to scale a product or feature out before you're confident there's product market fit. So, how this looks at Duolingo is we'll come up with a feature. We love it. It's great. Uh we'll launch it on iOS first. We'll test it. Looks great. Uh and then we'll immediately want to say, "Let's get more learners using this feature by having it on like web and Android immediately." And one of the things that does is it's the case that it it really actually slows down iteration on that. So, one of the things we've learned to do is try to be really patient with product market fit because, you know, the moment you want to make a change to that feature, you now need to change it on three different platforms instead of being able to scrappily prototype it online. Um so, it took a lot of discipline uh to to do this and when you don't have a lot of resources too, you know, that investment in having to port something three times is is a lot. Um so, just keep in mind iteration is

also a resource and the ability to iterate quickly is also a resource you need to account for when it comes to decisions like this. Um and then the the main thing here is not yet is a super, super strong response to both yes and no questions. Um so, we had to get so much better at saying not yet instead of no because honestly ideas, you know, can can become viable over time. So, the answer should never be no to to an idea. But it can absolutely be not yet. And then honestly, when it came to things like, "Let's do this new crazy feature, etc." also saying not yet instead of yes was also very powerful for us. So, highly encourage you to to make use of the not yet, um to put it in a backlog, to to think critically about it, but to feel free to say not yet to both yes and no questions. All right. So, we're going to run through organizational structure real quick. This is kind of the evolution of how our our org evolved over time. Um early days. So, we started with one

huge flat team with individuals in charge of certain metrics or projects. So, on my first day at Duolingo, I think I was employee like 15 or so. Um Luis came to my desk and said, "Hey, try to grow user retention." Like literally, I was just like a brand new engineer I sat at my desk and he was like, "Grow user retention." He just left. And then I that was it. So, we uh got more sophisticated over time with our onboarding, but you know, in the beginning, we we basically had projects and metrics that we were all in charge of. Uh everyone reported directly to the CEO in the early days and we had really excellent people. So, we didn't worry too much about titles, hiring very senior leadership, etc. That landscape might have changed. Like I said, this is just a story of how we started. Doesn't it's not meant to be prescriptive. Um but what this did allow us to do is actually set a very non-political foundation early on. People weren't too pre people people were preoccupied with their work and not too preoccupied with their status. Um and I think that really helped us. We hired our first manager, who's now our SVP of engineering, Natalie, my manager, um in early 2015. Now, if you remember, we launched in 2012.

Which means we had about 2 and 1/2 years without managers. Um I mean, technically, I guess Luis was everyone's manager, but um I thought, you know, I I wouldn't necessarily recommend this, but this is just what happened. Um and we continued to stay as flat as possible. So, I can say this, right, as an engineering director that we we really tried to not have more management than we needed at any given time. But everyone's use case is different. So, what I thought might be fun to share is a sort of timeline of of functions for this is fun to look up for myself cuz I didn't realize this was the case. So, at the beginning, we kind of only had design and engineering. Um I think we had like one person managing our finances, things like that. Um and then we hired marketing person in early 2013. I'll be transparent and say like marketing person was great, but like we actually decided we didn't want to invest in marketing at the time. So, we kind of like didn't hire any more marketing people for a while until probably like 2017 or so. Hire our first PM in 2014, our first manager in 2015. Uh it was an engineering manager. Learning scientist, that was like the phase of learning and and long-term bad investment in 2015.

Then there was talent acquisition HR. Notice there are four years there where it was like engineers were just hiring engineers and designers were hiring designers and we didn't have people for for explicit talent acquisition at the time. Um we hired data scientists even though we were AB test obsessed uh in 2017, so that was five years until we hired our first data scientist. And now we have excellent roles like program manager, technical program managers that we've hired in the past two to three years or so as well that really really help us grease the wheels. Um so I thought this was interesting because in some ways I think early on in a company you feel like you really need someone an expert in everything and certainly I can assure you that getting excellent people in these roles is fantastic. But you kind of also make do with what you had. I mean one of the first times I met our CEO he was carrying a box of t-shirts to like a tech recruiting conference. I was like, "Okay, this is a down-to-earth guy." But we all had to basically do whatever it took to to get things done and we in some ways hired these roles very late also because we wanted to wait for the right person to establish what that role and function

should be. Um and sometimes that took a long time. All right, so let me talk a little bit about the evolution of our organizational structure. After we just had this very flat structure for a long time, we introduced teams cuz we decided it'd be a good idea for more than one person to work on retention or or whatnot. Um so we base teams around metrics, features, or problem spaces and we wanted teams to be able to accomplish their goals pretty autonomously. So there are several ways to do this. Some people have like, you know, all the iOS engineers here, all the Android engineers here, etc. There's a design team, etc. We decided to do them vertically so that there were little pods of engineers and designers and program managers and learning scientists together. Um controversial, but we actually did this so we would minimize cross-team collaboration. I know that's like not a great thing to say, but it's because we didn't want teams to always feel blocked on other teams and their priorities. You wanted teams to feel like they had a sense of control over their own destiny and success. So we decided to build teams vertically with all of the components they would need to be successful at accomplishing their goals. Um so our first teams were actually like

learning, growth, and monetization, which if you remember maybe the the sort of phases of prioritization we went through. And we also at this point created the role of team lead uh someone who led the team and managed everyone on it. This will be relevant in a bit because learnings uh where eventually your teams are going to grow way too big and you need to fragment them and narrow their scope. Um so for example, our growth team fragmented into get teams like social, gamification, and retention as all elements of growth that had narrower scope. Um but honestly, our limiting factor when it came to making more teams was ready leadership um when it came to making new teams. And this is why I introduced the team lead concept earlier. I think one of our biggest learnings was, you know, we tended to set a very high bar for what it meant to be a leader and generally we'd rather have a more senior person double up and kind of be stretched than in lead two teams than have someone more junior as a team lead. Not saying this is necessarily right, but this is what we did. And here are the reasons. One of them is you really risk stunting technical growth. And when I say technical, it doesn't have to be engineering. It's sort of like all the all the things you learn from building

something, from doing your day-to-day job, you stunt early if you put someone in a leadership position that occupies all of their time. So being really sensitive to what growth are you potentially stunting when it comes to putting someone in that position early is really important. Um and it's also really hard to unmake someone uh uh a manager or a team lead. Once someone is in that position, it feels like a demotion even though it's really not um to to put them back on the individual contributor track. So be very careful about this as well. Um okay, after we had 33 teams I'm I'm saying like we kind of do things as late as possible. We had 33 teams all like this really flat structure. We decided we probably need some other layer. So now we have areas um composed of several teams. We now have area leads and this structure was really useful for sparse resources. So for example, we're very low on designers at Duolingo and we had a situation where a designer was embedded on like four different teams and just like stressed. Um so what areas allowed us to do were to group teams under this umbrella called area, have area level design, and be able to take care of the designers in terms of their priorities and just saying you're in

charge of this area, not these four teams. Um and area leadership was responsible for status, for summarization, goal setting, etc. Here are the areas that we have today and we also had something called business units um which were new verticals or little mini companies within the company. So it's possible, right, to have little mini companies in your company that are working on independent verticals as well. Um and then finally, not all teams are in areas at Duolingo. So anything that is a little more functional, marketing people, finance, strategy, business um or function-based teams that support the rest of the company just remain teams. And that's totally okay, not everything needs to fit in this structure. Um okay, so a few lessons learned is uh just like in coding, naming is strangely extremely important when it comes to teams. Like this is one of the things that you would not think is important at all, but somehow was really important for us. Um naming a team defines the assumptions about what area of ownership that team has and people will therefore reach out to teams that they feel like are closest to the information they need. And if

that is very obfuscated, people will not know who to go to um for things. Um it's also the case that employees will tend to remember team names, but not mission statements or the goals of each team if you have a lot of teams. So all they have to go off of for knowing what the company finds important is the teams that the company has, right? So if you have a team called retention, means the company cares about retention. Um so I would all this to say somehow naming has become very important and we run into problems in the past where we name something poorly and it just confuses everyone in the company. Uh so naming is important. The other thing we realized is sometimes we would do organizational um reorgs or have brilliant plans for how to have a very perfect organizational structure and then always always at the end we'll pare it down to the most minimum viable plan that changes the fewest number of things. Mainly cuz manager team changes um team restaffings are often very disruptive. Um so we tend to do smaller organizational changes, little tweaks like one person moving, etc. every one

to two quarters. We do large reorgs about once a year and we add a layer or every three years or so. Um so this is sort of like the cadence of of how we do things. Um all this to say be very practical. Um so we will never ever have a perfect organizational structure. This is literally what I personally work on almost all the time. Um but strive for slowly improving your organizational structure over time to keep the minimum acceptable level of chaos, I think is what I'd like to say here. Um there will always be overlap, gray areas, things will fall through the cracks, you won't be able to staff something, or there won't be the quite the right leadership in time and that is totally normal um and expected. Uh so there's no right answer. There are kind of two paths you can take and you can decide for your company where on the spectrum you want them to lie. You can either have a really stable, rigid teams that is great for building team expertise, loyalty, stability, a sense of ownership, etc. Um having sort of the same people on the same team for long period of time.

You can also have really scrappy, flexible, you're expected to change teams all the time, highly agile teams. That's great for innovation and always working on higher ROI things. If you want to always put your superstars on the most important thing, this structure is great, too. Um and there are pros and cons of each of these and maybe you actually have a few people on on each of these spectrum as well. Um there's no right answer, but here are the options available. All right. And then finally, yeah, different things work for different companies. Finally, processes. So originally there were like eight processes here and I'm only going to share two with you today cuz one, there's not enough time. Two, it's probably really boring to most people here. Um but one process that really drives uh Duolingo is OKRs. I know no one really likes hearing about OKRs, but here's why I'm talking about them. It's really good for setting goals and alignment. Um if you don't know what OKRs are, they're a set of objectives and the key results you want to uh accomplish towards that objective. Ideally, they are measurable and you will receive a grade at the end of the quarter.

Um the key thing here that we really had to untrain our perfectionists on is the goal is to get a weighted score. And when I say weighted, it's weighted based on based on priority of 0.7 at the end of a quarter. If you get higher than a 0.7, it's actually not great. It means you weren't ambitious enough with your goals. Um so it's and the other thing I just really want to emphasize, especially to leaders here, is this measure and this grade is a measure of how well you planned, not a measure of how good a team is. Um if you find that a team gets a 0.3 at the end of the quarter, the correct uh discussion to have after that is how did we fail this team in terms of planning? What were the unanticipated things that happened? Why were our estimations off? It is not about the performance of the team. It's about how we can plan and scope things better to prevent crunch. Um so problems addressed here. Align on goals, communicate priorities, have accountability on these teams, and practice much better planning and prediction when it comes to the goals and how realistic they are setting for your teams. Um so here's a a concrete example from I

just pulled this from like retention, obfuscated some numbers. You don't have to understand what this is, but this is kind of what an example of an OKR looks like here, right? Is the objective is just to improve the retention of one of our features um from X to Y, you know? And then here are the P0s, the two most important things we want to do for that. Here's the P1, here are two P2s. Um and one thing I just want to highlight here is note that these all kind of start with verbs and have things like reduce latency, run experiment, port to improvements, launch experiment, run experiment. And what I want to say here is one of the biggest tips I have is, again, naming and consistent terminology is actually super super important. Um so we have these terms at Duolingo. There's especially around our AB testing framework of launch, run, roll out, port, prototype, get stakeholder approval. And the reason precise terminology is important is it helps control scope. As a leader, you want to protect your team from scope creep. And that means not promising, let's just launch all these experiments, but really

diving into what are the right milestones I can deliver for these very arbitrary quarter boundaries, right? It's not always going to be the case that features should fit nicely within quarter boundaries. So how can you break that up um so that the right milestones are expected out of your team, whether it's just let's just get stakeholder approval, get on the same page about the product we're building versus let's just implement the thing, but we're not going to be able to roll it out to learners yet. It won't be ready or QA'd by time. If you have precise terminology, you prevent that scope creep and you protect your team from from crunch. Um, other thing we've learned is collaborative OKRs across two teams that have their own separate OKRs should be verbatim on both teams' OKRs with the same priority because it's really bad if this is the most important thing for this team to collaborate on and the least important thing for this other team to collaborate on and have different priorities cuz one of them is going to do it, the other is not. And then you screw over the team for which it's really important. So OKRs are a way to sync on these things very explicitly that we agree this is a priority for both of our teams.

Um, and this sort of last point here I think is is OKRs are very strong incentives for teams. So capture work you want to make sure gets done, especially traditionally thankless work. I am a huge proponent of this. Documentation, fixing bugs, stability, cleaning up technical debt, all really need to be represented and celebrated and given credit for when it comes to planning. They can't all just be top-line business metrics. Um, because if you want these things to happen, and it's very important for these things to happen, you need to reward and recognize that. Um, okay, something a little more fun outside of like the manager world is one of the things that we have learned is to really, really invest in greasing the wheels of the things that your company spends a lot of time doing. And since we run 300 experiments a month, right, we wanted to invest in what we call automated experiment reports. So every single test we run has a report automatically generated nightly including all the KPIs across the company that people have submitted they care their own teams care about. Um, and whether that experiment had a significant impact on any of them. So

here's a sort of example experiment report. I'm not sure if you could read it. You know, here I can look at my experiment learning multiplier in decay web, whatnot, and see like, ah, it seems statistically significant for increasing active users uh over time. Great. And if I scroll down this, there would be revenue metrics, retention metrics, and every metric every team in the company cares about to make sure I'm not screwing up someone else's metric over time. And this being automated meant it was super transparent um and and and prevented me from sort of messing with another team's metrics. Um, so the problems we addressed here, very, very fast feedback. These reports were generated nightly. Um, included all the guardrail metrics so that you didn't have to memorize them or know who to go to or know who what to look at to do it. Um, it makes experiment analysis super accessible, especially to non-technical people or non-statisticians like me. Um, and it vastly, vastly accelerated decision-making on whether or not to launch or shut down experiments. Great. Last section, we're almost through. I told you this will be rapid. So relatable growing pains that you will probably inevitably encounter.

Um, first we're going to start with engineering pains. Ownership voids and untenable backlogs. Hopefully relatable. Um, the way that we are trying to address this now is to form centralized bug triage groups with very senior leaders so that we could communicate this is actually important and and worthwhile time. We're also now forming decentralized groups per area. Um, and also adding a ton of automation for duplicate reports. So one of the things that we started drowning in is people reporting the same problems over and over again and it literally took all of our time to even dedupe those than to even start addressing them. So invest in technology that helps you be able to do these sort of lower ROI tasks. Um, and also as I mentioned, OKRs including bug squashing and triage in those OKRs means people will get protected time to do them and you account for them in your planning. Uh, we're still bad at this. Other thing, legacy code and technical debt. Um, occasionally we will actually just stop and do all hands-on-deck projects to clean up uh code base rewrites or redesigns. Hopefully you don't get to that point. You will probably get to that point and

it's important to kind of put the foot down and say we just really need to do this to accelerate how quickly we can code in the future. Um, and some quarters we actually ask all teams to include a KR dedicated to reducing technical debt. Um, technical debt will always be one of those high importance, low urgency things. So I think that the key thing to learn here is is to to really invest in those things. We also have an annual company grease week where everyone in the company takes a week to just clean up shop. Do those refactors you've always wanted to do, write that documentation you've always wanted to have, etc. No one's going to bother you that week. You won't have any meetings that week and it's really important that everyone does this at the exact same time because if it's only a few people, you're just going to get drowned in the work that other teams are asking of you anyway. Communication pains. Uh, so a lot of stakeholder man. This is like the one emoji I managed to like insert in this presentation and I decided on this one. I'm that stakeholder. I'm that problem, so I can talk about it. Um, so one of the things we did was you know, it was the case that teams really needed to know what other teams were doing if they affected their metric or their feature.

So we have this template now where there's just a list of all required collaborators or people you need to contact um in order to to get your feature approved if it touches any one of these domains and that's been really, really helpful. Um, and we're also just investigating different stages of feature development. When something's just an ideation, you don't want to put red tape on it. So we're sort of trying to set expectations now for how far along a project you need to be uh in order to start pulling in stakeholders to really minimize stakeholder management at the beginning of projects and only introducing stakeholders when they're actually needed. Um, other communication pain is onboarding and documentation. We have an internal content management system now. Was highly worth investing in where we just do automated training uh for for all of our new employees. Um, and we have a group that audits the quality regularly of this. Very, very worthwhile investment for us. Uh, we standardize our career ladders to share a common core so that all of our levels would be fairly consistent. And then we also hired people to lead this initiative uh starting with the leadership training.

And then the final pain I'm trying to talk about are emotional pains which are usually the hardest to deal with. Um, handling low performers and exits was honestly very, very hard for us and still is. Um, Duolingo needed to get a lot better at acting on low performance. We were often too kind. Um, so we created a playbook for managers on identifying and handling low performance and we found that often managers actually just lacked the confidence to even note or judge someone as a low performer. Um, so we gave much more objective guidelines for that so also everyone could understand what it means to to not be doing well. Um, and the other emotional pain was really pivots and sunsetting projects which I think a lot of people encounter early on. Uh, what we found is early communication transparency around the reasoning behind it is like if you believe in that reason, you shouldn't be ashamed to share it. Um, it and it often times we actually found that it was the team building the feature that ultimately decided it should be sunsetted and that is such a healthier way to go about it as well. So retrospectives are great and healthy and we really need to encourage blamelessness in those. Um, and I think the most important thing

is to reward innovation even if it results in failure. If you want an innovative company, if you want your best people working on innovation, they need to not have the fear that their careers would be ruined if it doesn't work out, right? That's not how innovation works. So we need to figure out ways to celebrate our not even failures, but to celebrate, you know, projects that didn't pan out and the things that we learned from it. And so many of our failures at Duolingo ended up turning into foundations for future successes. Um, great. So most people need to know um Yeah, right. Most people need to know that having a project not succeed or having a project sunsetted won't set back their career. All right, we are towards the end of this so I'm just going to rapid fire through the recap and takeaways here cuz we talked through a lot of things. So first we started with people, right? Hiring the right people, single most important thing you could do early on. Um, generalists, highly recommend. You want people to wear many who are willing to wear many hats as well. Um, keep that bar high and retain good talent. Uh,

retaining good talent just as good if not better than being able to hire new talent. Um, and ensure that employees understand the value of their time and prioritize that time accordingly as well. Um, one thing that we noticed your leadership layer will likely be your limiting factor so invest in growing good leaders and identifying potential there. Uh, and also relatedly, resist the temptation to force someone to be a leader before they are ready. You will stunt their growth. And then finally, it is better to have a hole than an a-hole. Like I said, I will send out these slides afterwards. Um, great. In terms of strategy, be aware of the trends. You know, we rode that mobile wave and it was excellent for us. Um, identify and preserve your moats. What are the competitive advantages that you uniquely want to invest in? Ours was a huge user base and AB testing. It could be very different for you. Uh, align as much as possible your business objectives, your mission, and your incentives. If they are aligned, you will have smooth sailing. If they are misaligned, there will be a lot of unhappiness. So do your best to align

these things. Um, and try to have a rational and emotionally detached lens when reading user feedback as well. Uh, try to listen to the data, listen to all of your users, not just the most vocal ones. Um, and make room for innovation and failure especially. Uh, finally, prioritization here. Focus. Don't try to do everything at once or be good at everything at once and really, really focus on things that are most important to you at the time. You will not have the resources to do everything at once. Um, get really good at saying not yet. Optimize for fast, well-informed decision-making. Decision-making tends to be the thing that drags things on. So, invest in making it speedy and objective. Uh and finally, use third-party tools when they are available. Do not invest in in-house things until you outgrow them and truly outgrow them. Uh and then finally here on the prioritization front, uh as a reminder from our CFO, you can like the whales without committing to saving the whales. So, when it comes to

wanting to do good, wanting to do philanthropic efforts, really focus on the things you're uniquely positioned to benefit. Um and that will give you the highest leverage and impact. Organizational structure, be patient. Don't jump the gun on these things. Inserting organizational layers, scaling out a feature before you find product market fit for it, putting someone in a leadership position they aren't ready for, hiring functions or positions before you need them. As we demonstrated, we actually waited a really long time to hire certain functions. Um and also name teams well. Uh try to optimize enabling team to get things done autonomously and to feel a sense of control over what their success can be. Um find the right balance between flexible flexible, agile, scrappy versus stability and building knowledge leaders and ownership. Um And finally, recognize that organizational structure is always going to be a work in progress and you might get grumpy about the cracks and holes, but it's okay. They will always They will always be there and uh building up a sort of tolerance for vagueness, I

think, is one of the the best things you can do. Uh and finally, I used to have a lot of bullets here, but the only thing I have to say about process as a takeaway is only implement and create them when you need it and no earlier. And if you want to actually look into the processes that I described today in all the hidden slides, uh I'm more than happy to share that after this talk. All right, we did it. I think we have 7 minutes for for Q&A. Thank you for riding along with me. 7 minutes for questions. I have to take my mic. Hi. Uh uh the talk is amazing, absolutely amazing. Uh can you share some light on the AB testing on unmeasurable things? I mean On Yeah, on unmeasurable Yes, because neutral results is not enough, right? We will know it's A or B. Uh sorry, can you repeat the last question? Uh just want to know more about the AB testing on unmeasurable things. Yeah, yeah, yeah, absolutely. Um so, the and and there's a I'm I'm happy

to share share my 2020 talk that actually dives into this. It has a whole section on this. But the main thing is so, uh the main thing we couldn't measure well was how well our learners were actually learning a language cuz learning takes a very long time. And most AB tests we run, we want to run within 2 weeks, but 2 weeks is not enough time to learn something, right? Um so, we spent a lot of time actually trying to invest in how can we quickly measure learning? And we honestly, I think, wasted like a year trying to do this. And instead, we flipped the approach to saying, "Okay, we have all of these what we call guardrail metrics. They're things like retention that we never want to hurt. They're things like revenue we don't want to hurt. And so long as we don't hurt metrics, let us implement changes that are described by a pedagogical expert and our learning scientists as good changes for learning and let's call it a win even if all the if if all the metrics are neutral, but an expert says this is better for learning, that is going to be our new win. It's not going to be an increase in user active, it's not going to be an increase in retention, it's going to be we removed more translation from our app

and are doing more monolingual exercises or we were able to serve learners harder content without them getting frustrated. Those were the things that we celebrated as wins even if they didn't move our numbers because we just knew it was better pedagogically from the experts that told us so. Does that make sense? All right, yeah. Thanks. That helps a lot, yeah. Hi. Hi. Um I'm wondering if you can talk a little bit more about how you guys think about monetization um because I mean, clearly you guys have to invest a lot in marketing and that's very expensive and you know, you guys have certain runways. Mhm. Um but at the same time, you do want to move on to uh efficacy, right? So, at what at what point do you say, "Okay, we're making {quote} enough money to maybe make that our secondary focus, right?" And start working on other things versus, "You know what? Don't worry about anything else. Focus on monetization." Like what How how do you find the right balance there? Yeah, that's a great question. Honestly, one I'm not an expert in at all, so I can only kind of give my personal take on it. Um I think for it really is about what is

the most important thing to your company and is absolutely not a bad thing at all for companies who want to make money, right? Um For us, it really was that based on the the people we've acquired over time and how mission-centric they were, our priority was being the best and biggest language learning company in the world and we knew we needed a ton of money to do it. So, everything when I when I talk about aligning sort of your mission, your business objectives, and your incentives, the sort of I'm talking about is we wanted to make money um and decided to continue prioritizing making money because we knew this was the amount of money we would need to be able to penetrate like the English learning markets and to be able to provide education to the world, etc. This is how much money we would need to do this. So, we decided um to invest in learning as well because we said we one, we actually actually want to teach languages to our learners, that's why we all work here. Um but two, it we would actually hit a market cap when it came to monetization and growth if we only ever served beginner courses or this number of courses, right? We wanted to teach

advanced English so people could get jobs um in in knowledge jobs in English even if it weren't their first language. So, in some ways everything kind of worked together in that we decided to invest in learning once we realized when it would uncap growth and monetization. In some ways, it was it was related to monetization. It wasn't really a if or or. Um it was more everything used to work simultaneously towards our one goal and monetization and learning were both a part of that. So, when it comes to making enough money, I think the question ask is how much do you need to accomplish your goals? And then also, is there anything that you actually need to prioritize a little bit more to uncap your market potential in the future as well? Got it. Thank you. Yeah. I don't know who was first. Okay, go for it. So, I was wondering, you know, back in 2012 when you started, what was the ways that you guys went about acquiring new users? Um you know, aside from just building a nice app. Yeah, sorry. What was the what before new users? Uh

back in 2012. Yeah, sorry. I I think I missed like the verb. What Oh, what what was your mechanisms for acquiring new users? Yes, yes. Um so, one really This is another thing we got lucky on is our CEO Luis gave a TED Talk um about this idea of free education that I think attracted a lot of people to our beta. So, one of the I think that was one of the biggest drivers of people who signed up for the beta version of Duolingo was this TED Talk and I think there was a lot of evangelism in in the more like academic Duolingo actually was born out of Carnegie Mellon University. So, I also think like we evangelized on campus and things like that, too. Um but truthfully, uh we didn't do that much and I think that's why we were lucky is we we actually there was a lot of word of mouth that actually did it for us and that is not something we ever want to take for granted, but somehow when people tried the product, it was it was new at the time. They were basically like, "Oh my gosh, I can actually learn a language like from my house for free." Um and word of mouth was by far the

highest driving factor even since recently of of our growth at Duolingo. We actually didn't invest in marketing for a very, very long time because of word of mouth. Um and I think in terms of organic growth, word of mouth only really comes when you are a very, very solid brand and authentic brand, right? So, we actually almost intentionally didn't do too much like marketing or commercials because we wanted that organic word of mouth to not become distilled. Um so, I would also say engaging with the community um and that TED Talk, I think, were our main drivers. Thank you. Thanks. Good. I think we only have a minute left, but I'm I'm happy to go out to the wrap-up room also for more questions. But go ahead. Uh so, here's a project management question. You I liked your concept of the you have no meetings and focus on those things. Um and with with you having hundreds of new tests a week and talking about or or a month, um talking about needing key stakeholders involved and with work from

home and you know, we've experienced an increase in meetings and fatigue with that. Um are there any key takeaways you guys have found um with, you know, reducing meetings or having some sort of structure around um that during work from home? Yeah, 100%. So, as someone who literally's whole job is to have meetings, like I feel the meet meeting fatigue even though I'm uh hardened to it. I think one of the main things is I think uh for me, at least, there are meetings that drain energy and there are meetings that give energy. Um and I think really thinking hard about inventorying what are the meetings that are draining energy from everyone? They're probably not very useful meetings. You know, and like who are the meetings useful for as well? And is there a different means other than a meeting for those people to get that benefit out? So, we reduced the number of meetings we had um and honestly, in some places, increased the number of meetings we had based on this is could this have just been a Slack message? Could this have been some other process? Can there be fewer people in this meeting? Um I think was was really important for us, especially during COVID when there

was so much meeting fatigue. Um the other thing to do is things like Greece weeks and also, I think we give our our managers a break at some point. I think when it comes to performance review season, we say, "No meetings this week." For for managers, I think literally having just one no meeting week every now and then really, really helps people to to reset um back to baseline. But I would say like the key the the key um culprit in this really is just useless meetings or meetings that are way too long. I have found, this might be controversial, but like 20-minute meetings are really good length for me. I have found that if you The difference between what you get done in a 20 versus 30-minute meeting is the same, but you give people a 10-minute break from meetings by having a 20-minute meeting. Um so I think just really optimizing for making the most of every minute of meeting time such as worthwhile for everyone was was something we had to look at a lot. Thank you. All right, thanks. I will I think give the room back to our lovely tech folks to get set up, but I am happy to to stand outside and answer any other questions. Thanks so much for joining again today.

Thank you.