Measuring the long-term effects of brand advertising (with Carl Mela and Ross Link)

Mobile game developers no longer need to pay up to 30% in major app store fees. With Exolo Web Shop, you can create a direct storefront, cut fees down to as low as 5% and keep players engaged with bundles, rewards, and analytics. Start today at exola.com. That's x ss o l.com or use the link in the episode show notes. The problem is that the distinction needs to be drawn between the competence of the economists and the correctness of their analysis.
Hello and welcome to the mobile dev memo podcast. I'm your host Eric Seertfort. I'm joined by two guests today, Carl Mila and Ross Link. Carl Ross, welcome to the podcast.
Thanks. Good to be here.
Yeah, thanks. I'm delighted to be here. I've listened to the podcast. It's excellent. I'm honored to be asked.
Well, I'm very grateful that you agreed

to join me. So, so Carl, you were actually the first guest that I've brought on as the result of request. So, someone requested that I bring you on and you know, obviously you're familiar with your work. You're a very prolific academic in the space. And I said, "Yeah, that's a great idea. Can you introduce us?" and he's he said, "I don't know him. I just think he'd be great to be on the podcast." So, fortunately, we do have a number of mutual acquaintances. So, so Garrett Johnson, my friend, connected us, but also you worked with Julian Run at at Duke. He he told me that you were the one who brought him out there. Is is that correct?
Yeah. Yeah, that's he's fantastic. It's been great interacting with him. I also loved your episode with John Lynch and JP Dubet on privacy. Very informative. You've had some. Yeah. Fabulous guest. Question. You have to ask yourself those who recommended me. Was it a friend or a foe?
Hopefully, it was a friend and someone who didn't want to take your ratings.
Well, I didn't I didn't I didn't

interrogate. I imagine just someone who has deep respect for your work. And then you recruited Ross to to come along as as more of like the practitioners voice. So, I'm very very happy to have you both. Maybe just to kick off the episode, you could both introduce yourselves to the audience. And maybe Carl, we'll start with you.
Yeah, sure. I have my PhD marketing from Colombia. I've been a Duke professor since 1999. I served as the executive director of MSI and I'm on the board of the Advertising Research Foundation. My early work was on measuring brand equity, which we're discussing today. And then I've shifted more into e-commerce and digital advertising, and those two streams are coming together in commerce media. I know you've covered that in your podcast, so that's another area of interest. I should also add that I'm from Boston, but I met my wife in Seattle. So, I'll let you figure out who I'm rooting for this weekend.
Uh, well, yeah, we'll keep that a secret. And then, Ross, how about yourself?
Yeah, sure. So, I'm the CEO of a company called Marketing Attribution. We're a measurement firm. We've been around for about seven years. I started out I got

my BS and O from Cornell, NBA University of Chicago. Worked at Booze Allen for a few years in their system strategy department. Worked at PNG in their media optimization group. And then had a company called Marketing Analytics. I ran that for 20 years. one of the first marketing mix modeling companies. sold that to Neielson in 2011 and then ran Neielson's marketing mix business globally and their emerging MTA multi-touch attribution business before I started marketing attribution in 2017 I guess around doing eight years now I guess almost nine may
maybe just kind of before we get into the questions what do you feel about the kind of renaissance of media mix modeling because it you know it it's become I you know and maybe tell me if I'm wrong about this observation too but it seems like it's it's become more relevant again especially with like a lot of the digital first advertisers who historically hadn't used it what do you make of that of the the kind of renewed interest.
Well, I sold my first company uh marketing analytics to Neil in 2011 mainly because I thought marketing mix modeling was going to die. I thought the way we do measurement was with multutionusion at the person level. It's definitely the way I would prefer to do it at the person in the household level.

But in the middle of some tests we were doing at Neielson. It was called digital media consortium. My I recruited a bunch of companies Coke, PNG, Pepsi, you know, lots of companies, Clorox and then you know other companies as well. We had we had Krux was part of it before Salesforce bought them. But the idea was to test marketing mix modeling multi-touch attribution exper some match panel test and try and find out which kind of things worked best and in the middle of the test Google said cuz the Google was part of it and they were giving us very useful information including person level impressions and said oh guess what we can't share that anymore because we're very concerned about data leakage well it turns out that data leakage really means revenue leakage because if you're giving out your person level impressions your Google or your Facebook you're giving out those you're letting you know companies like us get those person level impressions you are releasing your targeting list. That's your targeting list that you just gave out. I gave those impressions those people because they I studied them for a while. I know they are in the market for a luxury automobile or something. So that is what they're really concerned about. So I So that's when they stopped giving out person level impressions and that was kind of the beginning. It also was

around the 2016 election, Cambridge Analytica and all that stuff. But it really was and what what the thing I took away from it what it was that this isn't really privacy. This is self-interest from the platforms that then kind of merged into privacy. And so that's why I don't believe we're going to get person level impressions again from Google that we can then merge with Facebook. And so that's why I really believe that was kind of the beginning the end and we all are coming back to market modeling which I love and it's my second best thing after person level stuff but it does work and I think that that emergence is going to resurgence is going to kind of stay.
Yeah. Fascinating. All right. So we we kind of workshopped these questions. I'm I'm really happy with the way the questions came together. So So what just to kind of give the audience a sense of how we'll conduct this. So So usually when I have two people on, I'll try to have like I'll identify a main respondent for the question. Obvious of course the other guest is more than free to to kind of also chime in, but I'll have kind of the person I've identified as the as the primary respondent and then you know we'll get thoughts from

the other person too. So the first question we'll start with Carl and it it is what do most companies get wrong about brand measurement both from an analytical but also a conceptual standpoint.
It's a great question. I have several points to make on this dimension. Probably the first one that should be no surprise. Brands like Coke and Apple, they weren't built overnight. It took years and years to build these brands. Why it does it take so long to build a brand? It's not just changing somebody's belief about the product attributes, but you actually have to change their preferences. Changing preferences is hard. It takes a lot of work. It takes time. It takes repeated exposure. Takes giving people compelling reasons for purchase. You know, a discount isn't a compelling reason for purchase as an excuse. And it can't be measured with a small amount of data or an experiment. Like is it's something that takes a very long period of time. I think the second point I want to make on this domain is all too often I see brand strategy follow from what can be measured instead of basing their measurement around their

strategy and this leads to some bad strategies. So you know the canonical example of this in the 1980s was the release of store level scanner data. Prior to that there sales were measured through bi-weekly inventory counts in the back room. So they literally had someone go in the back room and figure out how many cans of beans sold. In fact, if you ever heard the term bean counter, that's literally where the term came from. And so if there was a discount for a given week, it would be sort of aggregated away over several weeks. And then scanner online or excuse me, digital scan data became immediately available. So companies could easily see the sales lift from a discount. And so they started discounting a lot. And advertising is advertising effects are hard to measure. We know that advertising elastes are low, but you know advertising is intended in many instances to build brands and that's not something that happens on a weekly cadence. And so what happened was companies shifted budgets from

advertising to discounting. And so that's when I started thinking about tools to measure long-term effects because it's pretty clear that that brands were shooting themselves in the foot. But again, these things are hard to measure. more recent incarnation of this phenomena is the advances in measuring direct response advertising. It's very easy to do AB tests. So ads are measured in terms of direct response. In fact, I think one of your past speakers was talking about one of their main KPIs as being clicks, but clicks and even purchase intent is not brand associations. It's not brand beliefs. I mean, I could drive clicks just by advertising a discount. So basically what's happening is because people can measure direct response so readily and so easily they've shifted ad budgets to direct response and I think that is that's an issue and that's again why I've worked on some of the long-term measurement tools that Ross and I will talk about shortly. A third point I think advertisers are are very tied up for similar reason in targeting which I

think is wonderful but it underestimates the value of reach. So let me give you a very specific example. Imagine you have a carbonated beverage brand and you know you you advertising that carbonated beverage brand. Sort of the best person to reach is the person who's trying a carbonated beverage for the first time on that day, right? You want them to be activated around your brand when they make that purchase because the customer lifetime value becomes enormous. But you don't know who's just about to start purchase for the first time. I mean, there may be some signals by like age, but not everybody's young when they first drink a carbonated beverage. And so, you know, some of the exacts I talked to in that industry say, "Hey, no, a really sensible strategy is to advertise to everybody." And I know that if over lifetime, you know, if someone's roughly 300 days a year, what times 30 years, looking at 9,000 days, 10,000 days, I know that on 9,999 days, my ads are wasted and only on one day it's

relevant. But it more than offsets the loss. So I think that again this notion of reach versus direct response something that's a concern and one last point in this area and that's the difference between marketing mixed models and media mixed models I think you know just I want to clarify this before we get into it media is only a a component of brand strategy and you know I have a paper I think it's a journal marketing research in 2010 it looked across 25 consumer package good categories about 709 brands over five years and we found that this short-term and long-term sales elasticity is 1.37 for product and 74 for distribution. In contrast, it was 103 for advertising and 00 for discounting. Now, ironically, like if you if you walk the halls of many research companies, a lot of money is being spent on ad and price measurement, but not nearly as much as being spent on things like product and distribution in terms of developing brands. And again, I think that relates

in part to to this this measurement. So, so how does some companies solve this? They have one phase where they use the marketing mix measurement. They use this to figure out the optimal media allocation and then they use a media mix measurement model. So, summing up, brands aren't built overnight. Strategy should follow from what can be measured instead of the other way around. Reach is underappreciated and underweighted in a lot of strategies. And we need to think about marketing mix modeling instead of just media mix modeling. Ross, I don't know if you have anything to add on that one.
Uh, well, no, I I agree with all that. We we I mean, you can't I can't I I think most most people can't do it actually in a media mixminded uh model that only has media in it. So, I think we always even if we're only focusing on the media, I believe everybody will put something in there about price and promotion or maybe they'll just take Neielson's estimates as as you know, I mean the uh no v syndicated data vendors estimates as as gospel. But yeah, you do have to control for those other things. But yeah, we do both types of miles and marketing mix miles preferred to us

definitely.
Carl, so you talked about, you know, this kind of shift to direct response just as a function of like, well, that's more measurable. And you kind of made a point earlier in your answer about the strategy is essentially as a mistaken approach. The strategy was downstream of of what was measurable, right? So you had companies building strategies just on the basis of like what they could measure and then that that would kind of that that would kind of result in this general shift towards like well we're going to spend on direct response because we can measure that and so the measurability of that medium becomes a a constraint on our media spend. Is that kind of are those the same thing? Is that is that a phenomen is that a phenomenon of that of that mistake?
I think so. And let me give you another example again related to a recent episode of yours. you know fundamental question a lot of companies are asking themselves AI make ads as well as people right and so you've had some groundbreaking seminal stud soon to be seinal studies on that on that very point where the answer is being sort of as long as people don't know it's

created by AI yes AI generally does better than people but that you know again the measures are things like clickability but that's not brand attitudes and brand associations so the jury's out like part of building a brand identity is relentless consistency about what you believe your brand to be. Like AI is not tuned to that issue. It's not clear that it can generate executions in a manner over a long period of time that reinforces brand messaging. And so like the jury's out as to whether or not I could use AI over the long term to augment branding. Maybe I can, maybe I can't. But the point is like we don't know because we've been focusing on short-term measures.
Right. Well, that's that's the kind of fundamental attention. Did you see that? There was that article that was circulating a few months ago, but it was a Mendelz. I might be mispronouncing the name, but they they they have they make all these different confections. So, they have like chocolate. It's a That's right. Manderlay. And I'm forgetting all the relevant details of this article, but but they had worked with like Deote

or something to build like a model that could supplement their creative output, but it was trained on their own internal data, right? like instead of surrendering this functionality to Facebook or whatever to to then you know we've seen some kind of scandalous results of that there was a case where Facebook was circulating an ad they were fighting an ad that they had built on behalf of a brand and they were using like an old woman as a subject of the creative when the the brand is a men's t-shirt brand so but anyway Manderlay had built their own model for generating creative which was trained on their own data so it should be sort of more attuned to their brand sensibilities do you think that's a possible solution there I mean Not everyone can do that. But if you can, is that a possible solution?
I think it's a step in the right direction for sure. Right. We we need to think about brand guard rails when we're developing an AI strategy towards creative.
Yeah, absolutely.
Got it. So So Raj, well kind of next question is for you. What are the most common internal political issues that you see arise with marketing measurement within organizations? Who are the less obvious stakeholders for marketing

measurement? And how do needs sometimes get misaligned with measurement?
Well, Ben, those questions are linked up with things. Carl does too quite a bit. But so I think the the biggest political issue in in measurement is short-term incentives of of brand managers. Their incentive system is set up to drive volume for the next one or two years. And the the owners of the company want the brand to be successful for, you know, decades. So that's that's a misalignment that causes them to, you know, focus on volume for the next couple years and not necessarily be building long-term brand loyalty. That's why I've worked with Carl before on doing long-term effects models is to try and try and change that. So that's the biggest one. That's one that's been around for a long time. Another one that we have is that you know we work I will say that there's less political conflict in what we do now than years ago though. I mean there there were there were times when we first started doing this. I mean I've been doing this since like 91. So at the beginning there were many people that just questioned the whole value of quantitative marketing measurement. So that was the problem. There would be marketing managers who just don't

believe it. It's a soft science. You can't really measure this stuff truthfully. When I was in business school, I that was I believed. You can't measure you're not going to be able to measure advertising, maybe promotion, but now you can do it. So that that one's kind of gone. But we do have some. So the agencies, the ad agencies, they also they have an incentive mainly to drive the top line because, you know, same with the brand managers, they're not really responsible for all the cost, for the cost of goods sold. They're they can't control all that stuff. So you can't necessarily compensate them on profit. So they are mostly incentive to drive the top line. and measurement firms if you you work through it if there somebody tells you I want to optimize profit it's very clear how to do that in some sort of quantitative optimization you know you you consider all the diminish returns you have all the response curves and you tell them to keep pushing you know spending on advertising until the next dollar doesn't drive enough revenue subtracted cost of goods sold to justify the cost of the media it just it's it's a very clear where how to optimize profit for us as a measurement firm how to optimize revenue I'll tell you how to optimize

revenue that's really it isn't real literally what people really want to do, but they'll say that I've just I'm focused on revenue. You focus on revenue, you spend a bazillion dollars all you got and then borrow more and spend that too. So what you really need is you need some sort of waiting between the two and then judgment comes in. So as a measurement firm, it's much more clear how to f optimize profit. We tend to focus on that and so therefore the agencies generally want the client to spend more money than we tell them to. So that's that's a conflict that we run in on that one. Another one is kind of near and dear to my heart is like who does the measurement? Okay, we're a measurement firm. And then everybody's got all the different people that do measurement will have their own incentives and and and biases and so on. You know, we measurement firms, their typical biases, not me, we don't do this. We we focus on straight shooting, just tell people the truth. That's how you stick around and keep doing this for 30 years. But there are a lot of firms that if you tell your incentive system is to tell the client the highest ROI, to re ring out the highest ROI you can get from that data. Give that to the client because the people that are paying you generally that's what they want to show. It's either brand

management they want to show their management or it's the agency or it's the platform. Pretty much almost everybody that's our stakeholder except maybe the CFO wants us to show a unre sometimes somewhat arguably unreasonably high ROI. So that's the measurement firms platforms. They also want to show a high ROI and they'll do that and using a couple methods that tend to show high ROI. I don't think they're all you know being evil about it. They're using methods that some people tell them is good but some of these methods do tend to give you higher ROI. One is match panels. If you just if you just look at a very simple here's people that are exposed, here's people that were not exposed. A reasonable person would think, well, that's a good way to do it. Problem is with all the sophisticated targeting mechanisms that there are today, people that were exposed were not exposed for some random reason. They were exposed, but they're likely to buy. So then you compare people exposed, people are not exposed. You know, oh wow, here I looked at a whole bunch of people that saw a loan ad recently because I mean they watched a loan television program or something and because of that I hit them with an ad. Well, geez, that's why, you know, that's why they took out more loans. They were

already more likely. That is a big problem. And that is in most platforms measurement systems. It's in there. It's not in some of the best ones like say Google's for example. I think they do a very good job with something called ghost ads. It's a complicated way. It's it's in an auction. Again, in their defense, I don't think everybody's evil out there, but auction trying to do an randomized control trial in an auction system is quite complicated. I think Carl knows about that. Really is. And the it's like when do you substitute in the control product? Before the auction or after the auction? You have to do it after the auction. And so Google does that. Most people do it before because it's so complicated. So that's and then and then so it's it's between the match panels, the sort of biased auctions. Measurement platforms tend to give biased estimates. So that's another thing about who's doing the measurement and what is their incentive system. And then you asked about less obvious stakeholders. We have a lot of stakeholders. Like at the beginning, you know, I've been doing this for a long time. So things have changed. beginning it was not everybody the CFO didn't really care when we first doing this stuff but now everybody cares and pretty much everybody supports quantitative marketing measurement and they support

marketing mix modeling too for whatever reason either because they they understand this thing I talked about about the about giving away your data is your is giving away your targeting list or for whatever reason they don't trust the platform so marketing mism is very accepted so our stakeholders includes key one is the insights of the analytics team them the brand management going up to the CMO, media buyers inside the company, finance and the CFO and we we do work with the C seuite CFO CMO. I know the CEO at some of our biggest like10 billion dollar plus companies are aware of the stuff that we do trade promotion management because we do marketing mix modeling. I've been doing that for a long time. We try to we try to model everything price and promotion and have it all be reportable. So therefore we work with trade promotion people with some of our clients revenue management on pricing. We do pricing models along with it and the ad ad agency the platforms and the platforms themselves also they pay us directly. We do a fair amount of studies for all the major not all of but like you know maybe five six of the biggest platforms to to

do best practice studies for them for the benefit of their client using like you know credit money or something like that but then they have their there's another stakeholder and finally our favorite stakeholder procurement just love them but with this many stakeholders if there's any less obvious ones I don't want to know about them basically what I'd say about that and I think you had one more question which was about alignment about misalignment or something like that yeah you can get misaligned when you're doing when you're measurement firm like us marketing mix modeling when you're working you're going from one client then you go another one another brand it's kind of it's kind of the same thing okay guess what today it's the same thing as yesterday we get all your marketing we put in the best model we can do with the most granular data we can do and we report on the ROI of it and recommend what you do next year so it's kind of the same thing every time but it is important still at the beginning to ask the client what is their needs what do they care about oh we're moving into a new sales channel oh we have a new product introduction oh we're concerned about this competitor things like that you do want to know what what is your kind besides the normal stuff what is it and then as long as you do that then generally your measurement is aligned with the needs

did you have anything to add
just three some reinforcements first I want to give a shout out to Garrett Johnson because Ross mentioned ghost ads I know Garrett's been a guest on your show and I think that's a really nice
thing that every marketer should be familiar with that's doing digital advertising I want to underscore his point about the challenges of an independent analyst giving an honest answer to firms who have invested interest in spend because your the number of employees that you have and your power in the company is correlated with your annual budget or spend. And so if your budget gets cut, your power gets cut. So there's a tremendous incentive for a lot of these companies not to want to listen. In fact, I co-chered the American Marketing Association School of Market Research for a couple of years and we always did a top 10 list where we surveyed the attendees on what they saw as the biggest problem in market research. Number one, and not by a small amount, every year was their managers telling

the design studies to show that whatever initiative they had worked. And that's a real danger. Finally, I also want to echo this point about brand managers. Unless you have a long-term measurement system, which we're going to get into shortly, in place, I tell my students who are brand banishers, I say, you know, the best thing you can do is ruin your brand. Just cut ads, discount the hell out of it, move a ton of cases. Yeah. So, what if people now think it's a cheap brand you've harvested, leave that mess to the next person because if you go in there and you invest in the brand and this brand equity and do a lot of brand building, that's a cost. It takes time to build brands. So, you get the cost, but you don't immediately get the benefit or you're competing against the other brand managers. Look, that is a really big problem. Um, the one year tenure brand managers unless you have a system in place to to deal with this. And again, we'll give some examples later on how to manage this. Yeah, I feel like marketing I mean especially

when you when you go to big firms like I mean imagine the kind of firms that Ross works with like the marketing dynamic is probably so rife for issues of principal agent problems because it's like to your point Carl it's cool to spend a lot of money. It's really fun to be invited to Super Bowl, you know, suites and get Taylor Swift concerts and to have your TV ad being talked about by your family and friends. It's really cool. It's a really nice thing. And so, you know, whereas, you know, shareholders maybe would have a different opinion if they were more attuned to the the kind of underlying performance, right? And so like there is I guess there's a little bit of an alignment just naturally inherently in that. But like then it's a question of like who wants the bad news, right? Because if the measurement's going to give you bad news, the external, you know, unbiased measurement company that you're working with is going to give you bad news and the

marketing team is the one paying them, then you might run into misalignment just naturally, right? So that I've seen a lot of cases where the CFO steps in and says, "I don't trust that. I'm going to be the one interfacing with the measurement company. I'm going to pick the measurement company. I'm going to audit their performance and I'm going to be the one that decides, you know, whether you're spending the appropriate amount of money. But I just one one quick question I wanted to follow up, Ross. So, you talked about agencies and kind of managers kind of having the same like a little bit of the same sort of bias to just driving that topline revenue because with with agencies, I think it's it's pretty easy to understand because that's how they get paid, right? And then with marketers, it's it's more just that what I just talked about is cool to spend a lot of money. Could you maybe just expand on that the idea with the agencies a little bit more because I think that's something that seems obvious, but it it kind of sneaks in in more nuanced ways than maybe some people understand. Well, I mean the way the way that's coming to to to to my world even in the last few weeks with two different platforms actually maybe the last couple months with two different platforms is that we

did a best practice studies for them and then identified you know different types of ads that have the highest ROI or you know yeah highest ROI there's different measures of ROI whether or not you include margin in or cost of goods sold or not depends sometimes that's confidential but you know we focus on ROI and then the well this was this was actually the platforms uh but but the agency does the same thing which is that they want to drive revenue and therefore they even will question whether they should be focusing on ROI or not. I've seen that they say well why why don't you tell us which ads were most effective and we'll do those effect and there's there's it's kind of getting in the weeds here but effectiveness in our world is generally defined as the incremental revenue per impression or thousand impressions regardless of how much those impressions cost. Then you add in the cost of the impressions and then it's incremental revenue divided by media spend. And then then you might you might might multiply the revenue by the margin to get the actual incremental profit. But but the agencies and the platforms are saying are asking us why don't you focus on effectiveness and forget about you know not ROI because you're going to stop you're going to stop spending way too soon if you focus

on ROI but you don't really maximize ROI. We're doing you know we do these market mix models then we put them in an optimization like a nonlinear optimization big ones that we work with professor at Stanford Mall on a an S-curve optimization algorithm spe specialized for S-curves. What what I'm saying is that you don't actually maximize ROI after you're done with your marketing model. You maximize profit and so or revenue. So anyway, I guess what I'm saying to you would ask them to get a little bit more into the details by the agency. It ends up, you know, having to convince them that you even do need to look at costs. You know, you can't just maximize effects. Forget about the costs. Somebody gives you a billion dollars to spend. I mean, our clients do spend some do spend a billion dollars on advertising globally. You have a billion dollars to spend. You don't just you can't forget about cost. What you have to do is you have to maximize the inc. If all you're focused on revenue, fine, focus on incremental revenue. Forget about margin. You still have to get the maximum incremental revenue per dollar so you get the most revenue for your billion dollars. But anyway, that's it's it's it gets in the weeds about that kind of stuff with the agency.

Yeah. CMOS love ROI because they can walk to the CFO and say, "Look, these numbers, they're eyepopping." Even if the marginal profit is negligible. Yeah.
Let's get real. Half the time you're not sure if your campaigns work. You just kind of hope they do. Attribution numbers look good, but what if those conversions would have happened anyway? Incremental helps you find out for real. No tracking people, no guessing games, just clear answers on which campaigns actually move the needle. If you want to stop guessing and start knowing, go to Incremental. Because good marketing isn't about getting clicks, it's about getting results. Check them out at incremental.com. That's ign.com or click the link in the show description. Carl, so you wrote kind of a canonical piece. If brands are built over years, why are they managed over quarters? And in that piece, you argue that a disproportionate focus on short-term data supports a constant cycle of price discounts that may end up eroding brand

health and pricing power by increasing consumers price elasticity, which could be economically destructive in the long term. You propose two measurements of brand performance that firms should incorporate into their strategy, quantity and price premium. Can you explain what those are and how would you suggest that firms adapt these metrics in their broader measurement portfolio? So let me start with the last question because it's easiest. How should firms adapt these metrics? To be blunt, call Ross or someone like Ross. The reality is the concept's actually very easy. But you know, when you're talking to someone with 20 years of experience, they know the benchmarks. They know the errors. They know the norms. There's a lot of trial and error over the years. And so much money is spent on advertising. You really need to be with firm that does things correctly. And to Ross's earlier point that you could trust, they'll tell you if something isn't working. In terms of the example and and what this is all about, let me give a a very specific anecdote that got me thinking about this

because when I was in MBA, I remember learning about, you know, how do I evaluate the profitability of a discount? So, I was taught you just look at the lift. If the lift is really big to outweigh sort of the loss in margin, then the promotion looks great. Go ahead and do the promotion. But let's think this through. See, I sell a lot more at smaller margins. So overall, my profits are up. But let's suppose I am discounting. What happens if I do it next month? Next month, consumers have learned to lie and wait for a discount. So the baseline sales where the sales and things are off promotion are very low and the incremental response is even bigger. So if I go through and I do the things I was taught when I was UCLA MBA on evaluating the profitability of promotion, the incremental lift is even higher. And so the promotion looks more profitable. Next thing you know, you're in a death spiral. And so what you really need to do is take a look at these baseline sales and these promotional lifts. And the

promotional lifts are basically price elasticities. Higher price elastity, higher promotional lift. So the idea is let's go run a marketing mix model and in the marketing mix model gives you two very very useful pieces of information. It gives me the baseline sales. The baseline sales is typically something like an intercept in the model. It tells me if the marketing all the marketing tactics and the price are the same which brand sells more. The idea of brand selling more is a measure of brand power. And then I could look at elasticities again how sales change with price. And the lower elasticity the higher you can raise your price and therefore the higher margins you have. you think of luxury goods. So if a brand does really well hopefully you see high baselines and low price elasticity and you can track these over time. So in one paper I wrote I remember doing these by plots where we looked at the path of brands over five six years where we just plot these two things and we saw real movements and interestingly enough as we'll talk shortly those movements can be tracked

back to your marketing strategies. So one of the things again I found with people in my research and I was very relieved when others found the same thing. Ross has more cases than I do. Not surprisingly congruent with the example I gave you a moment ago is that a lot of discounting tends to show up in subsequent periods as greater lift and lower baselines. Now advertising tends to lead to higher baselines. So linking this back to the product manager example. If you know, one of my students took my advice and trashed their brand, what would you see by the time that person's review is being written up, you would see baseline sales go down and price last go up. And so, even though they've sold a lot of cases, the managers are going to say, "Well, you trashed my brand." And this has created a very powerful incentive to get companies back on track to managing brand health over the long term. Ross, do you have anything you want to add on this? because I know you've done so many more of these studies than I have.

Well, I mean, this is a classic paper. I mean, one of our clients was in it, Steve Gary from Clorox. Um, no, it's a classic paper. I mean, Carl's the king of long-term effects, honestly. And so, no, we we I think I don't think it was this paper that the algorithm is that about the modeling the base volume and the price. I don't think it came out of there. I think it was a subsequent paper, but no, that is classic. And then we we built our algorithm on on another paper of Carlos later on.
Well, so what I loved about that, so in the paper you talk about you give it the example of Lost, right? And so like it's it's interesting because I've always known Lacass as like that kind of premium tennis brand. I I didn't know that that they were it was acquired. So the story of it is it it was this this French brand that was is known as kind of like I guess a luxury kind of you know tennis apparel brand and it got acquired and the acquirer just cut the price and tried to you know just increase the distribution and then well as a result you know it became worth a lot less because they just the okay they were selling more but at far lower prices and then the original owners bought it back and kind of returned it to the sort of luxury status it had. But

then you also talked about Nike and and so with Nike with the Foot Locker distribution, Nike had said, "Hey, look, here's the these are the terms. If you want to sell our shoes, you you have to position them as like kind of high quality goods." And and so the paper was from 2007, but Nike had the more recent experience I guess in the last like two years of they kind of shifted into DTOC and then they started relying more on just like you know direct response advertising and then pushing you know promotions to to make that more effective and that I think diluted the brand right made a lot more accessible and then and then you know they ran into trouble with that and I think they brought in a new CMO I think they brought in a new CEO if I remember correctly but but they had to kind of revert strategies there just maybe if you could just kind expand upon those. I just I just thought those two examples were really interesting, especially the Nike one with the more recent experience of having kind of taken the opposite approach as what they did as what you depicted in the paper.
So Nike's challenges have have evolved somewhat, but but let me just back up briefly and and and why did I bring up those specific examples? There's always

some conflict with channel in the case of Nike in the paper you know the channel wanted very liberal terms on credit from Nike and Nike didn't want to give liberal terms and credit you know so these conflicts are all about you know how do I allocate sort of the value I've created in the brands I've built so I create the value how do I allocate and to me this is all a function of who owns the customer So if the customer is loyal to the retailer or is the customer loyal to the brand and in fact one of the very reasons we see Shopify you gain so much market share in in ecom relative to Amazon is because Shopify sits on the back end. Amazon sits on the front. Amazon controls the customer experience. Doesn't give a company a chance to do any branding or brand building unlike if I use the backend tools at Shopify. So Nike was extremely successful in owning

the customer. Fast forward to now, it's a different world. The barriers to entry of DTC are pretty low. And so you have all sorts of fragmentation and ankle biting. So, you know, Nike is a really good brand. It has a great reputation, but maybe it isn't the single best brand for bike shoes, you know, and so the fact that I've got more DTC and ankle biters means that has to fight a little bit harder in thinking about how to own the customer and how that affects their advertising. I think that also the poster child recently for all of this is the acquisition of Craft Heights by private equity. Ross, I don't know if you've been part of the measurement in that space, but they went big on controlling costs and part of that was cutting advertising and you know, it's been it's been a really rough time for Craft Heights. So, you can't cost cut your way to excellence to be honest. You have to invest in build brands. I hope that answers the question, Eric. Longwinded.
No, no, no. That's great. And and I guess, you know, I I feel like private

equity would be especially bad at that at at at operating a a a brand centric company because, you know, the the whole goal of private equity is to buy an undervalued asset kind of get integrate some efficiencies and then flip it in a few years, right? Well, if you look at ad spend and you know, even if you've got these kind of provable results of like look, you know, when we support these brand initiatives, they drive real value. Yeah, fine. They drive real long-term value. We're not going to be the owners in the long term, right?
Yeah. By the way, that's one reason that analysts, some analysts, I don't know if you've worked with anyone else, some analysts have started adopting these measures too as they evaluate companies.
Yeah. I I I did work with some people at Vanderbilt that were helping out Wall Street firms evaluate mergers and acquisitions and using cross price elasticity. See, what happened if these companies came together? Would they be able to raise prices more? I think we've done that. Yeah.
Ross, I wanted to switch gears into measurement. What are the differences in approaches to marketing measurement that you see across verticals for successful

companies? So, how does effective marketing measurement differ for instance e-commerce retailers versus app developers versus large CPG brands or does it?
Okay. Okay. Well, we haven't worked with every single vertical but we have worked I mean between my first company marketing analytics and then when I was at Neielson and then now here at market attribution we have covered a lot a lot of verticals including gaming like you mentioned Eric uh we've done that. There's not many verticals that we haven't done. I I think some B2B things we haven't done probably because that that's a little bit that's kind of different. That's more like CRM which is not really our our specialty but I would say so how how they differ well first of all the KPI so we'll model sales in some verticals we'll model downloads for some loan applications sometime also we we may model different portions of the funnel of the purchase funnel. So we may so for a bank we might want you know model loan applica uh you know loan applications and then loans given out or something like that. There's different levels of the whole funnel we can measure the data differs e-commerce retail they generally have better data

and they have sales at the consumer level. If you if you have if you know who your consumers are, you can do a lot a lot better stuff with with modeling and because you know you know sales by consumer and you don't always not necessarily like e-commerce. Oh yeah, they would know everybody they sell to. They don't necessarily know a lot about them. They may sell through some sort of affiliate and and and then not know everything about the who it went to. But so the data differs and then the methodology differs. So if you do have consumer level information like randomized control trials are the gold standard in measurement of pretty much everything in medicine and in marketing and in all kinds of stuff they are they they are the gold standard and you can make sure you don't have any biases with them. But it's it's a lot easier to do if you do sell directly to consumer. So that's easier to do if you're e-commerce or you're a retailer. We've done some RCT random P trials that were I thought were super interesting. This was in CPG. We had a 10 million household panel basically that we could send ads to and see how they behaved. So 10 million households, something like 30 million people, you know, 100 million devices or

something. It was it was a a rather large experimental lab that we had. It was super awesome. We couldn't measure as many campaigns at the same time as we can do with marketing mix. We can only do like maybe six at a time, but it was so expensive cuz we had to pay an identity management firm. If you're if you're a retailer or you're e-commerce, you might not have to do that. So that's where practical in in those verticals. B2B, like I say, generally doesn't have enough volume. You can't just sell like one or two giant, you know, industrial machines a year and do some kind of model on that. That's kind of hard to do. You need some kind of volume. So B2B is kind of And then long-term effects. You know, this podcast is kind of focused on long-term effects. That also your ability to do that differs by vertical quite a bit. And well even even just any kind of measurement for let's say for automobiles that has to you can't automobiles have such a large long purchase cycle that it's really difficult to model sales with a marketing mix model for cars. So we generally model something like an attitude like purchase intent or some some kind of thing like that. There are some products that have a very long

purchase cycle like mattresses that we are able to model. I I don't know if it's because they tend to have really strong holidays. So like you know there's going to be a few periods where the people are going to buy a bunch of mattresses there President's Day like right now and Labor Day those are big seasons for that. So you you can you can measure you can see the effect of your advertising in some sort of short period of time. So I'd say that that that those are differences across across there. And then as far as like you know really successful you know some of the successful marketers that we work with I think an e-commerce company that we're working with is like really I think they're one of our more successful clients. And I would say here's some things about them that I think help well to quantify what they what how good they're doing. They went from about 250 million sales to about 550 million over the time that we've been working with them. More than double sales. Their CMO is a super awesome guy. Cracked us all up and he is always pushing us to to push the measurement envelope forward. He had us doing our first share of search models that were popularized by a guy named Les Benet. I think he's in the from the UK. But share of search did the

interesting thing. It's it's a potential long-term effects model. Although I think it has some pros and cons for that. They also pushed us to do our first affiliate marketing. That's an important thing for e-commerce where you sell through affiliates. You give them like a rather potentially large commission like 20% 30%. Or 2% depending on what kind of affiliate it is. So but so modeling that like how much extra value would I have sold without that affiliate referring it to me? You know, I'd prefer not to give them 20%. And then and then working tightly with their media guy. We turn things around really fast for them. I just I just I just love the way these guys are pushing everything. So for them after each of their major holidays, we will get the model turned around in five weeks. So 5 weeks after the holiday ends and they you know the Sunday of the holiday or whatever, the Monday of the holiday, five weeks after that, we're done. We've collected all their data, working with their media guy, we run the model, we give them the results, put it in optimization, recommended what they do, and we're done and we're ready for the next holiday. So I think guys like that that that really push, you know, try new things, push the envelope always, you know, I think those guys get good results. It's good to be good at

measurement.
If you're swimming in dashboards, but still arguing about what actually drove installs, this is for you. Branch is an AI powered MMP built for growth marketers who care about signal quality and outcomes, not just reports. You can quickly answer questions like, "How is my Tik Tok spend really performing?" or "Which partners are driving net new users?" And even launch campaigns that move users from offline to app without breaking attribution. Branch's AI proactively surfaces what's working, flags issues early, and takes care of the busy work like link creation and tagging so you can move faster and spend smarter. Learn more at branch.io. That's branch.io. So, next question is is for you both. Maybe I'll let Carl start and then and then Ross, you can follow up, but what are the pros and cons of the various methods of measuring long-term marketing effects? So let me outline the the ways to do it and then the pros and cons. And I think Russ will Ros Russ will probably

take a dive into sort of some applications. But the three basic approaches are using surveys to measure brand attitudes, brand associations, salesbased measures like baseline drivers as we discussed earlier and then you know some people use stock returns, Tobin's Q specifically. Tobin's Q is a measure of market value relative to assets. The idea is the market value is higher than the assets. Some of that is related to brain strength. Surveys are easy to measure, but they're often a weak link to purchases. So, for example, purchase intent R squares on purchases are not high, and you have serious response bias because, you know, who has time to fill in surveys. I don't know about you, but I haven't done many surveys in a long time. Sales, again, this is the marketing mix type things that Ross is talking about. a easy it's data that are typically part of the you know the standard tech stack martekch stack of a company it's behavioral it's field-based and so it's a a really nice tool to get

a read on how a brand is is trending the downside of course is it's difficult to r link directly to profits which is sort of the key financial KPI I mean it can be done the stock price it does go into to firm value which is again what you need to raise capital and sort of It's the end game, but it's it's a bit of a challenge like spend to stock price for a couple reasons. First, stock price isn't reported at the brand level, right? So, I can't figure stuff going out on the brand level. The second thing, you know, is costs aren't many costs aren't under the marketer's control like cost of goods. And then, you know, what is the result you're looking for? Ironically, if you have a null effect, then you're at optimal spend because if you're spending too little, you should spend more to raise profits. If you're spending too much, you spend less to raise profit. So, if I'm looking at a direct link between spend and profits, the only way you know you're doing it properly is a null result. You know, of course, null results have all sorts of problems, not the least of which being you can't really test the null result because of

power issues. So, those are sort of three and the pros and cons, Ross.
Yeah. No, Carl outlined them. I think there were there were attempts at measuring long-term effects that go way back. I I I came across a paper just in the last few days when I was talking to my team about getting updates from them about what they thought about long-term effects models that worked or didn't work. They told me about one way back from 1978. It was called assessing the long-term value of advertising. I don't know if you've ever heard that one, Carl, from some guy Naram Dala from J Walter Thompson from 1978. They told me about this one. So, it's it's been going back. I I don't really think that one's super promising. So, the different methods we've tried brand awareness, we tried using that. It's like a survey method. It had a nice We worked with Millard Brown together on that. I'm now part of Canar. We got a base awareness that was served as a long-term effect and it was big and so that was good. That was good about it. But it is survey based. It's it's a relatively small sample. People also get confused with awareness data. People get confused about brands that they've seen. So you'll ask them, you

know, are you aware of this brand or something or are you aware of this brand's advertising? And they will say they they will say they've seen ads that weren't even running at that time. But and then what we find out it is it's another brand that's kind of similar to it that they got confused about. So things like that make it survey data as Carl said it's like there's biases in it um selection biases and other things and there's people just don't remember stuff exactly and so that's one con of that one. We've had other survey data. There's a thing called brand asset valuator that that's interesting data. It goes back a long time. It's got a lot of brands in it. We use that data. You know it's perception data. We thought it got decent results. senior management wasn't convinced that that a survey data is going to let us double all all our ROIs. I mean the general rule of thumb is that what you measure in a market mix model is like the effect on sales in the first year maybe and then you should double it to get the total effect something like that and and so the the 2x multiplier whatever we got out of that study the the senior management didn't take much action as my is my recollection we've also we tried other

behavioral metrics like sheriff search that was the one that was popularized by less banana his first stuff was on using durables I think it was phones and something else and it looked really cool and I I do think They we it's a it's a it's a cool model to use, but it's not clear to us that branded what we're talking about is branded organic search. You're searching specifically for some sort of keyword that has the brand's name in it. You would like to your advertising to to drive people to search for your brand, learn about it, stuff like that. So, that's why branded organic search is a good thing to drive. But anyway, I I it's not clear that that's a good indication of long-term brand loyalty or anything in things like CPG or soap or, you know, toothpaste or something like that. But I I believe it in in durables. Neielson used a statesbased model for a while that was based on new and repeat buyers. That sounds quite promising to me. If you can track somebody, like if you're a phone company, a telecom company, and you track and you've got somebody that's a customer for several years, I think that that that could be promising if you watch somebody over a few years. I haven't personally done that one with a

phone company, but there there's been several attempts at using that like tracking buyers over time, see how many are new, how many repeat, and trying to have a model try and predict why how do I get new customers that then stick around for a long time. That those are very promising. Another one was done by NCS. That's Neielson. That's an alliance Neielson had with a company called Cataly Le and Catalina Solutions that's now doesn't do this anymore, but a woman named Leslie Wood, she's done a lot of cool stuff. She did a household panel long-term effects model that had new and repeat buyers only. Also, the problem with that is in CPG and you have like a panel, there's a lot of churn in these panels. So, like after a year, so many of the panelists have turned over that she could really not measure long-term effect for longer than one year. And that's not really what we call long-term effects. We generally call them long-term effects after that. So, we've used a lot of them and I will say that none of them are perfect and it's a hard thing to measure the effect on your sales. like you know again these some of these things I thought at first when I heard them is like I don't think it can be done trying to measure your effect on your sales two years out or 3 years out

but you know you can get decent estimates and and the best one the best model that we've done and like I say none of them are perfect but the best one in terms of like getting intuitive results consistently that are accepted by the client and that the client comes back and buys from us again I like to repeat purchases as much as the next guy is Carl's two-part model that one that I guess he did outline the kind of basics of it in a HBR article but then later on he develops some methods to try and estimate it. So we do two parts of it. We estimate the impact of your advertising on base core sales like on a quarterly we do quarterly data. So we don't we wash out the spikes and we just want to know total volume especially core volume that you're driving and like you know in the long term. So we model that and then we model your base elasticity and see what happens. How does your media how do your promotions affect those things? The core volume. Do you keep having giant promotional spikes that drive down the volume in between the promotions that you're selling at full price? Is that what you're doing? And how are you affecting price elasticity? Nobody wants a high price elasticity. That helps you cut price. People want a low price elasticity so you can raise it. So raise prices. So

that's a really good model. We run that model for 16 years. When I first started working with Carl was a long time ago. I guess it was 2010. We I brought it into Neielen when Neielson acquired us. They're still using it to this day to my to my knowledge. No other long-term effects model that we've ever used. We've used a bunch of them. I listed some. We've used more than that too. I think none of them lasted with more than two years with us anyway. maybe with somebody else, but both us went away. So, this is the only long-term effects model that I've had a long-term relationship with. So, I thank Carl for that. And and I think I'll say that one one thing I'll say about Carl's model is not all verticals support the price elasticity part of it. Sometimes we'll be in a vertical. You just can't get a good price elicity. So, we just do the core base volume part of it and that's good enough. And that that is the key is that we're modeling sales. There a lot of long-term effects models. If you're modeling some kind of attitude or something else that that then management doesn't really believe that that's connected to sales, model sales, model base core sales, everybody can agree that that's a good thing to do. And I think that's part of why it works really well.
Yeah, I remember that. I remember Ross coming down to Duke 2010. Remember the

time we were spending on the whiteboard? Oh, you had some great ideas. Ross mentioned Les Benet. For those who are not familiar with him, you probably should look him up. He's an icon in the advertising industry. is a stalwart on protecting branding in the advertising mix and I know he's done a lot of work in the space of brand management baseline sales not just saying less banana is the way he talks when he did this thing about share of search it was during co and I remember watching just saying oh this is so relaxing and nice he's just a great speaker nice guy
yeah so Russ that was that was a great segue to the next question I know we're just at time but maybe I'll try to squeeze this last question in so Carl you know we're kind of talking in the context of like a lot of, you know, brands that are sort of established. They've got long histories, maybe, you know, something like Coca-Cola going back more than a century. What tools does a CMO have for like a nation brand? So, imagine like a brand, a DTOC brand that's just starting up, right? What what tools does does a CMO have with a nent brand to make the case for investing in this to investing in brand

advertising to to build these kind of like long-term brand effects? How would they how would they make the case to like the CEO or like their investors that this is something that should be invested into? So I had one thought on that and Ross had one thought on that. I'll show my thought on that and jump to Ross. But analoges are really important here. The reality is you can't make up data. Remember it takes five years or more of data to really truly understand long-term effect. You could think of this in terms of data variation. You know how many long-term cycles of spend can you really have over a very short period of quarters? You can't. So if this stuff's being measured at quarterly variation baselines, I mean to have any statistical power, you need enough quarters really to to link long-term strategy to quarterly baselines. But there are analogous brands and channels. So one of the really interesting discussions that you've probably fielded on your on your show or you may soon is with ads woven into generative AI like what's their performance going to be relative to search engine marketing,

right? And then you start thinking, well, you know, are these analogous? You know, and they should be in any way. Like the SEO tools are same. I mean, I still have to have my website be discoverable. All those tools we did to to get higher SEO and better SEM tend to apply in this market. So like might be reasonable and say like as an initial guess, I can expect an analog. The same would be true for new social channels with existing social channels adjusted for demographic and those sorts of things. But I want to caution, you know, you you can't think of a brand new channel is brand, you can't prove a brand new channel is brand building until you have enough data to prove that it's brand building. You might have good faith, a good reason. Analogs might be part of that. And Ross, like I think you have some experience again with Lotus' old measures and rules around this. Yeah, I mean that that that the original 2x thing came from I I I meant to mention this during an earlier web discussion, but there was that 1978

paper that I only heard about until recently, but there's a classic paper that was done in 95 by Len Lodish, Jee Abraham, Bruce Richardson, a couple other people from IRI and Warden. Um it was a summary of 55 inmarket experimental estimates of long-term effective TV advertising. That was experiments like using behavior scan markets. Those were IRI now called Sakcon but way back when they had small cities Peoria or something little tiny cities that were kind of isolated from other media markets and then they would you know run some ads in one city and then do something else I did different or within the same city I think that's split cable that's right they within the same city they had some people saw one ad some people saw another ad or didn't see an ad and they did that and then watched for like three two or two years out they watched for two years out I think so that that was a very interesting that's where they got the 2x multiply first thing I said the long the total effect from an ad is generally twice the the measure that you get in the short term. That rule of thumb, all kinds of long-term effects models, we all go back and make sure we're somewhere near two. Otherwise, we're probably doing it wrong. But yes, I agree with Carl that that that for a

brand new brand, you probably should just use the 2x thing. There have been some attempts to do like Neielson after we brought Carl's model to Neielson, some other guy that that that came in with my team to Neielson, he took that model after I left and then tried to make it so it could work. Because like with Carl's model, we generally say we have to be at the client for three years cuz what we get there and then we have two years of back data and then once you've been there for three years, we have five years of data that then we can use and Carl's model works pretty good with five years of data. And so that's what we'll say. But you know that's like kind of hard to sell, especially if it's a new client coming in. So I can understand why Neielson tried hard to make it try and work for a new brand. And what they did was they took ad stock and just kind of extended ads stock way out. That's been done by some adsock is for anybody probably most people listening to this probably know what ads is but adock is you take GRPs or impressions and then you kind of do exponential smoothing on it just kind of smooth them out into the future and then you use that as your variable and then through that you get an effect going out in the future. You try different decay rates. It's pretty much the same thing as exponential smoothing except it has

some kind of s different people different companies have different kinds of proprietary or maybe they admit it different kind of saturation estimates they layer on top of the on top of the smoothing thing. But trying to get a new brand and just doing ad stock to go out in the future, it doesn't work too well because it it makes all the you spend it out so far in the future, it just looks like a pancake and it's highly correlated with the intercept itself. It doesn't work good. The one one thing that we did do that was that for for a new brand, Carl and I did find this one out just a couple days ago was we well actually I knew about it but I didn't really think how that is pretty good. It was for in in pharma or actually we're doing you were doing pharma or OTC over in Europe and over there they have a mixture of the things that we buy as OTC over there. They might some is pharma and so is OTC but they have a a measure that we could they're getting over there from some market research firms called average weekly recommendations. So we advertise to the health care professional and then we get data on how how much they're recommending that product to their patients and so like I say even some OTC things like nicotine replacement therapy that is you go to a

doctor over there to get a recommendation and so so we'll model AWRS and then we'll model the effect of AWRS on sales. So we get kind of two kind of long things. Effect of marketing on AWRS that has a big lag lag in it. The effect of AWRS on sales that has another lag. We get out about a year maybe in terms of the effect but you know like that's about the cut off of where you might call it long term. So bottom line I agree with Charl with Carl that benchmarks are the probably the best thing. There's a couple of things that are decent.
Yeah Ross I like your example that's actually called the surrogate variable technique you know by a bunch of academics. The problem you're use your application is fantastic. Problem is the assumption in these models is that this intermediate measure the surrogate fully mediates the effect of advertising on future outcomes which just is insensible in most applications but not yours. There's actually really good application.
Carl Ross, this was a fantastic episode. I I learned a lot. I I imagine everyone listening did too. How can people follow you? How can they consume your content? How can they find you on the internet?

I'm at www.marketingattribution.com. marketing attribution.com.
Yeah, and thanks for having us. It's great to inject long-term back in this again. I love that it's top of people's minds. In terms of my case, I like to say that if you're going to have a kit, think about how to name them and that people can search them and they're the only real result that shows up. Like to my knowledge, there's maybe five Carl MAS in the entire world. So to find me is pretty easy in a in a Google search. just you know enter my name and it'll pop and brands are built over years why the management of quarters just do a simple search on that it'll pop up at HBR I think that is a really good starting place for a lot of people if they want to take a read of what's going on here so thanks again
yeah thank you Eric
no you're you're w just uh pointing out to people that that is a HBR payw wall gated article but the PDF can be found through a Google search as well so they can they can find it that way I really appreciate both your time thank you very and enjoy your weekend.
Thank you.
Thanks. Bye.