Podcast: Quantifying the advertising value of cookies (with Garrett Johnson and Shunto Kobayashi)

[music]
Let's get real. Half the time you're not sure if your campaigns work. You just kind of hope they do. Attribution numbers look good, but what if those conversions would have happened anyway? Incremental helps [music] you find out for real. No tracking people, no guessing games, just clear answers on which campaigns [music] actually move the needle. If you want to stop guessing and start knowing, go to Incremental. Because good marketing isn't about [music] getting clicks, it's about getting results. Check them out at incremental.com. That's i n c r m n [music] t a l dot com, or click the link in the show description.
The problem is that the distinction needs to be drawn [music] between the competence of the economists and the correctness of their analysis.
Welcome to the Mobile Dev Memo podcast. I'm your host, Eric Seufert. And I'm joined today by Garrett Johnson and Shunto Kobayashi. Shunto and Garrett,

welcome to the podcast.
Hey there, Harvard man.
Oh, well, hey.
[laughter]
Congratulations on your recent graduation of Harvard's Masters of Data Science.
Yeah, well, Masters of Applied Computation and then the Data Science is like the specialization, yeah.
Well, congratulations.
Congratulations.
Thank you. It's funny because I follow the SEAS School of Engineering and Applied Sciences on LinkedIn and they have these profiles of all of my classmates of, you know, it's like so-and-so is now, you know, someone's moving to Amazon. I saw someone's moving to Palantir. I just thought it would be funny if they had a profile of me like, "And Eric Seufert, who just recently graduated, will continue to write a blog and run a podcast."
[laughter]
Yeah. Well, yeah, the treatment effect may not be huge here, but you learned a lot.
Yeah. Yeah, yeah, right.
[laughter]
Yeah, the increase in wage, zero. Okay, it's really nice to have you both on. So, you both co-authored a paper which was published recently, but I guess the preprint is like a year old, is that right?

Something like that, yeah.
The paper's fantastic. I think I'll I'm going to link to it in the show notes. Everyone should read it. But the paper's titled Can Privacy Technologies Replace Cookies? And I want to dive into what you found and the methodology. But I think before we get to that, maybe you can give like a little bit of a refresher or just an education about the specific topic which was the Privacy Sandbox. So, keep Give me some background on the Privacy Sandbox, the actual experiment, and then explain like how you acquired this data. Like what was the providence of the data that you used in the paper? And and also, please introduce yourselves as you do so.
I'm Shunto Kobayashi. I'm an assistant professor of marketing at Boston University. I'm an economist by training. I have a PhD in economics from Caltech, and I work on research about economic and privacy issues in digital markets, especially digital advertising.
And much of what Shunto said applies to me, so I'm an associate professor at Questrom School of Business at Boston University. We're in the same department in marketing, and we share the same

interests in privacy in digital marketing. This paper is trying to study the impact of Privacy Sandbox. Many of us have tried to forget about Privacy Sandbox because Google ultimately retired it in October 2025. But the big idea was to use privacy-enhancing technologies to try to have your cake and eat it, too. So, to try to have the measurement and targeting benefits of cross-site or cross-app identity in online advertising while giving better privacy protection to users to reduce cross-site tracking. So, there's a bunch of different technologies that went into the Privacy Sandbox. Probably the most aggressive technology was Protected Audiences API. And basically, what it did is it kind of inverted the industry. So, the the criticism of the industry right now is that it takes people's personal cookie IDs, and it shares them with, you know, dozens or hundreds of different companies, which could allow these companies to track users across sites. And so, what Protected Audience did is that it put the entire industry on a

user's device and auctioned off a targeted ad on the user's device, while not allowing that information to leave the browser. So, it was a very interesting proposal, but as I said, Google retired this and and all the other technologies in October 2025. But, you know, privacy-enhancing technologies do still live on. The W3C is proposing an attribution standard that would use privacy-enhancing technologies to provide attribution for the web. And Google said it was going to support the this proposal when it retired the Privacy Sandbox. And so, this created a lot of stir because Google obviously is a very important market participant, and they said that they were going to re-architect the industry. And so, the UK competition authority got involved and wanted to try to understand the impact of these technologies before Google would roll them out to the entire industry. And so, they did something really unusual. They created an experiment

ahead of time that would allow the industry and the regulators to evaluate how Privacy Sandbox performed. And what it did is it took 2% of Chrome users and it randomized them into three groups. One group was a status quo group that had third-party cookies and Privacy Sandbox turned off. Then there was a Privacy Sandbox group that had third-party cookies turned off and Privacy Sandbox turned on. And then there was a cookieless group that had neither of these technologies. And so, like unlike most experiments that are sort of just an AB comparison, you had kind of two goal posts. You had the status quo and the burn everything down goal posts, and you had to see like how Privacy Sandbox compared to those two goal posts. So, we were able to study this. In this project, we were able to study the impact on publishers. And in order to do so, we were able to collaborate with Raptive, which is the largest ad management company. They gave us data for this experiment, and they represent over 5,000 publishers. They get a lot of traffic, like a couple hundred million visitors in a month. And

so, this gave us great data to study the impact of Privacy Sandbox and of removing cookies on publisher revenue.
And I'll I'll note that Paul Bannister, friend of the podcast, who's been on at least two times, maybe three. So, he's he's I want to say the chief revenue officer at Raptive. Anyway, just a just an aside. Okay, so I guess that, you know, leads to the next question pretty naturally. What what did you find? What are the headline findings in the research, and how material are cookies to publisher revenues?
Uh yeah, and just Raptive was an outstanding partner. Paul and Patrick over there are great. We're grateful for their help. So, yeah, what we found is that when you get rid of cookies, ad prices fall by 29%, but actually a bunch of impressions already don't have cookies. So, when you take that into account and just look at those impressions, ad prices fall by 35%. So, that's that's significant. And then, you know, the question is how well does Privacy Sandbox do in recovering those losses? And unfortunately, the the news from this research was pretty dismal. It was recovering only 4% of this lost revenue. But, it's really important to provide

the context here that the adoption of Privacy Sandbox tools by the industry was pretty modest. And so, that puts a ceiling on just how much revenue movement you're expecting to get. You know, it's good news that we saw some movement. You would see more movement, we expect, if more of the industry adopted this. So, you know, it's it's not providing the the answer to the question that we all want to know, which is exactly what the future would look like when Sandbox is, you know, perfected and the industry is adopted as much as they are, but at least starts to to set the conversation.
And by by the way, I just want to acknowledge before we go any further, we had a wonderful co-author, Jangrong Gu. She just graduated from BU with her PhD, and she's now at McKinsey. So, she's the third member of our team and and not here today.
I think one really interesting revelation from the paper that was the latency that the Privacy Sandbox introduced and that and its impact on the publisher economics. Can you talk me through that? So, like the Privacy Sandbox again, these these are tools that in a lot of ways mediated the data

flow. And so, like enabling these tools, as you you know, sort of surfaced in the paper, introduced latency, which has like its own sort of separate effect or its own effect on the economics. Can you Can you talk me through that?
Yeah, exactly. So, this gets at what's fascinating about Privacy Sandbox. So, as Garrett explained, so auctions traditionally run on servers, right? Like ad exchanges. And in that process, users' data can get exposed in forms of like bid requests and so on. And Privacy Sandbox tries to prevent that from happening by moving the ad auction process into your browser. So, now your browser, Google Chrome, is doing this auction process. And, you know, this is great because this is going to minimize the exposure of your data outside of your device. So, but the problem here is that your device is now doing the auction process and also running these bidding algorithms and so on. And that takes time. That takes time in addition to like all the the time that already takes

for, you know, rendering ads and so on. And this is why in our data set on the median impression, we find that ad delivery takes twice as long under the Sandbox condition. And that's roughly 2 seconds versus 1 second in the status quo condition that has third-party cookies, but not Privacy Sandbox. So, users may see the ad for shorter amount of time, but also there's a hard deadline here, right? The This deadline is that users may navigate away, click away from the webpage they're browsing even before this ad gets rendered. And that means that this additional latency may be leading to zero ad revenue, like like failed impressions for the these publishers. So, we had to take this into account when we compute the sandbox effect. You know, this ad zero ad revenue coming from people leaving before the ad gets rendered. But, this is actually methodologically tricky to capture

because we have data set on impressions. And by definition, these are not really impressions, right? So, we don't have rows corresponding to these failed impressions. So, this is something we had to statistically back out using our knowledge about the data and also this knowledge about the experiment. So, the way we did it is by comparing the ratio of sandbox impressions to the non-sandbox impressions we see in our data set and see how it matches or doesn't match with the ratio of treatment rates, treatment assignments rates we know from the experimental design. So, as Garrett mentioned, you know, there's like assignment treatment assignments rates, you know, it's like 1% was task quo and then 0.25% for cookie less, 0.75% for the sandbox. This this balance, this distribution gives us an expectation about how many impressions we should see or like what's the balance of impressions we should see across these

groups if there's no missing impressions. And when we did this comparison of the empirical ratio and the theoretical ratio, we did indeed find that there's a gap coming from ads lost to latency. And we estimate that the missing rate is about 3%. So, this means that 3% of the time under the sandbox condition impressions do not actually lead to ad revenue for the publisher. So, 3% of time these users navigate away even before the ad is rendered. So, this is something that's accounted for in our headline effect that gets shared. And this latency, you know, it's there, but we also found that it's worse for smartphones. Probably because smartphones typically have more limited computation power. So, having this ad auction process and bidding algorithms, it's natural to think that's, you know, like this

competition is leading to longer latency. So, we think this is one issue that PIIs may face in general if they rely on this kind of on-device competition.
It's pretty meaningful 3%. So, I mean, if you would tell someone if you adopt this technology, essentially 3% of your users is not not 3% see a decline, but 3% go to zero, that's pretty meaningful, you know, along with alongside the decrease in publisher revenue as a result of just the loss of fidelity. That's a pretty meaningful just loss, right?
Yeah, absolutely. I know Rap-ID was really frustrated with this phenomenon. And you know, I think this is like real problem. Like this additional ability from Protected Audience API of, you know, like targeting in privacy-preserving way, that's good. But, you know, 3% of impressions go like go missing. That's I think that's pretty bad.
Yeah, just for those following, the reference to the API in ad tech circles was called You may refer to it by its

full name, but but
[laughter]
I just thought it was funny to call it PAPI.
Big PAPI.
Yeah, big PAPI.
AI is changing everything, [music] but it's only as good as the signals behind it. Branch connects customer interactions across paid, organic, [music] offline, email, web, and app touchpoints and turns them into the trusted context you need with links and attribution that capture the full journey. Learn more at branch.io. [music] And while you're there, check out Branch's AI search and discovery report covering insights from more than 300 [music] enterprise marketing, growth, and digital leaders to understand how the industry is responding [music] to the rise of AI search. That's branch.io.
Okay, so the paper also finds that there were fairly significant differences in impacts between EU and non-EU users. What were those differences and what do you think explains those differences?
Yeah, so again, we're looking at these 50, sorry, 5,000 publishers who are

serving ads through Raptive, and these publishers get impressions from across the world, mainly in the United States, but we also see impressions from Europe, for example. And we find that when we decompose the effect into different regions, there's a sharp contrast in the cookie removal effect between the US and EU traffic. And what we find is that the damage from removing third-party cookies on revenue is about twice as large on EU traffic. So, for impressions that carry cookies, publishers lose roughly 66% of their revenue on their Europe traffic versus 33% on their US traffic. So, it's a big contrast, and not only that, we also find that if you look at share of cookie impressions, share of impressions that carry cookies, it is also lower in the European Union in the first place. It is 57% versus 81% for US users.

And the reason for both is GDPR. We think it's GDPR because of how GDPR has these consent rules requiring you to get users explicit consent for enabling third-party cookies and also potentially using these kind of like alternative IDs like hashed identifiers. So, you know, we have more opt-out users in Europe. And also, it's harder to deploy alternative IDs there. And consistent with that story, we find that on average, there are fewer alternative IDs available for European impressions. On the flip side, we find that when we talk about Privacy Sandbox, Sandbox actually recovers more revenue from European users. We find that the recovery share that Garrett gave, this is about 23% for the European Union impressions. And that is versus 2.4% for US users.

So, you know, it's almost like 10 times difference when it comes to the ability of Sandbox to recover the lost ad revenue coming from having no cookies. So, why is Privacy Sandbox performing so much better in Europe for European Union impressions rather than these US impressions? We think there's two reasons. First, because Sandbox and traditional ad auctions actually compete for the same impressions. So, when these publishers get these impressions, they launch auctions through traditional channels like header bidding. And also, they're launching an auction through PAPI. And that means that when the traditional side is weaker, when, you know, you have no third-party cookies or when you do not have alternative IDs, the PAPI channel is actually performing better or it's more pivotal in getting ad revenue. So, and again, in the EU, we have fewer alternative IDs on average.

So, this is why Sandbox seems to be mattering more in this auction process. And the second reason is, again, you know, in the EU, we have GDPR. So, we have a tighter privacy environment. So, we think that European firms may have had more interest in adopting Privacy Sandbox or PETs in general, precisely because they want to target users, consumers in Europe.
So, first of all, thank you for referring to it by its spiritual name, Poppy. But we also saw more vocal resistance to the Privacy Sandbox from European publishers, right? So, there's that the sort of German consortium of of publishers. I want to believe it was like led by Axel Springer that had filed a complaint with the EC about the Privacy Sandbox. And their argument was anchored to competition. But the point was, look, this is going to this is going to be really painful. And and I do think that was there were there was just more there was just more vocal opposition from European publishers. Is that correct? Am I

remembering that correctly?
Yes and no. So, I think my my recollection is that they were less opposed to Privacy Sandbox per se and more opposed to getting rid of third-party cookies because they understood that getting rid of third-party cookies would be very costly to them. And they were probably had more ambiguous feelings about Privacy Sandbox. And you know, they didn't know how well that would perform. But that was the complaint. But you know, some several browsers had already blocked third-party cookies, right? Like Safari and Firefox had already did this. I guess what changed is that Chrome has got such large market share that once they do it, then it's you know, it's even more damaging to these companies.
Yeah, how is So, I remember one thing that Paul mentioned that was really interesting on a on a podcast a while back. But but it was that the fact that Chrome didn't deprecate third-party cookies or block third-party cookies and Safari did prevented like kind of a clean comparison because there was some place for that revenue to flow to. Like it was like you couldn't look to Safari

as like the comparison baseline because the publishers could then just prioritize Chrome support. And maybe that would just absorb some of those users. Like users would abandon Safari and go to Chrome because it had it offered like, you know, it just better functionality as a result of getting more money and therefore it was a better experience. And so, it's not like you couldn't look at Safari as like the baseline because it actually lost a lot of usage and engagement to Chrome as a result of Chrome continuing to support it. But, if you lost it in Chrome, it would be worse than what you'd expect if you just looked at Safari. Is it Do you think it's a a reasonable I don't I don't remember the exact argument. I might be mis- misarticulating it, but like I I remember that was just being really interesting that there's Okay, we don't actually have the clean comparison.
Yeah, I think this this is a little bit contentious. So, I think, you know, part of what Paul's saying, I think the model behind that would be that like advertisers have a fixed budget and they're going to spend, you know, X amount of dollars on the open web no matter what.
Right.
And so, you know, if you have policies that hurt identification for a minority

of browsers, then the money will just switch to the larger browsers. But, I think, you know, some of the money is not fixed, right? Like, a lot of your listeners are performance advertisers. And so, if you can't measure performance, then you're going to spend less on that channel. And so, the money, you know, maybe some of that money would go to Chrome, but not all of it would go to Chrome. So, probably the truth is somewhere in between, but I I guess that's Either way, Chrome being the last kind of big browser standing here getting rid of third-party cookies would would, you know, tip the market pretty substantially, I would think.
Got it. Okay, so so the paper the experiment you you conduct takes place in a world where publishers are already using hashed emails and you probably see like identifiers. So, if those alternatives had been removed, what is your intuition and you know, we don't know this, but what do you think would happen with revenue? How how much larger would the revenue gap be if if those were sort of taken away?
I agree it'd be larger. And I appreciate you raising this cuz this is a really key point. One of the criticisms of research on the value of cookies is that

if you were to get rid of cookies, then the industry would be forced to adapt and to innovate. And what we think we're capturing is a really sizeable revenue effect after there's been a lot of this adaptation and innovation, right? Like Raptive's value proposition for publishers is to maximize the revenue that they get, and something like a third of browser traffic doesn't have cookies, and so they've been trying to increase the revenue that they're getting from these impressions. And they've been doing that by making heavy use of alternative IDs, which are demonstrated through internal experiments to increase revenue. And so again, like the effect that we're showing is is net of that, and if you get back to your question of like, what would the effect be without any of these things, we have some other peer-reviewed research where we find that ad prices fell 52% about a decade ago when users opted out of behavioral targeting via the ad choices program. Similarly, Google had an experiment in 2019 that found that ad prices fell 52% when it

deactivated cookies. And actually the UK regulator re-estimated that and said because Google only turned off cookies for itself and not for the other companies in the marketplace that the loss could be even as high as 70%. And all this is consistent with if you look at like the prices fetched by Safari browsers relative to Chrome browsers, you know, you see these large differences. So, this is really critical from a policy perspective because if the concern is not cookies per se, but all forms of tracking, then any policy that would maximize privacy and would therefore restrict all IDs and create this larger revenue gap. And I think it's really important to emphasize, if we go back to this issue of industry adaptation, you know, whenever the industry's been squished by getting rid of cookies or getting rid of mobile ad identifiers, it doesn't innovate towards contextual advertising, it innovates towards more identifiers, whether it be probabilistic identifiers or deterministic identifiers. So, I think this undermines the argument that the industry can just

you know, away from these things. And it shows that there's a really difficult policy trade-off, right? That if you maximize privacy, that's going to affect publisher revenue. And so, solutions like a privacy-enhancing technologies like privacy sandbox are an interesting third way. There's not been a lot of like take-up from that from regulators, but it's interesting that actually the UK's privacy authority just released in the last few months a policy proposal that would waive the consent requirement for certain advertising use cases like attribution if they're enabled through PETs. So, there's, you know, the door is opening up to trying to create this compromise way for the industry to continue to to to function while doing a better job on privacy.
Just uh contribute another, you know, relevant data points. Facebook said the same thing as 50% of the value the impression value from FAN, which is which is its third-party or is its third-party ad network is derived from

the its targeting abilities. So, so like if we turn that off, we lose 50% of the value, which, you know, accrues to the the publisher. That policy proposal is interesting, but I guess the question is like what I was curious about with those kind of like policy prescriptions. So, like just talking about the the the UK privacy authority saying, "Okay, if if this goes through a PET, we waive the consent requirement." Okay, but like what are the conditions on the PET? Like so, do they say it has to like cuz cuz then how do you how do you sort of quantify that? Can you? To say it does this or it has this sort of functionality that it offers or like because at the end of the day I did like an argument I made against Apple with this piece I wrote called Pyrrhic Privacy. They they seemed to be quantifying the privacy value of this by the reduction in revenue. Like it seemed like that's the way they viewed it. Like this is private cuz it reduced revenue by this amount. Like if you're going to institute that as a policy like as as a as a regulator what's the scope of the PET that satisfies the requirements? And and how do you how do you how do you scope it in a way that's like quantifiable and generalizable? Because what do you say? Is it it applies these

frameworks? Or is it that there's some sort of like reduction in fidelity in the in the identifier? Like I guess if you were like advising a a regulator, how would you advise them to implement this in a way that's workable without having to resort to like, "Well, it destroys 50% of publisher revenue, therefore it must be private."
So, privacy-enhancing technologies, some of them do come with some technical guarantees. So, the idea for attribution is, you know, right now we are able to Well, if you're just cookie tracking, then you can conceivably track the user across websites to understand what impressions they saw on which website before purchasing. And at least with Privacy Sandbox, the notion was to instead of sharing that individual data was to kind of share that through a server and then put back to the advertisers some noise data and some aggregate data. And so, this has formal mathematical privacy guarantees when you add differential privacy noise to these things, which should be satisfying to

regulators. The cost on the firm side is is an open question, and that is a big headache when you're trying to use machine learning algorithms to work with this noise data. But, that is the idea is to have these technical guarantees in terms of adding noise to data and that improving privacy. And then another approach, so so the ICO also talked about fraud detection. And there they were referencing the private tokens approach from Privacy Sandbox, where you basically just get a like a binary signal or maybe a two-category signal that says, you know, we think this user is a real user or we're not sure or they're not a fraudulent user. And so, I think that actually has a pretty intuitive privacy definition, right? There's no like detailed personal information. It's just you're one of three numbers. So, I think that's pretty satisfactory to people.
So, regarding the the DP approach, like so you could you I guess one option would be to say, "Okay, we're we're going to mandate an epsilon that you have to use." And and so like cuz that was one of the complaints about about Apple, right? They said, "Okay, we we we

apply DP everywhere." But then they turned out that their epsilon value was like 17 or something, some very large value, which then doesn't provide those guarantees. I guess would that be an option to say, "Okay, epsilon has got to be, you know, whatever, some very low value, and that's the mandate." Would that be an option or is that Is that not accounting for the the idiosyncrasies of these various use cases?
I mean, that's where the devil goes into the details, right? Is like this epsilon term just refers to how permissive the privacy enhancing technology or differential privacy is. And if it's very large, then you're not changing the data much. And so it's a difficult thing, and and I think it would have to be a bit of a dialogue between firms and the regulators. That's been a big controversy within these discussions is that somebody somehow has got to decide what these parameters are, and incentives go in different directions to set these parameters, and nobody wants to be the one to to to to pick the number.
Well, yeah, I mean, and there's also I mean, with DP specifically, I think it's very divisive, right? So you know, we know that Trump admin has banned DP this month

and then the use of it in the Census Bureau. So, you know, I guess there's you know, people have differing views on that. But yeah, I guess it's just one of those issues I think it's like it's very hard to prescribe it, right? As a That is a very, very thorny policy issue, I guess, is my point.
Yeah, but I think you have to sort of take a step back and think like these are problems, and they're very serious problems, but is it realistic that we're going to have like user-level data in marketing data sets from now until eternity, right? It just seems like eventually there's going to be some bad thing that happens or sufficient policy will to try to get rid of these sorts of data sets, and we want to still have some of these benefits of this data-driven way of doing advertising. So, I think we may in some point in the future have to accept these sorts of trade-offs and look at Safari and Firefox, right? They don't allow any of this data to pass right now. If they get

behind this attribution proposal at the W3C, then you're actually going to start to see some data coming from these browsers, which however imperfect is probably good for industry and for monetizing these users.
Well, I wish my friends at Mozilla, by way of the anonym acquisition, best of luck in applying their talents and their domain expertise to that to that problem. You mentioned the word trade-offs. So, the privacy-enhancing technologies, they often assume some economic trade-off between, you know, privacy and monetization. So, based on the findings in the paper, do you think browser vendors have underestimated the magnitude of that trade-off or is the current generation of PETs simply PETs simply not mature enough yet, right? So, like that even like that latency question, is that a question of like, well, okay, if they spend more time on it, they get the latency down as it's a first it's a first pass. Or is it So, so where where do we stand on the the idea that the browser companies they underestimated or like we're just in this like an early innings here and and and they'll get there?
Yes, I think there's of course like optimization and like adaptation problems, you know, still left for these

PETs, but I think Google definitely knew the trade-off because sandbox is by design, you know, it has to track worse than cookies, right? Like we have these like aggregation and differential privacy and and these things that lead to basically, you know, some kind of noise or like less granular detail in the the tracking information. So, of course, I think, you know, they knew the trade-off, but I think what they underestimated was the severity of the adoption problem and how it would undercut the overall effectiveness of privacy sandbox. So, as we explained, privacy sandbox, you know, it can change how market transactions, like how impressions traded and like how they recorded like in fundamental ways. And that that means that it requires two-sided adoption both from the demand side and supply side. So, if demand side platform shows up in this, you know, papi channel and if there's no supply of impressions coming

from publishers because they haven't adopted privacy sandbox, then they will see nothing. And indeed, our data shows that only a few adtech firms adopted it adopted privacy sandbox and this really held back the overall effectiveness of privacy sandbox. So, what what held back adoption here, right? I think what held back adoption was uncertainty and the cost of adopting privacy sandbox. So, regulators, you know, they have been very definitely ambiguous about, you know, which direction we're going. Like, are the pads the future? Like, which are pads going to be, you know, like enforced or not? And also kept moving the deprecation deadline. So, it was hard for adtech firms to commit to using sandbox. And also that's because of how it is costly to adopt it, you know, like we described, privacy sandbox is not just a drop-in replacement for third-party cookies. Like, you have to change how

your algorithms run and also you may even need to change your infrastructure so that it is optimized for privacy sandbox. So, I think because of this cost of adoption and the uncertainty in the reward, I think that held back adoption. That said, I think there's reason for some optimism about the technology itself. So, Garrett Geron and I, we have separate work studying the advertiser side performance of privacy sandbox. There we use data from a DSP, demand side platform, covering 3,000 advertisers. And the goal of that paper is to obtain the incrementality from retargeting through the cookie-based channel and also the sandbox channel, which is a puppy channel, right? So, we get this broad-based evidence about incrementality. And what we find, we find that incremental conversions per dollar, if you adjust it for spending, we find that it is pretty similar across the

cookie channel and the sandbox channel. So, this illustrates that if you adjust for the fact that there's not a lot of supply of impressions, meaning you don't get to spend a lot of money through the sandbox channel, we seem to get more similar KPIs, similar performance. So, I think that that result implies that the technology can perform, but the problem is that this equilibrium of these ad tech firms, both the supply side and demand side, adopting it, it just never formed so far. And that's part why I think privacy sandbox, so as, you know, get or set, it's abandoned, but I don't think it's about it's abandoned for performance reasons. Google probably killed sandbox mainly because of the recent antitrust developments, like the DOJ ad tech case and also some of these investigations going on in the European Union. So, I don't read this abandonment as a

failure of PATs in general. And I I I'd like to reinforce what Garrett said. So, you know, cookies keep fading, right? And, you know, we have to choose between going to contextual advertising or we allow like more alternative IDs and, you know, ad tech firms making identity graphs from that or we reach to some kind of privacy-preserving compromise. You know, that that'll be great, but again, you know, you there's nuance, right? The challenge here is it's that PATs, you know, you have to get ad tech firms to adopt it and also abandon third-party cookies. And that takes uh standard bodies like the W3C, policymakers or big tech firms to force the move. And this is really hard because as this sandbox experiment shows, even green-lighting these technologies is hard, right? Like experimenting it and you know, like like satisfying different people, different entities, that's going to be hard and setting like, you know, like

what's the epsilon like what like we discussed, that is going to be hard. And I think that's more than technology itself is the real barrier to having privacy-enhancing technologies.
You well, when the new paper drops, I'll have to have you both back on. That sounds really fascinating.
Mobile game developers no longer [music] need to pay up to 30% in major app store fees. With Xsolla Web Shop, you can create [music] a direct storefront, cut fees down to as low as 5%, and keep players engaged with bundles, rewards, and analytics.
[music]
Start today at xsolla.com. That's xsolla.com [music] or use the link in the episode show notes.
All right, I want to I want to move on to the second paper on the docket for discussion today, which is COPPAcalypse. So, Garrett, you wrote or you published a paper recently about the COPPAcalypse with YouTube. So, maybe can you give me uh background on the paper, talk to me about the methodology and the and the headline findings?

Cool. So, yes, in September 2019, YouTube paid the largest fine ever under the child's children's privacy law, this $170 million settlement to settle charges it violated COPPA. And as part of that, beginning in January of 2020, YouTube identified kids' content and eliminated all personalized ads associated with that content, as well as on-site personalization features like commenting and playlists and subscriber notifications. So, the title of our paper Copocalypse comes from the panic that this created among made for kids creators. We don't know exactly what the impact was on personalized ads revenue because that data is not public. If anybody happens to have that, I would love to to talk with them. But, at least one creator shared their data with us and we saw that the ad prices fell 73% without personalized ads. So, this was very meaningful for these publishers. We studied this by looking at 5,000 top US YouTube channels and we compared the

made for kids content creators to the non-made for kids content creators using a difference-in-difference design. And what we showed is that the made for kids content creators produced 18% less content. They also started to pivot away from made for kids content and reduce the quality of the content they were releasing by releasing less original content, by reducing them their manual captioning, and also their video ratings started to fall. On the other side of the market, YouTube views of made for kids content fell by 20% and subscriptions fell by 25%. So, these are very meaningful movements and this work is actually co-authored with a different team. This is my colleague Tess Rylan, who is in my neighbor on the other side of my office, and Leon Jong from University of Hong Kong who graduated from BU, and James Cooper at George Mason University.
Yeah, that's fascinating. I mean, that's a pretty substantial drop. I So, that was actually that was part of my argument. I've been writing a series Netflix's YouTube opportunity and they've been poaching these these made for children

creators. Miss Rachel was the first, but they have also got Danny Go and they've got um Mark Rober that they just signed. And, you know, so it's probably much more enticing when you say, "Look, we're going to just give you we're just going to buy it. We're going to buy the content. You can maintain on YouTube, but if YouTube the revenues are deteriorating as a result of that's what I said in the piece. You know, look, the revenue is not it's not personalized ads right against this against the you know, I guess your content. And you don't you could argue nor should it be ever, but you know, it's it's probably more enticing than to go to a company like Netflix. Okay, well, I'm not going to it's not going to scale with my viewership, but I get a kind of guaranteed payment. Okay, but the So, the paper is ostensibly about COPPA, but it reads like a preview of what happens whenever platforms remove personalization. So, do you view the YouTube settlement as a useful analog for things like ATT, for privacy sandbox, or any future restrictions on behavioral advertising more broadly?
Yeah, thanks for mentioning this. As you were alluding to before, you know, this broad agreement that children's privacy is important. So, personally, I was less

excited about studying this topic. It's a bit of a third rail. But, there's few opportunities to study the impact of restricting behavioral ads on content creation, and we think that this YouTube settlement is an unusually clean case because the platform eliminated online behavioral advertising, and they're imposing compliance on other firms. So, you know, ATT is a good comparable because Apple's policy imposed the ask app not to track prompt on all the apps that were using the mobile ad ID, but you know, the YouTube settlement meant no behavioral ads, but there was still some minority of iOS users that were providing consent. And also, YouTube is a closed ecosystem, so there's no workaround for advertisers. But, you know, Eric, you've written about how iOS is implicitly or tacitly allowing some amount of fingerprinting to still happen on its ecosystem. So, it's also interesting to contrast this with the GDPR. You know, the GDPR did not force a ban on behavioral targeting in reality. You know,

arguably, that's what the law was intending, or that's what certain regulators are intending to do, but that's not what actually happened. And so, what we saw was that the industry tried to maximize cookie consent for as much as possible for for long as possible. So, undeniably, you know, now that there's a lot of greater ease for users to opt out of cookies in the EU, this would have some impact, but if you try to study this empirically, it's going to be very hard to see this blunted impact in the data.
Yeah, fascinating. I mean, you will keep in I mean, also the DMA wanted to eliminate behavioral advertising wholesale. That was originally that was in the original text. They watered that down. I mean, that went you know, that that went through the tripartite negotiations, but that was originally proposed there, and they watered it down. I mean, that's the that's the legislative process, right? And didn't pass it didn't make it through. So, presumably, there's some recognition at least in Europe that okay, there there'd be a an economic harm imposed if we did that.

Okay, one of the most noteworthy findings in my mind in the Copaculus papers is that views became more concentrated among the largest channels. So, should we interpret that as evidence that personalization disproportionately benefits the long-tail creators, and if so, does that have implications far beyond children's content?
Yeah, you brought up an important point about regulation in Europe. So, yes, European regulators have flirted with getting rid of behavioral targeting, and actually the the DSA does ban targeted advertising for minors under human this is defined as under 18, COPPA is under 13. So, these I mean, this YouTube policy we're talking about applied worldwide, so it wouldn't have an effect there, but you know, it does ramp up these um restrictions, which we could maybe debate where what the age level should be and and just what is being accomplished from a privacy perspective, but certainly it's impactful for creators. So, you going back to your question about like the impact on the largest channels, you know, it's interesting

because a recurring finding of the privacy literature is that smaller firms are more affected by privacy regulation, um and I think we know that personalization tends to benefit long-tail creators. Now, in this setting, actually personalized recommendations and personalized search remained on. It was these other elements that would kind of make YouTube sticky, like being able to add videos to playlists, commenting, end screens, subscriber notifications. That's what was deactivated. And it's not, you know, wholly clear why that would affect smaller channels more than larger. I mean, probably a little bit. But, it's important to note that there's also an effect on the content creation. So, the smaller channels reduced their content creation by more than the larger channels. And so, I think this creates this negative feedback loop where they have less content, which gives them less views, which gives them less money, which gives them, you know, less ability to create content. And, you know, you can speculate also that maybe these smaller creators were more dependent on YouTube revenue than say the Disney Kids

channels. You know, that you you brought up this point about some of these creators being able to work with Netflix. Those are going to be, you know, the larger creators. And it's probably harder for the smaller creators to get alternative revenue sources like sponsorships, too.
Right. I mean, I was going to bring that up. Like, if you're a Dude Perfect or you're a Mr. Beast, well, okay, I'll just go to influencer campaigns where I'm promoting the product.
I'll sell more chocolate bars.
So, yeah, exactly. And and versus just the ads that I'm depending on. Yeah, it's really fascinating finding. Okay, I want to maybe we could do like if it's, you know, if you're okay. So, you previewed the paper that you're working on. Maybe could you talk about some other stuff that you're working on? I mean, you kind of operate right in the nexus of the MDM content interest graph. So, so maybe some a little bit of a a sneak peek of other stuff you're working on?
Yeah, so Garrett and I and Jen Wong and another co-author called Yufei, we actually have a new project on Privacy Sandbox. So, we're using the same data as Raptive, but here we try to ask a question more

about like, okay, Privacy Sandbox, we see this effect, but, you know, can we decompose the effect into different APIs? So, Privacy Sandbox is it's not just, you know, Protected Audience API or PAPI. It's a suite of different technologies. We have things like Topics API and also Attribution Reporting API. And when we look at the Privacy Sandbox, we you know, mostly see PAPI, but there's some effects from Presumably, there's some effects from Topics API and also ARA, too. And in this project, we still working on it, so we don't have like results to share, but we're trying to understand what is the the effect coming from PAPI specifically and also what are the effects from these other APIs. So, we have this, you know, awesome field experiment, you know, to use and also Raptive, they also switched on and off Protected Audience APIs for certain occasions. And we can use this as like, you know, event studies or almost like,

you know, diff-in-diff design again to study the the differences made by different APIs. So, that's what we're working on in terms of Privacy Sandbox.
Fantastic. Well, I know how big of a lift these projects are. I know that it it's an extraordinary effort to to undertake a paper like this and get it published. I thank you both for your contributions to the space. I think I'm going to link to both papers, but I I recommend that everyone read them both. And I appreciate you coming on the podcast and sharing your wisdom.
Yeah, thank you so much for having us. It was a real It was really fun to be here.
Thanks for having us, Eric, and thanks for all you do. It's when I'm on the listener side, I always enjoy the great insights that you and your guests have to share. So, thanks for using your time in a way that benefits so many of us.
I appreciate that and appreciate you both coming on. Thank you very much.