S7, E25: The impact of AI Overviews on outbound clicks (with Ananya Sen and Saharsh Agarwal)

Mobile game developers no longer need to pay up to 30% in major app store fees. With Exolo Web Shop, you can create a direct storefront, cut fees down to as low as 5% and keep players engaged with bundles, rewards, and analytics. Start today at exola.com. That's x ss o l.com or use the link in the episode show notes.
The problem is that the distinction needs to be drawn between the competence of the economists and the correctness of their analysis. Hello and welcome to the mobile dev memo podcast. I'm your host Eric Sufort and I'm joined today by Ana Sen and Sahar Agarwal. Ana Sahar, welcome.
Thanks, Eric.
Thank you, Eric.
So, I asked you both to join me on the podcast today to discuss your preprint

paper, the impact of Google AI overviews on publisher traffic and user experience, evidence from a field experiment. I will have links to the paper in the show notes. So, you know, anyone listening along, maybe you want to pause the audio and and go quickly browse it, skim it or or read it very closely. It's very very fascinating field study that you did. I shared it on LinkedIn and Twitter maybe a week or two weeks ago and it generated a lot of conversation. So I'm I'm I'm excited to to talk through it at length today. But before we do that, why don't you both introduce yourselves and maybe Ana, we can start with you.
Sure. Thanks again, Eric, for having us. So I'm Ana Sen. I'm an associate professor at Carnegie Melon University's Hines College. So that's the Hines College of Information Systems and Public Policy. And in general over the past decade, my research has focused on topics at the intersection of technology, business, and society. I've particularly focused on digital

platforms, media platforms, and you know, media platforms in the context of the online information ecosystem more broadly. And you know, I've been at Carnegie Melon for about 7 8 years. And before that, I was a posttock at MIT Sloan School of Management. and I got my PhD from the Tulu School of Economics.
Great Sah.
Thanks Eric for having us here. So my name is Sah A Shagaral and I'm an assistant professor at the Indian School of Business. So broadly my research focuses on technology and how it changes the way we live, how we interact online, the offline implications of technological developments. And so broadly my research recently has focused around issues related to digital well-being online governance of platforms and competition policy and in fact the paper we'll be discussing today sits squarely in the space of platform governance and competition policy. So I've been at ISP for the last three or four years now and before that I was a PhD student at Carnegie Melon Science

College where I met Aran.
Great. Yeah. I mean I think this paper and in this topic broadly it's touching on a really important topic which is you know in some ways still able to be steered from a policy standpoint. My my sense is like a lot of the protestations about you know Google's behavior with you know in the adtech trial and the search trial were a little bit futile. I mean like there was just nothing to be done at that point. It was coming decades after the fact right or at least a decade after the fact. Whereas here I mean there can be like a policy intervention that does kind of steer things in the way that policy makers might feel is is best. And so it's I think it's it's really great that you've studied this. You know, I've I've written a and it's not it doesn't only pertain to Google either, by the way, but just to make that clear. But I've written a series called Google's Gambit where I I talked about well, you know, look, this is what they're doing. I kind of articulated I think in real time and

it seems a little bit silly to even to even talk about after the fact because it is after the fact now, but the first Google's Gambit article that I wrote was when they first introduced AI overviews and my point was this is where they're taking search. They want it to become this interface that doesn't lead you away, that keeps you engaged in the chatbot experience and that will, you know, deprive publishers of clicks. And it's great that you both have authored this to sort of study that empirically. Okay. So, please provide me an over just kind of talk me through an overview of the paper for the people that haven't read it. I mean, I read it and I I suggest anybody that's interested in this topic should read this. I believe it's it's the first kind of piece of academic work I've seen that has really explored this. I don't know if if you know of any others, but it's great that this is out there now, but could you just provide an overview of the paper and its principal findings?
So Eric, as you described the motivation for the paper search, the search market is traditionally very symbiotic. So

publishers depend on search engines and search engines depend on publishers and it's a very healthy ecosystem which has been created over the last couple of decades. However, in recent years, we've increasingly seen this trend towards Google trying to capture some of that value by trying to keep users on his platform and not send traffic downstream. And so, yeah, it's been a trend over the last few years with knowledge graphs, feature snippets, etc. So, this discussion has been pretty old. However, with AI overviews, it's taken an entirely different scale. Firstly, because of the prevalence of AI views, the prevalence is really high. also because of the nature of the answers that they generate which really reduces the need for people to click and so the paper is simply about providing very strong causal evidence on this topic and as you described yes there have been people who tried to shed some light on this topic and most of them have been fairly correlational for instance comparing before and after comparing

queries with overviews and without overviews and we know that that's not causal enough and so that's the gap that we really wanted to fill And so the best way to answer this question is just to collaborate with the platform and to run a field experiment on millions of users. But of course it's very hard to do something like that. I'm sure Google must have done that internally. But as researchers we didn't really have that kind of access. And so what we did was we ran a field experiment with live users and we had to create a custom browser extension. And so this browser extension did two things. The first was it passively tracked data from people and so in a very natural browsing setting we got data from about thousand people on all the websites that they're visiting all the searches that they're making the we get to see the search results that are presented to them whether an AI will be shown or not how much they're clicking where are they clicking and so on and also besides just passively tracking the key part was that we were able to actively manipulate the search interface and so in the main study we had a

control group where we just tracked and not did anything else and we also had a treat a treatment group where we hid the AI overviews whenever Google chose to show it. We because this was randomized therefore all else was equal and what we measuring was causal effects of showing AIO views on clicks and so we see that when we hide the AIO views the downstream clicks the organic clicks really increase a lot and the increase is fairly significant if we measure relative to the no AI overview count of you can say that showing AI overviews reduces clicks by 40% conditional on a happening which is a fairly big number. We also find that these clicks are not necessarily higher quality. The clicks which are being lost sorry the Google's argument has been that the that yes AI overviews even if they do cut clicks they only cut the noisy clicks the not so useful clicks but because we have data on the amount of time that people

are spending their interaction with the downstream page and so on. So we we're actually able to shed light on this and and we find that that the quality of the clicks were actually not different across the two groups and yeah one of the headline results that we find is that although we find such huge effects on downstream publishers in terms of clicks surprisingly user experience was absolutely unchanged even when we hit the AIO views. So in terms of overall search experience, in terms of the quality of information, in terms of ease of finding information, we found that users in both groups felt exactly the same. So yeah, that's a brief overview of a paper and happy to go into any details if you like.
Yeah, that's great. And I I mean I like I like the point you made about like you well the you know the the ideal way to do this is to institute an RCT at the service layer but you know okay it's like I think a lot of times researchers make the perfect the enemy the good in that in that sense it's like

well yeah that's that that would be the ideal way to do it but you can't do that and so I think you found a very very clever way of collecting this data. I think the methodology was pretty impressive. Okay, so one thing that struck me from the paper is that users didn't report any improvement in search quality despite the 40-ish% reduction in outbound clicks. How do we interpret that disconnect? I mean, you you kind of offered up one interpretation, which is that well, when Google says they're only removing noisy clicks, maybe that's not actually what's happening. But what's what's the right interpretation of that finding?
Yeah, that's a fair question because when users are shown views, they they're not clicking. So then you would expect that that must be really improving the search experience otherwise why would they not click. However, when we remove it, search experience is not really changing. We thought about this a bit. This sort of probably even relates to some of the results that we had about the position and that we can probably talk about later in this discussion as well. You know, we find that most of the times EI views are basically shown in

the prime spot at the top. Common experience suggests that whatever people see in like most of the times people don't even scroll too much like they just do a search and then within a few seconds they just click and they go away and so whatever they end up seeing in the prime real estate is what they're basically giving attention to and so when AI when an AI overview is sort of to use that word forced upon them at the prime real estate that's basically what catches their attention their question is answered and therefore they don't really feel the need to click however when the AI overview is not there they they they're equally happy visiting the various links. It's not like they're having a worse search experience. And so I think some of it is basically just to do with the fact that that yes, if the AI view is silent enough, then maybe the users engage with it and they find some value with it, but the moment it's not there, it's not as if users are looking out for it, they find that there there's something missing in the search experience. So yeah, it's one of those products which where it seems that unless you really make it salient and

you give it the prime spot, maybe like users don't care about it too much. So that would be the indication from what we have. But of course, like I I'll have to be a little careful here that our survey measures are self-reported and it's possible that yes, there are some relatively small improvements in the search experience in terms of convenience which our survey measure is not really able to capture. So I'll just have to be a little bit careful about that. Yeah, I mean maybe maybe we could just take a second and talk about you again like the best way to do this would be an RCT at the service layer. It can't do that. Like this is a very good way of doing it, I think. But like maybe talk to me about some nuance that should be applied to the interpretation of the self-reported data because it is self-reported, right? There's not like a not like a proxy or something. But maybe maybe just to add a little more color there like what is some nuance that we should approach the self-reported data with? Well, I mean it's it's possible that when I see the AI overview, my search experience improves slightly because of convenience. However, it's

not a very big shift and therefore when I'm asked about this at the end of two weeks. Remember like the survey measure was collected at the end of two weeks. It's not really collected in real time, right? Like as people are searching on the go. So it's possible that while you were searching maybe you had a slight improvement in your convenience but like it's not something that that you remember at the end of two weeks basically.
Yeah.
So I think that's the that's the nuance
but you know one thing I'd like to call out is that we also had a third arm in the paper where we forced people to use AI mode.
Right. Right.
And then we did see a shift in the in the experience numbers. Right. So it's not as if these experience numbers do not indicate anything like in the in the EI mode arm we saw a huge decline in all the experience numbers. So that clearly stuck with people right like even though we asked that question after 2 weeks we definitely saw a reduction in experience. I think that also says something that yeah this measure is is definitely capturing some signal but given that it's self-reported we just have to be a little bit careful hold it.
So the experiment measures the shortrun

effect on clicks. Do do you think click behavior might change meaningfully once AI overviews is no longer new and novel and and users have grown accustomed to it?
I think that's a great question because you know the space is moving pretty quickly. So you know we need to sort of zoom out from our study and think more broadly. But before I do that, maybe just like two thoughts about how we should think of these results, you know, and short run versus long run in general, you know. So our experiment lasts for 2 weeks and we observe behavior for 2 weeks and then you know we make the final payment for people who've installed the extension and then people are free to uninstall the extension as well. But a decent proportion of people about 60% still keep the extension installed. And so at the end of the twoe period which is the main experimental period we switch people's intervention. So people who were in the treatment group become the

control group. People who were in the control group the status quo now their AI overviews are hidden. So we have two weeks worth of main experimental data and then we have two weeks worth of post-experimental data where we switch around their treatment and control conditions. But in both these periods we find results that are pretty consistent which is that click-through rates on organic links increase when AI overviews are hidden. So and this this behavior sets in pretty quickly. We saw this even in you know some of the some of the smaller pilots that we had done before launching the main experiment. So given how AI overviews are currently I think this effect is pretty robust even though you know we measure something for 2 weeks and you know again m maybe stepping back a little our experiment is in a particular point in time. That

said, you know, these AI overviews have been around for, you know, over two years now, if I remember. Like I think they were first launched in like May 2024 and the percentage, you know, the percent of queries that have seen AI overviews has been increasing over time. So, I'm guessing people are getting used to it, but it's of course things can change as we go further down the road, you know, which goes to the heart of your question, right? what happens a as we move forward how might behavior change I can see it change in you know potentially two different ways two opposing ways let's say you know AI overviews and how they appear on Google search page remains you know similar to what we see then maybe people actually learn how to sort of you know read the AI overviews you know scroll past them just because they know it's there but they also know that you know they can

sort of ignore it right they don't always need to click on it that would mean that you know our treatment effects might be you know an upper bound overall and that you know things might actually become better for publishers down the road but it can also work the other way as Sah mentioned that you know we do have a third arm which is AI mode which is you know in some sense if you think about why Google might have introduced AI overviews in the first place which is you to compete with you know the LLM chat bots other LLM chat bots out there AI mode might be the natural culmination of this process in that case even based on our results we see that click-through rates reduce even more on organic links so you know it could go either way I would think that the reason why we had the AI mode arm was exactly because we were trying to think of okay what might it look like down the road and if it's I

think it's highly plausible that it might look something like AI mode in that case we see downstream traffic being even lower right yeah no I mean it's a very fair point that this is not new right you know May 20 you're right this was launched in May 2024 I mean people have had more than two years with it now to be fair I mean it launched in like a limited beta in May 2024 it got rolled out to the broader set of countries in 2025 so but even still it's been in more than a year I mean this isn't like a new idea And it's not as if chatbot interactions haven't become ubiquitous in the consumer experience just broadly, right? I mean, that's that's how people interact with information now. And so, like a point that I make a lot with respect to like commerce and ads is that the stuff that hasn't taken off yet probably is not going to because people seem to characterize like, you know, consumer interaction with chatbots as like this new phenomenon and it's not. it's years old. Like we're not in the early innings. We're, you know, and

some of this stuff is we're late. It's happened. And that's the argument that I made with Agent Commerce. It's h it happened. We tried it. It didn't work. That's been retired. It's been we moved on beyond that. Okay. But I think like one of the questions that kind of pointed a spotlight at the paper and a lot of people kind of reacted in predictable ways because there is like a lot of animosity towards Google, especially from like the traditional publisher sphere, you know, given maybe some sense of grievance around what happened on the ad side. But you know like the reality is Google in a way was responding with AI overviews and keep in mind people had written Google off. I mean people said search was going to go to zero right and so as a result of chat GPT and like the explosive growth that it experienced. When I've spoken to people at Google, I've made the case that like look, Google's gambit was very much like a deliberate choice. And maybe

the purpose of that wasn't to abandon the open web. Maybe the purpose of that was not to starve the open web of traffic, but that was going to be the outcome. And you know, people at Google have said, "Yeah, sure." I mean what are we going to lean into the open web or are we going to go the opposite direction and try to be the protector of the open web when our own business is under threat from chatp what option did we have I mean if we could recognize that that outcome it wasn't the purpose that wasn't the reason we did this and we knew that it was going to be a consequence of it but achieving that consequence or effectuating that consequence wasn't the point of this but what else could we have done right and that's that's kind of been the reaction when I've spoken to Google people about this which which feels like okay fair. This is just a tectonic shift in the consumer technology landscape and everyone is going to expect to interact with a chatbot. And I think you're right that the Google AI mode is the end point of this. It's the the AI overview's always felt like this kind of transition

phase is just to get people accustomed to it. But my sense has been you go to google.com and you land on Gemini at some point. You know those channels your colleagues keep bragging about, the ones getting all the credit? Yeah. They might be doing squat. Attribution makes every channel look like a hero, even when it's a zero. Incremental tells you who's actually doing the work. It's like a lie detector for your marketing budget. Start using incremental today. Get your demo at incremental.com. That's n cmt.com. Mention that you came through the mobile dev memo podcast for a special 15% discount for the first 6 months. So the deterioration in the click rate doesn't account for the fact that some publishers that have done data licensing deals with Google might be preferred in AI overviews in terms of outbound traffic and and that's not like a criticism that's just you couldn't parse that but that's just the fact it's just the reality. So given that that pretty

meaningful deterioration the click rate, could some publishers actually be seeing an even sharper decline because they're not participating in these data licensing deals and therefore you know the actual outbound clicks are preferencing the people that are and so if you would exclude the the companies that are doing the deals, the decrease is even more extreme,
right? I I think this is a super interesting point and it made me you know look up information about these licensing deals and actually I found based on the Google AI overview in fact that about 20 major sort of international outlets have signed licensing deals right now with Google as you said it's it's not a dimension that we explored in the paper because you know as soon as we start sort sort of digging into subsamples, we lose power. You know, we have a thousand people over a few weeks, but it's hard, right? It's hard to credibly detect effect as soon

as you start looking at heterogeneity. That said, I think, you know, I think one thing to keep in mind, at least right now, is that, you know, the proportion of clicks coming from AI overviews is pretty small. In our in our study, it was about like 78%. which is you know which is higher than what I've seen reported in the Pew Research study but it's still only you know 7 or 8%. I think there is the broader aspect of your question I think is also very interesting which is that given that leading news outlets are making these deals and let's take it to its logical end point. Let's say, you know, that's one of the models that comes through that there is a licensing play, then it's perfectly possible that outlets that aren't signing these deals don't show up in these AI overviews and then, you know, that eventually spills over onto search results and then it has

an even more negative downstream effect on them. And of course, you know, it's not only on the publishers to decide whether they want to sign these deals or not. Who does Google think is, you know, is worthy of of these deals? What is important to them? The thing that I do slightly worry about is that, you know, when I think of the open web, when I think of, you know, the internet, e-commerce, etc., we often think about the longtail and the benefits of the long tale to the online population. But I fear that here the the smaller the smaller outlets, the smaller publishers might lose out the most. But you know that that remains to be seen. Maybe that said, I'd also like to just you know take a step back and talk about the reason why I feel that using this Chrome browser extension based type of study is the right way to go because Google is an interested party. Downstream publishers are interested parties. Right? So th

this was our way of sort of providing reasonably neutral evidence as a neutral party per se. We don't really care which way the results go. Our our main job is to pin them down rigorously and report them and then you know contribute to these conversations. But yeah I mean I think there are various ways in which this would play out in the future.
Yeah. So you make a really good point about the data licensing deals. talked about this in the Google's Gambit series, which is that, you know, not only is it a little bit risky for a publisher to do that licensing deal in the first place because I mean, you're giving away there's no guarantee that you're going to get the clicks and so just doing the data licensing deal could ultimately result in, you know, you getting fewer clicks because your information, your data is going to appear in the AI overuse AI mode and it may obiate the need for a click. But at the same time, it's like Google kind of becomes kingmaker in that. I mean they've always been kingmaker in the sense that you know they had an algorithm that determined you know which

which websites got clicks right you know presumably that was to your point that was fairly and I I don't want to say it wasn't it was it was opaque for sure so it wasn't transparent in the least but you could maybe presume that it was in the best interest of the consumer because Google was incentivized to drive you away right I mean they want you clicking out in the first couple of links and so they're probably going to be pretty good at ranking and page rank. That was the whole point of that. And then then when ads got introduced and ads, you know, expanded to become the whole top of the fold, then then they you could make the argument which I've made is that they created a situation where actually it was better if for everyone everyone's interests were kind of aligned if the advertiser that got the placement would have been the top link anyway. like that everyone's interests are aligned in that way and so you could make the case that like look this auction mechanism still kind of ensures that the best information is being

placed up front now best is very qualitative in cases like shoes or whatever but but nonetheless right but now that's that's demonstrabably not the case I mean they want to keep you there presumably to show ads to you and keeping you in that environment in that engagement environment allows them to show more ads and so they have no incentive to drive you out. Now the question becomes who do they need for that purpose? They don't need a very small niche longtail publisher. Whereas before for a non-monetizable query that might have been the best destination and they might have preferenced you over some much bigger website because you know they crawled everything. Your the size of your user base or you know the number of page views you got every day was irrelevant to them. They just wanted the best information and they had ways of qualifying that. But but now they don't need you because you need a lot of data for to be useful to them. And if you're some very niche publisher without a lot of data, without a lot of content, you're not valuable to them and they're not going to do a licensing deal with you. And then there and therefore you're left completely out in the cold. And so I guess you get to this like Fouian

bargain situation where you say, "Look, I know if I give you my data, I can't control whether you're going to drive clicks to my website anymore, which would then enable me to monetize as a publisher through ads, but if I don't do the deal with you, I'm pretty sure you won't drive any clicks to me." and and I'm also therefore starved of the ads revenue while starving myself of any potential data licensing revenue. So it's very it's a very interesting situation that you know publishers have been put in but to your point it also probably disadvantages the smaller publishers the most. Okay so you talked about the intervention so you've got the intervention one intervention removes AI overviews entirely. Could a version of AI search exist that preserves most of the user benefit while restoring publisher traffic? Is there is there a version of AI overviews that kind of hits a sweet spot in terms of driving clicks out but also fulfilling that what I think is probably a need now to give consumers that natural text search opportunity.
Absolutely. I think that's a great

question and it also helps us appropriately clarify the scope of our study. So what we are basically studying is as per the current design which we are seeing the status quo what is that doing to downstream traffic and user experience right and so our intervention is basically studying the impact of the current design is not saying that AI search in itself is always going to lead to the same result and so therefore yes I do agree that what that a lot of what we are measuring is very typical to the design of the interface as well And it's quite possible that there might be a different version of AI search where yeah there are AI capabilities but at the same time publisher traffic is also not lost and in fact I think Google is also exploring some of this. So in in the recent core updates and recent Google IO as well they did mention some of these changes for instance they're trying to make the links more clickable make and in fact I think even have have like a garousel of

links in the AIO view and so yeah they're basically trying to make the links a little more prominent. So I think the impact of these is not yet known. So I think it's a it's a fairly open question and it's an interesting and it's an important question. So while our paper basically look looks at what is happening right now but the question that you pose is very different but very interesting which is that is there a version of search at all where we can get the best of both worlds. I think it's something which I think there would be a lot of platform design elements that would go in here. uh so I think there's more research required on on this front and in fact of course you know there there are a few other things that you can also think of like if if you don't want to go down the fancy route of maybe playing around with the with the font size of the views or of the prominence like how much of the snippet you're showing one simple thing could be just to reduce the prominence for instance rather than let them have primary real estate maybe just move them

to a less prominent position maybe maybe give users a choice right okay and AI movies available. Do you want to view it for instance? Now there you you still have some element of AI search but because it's not being sort of forced upon people maybe publisher loss will not be that heavy. I think these are all very interesting scenarios to examine but something which our paper doesn't do currently.
Do you have any sense of what that might look like? Like what might that entail?
Well Google will definitely not do it not to solve this one on their own. Of course they're making some of these interface changes on their own like for instance making the click making the links more prominent having having like corousal of link so all of those things Google is trying to do but of course Google is not really letting go of it prominent position so if we talk about reducing the prominence of the AIO views I don't think it's going to happen on its own because we we mentioned that incentives are very differently aligned here and I mentioned to reduce the prominence of AI views maybe not have it as a default maybe give users the choice to unroll it if they want to. I don't think those are going to happen without

regulation. I don't think we we we can get to that without regulation.
Right. Again, I know you're not like a product designer and neither am I. Right. But is there a way to have both? Like I mean, I guess you could I don't know. You just have a Yeah, like you said, a different colored background on the info box that contains the source links or something like that. Just something to draw the user's attention to the actual source of the information. I mean I feel like something like that might even as simple as that could work.
There has been decades of research on like nudging and basically just nudging people towards specific elements on the web page and so on. So yeah I think I think all of these definitely the possibilities are there. For instance like right now if you look at the AI overview they not just there at the top position but you also get to see a significant amount of the AI overview and then you you click on the show more button to see more. What if you just reduce that size? Like you just get to see one or two lines. Maybe user don't really care about it if it's not that prominent. So some of some of these

interventions can actually be fairly simple like you just have to make it less prominent and let the attention go organically through the other links. But as I said, it's not something which you can expect Google to do without some external pressure.
I think the result that surprised me the most was that the additional clicks weren't lower quality. Right. So, does that I mean I know we talked about this at the beginning, but like how do I reconcile that with what Google has said publicly? Because I don't think that they're mismeasuring this and I don't think that they're, you know, proaricating. I I there's I'm I'm just wondering if there's like a different interpretation or if the way that they've talked about, you know, these kind of false clicks or the lower quality clicks or the bad clicks is just a definition that might not be intuitive to somebody who doesn't work at Google. Is there is there an explanation for that that allows me to reconcile both of these things?
We've also seen those some of those statements and we also mentioned that in in the paper and again you know the

point of the exercise was to try and understand the impact of such an AI product on different stakeholders within this ecosystem. So one of the things that we've heard is that you know the quality of clicks that's coming through AI overviews is better but when we analyze it conditional on a click we don't find any difference based on you know the dimensions of quality that we can measure across treatment and control right so we don't we don't find any difference in terms of bounce rates or downstream time spent or the probability of clicking on the back, you know, to the go back button. So, on the face of it, it does seem to go against what Google is saying. And you're completely correct in speculating that maybe, you know, we're just talking past each other, right? It's like maybe Google is

measuring something else and what what we are measuring is different. And again, our objective is to just try and pin down the most rigorous evidence that we can and we're happy to take into account any sort of data that that Google provides on this, but I haven't seen anything yet. We have these measures as part of our experimental setup, but we can't look further down the funnel. you know we don't know whether people are purchasing more because that's not something that we can observe the proportion of clicks that are coming from within AI overviews for us is 8% but that you know 8% you know constitutes a reasonably small number just given our sample size so maybe Google has a significantly larger sample size and they're sort of measuring something differently or based off of a larger sample size or they might be looking at some certain types of segments which again we can't do. We

don't have enough power. So again this is not to say that the publishers are all right, Google is all wrong or vice versa. It's just that this is what we find and the good faith sort of implication of this is that we want to read these results in good faith and just contribute to the conversation around the online information ecosystem whether whether that's policy whether that's related publishers or or Google I think it's it's possible that they just have some very proprietary metric that would be impossible to you know intuit it from the outside that is a bad click but yeah I mean it just it does seem like it's difficult to reconcile.
Yeah. At the at the very least, for example, we would have to reconcile the fact that okay, maybe something let's say is happening down the funnel, right, for certain brands, for certain types of purchase. But at the top of the funnel, we are still seeing sort of no differences across treatment and control. So, you know, AI overviews versus no AI overviews. So, any

explanation has to sort of overcome that at least because these are effects that we can pin down reasonably well. AI is changing everything, but it's only as good as the signals behind it. Branch connects customer interactions across paid, organic, offline, email, web, and app touch points and turns them into the trusted context you need with links and attribution that capture the full user journey. Learn more at branch.io. And while you're there, check out Branch's AI search and discovery report covering insights from more than 300 enterprise marketing, growth, and digital leaders to understand how the industry is responding to the rise of AI search. That's branch.io. How much of what you're measuring is about AI versus just the value of that particular block of real estate, right? So like imagine if Google replaced AI overviews with something else in that position zero. Do you think you'd expect a similar effect like a m I mean I don't

know what that would be but you know you could just just a big block of a white pixel like if their logo was 10 times larger or something would would that you would you expect to see the same kind of effect?
Yeah I think it's a great question it's something we thought about a lot as well and in fact even discussed some of this earlier in this discussion. So so is it about AI is it about the real estate? I think it's a bit of both basically AI overviews they're combining two features. The first is that they're providing a synthesized answer but they're also occupying the most valuable real estate because of which all the other organic links are getting pushed down. Now in our experiment both of these are moving together. When we move the AI overview not only the AI feature is gone but also the the prime real estate is now taken up by the organic links. So therefore we cannot really separate one from the other. But however think about it like if let's say that instead of the EI we just had a block of white space for instance as you mentioned. So now what would happen is that yes a lot less people would now access the third or the fourth link

compared to if it was just thrown at them by default because we know that people don't scroll a whole lot and in fact people talk a lot about the phenomena of zero click searches but in our data we are also able to observe scrolls. We don't really talk too much about it in the paper but we find that a lot of the searches are actually zero scroll searches. So people just don't scroll not just they don't click and so so yeah search is fairly it's it's a very short process like you you put in a keyword within two seconds you click something you don't people are not very thoughtful about it so yeah if if you show me six results on the main page maybe I click on one or two of them if you show me only one or two because everything else is just white space so yes maybe I will now see less of the other results however I don't think that the effect is going to be the same as that of AI overviews because AI overviews do two things. The first is that they move all the organic links down also. Secondly, they they also satisfy your answer there itself. So now you feel much less the need to go down

as well. If you just had white space, then yes, you might not scroll as much as you would have earlier, but but now your your answer is not satisfied, right? So so I do think that yes, some of it is position, but it's not just position because you you're getting the answer there itself. So therefore, you have much less reason to go down. In fact, let's say you didn't have white space. You just had a bunch of other links like for instance a bunch of shopping carousels for instance. Now that is also occupying prime real estate but that that is leading you to a click right. So so yes although that that carousel might get a lot of attention because it's in the prime spot. However the the function of that carousel at the top is divert you downstream but the AI overview does the opposite. It takes the attention and then it satisfies your answer there itself. So I think it's a bit of both. I don't think it's only position, but I think that a lot of it is driven by position.
Yeah. I mean to be fair, I don't think the effect would be the same if it was a white space or just a larger I mean I think that's the the point is the AI

overviews is soaking up a lot of the the engagement. But it is interesting to tease that apart because I think that it's it's important because that's been optimized. I mean Google knows exactly how much vertical space they have to play with with anything non-answer related before they lose engagement. So my sense is like most of it is probably the AI. Okay. So I mean I I know we've got like seven minutes left. I I want to be respectful of your time, but like talk to me about how much have you thought about the sort of like the right policy response to this because I I do think it's important to kind of bang the drum on the idea that this is when you can actually influence things. If you wait a decade, you can't. And this is the moment to actually be paying attention. And I don't know that this FTC or this DOJ is interested in intervening on these types of issues. My personal sense, and I don't want to put words in your mouths, but my personal sense is that they are not. And so maybe there's just no policy prescription that could be received. Assuming there was, like what

would you say if you were advising the FTC or the DOJ? What what's the least interventionalist policy you'd recommend? Is this an antitrust problem, a copyright problem, a transparency problem? Is it something totally new that we haven't encountered yet? And what do you think the timeline is for addressing it? Because I do feel like these moments are very fleeting when they arise. Maybe we're still probably in that window, but I doubt it's going to last much longer. What What would be your policy prescription? I think at least in the paper you know where we don't go into policy proposals because you know the main the main goal of the paper is to provide evidence and based on the evidence I I don't know if we have an optimal sort of legal remedy but to be honest like you know based on some of the feedback on the paper you know based on this conversation as well we are thinking of writing up a separate piece just trying to sort of speculate about okay what might this evidence imply for different types of

interventions so you know let me take a crack at that speculation I think maybe maybe to begin with of course the DOJ or the FTC they haven't done much right now and they might you know they might not have the appetite but let's say in in England the the CMA has you know has introduced like an opt out policy where you know and I think that's from an antitrust or competition policy perspective where news publishers can opt out of being featured in the AI overviews, but that doesn't automatically lead to that exclusion from the organic search results based on our evidence. I don't know if that's the right approach because at the end of the day, the downstream publishers might still not be getting access to the traffic, right? And on top of that, if you know, let's say we head in in the direction of Google signing licensing

deals, then you know, news outlets lose out on that revenue as well. So, you know, I think transparency matters in general. Choice to an extent matters but we need to come up with a solution that potentially compensates publishers for providing content which form the basis of these AI overviews but also I think you know we sort of touched upon this earlier you know you might get compensated but if you don't get access to the traffic then it's very hard to build a brand to convert users into subscribers so going Back to the platform design conversation. Is there a way to make these links more salient such that some of the traffic still does go through to publishers so that they can build their brand? They know where you know they have data on the traffic so that they can optimize their content offerings while at the same time they

get compensated for providing the content that powers these AI overviews. I think maybe a combination of that would be the ideal scenario in my mind. That said, this is not the first time we've seen a legal back and forth between Google and downstream publishers. We've seen this with Google News like about a decade ago. And at that time I don't think that Google really wanted to pay news outlets for you know the tiny excerpts that they had along with the links on Google News. But I think this context is fundamentally different and I think everyone is recognizing that given the fact that we already have licensing deals between Google and and different publishers. So you know again the main point that we sort of highlight through the paper is that there is a diversion of traffic and that could have implications for you know antitrust copyright but we don't

have an exact sort of legal remedy but we we do hope to speculate about this in a separate piece.
Well please ping me when that is live. I would love to read it. I'll happily share it. Ana Sahar this was fantastic. I really appreciate you taking the time and I appreciate you. I appreciate you going through the effort to collect this data because I know this is not an easy thing to do, but this is like the time to be interrogating these questions because if you wait five, six years as a lot of times, you know, I I love h having academics on the podcast. I love reading academic papers. I summarize academic papers at least once a week in my socials, but academics tend to not be right at the forefront of these issues. they tend to to to wait a little bit, you know, before they dive in, which is totally understandable. But I I appreciate that you've published this now because this is the right time to be considering these questions. Thank you very much. Thank you for sharing your wisdom and your insights.
Thank you for having us on.