The Measurement Crisis (in mobile UA) | #1 of 5 in State of Measurement Series

hello everyone and welcome Joseph Kim and Chris Hoy from game makers here to introduce a new series on mobile user acquisition the series is called state of measurement today in a privacy first world and essentially the mobile user acquisition landscape has been going through a lot of changes with potentially huge impacts to game marketing and the success of games on mobile and Chris before we go ahead and jump into our first episode I thought it'd be great if you could explain to the audience why this topic is so critical and also some of the topics that we will cover as part of the series yeah thanks Joe and pleasure to be here and bring this content to you the audience we are seeing serious impacts around privacy and the ability to measure in a post-device ID kind of fingerprinting world so we wanted to put together a five-part series helping advertisers understand the measurement crisis and the different tools and measurement signals that will be available to you so let me break the five-part series down for you so you can

get excited about this education and information the first podcast in the series will be the overview of the measurement crisis the second podcast will be the real state of last touch and the real value that last touch is bringing you the third podcast will be an education on an incrementality testing and how it can be leveraged in a privacy first world the fourth one will be the promise of MTA 2.0 and data clean rooms so you understand some of the more fringe benefits of tools that are coming to you that may impact last touch and the last bit is just a kind of a summary of those things as a roadmap to measurement success helping you the advertiser take all that you've been educated on turn it into action and succeed so we're really excited to cover all this with you and think it'll be highly impactful to your ability to win in this privacy first world I think you've explained it really well Chris I know a lot of people in the

industry not just UA marketers but EXA product managers should learn this and with that we'll dive right into the first episode starting right now today's episode is sponsored by data. to access estimates rankings downloads Revenue usage or engagement for millions of apps on the App Store and Google Play sign up for data. all right hello everybody we are here with three experts on user acquisition and we're going to be about something I think is very relevant to Mobile gaming success and there has been a lot of speculation about user acquisition how effective distribution and marketing on the App Stores has been post idfa deprecation and on Google what will happen post GID or the Google advertising ID deprecation further we also don't know what will happen in a post fingerprinting deprecation environment are we there yet is that happening so

as far as I know ID despite idfa deprecation a number of marketing channels still have goal-based sort of user acquisition campaigns available to marketers we'll find out when we speak to these experts here so I believe you can still do specific roas targeting other forms of event value based UA campaign optimization even post idfa or perhaps not I'll ask you guys so today to speak to the current and future state of measurement the ability to attribute installs we have brought together first Andrew kavado founder and managing director of growth by science next we've got Warren Woodward Chief growth officer at uptick uh growth marketing services company and finally John Wright VP of publishing from leading hyper casual game Studio qual welcome guys I was just I was telling you guys I'm so excited to have you guys on because this is a very important topic one that I think there's a lot of confusion in the industry about

one I'm confused about and so it'd be great to like really go deeply into what the current state of the market is what the future is but I thought we could start by just kind of having a recap of what got us here why do we have confusion about the ability to attribute installs to do measurement what is a current state of measurement when it comes to things like you know Apple's privacy policy what's what is Google doing GDP all that kind of stuff maybe we could just open it up to you guys to to kind of set the stage in context for where we are today I'll give a quick opinion I know this is maybe not a good one to go first because it's pretty contrarian but I think measurement and attribution has been broken for a long time but we've had access to data and and we've developed an entire ecosystem and series of ecosystems around the subsession of user level data stitching together paths to conversion in Mobile but even Beyond

mobile as well and and unfortunately that's actually T taken us down a really bad path and and so when when all of the changes that have happened over the last few years and I'll just highlight them very quickly we've had regulatory changes at various states in the US in South America and Europe and then we've had OS changes so te you know starting with apple of course restricting access to user identifiers Ida in particular that what I thought would happen would be a catalyst for changing of the broken path to conversion Paradigm and it's not happened and so what we've had instead is is this this explosion of Technologies and methodologies that are trying to circumvent the restrictions of access to that user level data and so I think that's kind of taken the industry for a little bit of a loop and and people are kind of grasping at straws and sort of trying to recreate this keep the dream alive of what I call Post exposure attribution right so so my my thesis is that stuff has been broken now it's

it's hard to get a data and it's still broken but Andrew maybe I could ask you though when you say it was broken before what part of it was broken yeah great great Point what was broken before about it is that any kind of postexposure attribution is missing a causal impact of uh of what are the ads doing to drive a conversion that would not have otherwise occurred and there's there's a a light you know I would say that that it's not it's totally 100% broken but it's broken enough that if you if you go into that into that cycle and that feedback loop of something happens after a user sees an ad and therefore that that that ad gets credit that that you you end up leaving a lot of dollars on the table lot of conversion on the table and you're paying what I like to call an adte tax essentially paying for conversions that you would have gotten anyway but it looks like your marketing is driving that and so there's this whole just just non way of doing

marketing that that has been supported by the ability to look at data at a very granular level and now that's been broken by a lot of the privacy and Regulatory changes that we've seen U but people are still trying to go back to that so what I'm advocating for is let's shift that and let's actually use this as an opportunity to step outside that box okay and Andrew and I I'll hand it over to you Warren but just for our audience to to explain if I'm understanding you correctly Andrew what you mean is that even before you might see multiple ads on different platforms and there may have been a user that was going to install a game or something anyway and so it's not 100 it wasn't even before it wasn't 100% clear what caused an install or a conversion event or things like that and so even before even when you had like the user level information like the initial exposure to conversion that part was not 100% science is that what you that's correct I'd say a quick summary before I turn it to war is it used to be not accurate but

very precise and now it is not accurate and not precise okay Warren yeah first of all great to be here with you guys so Andrew as as you're describing that the phrase that comes to mind to me is paraphrasing but the the colloquialism of show me the incentive and I'll show you the outcome and one thing so like our our team here at uptick like I come from you know working at game Publishers running internal UA teams now we have you know our teamate uptic where we we help these teams and we build software to solve some of these problems but we've encountered many times where we'll meet a team or some somehow digging into the data of a project and we discover one of the one of the many examples of the things that you're talking about I'll give two examples right here like one is when there's clearly some sort of impression you know click spamming impression spamming some something that's basically over crediting an ad Network another very common one is is Apple search where we all know that you know that's that's poaching some percentage of users that would be there

anyway no one in the kind of chain of order really owns has an aligned incentive to push against these things other than a CEO or founder who doesn't have the time to get into that level of granularity so it's come up time and time again where like interacting with with teams actually doing growth where it's like hey the the cpis actually aren't this low or this row actually isn't this good here's why and it's kind of like yeah let's not talk about that because it makes my numbers better you know it's like they're being judged by like Hey how well are you hitting this kpi the these broken parts of the attribution system you know encourage bad behavior and and more of like it helps them achieve these kpis and if you look throughout the org there's there's I I think it's it's very fundamental to part of why mobile free to play is doing poorly right now is there's not in a typical company structure someone who owns the incentive to fight against these

things great points yeah totally I think it's worse from the ad platform side too I I think the point you're making is a point that a lot of Executives don't understand and I think it's it's such a not well understood issue that I think is absolutely critical if you really want to be successful and and profitable but John do do you have anything to add here or I would say I'm not going to talk as smartly as the guys just did because I'm just a lowly publishing person just trying to make his way in the gaming world but what I would say guys is that you know I I was in desktop before mobile I'm you know I'm a certain age of of a person with a certain career and in you know my opinion is like last click is just you know in 2024 are we really still working in such a fashion you know like in desktop 15 years ago we were talking about like the bathtub model right and how the person who gave the first click would get half and how the the one at the end would get a half

and I I it still sort of stumps me to this day that the predominance of the industry is still working such a fashion and I think it goes back to what you both just said in terms of there's no incentive for a lot of these ad networks to change and I think it's quite unfair to a lot of the people that are influencing the start of the funnel which are not getting rewarded in in such a way right so we've had all these changes as you guys have outlined at least at a high level and the the that I'm confused about right now is I'm hearing kind of different stories depending on who you are so some people have said you know roaz has decreased you know payback periods have increased they're seeing performance degrade by 25 to 30% I'm hearing from other folks that there has been no change to us from idfa deprecation and to to some I think to some degree I think War the issue that you talked about might be the culprit where people were doing their models and

they just kind of got exposed and was it idfa deprecation or was it your freaking aggressive model that actually was not you know what I'm saying but but I want to hear from you guys and John I think it's great that you're here as as a publisher seeing performance on games and things like that but in your view what has been the actual impact of IDF deprecation and these other things to the current state of the UA marketing landscape today there's there's two narratives going on at at the same time so if you talk to any individual company particularly the larger they are they'll say that they have a strategy they have a great data team great U UA team and that you know they've got they've got iOS solved but then if you zoom out and look at the macro and the trends in our industry the massive layoffs just anecdotally how many great growth Marketing in UA people that were like so in demand for so much of their careers that are now just hustling unable to get full-time jobs

and it's very clear that that part of the industry uh of of of doing profitable straightforward user acquisition for H half of the ecosystem on iOS it's very clear that it's not solved for the industry at large but there's a lot of emperor's new cloes where none of the none of the big companies will outwardly admit that they have problems with this but then you see it reflected in their actions um and there's been very little adaption of the industry for standardized solutions to to address this and again it's it's kind of that lack lack of incent aligned incentives for it yeah from my side guys I would say that we're like I'm definitely seeing it holistically across the space that you know ecpm have been affected right and I think there is a part of it which is like certain genres are hit harder than others but I think you can clearly see from the evolution of from hyper casual to hybrid casual and the fact that a lot of these Studios are progressing to you know larger more

robust more complicated games which have like meta and deeper engagements that you know need a higher LTV these companies wouldn't have you know been forced to make these changes and you wouldn't be seeing a lot of these Studios going under if you know the uh Golden Days of of you know getting xltv and a low CPI would have been continuing right and and John maybe it's important to just kind of spell that out I think you're you're probably more of an expert at the ad monetization side than I am but can you can just people that aren't in the UA or admon weeds like how do the things that we talked about directly correlate to you making less of the ads or any anyone running in-game ads making less on the ads in the games that ecpm being lower yeah sure so I think it's around the the quality of the actual conversion right so the standardized metrics for for this stuff on the creative side would be your CTR CVR which Powers IPM install per th000 mil and depending on your CPI CPI being cost per install times

installs per thousand give you your your total ecpm so ultimately what's what's generally happened is the there's one part of it which is the suppli has kind of been affected by the droing ecpm which has gone from a Dro in user quality so there is like a correlating factor of you know the ecms have gone down which means my LTV and my arpu my average revenue per user is also taken a hit which means that now you know where you could do say an install for 50 cents and you made 60 cents that same install is now costing you 70 cents or 80 cents which means that the acquisition is unprofitable this has had sort of a KnockOn effect in terms of the amount of Supply because there's less new games launching there's less you know actual um inventory to be to acquire and we all know when Supply goes down cost goes up and this has an effect on that buying power and I think very interestingly I don't know if you uh listened to the deconstruct fund podcast with the CEO of

C playay but they went from spending three million a month to 500k after idfa so there's no way that that isn't a corresponding factor of deprecation okay so well I got I got a I got a quick quick thing to to jump in and offer I don't disagree with any of that but I will I will offer maybe a different dimension to it so I've seen a lot of data that supports this which is yes if you look at say platform level uh CPI metrics or whatever and you look at and you CH chart that over time that you know Trends over time and you look at when I when idfa deprecated when iOS 14 launched you can see an absolute Spike now however when you jux toose that against an incrementality based CPI do anyone want to guess that that line is absolutely flat so so to me that that says two things one it's it's not necessarily a performance issue it's a perceived performance issue because the measure measurements impacted by the

idfa deprecation right you get less matches you get less High Fidelity matches it looks like fewer things are happening and so you know for the same spend you get less attributed installs right and so therefore the CPI goes up but the other thing that that it it tells me and that at least to hints at strongly is that goal-based bidding does not work from an incrementality perspective because if it did right you would you would assume that the incrementality would would decrease and the and the cost for increment install would increase after iOS 14 but it did not and therefore that that level of performance does not actually impact incremental installs incremental conversions right and so I think Andrew if I could just ask you to pause real quick and for our audience because we we don't have all UA experts and I I'd love to have this conversation be as accessible as possible so when you talk about incrementality and I I think people kind of get the notion of like goal-based targets but in terms of incrementality if you could just kind of quickly explain what you mean yeah absolutely sure so incrementality refers

to essentially installs conversions whatever that you would achieve or attain as a result direct result of those ads so they're incremental to some base level of of organic installs right these also referred to as causal installs usually they're measured by something like a holdout test right where you've got ads live in certain geographies or audiio segments or whatever and you've got a statistically comparable audience that you do not show ads to and you look at subsequent behavior of those two respective cohorts and when you do that the delta in conversions installs whatever that you that you realize is the incrementality the incremental installs now that differs from what you will see in a platform which would be a post exposure install last touch or u-shaped or whatever attribution model but but that th those two metrics are are very different and and so what I'm describing here is that when you you don't actually don't really care if you're doing Geo testing for example you don't really

care about idfa disappearing it doesn't impact necessarily your ability to do those incrementality tests but it obviously does impact the post exposure way of measuring and so the fact that Trend that I described where you see the CPI Spike after iOS 14 is indicative of it being a measurement issue a performance a perceived performance issue and not a true performance issue which is measured by the incrementality way right so if if I could simplify that what you might be saying please correct me if I'm wrong is that certain games might have a base level of organic installs and when you do like paid marketing campaigns you would attribute some installs to those paid campaigns some of some of them which are from those paid campaigns but because of like the inability to measure you're not some of that organic Baseline may be attributed to the paid or may or maybe they're not attributed enough and that inability to exactly attribute those insults to the paid campaigns is where there may be some confusion in terms of how you measure is that what you're saying Andrew almost yeah almost I would

say it's more that that when you've got a level of installs yeah because of the way that goal-based bidding works and because of the way that post exposure attribution works it looks like you like the ads are driving more installs and they're actually driving and so when you when you lose idfa and you lose the ability to connect uh uh conversions or installs to an ad exposure it it looks as though you're spending the same amount but getting fewer installs but actually all that's happening is fewer installs are getting Associated to the ad spend but if you if you take that and and put off the side and you measure the exact same situation with an incrementality approach you will typically see that there's not a lot of incrementality and that that that level of incrementality does not match to either the pre or post idfa level of of attributed installs and so the point is the point is that's what you know the in the first kind of diet tribe I said here where like measurement was broken even before iOS 14 that's that's what that's

referring to and so the the basically if advertisers if more advertisers focused on incrementality they wouldn't care as much about iOS 14 Andrew I've got a ran off pasted here a little bit John I'm sorry but like why do you think then and this is to you as well Warren but yeah why do you think that people are not focusing on this more like why are not more game companies and and businesses in my position working on incrementality model and why hasn't this you know adopted by the the masses I mean I've definitely got thoughts here but moreen as as somebody who's in a hot seat maybe you could uh tell us what you think about that yeah so I think a lot of what what you just said Andrew we see in you know conversations we have with with gaming companies and you know in our in our own data which to paraphrase like yes the CPI directly attributed to Facebook shoots up but your ECPI uh it it stays flat relatively on

on iOS the thing I would push back well I guess I would half agree is that you know goal-based buying doesn't work I would say it absolutely doesn't work with the existing tools out of the box you get from the MMP but to actually do profitable UA you absolutely must find a way to do something that proxies that and that can maybe touch on to to your point John like nothing to say nothing about the value I think plays a huge role in overall measurement but but our team took a different approach to solve the same problem and I just want to kind of like talk it through as another lens and why we did a different approach so what we saw is just like the data needs to be normalized that's like a huge part of what we're talking about like Android data is still aens basically normalized when you look at all the channels side by side to each other but in iOS just to give like three examples Apple search uses its own logic which is not it's not scan it's not fingerprinting they just have their own rules and then we have all of the networks

that still do fingerprinting where it's close to one: one although we find there's about a 30% variance which is you know still huge and then we have the last bucket which is driven primarily by uh Facebook where we see about one out of seven users opt in so you're only seeing about one seventh of the Revenue data and then Google ads lastly is the most extreme because YouTube doesn't even ask for consent for that that pocket of inventory so we see something like one to 100 to one out to three 00 I iOS Google ads users op in so that's all to say if you look at all of your channels IOS and Android next to each other and are just trying to see Raz you've got at least four different models floating around and it's completely unnormalized and this is kind of the Gap in the stack for us we had to solve this so like we built the system for this over two years ago because like we only get brought in if we can drive incremental profit from UA so when these changes happen we're like do we still have a business and we scrambled and tried different things what we landed on was approach that I would not

say is like contrary to incrementality if anything maybe it's supplementary but what we do is what I describe is dynamic polling so we use scan installs as a source of Truth for CPI and install count and then we normalize every channel so if if we already have most covered by probabilistic we'll basically DD that against the uh the scan installs but where it gets more interesting is Facebook for example so in that example that described earlier one out of seven users consent to be tracked at a high level what we're doing is for every campaign segment if you're a user that agreed to be tracked we're surveying you and applying your LTV curve to the rest of the segment so that lets us like still use goal-based systems but normalize them for the amount of op that we have in as individual segments because it is surve when you say segment when you say segment do you mean part of the same UA campaign or as part of a daily cohort how do you define both so we Bas we we as we've built out our our software for this we've kind of added layers to it so it'll basically kind of

roll up to the next subset where it has enough data so if like individual campaign has an accurate amount of data it'll model at that level if there's not enough there it will model at the the the ratio of the like country Plus ad Network plus OS got it okay and maybe just for our audience so I I think Warren you've done a great job of kind of giving us a high level overview of how people are trying to do measurement to the best of their ability given that these privacy changes broke what andreww you're calling the post exposure attribution system and just for our audience so basically what Warren's talking about and correct me if I'm wrong guys but because of Ida deprecation and the previous ability to get user level device level data and to attribute actions and behavior to users now they're doing all these things that Warren's talking about these four different method methodologies fifth including Warren's Dynamic what he's calling Dynamic polling system which I think is is probably unique to your company perhaps Warren but now maybe we

can maybe if you guys could also talk about why is this a big issue or is it because you can do these four or five different approaches how big of a deal is Ida deprecation are all a pain in the ass first it's like like doing it accurate it's a huge pain in the ass that that we like the amount of engineering time we had to do to productize the thing that that we do is pain in the ass like I think that's that's number one at the most basic level I mean Andrew what do you think why do you why do you think there's still the default is that most teams don't have systems for this yeah yeah I think it's interesting that you say that you're that it impacted the I feel like that's that's one of the things depending on how you're building it that that could be pretty insulated against it you know yeah I should clarify I think these are kind of independent so I think our Focus was more as like hey we still operationally need to make like op optimizations every day gr level but yeah yeah to be clear it's not like I

don't see it as either or yeah I I mean I think I think it does like what it has done at least from my vantage point is is it's made what what UA folks have done for like a decade or more it's made that virtually impossible right like you used to be able to go in and and look at all this granular data and and make tweaks kind of on the Fly do really interesting cohorting and segments and and all that you could trust that it would SN across all of the different ecosystem pieces and now that that doesn't work just like you described Warren it's you know everybody's using their own system and that the data is a you know it's it's asymmetric and all of that and so this is this is kind of going back to to you know my obsession of incrementality why I think moving to that approach which is agnostic to all of this is a lot better and and it's not it's certainly not easy to do that it does require a lot of Paradigm shifting right like like if you're looking at and just using metric Works Joe you

obviously you're familiar with these guys right as an example so they're you know for for folks that aren't familiar they're what I call a triangulation system of measurement for for in the mobile space right so they they have they've got ground truth incrementality and they still rely on on where available attributed data to to split things up so putting all that together you get a single metric that at the end of the day is calibrated incrementality so so something like that is is an awesome way for for ad iers to really start to lean into into that that style of measurement but what it what it you know any any kind of system that that relies on that what it's not really going to be it's not going to enable is like that micro tweaking that that I think a lot of UA folks have have relied on in the past to to to to look at like what that you know how that's going to change sort of an attributed outcome my feeling is that a lot of that wasn't adding a ton of value and and at least the data that I have seen that is is causal that is incremental has proven that but I mean there's there's whole

organizations and politics and compensation and bonuses and funding that's all based on that and so to to shift the industry so rap rapidly you know it's it's just not that easy to do and that's why you know Warren you pointed out earlier right there's there's you know layoffs and big changes and I think that is a symptom of the industry realizing that there has to be a shift but we haven't quite landed on what that is and maybe one One path is is some of these AI type campaigns I don't necessarily think so but maybe could be just you know taking a taking getting out of the weeds a little bit leaning into some of the inherent fuzziness and distribution and statistical nature of these more you know statistical type measurement techniques but in any case there has to be a shift and we just haven't fully gone through that shift as a as a UA industry yet yeah yeah so go ahead sorry John I was just about to say and I'll hand it over to you though it kind of sounds like there isn't a perceived best practice in terms of how how to do this

measurement stuff right so Warren you're using like a unique proprietary Dynamic polling system Andrew you're kind of advocating for like this incrementality m triangulation approach John I'm wondering what you do and what are your thoughts on this yeah I mean look guys if I'm honest I think the both what Warren and and Andrew are doing is is much more complex than what we are doing and what the most of the gaming sort of companies are doing right now I find what you're saying is is ultimately very very interesting guys I think for the most part again most companies are still working in the traditional fashion with the same mmps right the same names that we're used to working in the same ways in terms of like last click and there has obviously been you know dramatic shifts in the in the spending behavior and how we invest money in UA you know traditionally two years ago or over two years ago we would probably spend 70% of our budgets on iOS right and 30% on Android now we're spending 50/50 because again it has become much more difficult

and this is you know something that is across the entire board it's not just like certain channels that have been affected it was like the whole thing and you know we traditionally always did sort of like Source level optimization right so maybe some companies were very technical and did user level stuff and retargeting through the dsps ETC but a lot of the time it was Source level bidding which was going to a particular you know subsource or a supply source and adjusting the CPI based on the actual Rass of that Source right and effectively what this does is increase the averages of the campaigns or or for that game because you're bidding on better quality from this source that you know has better users versus a lower Source depending on gen Etc so all of that was done very very manually until a couple of years ago now there are some roas Optimizer tools in the likes of app loving and iron source and these companies have built this stuff but honestly and and to be BR yeah to be a bit brutal here I think a lot of people just don't know what the

going on you know we're learning every day we're trying to Pivot we're trying to do as much as we can but it's it's it's tough out there right like and I feel every company is feeling it right now and everyone's trying to do smart stuff we're trying to do smart stuff we're updating our models all the time you know and and we're winning in certain aspects I'm very fortunate that qual is a growing company right rather than one that's staying flat but it it's tough out there really right but John I think like to some degree your ability to continue to leverage existing ad networks whether it's a app l or Unity or some of these other companies my understanding correct me if I'm wrong is because these companies are using fingerprinting technology to still be able to get some level of attribution now when app so the the rumor is that Apple's going to crack down on fingerprinting and so I'm wondering from your perspective is the three of you guys what is going to be the impact of this because I'm hearing

different things oh no like app 11's going to be fine they're going to do they're going to work around do some other stuff yeah or the fingerprinting Crackdown is not actually gonna happen so I'm wondering kind of a two-part question to you guys what do you think will be the impact of whatever Apple does in a post fingerprinting deprecation world if that happens and then what can ad networks do to respond to this in some way will there be some new technology something with it SDK something with something else or are we pretty much just going to be in a world where we're really guessing when it comes to user acquisition um I'm yeah I was just going to say guys I think you you both will have much bigger points on this than me but from my perspective I think Apple do what they say right so I'm not looking I don't think you know and based on privacy and based on everything else that they've been doing over the last two years I think we are moving to a fingerprint less we right I if I'll be shocked if

anything changes I think they're going to they're going to get rid of it that's my personal opinion and I think obviously there's been I'm at Games 4 in Barcelona and there's been a lot of talks on this the last couple of days and essentially yeah a lot of people are saying that you know the the networks themselves will think figure out something smart because that's why they billion dollar companies and they employ all of these amazing engineers and if that's some sort of server to server set up or if that's some sort of something off the back of an SDK that's already indic integrated I I really don't know but I think categorically it's going to be bad for a lot of us and I really hope that either the networks themselves or a very smart Tech provider out there comes up with a solution because otherwise I think maybe there'll be more companies going out of business so yeah that's my take yeah to time in here John you you you alluded earlier it's just like it's all kind of a mess everyone's just is trying to do their best in the situation we're very intentionally focusing on the

known like when when there's rumors of like how a new version of scan is going to function or rumors of what's coming for Android we devote like literal zero Cycles to developing for that until we see how the data comes in how the ad networks are reacting to it but then it's like all hands on deck and it's just there's it there's such a gap in what people are doing today with the data and tools available um versus like what you can do that it's not even worth starting to think about these like what if or like I better like design this thing in case like you this happens with with with Android that that's kind of like our personal our personal approach to it it's like there's so much room for improvement with just the data as it flows today and like you know what what the current version of of scan is that we need to focus on solving those problems just to have some kind of competitive Advantage yeah I mean look fingerprinting is I think it is eventually going to go away yes so I do

believe that it's it's going to get crushed my opinion on on generally how to approach this evolving landscape of data is if you you can do one of two things so there's you know Warren has has one approach which is if you can do this and if you have the right resources to manage for the current situation and as things pivot you pivot right away I I don't think a lot of companies are really as well organized to be able to do that out there and so I would take and I've always advocated for for for companies that can't you know be as Nimble as that to to think where is the ball going to be you know three four months from now five six 12 months from now whatever and and my opinion is that all of this user level data will eventually go to zero or close close enough to it to effectively be zero so you should be operating in with that mindset and that doesn't mean necessarily ignore anything that you have at your disposal right now but assume that whatever you have is ephemeral it's not going to be around that that you might have to operate in a world eventually where there is no data

and plan and plan for that and so just as an example of that you know when I when I work with my growth marketing clients we we don't install any sdks we don't install any pixels we don't do anything like that all we do is we look at events that we can track from a first-party perspective from a website or app or whatever and we look at ad spend data and we design sophisticated causal learning agendas with incrementality with and and we buy only on signals that that in in aggregate so that means managing to reach in frequency that means relying on first-party targeting that we know are native to whatever platforms or networks we're we're we're buying ads on and and that means using a feedback loop of of really just with the incrementality testing and when you do it that way you get two things one you you start to understand just how truly difficult it is to get Roi positive incrementality results but you can do it and it just takes some time you can get there and then two just how how far apart any

platform metric so in the cases where we do have platform metrics still when you get like oh wow our incrementality you know numbers look great and we're profitable and then you look at the platform metrics they're like so unprofitable looking and you're just shocked and I think when you when you finally get that that's when I've seen growth marketers have the real aha moment it's like wow this shit's really really wrong and and it doesn't work and so what I'm hoping is that as more of this data dies more and more advertisers and growth marketers go towards that that more scientific approach and and start to see that that light Andrew I I think so much of that is is is true one one thing I'd love you to expand on because I think you're closer and more an expert in this area than than I am is like as or some sort of incrementality is becoming more a necess necessary like where are we today as far as like the limitations of that so like historically what I've what I've at least perceived as the limitations is like just time Horizon and amount of

historical data so like the there's really important edge cases at least in that historical limitation which is like cool I launched a game yesterday and needed to decide what moves to make to you be like very extreme on one end but like where where are we kind of at with sort of the evolution of these types of models and the operationalizing of them and what are still the current limitations to them like when when did they start adding value and and what are the the limitations before that can kick in great great question I do think there are limitations and I'm not suggesting that mmm are perfect by any s sense I think Mmm have come a long way in that you can update them more frequently you can often use less data it really depends on on you know the way that it's modeled the way that it's put together but if you think of like your Granddaddy's mmm right it was I took two or three years worth of data it took a year to build and it was primarily for big cpgs that had to wait for a lot of data to come in often was delayed by months or quarters before they could even do it it was a more of a retrospective thing a lot of the modern

ones are are more forward-looking they're more predictive they use a like a basically a type of methodology that you can calibrate with you know incrementality results or or other types of say known outcomes that you can you can plug into them and calibrate them and so they definitely have come a long way but the reality of it is even like the the the measurement style that I described it's got its it's got its blind spots and it's got its in it's it's sort of like lack of perfection right what I'm what I suggest and sort of my my perspective on it is that you still have that same the same you know blind spots and taxs uh if you don't use that and if you use something more granular it's just different and so I would my my assumption and sort of my my position is that the trade-offs of using a scientific based you know incrementality plus plus you know plus attribution combined has less EX extreme tradeoffs and less extreme of a measurement tax if you will than staying

in the in the old school way I just think people are so used to that that they don't quite they haven't quite gred that and they're they'd rather be able to see the data right away be able to act on it right away and and be able to see like a single tangible number that they can rate as either good or bad versus maybe having adding fuzziness around all that but being a lot more accurate in terms of the ROI and so that's really like as in a terms of a mental model what marketers need to get their not just marketers by the way but CFOs that are that are putting budgets on that are looking at you know reports from Marketing in UA and saying well you know I don't understand you I don't understand these like distributions and confidence intervals and all that so there is there is I think a broader education that's required but to your point it's not perfect there are there are fuzziness around it and it's just a matter of of understanding what that is and and what type of fuzziness do you prefer right and Andrew you kind of paint a pretty bleak portrait of the world coming up in a post finger printless kind of world but maybe I can

ask Warren and John so in this world that Andrew paints and I I think user data doesn't go to zero right because at least you have like some optin and and Warren I assume your Dynamic polling methodology would still work in this case but John and Warren what do you guys do Post you know the deprecation of fingerprinting um Warren you want to start or yeah I mean I I can start here so yeah I'm going to contradict myself a little bit here I mentioned earlier you know we're really focused on play play the field like you know where where things lay but at the same time we're very mindful of as we productize things make them generic and we've already had this happen several times where the attribution Logic for a given network will change and the nice thing with how we built it it's a generic enough system so it's just even even for you know an extreme sample like like Facebook Facebook as we saw the opt-in go down with adoption the model just updates in time and it's just you know you're

surveying a smaller percentage of the population so of course that that with with any surveying methodology you know it's going to decrease accurately but I think that's that's the way we do try to adapt is just say okay well what what do we know we know that we'll probably have this data in decreasing frequency in the future and in high variant so how can we make sure our tools like are built with that in mind and not for like the explicit setup that we have today yeah what about you John yeah no I I I completely agree Warren I think look JK I think all of us have been in this space for a long time we've seen a lot of adversity and we're all still here y right and you know hits the fan very often in this industry my friend and and like I said we still there's still billion dollar companies out there and there's still new business happening every day and new tech being built all the time and okay I'm an optimist I think this is another hurdle that we need to get through right as a as a collective as an industry but we are still a solid ecosystem we're still very

very smart people in the space and I think maybe in the long term this will be a good thing like what a lot of the stuff that Andrew and Warren have spoken about today is is really smart stuff stuff that I'm even only barely learning about now and if this is a forcing agent that creates a much fairer we then all of this trouble going through now in a year or in two years or in five years will just be for the benefit of all of us so like I said I think new product coming to Market new ways of working smart people figuring out that is you know where we're going to be so so look I'm whilst the immediate future might be a little bit Bleak I I'm still you know I've been in this game a long time and I'm still here so I don't think this is gonna be the thing that kills me yet John you're filling me with so much hope you with with for anyone watching on video there's an Angelic glow from the light above John and as he's giving this inspirational speech about how this is going to bring us all to a better space I can't help but be filled with hope I love it I'm a follower I love it

John I totally agree with you man that's that's it's a it's a great perspective I love the optimism but maybe we could also just as a question to you guys in this new world that we're facing what in terms of like who could be impacted just for just thinking strategically and for people to understand I I would think that companies that use fingerprinting for their user acquisition campaigns like an app l or a Unity some of these ad networks would be impacted maybe game studios with long payback periods with a lot of uncertainty in terms of whether the amount that they spend will pay back or not but can you guys comment quickly on like what companies do you think could be the most impacted by these changes coming up yeah I mean it's who's who's not I mean let's go through the list like anyone building a mobile game is impacted any AD network is massively impacted all of the technology providers that provide specifically attribution providers they've all been scrambling

and don't have normalized Solutions like it's hard to find uh part of the ecosyst like the better question is like who's who's not impacted yeah exactly yeah okay all right well maybe I could ask one last question for you guys so we are where we are and there's a lot of speculation in terms of what the ad networks would do maybe there's new technology maybe this maybe not who knows but in your specific perspective in your view what does the future state of measurement look like to you guys and I I think I don't know Andrew it seems like you kind of have a specific view but I'd love to just have it be specifically articulated by each of you guys about what this looks like in the future and and maybe starting with you Andrew sure yeah I mean I've kind of spoken about it I think there's G to be a a big segment of the market that moves away from user level data almost entirely and leans into triangulation Solutions incrementality aggregate based measurement however one thing I didn't

mention but I do think will be a big part of this ecosystem is is this idea of privacy enhancing technology clean rooms and I think that will be primarily the realm of really large Enterprise organizations and power very big Partnerships um that exist with with you know companies that have a large amount of first-party user data so I think there'll be some combination of those to probably the Torso and longer tale will be more the former and the big uh the big folks will lean into that privacy enhancing technology world uh maybe John I would love to say I believe what Andrew just said sadly I think a lot more of the same guys but with just these limiting factors in place I don't I would love to say I think that you know in the next couple of years an incrementality will become in you know the primary but I still think we're a little bit away from that I think there's some other factors that need to change in the industry before we

get to that as as the primary so I think things will stay the same but different you know like there going to be there G to be a few like more minor changes than a complete colossal shift but hope look hopefully I'm wrong hopefully Andrew's right yeah I'm trying to think of how to say this without just like being overly shilly of of uptic but I mean we we put our money in our resources like where we believe this is headed as far as like what we're building for it so I'll try to answer more generically which is that I think it's extremely clear at this point you can't operate your business with the two main data sources that we had in the past which was the data you get from the ad Network and the data you get from your MMP we're going to hit this era where there has to be a data science or modeling layer that's interpreting and normalizing that data that can be operationalized to make day-to-day decisions so I think Mmm and incrementality is part of that and also the kind of things that that you know our team's focusing on that are more for like day-to-day operations but one way or another not that it has to be you

know any one provider on the market or that you have to build it yourself but you can't the systems as we've been given them do not answer the questions that we need answered today and we have not built out that next leg as an industry or agreed upon it enough but like that it's inevitable that that has to be introduced in order for companies to survive this phase right all right guys well I want to thank you guys for your time expertise inputs I I do feel like we should probably do another podcast going more specifically into in incrementality so for the audience watch out for that but thank you guys so much I've learned a lot and for audience we'll catch you next time thanks guys thanks everybody thanks [Music] guys [Music]