S07, E33: An oral history of mobile user acquisition (with Nebojsa Radovic)

AI is changing everything, but it's only as good as the signals behind it. Branch connects [music] customer interactions across paid, organic, offline, email, web, and app touch points and turns them into the trusted context you need with links and [music] attribution that capture the full user journey. Learn more at branch.io. And while you're there, check out [music] Branch's AI search and discovery report covering insights from more than 300 enterprise marketing, growth, and digital leaders to understand how the industry is responding to the rise of AI search. [music] That's branch.io.
The problem is that the distinction needs to be drawn [music] between the competence of the economists and the correctness of their analysis.
Welcome to the Mobile Dev Memo podcast. I'm your host, Eric Suffort, and I'm very excited to be joined today by Nebo Reovvitz. Neibo, welcome to the podcast.

It's been a long time coming.
Yeah, it's such an honor to be here. We've known each other for over a decade. We work together and I'm really glad to spend some time with you and talk about the industry and the recent trends.
So, I'll let Nebo introduce himself, but I guess the way I would describe him is like a grizzled veteran mobile gaming user acquisitionist, mobile gaming UA marketer. He's been in gaming in mobile gaming since basically the dawn of of the category, worked at some of the most, I would say, cutting edge mobile gaming studios in terms of the way that they approached UA and is maybe like the house MD of mobile UA just seen everything I would imagine at this point. But we finally got the green light to get you to come on the podcast. So I just introduced you, but why don't you introduce yourself to the audience in your own words?
Yeah, I mean at this point everyone knows me as Nebo. I moved to the US in I studied computer science back in like back in Belgrade, Serbia where I'm originally from. I moved to the US 2012. Started my career in mobile gaming 2013

and I've been in mobile gaming ever since. Worked at like all kinds of different companies of all kinds of different sizes. Started my career at Nordius which got acquired by Take 2. Continued my career at Machine Zone which was likely the largest advertiser, a mobile gaming advertiser at the time. then continued network where the two of us had a chance to work together had a wonderful time over there and I now for the last six years almost have been bazinga so uh developers of different sizes all mobile gaming and obviously different eras of us acquisition
so I thought when I wanted to structure the podcast I was thinking about you know what would you ask someone who's just been in the space since the inception and I think one just really interesting perspective I'd like to get is like a history of mobile UA from you. So, so talk to me about mobile gaming UA in the MZ era. So, MZ, I think Yeah, I think they must have been the largest marketing spender in in the gaming category, but I think at one point they were the largest spender on Facebook. Is that right?
Could be. Yes. I mean, it was a massive like advertising operation at the time,

you know, hundreds of millions of dollars in monthly in like an annual spend. uh you know two Super Bowl commercials maybe even three like just like a a pretty remarkable point in time and like a pretty remarkable team in a very different era as well you know first of all like media buying at the time was super manual I I think it's hard for people to understand how manual things were like especially if you started your career postcoid you know we had to manually pull you know daily spend reports we had to stitch them together like there were no like reporting APIs creative reporting was pretty uh clunky at the time as well. So, so again, you know, uh when you think about user acquisition back in the in the MZ days, it was largely manual and having access to certain data and having better reporting was like a true true advantage in the market. So if you could get like again creative reporting or at the time publisher level reporting was a huge thing where we could you know like target a specific publisher let's say like a solitary game or like the number one downloaded game on the app

store that would really create like a a competitive give a give the team a competitive advantage. One thing about MZ that I don't think is has ever been talked about is that MZ really help push the ecosystem forward. So a lot of these things again reporting API creative reporting etc etc like came to like life because of MZ because MZ was very it was a big advertiser it was very pushy in terms of like asking for these different features and as a result of that I think the industry managed to capitalize and become better and kind of more sophisticated media buying compared to like today's age. One more thing is it was a really big team. I think we talked about this in person at least multiple times like it was over 100 people managing media buys which is kind of like really crazy to think about from this perspective where teams are generally pretty lean especially on the user acquisition side but on the flip side you know there were teams who are super specialized to specific channels and really felt like being part of the kind of Ustar team where you would work with the person who's like best in the

world at managing like Google or Pinterest even because we had a pretty pretty kind of like a we had a like our budget was again pretty big and we could test new channels even though that meant that those channels would not be immediately profitable.
Talk to me about the structure. So how is that organized? So you'd have channel specific people. So that person who's best in the world at Pinterest and then you'd have that channel specific person for essentially every channel. So I would imagine some channels had had more people than others. So Facebook probably had a number of people on that team. Is it did you have you had like sort of channel specific pods and then how did that all coalesce in terms of like reporting and then in terms of like managing those pods and then where did this all flow up into?
Yeah, I mean the teams like the Google team and the Facebook team were pretty big. I would say they had anywhere between five and 10 people managing Facebook and Google. I think it's also important to remember this is pre UAC pre-advantage plus era. So you know it was pretty manual on Google. You had web specific campaigns, keyword specific

campaigns, then like YouTube specific campaigns when YouTube became a thing on the advertising side. So, so like highly specialized teams, highly customized reporting because there was no like you know singular was a thing at the time. You could get like you could do cost aggregation, you use like tools for cost aggregation, but to get this kind of pub level or like placement level reporting from Facebook, from Google, you had to build a customized report. And that was like a real advantage because most small advertiser at the time a they didn't know about it, b they didn't have access to it because again it wasn't like accessible to everyone. That's why like having such a big team helped so much at the time. So like MC could stay ahead of the curve in terms of this kind of like like it maybe it's it's interesting to share what I did over there. I I focused almost solely on DSPs. So your programmatic buys and the thing is when you are such a big advertiser and you work with multiple DSPs and you have pub level access reporting pub level reporting across the board you can see what are the most efficient ways of like

buying from a certain publisher from a certain app and like that's even nowadays it's like it's very kind of you know sci-fi because we don't really get this level of insight even today but at the time because MC was such a big because he had so many people who could like build custom reporting and focus on things such is like what is the most efficient way to buy from the solitire app. We could do things that no one else could do at the time. So sometimes having more people is actually you know helpful because you can do things at a higher level of granularity with more precision than others. Obviously that's changing now with AI but at the time there was like a real advantage because we just had access to more data and more people to actually process that data and make better user acquisition decisions.
Talk to me about that because I think that's worth going into some detail on. So when you say the best way to buy from the solitary app you're so when you say pub level you're talking about first of all you're talking about app level
that's correct publisher yeah like I'm just I know that I just I just want to make sure that it's clear so so okay so you're talking about app app level reporting but I think and we have spoken about this what you're talking about is like what much more specific than that

it's like I want to get the third impression in a session in placement X in the solitire app
the third impression on the banner placement in the solitary app in Germany on iOS 14.4.
Yeah. But like versus the second in US on Tuesdays because you a you were spending so much data,
you had you you kind of sidestepped the curse of dimensionality in a lot of ways, but also because you had all this reporting, you were able to get enough data to to essentially look at all the combinations of things and say this is when that that third impression in Germany on Tuesdays on this iOS version is going to be more profitable than the second or the first, right? which is you can understand why because maybe the first is more competitive but you had that basically across the board you were operating against that across the board
for X number of apps
that's correct yeah and the apps would be again to your point these had to be like large EU apps because you needed

enough data to make make proper decisions but like for those who don't fully understand how apps how those apps are monetized they don't use just app login for monetization they use 10 different networks there's a waterfall or there's like header bidding where they dynamically bid at the same time and maybe it's cheaper and more cost efficient to buy like an impression from Unity than Applied and then we had to figure out and you know like you kind of reverse engineer what the margins are and try to find like more efficient ways again to buy. At the time it was called supply path optimization. It's not like a term that's like thrown around much nowadays, but it was really powerful. And this also meant that we could strike like deals with publishers and maybe buy directly in case they were big enough and their inventory was like valuable to us enough. Like nowadays I don't think anyone does it especially at scale but at the time there was a very very successful strategy. And that kind of like led us to the next stage of UA which was like acquiring DSPs and building first-party DSPs where you wanted this kind of like manual logic to turn into algorithmic bidding which is

essentially what started happening after 2015. So, so SPO is not a term that gets used a lot in mobile, but go to like a market event and you'll hear that from every second speaker on the It's just that's a web
like a programmatic web concept, but that's, you know, still very still very sort of common to hear that. But talk to me like who were the people producing those reports? I mean, because MZ would hire like, you know, they had a machine learning team and they had kind of like economists, right? They had, you know, people that would do that kind of analysis and produce that kind of reporting, right? Because it was big team across the board, not just on media buying, but analytics was really strong. Data science.
Yeah. There was like a dedicated market analytics team obviously helping like the market analytics or maybe it's better to call it marketing technology team. It's not that you just had to like aggregate all the data and build reports. You also had to build tools for the team to operate better like campaign naming tools, creative naming tools or renaming tools any sort of automation

like the I believe that you know the saying at the time was you know if anything takes you more than 60 minutes should be automated. I mean people don't understand how like forward thinking this was at the time and now it's [clears throat] kind of it kind of just makes sense but because there was like a marketing technology team who built these kind of automation workflows and tooling things were we were again ahead of the curve. We're moving much faster than most of our competitors and we're able to spend significant amounts of dollars on games that were not necessarily for large audiences. These games were pretty hardcore and but we found ways to find these high value customers.
And what was the channel landscape like at the time? Like what was the dominant channel?
2015 I believe that Facebook and Google were 60ish%. So Facebook was still pretty dominant because again of of like you know it was a large app. It was the only the only kind of media mobile app first media buying solution. Uh this is preUAC where Google was still fairly manual. Again, Google was probably in the 20ish% and DSPs or programmatic buying was pretty

big. It was you know 20 25% of the overall spend. Now the reason when I say programmatic buying uh essentially all the video networks or ad networks nowadays such as epavin and unity which are the biggest ones but also liftoff that acquire bungle iron source at the time was a separate network I referred to them because you could either buy directly by working with iron source like company or you can buy programmatically through a DSP. So we bundled all of these together because it's the same type of buying to sort of like DSP or programmatic buying. Yeah.
And that was around 25%. There was some some percentage of spend went to incentivized traffic which is known as PPS right now. They kind of like they were popular at time then they disappeared and they kind of like reappeared again in the past few years. And then preloads were also like probably in the fiveish% range. Uh Digital Turbine I think Pinite at the time there was like a handful preload companies that we also partnered with closely.
What was that company that sponsored the password for the Wi-Fi at all those

events? It was always the same company and they were some kind of media channel. You know what I'm talking about? I can't remember the name, but you would go to like a mobile UA conference and they would sponsor the Wi-Fi and the password would be their company name.
Yeah.
You remember what I'm talking about?
No, I I I mean, it's been a while. I I can't remember. Maybe it was Digital Turbine. I don't know. Um,
no, it wasn't. I want to say they sponsored the Mobile Growth Summit P email or sorry, Wi-Fi passwords and maybe I don't think Well, anyway, that's that's where we first met in person, right? It was Mobile Growth B the infamous Mobile Growth Summit Berlin, right? Yeah, I think so. Like there were like these LTV Greater than CPI events.
Oh, is that where we first met?
No, you you Yeah, you're right. No, we first met in Mobile Summit Berlin 2015. That's correct. When you did the standup and and you know, we had a wonderful time.
Yeah. Okay. So, what what's gotten easier in mobile gaming UA since then? Like, so maybe just to take a big step back. So, you're describing the most technically savvy mobile gaming UA operation being something like a hundred media buyers. this

large foundation of analytics infrastructure that works and that gives edge and competitive advantage to a team or to a company that can afford that and that could support that but that's not in the best interest of like a meta or Google right because they you know there's only a handful MZ stood alone at that time as having a an operation of that size and what and what those companies would like to do
would be to provide that same level of sophistication to everybody and so they should build those tools and that that's largely what happened, right? I mean, you're saying that that was pre UAC. UAC being universal app campaigns, which is like Google's endtoend automated product for for mobile app installs at the time, you know, which that predated advantage plus and PMAX and all and all those other systems. But so then so talk to me about that that sort of evolution where Google and Facebook now Meta and all these other companies wanted to bring that level of sophistication to their advertisers to give them the benefit of that that those those tools. Now, you probably would never find a team that

big a mobile even at a very large publisher. What's gotten easier? How how do things get easier as a result of that in mobile gaming UA? And then conversely, like what's gotten harder? I mean, because those bring challenges themselves. And so, talk to me about like in terms of just I'm spinning up a mobile UA team. I can do these things now that would have been unthinkable 10 years ago versus like okay these these challenges I'm gonna face are like totally new or they're a lot more acute now than they would have been 10 years ago.
Yeah, that's a great question. So for context where UAC was introduced, no one thought that would work. Uh it was just like this makes no sense. We want all those manual levers. We want to be able to like target specific publisher, target specific keywords. This makes no sense. And honestly, in the first probably two years, UAC was not particularly performant. But over time, they figured out how to find ways to find high quality customers. They introduced more sophisticated way ways of bidding. But most importantly, they made user acquisition much easier. So they democratized access to sophisticated UA tools and that is when

sort of like the the playing field got leveled and where like more advertisers could spend significant amounts of money on Google UAC, then Facebook, Advantage Plus that came after that. And you didn't really need to be this kind of like highly technical, highly sophisticated media buyer. And I'm not trying, you know, like when they talk about different NBA eras and how like whether I don't know, LeBron is better than Jordan. It's like it's generally a different era. You can't compare like 2015 to 2020. Just like a different buying tools. It's really that things got way easier on the kind of like touch point side of things. Like you didn't have to do as many manual like grunt worky things to manage campaigns, but that also meant more competition. More competition meant that again more people could like or more companies could spend money at scale on these channels. One thing that kind of that kind of made this situation even worse is consolidation started to happen. Iron source and unity are becoming one company. Liftoff and bangal are becoming one company. You know epin is acquiring max. Epin is acquiring mopub which is like the largest advertising exchange

mobile advertising exchange out there. So instead of like working with 20 30 plus partners that and you need like a large team to support that now you have four or five uh different channels that that have 95% coverage and you really like you need people on the team who understand these five channels really well and you need to do those five channels really well and you'll get you you'll get what you need instead of like trying to kind of do the the shotgun strategy and try to like work with 20 channels and and you know like not find one that's working. So, so basically consolidation is a starting point. Now there's fewer channels. It's slightly easier to manage user acquisition efforts because of that. The the flip side is that most of the buying nowadays is algorithmic. So you don't like move all those kind of knobs and like play around with different levers. It's the algorithms doing like doing the buying on your behalf and you're essentially in charge of three or four things. It's bits campaign type. Are you bid is like do you want to pay $10 or are you trying to get like a 10% return on ad spend by day seven? The second one is campaign

type. Are you optimizing towards cost per install which is not that common? Are you optimizing towards cost per retained user or like a day one return on ad spend or day one return on advertising? Well, basically like a certain advertising revenue by a certain date. So you can pick between different campaign types. Maybe this is a little confusing, but obviously you can you could just decide what are you optimizing your campaigns towards. The third one is budget. Are you spending $5,000 or like $100,000 a day? Uh changes things significantly in the sense whether the algorithm can hit those goals or not at that level of spend. And the last one is creative where creative essentially remain the only one kind of high touch point item that you can that you can optimize and that you can use to impact the the actual performance. So, so campaign execution is easier but hitting your goals is much harder because the competition significantly increased. One more thing that happened with consolidation that I I didn't read I I didn't find like many articles or couldn't find many articles writing about this is that given the fact that there's like four or five channels the

access to western audiences is significantly democratized. So if you are like a a Chinese developer, like an Asian developer who has like largely successful forex game doing really well in China, for you to like succeed in the western market, you just go to App Lavin, give them all of your money, and they'll bring as many high quality customers or like high quality players in your game as possible. And this wasn't really the case, you know, 10 years ago where Facebook was like not necessarily again it was like you could not like access Facebook without VPN. You could buy like Facebook inventory through agencies in China, etc., etc. So basically the competition opened up and and grew in its size and they're like the competition is thoroughly global nowadays and then because all of the buying is algorithmic. It's only on a handful of channels. Uh it's it's extremely competitive. It's not necessarily easy to to launch games and it's not necessarily easy to profitably run a user acquisition efforts. So that's the flip side of the democratization of media bike.
That's that's really great. Thanks. I

really enjoyed that answer. So, so I think like the way I kind of like characterize it is like you can't win by throwing more bodies at the problem anymore. It's like the the challenges have gotten a lot more data sciency, but in terms of like running your business, I think they've seeped a lot more deeply into the product, which maybe that's a good thing, right? Maybe it's a good thing. It's because that just like then that just forces everybody to make better products and really compete there because if you don't have to compete at the UA edge,
right? at the the DA frontier. So, everyone's there because they just use the four or five channels and they're and those are all, you know, very sophisticated. Then maybe it's just, okay, well, now I actually get to do what a consumer product company is designed to do, which is build a great consumer experience.
Yeah, that that's correct. like UA because it's so expensive and it's and it's like your business is so dependent on UA as well because organics are no longer like a a significant contributor to the overall business. You really need to make sure the products you're making

will work on these channels like if I think mate wrote today and I mentioned this few times in the past it's like you know if your product is not working on epin don't launch it. Like if your game is not able to successfully run epin campaigns at scale, it's like highly unlikely you'll have a successful product which informs like the the product development cycle. like you have in early you have to test these channels in early kind of product development cycles to see whether there's a product market fit or you need to change something either gameplay or like the art style theme in order to kind of like increase your product's chances of success through like lower CPI higher average CPI cost per install or high average revenue per user through just like finding the better kind of audience market or like product market fit. Most of the teams that doing user acquisition successfully nowadays and spending and scale are the teams that that run campaigns for the best products. I know it sounds very it sounds logical, right? Like the best products do best on the US side, but it wasn't the case a decade ago where there were still some

opportunities for arbitrage, but like truly if you look at Royal Match or games that are making over a billion dollars in revenue a year like Monopoly or Royal Match, those are like truly amazing games. They're great products with the, you know, best-in-class retention profile with the bestin-class monetization and that's why they are where they are. And essentially, UA just like amplifying great products and it's sort of like, you know, putting the the UA fuel on the on the great product fire to get them to where they where they need to be. So that that's how I see the current kind of landscape and who are the currently most successful usition buyers.
What's the process like? What's what does that look like? So what does your week look like? How do you structure that? I guess I guess with all this automation, what are you doing? [laughter]
Yeah.
If I mean if you know if the whole enterprise is like plugging some numbers into meta and just and then watching the chart go up like I you know not to sort of like denigrate the role and you know I did mobile gaming UA for a long time too but like what is the what's a week look like? How are you structuring the week? What are you focusing on? And how do you structure your process?

Yeah, it's a great question and you know I could give a funny answer but I probably shouldn't. So uh so yeah I mean the the change is like when you're leveraging algorithmic platforms or like algorithms for buying like what's being fed into that algorithm becomes the priority and you know we already talked about creative you really need to you know think about like hey how different creatives perform are they doing well or not like what's the retention profile for like a certain group of creatives etc etc but also like in general like one of the kind of key topics and it's it's not a new topic we talked about this at network and worked on this in network signal engineering essentially what type type of signal is being fed into the platforms. Uh I'll give an example of epven because it's probably the most relevant for the mobile gaming crew. It's like you need to drive like 20 purchases a day at a campaign level in order for the campaign to exit the learning phase and start performing well. Like how do you get to those 20 purchases a day? Like what is the budget that you're that is needed for you to hit 20 purchases a day? What is the starter pack in the game so it can actually happen? You have a $20 starter pack, maybe your conversion rate won't be high enough to get there. So this is

essentially why it's important to work closely with the team to explain them like why like certain number of purchase events and certain conversion rates are important for the campaigns so we get enough signal for the campaigns to kind of exit the learning phase and start performing. That's part one. So signal engineering understanding you know like like what's the what's how many purchases are being sent out every day if you're optimizing towards a specific revenue target. what's the daily average like average revenue per user or like you know average revenue per user on a certain day like day zero or day seven or day 28 which is kind of the dominant optimization like or conversion window nowadays that's that's kind of like one side of the story and then the second side of the story is creative creative again being such a huge lever it can help you essentially lower the cost it does two things it can either help you lower the cost which means that you can drive more purchases for the same budget which obviously helps the optimization process and speeds up the the feedback loop loop or the learning loop. And the second one, the second goal of the creative is audience discovery. It's like again working in tandem with the

product team trying to find especially for games that are like quite mature. You know the games I'm working with they're 10 plus years old. So you're trying to find different audiences that truly incremental to your game or to your product and then you know creative again helps you lower the cost or you know find different audiences that will spend maybe that will have higher arpo and then in that way kind of like hit again the campaign can hit their goals. So, so the the human oversight like portion of the job becomes really important because you need to make sure even though in theory you can like you can tell cloud to monitor whether there's 20 purchases a day or not, you need to kind of understand how these different things play like interact with each other and how they impact the overall campaign performance.
Well, and also the 20 purchases a day that's not totally within your control, right? It never was, but you know that that requires some coordination with the product team, I guess, right? I imagine that's and then if you're doing UA for a company like Zinga that's got a portfolio that's a lot of products to be coordinating with that. What does that part of the job look like?
Yeah. So, so again depending on the

budget size if you have like a smaller budget it takes more days to hit those numbers and tax the learning phase. So that will inform your cadence maybe you won't make daily changes you'll make weekly changes. It's absolutely depending on again the size of the product and the life cycle of the product. If it's a new product, you'll meet with the with them weekly, talk about the kind of the most important milestones, the the opportunities, the gaps, and try to like work with them to improve our again either like return on ad spend or you know overall product market fit by changing things in the game like maybe mini games that we're like introducing are just not performing well. So there's definitely that component of working very closely with the product team to inform them what are the things we need in order to like succeed. But also on you know outside of the product team they you know like it requires a lot of coordination with the UA team itself managing campaigns and then with the analytics team like trying to understand how different campaigns perform what is the LTV of those users because the LTV of a user coming from a coming from like a return on ad spend campaign versus the LTV of a user coming from like a blended return on ad spend

uh where by blended I mean you know we blend in a purchase and in a pads revenue is going to be very different because it's different profile of a user and then we have different LTV curves to estimate. Yeah.
So you have to work very closely with the marketing analytics team and then creative again plays a big role because different creatives drive different quality different user types. We need to understand what their LTV curves look like what their monetization profiles look like and then work closely with the creative analytics team and the creative strategy team to make more of those that work and fewer of those that don't. So, so it requires a lot of coordination and then multiply that times 8 or time 25 which is like the size of Zinga portfolio, you get a pretty complex operations in this case.
You know those channels your colleagues keep bragging about the ones getting all the credit? Yeah, they might be doing squat. Attribution makes every channel look like a hero, even when it's a zero. Incremental tells you who's [music] actually doing the work. It's like a lie

detector for your marketing budget. Start using Incremental today. Get your demo at incremental.com. That's [music] in n cmnt a l.com. Mention that you came through the mobile dev memo [music] podcast for a special 15% discount for the first 6 months. Let's let's sort of delve into that that creative process. Talk to me about that. So, so maybe just talk to me about how you do it, but but then also like what are the goals there? Like are you targeting volume of output? I imagine that a company that's, you know, publicly traded as Take 2 is, you know, there's there's guard rails or there's just constraints around what kind of creative you can launch. can't I imagine there's there are and and so there's limitations there and you're dealing with separate teams that may have different ideas of like what the representation of the product should be toward toward new customers that that maybe brings in some like qualitative sensibilities and and then yeah you're

you're you're having to navigate that that tension that can be sometimes difficult to communicate which is like look if we show you know this kind of a creative we're going to get this kind of audience and this kind of audience is going to behave in a different way. So you have to kind of segment these people out and think about them as groups. So talk to me about that. Like h how do you go about producing the creative that you use?
Obviously again different games, different budgets like have different creative requirements and uh different essentially uh creative refresh rates. Like on games that spend, you know, a few hundred thousand a month, you maybe refresh creatives once a month. On games that spend millions of dollars a month, you have to refresh creatives almost daily. One thing that changed as well, I think it's important to mention this before I like go into like the creative production process. It's that volume requirements keep you crazy. Uh the top spenders on Applavian introduce like thousands of ads a month. Arguably creative teams are much bigger than UA teams nowadays. If you talk about again like these Chinese like large imploving advertisers they have three or 500 plus person creative teams because creative volume became like a big thing and our

you know the platforms claim that they know which creatives will work in even before they like serve they serve one impression against those creatives. So it's that the volume imperative became like a big thing and we have to essentially try to produce more and you know for like bigger games we produce more for smaller games we produce fewer creatives. So it starts from there. What is the kind of success metric for creative? It's like hit rate. Like it's essentially kind of percentage of media spent. Uh some some teams call it hero rates. So if you build new creatives, those creatives are able to spend sufficient amount of money like let's say 10 or 20 or 30% of the overall share of wallet or the overall spend. Then that those creatives are considered successful creatives because they're able to hit specific return on net spend goal or cost per goal, whatever the campaign goal is. So So that's kind of like how the measurement framework works. Successful creatives are those creatives that spend and we need to make sure that we make more of those or have more heroes. So the creative production process like the goal of the successful creative production process is to like drive more heroes which is essentially

like we we essentially you know again have have this setup where their creative strategist, creative analysts, producers who make sure that ads get produced and they work closely with the user acquisition team to understand what's working, what's not working. There's also like a separate effort that's looking into like what what is working for our competitors, what are the most popular trends out there on social media on social media and on like for example sensor tower ad intelligence tools kind of what are the the the best performing ads overall and that that informs the creative production roadmap and we start producing those assets. Uh you brought up one more point which is really important. It's like what happens when you work with different licensers and IPs. That does complicate the process that essentially slows you down because it takes like three to four days to create like a video ad. It also takes another week to get approvals for those ads. So if you're working with licensers, you know, it things will take longer and also they might not be okay with certain themes. For example, misleading ads. They might not be okay with you like branching out of a specific world. For example, if you're like a Harry Potter game where we have a

Harry Potter puzzles and spells, you kind of have to stick within a specific Harry Potter world and also specific art style. So, there are some limitations and you have to understand those limitations before you kind of branch out and like start producing ads. But over time, you know, we got really good at this. We like work we've been working with these licensers for a very long time and we're like kind of like we know how to what to make and you know, what are the safest way to build creatives that will perform.
What did it look like at MZ? What was the creative production process? because I imag you know there were no image gen or video gen tools 2015 how were you making the creatives and how many were you making and I guess there was probably a more diversity of the types right so you probably you were working with banners and you were and you know even keywords and what were the major differences between then and now
yeah and nowadays it's mostly to your point video and playable or like interactive ads back in the days I I don't fully remember what was I think the team size was like around 75. But again, MC was like really like we had

like a lot of creative development was automated like localization was fully automated. Most of the static im static image creation was automated. So if you had like medium rectangles and banners and I don't know like interstitial images like localization and exporting was largely automated through like dripping tools. So it was really like unique at the time about machine zone and there were obviously two different there was outsourcing there was internal development anything that would that had kind of like to do with TV like working with Schwarzenegger if you remember it was a big thing working with Kate Upton we worked with Conor McGregor very early on yeah before he was like a big shot yeah we there's like a game of war Conor McGregor ad look it up uh so so there was you know there was like a professional team that it wasn't not everything was done in house but again Like people like don't give Gabe enough credit. He was really a visionary. And there was this famous famous I think you you definitely remember this video when he went to a conference and told everyone they have no idea how to do UA. I think he quoted that like we produce

200,000 ads in a given year
and and you know the mo the most important ad for machine zone was that famous play infamous playable that that probably was the most played game in the world at the time because it did really well and it did well because
MC thought about IPM and retention super early on we had like create reporting ahead of time and that that's why like the creative was like extremely how creative was what creative was made and run was like super data driven. And we had like this kind of like closed loop between creative strategy, creative analytics, user acquisition, and creative production. Uh, but again, a large team. I believe it was 75 people. It's been a while to be honest, so I don't I don't remember the actual number. And obviously a shout out to Gabe for, you know, like being a very important part of the mobile user acquisition history.
You talked about the kind of pods, right? So you've got the the Facebook pod, the Google pod, the, you know, Pinterest pod or whatever. Is that how you structure teams today? what what what's the right team structure in the in the current environment and and then

what are the roles that exist on a on a mobile UA team because I imagine like you know media buyer as such I mean I'm sure that there are media buyers but certainly not hundreds right I mean what are the sort of significant roles now how does how's the structure changed and then like who who's populating these teams
so what's different is that especially now with AI and and think of AI as an efficiency booster you need fewer people to do more things so if we structure the team and we had the same setup and network. You remember that it's like three pillars. It's basically you know media buying, creative analytics and then you need people who can like fit into one each one of these pillars and to understand how they you know interact with each other. So media buying is the same user acquisition. The only thing nowadays you need more of these kind of like T-shaped AI fluent UA leads who can run UA for like one game on their own. There's not you don't really need to make changes every day like on Facebook you move things around. to make changes every other day. So, so in terms of like actual grunt work, it's much easier, but you need someone senior who understands

AI, understands how to like harness the power of AI to kind of get the most out of of the tools and out of the out of the out of the products that we have. So, that's why having fewer AI fluent people can actually do the job. The goal right now is to build a team of of kind of super IC's who can do this but also to build systems that help them do their job extremely efficiently with the help of the AI tooling. The second part is creative. So again creative producers, creative strategists and and user acquisition folks work together to understand what needs to be made by you know what are is it video is it playable? Do we need to localize it culturalize etc. This is still fairly manual because again we don't know what's going to work even though there's so much data. You know how it is with creative. It's kind of like it's all about trusting the process and and creating a a sufficient amount of creative until you find the next hero. uh obviously it is data informed but it's not always like you know the hit rates are not always uh predictable and then and that's why you know the job of

a UA person UA lead is to work closely with creative strategies who sort of sits as a bridge in in between UA understands UA numbers understands like what the retention profile looks like what is the IPM and CPI correlation uh and then talks to creative production and actual marketing artists to develop those ads so they meet weekly uh they discuss what's being made they work together on briefs and then those briefs are turned into actual creative assets. The last point is market analytics. Uh market analytics can be consists of usually two teams. Creative analytics that provides all this creative level data and then market analytics helps the user acquisition team understand what are the opportunities uh works on predictive LTV models or like works with you know like adjacent teams such as like data science to build accurate predictive LTV models that help us understand what the performance will look like in the future and help us again invest money in the right bats and invest money in the right campaigns creatives etc etc. So again three pillars creative media buying uh analytics and then creative strategy sits between UA and creative uh

production and creative analytics sits between uh between these three sort of in the center between like analytics UA and creative that's how we structure it nowadays.
And then what about signal engineering is that like a dedicated role or is that just an analytics function?
It's not a dedicated role. That's essentially what this kind of like everyone on the team on the user acquisition side is aware of certain like purchase or event thresholds. So they're the ones who talk to the product team and trying to understand hey what can we do to like drive more purchases? Do we need more budget etc etc. So essentially that's the role of a UA person. The the market analytics person can like assist if we need some sort of custom reporting. I mean you know this really well that discrepancies are like a big part of our job. So partnering with market analytics to understand you know the discrepancies between partner networks or what the partner sees and what we see internally is super important. So that's why this like thing is not fully AI automated. There's just like human oversight is extremely important maybe more important than ever because wrong signal can break things and then you know both UA the UA team

and the analytics team like kind of like analyze this and try to like validate the data and make sure that it's accurate and helping us succeed.
And what about channel mix? You know, you mentioned you've got the four to five channels that give you sort of like, you know, 95% of your spend is concentrated there. Back in that 2015 era, testing new channels was 10% of your time or something, you know, it was you're just always on boarding a new and I feel like that's just probably not the case at all anymore. How do you think about that? And is it is there like is there like a sense that you need to optimize a channel mix or is it just, you know, hey, we're going to spend where we spend and it'll be driven by rorowaz. So when I think about new channels and you're right like you know a lot of teams I interacted with they had like a specific theme for new channels basically new channels testing team
I don't think that's the case anymore and when I think about new channels and channels in general I think about three things one is scale does that channel have sufficient scale you know like where Facebook is Facebook and Google are kind of high scale channels the other one is like return on ad spend or the cost like can I actually hit my return on ad spend targets on those

channels and you know like again plug in Facebook Google you you usually are able to hit your goal And then the last one is effort. If you have to like spend, you know, six hours a day optimizing a channel, no matter what the return on spend is is usually not worth it. And I think a lot about the RAR of effort because I I made this mistake multiple times in the past. Even when we worked at at I don't know like when we worked at network, I would spend X amount of like hours optimizing an incentivized network which drives like 10 purchases a day instead of optimizing Facebook that drives a thousand. So, so you really need in this new setup where there's like you know super IC's with an AI exoskeleton, you really need to think about where the time goes and is it like spent the most efficiently and the most efficient way to spending time right now is to focus on these high channel like high scale high RAI channels. Uh the other reason is targeting too like we do care about highly engaged high RPO customers and those are not it's not easy to find those customers outside of the wall gardens because they know exactly who's playing games. they know exactly who's spending in those games

and that's why like return on ad spend targeting has been uh so successful because they they have kind of this kind of like information as symmetry and know who high quality players are and it's it's usually testing that outside of these world gardens means that you're approaching like a shotgun kind of like uh you have a shotgun approach and and it's not necessarily easy to to succeed unless you have a lot of time on your hands to experiment. So this is probably something smaller teams do. We mostly focus on high scale high kind of return on ad spend channels. The only exception for gaming are these paper engagement networks that I mentioned briefly earlier like Edo, Misplay and Al Media. They're pretty big at this point and they are super relevant for the audience which is why like a lot of gaming advertisers focus on those and sometimes you can spend up to 50% of the budget on PPS.
Yeah, I had the CEO of of Misplay on a couple I don't know months ago. That was a that was an interesting episode. And talk talk to me about the the changing team structure, changing process, you know, how how the team evolves as you

scale up. So like when you're spending, you know, you mentioned you get you got the game that's spending six figures a month, you know, you creative is being refreshed once a month or something and you get into seven figures and and and you just have to probably re reconfigure the process. And so talk to me about that evolution, six figures to seven figures and then seven figures to eight figures. How how does the how does the process change? How does the workflow change? What does what does how does the focus change?
Yeah, my my recommendation for smaller companies who are just like kicking off if you if you have like a six figure budget, let's say anywhere, you know, from like a 100 to $300,000. Just do Google and Applin and do it well. Do Google and Android. It's the number one channel in Android. It does really well. It's truly incremental. Your measurement also becomes really really easy when you have only one channel.
Yeah. Uh so so essentially just do Google and Android app plug in on iOS and and you know make them do work really well and you know once you're ready to scale up start adding more channels as you go from like a few hundred,000 to like you know 500,000 to

to a million dollars you you're slowly adding more channels you start adding like unities of the world meta of the world meta is not the among a top three channel nowadays so like you usually add it much later in the mix and you can still do that with sort of like a factional team uh maybe like one UA lead who owns everything, but you have shared resources on the creative side and shared resources on the analytic side. Things start getting more interesting as you start scaling, you know, to like millions of dollars a month or tens of millions of dollars a month where essentially you have you need a dedicated team working on that game. You need multiple multiple UA people. You need a dedicated creative production team because you know you need to scale from let's say four creatives a week to 100 creatives a week and that requires a dedicated pod requires a dedicated creative strategist. It requires a dedicated market analytics person because it plays such a big role in this phase. One more important thing about like on the strategy side of things and how we scale campaigns. It's like again you start with two channels like $500,000 to a million you add more channels a million dollar to $5 million

you start adding uh more geos localization starts becoming a more important effort culturalization starts becoming a more important effort if you're like advertising in Japan like you know APAC or like Middle Eastern North Africa. So so you need like creative resources to actually support that or local agencies uh who can support you in in the culturalization efforts. And then as you go from like seven figures to eight figures, what becomes relevant is also audience segmentation like trying to find different audiences as you're sort of saturating the golden cohorts. So like audience segmentation becomes different. So different creative strategies and the second part is also brand building becomes more important because you want to you want to build a brand around the game. You're spending tens of millions of dollars a month which is why like we start seeing uh you know TV campaigns. We start seeing partnership with IP different licenses and IPs. we start seeing brand ambassadors like being attached to the game. So, so this is very common when like teams start like you know spending tens of millions of dollars a month and we have a handful of those in the market right now. Usually

they start partnering with licenses and IPs. The last thing is like measurement becomes incredibly complex and we talked about this like a few weeks ago. When you see like these kind of like large companies like you know prediction market apps they're literally advertising everywhere. It's like how do you measure this? like less click attribution will take you will only take you so far. So you really need to think about incrementality. You need to think about media mix models or like to think about marginal cost essentially. Does adding more channels actually drive more customers or drive more like you know uh purchasers. That that's again why like you need dedicated pods, dedicated teams working on one game because the spend level is so high that and and risk goes up with it that you need a dedicated team working on it. And we've seen this multiple times from DMZ era to like Monopoly Go nowadays. It's a very similar pattern.
Yeah. Yeah, I mean the measurement just becomes the job at that scale. I mean that's that is the job. It's just figuring out where you're actually spending incrementally. And I I I get kind of bombarded with all these, you know, pitches for these companies that are, you know, selling agents to, you

know, replace your marketing team. And it's like just I I I don't know that we'll never get there. I mean, I'm maybe to some extent. I mean, we will, but I'm always like, well, isn't I mean, Meta Ads Manager is an agent.
Yeah. Google PMAX is my agent there. If all of these companies, these platforms are like they have their agent. They've built it. They're doing I don't need an agent to coordinate those things. I guess I guess I do. But the reality is like the job is dealing with this consistent, steady, reliable stream of edge cases that break thing, you know, where things break and then interpreting the measurement. And like no one really knows how to build that from the outside. And then I think even on the inside it's going to be it's going to be very difficult to build a tool that just generalizes beyond
one single client like that. So the agent that you build that a probably requires you know human oversight no

matter what. But b probably just is this assemblage of humans that you've got dealing with fires always.
Yeah.
And all the complications of all the things that we just talked about. Right. And so like I've got the marketing agent and that's Advantage Plus. I've got the agent that's PAX. I've got, you know, I've got the Axon agent. I've got the Vector agent. I've got all, you know, in the mobile gaming space. I've got all these specific agents. They're specialized. What I need is a human to deal with this constant like relentless stream of
things that break. And that's the job. That is the job.
Yeah. It's still, you know, I always, you know, I was joking that I'm a plumber just like dealing with data pipelines, all the breakages. I think that's real that like what's changing, we touch upon this briefly, is like you like if you use offtheshelf solution, everybody's going to see the same results. It's really how you succeed is by finding like non-obvious patterns and exploiting them. You know, like finding different ways to like find an audience.

It used to be misleading ads or it used to be like new channels for example or new geos or finding some or like UGC ads for example over the time. So you have to always try things and not use things that already exist. And that's the kind of like that's the creativity part that's super important. It always be super important and we'll most of the time be human driven like or for the long time it will be human driven and like our job is to remove the friction and the way I think again about AI is like build workflows like you don't need to upload creative on Facebook anymore like you can build an agent where the producer pushes the ad directly to Meta and then Meta can also directly run those ads but again the pipeline stuff the data stuff the breakage that has to be done by a human and there has to be a strong human oversight who knows what they're and then they're doing it well. So that that's kind of how I see the current shift and transition from sort of like somewhat AI to full AI marketing stack.
Yeah, as you were talking I was just reminded of noob versus pro. Remember that was like we had this hero creative at network that just carried us for what

three months a noob versus pro
and it we didn't come up with the concept. We kind of ripped it off but like it just worked. It was just the absolute champion creative that we had for I mean just beat every other creative and we scaled our UA. I mean something like some unbelievable growth and spend during that period and that was it. It was just having that one creative is just carried us for that long. All right. I want to end on the events top. So we discussed this a while a couple a little bit ago few weeks ago but what are the events? How how's the events landscape changed? What events are people going to? Yeah, I mean, you know, we all became parents and a little older, so I don't go to a lot of events. You know, the the number one mobile kind of user acquisition event nowadays, it's pretty big. It's back to where it was before. The energy was amazing. You get to meet all the industry experts and all the most kind of relevant vendors. It is a really good conference and that's pretty much the only one I'm going to like regularly. Uh there's a lot of smaller local events like ASP Growth

Summit. uh they're doing a good job of like, you know, keeping the local communities together. Absolar did like a a decent job with MAMA events. I don't know if you had a chance to go to any of those. I I went to this year's and the previous years. They were amazing events, really nice venues. So So I enjoyed those and and that's about it on my end. I unfortunately don't go to a lot of a lot of events anymore, but these are kind of the most important ones for me. Again, there's games forum, there's all these other events, but I feel like most of the kind of senior people don't go to them that much anymore. And it's more of these kind of local smaller events that get the gang back together.
Well, and then I guess it seems like the the European event landscape is a lot richer. Everyone was at Gamescom. I don't Is Gamescom interesting? What's happening at Gamescom? I I went once and it was just I didn't really think it was that useful, but it seems like everyone was there.
It's also the most of the gaming move to Europe, so there's like a larger group of companies there, and I think they're just like starving for good events. Uh, Games Forum Barcelona was pretty busy. I

was there last week actually, last year. So, I think Games Forum, Gamescom are still somewhat relevant because there's not that many that many kind of UA specialized event. They're vendor events like Amplify. I know this MaddX has this kind of like UA Yak or something like that event. So, I know there's a lot of these vendor sponsored events, but I don't usually go to those.
Nepo, thank you. You took us on a trip down memory lane. It was nice to reminisce about the 2015 era, the frenetic 2015 era. How can people uh hear from you? How can they connect? How can they they consume your wisdom?
I'm pretty active on on LinkedIn still on on Axe. My username is slightly slightly hard to pronounce. It's INIAC like the first computer. So find me on X. Find me on LinkedIn. Yeah, just email me. Eric can leave the email. Happy to help. And yeah, thank you so much for the opportunity. It's always again great to talk to you and thank you for a decade plus worth of content and you know propping up this industry and making us all better at our jobs through mobile dev memo.
Oh well that's very kind of you. Let's

here's to another decade. Cheers.
Yeah. Cheers. [music]