Problems with A/B Testing in Mobile Marketing

A/B testing in mobile app development and marketing is statistically fraught, and marketing tests in particular carry distinct risks that make conventional approaches misleading. Drawing on a 2015 interview with King's head of data analytics, the article first outlines four problems with A/B testing in free-to-play games: Pareto-distributed monetization lets a small number of enthusiastic players skew results (while also requiring large samples to capture the full distribution), cross-device experiences are hard to test consistently, effects are difficult to verify and demand long test durations, and results are often too specific to apply across a game portfolio. Premature test termination upon observing significance is noted as a further common pitfall.

The analysis then argues that testing for user acquisition is an even more precarious undertaking. Most mobile ad networks run Bayesian bandit algorithms that automatically shift traffic toward the best-performing creative by acquisition cost. Because each variant is then served to a different pool of impressions, results from the two arms are not comparable. Facebook is cited as an exception that allows campaigns to run without automatic optimization, but most networks do not.

A second, more fundamental problem is that marketers tend to optimize for local metrics such as click-through rate or cost per install, which ignores the developer's ultimate objective of revenue. The "click-through-rate conundrum" illustrates that ad clicks can move opposite to store-page install conversion when creatives are tuned only for clicks. Click-to-install is a better measure, but still insufficient, since targeting breadth and total addressable market also matter.

The conclusion is that marketing A/B tests should identify which materials generate the traffic mix that maximizes total net revenue at scale, balancing user base size against acquisition cost and lifetime revenue. The winning variant is not necessarily the cheapest, broadest, or most clickable, nor the one yielding the most engaged or monetizing cohorts. Exceptions arise when strategic goals, such as growing the user base ahead of a funding round or exit, take precedence.