Incrementality Testing Guide: Measure True Marketing Impact

Attribution tells you who touched what. Incrementality tells you what actually worked. As privacy changes make deterministic attribution harder, understanding true incremental impact becomes essential for marketing efficiency.

What is Incrementality?

Incrementality measures the causal impact of your marketing—the conversions that would NOT have happened without your advertising. It separates true lift from conversions that would have occurred organically.

Why Traditional Attribution Falls Short

• Correlation ≠ causation: Seeing an ad before converting doesn't mean the ad caused the conversion
• Selection bias: You're targeting people already likely to convert
• Attribution wars: Multiple channels claiming credit for same conversion
• Privacy gaps: Increasing inability to track user journeys

Incrementality Testing Methods

Randomized Controlled Tests (RCTs)

The gold standard—randomly assign users to test (see ads) or control (don't see ads) groups.

ClicksFlyer Team

1. Define your test hypothesis and success metrics
2. Calculate required sample size for statistical significance
3. Randomly split audience into test and control
4. Run ads to test group only
5. Measure conversion difference between groups
6. Calculate lift and statistical significance

Ghost Bids / Intent-to-Treat

For auction-based environments, participate in auctions but don't show ads to control group.

• Bid on all eligible impressions
• Win auctions for both groups
• Serve ads only to test group
• Track conversions for both groups

Geo Experiments

Use geographic regions as test and control groups when user-level randomization isn't possible.

ClicksFlyer Team

• Match markets by size, demographics, and baseline behavior
• Account for seasonal and regional variations
• Run for sufficient duration (2-4 weeks minimum)
• Use synthetic control methods for analysis

"The best incrementality test is one you can act on. A statistically sound result that's too expensive to implement doesn't help anyone."

Designing Your Test

Hypothesis Formation

Start with a clear, testable hypothesis:

• "Facebook ads drive X% incremental installs"
• "Retargeting generates Y% lift in purchases"
• "Upper-funnel video increases Z% conversions"

Sample Size Calculation

Key factors determining required sample size:

• Baseline conversion rate: Lower rates need larger samples
• Expected lift: Smaller expected lifts need larger samples
• Confidence level: Usually 95%
• Statistical power: Usually 80%

Test Duration

• Minimum 1-2 weeks to capture conversion cycles
• Account for day-of-week effects
• Consider your typical time-to-convert
• Avoid major seasonality or events

Analyzing Results

Key Metrics

• Incremental lift: (Test - Control) / Control
• Incremental conversions: Total attributed × incremental rate
• True CPA: Spend / Incremental conversions
• iROAS: Incremental revenue / Spend

Statistical Significance

• Calculate p-value for observed lift
• Look for p < 0.05 (95% confidence)
• Report confidence intervals, not just point estimates
• Be wary of small sample sizes or short tests

Common Pitfalls

Contamination

When control group gets exposed to treatment:

• Cross-device exposure
• Spillover effects in geo tests
• Shared household exposure

Selection Bias

When test and control groups aren't truly comparable:

• Non-random assignment
• Pre-existing differences between groups
• Opt-in bias

Acting on Results

If Lift is Lower Than Expected

• Reduce budget on that channel/campaign
• Reallocate to higher-incrementality tactics
• Test different targeting or creative approaches
• Consider if you're over-investing in bottom-funnel

Building an Incrementality Culture

• Run regular tests across channels
• Build incrementality into planning models
• Train teams to think beyond attributed performance
• Invest in measurement infrastructure

Incrementality testing requires investment, but the insights are invaluable. Understanding true marketing impact allows you to allocate budget efficiently and prove value to stakeholders.