What is incrementality testing?

Last updated: May 5, 2026

Introduction

Marketing teams often ask, “Which part of our advertising is actually driving results?” Incrementality testing is a way to answer that question by scientifically isolating the effect of your marketing. Think of it like a controlled experiment for your ad spend – similar to a medical trial or a taste-test – where one group is exposed to the marketing (treatment) and another is not (control). By comparing outcomes, you can see what additional results (“incremental” sales, clicks, sign-ups, etc.) were caused by the marketing and wouldn’t have happened otherwise . In other words, incrementality testing reveals the true lift your campaigns deliver beyond what would occur naturally. This article will explain what incrementality testing is, the main ways to do it, how it differs from Marketing Mix Modeling (MMM), recent machine learning advances that make it more accurate, and how incrementality tests and MMM can work together.

What Is Incrementality Testing?

Incrementality testing is an approach to measure the causal impact of a marketing initiative by using a controlled experiment. Rather than relying on attribution models that credit every touchpoint (which often overstate impact by counting conversions that would happen anyway), incrementality tests deliberately hold back some audience or markets from seeing your ads and compare their results to those who did see the ads . The only difference between the two groups is the advertising, so any uplift in the test group can be confidently attributed to the campaign – that uplift is the incremental effect.

Put simply, incrementality = the lift driven solely by your marketing. For example, if you ran ads on a new podcast and sales rose 10% in regions where the ads ran (compared to no change in regions without the ads), that 10% is the incremental gain from the podcast ads. If no difference is detected, it suggests the campaign didn’t add any value (though it could also mean the test wasn’t sensitive enough or the execution needs improvement ).

Why is this important? Incrementality testing gives marketers a clear “cause-and-effect” read on performance, cutting through noise and overlaps in a multi-channel world. It’s like getting an X-ray of your marketing portfolio – revealing which tactics truly drive growth and which are just riding along with organic sales. This helps answer the classic Wanamaker dilemma (“Half the money I spend on advertising is wasted; I just don’t know which half”) with hard data, so you can confidently optimize budgets toward what actually works.

Forms of Incrementality Testing and How They Differ

There’s no one-size-fits-all way to run an incrementality test. Marketers use several experimental designs to find a suitable control group and measure lift. Each has strengths and limitations. Here are some of the most common forms:

Geo Experiments (Geographic Holdouts): This method uses different regions or markets as the test and control groups. For example, you might increase ad spend in a set of test cities while keeping it off (or at baseline levels) in a set of control cities. After the test period, you compare the outcomes (sales, site traffic, etc.) between those “exposed” markets and the “holdout” markets to estimate the incremental impact of the ads. Geo experiments are popular for channels like TV, radio, or out-of-home where you can’t easily randomize individual users, but you can vary marketing by location. A key benefit is that geo tests are conceptually simple and the results are straightforward to interpret . They also allow for real-world testing at scale (since each region encapsulates many customers). However, careful design is needed to ensure regions are truly comparable and isolated – factors like regional differences or spillover (people in a control area seeing a neighboring area’s billboard or ad) can bias results . Tech companies like Google and Meta provide frameworks for geo experiments (often called “geo lift” tests) because they’re systematic and effective when done right .
Synthetic Control Experiments: One of the more advanced techniques is using a synthetic control group. Instead of picking just one real-world control group, you algorithmically construct a blended control from multiple units. For example, imagine you’re testing an out-of-home ad in Detroit. Rather than using Milwaukee alone as the control city (which might be “close but not perfect”), a synthetic control might combine 70% of Milwaukee’s trend, 20% of Philadelphia’s, and 10% of St. Louis’ to create a composite that mirrors Detroit . It’s like creating a Frankenstein’s monster of control regions to get the best possible match for the test region’s behavior had it not received the ads. By weighting and blending several markets, synthetic controls can more closely replicate the counterfactual (what would have happened without the campaign) than any single market could . This leads to more precise lift estimates – one study found synthetic control-based tests were 4× more precise than simpler matched-market tests . The downside is complexity: it requires statistical modeling or machine learning to select and weight the control components. But modern tools (including open-source libraries from Meta and others) can handle this heavy lifting. Synthetic controls are especially useful in geo experiments to ensure your control is as similar as possible to your treated regions . They help account for external trends by using many data points, providing a more robust baseline for comparison.
User-Level Holdout Tests (A/B Tests) or Conversion Lift: In digital channels (such as Meta/Facebook or YouTube ads), it’s often possible to randomly split your audience itself. One group of users is shown your ads (treatment) and another group sees no ads (or sometimes sees a placebo PSA ad) as the control. Because assignment is random, both groups should be statistically identical upfront. This yields a clean apples-to-apples comparison of how the exposed users behave versus the unexposed users . User-level experiments are the gold standard for isolating causality – they largely avoid external confounders since both groups live in the same markets and time periods. This approach shines for digital performance campaigns (e.g. measuring incremental app installs from a mobile ad campaign by holding out a percentage of app users). The challenge is that you need platform or data access to implement the split. Many large ad platforms offer lift test tools that do this behind the scenes. The strength of user holdouts is precision (you’re zeroing in on individual-level impact) . A weakness is they can require large sample sizes to detect lift, and improper randomization or biased user selection can skew results . Still, when feasible, a randomized controlled trial on users gives extremely credible incrementality insights.
Time-Based Holdouts (On/Off Testing): Another variant is a holdout over time. For instance, you might pause a campaign for a controlled period (or turn off a specific channel for a week/month) and observe what happens to your metrics, then restart it and see if metrics lift again. This “on vs. off” approach can reveal the incremental contribution by noting the drop when advertising is absent. It’s essentially a before-and-after test. This method is easy to execute (just stop ads in the test period or region), and in attribution-limited environments (like a broad awareness campaign) it may be one of few options. However, it’s less robust because many external factors could change over time. To improve it, marketers often combine time-based holdouts with geo tests (e.g. turn off ads only in certain regions, while keeping other regions running as a comparison). Such blackout tests or matched market tests add more rigor than a simple before/after because you have a simultaneous control comparison.

Aside: Other methods – Occasionally, marketers also gauge incrementality through survey-based lift studies (for example, running brand lift polls to see if exposed audiences have higher brand awareness than control audiences) or panel-based approaches. These can measure the perceived impact of ads (useful for upper-funnel brand campaigns) , but they rely on self-reported data rather than observed behavior. For performance marketing outcomes (sales, conversions), the experimental methods above (geo, user holdouts, etc.) are preferred for concrete, behavioral lift measurement.

Each form of incrementality testing involves a trade-off between practical feasibility and experimental purity. Geo tests are practical for almost any channel (even offline media) and can cover large populations, but require careful control matching. User-level tests offer purity and granular insight, but only work where individual-level randomization is possible. More advanced constructs like synthetic controls improve accuracy at the cost of added complexity in design. The good news is that all these approaches share the same spirit: isolate a control vs. exposed group to get a true read on lift. Marketers can choose the design that best fits their channel mix, budget, and data access.

Incrementality Testing vs. Marketing Mix Modeling (MMM)

You may have also heard of Marketing Mix Modeling (MMM) as a way to measure advertising impact. Both MMM and incrementality testing aim to measure the contribution of marketing, but they are very different approaches that serve different needs. Here’s how they compare:

Methodology: Incrementality tests are experimental – you actively change something (e.g. hold out ads) and observe the difference. MMM is analytical/modeling-based – it uses historical data and statistical models (often regression or machine learning) to estimate each channel’s effect on sales over time, without intentionally holding out media . Think of incrementality testing as looking at a snapshot in time via a focused experiment, whereas MMM is looking at long-term patterns across many time periods.
Scope: Incrementality tests are typically micro-level and tactical. They zoom in on one channel or campaign at a time to answer “did this specific effort drive extra results?” . For example, you might test just your YouTube ads for a month in a few regions. MMM, on the other hand, is macro-level and holistic. It considers all major channels and external factors together – usually covering years of data – to tease out each factor’s contribution to sales . It’s great for big-picture questions like budget allocation: “What mix of TV, paid search, social, etc. is optimal for our growth?”
Granularity and Interaction Effects: Because MMM looks at the whole marketing ecosystem, it can capture cross-channel interactions and diminishing returns. For instance, MMM might reveal that your search ads perform better in weeks when TV is also running, or that a channel’s ROI falls off after a certain spend level . Incrementality tests, especially single-channel ones, usually hold other things constant and measure the isolated effect of one element – they don’t inherently show how channels work together. In other words, an incrementality test is like examining one puzzle piece in detail, whereas MMM assembles the entire puzzle to see how pieces fit together.
Speed and Actionability: Incrementality experiments often run for a few weeks or months and give near-immediate, campaign-specific insights. This makes them useful for quick tactical decisions – e.g. deciding if a particular campaign was worth it, or optimizing an ongoing strategy on the fly . MMM traditionally has been a longer-term project (taking weeks or months to build the model, and using data from past quarters or years). Its insights guide strategic planning (quarterly or annual budget shifts, channel mix strategy) rather than day-to-day tweaks . However, modern MMM tools are getting faster and more “always-on” (more on that later). Still, as a rule of thumb: use incrementality tests when you need a focused answer ASAP, and use MMM for broad analysis and forecasting.
External Factors: A well-built MMM will explicitly account for things like seasonality, economic trends, or competitor activity by including them in the model . This means MMM can tell you, for example, that sales were down due to a recession even if marketing was strong, or adjust for a big one-time event. Incrementality tests handle external factors by relying on simultaneous control groups. If your control group experiences the same external conditions as your test (e.g. both regions went through Black Friday, both saw the competitor’s new product launch), then the external noise cancels out in the comparison. But if something unexpected affects only one group, it can distort results. In practice, good test design and sufficiently large groups mitigate this, but MMM’s approach of using historical data and many variables provides a more explicit adjustment for external influences in measuring marketing impact .
Data Requirements and Cost: Running experiments can incur opportunity cost (e.g. withholding ads from a potential audience means potentially lost sales in the short term). They also require operational effort to design, execute, and analyze each test – which can be resource-intensive if you want to test many things . MMM requires lots of data (usually aggregated spend and sales data over time, plus any control variables) and statistical expertise to build the model, which historically made it an expensive undertaking mostly for large advertisers . New tools and automation are reducing this barrier, but MMM still works best with robust data and usually benefits from experts or advanced software to implement.

In summary: Incrementality testing offers direct, causal proof of a marketing tactic’s impact, but typically focuses on one piece of the puzzle at a time . MMM provides a comprehensive, big-picture view of all marketing efforts combined, accounting for their interplay and external factors . One isn’t “better” than the other – they’re complementary (as we’ll discuss shortly). A helpful analogy: incrementality tests are like using a microscope to zoom in on one specific campaign, whereas MMM is like a wide-angle lens capturing the whole landscape. Or as one industry expert put it, an incrementality test is akin to examining a single puzzle piece, while MMM assembles the entire puzzle to guide strategic decisions.

What hypotheses can be tested through Incrementality Testing?

One of the most powerful aspects of incrementality testing is its flexibility—it’s not just about testing if a channel works, but also how, where, for whom, and at what level. Here’s a breakdown of common hypothesis types marketers can test using incrementality testing, grouped by focus area:

1. Channel Hypotheses

Goal: Determine if a specific marketing channel drives incremental results beyond what would happen without it. Example Hypotheses:

“Running podcast ads increases incremental sales.”
“YouTube TrueView campaigns drive incremental brand search volume.”
“Connected TV ads generate net-new customers, not just shifting existing ones from other channels.”

2. Budget Level (Spend Elasticity) Hypotheses

Goal: Test the impact of different spend levels to understand diminishing returns or identify optimal budget thresholds. Example Hypotheses:

“Doubling our Meta spend from $500K to $1M yields more than double the incremental conversions.”
“There are diminishing returns in linear TV once we exceed $2M per month in spend.”
“A small spend in out-of-home still drives meaningful lift.”

3. Creative Messaging Hypotheses

Goal: Assess whether different creative concepts or ad formats generate varying levels of incremental impact. Example Hypotheses:

“Emotional storytelling ads on YouTube are more incremental than product demo videos.”
“Short-form podcast reads drive more lift than long-form host discussions.”
“User-generated content performs better than branded studio ads in Meta feed.”

4. Offer or Incentive Hypotheses

Goal: Measure how different promotions or value propositions affect incremental outcomes. Example Hypotheses:

“Offering free shipping increases conversions more than a 10% discount.”
“A limited-time bundle offer drives higher incremental lift than a standalone product ad.”
“Discounts are only incremental for lapsed users, not current subscribers.”

5. Targeting or Audience Hypotheses

Goal: Test whether specific audience segments are more or less incremental to reach. Example Hypotheses:

“Incremental lift is higher among Gen Z than Millennials on TikTok.”
“Retargeting past purchasers has lower incremental lift than prospecting new audiences.”
“High-income ZIP codes respond more incrementally to CTV ads than low-income areas.”

6. Geography or Market Hypotheses

Goal: Understand if location or market dynamics influence incrementality. Example Hypotheses:

“Urban markets show higher incremental lift from billboards than rural areas.”
“Newly entered markets are more responsive to influencer campaigns.”
“Digital ads' incrementality is lower in cities with strong retail presence.”

7. Timing and Flighting Hypotheses

Goal: See how ad timing, duration, or pacing affects incremental impact. Example Hypotheses:

“Weekend CTV ads drive more incremental signups than weekday placements.”
“Running fewer but longer TV flights yields more impact than continuous low-level exposure.”
“Early morning podcast ads perform better than evening placements.”

8. Format or Placement Hypotheses

Goal: Determine which ad formats or placements within a platform drive the most lift. Example Hypotheses:

“Instagram Stories drive more incremental conversions than Instagram Reels.”
“Pre-roll YouTube ads are more effective than mid-roll ads for product awareness.”
“CTV ads placed on premium content yield higher incremental sales than remnant inventory.”

9. Cross-Channel Interaction Hypotheses

Goal: Test if combining channels results in synergy or redundancy. Example Hypotheses:

“Combining podcast and YouTube ads increases overall lift more than each alone.”
“Paid search has greater incremental lift when supported by brand TV.”
“Out-of-home boosts performance of mobile campaigns in geo-targeted areas.”

Want to learn more about experimentation? Check out other Paramark experimentation articles here

What is incrementality testing?

Introduction