Experiment Design FAQ

For more information on Experiment Design methodology, see our Experiment Design Documentation.

What is incrementality testing and why should I run experiments?

Incrementality testing measures the true causal impact of your marketing by comparing a treated group (exposed to ads) versus a control group (not exposed to ads). This helps you understand what additional results were actually caused by your marketing and wouldn't have happened otherwise.

Unlike attribution models that show correlation, incrementality tests prove causation by isolating the effect of your marketing through controlled experiments.

What types of experiments can I run?

Geo Experiments (Geographic Holdouts): Use different regions as test and control groups. You might increase ad spend in test cities while keeping it off in control cities.

User-Level Holdout Tests: Randomly split your audience where one group sees ads and another doesn't. This is often available through platforms like Meta and Google.

Time-Based Holdouts: Pause campaigns for controlled periods to observe impact, then restart to see if metrics lift again.

Why do some experiment instructions lower spend in some markets instead of simply adding more budget to the test markets?

This usually comes down to which test type Paramark is designing. Not every experiment is a pulse-up where the goal is to add spend in the test cell while leaving the rest of the footprint untouched.

• Holdout design: Some markets intentionally spend less or go dark, while the remaining markets continue business as usual. This is useful when the question is “what happens if this channel is removed?” In that setup, lowering spend in the holdout markets is the treatment.

• Pulse-up design: Test markets receive more spend while control markets stay flat. This is useful when the question is “what happens if we scale this channel?” In that setup, adding more budget to the test markets is the treatment.

The reason Paramark does not simply "add more to the test budget" in every case is that doing so would answer a different measurement question. A holdout tells you the value of turning a channel down or off. A pulse-up tells you the value of increasing it. Those are related, but they are not interchangeable.

There is also a practical constraint: the experiment needs a clean comparison between cells. If you lower spend in one set of markets and also add unplanned spend elsewhere, you may no longer have a readable two-cell test. In some cases a three-cell design (holdout + business-as-usual control + pulse-up) is possible, but it requires more markets and more statistical power.

So the simplest rule of thumb is: if the design shows lower budgets in a subset of markets, that is usually because the experiment is a holdout, not because Paramark forgot that adding spend was an option. If your business question is specifically about scaling rather than holding out, ask your Growth Advisor whether the right design is a pulse-up instead.

Why would we lower budgets in the rest of the state instead of only adding more budget to the test markets?

It depends on the test design. In a pulse-up test, you usually do add spend in the test markets while keeping the control markets at business-as-usual. In a holdout test, you reduce or remove spend in the holdout markets so you can compare those markets against the ones that keep running normally.

The key principle is that the experiment needs two clearly different treatment conditions. If every market simply gets more budget, there is no stable baseline left to compare against. The result becomes much harder to interpret because you cannot tell what would have happened without the change.

A few practical ways this shows up:

• Pulse-up design: Increase spend in the test markets and keep the control markets flat. This is used when the question is whether more spend creates more incremental outcome.

• Holdout design: Reduce or pause spend in the holdout markets and keep the rest of the markets flat. This is used when the question is whether the existing channel is incremental at all.

• Three-cell design: If you want to both hold out some markets and add budget in others, that is usually a separate multi-cell design with a holdout cell, a pulse-up cell, and a business-as-usual control cell.

So the answer is not that Paramark always prefers lowering budgets elsewhere. It is that we choose the structure that matches the business question and preserves a clean comparison. If the goal is to learn whether additional spend drives lift, a pulse-up design is often the more natural setup. If the goal is to learn whether the channel is incremental at its current spend, a holdout is often the cleaner design.

How do experiments work with Marketing Mix Modeling (MMM)?

Experiments and MMM complement each other perfectly. MMM provides continuous "always-on" measurement across all channels, while experiments give deep, point-in-time insights for specific channels.

You can use experiment results to validate and calibrate your MMM findings. If your MMM suggests a channel has 3.0x ROI and an experiment shows similar lift, that increases confidence in the model.

How long do experiments take and what's the timeline?

Experiment Duration: Typically 4-12 weeks depending on the channel and required statistical power. Bottom-funnel channels like Google Search may need shorter periods (4 weeks) while top-funnel channels like TV may need longer.

Design Timeline: Experiment designs typically take 1-2 weeks to prepare once we have all necessary data and requirements.

Analysis Timeline: Expect 2-4 weeks after the experiment ends for complete analysis and results.

Would a true matched-market test usually take longer than a synthetic-control design?

Usually, yes. A synthetic control uses a weighted blend of control markets to mirror the test markets as closely as possible, which generally improves the precision of the comparison.

If you switch to a stricter matched-market setup instead of a synthetic control, the comparison is often less precise. That typically means you need one or more of the following to reach the same confidence level:

• A longer test window

• A larger spend differential or treatment size

• A willingness to detect only larger effects (a higher minimum detectable effect)

There is not a universal rule like "add two more weeks" — the exact tradeoff depends on the channel, target metric volume, number of eligible markets, and how closely the proposed markets match in the pre-period. In practice, if a customer wants to compare a matched-market approach against a synthetic-control design, Paramark would re-run the design assumptions and show the revised budget, duration, and power tradeoffs before launch.

When is it safe to start an experiment after a major campaign or platform change?

Usually, you should wait until the channel has stabilized enough that the test is measuring the experiment itself rather than the platform's relearning period. A good rule of thumb is to avoid launching immediately after a major structural change such as a campaign rebuild, a large budget reset, a major targeting change, or a platform-side change that materially affects delivery.

What matters most is whether performance and delivery have returned to something close to normal operating behavior. For many digital channels, that stabilization window is often a matter of days, not months, but it depends on how disruptive the change was.

• Small operational changes like minor budget tuning or creative swaps usually do not require a long delay.

• Larger changes like campaign restructures, new bidding logic, geo setup changes, or platform outages can require a longer wait before launch.

• If you are unsure, ask before starting the clock. It is much easier to delay launch briefly than to invalidate a full test window.

In practice, Paramark will usually look for a short steady-state period before the experiment begins, confirm that tracking and delivery look normal, and then start the test window. If a major change happens right before launch, the safest move is often to push the start date until the channel has settled.

For more on whether you can shorten the timeline with more budget, see Can I speed up an experiment by spending more?

Can I run a paid search experiment if the account is still recovering from a major disruption or rebuild?

Usually, only after the account has returned to a reasonably steady operating state. If paid search performance is still being distorted by a recent account deletion, campaign rebuild, tracking repair, bidding reset, or another major disruption, the experiment can end up measuring the recovery itself rather than the incremental effect of the test.

A practical rule of thumb is to separate recovery from experimentation. If spend delivery, campaign structure, geo targeting, conversion tracking, and bidding behavior are still normalizing, it is usually better to stabilize first and then launch the test on the version of the account you actually want to evaluate.

For paid search specifically, three checks matter most before launch:

• The account is serving normally again. Budget pacing, impression delivery, and campaign coverage should no longer be in a post-rebuild or post-outage recovery pattern.

• The primary conversion signal is finalized. If you plan to introduce lead scoring, Enhanced Conversions for Leads, offline conversion imports, or a new qualified-lead event, it is usually better to finish that change before the experiment starts rather than switch optimization signals mid-test.

• The target metric matches the business question. If lead volume is flat but downstream outcomes like SQLs, opportunities, or pipeline have deteriorated, the experiment should usually be judged on the most downstream metric that still has enough volume to measure credibly, with upper-funnel metrics treated as supporting diagnostics rather than the primary success metric.

If the account must be tested while some recovery is still underway, the safest framing is usually to treat that work as a feasibility or planning conversation first: confirm what has stabilized, identify which changes are still in flight, and decide whether the right next step is a short waiting period, a different metric, a longer test window, or a different methodology.

The simple decision rule is: start the experiment only once the account conditions and optimization signal reflect the setup you actually want to learn from. Otherwise, a later improvement or decline may be impossible to attribute cleanly to the experiment itself.

If I switched from a synthetic-control geo design to a true matched-market test, how much longer would I need to run it?

Usually longer — but there is no universal “add two weeks” rule.

A synthetic control is typically more statistically efficient than a rigid matched-market setup because it can build the counterfactual from a weighted blend of multiple control markets rather than relying on a single fixed match. In practice, that means a true matched-market version of the same test will often need more time, more spend, or both to reach the same confidence.

The size of that tradeoff depends on:

• The channel being tested. Faster-response channels can sometimes reach signal sooner; slower-response channels like CTV, TV, audio, and other upper-funnel media usually still need a longer in-market window.

• The target metric volume. If the KPI is sparse or delayed, the matched-market version usually needs meaningfully more observation time.

• The minimum lift you need to detect. Smaller effects require more precision, which makes the penalty for switching away from synthetic control larger.

A safe customer-facing rule of thumb is: expect a true matched-market test to run somewhat longer than the synthetic-control design, not shorter. But the exact duration should be sized by rerunning the design rather than estimated from a fixed multiplier.

If you are asking whether you can keep the same runtime by increasing spend, sometimes you can partially offset the change — but only up to a point. For more on that tradeoff, see Can I speed up an experiment by spending more?

What is a Meta holdout test, in plain English?

A Meta holdout test is a way to measure whether Meta ads are actually causing additional business results, not just showing up before conversions that would have happened anyway.

In the simplest version, Meta randomly withholds ads from one group and continues showing ads to another similar group. If the exposed group produces more purchases, signups, pipeline, or another agreed KPI than the holdout group, that difference is the incremental lift from Meta.

A useful TL;DR is: one group sees Meta ads, one group does not, and the gap in outcomes estimates what Meta truly caused.

This kind of test is especially useful when a team is taking over an account, re-evaluating whether Meta deserves more budget, or wants a causal read that is more trustworthy than platform-reported attribution alone.

A few practical expectations:

• It is not an instant read. Most tests still need enough runtime and volume to detect lift reliably.

• The KPI matters. The best metric is usually the business outcome you actually care about, as long as it has enough volume to measure.

• It answers a specific setup question. A result from one Meta setup does not automatically transfer forever if the spend level, audience mix, campaign objective, or account structure changes materially.

• It can complement geo experiments and MMM. Platform holdouts give a direct causal read inside Meta, while geo tests and MMM can help validate performance more broadly across channels and markets.

Should I use platform conversion lift tests or external geo tests?

Platform Tests (Meta Conversion Lift, Google Conversion Lift): These are powerful when available and should be used if you have access. They offer precise user-level randomization.

External Validation: We recommend external geo tests for additional validation, especially for larger budget decisions, as they provide unbiased measurement independent of the platform.

Can I run multiple experiments simultaneously?

Yes, but careful coordination is required to avoid interference between tests. We help design non-overlapping experiments across different:

• Geographic regions

• Channels

• Time periods

• Audience segments

The number of simultaneous tests depends on your market size and spend levels.

More detail: How many experiments can and should I run at once?

What happens if my experiment shows no incrementality?

No incrementality can be valuable information, indicating: - The channel may not be driving additional value beyond what would happen naturally - You might be overspending in that channel - Budget could potentially be reallocated to more incremental channels We provide detailed analysis of why incrementality may be low and recommendations for optimization.

Your Growth Advisor will walk you through what the result suggests and whether a follow-up test with different parameters makes sense. For a fuller discussion, see What a no lift result means (and why it matters).

If we pause or end a geo experiment, should we remove the state exclusions or blackout setup?

Usually, yes. Once the planned experiment window is over or the test has been intentionally paused, the state exclusions or blackout setup have finished their job as a measurement tool. If you continue running the channel outside those states while the experiment is no longer active, you are usually no longer preserving a clean control group — you are just limiting delivery in a subset of markets.

• Remove the exclusions if the experiment is paused or finished and you are returning to business-as-usual delivery. In most cases, that is the correct operational step.

• Keep the exclusions only if your Growth Advisor has explicitly told you the test is still running, the read window is still open, or a follow-on design depends on those markets staying dark.

• If the experiment showed no incrementality, that result does not by itself mean the exclusions should stay in place. The result informs the next budget decision; the geo setup should still revert unless there is a specific follow-on test plan.

A simple rule of thumb: state exclusions are usually a temporary experiment construct, not a permanent campaign setting.

How do you account for external factors during experiments?

External factors like seasonality, competitor actions, economic shifts, and news cycles affect test and control markets simultaneously. Because a geo experiment compares the two groups during the same window, those external conditions cancel out in the comparison.

A few specific examples of how this works in practice:

• Seasonality: If conversion volume naturally rises in December across all markets, both test and control rise together. The difference between them still reflects the channel's incremental impact.

• Competitor campaigns: If a competitor launches a national campaign during your experiment, it affects both groups equally and doesn't bias the read.

• Macro events: A news event, weather disruption, or economic shift that moves the whole market up or down affects test and control together.

Where external factors can affect the read:

• Geographically asymmetric events: A hurricane in the Southeast during your experiment will distort markets in that region. If those markets are unevenly split between test and control, we may need to exclude them from the analysis or extend the experiment.

• Customer-specific events: A product launch, pricing change, or PR moment happening in specific markets during the test can bias results. Flag these to your GA before they happen.

• Channel-specific external campaigns: If you run a separate campaign (say, a national brand push) that correlates with the channel being tested, it can interfere. Coordinate with your GA on timing.

The design itself also includes balance statistics showing how closely test and control markets tracked in the pre-period. A well-balanced design is the first line of defense against external factors.

How long until I see results, and what do I do while the test is running?

Most experiments run for 4-8 weeks. During that window, you should not expect meaningful interim results. Looking at the data early almost always produces a misleading read, because statistical power builds over the full test duration.

What to do during the test:

• Do not change the campaigns being tested. No creative swaps, audience shifts, bid changes, or budget adjustments in test or control markets beyond what was agreed in the design. Changes mid-flight can compromise the read.

• Maintain consistent spend in control markets. The control group is your baseline. Anything that affects spend or performance in those markets (including correlated campaigns on other channels) can contaminate the measurement.

• Keep running campaigns outside the experiment as normal. National campaigns and campaigns in non-experiment geographies are unaffected.

• Let your GA know if anything unexpected happens in the business (PR event, product issue, major external development) so it can be factored into the analysis.

If you have extra budget land during the experiment or want to redeploy the held-back budget from a holdout, see Managing budget during a live experiment for what's safe and what's not.

At the start of the experiment, your campaigns typically go through a short ramp-up or learning period (3-7 days for most digital channels, longer for newly launched campaigns or large creative or budget shifts) while the ad platforms calibrate delivery, audience targeting, and bidding. Performance during this window doesn't reflect steady-state and tends to underrepresent the channel's true contribution in the early days. To minimize the impact, we recommend pre-staging campaigns where possible, launching on a Monday rather than mid-week, and avoiding launches adjacent to major holidays, pricing changes, or other business events. The ramp-up period is included in the test window by default — your GA will flag if a specific test should handle it differently.

After the experiment window ends, there is typically a short cooldown period (1-2 weeks) before analysis, allowing any lagged conversions from the test period to land. Full analysis follows after that.

What could prevent a test from launching?

Sometimes a customer wants to test a channel but the design doesn't produce a valid answer at achievable budgets. Common reasons:

• The channel is too small to show a detectable effect. If the channel's current spend is a tiny fraction of the marketing mix, a holdout there produces too small a signal to detect statistically. Either the test window needs to be very long, or a different test type is needed.

• Too few eligible markets. Geo tests generally need at least 10 eligible geographies on each side. Customers operating in very few markets, or with heavy geographic restrictions, may not have enough to form a valid design.

• Target metric volume is too low. If your target metric (signups, purchases, pipeline) has low overall volume, detecting a 5-10% lift requires a very long window or a very large effect. We may recommend aggregating to a higher-funnel metric or running the test for longer.

• Required budget exceeds what the business can support. The minimum detectable effect you want to measure drives the required budget. If the math lands at a number that isn't workable, your GA will show you the tradeoff curve: larger MDE targets mean smaller required budgets.

• Data quality gaps. If the historical data we use for matching test and control markets is sparse, inconsistent, or has known issues, we may need to resolve those before a design can be finalized.

If any of these apply, your GA will walk you through the specific constraint, what it would take to make the design viable, and whether a different test type (platform conversion lift, time-based holdout, or waiting for more data) is a better path forward.

What metrics should I focus on for experiments?

Primary Metrics: Focus on business outcomes like:

• Conversions/purchases

• Revenue

• Qualified leads

• App installs

Secondary Metrics: Track supporting metrics like:

• Traffic/site visits

• Engagement metrics

• Cost per acquisition

We help you select the right primary metric based on your business model and statistical power requirements.

How should I choose metrics for a launch-oriented or full-funnel experiment?

For a major launch, the best measurement plan usually combines one decision metric with a small number of leading indicators. This gives you an experiment that is grounded in business impact without forcing you to wait for the slowest possible KPI before learning anything.

Start with one primary metric that reflects the real business decision. For many B2B launches, that is not final revenue in the test window. It is often something closer to revenue but available sooner, such as qualified leads, pipeline created, or pipeline value at deal creation. If the true business question is "did this launch create more downstream value?" the primary metric should usually sit as far down-funnel as volume and timing allow.

Then add one or two in-flight indicators. These are not replacements for the primary readout. They are directional checks that help the team understand whether the launch is moving. Common examples include landing-page visits, free signups, form fills, or other upper-funnel volume metrics. They are useful because they land faster, but they should not become the success metric just because they are available first.

A practical full-funnel pattern looks like this:

• Primary decision metric: pipeline created, qualified leads, purchases, or another outcome the business actually optimizes for

• Secondary leading indicators: site traffic, landing-page sessions, signup volume, content engagement, or other early-funnel signals

• Diagnostic metrics: spend delivery, impression volume, reach, and campaign setup checks to confirm the test launched correctly

Avoid two common mistakes. First, don't choose a primary metric so delayed that the experiment can't produce a timely answer. Second, don't overreact to mid-test movement in fast metrics. Early reads are often noisy, and the purpose of the test is to evaluate the pre-committed primary outcome once the full window is complete.

If your team is split between an upper-funnel launch metric and a lower-funnel ROI metric, the usual answer is not to pick only one. It is to define both explicitly: one as the decision metric, and the other as a leading or contextual metric. That keeps the analysis honest and helps stakeholders understand what the launch changed across the funnel.

What if my ad platform doesn't support location exclusions in a specific country or region?

This comes up occasionally, especially outside the US. Some ad platforms do not support the same geographic controls in every market, so the execution plan may need to change even if the experiment logic stays the same.

When that happens, the first question is not “can we exclude this one region?” but “can we still create a clean separation between treatment and control?” In many cases, the answer is yes — just not with the exact campaign structure you would use elsewhere.

Common workarounds include:

• Rebuilding the campaign structure around included geographies rather than excluded geographies. Instead of running one campaign broadly and excluding control markets, you may duplicate campaigns and positively target only the test markets that the platform allows.

• Changing the geographic unit. If state-level exclusions are unavailable, another geography level supported by the platform may still work for the test.

• Adjusting the experiment design before launch. If a specific region cannot be cleanly separated, your Growth Advisor may swap markets, redesign the cells, or recommend a different setup that preserves a valid comparison.

• Using a different methodology. If the platform cannot support a clean geo split in that market at all, a platform-native lift study, a different channel test, or a different measurement approach may be more appropriate.

The key principle is that the test and control groups must remain operationally distinct. If the platform limitation makes that impossible, we would not recommend forcing the launch as originally designed. Instead, we would revise the setup so the measurement remains credible.

If you already know there are geographic targeting limitations in a country or region, flag them before the design is finalized. That lets us incorporate the constraint early rather than rewriting the setup at launch time.

Can adjacent or non-contiguous regions still work for a geo holdout?

Yes. A geo experiment does not require test markets to be next to each other, and it also does not require them to be far apart just for appearances. What matters is whether the setup creates a clean operational split and whether the pre-period data shows the test and control groups are well matched.

In practice, a design using regions like Central Scotland and the Midlands can be completely valid if three conditions hold:

• The platform can target those regions cleanly. If the campaign setup cannot reliably include or exclude the selected regions, the design may need to be reworked before launch.

• The historical data supports a strong match. The chosen test regions need a credible control group that tracked similarly before the experiment started.

• Spillover is limited. If media delivery, store traffic, or audience behavior regularly crosses those regional boundaries, the read can become noisier and the design may need buffer regions or a different geographic unit.

The main question is not whether the regions look intuitive on a map. It is whether the experiment can preserve a clean difference in treatment and produce a statistically credible comparison. If those conditions are met, separated or adjacent regions can both be workable.

If you have a preferred region list for business reasons, share it early. Your Growth Advisor can usually test whether that constraint still produces a viable design and explain the tradeoff if it weakens the match.

How do we execute a geo experiment?

We will include instructions on how to set up your campaigns and set budgets appropriately. Often, you will use the ad platform’s targeting capabilities to create exclusions. Sometimes, this requires duplicating campaigns to ensure that different treatment groups are receiving the correct treatments.

What if my ad platform can't exclude or refine location targeting in some markets?

Some platforms and countries have geo-targeting limitations. For example, a platform may not support location exclusions or refinements in a market like South Korea. When that happens, we do not force the original market split. Instead, your Growth Advisor will usually adjust the design so the experiment only uses markets the platform can target cleanly.

In practice, that usually means one of three things:

• Revise the eligible market list. Remove unsupported geographies from the test and control pool, then rebuild the design around the remaining markets.

• Change the campaign setup. In some cases, duplicating campaigns or restructuring location targeting can still produce a clean split if the platform supports it.

• Switch methodologies. If the platform cannot support a credible geo split, we may recommend a different test type or a different channel rather than running a compromised experiment.

If you already know there are countries, states, or regions where the platform can't target or exclude cleanly, flag that before the design is finalized. Those constraints can usually be incorporated, but they reduce the eligible market pool and may change the required budget, duration, or test structure.

What’s the difference between a feasibility analysis and a proof of concept (POC)?

A feasibility analysis happens before a full experiment design or launch. It is a quick assessment of whether a channel, market setup, and target metric are likely to produce a useful read at all. It typically looks at the ingredients that make testing possible: how much historical data exists, whether there are enough eligible geographies or audiences, whether the target metric has enough volume, and whether the likely budget and duration are in a range the business can support.

A proof of concept (POC) is a larger commitment. It usually means moving from “is this testable?” to “let’s actually run a live measurement exercise and learn something decision-worthy.” In Paramark terms, that normally means an experiment design with a defined methodology, test and control setup, power expectations, budget, duration, and a pre-committed decision rule.

A useful shorthand is:

• Feasibility analysis: Can this be measured credibly, and what would it take?

• POC: Are we ready to spend the time and budget to run the live test and act on the result?

This is also why a simple monthly correlation exercise often does not showcase the value on its own. Correlation can help us understand whether there is enough signal in the data to justify a design conversation, but it does not establish causality. Its role in a feasibility pass is to help estimate whether the channel looks measurable, not to serve as the final proof that the channel is incremental.

If a feasibility analysis suggests the signal is too weak, the usual next step is not to force a POC. It is to change the test structure: choose a different metric, extend the duration, increase the treatment size, switch methodologies, or wait for more data.

Can my primary experiment metric be a custom business event like a publish, activation, or completed onboarding step?

Yes. In many businesses, the best primary metric is not a generic top-funnel conversion like a click or signup. It is the first downstream event that represents real value creation for the business — for example, a published fundraiser, an activated account, a completed onboarding milestone, or another event that marks a customer becoming truly live.

Choose the metric that matches the decision you are trying to make. If the business question is whether a channel drives more meaningful activations, the primary metric should usually be that activation event rather than a noisier proxy upstream.

For geo experiments, custom business events are often ideal. If the event can be measured consistently by geography over time, it can usually serve as the decision metric even when it happens later in the funnel.

For platform-native conversion lift studies, the key question is observability. The platform must be able to observe the event reliably through its tag, CAPI, server-side event feed, or another supported conversion integration. If the custom event cannot be observed cleanly, a geo-lift test is usually the more credible measurement path.

A practical pattern is to use one custom decision metric plus one or two faster leading indicators. For example, you might make publish the primary KPI, while also monitoring earlier funnel steps such as landing-page visits, signups, or form completions for directional context.

The main rule is to avoid choosing an upstream proxy just because it lands faster. If a downstream custom event is the KPI the business actually optimizes for, that is usually the metric worth designing the experiment around — as long as there is enough volume and the event can be measured credibly.

How many states or DMAs do I need in a holdout, and which ones should they be?

There is no universal minimum like "hold out 5 states" or "use 10 DMAs." The right number depends on whether the experiment can still produce a strong pre-period match and enough statistical power for your target metric.

In practice, Paramark sizes the holdout based on four things:

• How many eligible geographies you have. Geo tests usually work best when there are enough markets to form a credible test and control split. If the eligible pool is too small, the design can become fragile no matter how the markets are arranged.

• The volume of the KPI in each geography. Sparse outcome metrics require either more markets, more time, more spend differential, or some combination of the three.

• The lift you need to detect. If you want to detect a smaller effect, the design usually needs more signal and less restrictive geography choices.

• The targeting unit of the channel. State-level designs are common for digital channels, while DMA-level designs are more common when the media is naturally bought and controlled at the DMA level, such as linear TV.

Which states or DMAs should be in holdout? Not necessarily the biggest ones, and not necessarily a fixed percentage of the country. Paramark selects the markets that create the cleanest statistically matched design after accounting for business constraints, platform targeting limitations, and historical KPI behavior. Sometimes that means a small number of high-signal markets are enough; other times it means using a broader set of geographies to preserve power and balance.

A good customer-facing rule of thumb is: start from the business constraint (for example, "we want fewer holdout markets if possible"), then re-run the design to see the tradeoff in required budget, duration, and minimum detectable effect. If a narrower holdout weakens the match or makes the test underpowered, we will recommend broadening the geography set rather than forcing an unreadable design.

If my campaign had a messy launch or payment issue, can we reset the experiment start date?

Usually, yes — if the disruption happened right at launch and materially changed delivery, the safest move is to treat the experiment as starting only once the campaign is actually running as intended.

What matters is not the calendar date you originally planned. What matters is the first date when the treatment was live, stable, and representative of the conditions the test is supposed to measure.

• Good reasons to reset the start date: a payment failure, campaigns not serving, major launch-day setup errors, platform delivery problems, or another issue that prevented the intended spend or treatment from actually reaching the market.

• Usually not a reason to reset the start date: normal day-to-day variance, minor pacing differences, or small performance fluctuations once the campaign is already live and behaving normally.

• If the disruption created a mixed window, do not count the broken days as part of the official measurement period unless your Growth Advisor explicitly confirms that the test can still be read cleanly.

A practical rule of thumb is: start the clock when the experiment is truly in market, not when it was supposed to be in market. If launch-day issues mean the first few days were not representative, those days are usually better treated as pre-start noise rather than part of the measured test window.

If you think this happened in your test, tell your Growth Advisor what changed, when delivery became stable again, and whether any holdout or control conditions were affected. We can then confirm whether the right fix is a simple start-date reset, a longer runtime, or a revised analysis plan.

What data do you need from me before you can build an experiment design?

Usually, Paramark needs enough historical data to answer three basic questions: what outcome you want to measure, which geographies are eligible for the test, and how much channel activity has been running in those geographies.

In practice, that usually means sharing:

• historical target-metric data by geography and date (for example leads, purchases, pipeline, or another KPI)

• channel input data by geography and date such as spend, impressions, or another channel-volume field that reflects delivery

• business context that affects feasibility such as platform targeting limitations, required holdout constraints, or upcoming launches that could interfere with the test

If a few fields are still outstanding, Paramark can often begin the design conversation anyway. For example, if most of the geography-level data is already available but a field like inbound call counts or Meta impressions is still being finalized, we can usually review the structure, identify likely constraints, and clarify what remains blocking before the test can be fully sized or launched.

The key is to be explicit about what is complete versus still in flight. A partially complete dataset is often enough to start planning; a silently incomplete dataset is what creates confusion later.

What information do you need if I want to run a regional Performance Max incrementality test?

If you want Paramark to assess whether a regional Performance Max (PMax) incrementality test is viable, the most helpful starting point is a list of the exact campaigns you plan to include, the regions where each campaign is currently live, and the KPI you want the experiment to measure.

In practice, we usually want to confirm:

• The exact campaigns in scope. Share the campaign names you want included in the test. For example, a starting list might look like this:

  • T1-AO-EXP-DG-SS-Prospecting-Display-Google-Performance_Max_Revamp_Marketing

  • T1-TF-AO-DG-SS-Prospecting-Display-Google-Performance_Max_Revamp_Broad

  • US-TF-AO-DG-SS-Prospecting-Display-Google-Performance_Max_Revamp_Marketing

  • US-TF-AO-DG-SS-Prospecting-Display-Google-Performance_Max_Revamp_Broad

• The regional setup. Tell us which regions are live today, which regions could be test versus control, and whether any geographies must stay on or off for business reasons.

• The primary KPI. For example: purchases, signups, qualified leads, revenue, or another business event.

• Historical KPI data by geography. To move from an initial review to an actual experiment design, we usually need the target metric broken out by geography and time (for example by state or region over multiple weeks or months).

• Recent delivery context. Current spend, conversion volume, and whether the campaigns are stable or still ramping after a recent launch, rebuild, or major change.

• Execution constraints inside Google Ads. For PMax specifically, we need to know whether the geography split can be implemented cleanly and whether budget would stay stable enough during the test to preserve a valid comparison.

If you already have screenshots or a list of the live campaigns, that is often enough for an initial review. From there, Paramark can confirm whether the setup looks directionally workable and what additional geography-level KPI history, if any, is still needed before the design is finalized.

One important caveat for PMax: because Performance Max reallocates spend dynamically across Google properties, a clean geo experiment depends on holding the setup operationally stable while varying by geography. If the campaigns are still in a learning period, being restructured, or likely to shift materially during the test, it is usually better to stabilize first and then launch the experiment.

For PMax, should I run an experiment next or just gradually increase spend?

It depends on the decision you need to make and how stable the current PMax setup is.

Choose a fresh experiment first when the question is strategic and high-stakes: for example, whether PMax deserves materially more budget, whether a recent rebuild changed its true incremental value, or whether you need a causal answer before scaling. A pulse-up experiment is usually the cleaner path when you want to know whether additional PMax spend creates incremental lift.

A gradual spend increase can be reasonable when the budget change is modest, the campaigns are already stable, and the goal is operational scaling rather than a clean measurement read. But a gradual increase is not the same thing as an incrementality test. It may help you pace into higher spend safely, but on its own it does not tell you whether the added budget caused incremental business outcomes versus simply coinciding with them.

A practical rule of thumb:

• If you need a defensible measurement answer, run the experiment.

• If you mainly need to scale carefully and can tolerate less certainty, a gradual increase may be fine operationally.

• If the campaigns were recently restructured or are still learning, stabilize first before doing either.

For many teams, the best sequence is: stabilize PMax, decide whether the next question is about measurement or execution, and then either run a pulse-up experiment or increase spend gradually with the understanding that the result will be directional rather than causal.

How much budget would a U.S. Pinterest geo holdout need to reach statistical significance?

There is no reliable universal number like “you need $X on Pinterest” because the required budget depends on the signal in your data, not just the platform name. A U.S. geo holdout on Pinterest is usually sized from the expected lift, the current Pinterest spend level, the target metric volume, the number of eligible geographies, and how noisy those geographies are in the pre-period.

In practice, Paramark estimates the budget by re-running the experiment design with your actual inputs. The most important ones are:

• Current Pinterest spend by geography so we know the baseline treatment size available

• The KPI you want to measure such as purchases, revenue, signups, or another business event

• Historical KPI volume by geography and date so we can assess variance and statistical power

• The minimum lift worth detecting because detecting a smaller effect usually requires more budget, more time, or both

• Execution constraints such as whether the test must stay state-level, how many markets can be held out, and whether campaign targeting can cleanly separate test and control

A useful customer-facing rule of thumb is: the budget required to “hit stat sig” is whatever budget creates a detectable enough difference between treatment and control for the chosen KPI in the available geographies. If Pinterest spend is small relative to the business outcome being measured, the design may need a longer runtime, a larger treatment difference, a higher-funnel KPI, or a different test structure rather than simply a bigger headline budget.

If the question is specifically “can we test Pinterest in the U.S. at our current spend?”, Paramark would usually treat that as a feasibility/design question rather than answer with a generic benchmark. The right next step is to size the design using the customer’s actual geo-level spend and KPI history, then show the tradeoff among required budget, runtime, and minimum detectable effect.

Can Paramark design a US geo holdout test for Pinterest, and how much budget would it take to reach statistical significance?

Usually, yes — a US Pinterest geo holdout can be a good fit when Pinterest does not have enough scale or operational support for a platform-native lift study, or when the business wants an independent read on channel incrementality.

The important caveat is that there is no universal Pinterest budget threshold that guarantees significance. The required budget depends on the specific design inputs:

• your current Pinterest spend level and how much of a treatment difference the test can create

• the target metric volume by geography (for example purchases, signups, pipeline, or another KPI)

• how many eligible US geographies can be cleanly split into test and control

• the minimum detectable lift the business cares about

• the conversion lag of the metric, which often matters on Pinterest because users may convert well after the initial ad exposure

Because Pinterest often sits higher in the funnel and has a longer consideration cycle than channels like paid search, the tradeoff is usually not just budget. It is often budget plus runtime plus cooldown. A design that is underpowered at 4 weeks may become viable at a longer duration, or with a larger treatment difference, or with a higher-volume KPI.

A practical rule of thumb is: Paramark would size the budget from the desired lift threshold rather than start from a fixed Pinterest benchmark. If the design comes back too expensive, the usual levers are to target a larger detectable effect, extend the runtime, broaden the eligible geography pool, or choose a different KPI that has more volume.

For many Pinterest tests, the real question is not “what is the magic budget to hit significance?” but “what combination of spend, duration, and KPI gives us a readable test?” That is exactly what the experiment design process is meant to answer.

If you are trying to compare a Pinterest geo holdout against a Pinterest conversion lift study, a useful shortcut is this: platform-native lift studies often need substantial spend and rep support, while a geo holdout is usually the more flexible option when you want an independent measurement read on downstream outcomes.

How much would we need to spend in a place like Utah to measure lift from a digital out-of-home campaign?

There usually is not a reliable universal benchmark like “spend $X in Utah and you will see lift.” For a regional digital out-of-home (DOOH) test, the required spend depends on the signal in your data and the exact experiment design, not just the market name.

In practice, Paramark would size the design from a few inputs:

• the target metric volume in the market such as purchases, leads, signups, or another KPI by geography and date

• the minimum lift worth detecting because a smaller effect requires more spend, more time, or both

• the geographic unit of execution such as states versus DMAs, which matters for DOOH and other upper-funnel channels where spillover can blur adjacent markets

• the planned duration because longer tests can often work with a lower weekly spend than a short burst

• the current baseline media plan and whether the campaign can create a clear treatment difference versus control

A good customer-facing rule of thumb is: the right baseline is whatever spend creates a detectable enough treatment difference for the KPI you care about in the geographies you can cleanly test. If the proposed DOOH spend is too small relative to the business outcome being measured, the test may need a longer runtime, a larger treatment difference, a higher-volume KPI, or a different structure rather than just a bigger headline budget.

Because digital out-of-home is typically a more upper-funnel and geographically sensitive channel, the design conversation often focuses on budget plus duration plus geography setup, not just one minimum spend number. If the question is specifically “what should we spend in theory as a baseline in Utah?”, Paramark would usually treat that as an experiment design / feasibility question and size it using the customer's actual geography-level KPI history rather than answer with a generic market benchmark.

We fixed geo-targeting issues and rebuilt Google Ads campaigns. When is it safe to launch the experiment, and what should we validate first?

Usually, wait until the campaigns are operationally stable enough that the test is measuring the experiment rather than setup noise. If you have just rebuilt Google Ads campaigns, fixed geo-targeting problems, changed conversion-event mapping, or are still validating offline uploads, the safest move is usually to finish those checks before starting the experiment clock.

For Google channels like Search, YouTube, Demand Gen, and PMax, the pre-launch validation checklist is usually:

• Geo setup is clean. Test and control regions must be targetable exactly as designed, with no accidental overlap, missing exclusions, or unsupported regions that would blur the treatment split.

• Conversion events are finalized. The event you plan to optimize toward or measure should already be the event you trust. Do not change the primary conversion definition mid-test unless you are prepared to reset the launch timing.

• Offline uploads or enhanced conversion feeds are landing normally. If offline conversion imports are delayed, incomplete, or still being QA'd, wait until they are stable enough that platform optimization is using the intended signal.

• Campaign delivery has settled after the rebuild. A short post-change stabilization window is normal so bidding, audience discovery, and budget pacing can normalize before measurement begins.

A good rule of thumb is: start the experiment only after the geo execution, optimization signal, and reporting inputs all reflect the setup you actually want to test. Otherwise, the early part of the experiment may mostly capture launch turbulence rather than incremental lift.

If some pieces are complete but one dependency is still in flight (for example, geo targeting is fixed but offline upload validation is not done), it is usually better to delay the start briefly than to launch and later wonder whether the read was contaminated by setup changes.

This is especially true when recent fixes changed how geographic data is mapped or cleaned outside the U.S. A geo setup that looked workable before the fix may become materially more reliable after the underlying regional data issue is resolved, so it is worth re-checking the final market split, confirming the updated geo inputs are flowing correctly, and only then locking the launch plan. The same logic applies if you are improving Google Ads quality signals through better conversion-event mapping or offline uploads: finish the signal QA first, then start the experiment on the version of the setup you actually want to evaluate.

How much would we need to spend to test a regional digital out-of-home campaign in a market like Utah?

There is no reliable universal number like “spend $X in Utah and you will see lift.” For a regional digital OOH test, the required spend depends on the lift you want to detect, the volume of the KPI you are measuring, how much of the market your OOH plan can actually cover, and whether the geography can be split into a clean test and control design.

A practical way to think about it is: the campaign needs to create a large enough treatment difference in the selected market for the business metric to move beyond normal week-to-week noise. In practice, Paramark would usually size this from:

• The target metric and its baseline volume. If the KPI is sparse, the test needs more spend, more time, or both.

• The minimum detectable lift that matters to the business. Detecting a smaller effect requires a larger budget and/or longer runtime.

• The concentration of OOH delivery in the market. If the buy is spread too thinly across screens or sub-regions, the treatment may not be strong enough to measure cleanly.

• The geography structure. A single-state test like Utah may still work, but for OOH we also need to think about spillover, neighboring-market contamination, and whether the comparison should be built at the state, city, or DMA level.

• The experiment duration. Top-of-funnel channels like OOH often need a longer in-market window than lower-funnel digital channels to accumulate enough signal.

So if the question is “what baseline spend would you recommend in theory?”, the safest customer-facing answer is: treat this as an experiment design and feasibility question, not a generic benchmark question. Paramark would size the required spend from the customer’s actual geo-level KPI history and the proposed OOH footprint, then show the tradeoff among budget, runtime, and minimum detectable lift.

A good rule of thumb is that regional OOH usually needs enough spend to create meaningful local saturation, not just light presence. If the proposed Utah budget would only buy thin coverage, the better recommendation is often to either concentrate the spend more tightly, extend the runtime, choose a higher-volume KPI, or redesign the test rather than force a weak read.

Can we run a geo experiment outside the U.S., in places like South Korea, Japan, or Taiwan?

Sometimes, yes — but there is not a universal country whitelist where geo experiments always work the same way. The practical question is whether the ad platform and campaign structure can create a clean geographic split in the market you care about.

A simple rule of thumb is:

• United States: Usually the easiest place to run geo experiments because most major platforms support mature state- and DMA-level targeting controls.

• South Korea: Sometimes workable, but some platforms have geo-targeting or exclusion limitations there, so the original design may need to be reworked.

• Japan and Taiwan: Potentially workable, but feasibility depends on the platform's local geo controls, the geographic unit available, and whether the market can be split cleanly enough for test versus control.

In practice, Paramark checks three things before saying yes:

• The platform can target or exclude the needed geographies cleanly.

• There are enough eligible regions with enough KPI volume to build a credible design.

• The setup avoids spillover or execution blur between test and control regions.

If a country does not support the necessary geo controls, the answer is usually not "never test there." The answer is often to change the campaign structure, change the geographic unit, or use a different measurement method rather than force a weak geo split.

So the safest customer-facing answer is: yes, geo experiments can often be run outside the U.S., but market eligibility is platform- and setup-dependent rather than country-name-dependent. If you already know which platform and country are in scope, share that early so the design can account for any local targeting limitations before launch.

How should I design a paid search experiment if the account was recently disrupted or rebuilt?

If the paid search account was recently deleted, rebuilt, or otherwise disrupted, the first goal is to separate recovery from incrementality. A test launched too early can end up measuring the platform’s return to normal delivery rather than the effect of the experimental treatment itself.

A practical rule of thumb is: stabilize first, then test. Before starting the experiment clock, confirm that campaign structure, geo targeting, conversion tracking, bidding, and budget pacing are all operating the way you intend to keep them during the test.

For paid search specifically, the design conversation usually starts with three questions:

• Has the account returned to steady-state delivery? If the deletion or rebuild changed campaign structure, match types, bidding logic, or conversion inputs, wait until performance has normalized enough that the experiment is not mostly capturing relearning.

• What is the right primary metric? If lead volume is stable but downstream quality has changed, the better decision metric is often farther down-funnel — for example SQLs, opportunities, or pipeline — as long as there is enough volume to power the test.

• Are other major changes about to land? If lead scoring, offline conversion improvements, or Enhanced Conversions for Leads will materially change optimization or measurement, it is usually better to either launch before those changes or wait until after they are live and stable, rather than introduce them mid-test.

In practice, Paramark would usually recommend a short stabilization window after the disruption, confirm that the target metric and conversion inputs reflect the setup you actually want to evaluate, and then size the experiment around the downstream KPI that matches the business decision. If the account is still recovering or another major optimization change is imminent, the safest move is often to delay launch briefly rather than risk an unreadable test.

When will we know whether a partner acquisition campaign is working, and when should we decide on expansion or an activation follow-on?

Usually, you should separate this into three checkpoints: launch stability, measurement readout, and next-step planning.

• Launch stability: In the first few days after launch, focus on whether the campaign is serving correctly, geo setup is clean, budgets are pacing as intended, and tracking is landing. This is an execution check, not a performance verdict.

• Measurement readout: A real yes/no read on incrementality usually comes only after the planned test window has run and the analysis period is complete. For many experiments, that means several weeks in market plus a short post-period for lagged conversions and analysis. Looking too early often tells you more about ramp-up noise than true lift.

• Next-step planning: You do not need to wait until the final deck is delivered to plan what happens next. It is reasonable to line up the decision framework in advance: what result would justify expanding the partner campaign, what result would argue for holding flat, and what evidence would be needed before launching a separate activation campaign.

A practical rule of thumb is:

• Ask about campaign health immediately, if the concern is launch quality or spend delivery.

• Ask for a directional read only with caution, if the campaign is still mid-flight.

• Make the scale / no-scale decision after the pre-committed readout window, once the agreed primary metric has had time to mature.

For an activation campaign, the safest approach is usually not to treat it as an automatic next step just because a partner acquisition campaign launched. The better question is whether the first campaign has produced enough signal to justify a second test or expansion, and whether the activation campaign is answering a distinct business question rather than overlapping with the original measurement window.

If a team needs to coordinate timing in plain English, a simple way to frame it is: check execution now, expect the real measurement answer after the planned experiment window, and decide on expansion or an activation follow-on only once that readout is mature enough to support a budget decision.

How should I design a one-variable geo pulse-down test for non-brand search when the business risk is high?

When the question is specifically whether reducing Google non-brand search creates a negative business impact, the cleanest design is usually a single-variable geo holdout or pulse-down, not a simultaneous reallocation into another channel.

The reason is simple: if you cut non-brand search and increase another channel at the same time, you are no longer measuring one decision. You are measuring two moving parts together, which makes it harder to tell whether the result came from the search reduction, the added channel, or both. If the business wants a confident answer on the downside risk of lowering non-brand search, isolate that question first.

A practical setup usually looks like this:

• Treatment: reduce or pause non-brand search only in the selected test geographies.

• Control: keep non-brand search running as business-as-usual in the matched control geographies.

• Everything else: keep other meaningful channels and campaign settings as stable as possible across both cells during the test window.

If you plan to launch or rebuild campaigns right before the test, give them a short baseline period first. For paid search, it is often smart to let the relevant campaign setup run at normal spend long enough to exit the platform's learning or ramp-up period before starting the pulse-down. Otherwise, the early test window may reflect setup turbulence rather than the effect of reducing non-brand search.

What about kill switches or mid-test check-ins?

For a high-stakes pulse-down, it is reasonable to define those before launch as operational guardrails. For example, the team may agree on one or two scheduled checkpoints and a pre-committed rule for stopping the test if business performance drops beyond an unacceptable threshold. The important part is that the guardrail is decided in advance rather than improvised from noisy day-to-day fluctuations.

A good rule of thumb is:

• Use checkpoints for risk management, not for declaring the experiment “won” early. Looking for success or failure too early can be misleading because most experiments need the full planned window to produce a confident read.

• Base the kill switch on business protection, not statistical significance. The purpose is to cap downside risk if the impact is clearly unacceptable, even if the formal experiment readout would still come later.

• Document the trigger clearly before launch. For example: if the primary KPI or another agreed business safeguard metric falls below a defined threshold at a scheduled checkpoint, revert to business-as-usual and treat the test as intentionally stopped.

The practical takeaway is: if the business question is “can we safely turn down non-brand search?” start with a one-variable pulse-down test, allow the campaign setup to stabilize before the test begins, and define any kill-switch rules up front. If that test shows no meaningful downside, a separate follow-on experiment can then test whether reallocating budget into another channel creates more value.

Want to learn more about experimentation? Check out other Paramark experimentation articles here