Let’s say you’re trying to answer, “What’s the impact of this channel on our overall growth strategy?” That’s a broad, long-term question, which requires a universal holdout test that runs over time and captures the collective effect of your channel strategy. It’s a zoomed-out, strategic view.
But if you’re asking, “How did this one campaign perform against its control group?”, that’s a campaign-level test, and it’s all about the here-and-now. It’s narrower, more tactical, and focused on short-term lift that can inform creative and offer testing.
Both are valid questions, but they require different methodologies, different audiences, and different timeframes.
You can’t stitch together campaign-level results and call it a strategy
We see this mistake all the time: Marketers run multiple campaign-level incrementality tests and then try to average the results into some kind of channel-level insight. On paper, it feels logical—stack up a few lifts, do a little math, boom: strategy. But in practice? It doesn’t work.
Why? Because each campaign-level test is measuring a different audience at a different moment in time, under different conditions. One campaign may target first-time buyers in spring. Another might focus on lapsed customers during the holiday season. Even if they’re both technically running in the same channel, they’re measuring different things. Trying to average them out is like blending apples, oranges, and pineapples and calling it a banana. It’s a fruit salad, not a conclusion.
If your goal is to understand the overall impact of a channel, you need a universal holdout test. This means creating a single, consistent control group—ideally drawn from the pool of people you expect to target over the full campaign timeline—and then measuring performance across all campaigns against that same group. That’s how you capture real, comparable, channel-level lift.
There’s no such thing as a ‘single-channel consumer’ anymore
Unlike the story of a Peruvian tribe discovered in 2024 with no prior outside contact, there’s no segment of U.S. consumers who are only reachable through a single marketing channel. This means that the desire some marketers have for each new paid channel to only bring in net incremental consumers is not realistic.
When you’re measuring the lift from your Meta campaigns, for instance, you’re not measuring it in a vacuum. Those same consumers are likely seeing your YouTube ads, scrolling past your email, or receiving your postcards in the mail. This omnipresent media environment means that incrementality is increasingly about channel interaction, not just isolated performance.
Marketers must not expect to find the consumer version of the Peruvian tribe—a purely incremental group of buyers never reached before by paid media.
Don’t chase high incrementality—find your Goldilocks zone
It’s tempting to look at a high incrementality number and say, “Let’s put all our budget there.” But that’s not strategy—that’s tunnel vision.
Smart marketers look for balance. If one channel is doing all the heavy lifting while others lag behind, it’s not time to cut the weak links. It’s time to investigate. The point is to balance the entire mix—not just overfeed the strongest performer.