Incrementality Testing Without a Data Science Team: A Founder’s Guide
A founder told me his attribution model showed Google was driving most of his sales. His plan: cut Meta and double down on Google.
I asked one question: “What happens to your Google sales if we switch Meta off?”
He said: “Wouldn’t it just shift more sales to Google?”
He was wrong. And the only way to know that was to run a test.
Attribution models divide credit across touchpoints. They cannot tell you whether a conversion would have happened without the channel. That distinction matters more than most founders realise. When you optimise on attribution alone, you risk cutting channels that are quietly driving demand while doubling down on channels that are mostly capturing it.
Incrementality testing fixes this. It isolates the causal impact of a channel by comparing a group exposed to your marketing against one that is not. The answer it produces is simple: did this actually drive new revenue, or was it along for the ride?
This article gives you a practical, three-stage framework to run your first incrementality test. No data science team required.
1. Why Your Attribution Dashboard Cannot Answer the Most Important Question
Attribution models observe conversions and assign credit based on rules: last click, first click, data-driven. The problem is they cannot separate a channel that caused a conversion from one that simply touched someone already going to buy.
Consider brand search retargeting. Someone visits your site, leaves, and sees a Google ad. They convert. Google claims the credit. But that person was already in market. The ad may have done nothing except cost you money. Attribution has no way to flag this.
This is not a criticism of Google or Meta. Platform attribution is designed to help you optimise within a channel. It was never built to tell you whether the channel itself is earning its place in your budget.
💡 Key Takeaway: Attribution answers “where did the conversion touch?” Incrementality answers “would the conversion have happened without us?” Only one of those is causal. Only one should drive budget strategy.
2. The Three-Stage Test-and-Hold Framework
The core logic of incrementality testing is simple. You split your audience or geography into two groups. One group sees your marketing. The other does not. You compare the outcomes.
Here is the framework in three stages:
Stage 1: Prepare
- Choose one test variable: one channel, one campaign, or one message
- Split your audience or geography into a test group and a control group
- Make both groups as similar as possible in size and baseline revenue
- Set your measurement window: four to eight weeks works for most campaigns
Stage 2: Intervene
- Run your marketing only to the test group
- Keep everything else identical: budget, creative, other channels
- Do not adjust the control group during the test
- Record your baseline metrics before the test starts
Stage 3: Measure
- Compare outcomes between test and control at the end of the window
- Calculate incremental lift: (Test result minus Control result) divided by Control result
- Lift near zero means the channel is capturing existing demand, not creating new demand
- Meaningful lift means the channel is genuinely incremental
The fitness analogy makes the logic concrete. Incrementality testing is like pausing one supplement in your training plan for a month while keeping your workouts and diet identical, then comparing results with a friend who keeps taking it. Attribution is the label on the bottle claiming benefits. The test is the experiment that shows whether it does anything on top of your existing effort.
💡 Key Takeaway: The framework is Prepare, Intervene, Measure. You do not need a data science team. You need clean group splits, a fixed measurement window, and the discipline to leave the control group alone.
3. Geo Tests: The Easiest Version to Start With
The most accessible incrementality test for a founder-led company is a geo test. You split markets by geography instead of by individual users. This removes the complexity of building audience holdout logic inside ad platforms.
Here is how to set one up:
- Select two or three comparable regions with similar baseline sales volume
- Designate one region as test, one as control
- Run your campaign in the test region only
- Keep all other variables identical across both regions
- After four to eight weeks, compare sales data by region
The question you are answering: does running this channel in a market produce more revenue than not running it?
Geo tests work for a single channel audit. They also work for the channel stack question the founder in my opening story needed to answer: what happens to total revenue if we remove Meta entirely? Run Meta in one region. Hold it out in another. Measure the gap. You now have a causal read, not a platform attribution estimate.
You do not need advanced tooling. You need sales data tagged by region and a clean test setup.
💡 Key Takeaway: Split by region, not by user. Run for four to eight weeks. Compare sales data. That is the entire method.
4. What to Do When the Results Are Uncomfortable
The hardest outcome from an incrementality test is discovering that a high-performing channel is not incremental. Your attribution dashboard shows strong return on ad spend. Your test shows near-zero lift. This is not a data problem. It is a strategic problem.
Common findings and what they mean:
- Near-zero lift on brand search: You are paying to capture people who would have found you anyway. Consider reducing brand keyword spend and tracking whether revenue holds steady.
- Near-zero lift on retargeting: Your retargeting is reaching people already in the buying process. The ads are adding cost, not accelerating decisions.
- Positive lift on a channel you were about to cut: Your attribution model was under-crediting it. The channel was working. You nearly made the wrong call based on the wrong signal.
The goal is not to validate your current spending. It is to give you a causal read on what is actually driving growth. That read will sometimes force you to cut a channel you are comfortable with. That is the point.
Final Thoughts: You Do Not Need a Data Science Team to Make Causal Decisions
Most founders are flying on attribution data that tells them where credit goes. Very few are running tests that tell them whether credit is deserved. That gap is where budget gets wasted and growth stalls.
Incrementality testing is not a big-brand luxury. The three-stage framework: Prepare, Intervene, Measure. A four-to-eight-week geo test. Basic sales data by region. That is enough to answer the questions that attribution cannot.
If you are making channel budget decisions based on platform ROAS alone, you are optimising the wrong signal. Start with one test. Pick one channel. Hold out one region. The result will either confirm your instincts or save you from a costly mistake.
To explore what this looks like for your specific growth model, book a discovery call or connect with me on LinkedIn.
A note before you close this tab. The fact that you read this far tells me something. You already sense that the way you’ve been thinking about growth might be incomplete. That instinct is worth following.
Mervyn Chua is a growth-transformation consultant helping founders and CEOs build the strategic clarity and systems to grow in an AI-first world. If this raises questions worth exploring for your brand, let’s talk.
