The Incrementality Test You Can Run Without a Data Team

I once switched off a channel on purpose, just to see what would happen. Total sales held. Then the return on ad spend (ROAS) of every other channel went up.

The channel had been taking credit for sales that would have happened anyway. The dashboard had no way to show me that. Only the pause could.

Most growth leaders know they should test incrementality: the sales a channel causes, as opposed to the sales it gets credit for. Then they hear what it takes. A marketing mix model (MMM). A vendor. A data scientist with a free quarter. So the test waits, and the budget keeps flowing on attribution.

You can run this test yourself. The question that moves budget is blunt: is this channel’s real contribution close to zero? A blunt question needs a blunt test. Here are five steps, and the limits to know before you start.

1. The budget question needs a big answer, not a precise one

A model tries to price every channel to the decimal. Your budget decision rarely needs that. It needs to know which lines nobody would miss.

Google has run this kind of test for years. Its researchers describe geo experiments plainly: split regions into two groups, change the advertising in one, compare the two. They call the design “conceptually simple”, with results that are “easy to interpret”.

The trade-off is sensitivity. A simple test sees big effects and misses small ones. For a budget cut, that is enough. If switching a channel off moves nothing you can see, the burden of proof shifts to the channel.

💡 Key Takeaway: Ask the blunt question first: would anyone miss this channel? A simple test answers it well.

2. Pick one big line, then split cleanly

Test your largest budget line, or the one you trust least. Small lines produce small effects, and small effects vanish in the noise.

Then choose the split:

• By region, if you sell across enough places. Meta’s open-source GeoLift tool recommends 20 or more regions, with at least 25 periods of history before the test.
• By time, if you do not. Switch the channel off for a set period, back on, then off again. Compare each off period with the on periods around it. Time splits are cruder, so run more cycles.

A single-city market like Singapore rarely splits into 20 clean regions. There, time is your split.

Hold back a minority, never the whole market. In one published case, a brand switched off brand search in 30% of its US regions and left it running in the other 70%. The risk stays capped while you learn.

3. Write the decision down before you start

This step costs nothing and does most of the work. Before the test runs, write three lines and share them with whoever owns the budget:

• The result that cuts the channel.
• The result that keeps it.
• The loss at which you stop early.

Why first? Once the numbers arrive, everyone reads them in their own favour. The channel owner finds a reason to keep it. Finance finds a reason to cut it. A rule written in advance settles the argument before it starts.

Measure total sales, or your main business number, in each group. Leave the platform reports out of it. They count clicks and assign credit. The test measures what changed.

💡 Key Takeaway: Agree on the result that cuts the channel before anyone sees the result.

4. Run long enough, then keep watching

Short tests flatter the pause. Some people who saw your ads last month buy this month. GeoLift’s guidance is at least 15 days on daily data, or four to six weeks on weekly data. The test should also cover at least one full purchase cycle.

Then keep reading after the channel comes back on. In one testing firm’s Cyber Week data, 41% of the extra value from ads showed up after the test window closed. Judge too early and you undercount the channel.

One test is also one season. The brand in the brand-search case ran its test three times: in the slow season, at peak, and in new markets. The lift was 1%, then nothing, then nothing. Only then did it switch brand search off and move the money.

5. Know what a pause cannot tell you

A blunt test gives blunt answers, and those have limits.

It cannot see small lifts. If a channel adds 2% to sales, a simple test will likely miss it. A flat result from a small test tells you the effect is too small to see at that scale. So cut in steps, and test again at the new level.

It can also come back loud. Some channels carry real weight. Google’s own pause studies found that when advertisers paused search ads, an average of 89% of paid clicks were lost, and organic clicks did not make up the gap. That is Google measuring Google, and it counted clicks rather than sales. Still, take the warning. A review of 640 experiments found Meta ads lifted brands’ main business number by about 19% on average.

So the pause cuts both ways. You find out which way before the budget meeting, instead of during it.

💡 Key Takeaway: A pause can clear a channel or convict it. Run it before you know which.

Final Thoughts: attribution shows who took the credit, and a pause shows what caused the sale

When I switched that channel off, nothing in the dashboard had warned me. Every report said it was working. The pause was the first time anyone asked the question directly.

You do not need a model, a vendor or a quarter of a data scientist’s time to ask it. You need one big line, a clean split, a rule written down first, and the patience to read the result properly.

Start with the channel you would be most nervous to switch off. That nervousness is usually a sign nobody has tested it.

If this was useful, follow me on LinkedIn for more on running growth as one system.

---

A note before you close this tab. Most budget arguments are measurement arguments in disguise. A pause test settles them with evidence instead of opinion.

Mervyn Chua is a Singapore-based growth leader who writes about running growth as one system. Follow him on LinkedIn.